Addresses issue of assembly.zip contents not unzipped to current directory, preventing UDF failures by default. #622
Conversation
…bly or UDF's fail with serialization errors Code change to unzip contents of assembly.zip to current directory instead.
| //val driverDir = new File(workingDir, FilenameUtils.getBaseName(zipFileUri.getPath())) | ||
| // ZZ: By unzipping into the working directory, this ensures currentdir is the dir containing the files that is needed to properly handle UDF's, without the need to set ASSEMBLY_PATHS | ||
| // The alternative here would be to change current working directory to the subdirectory path, but that seems like it could have more unintended side effects | ||
| val driverDir = workingDir |
There was a problem hiding this comment.
I'm curious how extracting the zip file to the driver's current directory will allow the Microsoft.Spark.Worker to locate the udf assemblies.
There was a problem hiding this comment.
Is it expected that the working directory of the driver is accessible by the Microsoft.Spark.Worker that may be running on separate nodes in the cluster ?
There was a problem hiding this comment.
well, the assumption I have is that the assembly.zip bundle in question would have the assemblies containing the UDF's. That's how we're using it. And having that in the working directory empirically works, where as if the assembly is unzipped to a subdirectory from the working directory, i.e. assembly/* does not (causes serialization errors). I don't know whether that's by intention or caused by some other side effect, but this is what we've observed.
The issue is that, if the assemblies are not in the same directory in which you're launch DotnetRunner, then UDF's don't get picked up (unless it's in the ASSEMBLY_SEARCH_PATH, which we won't be able to set conveniently). But if it's unzipped to the working directory for DotnetRunner, then the SEARCH PATH won't need to be set at all.
Hope that makes sense?
There was a problem hiding this comment.
AssemblyLoader.cs will set up the paths for the Worker to search for UDF assemblies. Unless the working directory of the Worker is set to the driver's working directory and is also accessible by the worker, then I don't know if this would be the proper fix. Have you been able to test this in cluster mode and/or in other big data platforms like HDInsight/Databricks ?
There was a problem hiding this comment.
@zzhu-bh have you been able to test this PR in a spark cluster ?
There was a problem hiding this comment.
Thanks for the follow up @suhsteve! Sorry, we have not been able to work on this, our teams have been putting out some fires. I'll see if I can get someone on the team to follow up on this issue. Thank you for your patience.
…bly or UDF's fail with serialization errors
Code change to unzip contents of assembly.zip to current directory instead.
We are excited to review your PR.
So we can do the best job, please check:
Fixes #nnnnin your description to cause GitHub to automatically close the issue(s) when your PR is merged.