got Exception in thread "main" java.io.IOException: No FileSystem for scheme: C

[justincui]

As a USER, I want to user pixiedust in jupyter of windows 10 so that I got the “exception in thread "main" java.io.IOException: No FileSystem for scheme: C”。

Expected behavior

jupyter kernel pythonwithpixiedustspark24 should be able handle path: C:/spark-2.4.4-bin-hadoop2.7/python/pyspark/shell.py

Actual behavior

Exception in thread "main" java.io.IOException: No FileSystem for scheme: C
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
.......
[IPKernelApp] WARNING | Unknown error in handling PYTHONSTARTUP file C:/spark-2.4.4-bin-hadoop2.7/python/pyspark/shell.py:

Steps to reproduce the behavior

1. start from command line: jupyter lab
2. create a new ipynb page, input following:
   from pyspark import SparkContext
   sc = SparkContext("local", "First App")
3. run them. You may see the exception printed in the console of jupyter
4. switch to original kernel "Python3", these two lines can be run without error.

environment

Windows 10 (only one hard disk C:)
Anaconda python (3.7.3)
spark 2.44 with prebuilt hadoop 2.7 (winutil.exe included)
scala 2.11.8

tried but not working:

set windows enviornment variable:
SPARK_HOME=C:/spark-2.4.4-bin-hadoop2.7
change line 512 of createKernel.py as:
"PYTHONSTARTUP": "file:///{0}/python/pyspark/shell.py".format(self.spark_home),