Abhishek.
All posts
4 min read

Getting Spark to read and write files on Windows

sparkpysparkwindowsazureadls

Most of my Spark work runs on cloud clusters, but I keep a local Windows setup for fast iteration. Getting it to reliably touch the filesystem, local and ADLS alike, took a series of fixes that had nothing to do with my actual data or code. Here is the log, cleaned up.

You need hadoop.dll, not just winutils.exe

Writing parquet failed at commit time with:

java.lang.UnsatisfiedLinkError: 'boolean
org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(...)'

I already had HADOOP_HOME=C:\hadoop set and winutils.exe in place, so I assumed the native side was covered. It was not. Hadoop also needs the native library hadoop.dll to touch the local filesystem on commit, and that file was missing from both C:\hadoop\bin and System32. winutils.exe alone was enough to create the output folder, which is exactly why the write got partway before failing.

The fix was to drop a version-matched hadoop.dll in both locations:

  • C:\hadoop\bin\hadoop.dll, next to winutils.exe
  • C:\Windows\System32\hadoop.dll

Then restart the kernel, because native libraries load once at JVM startup. Match the DLL to the Hadoop version Spark actually bundles (check pyspark/jars/hadoop-client-api-*.jar). A mismatched DLL can crash the JVM outright instead of giving you a clean error.

Pin spark.local.dir so the temp cleaner cannot break you

Next, reads started failing during schema inference:

java.nio.file.NoSuchFileException:
...\AppData\Local\Temp\blockmgr-<uuid>\...

The data was fine. All 200 parquet files and the _SUCCESS marker were present. The problem was Spark's block-manager scratch directory, which lives under the Windows temp folder by default. After a few crashed and restarted sessions, that directory went stale, so the broadcast write during parallel schema inference had nowhere to land.

Fix: give Spark a stable, short scratch path of its own.

.config("spark.local.dir", r"C:\spark-tmp")   # create C:\spark-tmp first

Then restart the kernel, since this only applies at session creation. On Windows I now set this for every session so the OS temp cleaner and old sessions cannot pull the rug out.

ConnectionRefused from Py4J means the JVM is already gone

Sometimes the only thing I saw was this, raised from getOrCreate() and even from spark.stop():

ConnectionRefusedError: [WinError 10061] ... target machine actively refused it

This one is a trap. A previous job had crashed the Spark JVM, but the Python kernel still held stale references to it. The next call tried to attach to a dead gateway and got connection refused. It is a secondary error sitting on top of the real one. It hides the true failure so well that even rendering the original Java exception fails, because that also round-trips to the dead JVM.

Fix: restart the kernel. The dead references live in the Python process and cannot be cleared in place. A fresh kernel gives you a fresh JVM. Then re-run at small scale to surface the real Java error underneath.

ADLS SAS auth: two gotchas with FixedSASTokenProvider

Once local files worked, I wired up ADLS over abfss:// with a SAS token, and hit two separate issues.

First, the class path. Pointing the provider config at org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider throws ClassNotFoundException, because the class actually lives in the services package. Confirm it against the real jar rather than trusting memory:

jar tf hadoop-azure-3.4.1.jar | grep FixedSASTokenProvider
org/apache/hadoop/fs/azurebfs/services/FixedSASTokenProvider.class

Second, and more subtle: for a fixed token, do not set fs.azure.sas.token.provider.type at all. Hadoop's generic provider path instantiates whatever class you name via reflection, which requires a public no-arg constructor. In hadoop-azure 3.4.1 the class only has a (String) constructor, so reflection fails with NoSuchMethodException: FixedSASTokenProvider.<init>().

The working setup is to provide only the auth type and the token, and let Hadoop construct its internal holder itself:

hconf.set(f"fs.azure.account.auth.type.{ACCOUNT}.dfs.core.windows.net", "SAS")
hconf.set(f"fs.azure.sas.fixed.token.{ACCOUNT}.dfs.core.windows.net", SAS_TOKEN)

When a fixed token is present and no provider class is configured, Hadoop calls the (String) constructor internally. FixedSASTokenProvider is Hadoop's own fixed-token holder, not a user-pluggable provider, so naming it in provider.type is what breaks it. Reserve that setting for custom providers that genuinely expose a no-arg constructor.

The takeaway

None of these were logic bugs. They were environment papercuts: a missing DLL, a scratch directory, a stale process, a class in the wrong package. Local Spark on Windows is mostly a matter of getting the environment right once and writing down what you learned, which is exactly what this post is.