JAVA_GATEWAY_EXITED: one error, two root causes

If you run PySpark on Windows for long enough, you will meet this:

PySparkRuntimeError: [JAVA_GATEWAY_EXITED] Java gateway process exited
before sending its port number

It is one of the least helpful errors I know. It means the driver JVM died before it could hand Python its gateway port, and that is all it means. It never tells you why the JVM died. In a single week I hit it twice, for two causes that had nothing to do with each other.

What the error actually says

PySpark starts the Spark driver as a child JVM process, then talks to it over a Py4J gateway. If that JVM exits before publishing its port, Py4J gives up and raises JAVA_GATEWAY_EXITED. So read it literally: "the JVM did not come up." The real cause is always somewhere upstream of Spark itself.

Cause 1: the VS Code kernel had no JAVA_HOME

The exact same builder code worked every time from a plain CLI python, and failed every time inside the VS Code notebook. That contrast was the whole clue.

The VS Code kernel is spawned by the editor, and its environment does not inherit my shell profile. No JAVA_HOME, no java on PATH. Spark's launcher could not find Java, so the driver JVM never started. CLI runs worked only because they picked up JAVA_HOME from my user profile.

The fix is to force a known-good environment in the notebook before importing or using pyspark:

import os, pyspark
os.environ["JAVA_HOME"]  = r"C:\Program Files\Java\jdk-17"
os.environ["SPARK_HOME"] = os.path.dirname(pyspark.__file__)
os.environ["PATH"] = r"C:\Program Files\Java\jdk-17\bin;" + os.environ.get("PATH", "")

Lesson: do not assume a GUI-launched kernel inherits your shell environment. Set JAVA_HOME and SPARK_HOME explicitly. And when you want to reproduce an environment-specific failure, truly unset the variable (env -u JAVA_HOME) rather than just overriding PATH, or you will leave other inherited values in place and chase a ghost.

Cause 2: a SAS token with an ampersand broke spark-submit.cmd

Java was sorted, and it still failed. This time java -version and spark-submit --version both returned cleanly, so the environment was genuinely fine. The JVM was dying for a different reason.

The culprit was how I passed my ADLS SAS token. I had set it as a builder .config() value. On Windows, PySpark forwards every .config() entry to spark-submit.cmd as --conf key=value, and spark-submit.cmd is a batch file running under cmd.exe. A SAS token looks like se=...&sp=rl&sig=..., and in cmd.exe the ampersand is a command separator. The launch command got split apart, spark-submit received garbage, and the JVM never started.

I reproduced it exactly: a config value of se=2026&sp=rl&sig=ab%2Bcd produced 'sp' is not recognized as an internal or external command, followed by JAVA_GATEWAY_EXITED.

The fix is to keep the token off the command line entirely. Create the session first, then set the fs.azure.* values directly on the Hadoop configuration:

spark = SparkSession.builder.config("spark.jars.packages", "...").getOrCreate()
hconf = spark._jsc.hadoopConfiguration()
hconf.set(f"fs.azure.account.auth.type.{ACCOUNT}.dfs.core.windows.net", "SAS")
hconf.set(f"fs.azure.sas.fixed.token.{ACCOUNT}.dfs.core.windows.net", SAS_TOKEN)

Lesson: on Windows, builder configs travel through cmd.exe, so any value holding a shell-special character (& | < > ^ %) can corrupt the launch and surface as the same generic gateway error. Keep secrets, tokens, and URLs out of .config(). Set them on spark._jsc.hadoopConfiguration() after the session is already running.

The takeaway

When you see JAVA_GATEWAY_EXITED, stop treating it as a Spark problem. Treat it as "the driver JVM did not launch," and work backwards. Check the environment first (JAVA_HOME above all), then check whether any config value you are passing contains characters the shell will mangle. The message is identical in both cases, but the causes could not be further apart.