Fix
Managed Service for Apache Spark (formerly Dataproc on Compute Engine): A bug has been fixed to add-back py4j library in image version 2.1.120.
Managed Service for Apache Spark (formerly Dataproc on Compute Engine): A bug has been fixed to add-back py4j library in image version 2.1.120.
Read the original announcement