Configuration options for Spark to work with o3fs
After setting up o3fs, you can make configuration updates specific
to components such as Spark to ensure that they work with Ozone.
To run Spark jobs with o3fs on a secure Kerberos-enabled cluster, ensure
that you assign a valid URI by setting the value of the Spark Client Advanced
Configuration Snippet (Safety Valve) property for the
spark.conf or the spark-defaults.conf file through
the Cloudera Manager web UI.
For
example:
spark.yarn.access.hadoopFileSystems=o3fs://bucket1.vol1.securehost1.example.com:9862