Which statement regarding Spark configuration on the Databricks platform is true?
A.
The Databricks REST API can be used to modify the Spark configuration properties for an interactive cluster without interrupting jobs currently running on the cluster.
B.
Spark configurations set within a notebook will affect all SparkSessions attached to the same interactive cluster.
C.
Spark configuration properties can only be set for an interactive cluster by creating a global init script.
D.
Spark configuration properties set for an interactive cluster with the Clusters UI will impact all notebooks attached to that cluster.
E.
When the same Spark configuration property is set for an interactive cluster and a notebook attached to that cluster, the notebook setting will always be ignored.
I tried it myself, setting a spark conf on the cluster ui, will impact all notebooks attached to that cluster, for example i set the number of shuffle partitions to 4, and in every notebook when i inspect the number of partitions i find 4.
A. Incorrect: Modifying configurations through the Databricks REST API while jobs are running can lead to unexpected behavior or disruption. It's generally not recommended.
C. Incorrect: While global init scripts can be used, it's not the only way. Configurations can also be set within notebooks.
D. Incorrect: Configurations set through the Clusters UI apply to the entire cluster, but they might not necessarily override configurations set within notebooks attached to the cluster.
E. Incorrect: Notebook configurations can take precedence over cluster-level configurations for the same property, offering finer-grained control at the notebook level.
In Databricks, you can use the Databricks REST API to modify Spark configuration properties for an interactive cluster without interrupting currently running jobs. This allows you to dynamically adjust Spark configurations to optimize performance or meet specific requirements without the need to restart the cluster.
If you update the configuration of a cluster using the Databricks REST API or the Clusters UI while the cluster is in a RUNNING state, the cluster will be restarted to apply the new configuration. However, Databricks typically handles this situation in a way that minimizes disruption to running jobs.
upvoted 2 times
...
...
Log in to ExamTopics
Sign in:
Community vote distribution
A (35%)
C (25%)
B (20%)
Other
Most Voted
A voting comment increases the vote count for the chosen answer by one.
Upvoting a comment with a selected answer will also increase the vote count towards that answer by one.
So if you see a comment that you already agree with, you can upvote it instead of posting a new comment.
hamzaKhribi
Highly Voted 11 months, 4 weeks agoCurious76
Most Recent 9 months agovctrhugo
9 months, 3 weeks agoalexvno
11 months, 1 week agopetrv
12 months agoalexvno
11 months, 1 week agopetrv
12 months ago