Free Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Actual Exam Questions - Question 10 Discussion
failures or intentional shutdowns by continuing where the pipeline left off.
How can this be achieved?
D, checkpointLocation in writeStream saves offsets and state for recovery.
I think A can be ruled out since checkpointLocation in readStream doesn’t store progress info needed for recovery. But is there any config beyond checkpointLocation in writeStream for handling metadata or query state?
D vs A? CheckpointLocation is definitely used with writeStream to save offsets and state for recovery. Setting it in readStream doesn’t really help with fault tolerance, so D makes more sense here.
A imo, the key to recovery in Structured Streaming is checkpointing the state and offsets, which happens on the sink side during writeStream. So setting checkpointLocation there (option D) makes the most sense because that's where Spark keeps track of progress for recovery. The recoveryLocation options mentioned don’t exist as far as I know, and checkpointLocation in readStream doesn’t handle state or offset tracking for recovery. So it’s definitely about configuring checkpointLocation in writeStream, confirming that D is the right call here.
Guessing D since checkpointLocation for writeStream handles offsets and state for recovery.
A/D? I know checkpointLocation is key, but it needs to be set during writeStream, not readStream. So A can be ruled out even though it mentions checkpointLocation. D fits better.
Option D seems right to me-checkpointLocation during writeStream ensures the streaming job can pick up where it left off after a failure. The recoveryLocation options sound made up or at least not standard.