Free Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Actual Exam Questions - Question 4 Discussion

It’s D because using an iterator UDF means the model loads once per partition and stays loaded across all batches, cutting down repeated reloads unlike the other options.

It’s D because using an iterator UDF lets you initialize the model once per batch instead of on every single call, which really cuts down the loading overhead compared to the other options.

D, since iterator UDFs keep the model loaded across batches, unlike the others.

D imo, since iterator UDFs let you load the model once per batch instead of per row, cutting down reloads big time. The other options don’t batch process as efficiently.

A/B? Changing to a PySpark UDF (A) might not fix the issue since it could still load the model on every call. Switching to a Series → Scalar UDF (B) doesn’t really help either because the model would still reload per row. The key is minimizing model loading frequency, which neither A nor B directly address. So these seem less likely to improve performance compared to options that batch process like C or D.

D imo, because using the iterator UDF means the model gets loaded once per batch, not on every single row. That’s a huge performance boost compared to the current setup.

D imo, because an iterator UDF processes batches as chunks, so you load the model once per batch instead of once per row. A and B don’t really help since they still process data row-wise, causing repeated loads. C is closer but mapInPandas is more manual and a bit heavier to maintain compared to the iterator approach in D, which is designed exactly for this use case—efficient model loading and batch processing.

C/D? I see why D is better for loading once per batch, but C also lets you control model loading outside the row processing. Either way beats loading every row like the original.

C/D? I get why D is popular since iterator UDFs let you load the model once per batch, but mapInPandas (C) also processes data in batches and lets you handle the model loading outside the row-wise logic. Both reduce the overhead compared to loading per row. The main point is moving from per-row loading to per-batch or per-partition loading, which both C and D enable. A and B won't help here since they don't address how often the model loads. So between C and D, it’s about which approach fits better with your pipeline setup.

It’s D for sure. Using an iterator UDF means the model is loaded once per batch of data, not once per row like in the original code. That cuts down the overhead drastically. Plus, it avoids reloading the model multiple times within the same task execution, which is exactly what’s slowing things down here. Options A and C don’t directly address the load frequency, and B still loads per element, so they’re not as effective.

B tbh, changing to a Series → Scalar UDF means the model loads once per row still, so no big gain. The real win is loading once per batch or partition, not per element.

Option D, since loading the model once per partition is way better than every row.

D imo, the main issue is loading the model every time the function runs. Using an Iterator[Series] → Iterator[Series] UDF allows you to load the model once per partition instead of every row or batch, which should cut down on overhead a lot. The other options don’t really solve that repeated loading problem directly. PySpark UDFs could be slower, and changing input/output types won’t fix model instantiation frequency by itself. MapInPandas is nice but doesn’t inherently solve this unless you explicitly cache the model somewhere, which isn’t shown here.