Free Google Professional-Machine-Learning-Engineer Actual Exam Questions - Question 14 Discussion
different training strategies. First, you trained the model using a single GPU, but the training process
was too slow. Next, you distributed the training across 4 GPUs using tf.distribute.MirroredStrategy
(with no other changes), but you did not observe a decrease in training time. What should you do?
D imo, increasing batch size usually helps utilize multiple GPUs better.
I’m thinking option A might be worth reconsidering. If the dataset isn’t properly sharded across the GPUs, each GPU could be doing the same work, so no speedup happens. Distributing the dataset explicitly makes sure each GPU processes a different slice. Just switching to MirroredStrategy alone doesn’t automatically handle how data is fed. Could that explain why training time stayed the same? What if they did increase batch size but didn’t shard the data? Would that still cause no improvement?
A The question says they used MirroredStrategy but doesn't mention distributing the dataset explicitly. Without using experimental_distribute_dataset to shard the data across GPUs, each GPU might be redundantly processing the full dataset, causing no speedup. So, distributing the dataset properly is key before worrying about batch size or TPUs.
Option D, increasing the batch size, makes a lot of sense here. Smaller batches per GPU can cause overhead to dominate, so bigger batches help GPUs stay busy and speed up training.
A Using tf.distribute.Strategy.experimental_distribute_dataset ensures the data is properly split across GPUs; without this, the GPUs might be waiting on the same data, killing speed gains.
A/D? Proper dataset distribution matters, but bigger batch sizes usually help more with multi-GPU speedup.
D imo, increasing batch size is key to better GPU utilization here.
A/B? If the dataset isn't properly distributed, the GPUs might just be waiting around. But also, a custom training loop (B) gives more control over syncing and might help optimize performance beyond defaults.
It’s D for me. Even if you’re using MirroredStrategy, if the batch size stays the same as single-GPU training, each GPU ends up doing less work and the overhead of syncing can kill any speedup. Increasing batch size lets you feed all GPUs properly and really speed things up. A is important too, but assuming you’ve already got the dataset distribution right, the batch size is the main bottleneck here.
A/D? Distributing the dataset properly (A) is crucial so each GPU gets its own data chunk. Once that’s set, bumping batch size (D) can improve GPU utilization and speed up training further.
A, without distributing the dataset, each GPU does the same work, so no speedup.
A, the dataset might not be split correctly, so GPUs aren't utilized fully.
Not distributing the dataset properly could mean all GPUs are doing the same work, so no speedup. But increasing batch size (D) might also be necessary to keep GPUs busy enough. Could it be that both A and D matter here?
Maybe D here. Increasing batch size often helps better utilize multiple GPUs since small batches can bottleneck parallelism, so training won’t speed up without a bigger batch.
Option A makes sense since distributing the dataset ensures each GPU gets unique data batches.
B imo, creating a custom training loop lets you control exactly how data and gradients flow across GPUs. Just switching strategy without adjusting training code might not speed things up.
It’s A for me. Without explicitly distributing the dataset, the MirroredStrategy won't automatically split the data across GPUs, so each GPU could still be processing the full dataset redundantly. That would explain why training time didn't improve. Increasing batch size (D) helps but won’t fix the root problem if the data isn’t properly distributed. Setting up experimental_distribute_dataset ensures each GPU gets its own slice, which should speed things up without needing to change your batch size or switch hardware.
A/D? If the dataset isn’t properly distributed, the GPUs won’t be utilized fully. Also, increasing batch size (D) often helps make multi-GPU training more efficient. Either could fix the issue.
Sounds like the batch size might be too small, so D.