Exam Professional Machine Learning Engineer topic 1 question 177 discussion

B) The preprocessing step is already complete and its output is stored is in GCS, so a separate, smaller pipeline just for training is the most efficient solution. *A) Conditional logic still performs prepocessing steps when the logic points to not skipping it, increasing costs; C) While reducing preprocessing time, this solution would increase this step's cost; D) Would still include unnecessary preprocessing for each algorithm test before it's cached.

Caching Preprocessed Data: Since the preprocessed data (10 TB) is the same for different model training runs, enabling caching allows Vertex AI to reuse it for subsequent pipeline executions. This significantly reduces execution time and cost, especially for large datasets. Disabling Model Training Cache: Model training is typically non-deterministic due to factors like random initialization. Caching the model training step could lead to stale models and inaccurate results. Disabling caching ensures the model is re-trained each time with potentially updated code for different algorithms.

agree with guilhermebutzke

According to this documentation cited: https://cloud.google.com/vertex-ai/docs/pipelines/configure-caching it is possible to write a pipeline setting True or False for each task component, like this: # Model training step with caching disabled train_model_task = train_model_op() train_model_task.set_caching_options(False) # Disable caching for this step # Model training step depends on the preprocessing step train_model_task.after(preprocess_task) So, with this, letter D is the best option. Furthermore, letter A and, Adding a pipeline parameter and an additional pipeline step introduces unnecessary complexity when caching can handle conditional execution efficiently and in letter C, configuring a machine with more CPU and RAM for preprocessing does not address the goal of minimizing pipeline changes and reducing execution time/cost effectively.

Not A. Adding a pipeline parameter and new pipeline steps does not minimise pipeline changes. Not C. The idea is not to re-run the preprocessing step at all. Not B. Creating a whole new pipeline implies a significant investment of effort. I opt for D: Enabling caching only for preprocessing job (although it says “pipeline job” in the option, I think that is a typo). Quoting Vertex AI docs: “If there is a matching execution in Vertex ML Metadata, the outputs of that execution are used and the step is skipped. This helps to reduce costs by skipping computations that were completed in a previous pipeline run.” https://cloud.google.com/vertex-ai/docs/pipelines/configure-caching

The pipeline already generates the preprocessed dataset and stores, there's no need to preprocess again for another model