Exam Certified Data Engineer Professional topic 1 question 120 discussion

The correct answer is B. Schedule a job to execute the pipeline once an hour on a new job cluster. Explanation: In this scenario, the business reporting team needs the data to be updated every hour, and the processing time for the pipeline takes 10 minutes. To meet this requirement with the lowest cost, the best option is to schedule the job to run once per hour using a new job cluster. A job cluster is created specifically for the duration of the job, and once the job finishes, the cluster is terminated. This is cost-efficient because resources are only consumed while the job is running, and the cluster does not stay active when it is not needed.