Exam Professional Data Engineer topic 1 question 17 discussion

D is correct because it uses managed services, and also allows for the data to persist on GCS beyond the life of the cluster. A is not correct because the goal is to re-use their Hadoop jobs and MapReduce and/or Spark jobs cannot simply be moved to Dataflow. B is not correct because the goal is to persist the data beyond the life of the ephemeral clusters, and if HDFS is used as the primary attached storage mechanism, it will also disappear at the end of the cluster’s life. C is not correct because the goal is to use managed services as much as possible, and this is the opposite. E is not correct because the goal is to use managed services as much as possible, and this is the opposite.

The correct answer is D. Here is the explanation to why Data proc and why not Data flow. When a company wants to move their existing Hadoop jobs on premise to cloud, we can simply move the jobs in cloud data prod and replace hdfs with gs:// which is google storage. This way you are keeping compute and storage separately. Hence the correct answer is D. However, if the company wants to complete create a new jobs and don’t want to use the existing Hadoop jobs running on premise, the option is to create new data flow jobs.

D. Create a Cloud Dataproc cluster that uses the Google Cloud Storage connector. Dataproc clusters can be created to lift and shift existing Hadoop jobs Data stored in Google Cloud Storage extends beyond the life of a Dataproc cluster.

D. Create a Cloud Dataproc cluster that uses the Google Cloud Storage connector. Here's why: Cloud Dataproc allows you to run Apache Hadoop jobs with minimal management. It is a managed Hadoop service. Using the Google Cloud Storage (GCS) connector, Dataproc can access data stored in GCS, which allows data persistence beyond the life of the cluster. This means that even if the cluster is deleted, the data in GCS remains intact. Moreover, using GCS is often cheaper and more durable than using HDFS on persistent disks.

D. Create a Cloud Dataproc cluster that uses the Google Cloud Storage connector. Google Cloud Dataproc is a managed Hadoop and Spark service that allows you to easily create and manage Hadoop clusters in the cloud. By using the Google Cloud Storage connector, you can persist data in Google Cloud Storage, which provides durable storage beyond the cluster's lifecycle. This approach ensures data is retained even if the cluster is terminated, and it allows you to reuse your existing Hadoop jobs. Option B (Creating a Dataproc cluster that uses persistent disks for HDFS) is another valid choice. However, using Google Cloud Storage for data storage and processing is often more cost-effective and scalable, especially when migrating to the cloud. Options A, C, and E do not take full advantage of Google Cloud's services and the benefits of cloud-native data storage and processing with Google Cloud Storage and Dataproc.

Option D is incorrect, as it would not provide persistent HDFS storage within cluster itself. Rather B should be the correct answer.

Correct D

Hadoop --> Dataproc Persistent storage after the processing --> GCS

D Seems right. Cloud storage can be used to achieve data storage even after the life of cluster.

The answer is D! Dataproc have no need for use to manage the infra and cloudstorage also no need for us to manage too!

D. Create a Cloud Dataproc cluster that uses the Google Cloud Storage connector.

Seems like it is D. https://cloud.google.com/dataproc/docs/concepts/dataproc-hdfs Never saw they mentioned persistent disks, although they are not deleted with the clusters...

Correct Answer : D

Dataproc cluster set up will be ephemeral to run HDFS Jobs and can be killed after Job execution killing persistent storage with cluster

Anwer: D

Isn't it A and D both dataflow for reusable jobs and gcs for data peraistance?

Two key points: Managed hadoop cluster - dataproc Persistent storage: GCS (dataproc uses gcs connector to connect to gcs)