Exam Professional Data Engineer topic 1 question 84 discussion

C is the only way which all records will be compared.

Answer: C Description: Full comparison with this option, rest are comparison on sample which doesnot ensure all the data will be ok

Hash is always a good idea to compare the data

In practice, I will do B. That means it may have error due to randomness. But that is how we normally do validation/QA in general, i.e. we test random samples In this question, I will do C.

key words here- Hash or collect value on "EACH table", after sorting the table. Option C

C. Use a Dataproc cluster and the BigQuery Hadoop connector to read the data from each table and calculate a hash from non-timestamp columns of the table after sorting. Compare the hashes of each table. This approach will ensure that the data is read in a consistent order, and the hash function will provide a quick and efficient way to compare the contents of the tables and ensure that they are identical.

C is the answer.

All records

B is the only way which all records will be compared.

HASH() to compare data skipping dates and timestamps

options A B and D only will determine that it “might” be identical since is only a sample. HASH() can be helpful when doing bulk comparisons, but you still have to compare field by field to get the final answer. The only one left is C which looks good to me

C. The rest use RAND() at some point, which makes it hard to compare for consistency, unless there's a 'seed' option, which wasn't mentioned. So C.

Since there is no PK and it is possible that set of values is commons in some records which result in same hashkey for those records. But still Anwer is C

Vote for 'C"

B: Because said migrated to BigQuery, then we don't need Dataproc, and samples don't mean you don't compare all of data.

C Using Cloud Storage with big data Cloud Storage is a key part of storing and working with Big Data on Google Cloud. Examples include: Loading data into BigQuery. Using Dataproc, which automatically installs the HDFS-compatible Cloud Storage connector, enabling the use of Cloud Storage buckets in parallel with HDFS. Using a bucket to hold staging files and temporary data for Dataflow pipelines. For Dataflow, a Cloud Storage bucket is required. For BigQuery and Dataproc, using a Cloud Storage bucket is optional but recommended. gsutil is a command-line tool that enables you to work with Cloud Storage buckets and objects easily and robustly, in particular in big data scenarios. For example, with gsutil you can copy many files in parallel with a single command, copy large files efficiently, calculate checksums on your data, and measure performance from your local computer to Cloud Storage.

C is correct