Exam Professional Data Engineer topic 1 question 303 discussion

Should be A. Curated zone need Parquet, Avro, ORC format not CSV or JSON. Check the ref - https://cloud.google.com/dataplex/docs/add-zone#curated-zones

- Auto-Discovery Feature: Dataplex has an auto-discovery feature that, when enabled, automatically discovers and catalogs data assets within a zone. - Appropriate for Both Raw and Curated Zones: This feature is applicable to both raw and curated zones, and it should be tailored to the specific data governance and cataloging needs of the organization.

- Raw zones store structured data, semi-structured data such as CSV files and JSON files, and unstructured data in any format from external sources. Raw zones are useful for staging raw data before performing any transformations. Data can be stored in Cloud Storage buckets or BigQuery datasets. - Curated Zones do not support JSON / CSV

Auto-discovery needs to be enabled for the curated zone to ensure that Dataplex can scan and register the files. You can configure this setting at the zone or asset level. Option A, moving the JSON and CSV files to the raw zone, would not solve the issue of automatic discovery in the curated zone. The problem lies in the configuration of the curated zone, not the location of the files.

Raw zones store structured data, semi-structured data such as CSV files and JSON files, and unstructured data in any format from external sources. Curated zones store structured data. Data can be stored in Cloud Storage buckets or BigQuery datasets. Supported formats for Cloud Storage buckets include Parquet, Avro, and ORC.

Discovery configuration Discovery is enabled by default when you create a new zone or asset. You can disable Discovery at the zone or asset level. For each Dataplex asset with Discovery enabled, Dataplex does the following: Scans the data associated with the asset. Groups structured and semi-structured files into tables. Collects technical metadata, such as table name, schema, and partition definition. For unstructured data, such as images and videos, Dataplex Discovery automatically detects and registers groups of files sharing media type as filesets. For example, if gs://images/group1 contains GIF images, and gs://images/group2 contains JPEG images, Dataplex Discovery detects and registers two filesets. For structured data, such as Avro, Discovery detects files only if they are located in folders that contain the same data format and schema. Reference : https://cloud.google.com/dataplex/docs/discover-data#exclude-files-from-Discovery

While JSON and CSV can technically be stored in curated zones, it is not a common practice due to the reasons mentioned above. no where in the mention link its mention that there is a restriction.

While none of the original options (A, B, C, or D) directly address the issue, the closest solution is: Move the JSON and CSV files to a raw zone. (This was previously marked as the most voted option, but it's not ideal due to data organization disruption) Here's why this approach might be necessary (but not ideal): Dataplex curated zones currently don't support native processing of JSON and CSV formats. They are designed for structured data formats like Parquet, Avro, or ORC.

Option A https://cloud.google.com/dataplex/docs/add-zone#raw-zones Raw zones are the only zones that support CSV & JSON

Its B guys, i encounter this in my job, and I had to do B to make it work

GCP001 agree with him

The answer can be found reading a common config of Dataplex in this URL: https://medium.com/google-cloud/google-cloud-dataplex-part-1-lakes-zones-assets-and-discovery-5f288486cb2f

Dataplex does not allow users to create CSV files within a “curated zone”

According to this URL: https://cloud.google.com/dataplex/docs/discover-data, the auto-discovery can support CSV and Json in both Raw-Zone and Curated-Zone. I also open a console the verify it, both Raw and Curated zone can set up csv&json auto-discovery.

Discovery raises the following administrator actions whenever data-related issues are detected during scans : Inconsistent data format in a table. For example, files of different formats exist with the same table prefix. Inconsistent data format in a table. For example, files of different formats exist with the same table prefix.

I'd go for Option B, auto-discovery is enabled by default for any zone, including curated ones, so if a file is not automatically discovered it's due to the disabled auto-discovery

I will go with A, check the ref. Curated zones only store Parquet, Avro, and ORC in CS, and well-defined schema and Hive-style partitions in the BigQuery: https://cloud.google.com/dataplex/docs/add-zone#curated-zones