Exam DP-203 topic 3 question 14 discussion

1. Partition by 2. GeographyRegionID, Year, Month, Day as the pipelines are per region this seems right choice 3. Parquet

I suggest storing the data in parquet

ChatGPT 4o Given the requirement to support daily incremental loads for each GeographyRegionID, the optimal partitioning strategy would be: Partition by (GeographyRegionID, Year, Month, Day) This strategy allows the system to efficiently access the data based on GeographyRegionID first, and then further narrows down the partitions based on Year, Month, and Day. This approach is particularly useful for managing and querying large datasets in a time-series fashion, which is typical for sensor data.

The second option given in the answer is correct which is Year, Month, Day, GeographyRegionId Why? The question states that the "solution must minimize storage cost". One of the way to do that is to have lesser number of folders getting created. Here is an example: Let's says we go with the option of Year, Month, Day, GeographyRegionId where we consider 1 year of data and say we have 10 different region. So the number of folders that would get created would be 1(Year) + 12(Month) + 365 (Days) + 3650 (No of days multiplied by 10 regions as each day's folder would contain 10 sub folders for 10 regions) which gives 3663 as count. If we go with the option of GeographyRegionId, Year, Month, Day then 1(Year) + 12(Month) + 365 (Days) i.e, 378 folders would be repeated 10 times inside 10 different region folder which gives total count as 3780(378*10) which is definitely higher than 3663.

1. Partition by 2. GeographyRegionID, Year, Month, Day 3. Parquet

1. Patition by 2.Year,Month, Day, GeographyRegionID (it said to minimize storage cost, not query performance. if GeographyRegionID goes first, each regionID will have repeated folders for different dates) 3. Parquet

https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-best-practices

This question is similar to #36Topic 1, if you reference that question, the answer should be 1. Partitioned By, 2. GeographyRegionID, Year, Month, Day, 3. Parquet

For 3.) if parquet with partitions, then it should "overwrite" mode instead of "append". Since, it is "append" mode, I think saveAsTable sis more appropriate.

Agree with Pallavi 1. Partition by 2. GeographyRegionID, Year, Month, Day 3. Parquet

Parquet is must (offer higher compression rates)- "The solution must minimize storage costs."

it is the same question on Topic 1 Question 36. Then 1. Partition by 2. GeographyRegionID, Year, Month, Day 3. Parquet

// the correct answer is df.write.partitionBy("GeographyRegionID").mode("append").parquet("/DBTBL1") // or df.write.partitionBy("GeographyRegionID","Year","Month","Day").mode("append").parquet("/DBTBL1") // Question says "minimize storage costs" so I would select the first one

I was wondering if the incremental load is supported for parquet, but since "append" mode is used, this should be alright. The question asks to minimize costs, so I go for parquet (not saveAsTable). partitionBy GeopgraphyRegionID, Year,Month,Day (pipelines per region; daily load) parquet

its recommend to use partitions first before Y/M/D so that they can be managed eazily such as assigning security, or processing by business unit such as zone/country/area etc., GeographyRegionId/Year/Month/Day and Paraquet are answers

Mes chers amis: 1.Sortby 2.GeographyRegionId, Year, Month, Day 3.Parquet

only reason for using .parquet is option seems to be dataset path not table else saveastable is right.