Exam Certified Data Engineer Professional topic 1 question 110 discussion

"hundreds of pipelines with parallel updates of many tables" indicates updating many tables concurrently via many pipelines. A is the best solution for that. B is the answer for updating a few large tables with few partitions.

"Which of the following solutions" I'm sure this is a question with multichoice. A and B options are correct together.

High volume and high-velocity data ingestion often becomes a bottleneck due to limited write parallelism. By partitioning ingestion tables based on small time durations (e.g., hourly or even minutes), you create many smaller partitions. This allows concurrent writes to different partitions, significantly increasing the overall throughput of your data ingestion.

Since multiple pipelines are being used high concurrency cluster would give maximum resource utilization.

A. B is only useful to improve performance of large tables ingestions.

Why not D?

Both options A and B could be relevant depending on the specific details of the use case. If the emphasis is on optimizing concurrent queries and overall data throughput, option A might be more appropriate. If the primary concern is parallel updates of tables with high-volume, high-velocity data, option B is a more targeted approach.

The best way to deal with high volume and high velocity data is to use partitioning

Databricks High Concurrency cluster

1) Partitioning by Time: Partitioning tables by a small time duration allows for efficient parallelism in data writes. Each time partition can be processed independently, enabling parallel updates to multiple partitions concurrently. 2)Optimizing for Parallelism: By partitioning the tables based on time, data can be ingested and processed in parallel, providing the ability to handle high volume and high velocity data effectively. Regarding option A, Databricks High Concurrency clusters are more focused on supporting a large number of concurrent users, which might not directly address the requirement for parallel updates of many tables with extremely high volume and high velocity data

High Concurrency clusters in Databricks are designed for multiple concurrent users and workloads. They provide fine-grained sharing of cluster resources and are optimized for operations such as running multiple parallel queries and updates. This would be suitable for a solution that involves many pipelines with parallel updates, especially with high volume and high velocity data.