You need to design a solution that will process streaming data from an Azure Event Hub and output the data to Azure Data Lake Storage. The solution must ensure that analysts can interactively query the streaming data.
What should you use?
A.
Azure Stream Analytics and Azure Synapse notebooks
B.
Structured Streaming in Azure Databricks
C.
event triggers in Azure Data Factory
D.
Azure Queue storage and read-access geo-redundant storage (RA-GRS)
B. Structured Streaming in Azure Databricks is the best option for this scenario as it allows for processing of streaming data and outputting it to Azure Data Lake Storage, while also providing the ability for analysts to interactively query the data using Databricks notebooks.
Azure Stream Analytics and Azure Synapse notebooks (option A) can also process streaming data and output to Data Lake Storage, but they may not provide the same level of interactivity for analysts.
Event triggers in Azure Data Factory (option C) can help automate data movement between Event Hubs and Data Lake Storage, but they do not provide the necessary functionality for processing and querying streaming data.
Azure Queue Storage and read-access geo-redundant storage (RA-GRS) (option D) are not relevant for this scenario as they do not provide capabilities for processing and querying streaming data.
A. Azure Stream Analytics and Azure Synapse notebooks:
Azure Stream Analytics can be used to process the streaming data from Azure Event Hub and output the data to Azure Data Lake Storage. Azure Synapse notebooks provide interactive querying capabilities, and they can be integrated with the Azure Data Lake Storage to enable analysts to run their analytics on the stored data.
B. Structured Streaming in Azure Databricks:
Structured Streaming in Azure Databricks indeed supports streaming data and can write outputs to Azure Data Lake Storage. However, the question emphasizes "interactively querying" the streaming data, and while Databricks notebooks allow for interactive queries, Azure Synapse notebooks are better integrated with Microsoft's suite of data tools for broader analytics purposes.
@Andrew_Chen Azure Databricks with Structured Streaming is preferred over Azure Stream Analytics and Azure Synapse Notebooks for real-time streaming data processing from Azure Event Hubs due to its native support for continuous processing, live querying, and seamless integration with Azure Data Lake Storage.
B. Structured Streaming in Azure Databricks
Here’s why:
Structured Streaming in Azure Databricks:
Real-time Processing:
Interactive Querying: Databricks notebooks allow analysts to interactively query and visualize the streaming data, making it easy to gain insights in real-time.
Integration: It integrates seamlessly with Azure Event Hubs for data ingestion and Azure Data Lake Storage for data output.
Other Options:
A. Azure Stream Analytics and Azure Synapse notebooks: While Azure Stream Analytics is excellent for real-time processing and Azure Synapse notebooks for interactive querying, combining them might not be as seamless and efficient as using Databricks for both tasks.
selected answer : B
to visualize data in stream analytics you use SQL query in the Azure portal inside synapse , not a notebook , therefore the answer is B
What streaming sources and sinks does Azure Databricks support?
Databricks recommends using Auto Loader to ingest supported file types from cloud object storage into Delta Lake. For ETL pipelines, Databricks recommends using Delta Live Tables (which uses Delta tables and Structured Streaming). You can also configure incremental ETL workloads by streaming to and from Delta Lake tables.
In addition to Delta Lake and Auto Loader, Structured Streaming can connect to messaging services such as Apache Kafka.
https://learn.microsoft.com/en-us/azure/databricks/structured-streaming/
I don't see data lake in the list, so probably the answer is A.
I am in favour of B because of this piece of information I have encountered:
https://www.databricks.com/spark/getting-started-with-apache-spark/streaming
On the other hand, there is this: https://learn.microsoft.com/en-us/azure/event-hubs/process-data-azure-stream-analytics
So I believe both to be valid, Azure Stream Analytics seems to be more straightforward
B. Structured Streaming in Azure Databricks is incorrect because while it allows you to process streaming data using Spark's structured streaming API, it is not designed to directly output the data to Azure Data Lake Storage. Instead, it typically outputs the data to storage systems like HDFS, S3, or Cosmos DB. Additionally, Databricks is a separate service that does not integrate with Azure Synapse for interactive querying. While it's possible to use Databricks to read the data from Data Lake Storage and use Spark to process the data and then write it back to Data Lake Storage, it will not be as efficient as using Azure Stream Analytics for this use case as it is specifically designed for streaming data processing and also has built-in connectors to various data storage and analytics services like Data Lake Storage
Although this might be true, after some pondering, the given solution A. Azure Stream Analytics and Azure Synapse notebooks requires a Synpase workspace which is not implied.
So I guess it would be databricks.
It's implied. Solutions said Azure Stream Analytics and Azure Synapse Notebook, Azure Synapse notebook cannot be created without Azure Synapse Workspace.
Why not Azure Stream Analytics and Azure Synapse Analytics?
upvoted 2 times
...
Log in to ExamTopics
Sign in:
Community vote distribution
A (35%)
C (25%)
B (20%)
Other
Most Voted
A voting comment increases the vote count for the chosen answer by one.
Upvoting a comment with a selected answer will also increase the vote count towards that answer by one.
So if you see a comment that you already agree with, you can upvote it instead of posting a new comment.
esaade
Highly Voted 2 years agoAndrew_Chen
Highly Voted 1 year, 5 months agonadavw
1 week, 1 day agoExamDestroyer69
1 year, 2 months agof2a9aa5
Most Recent 7 months, 3 weeks agoAccountHatz
1 year agoSirstyle
1 year, 2 months agokkk5566
1 year, 6 months agoauwia
1 year, 8 months agovadiminski_a
1 year, 11 months agovadiminski_a
1 year, 11 months agoKate0204
2 years agoKarl_Cen
2 years, 1 month agoLestrang
2 years, 1 month agoLestrang
2 years, 1 month agoMal2002
1 year, 9 months agoalexnicolita
2 years, 2 months ago