Did you know that you can monitor Azure Data Factory pipelines by using annotations? Azure Data Factory annotations help you easily filter different Azure Data Factory objects based on a tag. You can define tags so you can see their performance or find errors faster.
Using annotations is a best practice for developing Azure Data Factory solutions. They complement Azure Data Factory User Properties that are more dynamic.
Azure Data Factory annotations are static and the values will not change during runtime.
Table of Contents
Why should I use Azure Data Factory annotations?
To begin, the easier it is to find the error, the faster you can fix it. Monitoring Azure Data Factory with annotations is a tool that enhances monitoring experience.
Now, imagine that it’s 3 am and you have received an SMS or a call because one of your company’s Azure Data Factory mission-critical data movements has failed. You need to find out what failed and what to do to fix it (I have been in this position; I know how it feels).
On top of that, imagine that your environment has hundreds of pipelines running on different schedules at the same time.
You might waste valuable time just to find the error.
What are Azure Data Factory annotations?
Azure Data Factory annotations are tags that you can add to your Azure Data Factory components to identify them. A tag allows you to classify or group different objects in order to easily monitor them after an execution. You can create multiple Azure Data Factory annotations.
In addition, you can add annotations to most Azure Data Factory components.
Pipeline example
Dataset example
Create Azure Data Factory Annotations
First, let’s add Azure Data Factory annotations to a component.
In any dataset, click the Properties option and add a few annotations.
Then, do the same for the Linked Service.
Now, save all the changes.
Let’s test them!
Monitor Azure Data Factory Pipelines with Annotations
Now, let’s monitor the pipelines by using Azure Data Factory annotations. Execute any pipeline where you have added annotations.
Go to the Monitor Hub and click annotations and add filter.
Azure Data Factory will link different annotations from different pipeline components.
You can also use the search box to find items related to your annotations.
Also, it’s possible to enhance the Azure Data Factory governance by using Azure Log Analytics.
Summary
In summary, Azure Data Factory annotations are one more tool to minimise the impact on your sleeping patterns when you are the one looking after the error of mission-critical Azure Data Factory data movements.
What’s Next?
In upcoming blog posts, we’ll continue to explore some of the features within Azure Data Services.
As always, please leave any comments or questions below.
If you haven’t already, you can follow me on Twitter for blog updates, virtual presentations, and more!
