Monitor Azure Data Factory with User Properties

Do you monitor Azure Data Factory with user properties? User properties help you easily filter different Azure Data Factory activities based on custom properties and values. You can define these properties manually or dynamically. 

What are some of the benefits of using Azure Data Factory User Properties? 

• Easily monitor Azure Data Factory, complements perfectly with Azure Data Factory annotations  
• Find errors faster so you can solve them 
• Enhance the governance of Azure Data Factory pipelines and datasets 

Using Azure Data Factory user properties is highly recommended and part of best practices for developing Azure Data Factory solutions. 

What is the difference between Azure Data Factory user properties and annotations?  

To begin, let’s compare annotations and user properties: 

• Azure Data Factory annotations are static values and tags that help you group and organize objects (pipelines, datasets, etc).  

• Azure Data Factory user properties are defined within the Azure Data Factory activities and the values can change during execution. They help you measure performance for pipeline activities. 

Why should I use Azure Data Factory annotations? 

Azure Data Factory use properties to enhance the monitoring experience of your Azure Data Factory solutions. 

Monitoring mission-critical data movements is critical for any company. You’ll want to take advantage of the Azure Data Factory built-in features (like user properties) to enhance the monitoring experience. 

Only a few years ago, if you wanted to enable these capabilities, you had to create log tables and write queries or create additional reports to consume this information. With Azure Data Factory user properties and Azure Data Factory annotations, there are a growing number of built-in features to mitigate some of the past challenges with other ETL or ELT tools. 

What are Azure Data Factory user properties? 

Azure Data Factory are key/value pairs. This means that you will have a key (user property name) and a value (user property value). You can have up to 5 user properties per activity.  

For example: 

• Key: Source 
• Value: dbo.Sales 

Copy activity example with dynamic values 

Create Azure Data Factory user properties 

First, select an Azure Data Factory activity and navigate to the user properties section, Then, click New. 

Add a few user properties (the expression builder is not available). You can copy them from previous activities. In this case, I am creating the following user properties: 

• Source: complete dynamic expression 

• “@concat(item().Table_Schema,’.’,item().Table_Name)” 

• Destination: concatenates a static string and dynamic expression 

• “@concat(‘Demos/AdventureWorksDW/parquet/‘,item().Table_Schema,’_’,item().Table_Name)” 

• Environment: uses a global parameter 

• “@pipeline().globalParameters.Environment” 

Is that it? Yes, now you can add user properties to any activities. 

A few tips: 

• Use expression and avoid using hardcoded values  
• Add user properties to copy activities 
• Keep names consistent so you can track all the activities for a specific data asset 

Don’t forget to save the changes. 

Monitor Azure Data Factory pipelines with user properties 

Now, trigger the pipeline and go to the Monitor Hub. Click on a pipeline so you see the activities. 

You will see the user properties values. 

If you click on the user properties, you can add any missing ones to the columns. 

The column will be added to the grid. 

You can also use Azure Log Analytics to monitor them. 

Summary 

In summary, Azure Data Factory user properties go well with Azure Data Factory annotations and they are part of the best practices for development. I highly suggest using them to improve the monitoring experience. 

What’s Next?  

In upcoming blog posts, we’ll continue to explore some of the features within Azure Data Services.    

As always, please leave any comments or questions below.  

If you haven’t already, you can follow me on Twitter for blog updates, virtual presentations, and more!