Azure Data Factory Solution to Start and Stop VM

If you are trying to start and pause/stop an Azure Virtual Machine (VM) using Azure Data Factory or Azure Synapse pipelines to optimize on-going costs, don’t look any further! Today’s post will cover the solution. 

Reasons to Start and Stop your Azure VM 

To begin, the main reason to start and stop your Azure VM with Azure Data Factory or Azure Synapse Analytics is to create an automated process, You can also decrease on-going Azure costs and optimize resources, using the resources when you need them. 

Let’s imagine some scenarios: 

• You might want to stop your Azure VM in non-production environments if you are not using them 
• You could be looking for a way to start an Azure VM when you need it and stop it when you don’t need it 
• When hosting Azure Data Factory / Azure Synapse Self-Hosted Integration Runtimes in Azure VMs, you can start the VM just to load data from the sources that need a Self-Hosted IR 

This solution complements perfectly with the following optimization solutions: 

• Pause and Resume Azure Synapse Analytics SQL Pool 
• Scale Azure Synapse Analytics SQL Pool with Azure Data Factory 

Azure VM Statuses 

Before we start, let’s summarize the key statuses/actions for Azure VMs in this blog post: 

Running: the VM is accessible and running, you are paying while the VM is in this status. You trigger this status by using the Start option in the Azure portal. 

Deallocated / Stopped: you stop the VM and resources are returned to the data centre, you stop paying for the VM. This is the Stop option in the Azure portal and the Deallocate command when using APIs (this blog post). 

Power off/stop: you can trigger this status by using APIs, the resources stay attached to the VM and you continue paying for the resources. 

Deallocating / Updating / Starting are transition states. 

For the purpose of simplifying the solution, we will use the following commands in our pipeline: 

• Start to resume the Azure VM 
• Deallocate to stop the Azure VM 

Pre-requirements (Access) 

First, to allow Azure Data Factory or Azure Synapse Analytics to start and stop your Azure VM, you need to grant Virtual Machine Contributor the following access. 

Download the Solution to Start and Pause/Stop Azure VMs 

You can download the solution to start and stop the Azure VM with Azure Synapse Analytics / Azure Data Factory from my GitHub repository. 

Start or Stop Azure VM solution 

Trigger Azure Data Factory Pipeline to Start and Stop Your Azure VM 

To use the pipeline, there are different parameters that you need to pass:  

• SubscriptionId: format XXXXXX-XXXXXX-XXXXXXX-XXXXXX  
• ResourceGroupName  
• Region: for example australiaeast  
• VMName 
• Command (Start or Deallocate)  

These parameters help you create the API request to start or stop your Azure VM.  

Create an Azure Data Factory Pipeline to Start or Stop Azure VM 

Next, to create an Azure Data Factory or Synapse Analytics solution to start or stop a VM, you need the following components: 

1. Add pipeline parameters required for execution 

2. A Web activity to get the status of your Azure Virtual Machine (VM)  

3. An If Condition activity to make sure the status of your Azure VM can be paused or resumed 

4. A Web activity (inside the If Condition) to Start or Stop your Azure VM. This takes advantage of the parameter “Command” 

Add the parameters to the pipeline. 

After that, add the Web Activity and modify the request to be GET. Change the retry options if you’d like to increase them.  

Go to the settings sections and define Method to GET and Authentication to MSI.  

Copy the following command for the URL 

@concat('https://management.azure.com/subscriptions/',pipeline().parameters.SubscriptionId,'/resourceGroups/',pipeline().parameters.ResourceGroupName,'/providers/Microsoft.Compute/virtualMachines/',pipeline().parameters.VMName,'/InstanceView?api-version=2020-12-01') 

Then, add an If Condition activity to start or stop the Azure VM. 

For the If Condition activity expression, copy the following command. It will: 

• Start the Azure Virtual Machine (VM) only if the status is ‘VM deallocated’ 
• Stop the Azure Virtual Machine (VM) if the status is ‘VM running’ 

@or(and(equals(activity('Check Azure VM Status').output.statuses[1].displayStatus,'VM deallocated'), equals(pipeline().parameters.Command,'Start')),and(equals(activity('Check Azure VM Status').output.statuses[1].displayStatus,'VM running'), equals(pipeline().parameters.Command,'Deallocate'))) 

Next, click the Activity editor. 

@or(and(equals(activity(‘Check Azure VM Status’).output.statuses[1].displayStatus,’VM deallocated’), equals(pipeline().parameters.Command,’Start’)),and(equals(activity(‘Check Azure VM Status’).output.statuses[1].displayStatus,’VM running’), equals(pipeline().parameters.Command,’Deallocate’))) . This activity will execute the POST request and start or stop an Azure VM. 

Configure the activity to POST and copy the following command. 

@concat('https://management.azure.com/subscriptions/',pipeline().parameters.SubscriptionId,'/resourceGroups/',pipeline().parameters.ResourceGroupName,'/providers/Microsoft.Compute/virtualMachines/',pipeline().parameters.VMName,'/',pipeline().parameters.Command,'?api-version=2020-12-01') 

That’s it! Now save your pipeline and you are ready to use it. 

Summary 

Today, you’ve had a look at starting and stopping an Azure Virtual Machine (VM) using integration pipelines in Synapse Workspaces or Azure Data Factory.   

Don’t forget to download the following solutions: 

• Pause and Resume Azure Synapse Analytics SQL Pool 
• Scale Azure Synapse Analytics SQL Pool with Azure Data Factory 

What’s Next?

In upcoming blog posts, we’ll continue to explore some of the features within Azure Data Services.       

Please follow Tech Talk Corner on Twitter for blog updates, virtual presentations, and more!     

As always, please leave any comments or questions below.     

Check out these other posts: