Recently I was on a security initiative to enhance a very large Azure subscription to use private networking (vnet). A requirement was to ensure that the numerous existing Azure DevOps pipelines worked with minimum changes.

Initial Effort

At first glance, creating self-hosted build agents is a trivial task.

My initial effort resulted in Docker + Terraform + Azure Container Instance (aci) to deploy agents in a clean an automated way. It was perfect… until I ran a pipeline and it failed immediately. Turns out the base image doesn't have PowerShell Core… or Azure Cli… or Python…

Naked OS Agent

I discovered the resulting 'naked OS' agent to be quite limited. It makes sense, of course. One of the benefits of a self-hosted agent is total control of all the SDKs on it. You are free to include only what you need. Unfortunately, building an image from scratch would require an audit of all the existing pipelines and compile a list of required software, SDKs, etc. An audit was a daunting proposition. Not only were there hundreds of tasks within the dozens of pipelines, there were dozens of PowerShell scripts and a handful of bash scripts called by those tasks that have their own dependencies (think a PS script referencing a Module, bash script using specific version of Python).

Impressive Capabilities

It is easy to take for granted just how capable the Microsoft-hosted agents have become. They are absolutely stuffed to the gills with an impressive array of languages, SDKs, and tools - including side-by-side versioning support. Just look at the support for Python alone:

side-by-side Python support

Replicating Microsoft-hosted Agents

Eventually I located the scripts that Microsoft uses to build their build agent virtual machines. With these, I was able to build the same exact images that are used for Microsoft-hosted agents. The many existing pipelines can continue to run as-is with no changes other than switching out the agent.

Building the Images

Clone the repo

git clone https://github.com/actions/virtual-environments.git

Import the helper modules

 Import-Module .\helpers\GenerateResourcesAndImage.ps1

Kick off the long running (6+ hours) image creation process. Here I had issues if I didn't pass a Github token. Note that this will provision a temporary environment in Azure to build everything.

GenerateResourcesAndImage -SubscriptionId {sub-guid} -ResourceGroupName {rg-name} -ImageGenerationRepositoryRoot "$pwd" -ImageType windows2019 -AzureLocation {location} -GithubFeedToken 'REDACTED'

After vhd creation finally completes, create a custom virtual machine image from the vhd that was generated from the previous step. Note the name of the vhd is randomly generated, so you'll want to make sure to grab the correct one.

Windows:

az image create --resource-group {rg-name} --name agent-win2016 --source https://agentimages001.blob.core.windows.net/images/pkrosr2nmqwucz7.vhd --os-type Windows

Linux:

az image create --resource-group {rg} --name agent-ubuntu1804 --source https://agentimages001.blob.core.windows.net/images/pkrosr232dfehwe.vhd --os-type Linux

Next, create the Virtual Machine Scale Set (vmss).

az vmss create --name devops-agent-pool-win2016 --resource-group {rg} --image "/subscriptions/{subid}/resourceGroups/{rg}/providers/Microsoft.Compute/images/agent-win2016" --vm-sku Standard_D2s_v4 --storage-sku StandardSSD_LRS --authentication-type password --instance-count 2 --disable-overprovision --upgrade-policy-mode manual --single-placement-group false --platform-fault-domain-count 1 --load-balancer '""' --subnet "/subscriptions/{subid}/resourceGroups/{rg}/providers/Microsoft.Network/virtualNetworks/{vnet-name}/subnets/{sub-name}" --admin-username azdoadmin --admin-password XXXXX

Finally, navigate over to the Azure DevOps Project settings, select Agent pools under Pipelines, and select Add pool to create a new agent pool.

Congratulations, you've now got a replica of a Microsoft-hosted agent on your vnet!