Running Jmeter Load Tests and Publishing Jmeter Report Within Azure DevOps

This blog has solutions to 2 problem statements:

1. Running Jmeter Load tests in a pipeline through azure devops.
2. Publishing Jmeter HTML reports in azure devops and rendering it in a seperate tab (Parallel to Summary tab).

The second problem statement is a hot topic in the developer community as many developers want to publish Jmeter Reports(Or any other report) within Azure Devops. See below link:

Support for generic HTML Publishing inside Build and Release Pipelines

We have created a custom extension which solves the problem of sending Jmeter (Or any) HTML report to azure devops. If you want to skip directly to how to use this extension, go to “PublishHTMLReport extension” section directly in this blogpost.

But before everything else, Why run Jmeter tests from Azure DevOps Pipelines?

1. Repeatability- We wanted to Run our load tests repeatedly, we were fine tuning our applications which meant we want to do short duration load tests multiple times per day. This needed automation as doing ssh to a VM every time we wanted to load test, was not feasible.
2. Traceability- What happens when we are running 50 load tests per day while using different permutations of configuration and builds ? We mess up the traceability of what changes lead to what affects. How do we stitch changes done in code or configuration with the load test results without maintaining excel sheets?
3. Configuration driven- What happens when we want to test with 10 users and then 50 and then back to 5? And what about when we want to change the ramp up time, load duration of test, what if we want to make only 10 calls to the api? These changes can be done in the jmx file to support all kinds of scenarios, but making all these parameters and much more configurable through the pipeline made our job much easier.

Overview of the Load Test Setup in Azdo

We used a Centos VM in azure as a self hosted agent and created an azure devops pipeline to run on the self hosted agent. The azure devops pipeline orchestrated the following 3 steps:

1. The pipeline takes input variables like RAMPUPTIME, LOADDURATION, SHUTDOWNTIME, PARALLELUSERS etc from the user and configures the jmx file as per the above inputs.
2. Then it triggers build.sh script which downloads relevant Jmeter version, creates a docker container from it and runs the load tests as per what is defined in the jmx file. It then makes the results available in the LoadReports folder.
3. This LoadReports folder contains the end result of the load run which is then published by using the PublishBuildArtifacts task.

Detailed guidance on how you can create your own setup:

1. Create a VM in your azure account. I like Linux so i did it with Centos, but it will work on Windows as well. Register this VM as a self hosted agent in azure devops. Everything needed to do this is here: https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/v2-linux?view=azure-devops.
2. Clone this repo: https://github.com/lakshaykaushik/JmeterinAzdo.git, it has a dockerfile which will essentially build a docker image with Jmeter installed. Something like this:

3. This repo contains run.sh which contains the Jmeter run command along with required arguments. This is also the entry point script when the container is spun from the docker image.

4. This also contains a build.sh script, which will download a version of jmeter which you specify, run docker build and runs a container triggering run.sh which initiates the load run. When this script finishes up, you will have a folder named LoadReports containing your load test results.

5. Run_Load_Test.yml file contains the azure devops pipeline which orchestrates the following steps:

1. Inserts inputted variables into jmx to control duration of load run, ramp up time etc. file from azure devops.
2. Runs build.sh, which spins the container running Jmeter.
3. Runs Junit_Converter.py, which converts the jtl result file into Junit report which can be uploaded and rendered within azure devops. Link to Junit_Converter.py is https://github.com/Azure-Samples/jmeter-aci-terraform/tree/main/scripts.
4. Uploads the complete LoadReports folder to azdo, this folder can be downloaded from artefacts section and analysed.

To build end to end traceability between load runs and code and config changes we relied on good old Commit messages, how? Let’s see below.

All changes to code or config done for running load test are committed with a meaningful commit message. So this is what we used to do:

1. Change code for tuning the system. Commit it.
2. Go to AZDO, provide needed variables like load duration, no of threads, ramp up time etc and run the pipeline.
3. Pipeline sets up Jmeter, runs the load test and publishes the load test results as a zip and makes it available for download.
4. Summary page has a commit message which mentions what change we did, what was the commit number( What files changed) and the relevant load test results are available for download. See below.

Summary Page of Load Run

Notice the “1 Published” section of the summary, when you click on it, you will get the load test results zipped into a folder, ready for you to analyse:

Contents of the zipped folder containing load test results

Download the folder, unzip it and you will be able to open index.html with rich graphs and data of your load run:

index.html

But….Wouldn’t it be great if the complete Jmeter report is available within azure devops and there is no need to download the zipped report to analyse test results? How will that be for traceability of load test results? Pretty neat right? Incomes, the extension we developed for this.

PublishHTMLReport extension

To solve exactly the above problem, there is a cool extension on Azure DevOps marketplace (Disclaimer: I am publisher of the extension )which does exactly that. It renders the entire Jmeter report (all the cool graphs) as a separate tab parallel to Summary tab in the build results pipeline. We have named it “PublishHTMLReports”, because not only it can publish and render Jmeter reports in azure devops, but it can render any HTML within azure devops. You can download the extension from marketplace using:

Extension Name: PublishHTMLReports

Publisher: LakshayKaushik

Link: https://marketplace.visualstudio.com/items?itemName=LakshayKaushik.PublishHTMLReports

How to use this extension? Very simple.

1. Download and install the extension in your azdo organisation. This extension contains all what you will need to render Jmeter reports on azure devops and a pipeline task which will let you publish the reports (Remember LoadReports directory above)
2. Create a pipeline and use the task like below. The task publishhtmlreport@1 is part of the extension which was downloaded.

htmlType: Is the type of HTML you want to publish. Currently, it can be ‘Jmeter’ for Jmeter reports or ‘genericHTML’. for any other generic HTML page. We can extend this to include other complex static websites (How to extend this will be part of another blog post).

JmeterReportsPath: It is the path where your Jmeter reports are. When Jmeter is run, it creates a folder containing all sorts of files(css, js, html etc). Provide path of that folder to ‘JmeterReportsPath’.

3. Run the Pipeline and behold!!. Complete Jmeter report within Azdo.

4. What if you want to use the extension for publishing and rendering some other HTML and not Jmeter Reports? No worries- we have got it covered. Just pass the html page to “htmlPath” and change “htmlType” to genericHTML.

See Published Result below:

Generic HTML Page being published on Azure Devops using the extension.

Source code of the extension: https://github.com/lakshaykaushik/PublishHTMLReport

Thanks to my awesome team mates: Maninderjit (Mani) Bindra, Sapinder Pal Singh, anand chugh and Ritesh Modi for working with me to make this extension available to the public and Azure Devops users.