Run and report JOSF tests in Azure DevOps

Running your tests during your pipeline release cycle should be a breeze, and luckily with JOSF, that is the case. For this example we will require:

• An Azure DevOps git repository, where your JOSF project already resides.
• A playbook that you’d like to run.
• Azure DevOps

For this example, we will use Azure DevOps as our pipeline platform. We already have a project in which we’d want to run our tests, so make sure you have one in Azure DevOps for yourself.

Note: If you haven’t already stored your test cases in Azure DevOps, please follow these instructions first and look under the “Import a GIT project from Microsoft Azure DevOps” section.

Add JOSF to Azure DevOps

To use JOSF in your pipelines, we need to add it as an artifact and reference to it in our pipeline script

1. Install the Azure CLI on your local machine: If you haven’t already, download and install the Azure CLI from the official Microsoft documentation.
2. Create a new feed: New artifacts require to be put into a Feed. Create a new feed in Azure DevOps by going to Artifacts and then press Create Feed. We are naming it josf-installation and keep the artifact scoped to this project.
   

   

3. Start the Azure CLI and Authenticate: Open a command prompt or terminal and run the following command to sign in to your Azure DevOps account:

   az login 

   or use “az devops login” if you want to login with a PAT token. How to create a PAT token.
   

   az devops login 

4. Create a new artifact source from the JOSF server variant: The new feed can hold our artifact which we can publish via the Azure CLI. Be aware that if you are working within an organzation where you manually have to download your webdrivers, you also should add these webdrivers to the artifact, put it somewhere on the Azure machine you are using or do nothing if you are using a windows-latest machine (Chromedriver or Edge drivers by default on the machine). See chapter 2. Download and install JOSF – Add Chromedriver manually how to set this up.

   az artifacts universal publish --organization "{YOUR_ORGANZATION}" --project="{YOUR_PROJECT_NAME}" --scope project --feed josf-installation --name josf-server --version "{JOSF_VERSION}" --description "JOSF Test automation" --path "{PATH_TO_JOSF_SERVER_DIRECTORY}"

   Replace {YOUR_ORGANIZATION} with your Azure DevOps organization URL (see image below), and {YOUR_PROJECT_NAME} with the name of your project (see image below), {JOSF_VERSION} to the corresponding JOSF version (always three numbers: major/minor/patch. E.g. “3.1.0”) and{PATH_TO_JOSF_SERVER_DIRECTORY} with the JOSF folder that contains a clean JOSF server installation, not-including the projects folder. For an optimized result, download a fresh installation (JOSF server), unzip that folder and point the path to the /JOSF folder.

   
   
5. Verify successful publish: Open your feed in Azure DevOps and verify that JOSF has uploaded correctly.

   

That’s it! You have successfully added JOSF as a new artifact to Azure DevOps using the Azure CLI.

Use JOSF in your pipeline script

1. Navigate to Pipelines: Once inside your project, navigate to the “Pipelines” section using the left-hand side menu.
2. Create a new pipeline: Click on the “New Pipeline” button to start creating a new pipeline.
3. Select your source control: Choose your source control system where your code is hosted. Azure DevOps supports popular options like GitHub, Azure Repos, Bitbucket, and others. For this tutorial, we will use Azure Repos as our test cases are stored in the Azure repository and select your repository.
4. Choose your pipeline configuration: Azure DevOps offers different options for configuring your pipeline. Select Starter pipeline and continue.
5. Configure your pipeline: The pipeline is made up of a combination of tasks and powershell executions. In a birds-eye-view, we’ll be executing the following steps:
   • Install JOSF from the Azure Artifacts on that machine.
   • Place the testcases into JOSF.
   • Start JOSF and wait for it to be started.
   • Execute the playbook.
   • Publish the test results.
   • Commit the playbook results back into GIT repository.
6. Switch off Continuous integration: If this is not disabled, the pipeline will run after every commit in the master branch.
   • Edit the pipeline. The YAML file is in edit-modus.
   • Click on the three dots next to the “Run” button and click on “Triggers”.
   • Disable the “Continuous integration” option.
   • Click on the Save & queue button and click on Save.

The pipeline script

Your pipeline is executed based on an azure-pipelines.yml file, which contains the steps that should be executed. If you’d like to see the end result: go to the end of this page for the entire YAML file.

1. Switch to windows and setup an environment variable

By the time of writing this post, Azure generates a pipeline script which has a trigger and a pool of virtual machines, which it uses to execute the scripts on. For now, clear the YAML file and start from the beginning. Starting with setting the windows-latest machine so JOSF can execute correctly.

Next to the used vmImage, we need to set an environment variable which tells Selenium not to collect and send statistics to their statistics monitor.

# Pipeline for installing, starting and executing a JOSF playbook
# https://www.josf.nl/?p=3099

pool:
  vmImage: windows-latest

variables:
- name: SE_AVOID_STATS
  value: true

2. Download and install JOSF

The first step is using the correct JOSF installation, which you’ve added as an artifact, by going to the “Show assistant” part and search for Download package.

Step 1: Click on “Show assistant”.

Step 2: Search for “Download package” and click on it.

Step 3: Select the options.

Step 4: Make sure the downloadPath is set to $(System.DefaultWorkingDirectory)\JOSF and press the Add button.

Step 5: Add “extract: true” to the YAML.

This results in the following YAML snippet:

- task: DownloadPackage@1
  displayName: 'Unpack the JOSF artifact'
  inputs:
    packageType: 'upack'
    feed: '{FEED-ID}/{PACKAGE-ID]'
    view: '{VIEW-ID}'
    definition: '{DEFINITION-ID}'
    version: '{JOSF-VERSION}'
    downloadPath: '$(System.DefaultWorkingDirectory)\JOSF'
    extract: true

Add Chromedriver manually

If you are uploading your webdrivers manually in JOSF, it is mandory to copy the web-drivers into the JOSF installation web-drivers folder. In this example we are using Chromedriver on a windows-latest machine. Use the code below and put it in the YAML file between the installation and the start JOSF part.

- powershell: |
      mkdir $(System.DefaultWorkingDirectory)\JOSF\api\web-drivers
      Copy-Item -Path "$env:CHROMEWEBDRIVER\chromedriver.exe" -Destination "$(System.DefaultWorkingDirectory)\JOSF\api\web-drivers"
  displayName: 'Copy default windows-latest Chromewebdriver to JOSF web-drivers folder'

3. Start JOSF and wait for it to be online

Next is to start JOSF and wait for it to be online. We do this with the help of powershell, which starts the JOSF start command, and performs a ping on the localhost port where JOSF will respond, once it’s up and running.

Note how we’ve setup a timeoutInMinutes, as this prevents any infinite loops during failures of starting JOSF.

- powershell: |
   start-process '$(System.DefaultWorkingDirectory)\JOSF\api\start.bat'
   Write-Host 'Waiting for JOSF to be online'
   $statusCode = 0;
   Do {
       Write-Host -NoNewline '.'
       try {
         $statusCode = (Invoke-WebRequest 'http://localhost:8090/api/version').StatusCode;
         Start-Sleep -Milliseconds 750
       } catch {
         $statusCode = 0;
       }
   } While (!$statusCode -eq 200)
  displayName: 'Start JOSF and wait for it to be online'
  timeoutInMinutes: 2

5. Execute the playbook

Next is executing the command for running a certain playbook. We are only using the playbook name, projectpath (mandatory for this pipeline file) and the environment name, but other options are documented here or can be found in JOSF under the CLI menu-item (CLI wizard).

Note that we use our playbook named ‘Smoke’ on the environment named ‘local’. Change this to your own playbook and environment names. If you do not use environments, you can omit the entire environment option.

Note how this powershell task also contains the timeoutInMinutes, which you need to set to something which makes sense in your situation. Another important boolean is the continueOnError. If any tests fail, we want the pipeline to continue to its next and last mandatory task, which is publishing the report back to Azure DevOps.

- powershell: |
   $client = '$(System.DefaultWorkingDirectory)\JOSF\cli\josf-cli.jar'
   $arg = "$(System.DefaultWorkingDirectory)\JOSF\api\jre\bin\java.exe -jar '$client' -r playbook `"'Smoke'`" projectPath `"'$(System.DefaultWorkingDirectory)'`" environment `"'local'`""
   Invoke-Expression "cmd /c $arg"
  displayName: 'Execute playbook Smoke on environment local'
  continueOnError: true
  timeoutInMinutes: 60

6. Publish the test results

After JOSF executes a playbook via its command line interface, a JUnit report is generated. The task for this is found via the task section, but comes down to this yaml snippet:

- task: PublishTestResults@2
  displayName: 'Publish Test Results'
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '**/*junit-report.xml'

1. Review and save your pipeline: Once you have finished configuring your pipeline, review the settings and make any necessary changes or adjustments. Once you’re satisfied, click on the “Save” or “Create” button to save your pipeline.
2. Run your pipeline: After saving your pipeline, you can manually trigger a run or set up triggers based on different events like code commits, pull requests, or scheduled intervals. Once triggered, your pipeline will start running, and you can monitor the progress and view the execution logs.

7. Commit the playbook results back into the GIT repository

The last step is to commit the playbook results back into the GIT repository so the results will be available on the JOSF dashboard.

Steps:

1. Give the Azure user the rights to push and commit the playbooks into the GIT repository
   • Go to Project settings
   • Click on Repos->Repositories
   • Click on the repository you are using for JOSF
   • Click on Security
   • Search for the “Build Service” user. This username starts with the name of the project or the repository.
   • Set Contribute to “Allow”.
   • 
2. Add this script at the end of the YAML file. Fill in user.name, user.email and the GIT Repo URL for the git pull and push. You can find the GIT Repo URL really easy by clicking on the Repos button in Azure DevOps and copying the URL from your browser address bar.
3. Git push: The HEAD:main could also be the HEAD:master or a branch you are working on.

- powershell: |
    # Configure the git user
    git config user.name "Name of user"
    git config user.email "Email@address.com"
    
    # Stage every playbook run result file
    git add play-books/*
    
    # Commit the staged change
    git commit -m "Commit pipeline playbook results"
    
    # Pull any possible changes and then push to the branch
    git pull https://$(System.AccessToken)@dev.azure.com/{GIT Repo URL}
    git push https://$(System.AccessToken)@dev.azure.com/{GIT Repo URL} HEAD:main
  displayName: Git Commit pipeline playbook results

The entire YAML file (without manual uploading webdrivers)

# Pipeline for installing, starting and executing a JOSF playbook
# https://www.josf.nl/?p=3099

pool:
  vmImage: windows-latest

variables:
- name: SE_AVOID_STATS
  value: true

steps:
- task: DownloadPackage@1
  displayName: 'Unpack the JOSF artifact'
  inputs:
    packageType: 'upack'
    feed: '{GENERATED IN STEP 2. Download and install JOSF}'
    view: '{GENERATED IN STEP 2. Download and install JOSF}'
    definition: '{GENERATED IN STEP 2. Download and install JOSF}'
    version: '3.1.0'
    downloadPath: '$(System.DefaultWorkingDirectory)\JOSF'
    extract: true

- powershell: |
   start-process '$(System.DefaultWorkingDirectory)\JOSF\api\start.bat'
   Write-Host 'Waiting for JOSF to be online'
   $statusCode = 0;
   Do {
       Write-Host -NoNewline '.'
       try {
         $statusCode = (Invoke-WebRequest 'http://localhost:8090/api/version').StatusCode;
         Start-Sleep -Milliseconds 750
       } catch {
         $statusCode = 0;
       }
   } While (!$statusCode -eq 200)
  displayName: 'Start JOSF and wait for it to be online'
  timeoutInMinutes: 2


- powershell: |
   $client = '$(System.DefaultWorkingDirectory)\JOSF\cli\josf-cli.jar'
   $arg = "$(System.DefaultWorkingDirectory)\JOSF\api\jre\bin\java.exe -jar '$client' -r playbook `"'Smoke'`" projectPath `"'$(System.DefaultWorkingDirectory)'`" environment `"'local'`""
   Invoke-Expression "cmd /c $arg"
  displayName: 'Execute playbook Smoke on environment local'
  continueOnError: true
  timeoutInMinutes: 60

- task: PublishTestResults@2
  displayName: 'Publish Test Results'
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '**/*junit-report.xml'

- powershell: |
    # Configure the git user
    git config user.name "Name of user"
    git config user.email "Email@address.com"
    
    # Stage every playbook run result file
    git add play-books/*
    
    # Commit the staged change
    git commit -m "Commit pipeline playbook results"
    
    # Pull any possible changes and then push to the branch
    git pull https://$(System.AccessToken)@dev.azure.com/{GIT Repo URL}
    git push https://$(System.AccessToken)@dev.azure.com/{GIT Repo URL} HEAD:main
  displayName: Git Commit pipeline playbook results