Posted on Leave a comment

CI-CD Results API

This page lists the API calls that are meant for external integration. Currently there is one endpoint to get information in JSON format.

Also look at Integrate testup into the CICD pipeline, for API calls to start schedules and get response in plain

Authentication

All calls to our api can be authorized via the ApiKey. You can get this key from your user, or your project settings page. Please check for a description on to get this ApiKey in step one of cicd integration. For following we assume that $TESTUP_APIKEY is set to your key.

To access any REST endpoints you need to set the following header

Authentication: ApiKey-v1 $TESTUP_APIKEY

In curl commands you can set the Header like this:

curl … -H “Authentication: ApiKey-v1 $TESTUP_APIKEY”

Project status

The project status call returns the result of the previous test runs. You can call the following address to the get the current status

https://app.testup.io/cicd/project/$PROJECT_ID/results


The following query parameters are possible:

limitNumber of past executions that are included in the response (default 20)
fromDatestart time of the execution list (Iso format: YYYY-mm-DDTHH:MM:SS)
toDateend time of the execution list (Iso format: YYYY-mm-DDTHH:MM:SS)

Example:

curl “https://app.testup.io/cicd/project/$PROJECT_ID/results?limit=1” -H “Authorization: ApiKey-v1 $TESTUP_APIKEY”

Response:

{
  "project": {
    "id": 1,
    "name": "Project name"
  },
  "executions": [
    {
      "id": 2,
      "name": "Run by Schedule",
      "startTime": "2023-10-18T16:15:00.239571Z",
      "tests": [
        {
          "id": 3,
          "name": "Test case one",
          "status": "PASSED",
          "failedRecordings": []
        },
        {
          "id": 4,
          "name": "Test case two",
          "status": "PASSED",
          "failedRecordings": []
        }
      ]
    }
  ]
}
Posted on Leave a comment

How to integrate tests into Gitlab CI/CD pipelines

This tutorial shows how you can easily run all your tests from a Gitlab pipeline. We assume that your current pipeline is already able to build and deploy your software. This tutorial contains the following steps to complete this task:

  1. Get your ApiKey from Testup to authenticate your Gitlab pipeline
  2. Insert the ApiKey into a secure space as a Gitlab variable.
  3. Extract the Id of your schedule
  4. Setup the Gitlab pipeline to run your tests
  5. Optional: Make temporary changes to the settings in your test

1) Get your ApiKey from Testup

Go to your Testup start page and click on the “Profile” tab. Find your ApiKey und the section “Api Keys”

2) Provide your ApiKey to Gitlab

Since the api key gives full access to your testup settings it should be provided to the Gitlab pipelines in a secure way. To do this open Gitlab and follow these steps:

  1. Open the project settings
  2. Open the CI/CD section
  3. Expand the Variables section
  4. Press “Add variable”

Once you successfully opened the add variable dialog you can provide your details as follows:

  1. Change your variable’s name to “TESTUP_APIKEY”
  2. Paste the Testup ApiKey into the value field
  3. Uncheck the “Protect variable” box (or alternatively protect your pipelines that should run your tests)
  4. Press “Add variable” to complete the setup

3) Find your Testup schedule Id

In this step you must find the numeric Id of the schedule running your tests. For this step you navigate to your project in the browser. From there you open (or create) the schedule, such that you see a domain that it has the form http://app.testup.io/schedule/<scheduleId>?… Keep that schedule Id ready as you will need it in the next step.

4) Setup your Gitlab Pipeline

We assume that you already have a running pipeline that builds and deploys your project. Testup can only run your tests with access to a deployed version of your software. Therefore, you need to add the end-to-end test as a final stage in your build pipeline. Go to your stages section and add e2e-test. Your file .gitlab-ci.yml will probably look something like this:

stages:          # List of stages for jobs, and their order of execution
  - build
  - unit-test
  - deploy
  - e2e-test     # Add this stage here

As a next step you add your end-to-end test step at the end of your pipline description. Your new pipeline step can be inserted as shown below. Don’t forget to provide the correct schedule Id.

Testup:   # Run this job after deployment completed successfully
  stage: e2e-test
  image: curlimages/curl
  variables:
    SCHEDULE_ID: <YourScheduleId>
  script:
    - URL="https://app.testup.io/cicd/schedule/$SCHEDULE_ID/run/$CI_JOB_ID"
    - curl $URL -H Authorization:\ ApiKey-v1\ ${TESTUP_APIKEY} -s --retry 12
    - curl $URL -H Authorization:\ ApiKey-v1\ ${TESTUP_APIKEY} -s --fail

How exactly does this step work technically? First, it starts with an image that provides the “curl” command. Then it builds the url that triggers the start of the pipeline. This url contains the schedule id to run as well as the Gitlab job id to distinguish update requests from new runs. Following these preparations two curl commands are issued. The first calls Testup and retries until the test is either marked as failed or passed. Until ready the endpoint returns a 504 timeout code along with some early debug information. This first curl also makes sure that your pipeline’s debug messages contain useful information and a link to the corresponding resource in Testup. The second curl is necessary to make the pipeline fail if the tests failed.

Once your pipeline is set up you will see a debug message in your pipeline that looks something like this:

5) Optional: Provide additional settings to your test

Very often it is necessary to run your pipeline tests with other values than the ones used in interactive editing. Common cases are temporary domains or changing parameters for users, passwords etc. You can provide additional parameters in the message body of the curl. It is possible to replace urls and text contents that occur in your test. Your pipeline would then look like this:

Testup:   # Run this job after deployment completed successfully
  stage: e2e-test
  image: curlimages/curl
  variables:
    SCHEDULE_ID: <YourScheduleId>
  script:
    - URL="https://app.testup.io/cicd/schedule/$SCHEDULE_ID/run/$CI_JOB_ID"
    - curl $URL 
      -H "Authorization:\ ApiKey-v1\ ${TESTUP_APIKEY}"
      -H "Content-Type: application/json"
      -s --retry 12
      --data '{
       "message": "Execution title shown in testup.io",
       "variables": [{"name":"some_var", "value":"some value"}],
       "urlMap":[{
        "old":"https://stage.example.com",
        "new":"https://qa.example.com",
        "regex":false
       }],
       "textMap":[{
        "old":"OriginalValue",
        "new":"NewValue",
        "regex":false
       }]d
      }' 
    - curl $URL -H Authorization:\ ApiKey-v1\ ${TESTUP_APIKEY} -s --fail

Posted on Leave a comment

Running Tests from a Script using REST API

There are many use cases where you want to start a test from a script. A common scenario is a regular execution from a server cron job or within a CI/CD build pipeline.

The first thing you need is your personal API-key. In the Testup Web-App (https://app.testup.io) you will find the API Key under the tab “Profiles”:

Now you can use the following APIs in a shell, for example:

Run a test.

curl -X POST "https://app.testup.io/api/v1/project/run-all/<Project ID>?testIds=<Test Case ID>"     
         -H "accept: */*" 
         -H "Content-Type: application/json"              
         -H "Authorization: ApiKey-v1 <APIKEY>"              
         -d "{\"message\":\"New run by curl\",\"threads\":1}"

where 

<APIKEY> is your API-key (see above),
<Project ID>  is the Project ID of your test (see URL of your project, e.g. 6607334),
<Test Case ID>  is the Test Case ID of your test (see URL of your test).
"New run by curl\" is the name of the new execution the command will create. You can change this freely.
"threads\" is the number of parallel test executions. 

This curl returns a JSON with the execution id <id>. Using this <id> you can get the status of the results:

curl -X GET "https://app.testup.io/api/v1/execution/<id>/details"              
     -H "accept: */*"              
     -H "Authorization: ApiKey-v1 <APIKEY>"

If you want to run all test cases at once, you can use

curl -X POST "https://app.testup.io/api/v1/project/run-all/<Project ID>"                
     -H "accept: */*" 
     -H "Content-Type: application/json"                
     -H "Authorization: ApiKey-v1 <APIKEY>"                
     -d "{\"message\":\"New run by curl\",\"threads\":2}"

Another feature you can use is to replace values in the test with new values using textMap. Or to change the url using urlMap:

PROJECTID=<Project ID>
CONFIG='{
    "message":"Run by Curl",
    "textMap":[{
       "old":"Beanie",
       "new":"Weenie"
    }],
    "urlMap":[{
       "old":"https://does-not-match",
       "new":"https://new-url"
    }]
}'
echo $CONFIG
curl -X POST https://app.testup.io/api/v1/project/run-all/$PROJECTID \   
     -H "accept: */*" \
     -H 'Content-Type: application/json' \
     -H "Authorization: ApiKey-v1 <APIKEY>" \
     -d "$CONFIG"