Posted on Leave a comment

Testing with a Twist: The Testup Build Pipeline

How do you write software that tests itself? We at Testup are asking us this question every day, because we are writing software that allows everybody to test their software. And that obviously includes ourselves. One of our main missions is to write software that we love to use for testing our own software. Sounds twisted? Well, ….

Let’s first look at the attributes that we love and who is the hero in that discipline.

AttributeHero technologyIdeal
Fast responseStatic code analysisGet feedback as you type
Full historyStack trace
(Log file)
See everything that happened in run up to the failure
Transparent stateDebuggerSee all variable values at the time of failure
Production like environmentDocker,
Infrastructure as code
Quick reproduction of deployment settings
Smart assessmentsHuman
AI
Make informed decisions about the validity of the observed behavior

How can we achieve all these qualities at once? Unfortunately there is no hammer that hits all nails, at least not all at once. Let’s look at our build pipeline and see how we are doing.

There are three different environments to run the tests:

  • Build environment
    • Triggered with every commit (or locally on request)
    • Has access to only the artefacts of one repository at a time
    • Should finish within few minutes at most
  • Integration environment
    • Replicates all components of the productive environment
    • Mocks/disables external components like payment, messaging, …
    • Automated tests finish within an hour or less
  • Staging environment:
    • Usable environment for humans to interact
    • Final validity check for new features
    • Should be completed within a day

Below you can see the the build process from first commit up to production release. There are only two manual steps:

  • Pull request: Manual inspection of code quality
  • Acceptance test: Manual inspection of the feature’s usability before release

We will cover all aspects of the build pipeline in future posts. Next to come: “Automated UI tests”

“Testing with a Twist” is a series of articles about testing with Testup. Up to now the following articles were published: