Speeding up Cypress tests

On our recent project for railway topology verification at Schweizer-Electronic we used Cypress for E2E testing. All our workflows require importing or verifying data from various 3rd party systems and then building and re-building the topology models numerous times. Because of that, our tests became very quickly hard to maintain as they needed to be executed in one go and took tens of minutes. At this point, we started to look for ways to speed up the execution in the CI and find a way to debug and fix the tests without executing all of them again. As this can be challenging, in this article I would like to sum up what you can do in a similar case so you can work more effectively and not wait on your test runs as this guy:

1. Stop the tests on the first failure

As our tests are required to run in a very specific order and we cannot test use cases without testing the previous use cases first. Meaning that if one test like the data import has failed, it is very unlikely that the next tests will succeed or at least report trustworthy results. One failure can cause this:

Moreover, Cypress took much longer for failing builds than for successful ones, simply because of all the retries and timeouts on commands. The difference was about an extra 10 minutes on 35 minutes long build. Therefore, we Googled “abort” or “fail on first failure” solutions and not many of them really worked as it is not supported by default.

In the end, we ended up adding a plugin that stores whether there was a failed test or not:

cypress/plugins/index.js

/// <reference types="cypress" />

/**
 * @type {Cypress.PluginConfig}
 */
module.exports = (on, config) => {
  on('task', {
    setFailure: (val) => {
      return (failure = val);
    },
    getFailure: () => {
      try {
        return failure;
      } catch {
        return null;
      }
    },
  });
};

And overriding before and afterEach executions that abort the runner immediately after a failure:

cypress/support/index.ts

before(() => {
  if (!Cypress.browser.isHeaded) {
    cy.task('getFailure').then((failure) => {
      if (failure) {
        // @ts-ignore
        Cypress.runner.stop();
      }
    });
  }
});

afterEach(function onAfterEach() {
  if (
    this.currentTest?.state === 'failed' &&
    // @ts-ignore
    this.currentTest?._retries === this.currentTest?._currentRetry
  ) {
    cy.task('setFailure', true);
    // @ts-ignore
    Cypress.runner.stop();
  }
});

And even though this is not supported by Cypress yet, you can this way say to your CI that it is time to move to the next job as this is predestined to fail.

2. Running the tests in parallel

The more challenging speed-up technique was to split the single scenario tests into thirds and then run them in parallel. As simple as it sounds, you might hit a couple of challenges, so I will try to sum them up.

2.1. Creating and using DB images

In the case of parallelization, tests need to be executable in a row, but also in groups. This means that we need to be able to create DB images at certain points of the test execution and then be able to start the application from there again and again.

If you think about it, this will also improve maintainability, as the error can occur almost at the end and you don’t need to re-run all the tests to verify that your fix works.

In SpringBoot we locally use H2 and in order to keep snapshots you can, for example, use a file database like this:

topo.datasource.url=jdbc:h2:file:C:/Users/marti/git/topo/projects/topo-frontend/cypress/db-images/debug;FILE_LOCK=NO;TRACE_LEVEL_FILE=0;MODE=Oracle

Once you start the app with the file database you can run the series of tests that should be executed in parallel and then stop the application and make a copy of the database for the next series of tests. E.g. start the app with a clean database, run the first third of the tests and create a copy of the database which we can use to start a second third of the tests.

This will however require keeping the database images somewhere consumable by developers and CI jobs as well. As the database image can be large in size, it makes sense to git ignore them, upload them to your artifactory, and download from there.

This can be scripted so that before you start the backend you download the database image first.

cypress/e2e-serve-backend.sh

#!/usr/bin/env bash

cd $WORKSPACE

curl -u $NEXUS_USERNAME:$NEXUS_PASSWORD https://artifacts.seag.cloud/repository/cypress-h2-images/$2.mv.db -o $WORKSPACE/projects/topo-frontend/cypress/db-images/$2.mv.db --create-dirs

mvn -s $MAVEN_SETTINGS --activate-profiles h2 -pl :topo-server org.springframework.boot:spring-boot-maven-plugin:run -Dspring-boot.run.arguments="--info.map.mode=json --server.port=$1 --spring.profiles.active=MockedMessaging,cypress --topo.datasource.url=jdbc:h2:file:$WORKSPACE/projects/topo-frontend/cypress/db-images/$2;MODE=Oracle --topo.liquibase.user=sa"

2.2. Serve frontend multiple times

Unfortunately, Angular CLI does not provide a way to serve the application multiple times without rebuilding it over and over, therefore, you need to find a way to serve it from a single build and configure middleware such as proxies differently. In this case, we can build an Angular application and then serve it using browser-sync and http-proxy-middleware multiple times, on different ports.

cypress/e2e-serve-frontend.js

var browserSync = require('browser-sync').create();
var createProxyMiddleware = require('http-proxy-middleware').createProxyMiddleware;

var port = Number.parseInt(process.env.CY_PORT);
var backendPort = 38080 + (port || 0);
var frontendPort = 4110 + (port || -10);

browserSync.init({
  single: true,
  port: frontendPort,
  server: {
    baseDir: './dist',
    port: frontendPort,
    middleware: ['/api', '/actuator'].map(
      function(path) { return createProxyMiddleware(path, {
        target: 'http://localhost:' + backendPort,
        changeOrigin: true,
        logLevel: 'debug',
        secure: false,
      });
    })
  },
  ui: false,
  open: false,
  logLevel: 'debug'
});

2.3. Running the tests

In the previous steps, we created scripts to start the frontend and backend on specified ports and download the database image it should use. Now is the time to write some configurable scripts to run both of them at the same time, specifying the port, database image name, and tests it should execute.

"e2e-serve": "start-test 'sh ./cypress/e2e-serve-backend.sh 3808$CY_PORT $CY_IMAGE' http://localhost:3808$CY_PORT/swagger-ui.html 'node cypress/e2e-serve-frontend.js' 411$CY_PORT",
"e2e-run": "npm run e2e-serve 'cypress run --spec \"$CY_SPEC\" --browser chrome --config baseUrl=http://localhost:411$CY_PORT --reporter cypress/reporters/junit-polarion.js'",

As you can notice, we use environment variables to configure the port CY_PORT,CY_IMAGE, and what spec files should be executed with CY_SPEC.

As we build the app using Maven, the step that is then used in the CI is specified and configured in the POM file. E.g. following execution is running the second third of the tests which are placed in a folder with the name 02, and specifies port increment 2 (38080 + 2 on the backend and 4100 + 2 on frontend)

<execution>
  <id>e2e-2</id>
  <goals>
    <goal>npm</goal>
  </goals>
  <phase>integration-test</phase>
  <configuration>
    <arguments>run e2e-run</arguments>
    <skip>${skipE2ePatchset}</skip>
    <environmentVariables>
      <CY_PORT>2</CY_PORT>
      <CY_IMAGE>00_part2_v1</CY_IMAGE>
      <CY_SPEC>cypress/integration/02/**/*.spec.ts</CY_SPEC>
    </environmentVariables>
  </configuration>
</execution>

2.4. CI job

The only thing left at this point is creating your CI job so it executes all the steps with eager fail if any of the steps or parallels fail.

E.g. Our Jenkins job builds frontend and backend in parallel and then runs all Cypress test groups in parallel. As you can see from the first build, it saves us about half an hour as all thirds take around 15 minutes. The costs however go up, as these two out of three parallels use their own agent. In the second build, you can notice that we skip the parallels in case one of them fail and doesn’t run the build any further.

Further improvements and gotchas

In case you don’t write a proper database migration you can apply on top of the database images created with an older version of the software, you would need to create new images over and over. As a further improvement, I would suggest automating the creation of the images with custom Cypress commands, but one would need to get around the DB file lock as it is not clonable during the application run.
Last but not least, let me know if you do something similar, or if you think this might help your testing processes despite the higher execution costs.