Github Actions, Bitbucket Pipelines and CircleCI are cost-effective solutions for smaller projects and teams, but when your test suites grow in size and complexity the costs can start to mount. This is particularly true with Cypress and Playwright tests due to their inherently asynchronous nature: when a condition fails, both libraries will automatically retry for a number of seconds before marking the test as failed, so any failures will increase the overall runtime and cost.
Self-hosting the tests on your own infrastructure can be a good way to cut costs, but configuring the platform to autoscale properly can be complex, as ideally we want the platform to spin up a number of runners for each run to ensure that we can run the tests in parallel, and then to scale back down afterwards. Also while Playwright has built-in support for sharding, Cypress does not, and so you would need an additional subscription to Cypress.io to enable this.
Having used Kubernetes in previous backend projects I realised that Kubernetes Jobs would be a perfect solution to this, particularly if we could use Spot VMs for the lowest possible costs. Kubernetes Job can be configured to run multiple parallel tasks, and clusters autoscale to match demand, so we shouldn’t incur costs when we’re not running tests.
All that I needed was a system that could parcel out the set of test files to the available set of test runners and could handle Spot VM behaviour. Unfortunately such a system didn’t exist, so I had to create it…
Step forwards Cykubed
Cykubed dramatically simplifies setting up a CI/CD system for both Cypress and Playwright. Simply sign up at app.cykubed.com using your Github or Bitbucket account, copy and paste a single Helm command to deploy the Cykubed agent onto your Kubernetes cluster, and from then on all your test builds will be run automatically as Kubernetes Jobs. Cykubed will collect the test results and display them in the dashboard, and it seamlessly integrates with Slack and Github / Bitbucket pull requests.
The built-in auto-scaling in Kubernetes means that when you’re not running tests, your cluster will just scale down to the minimum number of nodes. Another huge advantage of running in Kubernetes is Spot VMs: they are often up to 70% cheaper than the standard VM price, with the caveat that they can be reclaimed at any time when the resources are required.
In practice that’s not an issue for Cykubed: a worker will be notified ahead of time that it needs to terminate, which gives it time to return the file it’s working on back to the queue to be handled by another worker.
Comparing Cykubed with hosted CI
Let’s assume that you have a large suite of Cypress tests (for example 200 test results i.e “it()” clauses), that take 60 minutes to complete when run end-to-end.
We’ll assume for this example you have a team of 10 developers and that they are pushing commits twice a day. Cypress recommend a minute of 2 CPUs and at least 4GB ram (although for more complex apps you may well need more ram, which can force you onto a more expensive VM).
CPU minutes used daily = 60 x 10 x 2 x 2
= 2400 CPU minutes
CPU minutes per month (24 days) = 24 * 2400
= 57600 CPU minutesComparing like-for-like on different CI/CD platforms is tricky as they’re all structured differently and have different free tiers. I’ll pick 3 of the most popular: Github Actions, CircleCI and Bitbucket Pipelines, normalise to CPU minutes, and calculate the cost after the monthly free tier is used up:
Github | Bitbucket | CircleCI | |
Free minutes | 3,000.00 | 2,500.00 | 6,000.00 |
Per cpu minute | $0.004 | $0.010 | $0.003 |
Monthly cost | $218 | $551 | $154 |
As you can see, the free tier doesn’t really go very far!
Now let’s compare using Spot VMs on a Google Autopilot Kubernetes cluster. Using the cheapest region (us-east1), the current spot prices for a single CPU core and 1 Gb of memory is $0.0133 and $0.00148 per hour respectively.
Monthly cost = (96,000/60) 0.0133 + 4 10 2 24 * 0.00148
= $22!Cykubed needs to install a single agent into the cluster. The agent itself requires minimal resources to run, but Autopilot requires a minimum of 0.25 vCPU and 0.5 GiB ram, so that will add another $16 per month. This still leaves it vastly cheaper than any of the alternatives!
Now we need a subscription to either cypress.io or cykubed.com to orchestrate the tests (i.e to run them in parallel and display the results on a webpage). Both platforms base their pricing on the number of test results (i.e “it()” test clauses), and both include a certain number in each tier before they start charging for additional tests each month.
As we have 200 tests in our project, we can calculate the number of test results each month:
Monthly test results = 200 x 10 x 2 x 24
= 96,000 tests We would need the Team subscription on both platforms due to the number of users, but the prices are wildly different:
Cypress.io = $615 / month
(10k test results included plus
$540 for the additional 90k test results)
Cykubed.com = $120 / month
(60k test results included plus
$20 for the additional 36k test results)Summary
To sum up: if you went with the cheapest CI/CD option (i.e Github) and used Cypress.io you would be paying $833 per month.
If you use Cykubed to run the tests in your own Kubernetes cluster using Spot VMs, you could pay $160 per month: that’s a saving of over 80%! You can also run as many tests in parallel as you like, which means your tests take far less time to complete, for the same cost.
You can try Cykubed for free: sign up at https://app.cykubed.com today.
Commentaires