Continuous delivery and testing
GoodData, and our CloudConnect team in particular, is always working to deliver valuable and well-tested features in the shortest possible timeframe. Following the Continuous delivery principles.
To make this possible, we use a number of tests:
- unit tests
- integration tests
- acceptance tests (aka “rest tests” in our terminology)
- UI tests
These tests allow us to remain confident that our code is well designed and works properly.
Why we need performance tests? (A true story)
It all started when one of our REST API resources displaying aggregated data from all users’ projects had to be rewritten to support mutliple datacenters (both AWS and Rackspace) – the goal being to see data from GoodData projects located on both datacenters. This change required to make more (parallel) HTTP calls than ever before.
To compare performance of the new solution to the previous one I performed some manual (and quite silly) performance checks. For one user I loaded a lot of data and called the resource multiple times while checking the response time. So far so good: the response times seemed to be pretty similar for both solutions, so we decided to deploy the new version to production…
Things always go wrong
Immediately after the deployment of the new version new problems arose. The number of requests that failed with Internal Server Error (500) status was growing exponentially. We had only one option - revert to the previous version.
Back to the drawing board
To learn from our latest experience I decided to take a different approach:
1. Test first
A number of considerations were accounted for when deciding what kind of test to use:
- As with TDD, we now knew to create a performance test for the current solution before actually changing any code.
- More importantly, we’d have to create a realistic test case simulating concurrent users. Testing performance on a single user would not be sufficient.
- Performance tests would have to be automated. Manual testing is cumbersome, error prone, and inaccurate.
- Automation would need to be done via our Jenkins infrastructure so candidates for performance tools should be able to integrate with Jenkins.
- Ideally, the test would be able to generate various reports such as standard junit test results.
- The tool shouldn’t eat too many resources since our Jenkins slaves are used for other tasks as well.
- It should be open source – no heavy commercial testing tools.
Gatling - The Winner!
Having a little experience with both JMeter and Gatling before, the choice soon became obvious:
Gatling’s clean DSL API (written in Scala) as opposed to the JMeter misuse of XML. XML is not a programming language and should not be used that way. Using DSL is a great advantage since we want to treat performance tests as a code.
High performance - gatling uses asynchronous concurrency (Scala, Akka, Netty) and asynchronous IO. This approach scales a lot better than the standard stress tools using one thread per user.
Gatling is more modern and viable alternative, it appeared on ThoughtWorks technology radar in 2013 and 2014.
As mentioned in Gatling stress tool made efficient, the primary motivation for Gatling’s creators was to make a stress tool that was more efficient and had a clean API.
JMeter and Gatling benchmark reveals real difference between JMeter and Gatling scalability.
Gatling project setup & implementation
Gatling is available as a standalone package from Gatling project page. Right now we are using the stable version 1.5.2. It comes with a handy GUI to defining the simple test cases. However this is not the case we are going to describe here - instead, we want to focus on real code.
Because most of our projects are using Maven as a build tool and Jenkins supports it out of the box, I decided to use it for our upcoming performance tests too - check simple gatling maven project
The test itself is relatively simple and concise (some parts omitted for brevity):
I won’t cover the Gatling basics - to get a quick grasp about Gatling check the Getting Started guide.
- Load user accounts for testing from csv file.
- Get user ID from REST API and save it to the
userIdvariable for later reuse.
- Repeat REST API calls for
requestNumbertimes with pause
requestPauseSecondsbetween each call and do it concurrently for given users.
Most of the calls to our API have to be authenticated. That’s not a big deal - I can simply add basicAuth with username and password read from CSV file. The interesting part is construction of the full URI because I need the to fetch the user ID via rest api at first. This is done in a separate request before the
.repeat(requestsNumber) part. Id is saved to the variable
userId and reused afterwards via the handy syntax
Gatling jenkins plugin can help to visualize the mean response times:
Gatling test results in Xunit format
What’s more important for us is to be able to determine and easily display the overall result of performance tests run as well as the number of failed requests in Jenkins. The standard way for doing this in Jenkins is to produce test results in Xunit format. Unfortunately, in the case of Gatling the only common place where the test results can be found is gatling log.
After digging into this problem a little bit deeper I decided to create groovy script to transform results from the Gatling log to the XUnit format supported by Jenkins.
This way we can get better and quicker overview overall test results.
Test results trend
The following image shows overall trend of one performance results. The red color represents failed requests. We can see that after some problems we were able to stabilize our resource, lowering the number of failed requests significantly.
Result of one execution of performance tests:
This image demonstrate a result of one test execution. The total number of requests is 480, with 10 requests failed (because of timeout).
Gatling and performance testing helped us to reveal and (partially) fix problems in our infrastructure. We were able to simulate production load pretty closely and after the deployment of a new improved version of the problematic resource, the number of failed requests came close to zero.
There are still some problems that needs to be solved but performance tests helped us discover those issues and identify potential bottlenecks.
- simple gatling maven project
- Gatling Wiki - Getting Started guide
- Stress testing JMeter and Gatling shows an interesting comparison of JMeter and Gatling performance.
- ThoughtWorks radar mentions gatling as a tool that is worth to try.
- Gatling: stress tool made efficient is an interesting story about the gatling development and original reasoning behind it
- Gatling 2 - What’s up Doc is a good presentation about gatling on scala.io conference. Mentions some internals and future plans.
- Gatling jenkins plugin
- Jenkins junit xml support
- Gatling gun