Page tree
Skip to end of metadata
Go to start of metadata

Continuous delivery and testing

GoodData, and our CloudConnect team in particular, is always working to deliver valuable and well-tested features in the shortest possible timeframe. Following the Continuous delivery principles.

To make this possible, we use a number of tests:

  • unit tests
  • integration tests
  • acceptance tests (aka “rest tests” in our terminology)
  • UI tests

These tests allow us to remain confident that our code is well designed and works properly.

Why we need performance tests? (A true story)

It all started when one of our REST API resources displaying aggregated data from all users’ projects had to be rewritten to support mutliple datacenters (both AWS and Rackspace) – the goal being to see data from GoodData projects located on both datacenters. This change required to make more (parallel) HTTP calls than ever before.

To compare performance of the new solution to the previous one I performed some manual (and quite silly) performance checks. For one user I loaded a lot of data and called the resource multiple times while checking the response time. So far so good: the response times seemed to be pretty similar for both solutions, so we decided to deploy the new version to production…

Things always go wrong

Immediately after the deployment of the new version new problems arose. The number of requests that failed with Internal Server Error (500) status was growing exponentially. We had only one option - revert to the previous version.

Back to the drawing board

To learn from our latest experience I decided to take a different approach:

1. Test first

A number of considerations were accounted for when deciding what kind of test to use:

  • As with TDD, we now knew to create a performance test for the current solution before actually changing any code.
  • More importantly, we’d have to create a realistic test case simulating concurrent users. Testing performance on a single user would not be sufficient.
  • Performance tests would have to be automated. Manual testing is cumbersome, error prone, and inaccurate.
  • Automation would need to be done via our Jenkins infrastructure so candidates for performance tools should be able to integrate with Jenkins.
  • Ideally, the test would be able to generate various reports such as standard junit test results.
  • The tool shouldn’t eat too many resources since our Jenkins slaves are used for other tasks as well.
  • It should be open source – no heavy commercial testing tools.

Gatling - The Winner!

Having a little experience with both JMeter and Gatling before, the choice soon became obvious:

  • Gatling’s clean DSL API (written in Scala) as opposed to the JMeter misuse of XML. XML is not a programming language and should not be used that way. Using DSL is a great advantage since we want to treat performance tests as a code.

  • High performance - gatling uses asynchronous concurrency (Scala, Akka, Netty) and asynchronous IO. This approach scales a lot better than the standard stress tools using one thread per user.

  • Gatling is more modern and viable alternative, it appeared on ThoughtWorks technology radar in 2013 and 2014.

As mentioned in Gatling stress tool made efficient, the primary motivation for Gatling’s creators was to make a stress tool that was more efficient and had a clean API.

JMeter and Gatling benchmark reveals real difference between JMeter and Gatling scalability.

Gatling project setup & implementation

Gatling is available as a standalone package from Gatling project page. Right now we are using the stable version 1.5.2. It comes with a handy GUI to defining the simple test cases. However this is not the case we are going to describe here - instead, we want to focus on real code.

Because most of our projects are using Maven as a build tool and Jenkins supports it out of the box, I decided to use it for our upcoming performance tests too - check simple gatling maven project

The test itself is relatively simple and concise (some parts omitted for brevity):

/**
* Perform concurrent requests to resource with different users from user_credentials.csv.
* For each user the requests are repeated with given pause and count.
*/
class ConcurrentUsersSimulation extends Simulation {
val hostUrl = Config.host
 
// how many requests per user
val requestsNumber = 20
 
// pause between two subsequent requests for the same user
val requestPauseSeconds = 15
 
// load users' (name, password) combination
val usersDataSource: Array[Map[String, String]] = csv("user_credentials.csv")
 
// perform requests for each user from credentials files in parallel
    private val usersCount: Int = usersDataSource.length
      val scn = {
        scenario("resource test: host=" + hostUrl + ", users_count=" + usersCount)
          .feed(usersDataSource)
 
            // at the beginning we need to find user id - note the "trick" with saving the result into the "userId" variable
            .exec(http("getUserId")
              .get("/gdc/app/account/bootstrap")
              .basicAuth("$evgenia.bogranskaya", "${password}")
              .check(jsonPath("$.bootstrapResource.current.loginMD5").find.saveAs("userId"))
            )
 
            /*
             * the performance testing itself - perform multiple requests for each user
             * - requests for one user are not concurrent, there is only one request per user at time
             * - there are multiple concurrent requests for different users
             */
          .repeat(requestsNumber) {
          exec(http("processesView GET")
            .get("/gdc/app/account/profile/${userId}/dataload/processesView").basicAuth("$evgenia.bogranskaya", "${password}"))
            .pause(requestPauseSeconds)
        }
  }
 
    setUp(scn.users(usersCount).ramp(20).protocolConfig(httpConf))
 
  }

 

I won’t cover the Gatling basics - to get a quick grasp about Gatling check the Getting Started guide.

Test steps:

  1. Load user accounts for testing from csv file.
  2. Get user ID from REST API and save it to the userId variable for later reuse.
  3. Repeat REST API calls for requestNumber times with pause requestPauseSeconds between each call and do it concurrently for given users.

Most of the calls to our API have to be authenticated. That’s not a big deal - I can simply add basicAuth with username and password read from CSV file. The interesting part is construction of the full URI because I need the to fetch the user ID via rest api at first. This is done in a separate request before the .repeat(requestsNumber) part. Id is saved to the variable userId and reused afterwards via the handy syntax ${userId}

Results

Gatling jenkins plugin can help to visualize the mean response times:

Gatling test results in Xunit format

What’s more important for us is to be able to determine and easily display the overall result of performance tests run as well as the number of failed requests in Jenkins. The standard way for doing this in Jenkins is to produce test results in Xunit format. Unfortunately, in the case of Gatling the only common place where the test results can be found is gatling log.

After digging into this problem a little bit deeper I decided to create groovy script to transform results from the Gatling log to the XUnit format supported by Jenkins.

This way we can get better and quicker overview overall test results.

Test results trend

The following image shows overall trend of one performance results. The red color represents failed requests. We can see that after some problems we were able to stabilize our resource, lowering the number of failed requests significantly.

Result of one execution of performance tests:

This image demonstrate a result of one test execution. The total number of requests is 480, with 10 requests failed (because of timeout).

Summary

Gatling and performance testing helped us to reveal and (partially) fix problems in our infrastructure. We were able to simulate production load pretty closely and after the deployment of a new improved version of the problematic resource, the number of failed requests came close to zero.

There are still some problems that needs to be solved but performance tests helped us discover those issues and identify potential bottlenecks.

Resources

  • No labels