Benchmark web applications alternatives

There are a few different ways to do benchmarking and load testing on parts of your web application stack such as the web server, the database, the cache layer etc. This article shows a somewhat comprehensive list of the alternative solutions along with some pros and cons. You are encouraged to explore those more in depth and find the solution that suits you best.

Benchmark – two load testing approaches

There are two roads you can take doing that: using a single machine for the load testing or distributing it. Using a single machine to run the tests is generally easier and more hassle free, since all that is required is installing the software on that one machine and running the test suite (ensuring good internet connection). Its generally done from another server which is not in the same datacenter to avoid biased results. More recently however with the widespread adoption of the cloud many third-part service providers have appeared that offer distributed load testing services very cheap or for free (with some limitations). There are also some tools that can be used to roll out your own distributed test rather easily.

Distributed load testing tools

One of the more notable providers – loader.iois a service provided by SendGrid Labs, the guys who came up with the hugely popular mail delivery service SendGrid. The nice thing about loader.io is that it has a free tier (limitations might change, so check the link above for the pricing and restrictions), but generally even with the limitations is very usable and can provide a good insight, if you are testing only one application and/or a couple of URLs and dont need anything fancy. Being a third party service, you don’t need any specific setup, but note that you will have to verify your domain (similarly to google domain verification), before you can run any tests against it. Their pro tier is currently only available with a monthly subscription and it’s not on the cheap side ($99 at the time of writing this), which makes it not very suitable for people who want to run tests ad-hoc here and there.

A competitive service is Blits.IO. It’s very similar to loader.io, but their free tier is a bit worse. That said, they offer a pay-as-you go plan, based on credits, so you don’t need to subscribe in order to use the service for more comprehensive tests. Subscriptions are also based on credits, but they come up at a discount. The UI for both services is pretty good and user friendly, however the development tools and API that blitz offers is much better and comprehensive. The service is entirely oriented towards developers and integrating it in the development lifecycle, with api client libraries for most popular languages. As an added bonus, it also supports application monitoring with different types of loads.

If you are the kind of person who doesn’t want the fancy graphs and reports and prefers to do it himself, you may want to have a look at the beeswithmachineguns python load testing tool. It was originally developed for use by the Chicago Tribune and was later open-sourced. This tool requires python (with some other dependencies) and an AWS account. It basically spins as many AWS micro instances as you need and runs ApacheBench(ab, which we will review later on) on every single one of them and provides a cumulative report from all the instances. Depending on the length of tests and number of machines, this can be the cheapest of the three approaches, but it requires some setup and is limited by all the limitations that ApacheBench has because of its reliance on it.

Note: If you are enthusiastic, it will be fairly simple to modify the code to run another benchmark tool, instead of ApacheBench and distribute it across AWS instances.

Local load testing tools

The alternative approach is to run some applications, either on your development machine or better on some dedicated machine for that purpose with solid hardware and internet connection (depending on the amount of loads you will be putting, generally the more cores the better).

By far the most widely used tool for the purpose is ApacheBench (ab) which ships with most Apache web server distributions. The tool is very easy to use for quick and dirty tests but has some limitations:

  • you can only test against one url at a time (there are workarounds, but they aren’t really worth it)
  • there is a hard set limit on the maximum concurrency
  • it’s quite unreliable on OSX and Windows

Another alternative is Siege. It gets around some of the ApacheBench limitations, mainly by allowing you to test multiple URL at once, specified in a text file. It supports basic authentication, http and https and can run tests both based on number of requests and time. A big limitation to siege is that its implementation is based on threads, which means you cannot run many concurrent requests (thousands) and there is quite a bit of a performance hit due to the context switching. There is no support for Windows. Generally though, Siege is good enough for most purposes and unless you are testing very high loads, it might a be a good fit for you.

If all you need is maximum throughput, you can try a small neat tool called gobench. Its implemented in Go, and doesn’t include any fancy features, its sole purpose is high throughput. There are some interesting benchmarks on the project page comparing results to Siege and ApacheBench.

Prefer a gui?

If you prefer to have nicer graphs and a decent GUI, there are a few other options that provide A LOT more flexibility and better UI for testing. The two most popular once are Jmeter and Grinder. Both are Java solutions, so they work cross-platform, but both are also based on threads, which means they have the same constraints as mentioned above (limit on the concurrency and performance hit with higher concurrency numbers). The two are fairly easy to run, but you will have to do some documentation reading, since they come with a tremendous amount of options. Jmeter can be customized in a million different ways to simulate a real-world application usage during testing. The same can be done with Grinder, however it also supports a built-in scripting language for even finer control over the test (you can script your tests in Python).

A little bit better alternative to those tools in my opinion is Gatling. It also is a Java application, but it provides a neat web-based interface. It doesn’t have as many options as Jmeter to specify from the UI, but it has a powerful DSL language that you can use to script your tests (similar to Grinder, but simpler). The biggest advantage though, is that this application is not based on threads but on events and async IO, which means it can put higher loads with better concurrency.

Some of the other tools that you can check out:

  • mk

    I start to use gobench due this post, thanks!