One of the most common load testing mistakes made by beginners is their configuration of the user ramp rate. This is best illustrated with an example configuration that we see frequently. This chart (from the load test report produced by our load testing software) shows the min/max/average page durations (pink, red, tan) along with the number of virtual users (blue) running the test:
What information does this chart give us about the system being tested?
By contrast, look at this chart from another load test report:
This chart tells us:
To summarize – the first test tells us only that performance is not good at 30 users. If 30 users was the goal, this test has not provided any information to indicate how far away we are from reaching that goal. If the goal in the second test was to support 500 users, it is easy to see that the system can currently support at least 40% of the goal – and there is probably a lot of optimization work remaining.
Note that the test configuration shown in the first example can be useful. This kind of test is usually referred to as a soak test. The purpose of a soak test is to run for a long time at the anticipated load to see if the performance degrades over time. This is frequently used to check for memory leaks or limits on other resources. However, soak tests are generally not used until the system is meeting the performance requirements.
When using a stepped ramp, it is important to hold at each user level for long enough to allow every page in all the scenarios to be exercised – usually at least several times. If all pages are not measured during the step, conclusions drawn from the results may be faulty.
Chris Merrill, Chief Engineer