TestnScale Blog

In this article, we will take a look at how typical load generation tools measure response times and point out the issues with this method :

The typical load generation script executes a path as illustrated in this simplified pseudo-code below:

   while (!done) {
        x = get time
        response = make request
        y = get time
        response time = y - x
        sleep thinktime
    }

Looks reasonable! But the devil is in the details. Let’s look into some of the issues:

• When ‘make request’ is called, is it really making a request at that time? Assuming this is a http request, the client system will need to format your request, create the http headers, and write the request to the wire. After the response is received, the client needs to check the response, strip the headers, and return the response data. While this all looks trivial, the response time you’re getting actually includes a good amount of client processing. A slow client will give you slow response times while a fast client interacting with the same server will give you fast response times. This issue will be exaggerated as you deal with more complicated client logic, such as sending a SOAP web service request. The client will need to deal with generating the XML before making the request and parsing the XML after getting the response. The difference between a slow and fast client becomes much more apparent in such cases.
• When you ask the load generator to sleep for a certain amount of time, we normally assume the process/thread is awakened exactly after the specified time elapses. On all but a very few systems, the sleep contract is actually a minimum sleep. If you ask to sleep for 10 seconds, you’ll wake up at least 10 seconds after the time you go to sleep. You may actually wake up any time after 10 seconds. It could be 10.1 seconds, 11 seconds, or even 15 seconds. Well, I’m exaggerating in the last case. But that would still be legal. When you deal with sleeps in the millisecond level, these effects become rather serious. It is actually very common for a sleep fo 10 milliseconds to wake up only after 15 milliseconds. The more you sleep, the less load you place on your test. Moreover, most tools don’t even tell you how much time was really spent sleeping (i.e what the actual think time was). So you may get better response times and less throughput not knowing that this is caused by the load generator spending all the time sleeping.

Now, you may start wondering how to solve such issues. If I got your heads spinning and your brows furrowed now, good. Just stay tuned. We’ll get to our solution, soon.

Tags: loadtesting, tool

This entry was posted on Sunday, August 1st, 2010 at 12:12 pm and is filed under Tools. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.