Apache Bench To Blame, Not AWS

Some days ago I reported on what appeared to be a problem with the Ada Web Server (AWS) project:

After a lot of testing, it now appears that this specific behavior has nothing to do with AWS, and everything to do with Apache Bench.

I grabbed a copy of the Apache Bench sources, and took a good look at them, and I think I actually managed to track down what looks like a classic race condition. When things are going really fast, Apache Bench can actually start more connections than requested by the -n parameter, and the data returned on these excessive connections are happily added to the accumulated totals by Apache Bench.

At this point it’s probably a good thing to note that my C skills are, at best, pretty bad, which is why I’ve send my speculations to the AWS maillist, where I’m sure some more experienced developers will let me know if I’m wrong or right.

What does support my findings, is that httperf does _not_ exhibit this behavior. Both I and Pascal have not been able to reproduce this problem using httperf.

So at this point, the finger of blame is pointing at Apache Bench.

AWS Bug Confirmed By Pascal Obry

### UPDATE ###

After more tests and a closer look at the ApacheBench sources, we now know that this problem stems from a race condition in ApacheBench.

### END UPDATE ###

The AWS bug I mentioned in my previous post has been confirmed by lead AWS developer Pascal Obry:

Just to let you know that I have been able to reproduce on a fast
GNU/Linux machine.
Feel free to debug that on your side, it will help understanding and
fixing this issue.

Very nice. Now all we have to do is wait for a fix, which I’m sure is right around the corner. The power of open source is amazing!

AWS being very generous with its bytes

### UPDATE ###

After more tests and a closer look at the ApacheBench sources, we now know that this problem stems from a race condition in ApacheBench.

### END UPDATE ###

While doing my AWS vs node.js test I stumbled on a somewhat weird issue: Sometimes AWS returned too many bytes.

I reported this to the AWS authors, who came back with this response:

This looks very very bad and does not correspond to any experience I have had with AWS. I’m really concerned by these results. How does your AWS server look like?

Luckily I’d tested this using both my homegrown AWS powered web-framework, and the demos/hello_world example from the AWS source package. Both exhibit the same behavior.

I did some more tests, this time using both 32 and 64 bit systems, and I even threw in a virtualized Slackware (VirtualBox 3.2.12) for completeness. I then reported back to the AWS maillist.

This is a very odd issue indeed, and I truly hope the AWS developers can figure out the what and why, but until that happens, I’ll soldier on. AWS is no less of a marvelous piece of software because of this minor glitch.