A benchmark comparing JavaEE and NodeJS made the buzz lately on the web. I was surprised by the results and decided to reproduce it to verify an intuition. Also, the article was followed by multiple comments that are themselves worthy to be commented. Which brings us to the current blog post.
But what is a benchmark? A benchmark is meant to measure the performances a piece of software. An attempt to reproduce in laboratory what will happen in production. If you ask the domain experts, they will tell you that benchmarks are a really dangerous, even though useful, tool. And that they are pretty much providing false results. From my point of view, it is a bit exaggerated but not that much as we will see below.
However, one thing is certain. Benchmarks must not be taken lightly and some precautions are in order. For instance:
- Have database volumes similar to the target
- Have a data set sufficiently heterogeneous and meaningful to prevent any cache induced bias
- Hardware with at least proportional throughput when compared to the target
Note that, as opposed to popular belief, benchmarks are generally optimistic. If you have bad response times during your benchmark, you can bet that it won’t improve in production. You should beware the opposite. Good response times during a benchmark do not guarantee good response times in production. Because the benchmark might be flawed.
Another interesting topic is (more…)