And here comes the first experiment with DBTestingSuite :).
Before going through the details, I want to tell about two tests I have written on the suite.
- KeyValueCorrectness: Actually this is a straightforward test to check the correctness of a Key/Value DB or its binding. It writes and reads random values on random keys and asserts the equality of value that was written and read. This seems somewhat ridiculous (a DB should achieve this, why should I use a DB that cannot even give me what I have supplied before :)), but this test can also be used to test the bindings or clients of a DB brand, some of which are implemented by community people (I may need this :)). The test is really dummy and extends the non-threaded version of Key/Value DB test abstract.
- KeyValuePerformance: I have been reading about load testing on DBs. If you want to push a load on a DB and test the performance of it, you should avoid overheads and focus on the pure performance of the DB. Using the local loopback (127.0.0.1) for requests achieves minimum ping time. Of course the size of the values in Key/Value DBs is a point to consider. So I left it customizable. Most DBs respond to the requests via multi-threading (the number is decided on starting time or dynamic). KeyValuePerformance is a threaded test composed of write and read phases. When testing the write performance, it fires x threads to write n data values at total, of length l. The number of threads can be chosen considering the number of cores on the machine, or just a number that supplies enough CPU load, where “enough” equals “maximum” :) (to be watched from Activity Monitor :)). n majorly decides on the test duration, but of course test calculates the average writes per sec, which is the actual number that we are looking for. The read phase works similar. I checked the performance test overhead on a large n (and an average l), and total write overhead was 86 millisecs and read was 46 millisecs, which is really negligible when a proper test’s duration is 5-10 secs. Also these overheads will be equal for all DB brands (or clients).
I have used JRedis as the java client for Redis.
The testing environment is again my little white macbook :)
2.4 GHz Intel Core 2 Duo (3MB L2 Cache, Bus at 800 Mhz)
4 GB DDR2 SD Ram at 667 Mhz
150 GB Hitachi Disk on SATA (1.5 Gigabit)
And here are the results for Redis, where x = 2, n = 10000, and l = 32:
Redis connection initialized.
KEY-VALUE CORRECTNESS TEST
Correctness OK!
Redis connection shut down.
Redis connection initialized.
Redis connection initialized.
KEY-VALUE WRITE PERFORMANCE TEST (2 threads)
Total write time: 742 milliseconds (10000 requests)
Write performance: 13477 writes per sec
KEY-VALUE READ PERFORMANCE TEST (2 threads)
Total read time: 757 milliseconds (10000 requests)
Read performance: 13210 reads per sec
Redis connection shut down.
Redis connection shut down.
Actually the result surprised me a little bit. As far as I now (it was also told on the Redis project site), Redis can achieve speeds like 110000 writes per sec and 80000 reads per sec. Then I ran the Redis Benchmark that comes with the package:
atasay-gokkayas-macbook:redis-1.0 rincewind$ ./redis-benchmark -q -n 10000
SET: 28194.37 requests per second
GET: 26525.20 requests per second
INCR: 28850.14 requests per second
LPUSH: 30183.74 requests per second
LPOP: 29618.34 requests per second
PING: 31734.18 requests per second
The benchmark results shown that Redis was performing about 28000 writes per sec and 26500 reads per sec on my environment, which dramatically relieved my surprise. The difference between the numbers on project site and numbers from the ./redis-benchmark was probably due to the environment. And I think the speed difference between my results and the ./redis-benchmark was sourced from the language (benchmark was written in C), and/or the client that I used.
Another important point that both my tests and ./redis-benchmark together show is that Redis performs better in writes. In default configuration, Redis writes on memory and periodically syncs with the disk. This can be very helpful in some use-cases and shows a strong difference of Redis, while it also constitutes a trade-off.
If I have time, I’ll try the same tests on Redis with different clients. And of course with other Key/Value DBs :).
Kthxbai.
