Loving me some redis…

I just had to tell the world about this sweet new database system.  redis is a non-volatile key-value database with speeds comparable to memcached.  For the right types of applications, it’s just perfect.

I am building an application that needs to be able to move objects out of and back into a key-value storage subsystem at extremely high rates.  I don’t need an ACID-compliant database; this application can tolerate some data loss.  But I do need extremely high throughput and scalability to handle bursts in traffic.  And not only does my application read a whole lot, it writes a whole lot, too.  Read/write ratio is nearly 1:1.

 

I’ll do a quick rundown of the alternatives I examined:

  • MySQL memory tables – would be really cool, but don’t support blobs or large text values, so storing large data objects is not possible in a memory table
  • MySQL InnoDB tables with massive RAM buffer – I have had some good luck with InnoDB in other high-volume applications — it’s quite good about using its buffers to minimize the amount of disk I/O it has to perform.  But for this application, it couldn’t keep up
  • memcached with application-layer logic to keep track of keys in a MySQL memory table – ugh.  It could work, but it’s absolutely nasty.

Just when I was about to resign myself to this last hopelessly inelegant solution (in fact, I built a working version), I did one last scan around the Web for a memcached-like application that is persistent.

I ran across Tokyo Tyrant and memcachedb, but neither really seemed like what I wanted.   Memcachedb requires really current Berkeley DB libraries, which is an issue if you’re using a stable enterprise Linux distro.  Also, BDB has maximum record sizes, and I want to be able to store arbitrarily large data objects in my application.  Tokyo Tyrant looks more promising, but it seems to be lacking documentation, and I don’t really feel like blazing new trails to get it built and installed.  Ultimately, I settled on redis, by Salvatore Sanfilippo.

Redis is very accessible for a newcomer to the system.  For such a new project, it has solid documentation.  It’s also ridiculously simple in its structure — you don’t even need autoconf to compile it under Linux.  In fact, its source is so streamlined, it’s hard to believe.  It only consists of 21 *.[ch] files, and fewer than 7500 lines of code.  On my 64-bit system, the compiled server binary is only 230KB!

Keeping with the simple theme, redis doesn’t even need a config file to run (although it does support one).  It also ships with a handy command-line client that you can use to issue commands to the redis server, which really comes in handy when you’re testing or monitoring a redis server.

What’s more, it has some really sweet primitives that let you do more than just store simple strings as values.  Its support for sets and lists is really intruiging.  I haven’t used them directly, but it doesn’t take much imagination to see the potential applications for such in-server data structures.

Redis has a wide variety of client libraries: PHP, Perl, Java, Python, TCL, Ruby, and more.  These haven’t all solidified, but I think they’re moving forward fairly quickly.

I’m using redis now in some limited experiments on our production servers.  It’s doing about 100 commands per second, moving objects of about 1KB in and out of redis.  The redis database has almost 300K key/value pairs in it, and it’s taking up about 290MB of RAM (and disk, of course).

redis works by storing everything in RAM.  Think of it as a memcached key/value store where things don’t ever get bumped out.   And, oh, yeah, you can query the system for all the keys (a feature request the memcached guys are constantly batting back — for many good reasons).  Periodically, redis will launch a thread (or maybe a subprocess, not sure) to write the contents of the memory to disk.

If redis were to crash, you would lose any changes to the database since the last flush to disk.  You can tune it to minimize the risk, but if your application can’t tolerate any data loss, this is the wrong tool for you.  But for some classes of applications, this might be the perfect database system.

Leave a Reply

Your email address will not be published. Required fields are marked *