Snappy Testing

Alright. I've finally got around to doing some Snappy testing. I've gotten some amazing results, but also noticed a bug. My current code seems to hang after a large amount of compression. It specifically hangs when i am doing an iozone benchmark. The 500MB pass (I set it to 4GB). But before I ran iozone, i did some paliminary testing vs lzjb, and I have to say, the compression performance is higher with Snappy. I was able to get a 23.14  compression ration instead of the 15.22 ratio from lzjb, with similar cpu usage.

I suspect this has to do with the snappy_env_init method in the snappy-c library i found. It seems fishy having to initializing the environment for every compression event. I may look into it further but it does not seem very scaleable. and given that ZFS compression works at the block level, and not the file level, there are a lot of compression events happening for a single file (instead of one event, compressing a 400MB file...it would be 400 events with a zfs file sysem that is allocated using 4k blocks).

Anyway, I may or may not delve further into this. I think I could actually call this a finished project. Its in a usable state. Anyone can grab it from github and run snappy with zfsonlinux. Snappy could be a real contender in the ZFS space. I might even think about adding other algorithms to zfs. 7z, bzip and ARJ would be fun.

Comments

Post a Comment

Popular posts from this blog

So I've again been bitten by the Transcoding/transpiling bug...Doh! My history with Transcoding  A few years back (around mid-2007), I was enamored with a new language on the block, JavaFX (now vala). Alas, I had moved on from the Java platform to the .Net platform during that time, and Silverlight just hit the stage (and WPF). I was more of a fan of the declarative and expressive nature of JavaFX than I was of XAML, so I looked into porting JavaFX script to the .Net platform (and later the DLR). I halted my work in 2008-2009, as JavaFX script was dying, and the Parser technology I was using (a great C# based parser called Irony) was giving me issues when integrating with the DLR. The LLVM Project I've been a fan of the LLVM project since about 2009. I have also watched its MSIL backend go into disrepair and interest in C# as part of the vmkit wane. I answered some stack overflow questions in 2011 about this http://stackoverflow.com/questions/5054938/how-to-translate-
Read more

AutoConf and M4 bliss

Just wanted to make a statement that the CLI tooling for building software in linux (and really any *nix) has changed dramatically. In my mind, configure and make were all black boxes and it was very very hard to actually effect them as an independent developer once they were baked. This is not the case anymore. I was able, with autoconf, automake, and m4 templates (and of course some C++ glue) to actually integrate two libraries (zfs and snappy-c) that know nothing about each other. This is a testament not just to the auto tooling, but the level of maturity of a lot of open source software out there. While make is still technically a black box (yes, make scripts are simple in concept, fine...but go look at Illumos makefiles and we'll talk...), it may be ok for make to stay in the shadows...like some x86 assembly that gets generated on the fly... I just might, after this zfsonlinux + snappy release, revisit illumos. But from what I recall of the original opensolaris documentati
Read more