Snappy Testing

Alright. I've finally got around to doing some Snappy testing. I've gotten some amazing results, but also noticed a bug. My current code seems to hang after a large amount of compression. It specifically hangs when i am doing an iozone benchmark. The 500MB pass (I set it to 4GB). But before I ran iozone, i did some paliminary testing vs lzjb, and I have to say, the compression performance is higher with Snappy. I was able to get a 23.14  compression ration instead of the 15.22 ratio from lzjb, with similar cpu usage.

I suspect this has to do with the snappy_env_init method in the snappy-c library i found. It seems fishy having to initializing the environment for every compression event. I may look into it further but it does not seem very scaleable. and given that ZFS compression works at the block level, and not the file level, there are a lot of compression events happening for a single file (instead of one event, compressing a 400MB file...it would be 400 events with a zfs file sysem that is allocated using 4k blocks).

Anyway, I may or may not delve further into this. I think I could actually call this a finished project. Its in a usable state. Anyone can grab it from github and run snappy with zfsonlinux. Snappy could be a real contender in the ZFS space. I might even think about adding other algorithms to zfs. 7z, bzip and ARJ would be fun.

Comments

Post a Comment

Popular posts from this blog

Linq To SQL and Change Tracking

I have been using Linq To SQL in a project recently. Linq is a great way to traverse through a collection of objects. Its a very nice utility and can get rid of those unsightly nested foreach loops with if/elses in your code. Linq To SQL is also a nice basic Object Relational Mapper. The issue is "when the rubber meets the road", as it were. Linq seems a slight bit incomplete. Take this example: A simple ChangeLog class. It has an Enum ChangeLogStates that has 4 states, Created, Modified, StatusChanged (a disposition), Shelved(basically deleted). Now, i could do an ChangeLog.add(LogStates.Created/Modified/Changed/Shelved,Object Key, strDescription) to every line of code where there is a change in the database....everywhere i insert, update the underlying entity. But Using the beauty of OOP and Linq to SQL, i can impletement the Datacontext's insert and update partial methods: partial void insertLinqItem(LinqItem item); partial void updateLinqItem(LinqItem item); in...
Read more

BB10 and Mono

I am back, with my usual mono antics! RIM recently had a conference and showed off more of BB10. I have decided I will attempt (albiet in my spare time) to use C# to create a Cascade application. I have a few ideas how i'll accomplish this. I'll post more info as time goes on.
Read more

Haiku + Mono: Almost there

After trying to compile and install mono 2.10.8 on the latest builds of Haiku, its apparent that Haiku is now getting some Mono love. Its not 100% complete, but while spelunking through source I saw specific __HAIKU__ #ifdef tags, which didnt exist when i first started looking into getting mono working. It looks like by the next alpha or release candidate mono will be up and running. Its doubtful my endeavors really helped. Though going forward I will be,  at the very least, publishing the majority of hacks so they may be used as information for patches going forward. I will also attempt (where i see the need) to get patches committed into Mono (and any other projects/tech that I hack regularly). I think the knowledge gained from hacking Mono/WebOS, Mono/Haiku and ZFS/Snappy (among others) will allow me to start creating useful patches. I've learned a lot about autoconf, automake, et al.
Read more