Digital Wasabi

Alright. I've finally got around to doing some Snappy testing. I've gotten some amazing results, but also noticed a bug. My current code seems to hang after a large amount of compression. It specifically hangs when i am doing an iozone benchmark. The 500MB pass (I set it to 4GB). But before I ran iozone, i did some paliminary testing vs lzjb, and I have to say, the compression performance is higher with Snappy. I was able to get a 23.14  compression ration instead of the 15.22 ratio from lzjb, with similar cpu usage. I suspect this has to do with the snappy_env_init method in the snappy-c library i found. It seems fishy having to initializing the environment for every compression event. I may look into it further but it does not seem very scaleable. and given that ZFS compression works at the block level, and not the file level, there are a lot of compression events happening for a single file (instead of one event, compressing a 400MB file...it would be 400 events with...

I've completed my first zfs + snappy build. Its up on github at  https://github.com/techsaint/zfs-snappy . I tried to test it on my Virtualbox instance, but it doesn't look like there is very good VDI support for zfs. It seems to freeze when i try to write to a file system. I tried with the non snappy version too, just in case I mucked up something, but the vanilla version also froze. Now that I have a compatible version, I will test it on actual hardware, instead of in a VM. This build actually has snappy baked into zfsonlinux using the snappy-c library I found. If this route works, I will look at using the snappy-c port in an illumos build. I have all the changes out on bitbucket, and I did spend a few months learning the opensolaris/illumos build files....

Just wanted to make a statement that the CLI tooling for building software in linux (and really any *nix) has changed dramatically. In my mind, configure and make were all black boxes and it was very very hard to actually effect them as an independent developer once they were baked. This is not the case anymore. I was able, with autoconf, automake, and m4 templates (and of course some C++ glue) to actually integrate two libraries (zfs and snappy-c) that know nothing about each other. This is a testament not just to the auto tooling, but the level of maturity of a lot of open source software out there. While make is still technically a black box (yes, make scripts are simple in concept, fine...but go look at Illumos makefiles and we'll talk...), it may be ok for make to stay in the shadows...like some x86 assembly that gets generated on the fly... I just might, after this zfsonlinux + snappy release, revisit illumos. But from what I recall of the original opensolaris d...

Just wanted to give a quick update. On the ZFS front i have really made inroads. I now have to decide whether to create a stand alone snappy kernel module (which could assist folks outside of my little tinkerings) or roll snappy into zfsonlinux, so i can do some benches, and see if its a viable alternative to the compression included in zfs. I am leaning toward kernel module, but I've found a project called snappy-c that is a completely C port of snappy. This will make option #1 super easy. On the tinkering front, Xamarin has ported the java layer of android to C# . I have already gotten my hands on the source. Planning to put it on a phone or tablet soon.

I didnt expect to have the time, but I really became interested in answering the question of how easy working with the linux port of ZFS would be. I am here to say that I got everything compiled and installed last night. I am now trying to track down an shared library file that doesnt seem to be found. It exists, but my guess would be that the zfs tool is looking for it in /usr/lib instead of /usr/local/lib. The specific error is: /home/techsaint/files/zfs-project$ zpool zpool: error while loading shared libraries: libnvpair.so.1: cannot open shared object file: No such file or directory

My ZFS/Snappy project was put on hold, because of technical issues and lack of understanding (and documentation) of how to integrate a library into an Illumos build. I may have found a compromise. The zfs on linux project has become a bit more mature. I may switch up and use linux to create a snappy proof of concept.   I understand linux build structure much more than open solaris build structure, and the things i dont understand are fairly well documented. Because zfsonlinux is on github , I'll most likely be using Github and forking the project.