Digital Wasabi

Alright. I've finally got around to doing some Snappy testing. I've gotten some amazing results, but also noticed a bug. My current code seems to hang after a large amount of compression. It specifically hangs when i am doing an iozone benchmark. The 500MB pass (I set it to 4GB). But before I ran iozone, i did some paliminary testing vs lzjb, and I have to say, the compression performance is higher with Snappy. I was able to get a 23.14  compression ration instead of the 15.22 ratio from lzjb, with similar cpu usage. I suspect this has to do with the snappy_env_init method in the snappy-c library i found. It seems fishy having to initializing the environment for every compression event. I may look into it further but it does not seem very scaleable. and given that ZFS compression works at the block level, and not the file level, there are a lot of compression events happening for a single file (instead of one event, compressing a 400MB file...it would be 400 events with...

Just wanted to make a statement that the CLI tooling for building software in linux (and really any *nix) has changed dramatically. In my mind, configure and make were all black boxes and it was very very hard to actually effect them as an independent developer once they were baked. This is not the case anymore. I was able, with autoconf, automake, and m4 templates (and of course some C++ glue) to actually integrate two libraries (zfs and snappy-c) that know nothing about each other. This is a testament not just to the auto tooling, but the level of maturity of a lot of open source software out there. While make is still technically a black box (yes, make scripts are simple in concept, fine...but go look at Illumos makefiles and we'll talk...), it may be ok for make to stay in the shadows...like some x86 assembly that gets generated on the fly... I just might, after this zfsonlinux + snappy release, revisit illumos. But from what I recall of the original opensolaris d...

Just wanted to give a quick update. On the ZFS front i have really made inroads. I now have to decide whether to create a stand alone snappy kernel module (which could assist folks outside of my little tinkerings) or roll snappy into zfsonlinux, so i can do some benches, and see if its a viable alternative to the compression included in zfs. I am leaning toward kernel module, but I've found a project called snappy-c that is a completely C port of snappy. This will make option #1 super easy. On the tinkering front, Xamarin has ported the java layer of android to C# . I have already gotten my hands on the source. Planning to put it on a phone or tablet soon.

My ZFS/Snappy project was put on hold, because of technical issues and lack of understanding (and documentation) of how to integrate a library into an Illumos build. I may have found a compromise. The zfs on linux project has become a bit more mature. I may switch up and use linux to create a snappy proof of concept.   I understand linux build structure much more than open solaris build structure, and the things i dont understand are fairly well documented. Because zfsonlinux is on github , I'll most likely be using Github and forking the project.

Woo!. Seems everything compiles perfectly. I had to copy the snappy libraries into the closed binary section of the build. Also, i had to build a 64bit and 32bit version of snappy (a bit difficult). I ran into a libtool bug (yes, another road block) when trying to compile the 64bit version of snappy. Right now, I use iscsi with virtualbox. So its difficult to import that into a different VM product (most desktop VM products dont support iSCSI booting).. I think i will copy over my illumos-gate directory to a now OpenIndiana installation, using a virtualHD, instead of iscsi. Then i should be able to import it into VMware or VirtualPC...and avoid the virtualbox bug I am experiencing with the latest illumos build. https://bitbucket.org/techsaint/zfssnappy

I am currently trying to add the snappy compression into ZFS for fun. You can check out my code here:  https://bitbucket.org/techsaint/zfssnappy I've been working on this for a few months (started around august). I've had massive roadblocks every step of the way. The first was the oracle purchase of Sun. Because of the purchase, the Sun Compiler tool patches (which were needed for an illumos build) were removed from the public. It took a month or so for the community to post instructions on using GCC instead of the Sun Compiler. After that, there was slight bug in the illumos build configuration that didnt allow for working builds if your new build version was older than the whatever version of illumos you were on. See  https://www.illumos.org/issues/1118 Now this new bug, is a virtualbox bug, which just appeared out of no where. Here is the ticket:  https://www.virtualbox.org/ticket/9651 . Since I rebased with the latest illumos code (as per build instructions) I ...

Just an update. I've been down in the "trenches", poking around with various technologies. Below is a short list of things i am checking out: Mono and SWT I am checking out mono and SWT as a UI toolkit vs Windows.Forms, or GTK# or WxWidgets or some other toolkit. SWT is an interesting UI toolkit that uses native widgets. Originally from IBM and now a major part of the eclipse project. It seems that IKVM allows SWT to be run without much effort. I'll post some of my progress on this later IKVM and Eclipse I am also checking out running the current version of eclipse with IKVM. Why? Its an interesting test on how far the project has come, its also could provide an interesting alternative to MonoDevelop or Visual Studio. WindowBuilder is also an interesting feature i'd like to check out. What if WindowBuilder could produce C# compatible SWT code? Could be a boon for Desktop mono linux apps. Solaris ZFS development I am looking at doing some Solaris Develop...