Share It

Saturday, February 1, 2014

Finding Libc

So,  as you know, I've been tooling around with the LLVM IR syntax and the Gold Parser framework. I have the scaffolding for the CIL conversion 90% done, and I am almost ready to parse the "Hello world" code found in the LLVM documentation. There is just one catch. In the Hello world, there is a call to an external standard io library function called puts(). Those fimilar with C/C++ would know that you include the standard header, #include <stdio> or #include <stdio.h> and you magically gain access to this function.

Porting that specific line into CIL would be easy, i'd just replace any puts call with a WriteLine call included in the standard System Library. This is fine to get the sample code running, but it is not a solution going forward. I need to get a list of basic libc commands and rewrite them to work with the System library in the CLI.


DotGnu Project Resurrected?

Years ago (around 2003-2004), the GNU community was really interested in CIL (at the time called MSIL) and its potential to have a common assembly language across disparate systems. One very real problem with *inx development is the fragmentation of libraries. C# was going to solve that problem (I posit that LLVM IR has effectively solved the problem....but thats my opinion). Many open source projects rose to the occasion of "Freeing" .net and MSIL from the hands of Microsoft. From that rush, we have great frameworks like Mono (and i believe Microsoft felt massive pressure, and decided to release most of the .net framework source under the MS-PL, a fairly open license). Lots of these frameworks died or became less interesting, especially as LLVM finished CLANG, etc.

Fast forward to today, and my need to basically port large portions of libc to .net. It seems DotGnu has already done this. Its under the GPL, but for my purposes, I dont care. This is a proof of concept and once i've finished getting things working, there could be a discussion of the merits of using MIT/X11 based licenses, etc. DotGnu has a module called pnetC, that includes a fairly feature complete port of libc to CIL.

Now that I have a CIL port of libc, its a matter of getting a list of all libc functions and adding the namespace to the call of that method. I'll test this functionality with the puts() function in the example code. This is a much better use of time, then just getting WriteLine to work (though i still may do that, just to get a clean parsing structure finished in the next few hours).

Tuesday, January 28, 2014

Goodbye ZFS?


ZFS my one true love

I've used ZFS at home since about 2007. Thats a long time for a technology. I have used it through Solaris 9, 10, 11, Nenenta, Illumos, at al. It is an amazing feat of engineering. Dtrace is another little thing i use, from time to time, on Solaris as well...but not so much in recent times.


Time for an annulment?

While I could say its been all roses, it hasnt. There have been issues with running third party software on solaris. FFMPEG was a beast in the early days to get compiled an running. I used my storage array for many things, and transcoding was one of them. I wanted to move away from running things like tversity on my desktop. I also wanted to get away form development on my desktop, so I started moving toward VMs for everything, which turned out to be the best thing ever....except for a few issues....USB support was shabby for awhile on virtualbox so deploying software onto a device was difficult (to say the least). Things have gotten better. I attempted to mitigate this by running an experiment. Trying to get iSCSI boot working on Windows Vista (and later, attempted with Win 7). I was partially there,  but later found out my pool, which was designed for long term, large file storage...wasnt cut out for the random I/O needed for an OS to run adequtely (I used a single vdev!!!).

Commodities are a beautiful thing

I am at a crossroads now. SSDs are cheap. Ram is cheap, Spinning Storage is cheap. I can now have a high performance development environment booted straight from my storage array. an environment that supports zfs snapshots and a host of  other ZFS features (de-dup, etc).

Why the long story about virtualization and Solaris? Well, i am looking to change my core OS. One option is to run ESXi Hypervisor and enjoy ZFS, but allow for me to run other virtual instances... I would no longer have to store them locally and could have them all centrally managed.

As I went down this road, I remembered that Windows has Hyper-v...and that RemoteFX for Windows Server 2012 allows for full local GPU use. Not only that, Microsoft has released a ZFS clone of sorts with ReFS and Storage Spaces. It has self healing and many other great features that I use on ZFS. Its missing "Free" Snapshots and its still unproven with large installations.


To Be continued!!





Quick Progress!


Parsing for fun and profit

I remember creating parsers to be a bit harder. I tried using Antlr when doing the JavaFX DLR project and failed at it miserably. It felt messy. I couldnt get a feel for how to structure the document, etc. I was drawn to the Irony parser framework because declaring the grammar in C# somehow felt (at the time) cleaner and it seemed to allow for more expressive statements and character logic.

I am not sure if it was my previous experience with parsing and Antlr (or Irony), or how clean and expressive the GOLD parser framework is, but I've been able to create a working parser in just a few days of coding. And by working, I mean it parses and understands valid LLVM IR straight from the project's documentation: 

I actually implemented a a lot more operations and keywords needed to parse the example. It was just easier that way. I am going to now start building an AST. I've decided to be a bit aggressive because I think the AST will drive some Grammar refactoring. I'd like to do the refactoring sooner rather than later. Right now the grammar is fairly small and I'd like to refactor using the AST as a guide. (Think BDD, but the behavior is defined for me, in CIL...i just need to start outputing CIL to validate/test!). This will also allow me to structure the document better and add more comments for posterity.

So my next steps:

  1. Create an AST
  2. Generate IL with the AST
  3. Create a running and valid CIL
  4. Refactor GOLD Grammar
  5. Add missing Operations and Instructions to Grammar
  6. Add respective AST logic
  7. Find small C (or C++) library to test Grammar and IL generator (lzip is a contender)
  8. Find a libc to bootstrap: ulibC or newlib are contenders


I am hoping to have 1-3 done in the next few days. 4-6 soon after that. I have very little experience with bootstrapping, so i'll probably have a look at the emscripten project as a guide. For those who dont know (or if i am using the wrong terminology), basically a general C/C++ library will use standard calls (syscalls, printf, cout, etc, etc) out to what is known as libc (though thats a specific library, there are many implementations). I will need to create a shim routes those calls to the respective mscorelib class/method or find a sutible replacement library on the .net framework (take SDL or Opengl for example...)



Note to those trying to create an grammar for LLVM IR. you see that c"<String>" notation? Its not mentioned ANYWHERE in their documentation. I am assuming it is short hand to create a character array from the given string. I've found a few other little syntax surprises hidden in the example code of the documentation. Its not too bad...yet. Because I am not worrying about my parser validating the syntax, i think I may even be able to ignore some of those surprises. 






So I've again been bitten by the Transcoding bug...Doh!

My history with Transcoding

 A few years back (around mid-2007), I was enamored with a new language on the block, JavaFX (now vala). Alas, I had moved on from the Java platform to the .Net platform during that time, and Silverlight just hit the stage (and WPF). I was more of a fan of the declarative and expressive nature of JavaFX than I was of XAML, so I looked into porting JavaFX script to the .Net platform (and later the DLR). I halted my work in 2008-2009, as JavaFX script was dying, and the Parser technology I was using (a great C# based parser called Irony) was giving me issues when integrating with the DLR.


The LLVM Project


I've been a fan of the LLVM project since about 2009. I have also watched its MSIL backend go into disrepair and interest in C# as part of the vmkit wane. I answered some stack overflow questions in 2011 about this http://stackoverflow.com/questions/5054938/how-to-translate-msil-to-llvm-li/8272816#8272816

Around 2011 I was looking at resurrecting the MSIL backend and decided while it would be of great use to the .net community (FFMPEG as a .net DLL would do wonders alone), it was something I couldnt commit to. And as of today, the MSIL backend has been completely removed in the 3.x branch of LLVM, and would need a major commitment to get it back up to usable levels. (Which one day, I may contribute to..)

LLVM IR....a better CIL


So fast forward to now. MSIL is now referred to as CIL and the LLVM project's intermediate language LLVM IR is amazing. Now I am interested in parsing LLVM IR and creating CIL. Apparently the Mono project has this going the other way, they have CIL code that gets transcoded into LLVM IR.


Of Parsers and Men



I've decided to create a non validating parser for LLVM IR using the GOLD Parser project (I will assume wellformatted IR for this experiment). And then from there, build an AST that outputs CIL!

Friday, May 31, 2013

Fun with the Cloud

Update!

I've been busy these last few months. I've Worked on porting Mono to the BB10 platform. worked on porting Google's Snappy compression to ZFS and a few other little side projects.

Recently I came across an intriguing open source project, GamingAnywhere. GA basically a private cloud solution for Cloud Gaming services like Gaikai or OnLive. Cloud gaming has always interested me. What is interesting about private cloud gaming is that you can use your general purpose PC as a gaming hub for your house. You can have a fairly nice Mid to high end PC and, in theory, not have to care about how it fits in with your entertainment system etc.

When you set up a GamingAnywhere server, it pipes your full desktop, or a specific named window'ed application to a GamingAnywhere client. It currently uses lag free x264 settings, at 24fps (Which if course brings up bandwidth and latency issues...the same issues that OnLive et al, have been working on). Currently both the client and server support windows and Linux.

More to come on this front, as I am still diving in. 





Thursday, September 27, 2012

BB10 and Mono

I am back, with my usual mono antics! RIM recently had a conference and showed off more of BB10. I have decided I will attempt (albiet in my spare time) to use C# to create a Cascade application. I have a few ideas how i'll accomplish this. I'll post more info as time goes on.

Monday, May 28, 2012

Tracking down Signal codes

Ok. I've been trying to tracked down the error's meaning. specifically SIGSEGV code=2. I found it listed in the header file located at  sys/siginfo.h :

* SIGSEGV codes (si_addr == address of faulting memory reference)
 */
#define SEGV_MAPERR     1       /* Address not mapped */
#define SEGV_ACCERR     2       /* No permissions */
#define SEGV_STKERR     3       /* ?? Stack exception */
#define SEGV_GPERR      4       /* ?? General protection */
#define SEGV_IRQERR     5       /* ?? Interrupt handler fault */
#if defined(__EXT_QNX)
#define NSIGSEGV        5
#endif

It seems to be SEGV_ACCERR. No permissions. Very interesting. I am not sure what it is trying to access, or why i dont have permissions as a developer. Hmmm...


Update:  Found addr2line. seems like there are some errors in aot-compiler.c, and some errors in mini-arm.c. I may just do a disable-aot and focus on the mini-arm.c issue...