During Christmas I did manage to register another compression method into zfs. All in all it was quite simple - few simple modifications. It did compile and it did worked on my laptop. Actually it wasn't new compression I used lzjb (copy&paste) and registered different name. So basically now I can change algorithm to whatever I want - and this is much hard to accomplish (mostly due to lack of free time). I did play a little bit with zlib.
Now I have to find some time and try to implement zlib or any other algorithm - well, not to implement but rather adopt something from OSS and make some changes (so it will work in-kernel). Ideally I would like to get some compression functions written in C ready to use in-kernel and it would be best if they were memory bounded. Do you know any? Urls?
ps. looks like zlib will be implemented in-kernel by Sun soon and adopted by ZFS.