Solaris 11: IDRs

One of the lacking features in Solaris 11 was  the way it dealt with IDRs when performing OS update. Essentially one had to manually back out an IDR to perform an update as a separate step before an update or by adding a --reject IDRxxx (but this wouldn't necessarily be 100% correct). However, recently Oracle started publishing  IDRs in their support repo once they are integrated as an obsoleting package - long story short, it means that now one can just update OS without worrying how to back out an IDR if it has been fixed in a later release. For more details see here.

The Importance of Hiring

Adam wrote about his experience at Delphix - the main point I agree with is the importance of hiring the right people. It takes time but it pays off in the long term.

Coming Back of Big Iron?

In the past decade servers have become boring - almost everything runs on a cheap x86 servers which mainly differ by color. Now, 96 CPU sockets, 152 cores and 9,216 threads and up-to 96TB of RAM in a server? I wouldn't mind to play with such a monster... read more

What's New in OpenZFS

Matt Ahrens talks about new features in Open (Illumos) ZFS. Some of the performance improvements Matt is talking about have their equivalents in Solaris 11 for some time now though and there are many more, for example:

6282155 arc doesn't always need to make a copy
15613053 : SUNBT6913905 IMPLEMENT ATA TRIM, SCSI WRITE SAME / UNMAP , THIN RECLAMATION
6281079 ZFS I/O priority inversion
6914162 Dedup of null blocks could use special treatment
6662450 L2ARC in memory overhead should be reduced
6957289 ARC metadata limit can have serious adverse effect on dedup performance
6896307 arc_meta_limit modernization
...

Then Solaris 11 ZFS also has encryption, up-to 1MB recordsize, RAID-Z/mirror hybrid allocator, it does support 4k sector, and there are more new features and improvements to ZFS. Then I like LZ4 support in Illumos which is not in Solaris 11... It is good to see though that both Illumos and Oracle are innovating around ZFS. From end-user perspective it is a shame in a way that they do not actively share code though. On the other hand a little bit of competition might be good after all. We will see. ps. notice that the video contains more updates on different technologies around Illumos and they are worth watching as well.

Deduplication on ZFS and NetApp

Recently I came across a case where deduplication ratio for the same data is lower on NetApps than on ZFS. This document maybe explains why - see limits for dedup on NetApp starting with page 26. Apparently NetApp will silently stop deduping data after a specific limit which varies for different models and Ontap versions is reached.

Anyone has some other ideas why effectiveness of dedup on ZFS might be higher for the same data? (assuming the same or similar blocksize).

ZFS/SLOG on SAN

What if you have ZFS deployed on SAN in a clustered environment and you require a dedicated SLOG? It would be really helpful if you could create a small LUN (2-4GB) directly out of disk array's cache. This would be perfect for SLOG. All reads and writes to such LUN would be serviced entirely from array's cache - meaning low latency and no double-writes to backend disks/SSDs for synchronous I/O. Some disk arrays actually do provide such feature, for example see Hitachi's Cache Residency Manager.

Manta

Joyent announced Manta -Object Store with local compute. Pretty cool.
You can read more about it here and here and here. And if you want to see how many times Bryan can write f** then read this one as well.

But seriously, it looks really interesting.

NSS_OPTIONS

Sometimes developers put undocumented options in their code to help with debugging issues. This morning I came across one of such options which prints extra debug information when executing NSS queries.

First you need to disable nscd:

# svcadm disable -t name-service/cache

Then you need to set env variable debug_eng_loop to >0, for example:

# NSS_OPTIONS="debug_eng_loop=2" ping -I 1 wp.pl
NSS_retry(0): 'ipnodes': trying 'files' ... result=NOTFOUND, action=CONTINUE
NSS: 'ipnodes': continue ...
NSS_retry(0): 'ipnodes': trying 'dns' ... result=SUCCESS, action=RETURN
NSS: 'ipnodes': return.
PING wp.pl: 56 data bytes

What's good about it is that it tells you which database configured in NSS returned the result.

Setting RPATH

Today I was made aware that elfedit tool in Solaris 11 allows for setting RPATH (among other things). The only caveat is that a binary had to be linked on Solaris 11. It is very easy to use:

 # elfedit -e 'dyn:runpath $ORIGIN/../lib' /opt/bin/myprog 

There is a nice blog entry about it from Ali Bahrami.

Btrfs: "are we there yet?"

Not in the foreseeable future...

OpenAFS on Solaris 11 x86

Two days ago I presented at the UK Solaris SIG meeting on running OpenAFS on Solaris 11 x86. This is essentially the same talk I gave last year in Edinburgh, I just added few slides explaining what OpenAFS is.

-ck hacking: Other schedulers? Illumos?

-ck hacking: Other schedulers? Illumos?

ZFS Write Performance

Interesting blog entry from guys at Delphix on ZFS write performance - impact of fragmentation.

ZFS: no-op overwrites

There is an interesting new feature in ZFS in Illumos.

https://www.illumos.org/issues/3236

When overwriting a block which is check summed with a cryptographically secure hash function we can compare the old and new checksums for the block to determine if they differ (at almost no cost since we were going to do the checksums anyway). If they do not differ we don't actually need to do the write. This:
1) Reduces I/O
2) Reduces space usage, because if the old block is referenced by a snapshot we will need to keep both copies of the block around even though they contain the same data.
This functionality is only enabled if:
1) The old and new blocks are checksummed using the same algorithm.
2) That algorithm is cryptographically secure (e.g. sha256)
3) Compression is enabled on that block.

 Philosophical question - should we just trust sha256?

(it seems this can't be disabled nor there is an option similar to verify=on in dedup).

There are more interesting new zfs features in Illumos (for example this one which does a similar thing to what Solaris 11 does). The only regret is that unless one wants to play with one of the appliances based on Illumos the only way to use these features is to use FreeBSD or Linux, which is rather ironic. But on the other hand - why not? At least at home.

 

Whose bug is this anyway?!?

Interesting post on bug fixing.