milek's blog

NSS_OPTIONS

noreply@blogger.com (milek) — Tue, 28 May 2013 08:26:00 +0000

Sometimes developers put undocumented options in their code to help with debugging issues. This morning I came across one of such options which prints extra debug information when executing NSS queries.

First you need to disable nscd:


# svcadm disable -t name-service/cache

Then you need to set env variable debug_eng_loop to >0, for example:


# NSS_OPTIONS="debug_eng_loop=2" ping -I 1 wp.pl
NSS_retry(0): 'ipnodes': trying 'files' ... result=NOTFOUND, action=CONTINUE
NSS: 'ipnodes': continue ...
NSS_retry(0): 'ipnodes': trying 'dns' ... result=SUCCESS, action=RETURN
NSS: 'ipnodes': return.
PING wp.pl: 56 data bytes

What's good about it is that it tells you which database configured in NSS returned the result.

Setting RPATH

noreply@blogger.com (milek) — Tue, 21 May 2013 14:09:00 +0000

Today I was made aware that elfedit tool in Solaris 11 allows for setting RPATH (among other things). The only caveat is that a binary had to be linked on Solaris 11. It is very easy to use:

 # elfedit -e 'dyn:runpath $ORIGIN/../lib' /opt/bin/myprog

There is a nice blog entry about it from Ali Bahrami.

Btrfs: "are we there yet?"

noreply@blogger.com (milek) — Sat, 18 May 2013 18:19:00 +0000

Not in the foreseeable future...

OpenAFS on Solaris 11 x86

noreply@blogger.com (milek) — Fri, 22 Mar 2013 12:47:00 +0000

Two days ago I presented at the UK Solaris SIG meeting on running OpenAFS on Solaris 11 x86. This is essentially the same talk I gave last year in Edinburgh, I just added few slides explaining what OpenAFS is.

-ck hacking: Other schedulers? Illumos?

noreply@blogger.com (milek) — Mon, 18 Mar 2013 23:28:00 +0000

-ck hacking: Other schedulers? Illumos?

ZFS Write Performance

noreply@blogger.com (milek) — Fri, 08 Mar 2013 12:13:00 +0000

Interesting blog entry from guys at Delphix on ZFS write performance - impact of fragmentation.

ZFS: no-op overwrites

noreply@blogger.com (milek) — Tue, 05 Mar 2013 09:28:00 +0000

There is an interesting new feature in ZFS in Illumos.

https://www.illumos.org/issues/3236

When overwriting a block which is check summed with a cryptographically secure hash function we can compare the old and new checksums for the block to determine if they differ (at almost no cost since we were going to do the checksums anyway). If they do not differ we don't actually need to do the write. This:
1) Reduces I/O
2) Reduces space usage, because if the old block is referenced by a snapshot we will need to keep both copies of the block around even though they contain the same data.
This functionality is only enabled if:
1) The old and new blocks are checksummed using the same algorithm.
2) That algorithm is cryptographically secure (e.g. sha256)
3) Compression is enabled on that block.

Philosophical question - should we just trust sha256?

(it seems this can't be disabled nor there is an option similar to verify=on in dedup).

There are more interesting new zfs features in Illumos (for example this one which does a similar thing to what Solaris 11 does). The only regret is that unless one wants to play with one of the appliances based on Illumos the only way to use these features is to use FreeBSD or Linux, which is rather ironic. But on the other hand - why not? At least at home.

Whose bug is this anyway?!?

noreply@blogger.com (milek) — Fri, 04 Jan 2013 19:27:00 +0000

Interesting post on bug fixing.

SPARC at 25: Past, Present and Future

noreply@blogger.com (milek) — Tue, 20 Nov 2012 12:44:00 +0000

vmtasks explained

noreply@blogger.com (milek) — Fri, 09 Nov 2012 12:23:00 +0000

Solaris 11 introduced a new kernel process called "vmtasks" which accelerates some operations when working with shared memory. For more details see here.

20 Years of Solaris

noreply@blogger.com (milek) — Fri, 09 Nov 2012 11:55:00 +0000

Nice video from Oracle celebrating 20 years of Solaris.

Running OpenAFS on Solaris 11 x86 / ZFS

noreply@blogger.com (milek) — Mon, 22 Oct 2012 23:55:00 +0000

Recently I gave a talk on running OpenAFS services on top of Solaris 11 x86 / ZFS. The talk was split in two parts - first part about $$ benefits of transparent ZFS compression, when running on 3rd party x86 hardware (but it also makes sense when running on Sun/Oracle kit - in some cases even more so). This part also discusses some ideas about running AFS on internal disks instead of directly attached disk arrays, which again, thanks to ZFS built-in compression makes it worthwhile and delivers even more $$ savings.

The main message of this part is, that if your data compresses well (above 2x), running OpenAFS on ZFS can deliver similar or even better performance but most importantly it can save you lots of $$, both in acquisition costs, and in cost of running AFS plant. In most cases you should even be able to re-use the current x86 hardware you have. The beauty of AFS is, that we were able to migrate data from Linux to Solaris/ZFS, in-place, by re-using the same x86 HW, and all of this was completely transparent to all clients (keep in mind we are talking about PBs of data) - this is truly the cloud file system. I think OpenAFS is one of the under-appreciated technologies in the market.

The second part is about using DTrace, both in dev and in production systems, to find scalability and performance bottlenecks, and other bugs as well. Two easy and real-life examples are discussed, which resulted in considerable improvement in scalability and performance of some operations in OpenAFS, along with some other examples of D scripts which provide top-like output with some statistics (slide #32 is an example from a Solaris NFS server, serving VMWare clients and displaying different stats per VM from a single file system...). DTrace has proven to be a very powerful and helpful tool for us, although it is hard to put a specific $ value it brings.

The slides should be available here.

Open Indiana is dead

noreply@blogger.com (milek) — Wed, 29 Aug 2012 18:54:00 +0000

With the main guy behind the project resigning, OI is essentially dead. It's been dead for some time though and with no commercial backing it never really had much chance. This is sad news indeed (although I haven't really used OI). It marks the end of Open Solaris era.

Can Illumos survive in the long term? Can it become relevant outside of couple of niche use cases?

Ironically, it is Oracle's Solaris which will probably outlive all of them.

1,000,000,000,000 Frames/Second Photography - Ramesh Raskar

noreply@blogger.com (milek) — Fri, 24 Aug 2012 18:26:00 +0000

SmartOS: An SA Primer

noreply@blogger.com (milek) — Fri, 24 Aug 2012 16:07:00 +0000

Ben Rockwood on SmartOS and Illumos. Ben - IPS is actually cool! :P

Locking a running process in RAM

noreply@blogger.com (milek) — Fri, 27 Jul 2012 12:04:00 +0000

Recently I was looking at a possibility of locking some or all memory mappings, for an already running process, in RAM. Why would you want to do it? There might be many reasons. One of them is to prevent a critical process to be swapped out.

Now, how can we do it without restarting a process or changing its code? There is memcntl() and similar calls like plock(), mlock(), etc. - but they only work for a process which is calling them. However libproc on Solaris allows you to run memcntl() in a context of other process, among many other cool things. By the way - libproc is used by tools like truss, ppgsz, etc.

First, lets see that it actually works. Here is a bash process running as a non-root user, see that no mappings are locked in RAM.

cwafseng3 $ pmap -ax $$
27709:  bash
 Address  Kbytes     RSS    Anon  Locked Mode   Mapped File
08050000     968     968       -       - r-x--  bash
08151000      76      76       8       - rwx--  bash
08164000     140     140      56       - rwx--    [ heap ]
F05D0000     440     440       -       - r-x--  libnsl.so.1
F064E000       8       8       4       - rw---  libnsl.so.1
F0650000      20      12       -       - rw---  libnsl.so.1
F0660000      56      56       -       - r-x--  libsocket.so.1
F067E000       4       4       -       - rw---  libsocket.so.1
FE560000      64      16       -       - rwx--    [ anon ]
FE577000       4       4       4       - rwxs-    [ anon ]
FE580000      24      12       4       - rwx--    [ anon ]
FE590000       4       4       4       - rw---    [ anon ]
FE5A0000    1352    1352       -       - r-x--  libc_hwcap1.so.1
FE702000      44      44      16       - rwx--  libc_hwcap1.so.1
FE70D000       4       4       -       - rwx--  libc_hwcap1.so.1
FE710000       4       4       -       - r-x--  libdl.so.1
FE720000       4       4       4       - rw---    [ anon ]
FE730000     184     184       -       - r-x--  libcurses.so.1
FE76E000      16      16       -       - rw---  libcurses.so.1
FE772000       8       8       -       - rw---  libcurses.so.1
FE780000       4       4       4       - rw---    [ anon ]
FE790000       4       4       4       - rw---    [ anon ]
FE7A0000       4       4       -       - rw---    [ anon ]
FE7AD000       4       4       -       - r--s-    [ anon ]
FE7B4000     220     220       -       - r-x--  ld.so.1
FE7FB000       8       8       4       - rwx--  ld.so.1
FE7FD000       4       4       -       - rwx--  ld.so.1
FEFFB000      16      16       4       - rw---    [ stack ]
-------- ------- ------- ------- -------
total Kb    3688    3620     116       -

Now, I will use the small tool I wrote, to lock all mappings with RX or RWX protections on them. The tool requires for a PID to be specified.

$ ./pr_memcntl 27709
pr_memcntl() failed: Not owner

Although I run it as root, it failed. Remember when I wrote a moment ago, that libproc would call memcntl() from a contetx of the target process? The target process here is bash with pid 27709 and it is running as a standard user, and by default a standard user cannot lock pages in memory. We can add the required privileges for locking pages in RAM. Lets see what's missing. I enabled privilege debugging for the bash process and run the pr_memcntl tool again

$ ppriv -D 27709
bash[27709]: missing privilege "proc_lock_memory"
            (euid = 145104, syscall = 131) needed at memcntl+0x140

Now, lets add the missing privilege to the bash process and then try again locking the mappings:

$ ppriv -s EP+proc_lock_memory 27709
$ ./pr_memcntl 27709
$

This time no error, lets see the pmap output again:

$ pmap -ax $$
27709:  bash
 Address  Kbytes     RSS    Anon  Locked Mode   Mapped File
08050000     968     968       -     968 r-x--  bash
08151000      76      76       8      76 rwx--  bash
08164000     140     140      48     140 rwx--    [ heap ]
F05D0000     440     440       -     440 r-x--  libnsl.so.1
F064E000       8       8       4       - rw---  libnsl.so.1
F0650000      20      12       -       - rw---  libnsl.so.1
F0660000      56      56       -      56 r-x--  libsocket.so.1
F067E000       4       4       -       - rw---  libsocket.so.1
FE560000      64      64       -      64 rwx--    [ anon ]
FE577000       4       4       4       - rwxs-    [ anon ]
FE580000      24      24       4      24 rwx--    [ anon ]
FE590000       4       4       4       - rw---    [ anon ]
FE5A0000    1352    1352       -    1352 r-x--  libc_hwcap1.so.1
FE702000      44      44      16      44 rwx--  libc_hwcap1.so.1
FE70D000       4       4       -       4 rwx--  libc_hwcap1.so.1
FE710000       4       4       -       4 r-x--  libdl.so.1
FE720000       4       4       4       - rw---    [ anon ]
FE730000     184     184       -     184 r-x--  libcurses.so.1
FE76E000      16      16       -       - rw---  libcurses.so.1
FE772000       8       8       -       - rw---  libcurses.so.1
FE780000       4       4       4       - rw---    [ anon ]
FE790000       4       4       4       - rw---    [ anon ]
FE7A0000       4       4       -       - rw---    [ anon ]
FE7AD000       4       4       -       - r--s-    [ anon ]
FE7B4000     220     220       -     220 r-x--  ld.so.1
FE7FB000       8       8       4       8 rwx--  ld.so.1
FE7FD000       4       4       -       4 rwx--  ld.so.1
FEFFB000      16      16       4       - rw---    [ stack ]
-------- ------- ------- ------- -------
total Kb    3688    3680     108    3588

It works! :)

Notice that only mappings with RX or RWX protections are locked (as hard-coded in the tool, but any mappings or entire process can be locked if desired). The tool can obviously easily be expanded so it automatically adds the privilege if needed.

The C code below is a prototype - it is not idiot proof nor does it handle all errors, and it could be more user friendly - but it works and it is trivial to extend it. It should work on Solaris 10 and Solaris 11 and also on all Illumos based distributions.

Notice, that it sets MCL_CURRENT|MCL_FUTURE flags, meaning that not only current rx|rwx mappings are locked but also all future mappings, with the rx|rwx protections set, will be locked as well. In order to compile the program you need libproc.h which is currently not distributed with Solaris (hopefully it will change soon). You can get a copy from here.

Some ideas on how the tool could be easily extended:

add an option to add the proc_lock_memory automatically (and perhaps update resource limit as well)
add an option to remove a lock on specific mapping or from all mappings
add an option to specify what should be lock
add options to specify if only current mappings and/or future mappings should be locked as well

ps. putting a process in RT class would achieve a similar result, although it wouldn't give a control over which mappings should be locked, and running in RT might not be desirable for other reasons as well

// gcc -m64 -lproc -I. -o pr_memcntl pr_memcntl.c
 
#include <sys/types.h>
#include <sys/mman.h>
#include <stdio.h>
#include <stdlib.h>
 
#include <libproc.h>


int main(int argc, char **argv) {
  pid_t pid;
  int perr;
  static struct ps_prochandle *Pr;
 
  pid = atoi(argv[1]);
 
  if((Pr = Pgrab(pid, PGRAB_NOSTOP, &perr)) == NULL) {
    printf("Pgrab() failed: %s\n", Pgrab_error(perr));
    exit(1);
  }
 
  if(pr_memcntl(Pr, 0, 0, MC_LOCKAS, (caddr_t)(MCL_CURRENT|MCL_FUTURE), PROC_TEXT, (int)0)) {
    perror("pr_memcntl() failed");
    Prelease(Pr, 0);
    exit(1);
  }
 
  if(pr_memcntl(Pr, 0, 0, MC_LOCKAS, (caddr_t)(MCL_CURRENT|MCL_FUTURE), PROC_TEXT|PROT_WRITE, (int)0)) {
    perror("pr_memcntl() failed");
    Prelease(Pr, 0);
    exit(1);
  }
 
  Prelease(Pr, 0);
  Pr = NULL;
 
  exit(0);
}

Physical disk locations

noreply@blogger.com (milek) — Wed, 02 May 2012 15:22:00 +0000

zpool(1M) has a very handy option to display physical disk locations on some hardware.

cwafseng3 $ zpool status -l cwafseng3-0
  pool: cwafseng3-0
 state: ONLINE
  scan: scrub canceled on Thu Apr 12 13:52:13 2012
config:
 
        NAME                                       STATE     READ WRITE CKSUM
        cwafseng3-0                                ONLINE       0     0     0
          raidz1-0                                 ONLINE       0     0     0
            /dev/chassis/i86pc.unknown/HDD02/disk  ONLINE       0     0     0
            /dev/chassis/i86pc.unknown/HDD23/disk  ONLINE       0     0     0
            /dev/chassis/i86pc.unknown/HDD22/disk  ONLINE       0     0     0
            /dev/chassis/i86pc.unknown/HDD21/disk  ONLINE       0     0     0
            /dev/chassis/i86pc.unknown/HDD20/disk  ONLINE       0     0     0
            /dev/chassis/i86pc.unknown/HDD19/disk  ONLINE       0     0     0
            /dev/chassis/i86pc.unknown/HDD17/disk  ONLINE       0     0     0
            /dev/chassis/i86pc.unknown/HDD15/disk  ONLINE       0     0     0
 
errors: No known data errors
cwafseng3 $

The HDDXX entries directly correspond to the physical slots, in this case in Sun x4270M2 server. Compare it to the standard output:


cwafseng3 $ zpool status cwafseng3-0
  pool: cwafseng3-0
 state: ONLINE
  scan: scrub canceled on Thu Apr 12 13:52:13 2012
config:

        NAME                       STATE     READ WRITE CKSUM
        cwafseng3-0                ONLINE       0     0     0
          raidz1-0                 ONLINE       0     0     0
            c3t5000CCA00AC87F55d0  ONLINE       0     0     0
            c3t5000CCA00AAA0D1Dd0  ONLINE       0     0     0
            c3t5000CCA00AA95559d0  ONLINE       0     0     0
            c3t5000CCA00AAAD155d0  ONLINE       0     0     0
            c3t5000CCA015214845d0  ONLINE       0     0     0
            c3t5000CCA015214F85d0  ONLINE       0     0     0
            c3t5000CCA01521070Dd0  ONLINE       0     0     0
            c3t5000CCA0151A287Dd0  ONLINE       0     0     0

errors: No known data errors
cwafseng3 $

See also croinfo(1M) for how to get this information (and more) for all disk drives regardless if they are part of zpool or not.

Illumos: Google Summer of Code

noreply@blogger.com (milek) — Fri, 23 Mar 2012 11:35:00 +0000

Are you a student and would like to do some interesting coding during the summer? If yes, then check this.

New Comparison of Memory Allocators

noreply@blogger.com (milek) — Fri, 23 Mar 2012 11:08:00 +0000

MWAC in Global Zone

noreply@blogger.com (milek) — Fri, 20 Jan 2012 17:50:00 +0000

Solaris 11 has a new cool feature called Immutable Zones. Darren Moffat presented new features in Solaris 11 Zones at the last LOSUG meeting in London. Immutable Zones basically allow for read-only or partially read-only Zones to be deployed. You can even combine it with ZFS encryption - see Darren's blog entry for more details. The underlying technology to immutable zones is called Mandatory Write Access Control (MWAC) and is implemented in kernel. So for each open, unlink, etc. syscall a VFS layer checks if MWAC is enabled for a given filesystem and a zone and if it is it will check white and black lists associated with a zone and potentially deny write access to a file (generating EROFS). The actual definitions for different default profiles are located in /usr/lib/brand/solaris/config.xml file. It is *very* simple to use the pre-defined profiles when creating a zone and it just works. Really cool. Thanks Darren for the great demo.

Now MWAC only works with non-global zones... at least by default. There is no public interface exposed to manipulate MWAC rules directly or to enable it for a global zone but it doesn't mean one can't try to do it anyway. DTrace, objdump, mdb, etc. were very helpful here to see what's going on. The result of having couple of hours of fun is below.

root@global # touch /test/file1
root@global # rm -f /test/file1
root@global # ./mwac -b "/test/file1"
MWAC black list for the global zone installed.

root@global # touch /test/file1
touch: cannot create /test/file1: Read-only file system
root@global # touch /test/file2 ; rm /test/file2
root@global #

Now lets disable MWAC again:

root@global # mwac -u
MWAC unlock succeeded.

root@global # touch /test/file1 ; rm /test/file1
root@global #

You can even use patterns:

root@global # mwac -b "/test/*"
MWAC black list for the global zone installed.

root@global # touch /test/a ; mkdir /test/b
touch: cannot create /test/a: Read-only file system
mkdir: Failed to make directory "/test/b"; Read-only file system
root@global #

ReFS

noreply@blogger.com (milek) — Thu, 19 Jan 2012 12:26:00 +0000

Next generation file system for Windows: ReFS
It looks pretty interesting and promising. Something like ZFS lite for Windows.

"The key goals of ReFS are:
Maintain a high degree of compatibility with a subset of NTFS features that are widely adopted while deprecating others that provide limited value at the cost of system complexity and footprint.
Verify and auto-correct data. Data can get corrupted due to a number of reasons and therefore must be verified and, when possible, corrected automatically. Metadata must not be written in place to avoid the possibility of “torn writes,” which we will talk about in more detail below.
Optimize for extreme scale. Use scalable structures for everything. Don’t assume that disk-checking algorithms, in particular, can scale to the size of the entire file system.
Never take the file system offline. Assume that in the event of corruptions, it is advantageous to isolate the fault while allowing access to the rest of the volume. This is done while salvaging the maximum amount of data possible, all done live.
Provide a full end-to-end resiliency architecture when used in conjunction with the Storage Spaces feature, which was co-designed and built in conjunction with ReFS.
The key features of ReFS are as follows (note that some of these features are provided in conjunction with Storage Spaces).
Metadata integrity with checksums
Integrity streams providing optional user data integrity
Allocate on write transactional model for robust disk updates (also known as copy on write)
Large volume, file and directory sizes
Storage pooling and virtualization makes file system creation and management easy
Data striping for performance (bandwidth can be managed) and redundancy for fault tolerance
Disk scrubbing for protection against latent disk errors
Resiliency to corruptions with "salvage" for maximum volume availability in all cases
Shared storage pools across machines for additional failure tolerance and load balancing

"

What is Watson?

noreply@blogger.com (milek) — Wed, 04 Jan 2012 19:42:00 +0000

You probably heard about Watson from IBM. Michael Perrone gave a very entertaining presentation on Watson at LISA. Enjoy.

Solaris 11 Source Code

noreply@blogger.com (milek) — Fri, 30 Dec 2011 12:00:00 +0000

It seems that Oracle has "published" S11 source code after all...
And a little bit more on the story.

From SunOS thru Solaris and OpenSolaris to illumos

noreply@blogger.com (milek) — Fri, 16 Dec 2011 13:01:00 +0000

Last week I attended LISA '11 conference and one of the great and fun presentations was Bryan Cantrill's talk titled "Fork Yeah! The Rise and Development of illumos".
See the video and the slides.

Also, if you are interested in the whole DevOps transformation watch Ben Rockwood's talk - see the video and the slides.

There were more interesting talks - you can find them at http://www.usenix.org/events/lisa11/tech/

Solaris 11 - hostmodel

noreply@blogger.com (milek) — Fri, 11 Nov 2011 13:31:00 +0000

Solaris 11 Express and now Solaris 11 have a new functionality which was long missing in Solaris. It allows to force an application which binds to a given IP address to only use a default gateway configured on the same subnet (hostmodel set to strong), or to prefer such gateway if exists (src-priority) or the default behavior which is to choose "randomly" (weak). This is very useful if you have an additional interface (10GbE for example) and you want a guarantee that all outgoing packets from all applications which bind to an IP address on that interface go thru that dedicated interface even if there are other default gateways on other interfaces/subnets.

From ipadm(1M) man page:

hostmodel (IPv4), hostmodel (IPv6)
Control send/receive behavior for IP packets on a multi-homed system. The value of hostmodel can be set to strong or weak, corresponding to the equivalent end-system model definitions of RFC 1122. In addition, a third value of src-priority is also supported. In the src-priority hostmodel scenario, a packet will be accepted on any interface, as long as the packet's destination IP address is configured and marked UP on one of the host's interfaces. When transmitting a packet, if multiple routes for the IP destination in the packet are available, the system will prefer routes where the IP source address in the packet is configured on the outgoing interface. If no such route is available, the system will fall back to selecting the “best” route, as with the weak ES case.