USDT provider

There's nothing like a real example. Official documentation here.
Imagine MySQL or Apache with USDT probes... I wish I had enough time :)

Another DTrace scripts repository

Thanks to Richard McDougall I found this repository of DTrace scripts. I'm sure I'll find many interesting things in those scripts.

IP storage

Commentary about IP Storage Directions posted by Richard McDougall - really interesting post.
I can only agree with Richard - NFS became worth solution in data centers, mainly because it's cheaper, easier and more flexible. Probably the same will be with iSCSI - the problem with it, as I see it now, is that it's not mature technology (yet) - it's not well tested in many different environments.
In our data center we were skeptical about NFS few years ago - we did some testing with Linux but it wasn't working well - too much kernel panics, etc. Then, some time later, we did try again - this time with Solaris. Well, it's working really well - right now we use NFS with Solaris with many TBs of data. We do use Linux with NFS too - it's working acceptable these days. The problem with Linux NFS client (both 2.4 and 2.6 kernels) is that if NFS server goes somewhat 'crazy' then Linux simply panics while Solaris with the same NFS server (which is not Linux and not Solaris) keeps running. We had to get used to daily Linux panics. Once the problem with NFS server was resolved (unfortunately it took many months) Linux clients started to work properly. On the other hand Linux NFS client with Linux NFS server works quite well these days - I mean stable. Solaris NFS servers just work. The biggest problem with Linux is observability - you can almost say there's none.

Is SAN dead? Definitely not now, and probably not in a next 5-10 years. Why? Because people know it - how to setup HA, tune it, secure etc. And it's very well tested technology. I think that iSCSI will be mostly used in low-end at the beginning, then will steadily go into mid-range and high-ends. What we see now is a combination of SAN and iSCSI. And people won't throw away their current SAN hardware for a long time. But it looks like IP Storage has a bright future and we will see many deployments in low-end in coming years (months?).

ps. What I'm gonna test in coming days is a combination of NFS server with ZFS in a SAN with MPxIO and IPMP :) I'm sure that's gonna be interesting!

DTracing Java!

Wow! Finally some bits available to DTrace Java. Imagine you can trace Java methods similar way as C functions with PID provider! And correlate them to syscalls, in-kernel functions, IOs, etc... Whole bunch of profiling is now possible. Although it looks like it's not perfect (yet) it's definitely worth looking into.
You can find some examples on Bryan's and Adam's blogs.
You can download Java provider for DTrace from project page. I'm sure Java developers will love it!

btw: this is available under CDDL!

Fault monitoring in Solaris

Here you can download Blue Print about Predictive Fault Monitoring in Sun Fire Servers.

ZFS is cool

I don't know why - but I feel it's cool somehow :)
Creating such a pool is not only easy - it's damn fast - I mean creating pool took less then 20s and creating filesystem took less than 3s!

What can I say - it just works! And all of this with multipathing (MPxIO) on Solaris 10 FCS.

sh-3.00# zpool create big c4t600A0B8000169D170000003B425BE048d0 c4t600A0B8000169D170000003D425BE3BAd0 c4t600A0B8000169D170000003E425BE3E0d0 c4t600A0B8000169D170000003F425BE404d0 c4t600A0B8000169D170000004A425BE6C4d0 c4t600A0B8000169D170000004B425BE6FCd0 c4t600A0B8000169D170000004C425BE71Ed0 c4t600A0B8000169D170000004D425BE8EEd0 c4t600A0B8000169D170000004F425BE94Ad0 c4t600A0B8000169D1700000041425BE4ECd0 c4t600A0B8000169D1700000044425BE5ACd0 c4t600A0B8000169D1700000045425BE5D4d0 c4t600A0B8000169D1700000051425BE9BAd0 c4t600A0B8000169D1700000052425BE9FEd0 c4t600A0B8000169D1700000040425BE44Ed0 c4t600A0B8000169D1700000043425BE55Ed0 c4t600A0B8000169D1700000046425BE600d0 c4t600A0B8000169D1700000047425BE666d0 c4t600A0B8000169D1700000049425BE694d0 c4t600A0B8000169D1700000050425BE982d0
bash-3.00# zfs create big/test1
bash-3.00#

bash-3.00# zpool df
Pool                   size   used  avail capacity
-------------------- ------ ------ ------ --------
big                   10.9T  68.5K  10.9T     1%

bash-3.00# df -h /zfs/big/test1
Filesystem             size   used  avail capacity  Mounted on
big/test1               11T     6K    11T     1%    /zfs/big/test1
bash-3.00#

update:bash-3.00# zfs list
NAME                   USED  AVAIL  REFER  MOUNTPOINT
test                  26.0K  21.5T      -  /zfs/test

This is biggest fs I have created so far - 21TB :)

SUPerG

Unfortunately I won't be at SUPerG this year - I really wish I were there. This is the best technical conference I've ever seen. This time SUPerG is being held in Washington, D.C. on April 18-21. Thanks to Glenn Fawcett you can download two presentations he's going to present this year - if you are interested in database scaling on big iron then go and download them here.

Jim Mauro has a blog too!

I've just found Jim Mauro blog!
So both authors of Solaris Internals are blogging now - it's gonna be interesting!

Solaris 10 Security Overview

Here you can download PDF presentation on Solaris security by Glenn Brunette. Nice summary what's in Solaris about security.

Richard McDougall has a blog!

Richard McDougall has started a blog! I'm sure it will be worth reading.

VxVM is cool

I had to grow filesystem on a production database by adding more disk to raid 10 but not thru concatenation as it would hit a performance but rather by recreating raid so proper config would be preserved. The trick is how to do it without any downtime. With VxVM you can do all of this on-the-fly without stopping database, unmounting filesystems, etc. and all of this on rather old Solaris 8 in a cluster environment (Sun Cluster). It took some time for VxVM to relayout data but at the end it worked without any problems or downtime of the database.

On the other hand it makes less sense to use VxVM (due to costs) as more features are put in SVM which is free with Solaris and as more features are put in mid-range and low-end arrays so a volume manager is not needed at all or SVM is good enough.
It doesn't mean of course that VxVM doesn't make sense in all cases - in many it does and it's really good software.

More than 10 args in PID provider

We run into a problem that we couldn't catch more than 10 arguments in a function using PID provider in DTrace. Thanks to Adam Leventhal for workaround. I've checked it for SPARC 32/64 bit programs and it works. Ideally it would be getting args[] working with PID provider and allowing it to get more than 10 arguments - as Adam wrote to me they do work on this and consider allowing more arguments - great!

Workaround:

sparc 32-bit:
        this->argN =
            *(uint32_t *)copyin(uregs[R_SP] + sizeof (uint32_t) * (N + 17),
            sizeof (uint32_t));
        (only applies to args past the first 6)

sparc 64-bit:
        this->argN =
            *(uint64_t *)copyin(uregs[R_SP] + 0x7ff +
            sizeof (uint64_t) * (N + 16), sizeof (uint64_t));
        (only applies to args past the first 6)

x86:
        this->argN =
            *(uint32_t *)copyin(uregs[R_SP] + sizeof (uint32_t) * (N + 1),
            sizeof (uint32_t));

amd64:
        this->argN =
            *(uint64_t *)copyin(uregs[R_SP] + sizeof (uint64_t) * (N - 5),
            sizeof (uint64_t));
        (only applies to args past the first 6)

Performance tests in Solaris development

Here you can find interesting blog entry by Fintan Ryan about "Sun's Performance Lifestyle".