Production debugging for sys admins

Well many people are curious if SystemTap is ready for production. On last UNIX DAYS we had an presentation about SystemTap prepared by our friend. Well during his preparations for the conference he almost got used to many system crashes a day. That alone speaks for itself. Then SystemTap currently lacks user space tracing and many many more things. I wouldn't put it in a production anytime soon and I don't know anyone who is using it.

James posted a well balanced blog entry about it.

Solaris 10 06/06

Finally Solaris 10 06/06 (update 2) is available. Read What's New.

ps. yes, ZFS is included and officially stable :)

NexentaOS Alpha 5

NexentaOS Alpha 5 is available - just in time to celebrate Open Solaris One Year Anniversary.
You can download it here. I must say that it's truly amazing what people behind Nexenta are doing. Just after one year Open Solaris hit the streets they provide almost fully working GNU distribution based on Open Solaris and Debian. I'm really impressed.

"This release of NexentaOS brings to you fully integrated Ubuntu/Dapper Drake userland. Today NexentaOS APT repository contains more than 11,000 Ubuntu/Dapper packages. This number is constantly growing, driven mostly by our industry-strength AutoBuilder."

In addition, Alpha 5 contains:

• Sun's Java SE 5.0 Java Development Kit (JDK(tm)) distributed under the new Distributor's License for Java. Available via NexentaOS APT repository.

• Live Upgrade. Starting from Alpha 5 we are supporting out-of-APT upgrade of the OpenSolaris core. Use Debian tools to bring your system up to the bleeding edge..

• Minimal and full installation, safe mode boot option, removable drive support.

• OpenOffice.org 2.0, natively compiled on NexentaOS.

• OpenSolaris build #40, non-DEBUG kernel.

And also:

• Xorg 7.0.x.

• GNOME 2.14.x with a bunch of neat features, in particular Application Add/Remove.

• KDE 3.5.2 and XFCE 4.3 alternative desktop environments.

And on top of that:

• Samba 3.0 (server and client included with InstallCD), iSCSI, ZFS with the latest fixes and updates.

* Graphical .deb Package Installer

* Apache, MySQL, and Perl/Python/PHP

* Rhythmbox Notification

* Better Applications Menu Organization

* Firefox 1.5, Thunderbird 1.5

* Search Results in Nautilus

* New Log Out Dialog

* New polished look and feel

• And 11,000 more packages, including the most popular office applications, graphics, and multi-media.

Putting your Code into Open Solaris

I'm definitely not a developer but still I can do some C programming. In order to better understand ZFS I was playing/looking with its sources like adding my own "compression" to ZFS during Christmas (I know... but it was really late night and I didn't want to sleep) or later I wanted to implement RFE: 6276934 ability import destroyed pools as I think that in some cases this would be very useful. Additionally while playing with ZFS sources I already knew that implementing this should be really simple and I wanted to test how in practice it's easy (or not) to get your code integrated into Open Solaris (and later into Solaris). I signed Contributor Agreement, made necessary code changes, tested it then made manual changes. Now I requested a Sponsor - Darren Moffat offered his help. ARC case was needed as new options were added, code review was also needed and some paper work. Thankfully for me Darren took care of all of this - thank you. A while later my code changes were integrated (snv_37) into Open Solaris and will also be available in upcoming Solaris 10 Update 2. You can read more about my changes here.

The point is that it's easy to get your code integrated to Open Solaris and you don't have to be a developer - if you are for example a system admin and you find something annoying (or lack of something) in Open Solaris you can easily fix it and share your fix with others. And that's one of the main goals of Open Solaris, isn't it?

There are people afraid that contributing code to Open Solaris could actually mean worse code - fortunately it's NOT the case as even if you are not from Sun you have to submit your changes to code review, ARC, follow coding style in Open Solaris, etc. and fortunately for you (submitters) Sun people will take care of this - you just write changes. That way a high quality of code in Open Solaris is preserved.

Here you can find other bug fixes by non-Sun people into Open Solaris. There are quite a lot of them just after one year Open Solaris is here.

Open Solaris Anniversary

Yesterday, June 14th, was a first year Open Solaris anniversary! What a year - lot of things happened and most of them good. Just quick glance at stats shows that Open Solaris already is a success with more people interest that everyone thought. I think that Open Solaris just after one year is far ahead of that we all expected it to be which is very good.

To celebrate PLOSUG formation and Open Solaris anniversary we had a first PLOSUG meeting yesterday.

Polish Open Solaris User Group

Hi.

Just after last Unix Days - Andrzej and I decided to create Polish Open Solaris User Group - PLOSUG. We were supposed to do it few weeks ago but... Anyway here we are. Tomorrow (June 14th) is the one year anniversary of Open Solaris so we think it's a good opportunity to get together and celebrate both: the anniversary and a creation of PLOSUG.

You can find more info about tomorrow meeting on PLOSUG page.

If you are from Poland and want to participate, talk, etc. about Open Solaris and its technologies, and also meet from time to time then please join to us.

PL-OSUG mailing-list is here.
PL-OSUG archives are here.

ps. this entry is in English however we'll mostly talk in Polish on PLOSUG mailing-list I guess.

fsck, strange files, etc.

In some places I use SATA drives for data storage. From time to time there is a problem with filesystems and I have to fsck, sometimes files are a little bit garbled, etc.
Last time I tried ZFS on SATA disks and to my surprise - just after 3 days I got few hundreds checksum error - well, that explains a lot. Then it stabilized and now from time to time I see occasional errors.

Sometimes we had to live with some problems for so long without any alternative that we have forgotten about a problem and got accustomed to fsck, some bad files, etc.


I would say that ZFS changes that picture radically.
Thanks to ZFS there's no need to fsck, proper data are returned to applications, no mangled fs entries, etc.
It already save our data :)

ps. I haven't yet seen checksum errors reported by ZFS on SCSI, FC or SAS disks...

Coolthreads warranty

I can't understand Sun's policy about warranty for Coolthread servers - only 90 days??!??! Their other entry level servers have at least 1yr - and all their opteron server have 4yrs by default (even x2100). So why T1000 has only 90 days of warranty? (T2000 was changed lately to 1yr).

To make things worse you can't buy T1000 with a Bronze support - you have to buy Silver at least - but that means quite a cost in a 2nd and 3rd year if you want 3 years "warranty" - I know you get more but sometimes you don't need more and all you need is a simple (cheap) warranty.

IMHO it should be corrected as soon as possible so Niagara servers are treated at least the same way as Opteron servers - 3yrs warranty by default. Bronze support should also be offered (at least it's not possible to buy bronze support for T1000/T2000 here in Poland).



Sun's warranty matrix for entry level servers.

Hot Spares in ZFS

Finally Hot Spare support is integrated.

RAID-Z not always a good solution

We all know ZFS is great. RAID-Z means fast RAID-5 without HW RAID controller. However the devil is in the details - while raid-z is great for writing speed, data integrity, etc. its read performance could be really bad if you issue many small random reads from many streams and your dataset is big enough that your cache hit ratio is really small. Sometimes the solution could be to make a pool with many raid-z groups - it means less available storage, but better performance (in terms of IO/s).

So if you want to use raid-z in your environment first carefully consider your workload and if many raid-z in one pool aren't good solution for you then use different raids offered by ZFS. Fortunately other raids in ZFS are NOT affected that way.

If you need more details then read Roch's blog entry on it.

Minimizing Memory Usage for Creating Application Subprocesses

Interesting article on fork()/system()/popen()/posix_spawn() and memory overcommit.