Production debugging for sys admins

Well many people are curious if SystemTap is ready for production. On last UNIX DAYS we had an presentation about SystemTap prepared by our friend. Well during his preparations for the conference he almost got used to many system crashes a day. That alone speaks for itself. Then SystemTap currently lacks user space tracing and many many more things. I wouldn't put it in a production anytime soon and I don't know anyone who is using it.

James posted a well balanced blog entry about it.

Solaris 10 06/06

Finally Solaris 10 06/06 (update 2) is available. Read What's New.

ps. yes, ZFS is included and officially stable :)

NexentaOS Alpha 5

NexentaOS Alpha 5 is available - just in time to celebrate Open Solaris One Year Anniversary.
You can download it here. I must say that it's truly amazing what people behind Nexenta are doing. Just after one year Open Solaris hit the streets they provide almost fully working GNU distribution based on Open Solaris and Debian. I'm really impressed.

"This release of NexentaOS brings to you fully integrated Ubuntu/Dapper Drake userland. Today NexentaOS APT repository contains more than 11,000 Ubuntu/Dapper packages. This number is constantly growing, driven mostly by our industry-strength AutoBuilder."

In addition, Alpha 5 contains:

• Sun's Java SE 5.0 Java Development Kit (JDK(tm)) distributed under the new Distributor's License for Java. Available via NexentaOS APT repository.

• Live Upgrade. Starting from Alpha 5 we are supporting out-of-APT upgrade of the OpenSolaris core. Use Debian tools to bring your system up to the bleeding edge..

• Minimal and full installation, safe mode boot option, removable drive support.

• OpenOffice.org 2.0, natively compiled on NexentaOS.

• OpenSolaris build #40, non-DEBUG kernel.

And also:

• Xorg 7.0.x.

• GNOME 2.14.x with a bunch of neat features, in particular Application Add/Remove.

• KDE 3.5.2 and XFCE 4.3 alternative desktop environments.

And on top of that:

• Samba 3.0 (server and client included with InstallCD), iSCSI, ZFS with the latest fixes and updates.

* Graphical .deb Package Installer

* Apache, MySQL, and Perl/Python/PHP

* Rhythmbox Notification

* Better Applications Menu Organization

* Firefox 1.5, Thunderbird 1.5

* Search Results in Nautilus

* New Log Out Dialog

* New polished look and feel

• And 11,000 more packages, including the most popular office applications, graphics, and multi-media.

Putting your Code into Open Solaris

I'm definitely not a developer but still I can do some C programming. In order to better understand ZFS I was playing/looking with its sources like adding my own "compression" to ZFS during Christmas (I know... but it was really late night and I didn't want to sleep) or later I wanted to implement RFE: 6276934 ability import destroyed pools as I think that in some cases this would be very useful. Additionally while playing with ZFS sources I already knew that implementing this should be really simple and I wanted to test how in practice it's easy (or not) to get your code integrated into Open Solaris (and later into Solaris). I signed Contributor Agreement, made necessary code changes, tested it then made manual changes. Now I requested a Sponsor - Darren Moffat offered his help. ARC case was needed as new options were added, code review was also needed and some paper work. Thankfully for me Darren took care of all of this - thank you. A while later my code changes were integrated (snv_37) into Open Solaris and will also be available in upcoming Solaris 10 Update 2. You can read more about my changes here.

The point is that it's easy to get your code integrated to Open Solaris and you don't have to be a developer - if you are for example a system admin and you find something annoying (or lack of something) in Open Solaris you can easily fix it and share your fix with others. And that's one of the main goals of Open Solaris, isn't it?

There are people afraid that contributing code to Open Solaris could actually mean worse code - fortunately it's NOT the case as even if you are not from Sun you have to submit your changes to code review, ARC, follow coding style in Open Solaris, etc. and fortunately for you (submitters) Sun people will take care of this - you just write changes. That way a high quality of code in Open Solaris is preserved.

Here you can find other bug fixes by non-Sun people into Open Solaris. There are quite a lot of them just after one year Open Solaris is here.

Open Solaris Anniversary

Yesterday, June 14th, was a first year Open Solaris anniversary! What a year - lot of things happened and most of them good. Just quick glance at stats shows that Open Solaris already is a success with more people interest that everyone thought. I think that Open Solaris just after one year is far ahead of that we all expected it to be which is very good.

To celebrate PLOSUG formation and Open Solaris anniversary we had a first PLOSUG meeting yesterday.

Polish Open Solaris User Group

Hi.

Just after last Unix Days - Andrzej and I decided to create Polish Open Solaris User Group - PLOSUG. We were supposed to do it few weeks ago but... Anyway here we are. Tomorrow (June 14th) is the one year anniversary of Open Solaris so we think it's a good opportunity to get together and celebrate both: the anniversary and a creation of PLOSUG.

You can find more info about tomorrow meeting on PLOSUG page.

If you are from Poland and want to participate, talk, etc. about Open Solaris and its technologies, and also meet from time to time then please join to us.

PL-OSUG mailing-list is here.
PL-OSUG archives are here.

ps. this entry is in English however we'll mostly talk in Polish on PLOSUG mailing-list I guess.

fsck, strange files, etc.

In some places I use SATA drives for data storage. From time to time there is a problem with filesystems and I have to fsck, sometimes files are a little bit garbled, etc.
Last time I tried ZFS on SATA disks and to my surprise - just after 3 days I got few hundreds checksum error - well, that explains a lot. Then it stabilized and now from time to time I see occasional errors.

Sometimes we had to live with some problems for so long without any alternative that we have forgotten about a problem and got accustomed to fsck, some bad files, etc.


I would say that ZFS changes that picture radically.
Thanks to ZFS there's no need to fsck, proper data are returned to applications, no mangled fs entries, etc.
It already save our data :)

ps. I haven't yet seen checksum errors reported by ZFS on SCSI, FC or SAS disks...

Coolthreads warranty

I can't understand Sun's policy about warranty for Coolthread servers - only 90 days??!??! Their other entry level servers have at least 1yr - and all their opteron server have 4yrs by default (even x2100). So why T1000 has only 90 days of warranty? (T2000 was changed lately to 1yr).

To make things worse you can't buy T1000 with a Bronze support - you have to buy Silver at least - but that means quite a cost in a 2nd and 3rd year if you want 3 years "warranty" - I know you get more but sometimes you don't need more and all you need is a simple (cheap) warranty.

IMHO it should be corrected as soon as possible so Niagara servers are treated at least the same way as Opteron servers - 3yrs warranty by default. Bronze support should also be offered (at least it's not possible to buy bronze support for T1000/T2000 here in Poland).



Sun's warranty matrix for entry level servers.

Hot Spares in ZFS

Finally Hot Spare support is integrated.

RAID-Z not always a good solution

We all know ZFS is great. RAID-Z means fast RAID-5 without HW RAID controller. However the devil is in the details - while raid-z is great for writing speed, data integrity, etc. its read performance could be really bad if you issue many small random reads from many streams and your dataset is big enough that your cache hit ratio is really small. Sometimes the solution could be to make a pool with many raid-z groups - it means less available storage, but better performance (in terms of IO/s).

So if you want to use raid-z in your environment first carefully consider your workload and if many raid-z in one pool aren't good solution for you then use different raids offered by ZFS. Fortunately other raids in ZFS are NOT affected that way.

If you need more details then read Roch's blog entry on it.

Minimizing Memory Usage for Creating Application Subprocesses

Interesting article on fork()/system()/popen()/posix_spawn() and memory overcommit.

T1000 arrived

Another T1000 arrived few days ago for testing - just after UNIX DAYS I'm going to start testing it.

USDT enhancements

Adam Leventhal has added new features to USDT probes in DTrace - especially is-enabled probes are a great add-on.

PSH + SMF = less downtime

Richard's Ranch has posted some info on Memory Page Retirement in Solaris which is part of Predictive Self Healing. If you want more details you should read: Assessment of the Effect of Memory Page Retirement on System RAS Against Hardware Faults.

What a coincidence because just one day earlier one of our servers encountered uncorrectable memory error. Fortunately it happened in user space so Solaris 10 just cleared that page, killed affected application and thanks to SMF application was automatically restarted. It all happened not only automatically but also quick enough that our monitoring detected problem AFTER Solaris already took care of it and everything was working properly.

Here we have a report in /var/adm/messages about problem with memory.

May  8 22:47:03 syrius.poczta.srv SUNW,UltraSPARC-II: [ID 321281 kern.warning] WARNING: [AFT1] Uncorrectable Memory Error on CPU0 Data access at TL=0, errID 0x000c303b.ed832017
May  8 22:47:03 syrius.poczta.srv     AFSR 0x00000000.00200000 AFAR 0x00000001.f0733b38
May  8 22:47:03 syrius.poczta.srv     AFSR.PSYND 0x0000(Score 05) AFSR.ETS 0x00 Fault_PC 0xffffffff7e7043c8
May  8 22:47:03 syrius.poczta.srv     UDBH 0x00a0 UDBH.ESYND 0xa0 UDBL 0x02fc UDBL.ESYND 0xfc
May  8 22:47:03 syrius.poczta.srv     UDBL Syndrome 0xfc Memory Module Board 6 J????
May  8 22:47:03 syrius.poczta.srv SUNW,UltraSPARC-II: [ID 714160 kern.info] [AFT2] errID 0x000c303b.ed832017 PA=0x00000001.f0733b38
May  8 22:47:03 syrius.poczta.srv     E$tag 0x00000000.18c03e0e E$State: Exclusive E$parity 0x0c
May  8 22:47:03 syrius.poczta.srv SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2] E$Data (0x00): 0x2d002d01.2d022d03
May  8 22:47:03 syrius.poczta.srv SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2] E$Data (0x08): 0x2d672d68.2d692d6a
May  8 22:47:03 syrius.poczta.srv SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2] E$Data (0x10): 0x2d6b2d09.2c912c92
May  8 22:47:03 syrius.poczta.srv SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2] E$Data (0x18): 0x2c932d0d.2d0e2d0f
May  8 22:47:03 syrius.poczta.srv SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2] E$Data (0x20): 0x2d102d11.2d122d13
May  8 22:47:03 syrius.poczta.srv SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2] E$Data (0x28): 0x00000000.09040000
May  8 22:47:03 syrius.poczta.srv SUNW,UltraSPARC-II: [ID 359263 kern.info] [AFT2] E$Data (0x30): 0x000006ea.00002090
May  8 22:47:03 syrius.poczta.srv SUNW,UltraSPARC-II: [ID 989652 kern.info] [AFT2] E$Data (0x38): 0x2091001c.2d1d2d1e *Bad* PSYND=0x00ff
May  8 22:47:03 syrius.poczta.srv unix: [ID 321153 kern.notice] NOTICE: Scheduling clearing of error on page 0x00000001.f0732000
May  8 22:47:03 syrius.poczta.srv SUNW,UltraSPARC-II: [ID 863414 kern.info] [AFT3] errID 0x000c303b.ed832017 Above Error is in User Mode
May  8 22:47:03 syrius.poczta.srv     and is fatal: will SIGKILL process and notify contract
May  8 22:47:20 syrius.poczta.srv unix: [ID 221039 kern.notice] NOTICE: Previously reported error on page 0x00000001.f0732000 cleared

Then by just using 'svcs' I learned which application was restarted and looked into the application's smf log file which has (XXXXX put instead of application path):

[ May  8 22:47:03 Stopping because process killed due to uncorrectable hardware error. ]
[ May  8 22:47:03 Executing stop method ("XXXXXXXX stop") ]
[ May  8 22:47:04 Method "stop" exited with status 0 ]
bash: line 1: 22242 Killed                  LD_PRELOAD=libumem.so.1 XXXXXXXX
[ May  8 22:48:44 Executing start method ("XXXXXXXXX start") ]
[ May  8 22:48:46 Method "start" exited with status 0 ]

UNIX DAYS - Gdansk 2006

UNIX DAYS - Gdansk 2006. This is a second edition of a conference about What's New in UNIX (on a production). The conference will be at Gdansk, Poland on May 18-19th. This time we managed to get Wirtualna Polska, Sun, Symantec, EMC, Implix and local university involved - thank you.

What is it about and how it started?
In October 2005 I thought about creating conference in Poland about new technologies in UNIX made by sysadmins for sysadmins (or by geeks for geeks). The idea was to present new technologies without any marketing crap - just technical. So I asked two of my friends to join me and help me make it real. Then I asked my company, Sun, Veritas and local university to sponsor us - and they did. That way UNIX DAYS 2004 was born. The conference took two days and all speakers were people who were actually using technologies they were talking about in a production environments. I must say that conference was very well received. Unfortunately due to lack of time there was no UNIX DAYS in 2005.

See you at UNIX DAYS!

update: well, we had to close public registration just after one day - over two hundred people registered in one day and we do not have free places yet. It really surprised us.

ps. our www pages are only in Polish - sorry about that.