Wednesday, August 31, 2005


I've got new workstation on my office desk! Actually it arrived two weeks ago but I had not much time to completly migrate my enviroment from old workstation (Sun blade 150) to this new one - W1100z. This one has AMD Opteron 150m 2GB RAM, with two internal disks (mirrored), 19" LCD Sun's monitor, DVD dual-layer burner :)))

I've been (well, I did...) working on my old workstation using Solaris 9 with CDE - it's been really STABLE enviroment and I mean it. I've been using it every day for my job (and notebook for WWW) and it has uptime of over 660 days! I haven't even logout from X session (CDE) for the same time period. Additionally some users were working remotly on this workstation.

bash-2.05$ uptime
8:35am up 663 day(s), 22:13, 5 users, load average: 0.25, 0.25, 0.30

bash-2.05$ ps -ef -o comm,etime|grep Xsun
/usr/openwin/bin/Xsun 662-21:48:03

I know it's silly but I feel sorry I have to turn it off - such a lovely uptime for a workstation. Anyway they are people waiting for this one so I'm gonna migrate completly to new one in next few days.

On my new workstation I put Solaris 10 with NVIDIA drivers Thanks to NVIDIA HW OpenGL and I'm going to try JDS instead of CDE. Well, I'm not expecting the same stability as with CDE but maybe it's good enough. And CDE is getting obsolete anyway. It's been working quite well for last two weeks (except some FireFox crashes everything else is working properly). I tried mplayer from BlastWave - works ok, but I decided to compile it myself so I get xv and CPU extensions - works just great. Xine is working too, so is Xmms. Playing four divX movies at the same time using mplayer is cool :) is working. Burning DVDs also works.

So far so good.

btw: I really like the way disks or DVD/CD are mounted in this system - no screwdrivers - for disks you get special bracket, for DVD/CD there's quite simple and clever solution - anyway no screwdrivers.

This workstation is really fast - I love it!

Monday, August 29, 2005

Wednesday, August 24, 2005

Oracle advertising Solaris x64

Oracle & Solaris x64 ad - well, that's something new.

Dynamic Reconfiguration & Solaris 10

I tried DR yestarday on Solaris 10 - and wow, I'm impressed. With older Solaris releases you couldn't unconfigure&detach CPU/MEM board on which kernel resided - usually it meant lowest CPU/MEM board in a system. I tried to unconfigure such a board on Solaris 10 - and wow! it works!
However there's an issue - system freezes for a moment (2-4 minutes) and then continues to run. It's not a case when you unconfigure a board without kernel on it. It's much better then in previous releases of Solaris.

bash-3.00# prtconf | grep Memory
Memory size: 32768 Megabytes
bash-3.00# psrinfo |wc -l

bash-3.00# cfgadm -alv|grep -i permane
N0.SB2::memory connected configured ok base address 0x0, 16777216 KBytes total, 1447408 KBytes permanent

bash-3.00# cfgadm -c unconfigure N0.SB0
bash-3.00# prtconf | grep Memory
Memory size: 16384 Megabytes
bash-3.00# psrinfo |wc -l

bash-3.00# cfgadm -c configure N0.SB0

bash-3.00# cfgadm -c unconfigure N0.SB2
System may be temporarily suspended, proceed (yes/no)? yes

bash-3.00# prtconf | grep Memory
Memory size: 16384 Megabytes
bash-3.00# psrinfo |wc -l
bash-3.00# cfgadm -alv|grep -i permane
N0.SB0::memory connected configured ok base address 0x0, 16777216 KBytes total, 1447408 KBytes permanent

Monday, August 15, 2005

Sun Cluster 3.1 8/05

New release of Sun Cluster is out! - just in time, as I'm gonna setup new NFS cluster in coming days.

Some selected What's New:
  • support for NAS (NFS v3/v4)
  • support for tagged Virtual Local Area Networks (VLANs) to share an adapter between the private interconnect and the public network
  • HA for Solaris Containers (Zones)
  • support for SMF services
  • support for AMD 64-bit platform
  • support for Oracle 10g (+RAC)
Sounds good. Although there's a problem - HA-NFS doesn't support NFS v4 :(((

Sun Cluster 3.1 8/05 Documentation
Sun Cluster 3.1 8/05 BASE
Sun Cluster 3.1 8/05 Agents

SX b20 is out

Solaris Express - Community Release build 20 is out.

Friday, August 12, 2005

v20z and large internal disks

If you want to make use of all available space on some larger disks (like 300GB) in v20z you need to apply 119375-03 on Solaris 10. Now what if you want to install system over a network and you're using JumpStart? Well, this is something I had to do today.

First, you have to patch net install image on a install server. To do this just run:

# patchadd -C /install/s10-GA-x86/Solaris_10/Tools/Boot 119375-03

This allows to install system using entire disk. But then you need to apply the patch just after a system is installed and before first boot. The best place for it is finish script. All you have to do is to add something like:

inst_arch=`uname -i`
if [ $inst_arch -eq "i86pc" ]; then
echo ' - adding patch 119375-03 for big disks'
patchadd -R /a/ ${SI_CONFIG_DIR}/patches/119375-03

Compiling Open Solaris for fun

I've just compiled opensolaris-src-20050720 on E2900 in less then an hour! (snv_19, SOS, DEBUG).

bash-3.00# time nightly ./

real 51m48.739s
user 199m27.220s
sys 58m57.491s

I tried again but this time put all files to compile on tmpfs filesystem and there's slight improvement.

bash-3.00# time nightly ./

real 46m9.597s
user 197m54.354s
sys 56m52.978s

Wednesday, August 10, 2005

Linux and Solaris

I've found this article from IBM which tries to compare some of Solaris 10 features to those available in Linux. Well, in my opinion the article is very biased and not fair to Solaris. Here are some examples.
Dtrace uses an in-kernel interpreter whereas SystemTap uses compiled native code. Compiled native code is faster than interpreted code. Therefore, using SystemTap will not affect the performance of the system while performing performance measurements. The in-kernel interpreter has to be completely bug free, otherwise problems in the interpreter itself can cause the system to crash.
That is interesting 'coz from SystemTap reference:
When complete, the generated C code is compiled, and linked with the runtime, into a stand-alone kernel module. [...] To run the probes, the systemtap driver program simply loads the kernel module using insmod. The module will initialize itself, insert the probes, then sit back and let the probe handlers be triggered by the system to collect and pass data. It will eventually remove the probes at unload time.
Well, so probe points are executed in-kernel too. I wonder how would SystemTap protect from null pointer dereferences and so on. DTrace uses interpreter so it does check for security and safety execution on the fly and catches possible problems when they occures.

When it comes to security SystemTap reference says:

DProbes exposes the KProbes layer in such a way that it is not crashproof, as it does allow invalid instrumentation requests.

Well, that doesn't sound safe. And then DTrace makes use of Priveleges so you can give DTrace to common (non-root) users without a security risk.

SystemTap doesn't aggregate data at a source so it has to transfer all data from kernel to user space and then filter only what you need. This could be performnace problem. Dtrace on the other hand aggregates data at a source so only revelant data are copied - this is clear performance win.

DTrace has already a lot of useful providers which let you know what's going on system or application in a fast and secure manner. I mean IO provider which lets you measure IO characteristics of your system and applications without knowig the details of how these IOs are generated. Then there're PID, SCHED, PLOCKSTAT, ... The main advantage of using these providers is that you can measure sometinh really useful and solve a problem without knowing exact implmentation in kernel. Hey, we're sys admins and application developers and not kernel programers.

And one more thing

Moreover, the interpreter is newly developed and not as mature as the compiler, hence there is a higher possibility of encountering bugs.
Well, DTrace has been tested in many PRODUCTION enviroments for quite some time (well over a year) and it proved itself excellent. I would say that it's SystemTap that has to yet prove itself. Not to mention that it looks like SystemTap is available for PPC64 starting with lates Linux kernels like 2.6.13-rc1 - which means that it's almost not tested at all and it probably isn't even included in RHEL for PPC while DTrace is already in a stable and production used Solaris 10 not to mention DTrace was available long before and used by many.

Then it looks like SystemTap can trace ONLY kernel functions. Well DTrace can do both: kernel and user space. And this is one of the big advantages of DTrace - you can easly correlate and follow different events from both: kernel and your applications. This is far more complete approach and quite useful in practice. Looking at porting status matrix of SystemTraps it looks like there's no active development of user space tracing.

Then in Table 1 there's 'Ability to leverage the hardware performance counters of the CPU' and it says 'No' To DTrace and Yes to Linux - however there's footnote (4) that this feature is not currently available on Linux. I can't find footnote for (*) but there should be Yes in DTrace column as well - 'coz it's a planned feature (look at USENIX'04 paper - Future Work section).

There are some entries in the table about optimizing programs for a particular workload and about generating a flos of instructions. Other tools than SystemTap are specified so to be fair there should be Yes in Solaris column too (for example: Improving Code Layout Can Improve Application Performance and SOS10).

Then last two entries in the table are: "Ability to write arbitrary locations in kernel memory" and "Ability to invoke arbitrary kernel subroutines" - of course again 'No' to DTrace and 'Yes' to Linux. Well, it's not perfect but actually you can use system procedure in DTrace to call mdb and then you can read/modify kernel memory and/or call kernel subroutines.

Well, SystemTap is lacking many of the features, is not well tested, is lacking good documentation - it looks like SystemTap is currently more a prototype then a working solution, especially on PPC64 platform.

Then in Multithreading Enhancements section they forget to add that there's also TLS support for Solaris x86 in GCC and not only in Sun's CC. Then there's table comparing SPECjbb2000 on v40z and some IBM's server. I don't know what point there's in that comparison as they compare: 64 bit JVM on Solaris with 32bit JVM on Linux, and on SOlaris it's 1.5.0 while on Linux it's 1.4.2. Not to mention that this is on different CPU architectures. This comparison is pointless regarding to this article (I quickly look at some prices of these servers and looks like IBM solution costs much more).

And at the end of the article there's:
The conclusion is clear; Linux provides a wide range of tools and technologies that are technically comparable, or better, than those offered in Solaris 10.
Well, this is really interesting. Let's see it again - article discusses thre features: Dynamic Tracing (DTrace and SystemTap), MPO and threads. Well, SystemTap is still much a work in progress and lacking many features of DTrace and it still has to prove itself in a production. While DTrace has a rich set of additional providers, proved itself in a many production enviroments to be not only helpful but to be secure and scalable. Then I can't find anything in the article which states any possible advantade using MPO or threads on Linux. Rather I find Solaris threads more mature as they've been used for many commercial and in-house applications for years and Solaris is well known for it's threads scalability. Not to mention entire ecosystem of additional applications both for sys admins and developers which makes Solaris threads more mature.

Don't get me wrong - Linux is great. SystemTap is interesting but looking at documentation SystemTap still is not even close to DTrace. It's just that I find this article being really biased toward Linux - but probably I shouldn't be that much suprised - after all it's IBM who wrote it.

And IMHO it's good that Linux develops it's own tools similar to DTrace - after all healthy competition is a good thing.

update: Casper Dik has posted some more comments on the same article. All I cansay is that I completly agree with Casper.

update: James Dickens posted his own comments

And one more thing as Casper noticed - SystemTap compiles and loads C code in a kernel as a module! While DTrace runs in kernel only its own code. Now what is more secure? Putting arbitrary code into a kernel (SystemTap) or run only well tested code in a kernel? And it seems that I was right - just by dereferencing null pointer you can still panic whole Linux using KProbes which is not a case with DTrace.

Tuesday, August 09, 2005

Thursday, August 04, 2005

I/O frameworks

I was looking for something completly different and find an interesting article about different I/O architectures. It's a little bit outdated but still it's worth reading.

Top 10 DTrace scripts

Matty has posted his own Top 10 DTrace scripts. These are really useful and easy to use tools based on DTrace - if you don't know DTrace and want to know more about what's going on in your system you should check them. Even if you know DTrace you should probably look at them - there's always something to learn about.

Another collection of useful tools based on DTrace can be found here. And againa - these are ready to use tools for every sys admin even without understanding DTrace.