I've found this
article from IBM which tries to compare some of Solaris 10 features to those available in Linux. Well, in my opinion the article is very biased and not fair to Solaris. Here are some examples.
Dtrace uses an in-kernel interpreter whereas SystemTap uses compiled native code. Compiled native code is faster than interpreted code. Therefore, using SystemTap will not affect the performance of the system while performing performance measurements. The in-kernel interpreter has to be completely bug free, otherwise problems in the interpreter itself can cause the system to crash.
That is interesting 'coz from SystemTap
reference:
When complete, the generated C code is compiled, and linked with the runtime, into a stand-alone kernel module. [...] To run the probes, the systemtap driver program simply loads the kernel module using insmod. The module will initialize itself, insert the probes, then sit back and let the probe handlers be triggered by the system to collect and pass data. It will eventually remove the probes at unload time.
Well, so probe points are executed in-kernel too. I wonder how would SystemTap protect from null pointer dereferences and so on. DTrace uses interpreter so it does check for security and safety execution on the fly and catches possible problems when they occures.
When it comes to security SystemTap reference says:
DProbes exposes the KProbes layer in such a way that it is not crashproof, as it does allow invalid instrumentation requests.
Well, that doesn't sound safe. And then DTrace makes use of Priveleges so you can give DTrace to common (non-root) users without a security risk.
SystemTap doesn't aggregate data at a source so it has to transfer all data from kernel to user space and then filter only what you need. This could be performnace problem. Dtrace on the other hand aggregates data at a source so only revelant data are copied - this is clear performance win.
DTrace has already a lot of useful providers which let you know what's going on system or application in a fast and secure manner. I mean IO provider which lets you measure IO characteristics of your system and applications without knowig the details of how these IOs are generated. Then there're PID, SCHED, PLOCKSTAT, ... The main advantage of using these providers is that you can measure sometinh really useful and solve a problem without knowing exact implmentation in kernel. Hey, we're sys admins and application developers and not kernel programers.
And one more thing
Moreover, the interpreter is newly developed and not as mature as the compiler, hence there is a higher possibility of encountering bugs.
Well, DTrace has been tested in many PRODUCTION enviroments for quite some time (well over a year) and it proved itself excellent. I would say that it's SystemTap that has to yet prove itself. Not to mention that it looks like SystemTap is available for PPC64 starting with lates Linux kernels like 2.6.13-rc1 - which means that it's almost not tested at all and it probably isn't even included in RHEL for PPC while DTrace is already in a stable and production used Solaris 10 not to mention DTrace was available long before and used by many.
Then it looks like SystemTap can trace ONLY kernel functions. Well DTrace can do both: kernel and user space. And this is one of the big advantages of DTrace - you can easly correlate and follow different events from both: kernel and your applications. This is far more complete approach and quite useful in practice. Looking at
porting status matrix of SystemTraps it looks like there's no active development of user space tracing.
Then in Table 1 there's 'Ability to leverage the hardware performance counters of the CPU' and it says 'No' To DTrace and Yes to Linux - however there's footnote (4) that this feature is not currently available on Linux. I can't find footnote for (*) but there should be Yes in DTrace column as well - 'coz it's a planned feature (look at
USENIX'04 paper - Future Work section).
There are some entries in the table about optimizing programs for a particular workload and about generating a flos of instructions. Other tools than SystemTap are specified so to be fair there should be Yes in Solaris column too (for example:
Improving Code Layout Can Improve Application Performance and
SOS10).
Then last two entries in the table are: "Ability to write arbitrary locations in kernel memory" and "Ability to invoke arbitrary kernel subroutines" - of course again 'No' to DTrace and 'Yes' to Linux. Well, it's not perfect but actually you can use
system procedure in DTrace to call
mdb and then you can read/modify kernel memory and/or call kernel subroutines.
Well, SystemTap is lacking many of the features, is not well tested, is lacking good documentation - it looks like SystemTap is currently more a prototype then a working solution, especially on PPC64 platform.
Then in Multithreading Enhancements section they forget to add that there's also TLS support for Solaris x86 in GCC and not only in Sun's CC. Then there's table comparing SPECjbb2000 on v40z and some IBM's server. I don't know what point there's in that comparison as they compare: 64 bit JVM on Solaris with 32bit JVM on Linux, and on SOlaris it's 1.5.0 while on Linux it's 1.4.2. Not to mention that this is on different CPU architectures. This comparison is pointless regarding to this article (I quickly look at some prices of these servers and looks like IBM solution costs much more).
And at the end of the article there's:
The conclusion is clear; Linux provides a wide range of tools and technologies that are technically comparable, or better, than those offered in Solaris 10.
Well, this is really interesting. Let's see it again - article discusses thre features: Dynamic Tracing (DTrace and SystemTap), MPO and threads. Well, SystemTap is still much a work in progress and lacking many features of DTrace and it still has to prove itself in a production. While DTrace has a rich set of additional providers, proved itself in a many production enviroments to be not only helpful but to be secure and scalable. Then I can't find anything in the article which states any possible advantade using MPO or threads on Linux. Rather I find Solaris threads more mature as they've been used for many commercial and in-house applications for years and Solaris is well known for it's threads scalability. Not to mention entire ecosystem of additional applications both for sys admins and developers which makes Solaris threads more mature.
Don't get me wrong - Linux is great. SystemTap is interesting but looking at documentation SystemTap still is not even close to DTrace. It's just that I find this article being really biased toward Linux - but probably I shouldn't be that much suprised - after all it's IBM who wrote it.
And IMHO it's good that Linux develops it's own tools similar to DTrace - after all healthy competition is a good thing.
update: Casper Dik has
posted some more comments on the same article. All I cansay is that I completly agree with Casper.
update:
James Dickens posted his own comments
And one more thing as Casper noticed - SystemTap compiles and loads C code in a kernel as a module! While DTrace runs in kernel only its own code. Now what is more secure? Putting arbitrary code into a kernel (SystemTap) or run only well tested code in a kernel? And it seems that I was right - just by dereferencing null pointer you can still panic whole Linux using KProbes which is not a case with DTrace.
