milek's blog

MWAC in Global Zone

2012-01-20T18:50:00.001+01:00

Solaris 11 has a new cool feature called Immutable Zones. Darren Moffat presented new features in Solaris 11 Zones at the last LOSUG meeting in London. Immutable Zones basically allow for read-only or partially read-only Zones to be deployed. You can even combine it with ZFS encryption - see Darren's blog entry for more details. The underlying technology to immutable zones is called Mandatory Write Access Control (MWAC) and is implemented in kernel. So for each open, unlink, etc. syscall a VFS layer checks if MWAC is enabled for a given filesystem and a zone and if it is it will check white and black lists associated with a zone and potentially deny write access to a file (generating EROFS). The actual definitions for different default profiles are located in /usr/lib/brand/solaris/config.xml file. It is *very* simple to use the pre-defined profiles when creating a zone and it just works. Really cool. Thanks Darren for the great demo.

Now MWAC only works with non-global zones... at least by default. There is no public interface exposed to manipulate MWAC rules directly or to enable it for a global zone but it doesn't mean one can't try to do it anyway. DTrace, objdump, mdb, etc. were very helpful here to see what's going on. The result of having couple of hours of fun is below.

root@global # touch /test/file1
root@global # rm -f /test/file1
root@global # ./mwac -b "/test/file1"
MWAC black list for the global zone installed.

root@global # touch /test/file1
touch: cannot create /test/file1: Read-only file system
root@global # touch /test/file2 ; rm /test/file2
root@global #

Now lets disable MWAC again:

root@global # mwac -u
MWAC unlock succeeded.

root@global # touch /test/file1 ; rm /test/file1
root@global #

You can even use patterns:

root@global # mwac -b "/test/*"
MWAC black list for the global zone installed.

root@global # touch /test/a ; mkdir /test/b
touch: cannot create /test/a: Read-only file system
mkdir: Failed to make directory "/test/b"; Read-only file system
root@global #

ReFS

2012-01-19T13:26:00.000+01:00

Next generation file system for Windows: ReFS
It looks pretty interesting and promising. Something like ZFS lite for Windows.

"The key goals of ReFS are:
Maintain a high degree of compatibility with a subset of NTFS features that are widely adopted while deprecating others that provide limited value at the cost of system complexity and footprint.
Verify and auto-correct data. Data can get corrupted due to a number of reasons and therefore must be verified and, when possible, corrected automatically. Metadata must not be written in place to avoid the possibility of “torn writes,” which we will talk about in more detail below.
Optimize for extreme scale. Use scalable structures for everything. Don’t assume that disk-checking algorithms, in particular, can scale to the size of the entire file system.
Never take the file system offline. Assume that in the event of corruptions, it is advantageous to isolate the fault while allowing access to the rest of the volume. This is done while salvaging the maximum amount of data possible, all done live.
Provide a full end-to-end resiliency architecture when used in conjunction with the Storage Spaces feature, which was co-designed and built in conjunction with ReFS.
The key features of ReFS are as follows (note that some of these features are provided in conjunction with Storage Spaces).
Metadata integrity with checksums
Integrity streams providing optional user data integrity
Allocate on write transactional model for robust disk updates (also known as copy on write)
Large volume, file and directory sizes
Storage pooling and virtualization makes file system creation and management easy
Data striping for performance (bandwidth can be managed) and redundancy for fault tolerance
Disk scrubbing for protection against latent disk errors
Resiliency to corruptions with "salvage" for maximum volume availability in all cases
Shared storage pools across machines for additional failure tolerance and load balancing

"

What is Watson?

2012-01-04T20:42:00.000+01:00

You probably heard about Watson from IBM. Michael Perrone gave a very entertaining presentation on Watson at LISA. Enjoy.

Solaris 11 Source Code

2011-12-30T13:00:00.001+01:00

It seems that Oracle has "published" S11 source code after all...
And a little bit more on the story.

From SunOS thru Solaris and OpenSolaris to illumos

2011-12-16T14:01:00.001+01:00

Last week I attended LISA '11 conference and one of the great and fun presentations was Bryan Cantrill's talk titled "Fork Yeah! The Rise and Development of illumos".
See the video and the slides.

Also, if you are interested in the whole DevOps transformation watch Ben Rockwood's talk - see the video and the slides.

There were more interesting talks - you can find them at http://www.usenix.org/events/lisa11/tech/

Solaris 11 - hostmodel

2011-11-11T14:31:00.000+01:00

Solaris 11 Express and now Solaris 11 have a new functionality which was long missing in Solaris. It allows to force an application which binds to a given IP address to only use a default gateway configured on the same subnet (hostmodel set to strong), or to prefer such gateway if exists (src-priority) or the default behavior which is to choose "randomly" (weak). This is very useful if you have an additional interface (10GbE for example) and you want a guarantee that all outgoing packets from all applications which bind to an IP address on that interface go thru that dedicated interface even if there are other default gateways on other interfaces/subnets.

From ipadm(1M) man page:

hostmodel (IPv4), hostmodel (IPv6)
Control send/receive behavior for IP packets on a multi-homed system. The value of hostmodel can be set to strong or weak, corresponding to the equivalent end-system model definitions of RFC 1122. In addition, a third value of src-priority is also supported. In the src-priority hostmodel scenario, a packet will be accepted on any interface, as long as the packet's destination IP address is configured and marked UP on one of the host's interfaces. When transmitting a packet, if multiple routes for the IP destination in the packet are available, the system will prefer routes where the IP source address in the packet is configured on the outgoing interface. If no such route is available, the system will fall back to selecting the “best” route, as with the weak ES case.

Live your life

2011-10-14T13:03:00.000+02:00

Oracle porting DTrace to Linux?

2011-10-06T09:51:00.001+02:00

Apparently Oracle announced they will port DTrace to Linux. See comments from Adam Leventhal (co-creator of DTrace).

KVM on Illumos

2011-08-16T18:39:00.001+02:00

This is probably the first big feature/integration into Illumos. See also Bryan's slides on Experiences Portin KVM to SmartOS.

Cool Exadata Advert

2011-06-24T10:53:00.000+02:00

Ilumos Panel Session - Videos

2011-06-18T14:58:00.001+02:00

A little bit too much of unnecessary bitterness towards Solaris 11 but other than that worth watching.

1 http://www.youtube.com/watch?v=NArlja-RXVA

2 http://www.youtube.com/watch?v=PBZRdojTik4

3 http://www.youtube.com/watch?v=lmvPJiYih80

4 http://www.youtube.com/watch?v=Gle4CU_lnls

5 http://www.youtube.com/watch?v=MyDaeQvp6O4

6 http://www.youtube.com/watch?v=eYxQO_OOcSQ

Oracle Corporation on behalf of Dell, Inc

2011-06-07T22:25:00.001+02:00

Out of curiosity I checked the Solaris 11 HCL list. One of the recently added entries is for Dell PE C1100 server and what caught my eye was:

Submitter Company: Oracle Corporation on behalf of Dell, Inc.

There are similar entries for other Dell servers and also for IBM servers (for example: HS22, HX5).
Then there are servers submitted directly by HP (for example: DL585 G7, BL680c G7).

This is a good signal that Oracle is behind Solaris on 3rd party x86 servers - good.

Stopping a thread

2011-05-24T21:15:00.000+02:00

Solaris 11 Express allows to stop/resume a single thread - really cool!
See man pages for pstop and prun.

Solaris 11 Introduction Video

2011-03-21T15:51:00.000+01:00

ZFS returns to MacOS?

2011-03-16T11:11:00.000+01:00

http://info.tenscomplement.com

"We're not quite there yet — but we have some exciting products in development. Our version of ZFS for Mac OS X, Z-410 Storage, started external beta testing last week. Expect more announcements soon."

http://z410.tenscomplement.com

"Our foundational release of ZFS for Mac OS X is targeted at early adopters and those who can't wait to combine the world's most innovative operating system with the world's most advanced file system.
The initial beta evaluation program is in progress (thank you to those participating). We hope to have more product details soon. Those interested in participating in our future test programs can leave their email below."

Read about it also on ZDNet.

SCSI UNMAP in iZFS

2011-03-05T03:12:00.000+01:00

Solaris ZFS got encryption recently and now Illumos ZFS gets SCSI UNMAP support. Are we going to end up with two diverging implementations of ZFS (ZFS proper and iZFS?) in the long term? I guess it is mostly up to Oracle now - it always has been.

Forced Privileges

2011-01-18T12:46:00.000+01:00

Solaris 11 has a new feature called Forced Privileges which allows for a specific set of privileges to be assigned when a given program is run. This makes some setuid binaries more safe and they are no longer really setuid in the traditional sense. There is an excellent blog entry by Darren who explains it in a little bit more detail.

ZFS on Linux - GA Release

2011-01-12T20:47:00.002+01:00

Below announcement was posted earlier today on Open Solaris mailing list:

"
Hi All,

Happy New Year !

First of all, a big thanks to you all for the tremendous response to the beta of ZFS port on Linux.It has been a continuous and arduous effort, not only from our team, but also from you all to rigorously test the beta and make the ZFS port on Linux code fit for use on production servers. Finally, after a three month effort on the testing, making changes and fixing bugs, we now come closer to the GA release.

I am glad to announce that KQ is releasing the GA code for ZFS port on Linux on January 14th. It is the confirmed date and everyone here would be glad to know that majority of the bugs reported have been fixed. The code base is on zpool 28, which means, you have access to dedup feature which wasn't available in the beta earlier !

Also, we are hosting a webinar on January 14th with an interactive conversation with our CEO answering your questions. You can find the details of the webinar on www.kqstor.com

Once again, thank you all very much for the support and patience. We have finally made it!

For any questions, feel free to contact me at darshin@kqinfotech.com

Cheers,
Darshin

Despite anti-Oracle hysteria, firm is an Open Source powerhouse

2011-01-03T15:41:00.001+01:00

"In the end, the key when it comes to deciding who helps and who hurts Open Source, is looking at the big picture, not the vitriol-filled comments, the fireworks and scare tactics of a few competitors who have been trying to destroy Java, OpenOffice.org - and even Oracle - for years.
"

Religion in IT

2010-12-03T01:52:00.002+01:00

Joerg posted:

Interesting statement in a searchdatacenter article about the IDC numbers
“When you sell against Dell, you sell against price. When you sell against HP, you sell against technical stuff -- the feeds and speeds. When you're up against IBM, you're not selling against boxes but against solutions or business outcomes that happen to include hardware. But, when you get to the Sun guys, it's about religion. You can't get to those guys. One guy told me last year that he would get off his Sun box when he dies."

Linux, O_SYNC and Write Barriers

2010-12-02T22:39:00.004+01:00

We all love Linux... sometimes it is better not to look under its hood though as you never know what you might find.

I stumbled across a very interesting discussion on a Linux kernel mailing list. It is dated August 2009 so you may have already read it.

There is a related RH bug.

I'm a little bit surprised by RH attitude in this ticket. IMHO they should have fixed it and maybe provide a tunable which would enable/disable new behavior instead of keeping the broken implementation. But at least in recent man pages they have clarified it in the Notes section of open(2):

"POSIX provides for three different variants of synchronized I/O, corresponding to the flags O_SYNC, O_DSYNC, and O_RSYNC. Currently (2.6.31), Linux only implements O_SYNC, but glibc maps O_DSYNC and O_RSYNC to the same numerical value as O_SYNC. Most Linux file systems don't actually implement the POSIX O_SYNC semantics, which require all metadata updates of a write to be on disk on returning to userspace, but only the O_DSYNC semantics, which require only actual file data and metadata necessary to retrieve it to be on disk by the time the system call returns."

Then there is another even more interesting discussion about write barriers:

"All of them fail to commit drive caches under some circumstances;
even fsync on ext3 with barriers enabled (because it doesn't
commit a journal record if there were writes but no inode change
with data=ordered)."

and also this one:

"No, fsync() doesn't always flush the drive's write cache. It often
does, any I think many people are under the impression it always does, but it doesn't.

Try this code on ext3:

fd = open ("test_file", O_RDWR | O_CREAT | O_TRUNC, 0666);

while (1) {
    char byte;
    usleep (100000);
    pwrite (fd, &byte, 1, 0);
    fsync (fd);
}

It will do just over 10 write ops per second on an idle system (13 on
mine), and 1 flush op per second.

That's because ext3 fsync() only does a journal commit when the inode has changed. The inode mtime is changed by write only with 1 second granularity. Without a journal commit, there's no barrier, which translates to not flushing disk write cache.

If you add "fchmod (fd, 0644); fchmod (fd, 0664);" between the write
and fsync, you'll see at least 20 write ops and 20 flush ops per
second, and you'll here the disk seeking more. That's because the
fchmod dirties the inode, so fsync() writes the inode with a journal
commit.

It turns out even _that_ is not sufficient according to the kernel
internals. A journal commit uses an ordered request, which isn't the
same as a flush potentially, it just happens to use flush in this
instance. I'm not sure if ordered requests are actually implemented
by any drivers at the moment. If not now, they will be one day.

We could change ext3 fsync() to always do a journal commit, and depend on the non-existence of block drivers which do ordered (not flush) barrier requests. But there's lots of things wrong with that. Not least, it sucks performance for database-like applications and virtual machines, a lot due to unnecessary seeks. That way lies wrongness.

Rightness is to make fdatasync() work well, with a genuine flush (or
equivalent (see FUA), only when required, and not a mere ordered
barrier), no inode write, and to make sync_file_range()[*] offer the
fancier applications finer controls which reflect what they actually
need."

This is really scary. I wonder how many developers knew about it especially when coding for Linux when data safety was paramount. Sometimes it feels that some Linux developers are coding to win benchmarks and do not necessarily care about data safety, correctness and standards like POSIX. What is even worse is that some of them don't even bother to tell you about it in official documentation (at least the O_SYNC/O_DSYNC issue is documented in the man page now).

ZFS for Linux

2010-11-25T01:28:00.001+01:00

Solaris 11 Express

2010-11-15T21:02:00.002+01:00

It is based on build 151. The interesting thing is that you can buy a standard support for it
and it doesn't look like Oracle is treating it as beta - at least not officially.
But then why not call it Solaris 11? I guess it is partly due to marketing reasons and partly because it is not entirely ready yet and there are some components which require further work.

ZFS Encryption

2010-10-04T13:59:00.003+02:00

It looks like ZFS crypto project has finally been integrated. It's been reported by other blogger here and the bug id: 4854202 ZFS data set encryption has been updated to reflect that it is in snv_149. Congratulations to Darren Moffat! - the guy behind the project.

It took much longer than expected. If I were to speculate I would say the integration was delayed on purpose so it gets integrated only after the public access to onnv gate was closed so 3rd parties like Nexenta could not take advantage of it before Oracle can. Hopefully once Solaris 11 is out we will see the source code as well. It also probably means that it will be in Solaris 11 Express as well.

Nexenta's take on ZFS, Oracle and NetApp

2010-09-29T09:21:00.004+02:00

Interesting post from Nexenta on recent developments in regards to ZFS, Oracle and NetApp. There is also a podcast which is a Q&A session with Evan Powell, CEO of Nexenta.

Oracle to buy NetApp?

2010-09-24T21:43:00.002+02:00

The Register:

"Oracle boss Larry Ellison said he'd love to have the 60 per cent of NetApp's business that plugs NetApp boxes into Oracle software, hinting that a NetApp purchase could be on his mind.

[...]
A combination of Oracle and NetApp would be a truly formidable server/storage play, even if the IBM NetApp reseller deal subsequently collapsed. Will Oracle actually make an approach? Will NetApp respond positively? The key there is whether it truly believes an independent, best-of-breed stance is viable or whether the stack-in-a-box, centralised IT approach is a sustained tank attack that threatens to flatten its business.
A future inside Oracle in those circumstances might look a lot better than a slow fade into a Unisys-like state while EMC, buoyed up with VMware and products above raw storage arrays, grows and prospers."

Bloomberg also reports on Oracle acquisitions plans.

SPARC T3 On-Chip Security Accelerator

2010-09-22T23:44:00.000+02:00

New SPARC T3 Systems

2010-09-22T23:15:00.002+02:00

Joerg reports:

Oracle announced several new SPARC systems based on the SPARC T3 processor. SPARC T3 is the next iteration of SPARC in the throughput arena: 16 cores running at 1.65 GHz., so 128 threads per socket, on-chip PCIe 2.0, on-chip 10GBe ...

It starts with one 16-core 1.65GHz SPARC T3 processor on the SPARC T3-1 Server . So you get 128 Threads packaged in a 2 rack units chassis.

Above this you will find the SPARC T3-2 with two SPARC T3 processors. It gives you 256 threads and 256 GB. You get this amount of power in 3 RU.

At the high-end is the T3-4 with four of the SPARC T3 CPU giving you 512 threads, 512 GB of main memory with 8 GB DIMMs and 16 PCI Express Modules. This system is somewhat larger: 5 RU.

At last there is a new blade for the Blade 6000 chassis: SPARC T3-1B. As the name suggests it provides 1 SPARC T3 processor, so 128 Threads on one blade.

The Register reports on the new systems as well.

If you are interested in a brief summary of what T3 is read Joerg post about it here.

OpenIndiana

2010-09-14T21:17:00.003+02:00

See the announcement and slides. You can also download ISOs here which are based on snv_147.

So it has begun... Is it going to survive in the long term?

OpenIndiana

2010-09-10T22:44:00.001+02:00

http://openindiana.org

OpenIndiana is a continuation of the OpenSolaris operating system. It was conceived during the period of uncertainty following the Oracle takeover of Sun Microsystems, after several months passed with no binary updates made available to the public. The formation proved timely, as Oracle discontinued OpenSolaris soon after in favour of Solaris 11 Express, a binary distribution with a more closed development model to debut later this year.

OpenIndiana is part of the Illumos Foundation, and provides a true open source community alternative to Solaris 11 and Solaris 11 Express, with an open development model and full community participation.

Announcement Details

We will be holding a press conference in London (UK) at the JISC offices at 6:30pm UK Time (BST, GMT+1) to formally announce the project and provide full details, including information on gaining access to download our first development release. In addition, we will be broadcasting live on the internet – please see http://openindiana.org/announcement for details of attending in person or on the web.

We believe this announcement will deliver the distribution the community has long sought after.

Solaris 10 License Change

2010-09-10T22:03:00.001+02:00

Joerg reports on Solaris 10 license change.

Solaris 10 9/10

2010-09-10T21:59:00.001+02:00

What's New:

Installation Enhancements

SPARC: Support for ITU Construction Tools on SPARC Platforms - In this release, the itu utility has been modified to support booting a SPARC based system with the install-time updates (ITU) process.
Oracle Solaris Auto Registration - before you ask ... you can disable it
Oracle Solaris Upgrade Enhancement for Oracle Solaris Zone– Cluster Nodes

Virtualization Enhancements for Oracle Solaris Zones

Migrating a Physical Oracle Solaris 10 System Into a Zone - cool, my RfE found its way into Solaris 10
Host ID Emulation
Updating Packages by Using the New zoneadm attach -U Option

Virtualization Enhancements for Oracle VM Server for SPARC (formerly known as LDOMs)

Memory Dynamic Reconfiguration Capability
Virtual Disk Multipathing Enhancements
Static Direct I/O
Virtual Domain Information Command and API - the virtinfo command

System Administration Enhancements

Oracle Solaris ZFS Features and Enhancements

ZFS device replacement enhancements - namely autoexpand
some changes to the zpool list command
Holding ZFS snapshots
Triple parity RAID-Z (raidz3)
The logbias property
Log device removal - at last
ZFS storage pool recovery
New ZFS system process – In this release, each storage pool has an associated process, zpool-poolname
Splitting a mirrored ZFS storage pool (zpool split)

Fast Crash Dump
x86: Support for the IA32_ENERGY_PERF_BIAS MSR
Support for Multiple Disk Sector Size
iSCSI Initiator Tunables
Sparse File Support in the cpio Command
x86: 64-Bit libc String Functions Improvements With SSE
Automated Rebuilding of sendmail Configuration Files
Automatic Boot Archive Recovery

Security Enhancements

net_access Privilege
x86: Intel AES-NI Optimization

Language Support Enhancements

New Oracle Solaris Unicode Locales

Device Management Enhancements

iSCSI Boot
iSER Initiator
New Hot-Plugging Features
AAC RAID Power Management

Driver Enhancements

x86: HP Smart Array HBA Driver
x86: Support for Broadcom NetXtreme II 10 Gigabit Ethernet NIC Driver
x86: New SATA HBA Driver, bcm_sata, for Broadcom HT1000 SATA Controllers
Support for SATA/AHCI Port Multiplier
Support for Netlogic NLP2020 PHY in the nxge Driver

Freeware Enhancements

GNU TAR Version 1.23
Firefox 3.5
Thunderbird 3
Less Version 436

Networking Enhancements

BIND 9.6.1 for the Oracle Solaris 10 OS
GLDv3 Driver APIs
IPoIB Connected Mode
Open Fabrics User Verbs Primary Kernel Components
InfiniBand Infrastructure Enhancements

X11 Windowing Enhancements

Support for the setxkbmap Command

New Chipset Support

ixgbe Driver to Integrate Intel Shared Code Version 3.1.9
Broadcom Support to bge Networking Driver
x86: Fully Buffered DIMM Idle Power Enhancement

Fault Management Architecture Enhancements

FMA Support for AMD's Istanbul Based Systems
Several Oracle Solaris FMA Enhancement

Diagnostic Tools Enhancements

Sun Validation Test Suite 7.0ps9
Enhancements to the mdb Command to Improve the Debugging Capability of kmem and libumem

You wil find a more in-depth description at docs.sun.com.

NetApp, Oracle Drop ZFS LawSuits

2010-09-10T21:53:00.002+02:00

The Wall Street Journal:

"NetApp Inc. and Oracle Corp. have agreed to dismiss patent lawsuits against each other, putting to bed a battle that had been ongoing since 2007.

The terms of the agreement weren't disclosed, and the companies said they are seeking to have the lawsuits dismissed without prejudice.

NetApp President and Chief Executive Tom Georgens said Thursday the companies would continue to collaborate in the future.

NetApp, which sells data-storage systems, and Sun Microsystems, which has since been bought by Oracle, began the fight in 2007, with each company alleging the other was infringing some of its patents.

Lawsuits were filed on both companies' behalf, with NetApp alleging that Sun's ZFS file-system-management technology infringed a number of its patents. Sun at the time also said NetApp products infringed its patents."

The Future of Solaris

2010-08-28T00:40:00.003+02:00

Below is a collection of blog entries on what's going on recently in regards to Open Solaris and Solaris 11.

Adam Leventhal's take on Solaris future
Bryan Cantrill's comments on Illumos
Garrett D'Amore - The Hand May Be Forced
Garrett D'Amore - Illumos
Ben Rockwood - Illumos Shines New Light
Ben Rockwood - OpenSolaris R.I.P.
Steve Stallion - OpenSolaris is Dead
Jim Laurent - Solaris 11 in 2011
The Register - Next Solaris prepped for 2011

Illumos

2010-08-04T00:01:00.001+02:00

Bryan Cantrill's comment.

SMF/FMA Update

2010-07-31T11:40:00.002+02:00

A rather large and interesting putback for SMF/FMA related technologies went into Open Solaris yesterday. It will be available in build 146.

PSARC/2009/617 Software Events Notification Parameters CLI
PSARC/2009/618 snmp-notify: SNMP Notification Daemon for Software Events
PSARC/2009/619 smtp-notify: Email Notification Daemon for Software Events
PSARC/2010/225 fmd for non-global Solaris zones
PSARC/2010/226 Solaris Instance UUID
PSARC/2010/227 nvlist_nvflag(3NVPAIR)
PSARC/2010/228 libfmevent additions
PSARC/2010/257 sysevent_evc_setpropnvl and sysevent_evc_getpropnvl
PSARC/2010/265 FMRI and FMA Event Stabilty, 'ireport' category 1 event class, and the 'sw' FMRI scheme
PSARC/2010/278 FMA/SMF integration: instance state transitions
PSARC/2010/279 Modelling panics within FMA
PSARC/2010/290 logadm.conf upgrade
6392476 fmdump needs to pretty-print
6393375 userland ereport/ireport event generation interfaces
6445732 Add email notification agent for FMA and software events
6804168 RFE: Allow an efficient means to monitor SMF services status changes
6866661 scf_values_destroy(3SCF) will segfault if is passed NULL
6884709 Add snmp notification agent for FMA and software events
6884712 Add private interface to tap into libfmd_msg macro expansion capabilities
6897919 fmd to run in a non-global zone
6897937 fmd use of non-private doors is not safe
6900081 add a UUID to Solaris kernel image for use in crashdump identification
6914884 model panic events as a defect diagnosis in FMA
6944862 fmd_case_open_uuid, fmd_case_uuisresolved, fmd_nvl_create_defect
6944866 log legacy sysevents in fmd
6944867 enumerate svc scheme in topo
6944868 software-diagnosis and software-response fmd modules
6944870 model SMF maintenance state as a defect diagnosis in FMA
6944876 savecore runs in foreground for systems with zfs root and dedicated dump
6965796 Implement notification parameters for SMF state transitions and FMA events
6968287 SUN-FM-MIB.mib needs to be updated to reflect Oracle information
6972331 logadm.conf upgrade PSARC/2010/290

Dell and HP Continue Supporting Solaris

2010-07-29T21:13:00.002+02:00

New announcement about Solaris support on non-Oracle servers:

Oracle today announced Dell and HP will certify and resell Oracle Solaris, Oracle Enterprise Linux and Oracle VM on their respective x86 platforms.

Customers will have full access to Oracle’s Premier Support for Oracle Solaris, Oracle Enterprise Linux and Oracle VM running on Dell and HP servers. This will enable fast and accurate issue resolution and reduced risk in a company’s operating environment.

Customers who subscribe to Oracle Premier Support will benefit from Oracle’s continuing investment in Oracle Solaris, Oracle Enterprise Linux and Oracle VM and the resulting innovation in future updates.

What Motivates Us

2010-07-21T18:02:00.001+02:00

Trailer: Java 4-ever

2010-06-26T12:07:00.001+02:00

zpool scrub - 4GB/s

2010-06-16T16:27:00.004+02:00

Open Solaris 2009.06 + entry level 2U x86 server Sun Fire x4270 + 1U xxxx array.

# iostat -xnzCM 1|egrep "device|c[0123]$"
[...]
                 extended device statistics           
 r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
8182.1    0.0 1022.1    0.0  0.1 152.8    0.0   18.7   0 1077 c0
8179.1    0.0 1021.7    0.0  0.1 148.7    0.0   18.2   0 1076 c1
8211.0    0.0 1025.9    0.0  0.1 162.8    0.0   19.8   0 1081 c2
8218.0    0.0 1026.8    0.0  0.1 164.5    0.0   20.0   0 1085 c3
                 extended device statistics           
 r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
8080.2    0.0 1010.0    0.0  0.1 168.3    0.0   20.8   0 1070 c0
8080.2    0.0 1010.0    0.0  0.1 167.6    0.0   20.7   0 1071 c1
8165.2    0.0 1020.3    0.0  0.1 166.0    0.0   20.3   0 1079 c2
8157.2    0.0 1019.3    0.0  0.1 151.4    0.0   18.6   0 1080 c3
                 extended device statistics           
 r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
8192.0    0.0 1023.4    0.0  0.1 174.6    0.0   21.3   0 1085 c0
8190.9    0.0 1023.1    0.0  0.1 174.2    0.0   21.3   0 1085 c1
8140.9    0.0 1016.9    0.0  0.1 145.5    0.0   17.9   0 1078 c2
8138.9    0.0 1016.7    0.0  0.1 142.7    0.0   17.5   0 1075 c3
                 extended device statistics           
 r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
8129.1    0.0 1015.6    0.0  0.1 153.0    0.0   18.8   0 1066 c0
8125.2    0.0 1015.1    0.0  0.1 155.1    0.0   19.1   0 1067 c1
8156.2    0.0 1018.8    0.0  0.1 162.1    0.0   19.9   0 1074 c2
8159.2    0.0 1019.2    0.0  0.1 162.0    0.0   19.9   0 1076 c3
                 extended device statistics           
 r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
8177.9    0.0 1022.0    0.0  0.1 165.0    0.0   20.2   0 1088 c0
8184.9    0.0 1022.9    0.0  0.1 165.2    0.0   20.2   0 1085 c1
8209.9    0.0 1026.1    0.0  0.1 162.4    0.0   19.8   0 1085 c2
8204.9    0.0 1025.5    0.0  0.1 161.8    0.0   19.7   0 1087 c3
                 extended device statistics           
 r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
8236.4    0.0 1029.2    0.0  0.1 170.1    0.0   20.7   0 1092 c0
8235.4    0.0 1029.0    0.0  0.1 170.2    0.0   20.7   0 1093 c1
8215.4    0.0 1026.4    0.0  0.1 165.3    0.0   20.1   0 1091 c2
8220.4    0.0 1027.0    0.0  0.1 164.9    0.0   20.1   0 1090 c3


Then with a small I/O it can sustain over 400k IOPS - more HBAs should deliver even more performance.

It is really amazing how fast technology is progressing.
To achieve above numbers 10 years ago it would have cost a small fortune.

Heat Maps

2010-06-12T12:05:00.003+02:00

Brendan Gregg wrote an article in ACM Queue about Visualizing System Latency as heat maps. The article explains really well what latency heat maps are and how to read them. It is also a good read if you want to learn about a rainbow pterodactyl (shown below) flying over an icy lake inside a disk array.

WAN Emulation - hxbt

2010-06-12T10:31:00.001+02:00

I didn't know about it :)

Open Solaris Roadmap

2010-06-10T11:21:00.002+02:00

It is rather brief but at least it is something.
See also a nice commercial below.

DTrace TCP and UDP providers

2010-05-28T11:31:00.002+02:00

Last night two new DTrace providers were integrated.
They should be available in a build 142 of Open Solaris.

PSARC 2010/106 DTrace TCP and UDP providers
"This case adds DTrace 'tcp' and 'udp' providers with probes
for send and receive events.  These providers cover the TCP
and UDP protocol implementations in OpenSolaris respectively. In
addition the tcp provider contains probes for TCP state machine
transitions and significant events in connection processing
(connection request, acceptance, refusal etc). The udp provider
also contains probes which fire when a UDP socket is opened/closed.
This is intended for use by customers for network observability and
troubleshooting, and this work represents the second and third
components of a suite of planned providers for the network stack. The
first was described in PSARC/2008/302 DTrace IP Provider."

The tcp provider is described here:

http://wikis.sun.com/display/DTrace/tcp+Provider

...and the udp provider is described here:

http://wikis.sun.com/display/DTrace/udp+Provider

ZFS - synchronous vs. asynchronous IO

2010-05-04T14:06:00.010+02:00

Sometimes it is very useful to be able to disable a synchronous behavior of a filesystem. Unfortunately not all applications provide such functionality. With UFS many used fastfs from time to time, however the problem is that it can potentially lead to a filesystem corruption. In case of ZFS many people have been using an undocumented zil_disable tunable. While it can cause a data corruption from an application point of view it doesn't impact ZFS on-disk consistency. This is good as it makes the feature very useful, with a much smaller risk but can greatly improve a performance in some cases like database imports, nfs servers, etc. The problem with the tunable is that it is unsupported, has a server-wide impact and affects only newly mounted zfs filesystems while has an instant effect on zvols.

From time to time there were requests here and there to get it implemented properly in a fully supported way. I thought it might be a good opportunity to re-fresh my understanding of Open Solaris and ZFS internals so a couple of months ago I decided to implement it under: 6280630 zil synchronicity.

And it was a fun - I really enjoyed it. I spent most of the time trying to understand the interactions between ZIL/VNODE/VFS layers and the structure of ZFS code. I was already familiar with it to some extend as I contributed a code to ZFS in the past and I also do read the code from time to time when I do some performance tuning, etc. Once I understood what's going on there it was really easy to do the actual coding. Once I got a basic functionality working and I asked for a sponsor so it gets integrated. Tim Haley offered to sponsor me and help me to get it integrated. Couple of moths later, after a PSARC case, code reviews, email exchanges, testing it got finally integrated and should appear in build 140.

I would like to thank Tim Haley, Mark Musante and Neil Perin for all their comments, code reviews, testing, PSARC case handling, etc. It was a real pleasure to work with you.

PSARC/2010/108 zil synchronicity

ZFS datasets now have a new 'sync' property to control synchronous behavior.
The zil_disable tunable to turn synchronous requests into asynchronous requests (disable the ZIL) has been removed. For systems that use that switch on upgrade you will now see a message on booting:


sorry, variable 'zil_disable' is not defined in the 'zfs' module

Please update your system to use the new sync property.
Here is a summary of the property:

-------

The options and semantics for the zfs sync property:


sync=standard
  This is the default option. Synchronous file system transactions
  (fsync, O_DSYNC, O_SYNC, etc) are written out (to the intent log)
  and then secondly all devices written are flushed to ensure
  the data is stable (not cached by device controllers).

sync=always
  For the ultra-cautious, every file system transaction is
  written and flushed to stable storage by a system call return.
  This obviously has a big performance penalty.

sync=disabled
  Synchronous requests are disabled.  File system transactions
  only commit to stable storage on the next DMU transaction group
  commit which can be many seconds.  This option gives the
  highest performance.  However, it is very dangerous as ZFS
  is ignoring the synchronous transaction demands of
  applications such as databases or NFS.
  Setting sync=disabled on the currently active root or /var
  file system may result in out-of-spec behavior, application data
  loss and increased vulnerability to replay attacks.
  This option does *NOT* affect ZFS on-disk consistency.
  Administrators should only use this when these risks are understood.

The property can be set when the dataset is created, or dynamically, and will take effect immediately. To change the property, an administrator can use the standard 'zfs' command. For example:


 # zfs create -o sync=disabled whirlpool/milek
 # zfs set sync=always whirlpool/perrin

Have a fun!

Gartner on Oracle/Sun New Support Model

2010-04-29T15:07:00.001+02:00

"Oracle announces its new Sun support policy that has the potential to radically change the way in which OEMs offer support and may make third-party maintenance offerings for Sun hardware unprofitable."

Memory DeDuplication in Linux

2010-04-27T02:24:00.001+02:00

Oracle Solaris on HP ProLiant Servers

2010-04-24T13:04:00.002+02:00

Recently there has been lots of confusion regarding running Solaris 10 on non-Sun servers.

HP Oracle Solaris 10 Subscriptions and Support:

"Certifying Oracle Solaris on ProLiant servers since 1996, HP is expanding its relationship with Oracle to include selling Oracle Solaris 10 Operating System Subscriptions and support from HP Technology Services on certified ProLiant servers.

HP will provide the subscriptions and support for the Oracle Solaris 10 Operating System on certified ProLiant servers and Oracle will provide patches and updates directly to HP's customers through Oracle SunSolve.

As part of this expanded relationship HP and Oracle will work together to enhance the customer experience for Oracle Solaris on ProLiant servers and HP increase its participation in the OpenSolaris community."

And of course you can also run Open Solaris on any x86 hardware, including HP servers, entirely for free if you want. I wonder though if it would make sense for HP to also offer support for Open Solaris - more and more customers are deploying Open Solaris instead of Solaris 10 on their servers and Oracle already offers a support for it on their own servers.

ZFS diff

2010-03-29T17:47:00.002+02:00

PSARC/2010/105:


        There is a long-standing RFE for zfs to be able to describe
        what has changed between the snapshots of a dataset.
        To provide this capability, we propose a new 'zfs diff'
        sub-command.  When run with appropriate privilege the
        sub-command describes what file system level changes have
        occurred between the requested snapshots.  A diff between the
        current version of the file system and one of its snapshots is
        also supported.

        Five types of change are described:

        o    File/Directory modified
        o    File/Directory present in older snapshot but not newer
        o    File/Directory present in newer snapshot but not older
        o    File/Directory renamed
        o    File link count changed

      zfs diff snapshot  snapshot | filesystem

         Gives a high level description of the differences between a
         snapshot and a descendant dataset.  The descendant may either
         be a later snapshot of the dataset or the current dataset.
         For each file system object that has undergone a change
         between the original snapshot and the descendant, the type of
         change is described along with the name of the file or
         directory.  In the case of a rename, both the old and new
         names are shown.

         The type of change is described with a single character:

         +   Indicates the file/directory was added in the later dataset
         -   Indicates the file/directory was removed in the later dataset
         M   Indicates the file/directory was modified in the later dataset
         R   Indicates the file/directory was renamed in the later dataset

        If the modification involved a change in the link count of a
        file, the change will be expressed as a delta within
        parentheses on the modification line.  Example outputs are
        below:

         M       /myfiles/
         M       /myfiles/link_to_me   (+1)
         R       /myfiles/rename_me -> /myfiles/renamed
         -       /myfiles/delete_me
         +       /myfiles/new_file

Project Brussels Phase II

2010-03-27T14:25:00.002+01:00

This project introduces a new CLI utility called ipadm(1M) that can be used to perform:

* IP interfaces management (creation/deletion)
* IP address management (add, delete, show) for static IPv4& IPv6 addresses, DHCP, stateless/stateful IPv6 Address configuration
* protocol (IP/TCP/UDP/SCTP/ICMP) tunable management (set, get, reset) global (ndd(1M)) tunables, as well as per-interface tunables
* provide persistence for all of the three features above so that on reboot the configuration is reapplied

Please see the case materials of PSARC 2010/080 for the latest design document and read ipadm(1M) man page for more information.

This has been integrated into Open Solaris.

CPU/MEM HotPlug on x86 in Open Solaris

2010-03-26T14:27:00.001+01:00

The integration of:

 PSARC/2009/104 Hot-Plug Support for ACPI-based Systems
 PSARC/2009/550 PSMI extensions for CPU Hotplug
 PSARC/2009/551 acpihpd ACPI Hotplug Daemon
 PSARC/2009/591 Attachment Points for Hotpluggable x86 systems
 6862510 provide support for cpu hot add on x86
 6874842 provide support for memory hot add on x86
 6883891 cmi interface needs to support dynamic reconfiguration
 6884154 x2APIC and kmdb may not function properly during CPU hotplug event.
 6904971 low priority acpi nexus code review feedback
 6877301 lgrp should support memory hotplug flag in SRAT table

Introduces support for hot-adding cpus and memory to a running Xeon 7500 platform.

ReadyBoost

2010-02-28T23:18:00.002+01:00

I didn't know that Windows has a similar technology to ZFS L2ARC which is called ReadyBoost. Nice.

I'm building my new home NAS server and I'm currently seriously considering putting OS on an USB pen drive leaving all sata disks for data only. It looks like with modern USB drives OS should actually boot faster than from a sata disks thanks to much better seek times. I'm planning on doing some experiments first.

ZVOLs - Write Cache

2010-02-25T15:05:00.002+01:00

When you create a ZFS volume its write cache is disabled by default meaning that all writes to the volume will be synchronous. Sometimes it might be handy though to be able to enable a write cache for a particular zvol. I wrote a small C program which allows you to check if WC is enabled or not. It also allows you to enable or disable write cache for a specified zvol.

First lets check if write cache is disabled for a zvol rpool/iscsi/vol1


milek@r600:~/progs# ./zvol_wce /dev/zvol/rdsk/rpool/iscsi/vol1
Write Cache: disabled

Now lets issue 1000 writes


milek@r600:~/progs# ptime ./sync_file_create_loop /dev/zvol/rdsk/rpool/iscsi/vol1 1000

real       12.013566363
user        0.003144874
sys         0.104826470

So it took 12s and I also confirmed that writes were actually being issued to a disk drive. Lets enable write cache now and repeat 1000 writes


milek@r600:~/progs# ./zvol_wce /dev/zvol/rdsk/rpool/iscsi/vol1 1
milek@r600:~/progs# ./zvol_wce /dev/zvol/rdsk/rpool/iscsi/vol1
Write Cache: enabled

milek@r600:~/progs# ptime ./sync_file_create_loop /dev/zvol/rdsk/rpool/iscsi/vol1 1000

real        0.239360231
user        0.000949655
sys         0.019019552

Worked fine.

The zvol_wce program is not idiot-proof and it doesn't check if operation succeeded or not. You should be able to compile it by issuing: gcc -o zvol_wce zwol_wce.c


milek@r600:~/progs# cat zvol_wce.c

/* Robert Milkowski
  http://milek.blogspot.com
*/

#include <unistd.h>
#include <stropts.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stropts.h>
#include <sys/dkio.h>


int main(int argc, char **argv)
{
 char *path;
 int wce = 0;
 int rc;
 int fd;

 path = argv[1];

 if ((fd = open(path, O_RDONLY|O_LARGEFILE)) == -1)
   exit(2);

 if (argc>2) {
   wce = atoi(argv[2]) ? 1 : 0;
   rc = ioctl(fd, DKIOCSETWCE, &wce);
 }
 else {
   rc = ioctl(fd, DKIOCGETWCE, &wce);
   printf("Write Cache: %s\n", wce ? "enabled" : "disabled");
 }

 close(fd);
 exit(0);
}

60 Disks in 4U

2010-02-16T12:01:00.001+01:00

Chip and PIN is Broken

2010-02-12T15:24:00.000+01:00

Dell - No 3rd Party Disk Drives Allowed

2010-02-10T16:55:00.002+01:00

Third-party drives not permitted:

"[...]
Is Dell preventing the use of 3rd-party HDDs now?
[....]
Howard_Shoobe at Dell.com:
Thank you very much for your comments and feedback regarding exclusive use of Dell drives. It is common practice in enterprise storage solutions to limit drive support to only those drives which have been qualified by the vendor. In the case of Dell's PERC RAID controllers, we began informing customers when a non-Dell drive was detected with the introduction of PERC5 RAID controllers in early 2006. With the introduction of the PERC H700/H800 controllers, we began enabling only the use of Dell qualified drives. There are a number of benefits for using Dell qualified drives in particular ensuring a positive experience and protecting our data. While SAS and SATA are industry standards there are differences which occur in implementation. An analogy is that English is spoken in the UK, US and Australia. While the language is generally the same, there are subtle differences in word usage which can lead to confusion. This exists in storage subsystems as well. As these subsystems become more capable, faster and more complex, these differences in implementation can have greater impact. Benefits of Dell's Hard Disk and SSD drives are outlined in a white paper on Dell's web site at http://www.dell.com/downloads/global/products/pvaul/en/dell-hard-drives-pov.pdf"

I understand they won't support 3rd party disk drives but blocking a server (a RAID card) from using such disks is something new - an interesting comment here.

IBM 2010: Customers in Revolt

2010-02-10T01:23:00.002+01:00

From my own experience their sales people are very aggressive with an attitude of sale first and let someone else worry later. While I always take any vendor claims with a grain of salt I learnt to double or even triple check any IBM's claims.

Power 7

2010-02-09T12:52:00.002+01:00

Now lets wait for some benchmarks. I only wish Solaris was running on them as well as right now you need to go the legacy AIX route or not so mature Linux route - not an ideal choice.

Data Corruption - ZFS saves the day, again

2010-02-04T14:22:00.007+01:00

We came across an interesting issue with data corruption and I think it might be interesting to some of you. While preparing a new cluster deployment and filling it up with data we suddenly started to see below messages:


XXX cl_runtime: [ID 856360 kern.warning] WARNING: QUORUM_GENERIC: quorum_read_keys error:
 Reading the registration keys failed on quorum device /dev/did/rdsk/d7s2 with error 22.

The d7 quorum device was marked as being offline and we could not bring it online again. There isn't much in documentation about the above message except that it is probably a firmware problem on a disk array and we should contact a vendor. But lets investigate first what is really going on.

By looking at the source code I found that the above message is printed from within quorum_device_generic_impl::quorum_read_keys() and it will only happen if quorum_pgre_key_read() returns with return code 22 (actually any other than 0 or EACCESS but from the syslog message we already suspect that the return code is 22).

The quorum_pgre_key_read() calls quorum_scsi_sector_read() and passes its return code as its own. The quorum_scsi_sector_read() will return with an error only if quorum_ioctl_with_retries() returns with an error or if there is a checksum mismatch.

This is the relevant source code:


406 int
407 quorum_scsi_sector_read(
[...]
449  error = quorum_ioctl_with_retries(vnode_ptr, USCSICMD, (intptr_t)&ucmd,
450      &retval);
451  if (error != 0) {
452   CMM_TRACE(("quorum_scsi_sector_read: ioctl USCSICMD "
453       "returned error (%d).\n", error));
454   kmem_free(ucmd.uscsi_rqbuf, (size_t)SENSE_LENGTH);
455   return (error);
456  }
457
458  //
459  // Calculate and compare the checksum if check_data is true.
460  // Also, validate the pgres_id string at the beg of the sector.
461  //
462  if (check_data) {
463   PGRE_CALCCHKSUM(chksum, sector, iptr);
464
465   // Compare the checksum.
466   if (PGRE_GETCHKSUM(sector) != chksum) {
467    CMM_TRACE(("quorum_scsi_sector_read: "
468        "checksum mismatch.\n"));
469    kmem_free(ucmd.uscsi_rqbuf, (size_t)SENSE_LENGTH);
470    return (EINVAL);
471   }
472
473   //
474   // Validate the PGRE string at the beg of the sector.
475   // It should contain PGRE_ID_LEAD_STRING[1|2].
476   //
477   if ((os::strncmp((char *)sector->pgres_id, PGRE_ID_LEAD_STRING1,
478       strlen(PGRE_ID_LEAD_STRING1)) != 0) &&
479       (os::strncmp((char *)sector->pgres_id, PGRE_ID_LEAD_STRING2,
480       strlen(PGRE_ID_LEAD_STRING2)) != 0)) {
481    CMM_TRACE(("quorum_scsi_sector_read: pgre id "
482        "mismatch. The sector id is %s.\n",
483        sector->pgres_id));
484    kmem_free(ucmd.uscsi_rqbuf, (size_t)SENSE_LENGTH);
485    return (EINVAL);
486   }
487
488  }
489  kmem_free(ucmd.uscsi_rqbuf, (size_t)SENSE_LENGTH);
490
491  return (error);
492 }

With a simple DTrace script I could verify if the quorum_scsi_sector_read() does indeed return with 22 and also I could print what else is going on within the function:


56  -> __1cXquorum_scsi_sector_read6FpnFvnode_LpnLpgre_sector_b_i_ 6308555744942019 enter
56    -> __1cZquorum_ioctl_with_retries6FpnFvnode_ilpi_i_ 6308555744957176 enter
56    <- __1cZquorum_ioctl_with_retries6FpnFvnode_ilpi_i_ 6308555745089857 rc: 0 
56    -> __1cNdbg_print_bufIdbprintf6MpcE_v_ 6308555745108310 enter
56      -> __1cNdbg_print_bufLdbprintf_va6Mbpcrpv_v_ 6308555745120941 enter
56        -> __1cCosHsprintf6FpcpkcE_v_      6308555745134231 enter
56        <- __1cCosHsprintf6FpcpkcE_v_      6308555745148729 rc: 2890607504684 
56      <- __1cNdbg_print_bufLdbprintf_va6Mbpcrpv_v_ 6308555745162898 rc: 1886718112 
56    <- __1cNdbg_print_bufIdbprintf6MpcE_v_ 6308555745175529 rc: 1886718112 
56  <- __1cXquorum_scsi_sector_read6FpnFvnode_LpnLpgre_sector_b_i_ 6308555745188599 rc: 22

From the above output we know that the quorum_ioctl_with_retries() returns with 0 so it must be a checksum mismatch! As CMM_TRACE() is being called above and there are only three of them in the code lets check with DTrace which one it is:


21  -> __1cNdbg_print_bufIdbprintf6MpcE_v_   6309628794339298 quorum_scsi_sector_read: checksum mismatch.

So now I knew exactly what part of the code is casing the quorum device to be marked offline. The issue might have been caused by many things like: a bug in a disk array firmware, a problem on an SAN, a bug in a HBA's firmware, a bug in a qlc driver or a bug in SC software, or... However because the issue suggests a data corruption and we are loading the cluster with a copy of a database we might have a bigger issue that just an offline quorum device. The configuration is a such that we are using ZFS to mirror between two disks arrays. We have been restoring a couple of TBs of data into and we haven't read almost anything back. Thankfully it is ZFS so we might force a re-check off all data in the pool and I did. ZFS found 14 corrupted blocks and even identified which file is affected. The interesting thing here is that for all blocks both copies on both sides of the mirror were affected. This almost eliminates a possibility of a firmware problem on disk arrays and suggest that the issue was caused by something misbehaving on the host itself. There is still a possibility of an issue on SAN as well. It is very unlikely to be a bug in ZFS as the corruption affected reservation keys as well which has basically nothing to do with ZFS at all. Then we are still writing more and more data into the pool and I'm repeating scrubs and I'm not getting any new corrupted blocks nor quorum is misbehaving (I fixed it by temporarily adding another one, removing the original and re-adding it again while removing the temporary one).

While I still have to find what caused the data corruption the most important thing here is ZFS. Just think about it - what would happen if we were running on any other file system like: UFS, VxFS, ext3, ext4, JFS, XFS, ... Well, almost anything could have happened with them like some data of could be corrupted, some files lost, system could crash, fsck could be forced to run for many hours and still not being able to fix the filesystem and it definitely wouldn't be able to detect any data corruption withing files or everything would be running fine for days, months and then suddenly the system would panic, etc. when application would try to access the corrupted blocks for the first time. Thanks to ZFS what have actually happened? All corrupted blocks were identified, unfortunately both mirrored copies were affected so ZFS can't fix them but it did identified a single file which was affected by all these blocks. We can just remove the file which is only 2GB and restore it again. And all of these while the system was running and we haven't even stopped the restore or didn't have to start from the beginning. Most importantly there is no uncertainty about the state of the filesystem or data within it.

The other important conclusion is that DTrace is a sysadmin's best friend :)

The EC gives a green light

2010-01-21T12:34:00.002+01:00

The European Commission clears Oracle's proposed acquisition of Sun Microsystems:

"The European Commission has approved under the EU Merger Regulation the proposed acquisition of US hardware and software vendor Sun Microsystems Inc. by Oracle Corporation, a US enterprise software company. After an in-depth examination, launched in September 2009 (see IP/09/1271 ), the Commission concluded that the transaction would not significantly impede effective competition in the European Economic Area (EEA) or any substantial part of it."

MySQL TOP

2010-01-15T12:16:00.014+01:00

This blog entry has been updated:

added NCQRS, NCQRS/s columns
fixed issue with dtrace dropping variables if the script was running for extended time periods
cosmetic changes re output

I need to observe MySQL load from time to time and DTrace is one of the tools to use. Usually I'm using one-liners or I come up with a short script. This time I thought it would be nice to write a script so other people like DBAs could use without having to understand how it actually works. The script prints basic statistics for each client connecting to a database. It gives a nice overview for all clients using a database.


CLIENT IP                 CONN   CONN/s     QRS   QRS/s    NCQRS  NCQRS/s   TIME   VTIME
10.10.10.35                 10       0       61       0       32       0       0       0
10.10.10.30                 17       0       73       0       73       0       0       0
10.10.10.100                52       0       90       0       90       0       0       0
xx-www-11.portal            92       0      249       0       48       0       0       0
xx-cms-1.portal             95       0     1795       5     1669       4      48      48
xx-www-9.portal            198       0      634       1      278       0       0       0
xx-www-13.portal           239       0      986       2      366       1       0       0
xx-www-3.portal            266       0     1028       2      455       1       1       0
xx-www-12.portal           266       0     1070       2      561       1       3       2
xx-www-5.portal            300       0     1431       3      593       1       2       2
xx-www-10.portal           333       0     1221       3      678       1       3       2
xx-www-6.portal            334       0      989       2      446       1       1       0
xx-www-8.portal            358       0     1271       3      497       1       1       0
xx-www-4.portal            395       1     1544       4      744       2       0       0
xx-www-2.portal            445       1     1729       4      764       2       3       2
xx-www-1.portal            962       2     3555       9     1670       4      22      21
xx-www-7.portal           1016       2     3107       8     1643       4     117     115
                        ======   =====   ======   =====   ======   =====   =====   =====
                          5378      14    20833      58    10607      29     207     199
Running for 359 seconds.

         CONN       total number of connections
         CONN/s     average number of connections per second
         QRS        total number of queries
         QRS/s      average number of queries per second
         NCQRS      total number of executed queries
         NCQRS/s    average number of executed queries per second
         TIME       total clock time in seconds for all queries
         VTIME      total CPU time in seconds for all queries

The NCQRS column represents the number of queries which were not served from the MySQL Query Cache while QRS represents all queries issued to MySQL (cached, non-cached or even non-valid queries). If values of VTIME are very close to values of TIME it means that queries are mostly CPU bound. On the other hand the bigger the difference between them the more time is spent on I/O. Another interesting thing to watch is how evenly load is coming from different clients especially in environments where clients are identical www servers behind load balancer and should be generating about the same traffic to a database.

All values are measured since the script was started. There might be some discrepancies with totals in the summary line - this is due to rounding errors. The script should work for MySQL versions 5.0.x, 5.1.x and perhaps for other versions as well. The script doesn't take into account connections made over a socket file - only tcp/ip connections.

The script requires PID of a mysql database as its first argument and a frequency at which output should be refreshed as a second argument, for example to monitor mysql instance with PID 12345 and refresh output every 10s:


       ./mysql_top.d 12345 10s


# cat mysql_top.d
#!/usr/sbin/dtrace -qCs

/*
Robert Milkowski
*/

#define CLIENTS self->client_ip == "10.10.10.11" ? "xx-www-1.portal" : \
    self->client_ip == "10.10.10.12" ? "xx-www-2.portal" : \
    self->client_ip == "10.10.10.13" ? "xx-www-3.portal" : \
    self->client_ip == "10.10.10.14" ? "xx-www-4.portal" : \
    self->client_ip == "10.10.10.15" ? "xx-www-5.portal" : \
    self->client_ip == "10.10.10.16" ? "xx-www-6.portal" : \
    self->client_ip == "10.10.10.17" ? "xx-www-7.portal" : \
    self->client_ip == "10.10.10.18" ? "xx-www-8.portal" : \
    self->client_ip == "10.10.10.19" ? "xx-www-9.portal" : \
    self->client_ip == "10.10.10.20" ? "xx-www-10.portal" : \
    self->client_ip == "10.10.10.21" ? "xx-www-11.portal" : \
    self->client_ip == "10.10.10.22" ? "xx-www-12.portal" : \
    self->client_ip == "10.10.10.23" ? "xx-www-13.portal" : \
    self->client_ip == "10.10.10.29" ? "xx-cms-1.portal" : \
          self->client_ip

#define LEGEND  "\n \
        CONN       total number of connections \n \
        CONN/s     average number of connections per second \n \
        QRS        total number of queries \n \
        QRS/s      average number of queries per second \n \
        NCQRS      total number of executed queries \n \
        NCQRS/s    average number of executed queries per second \n \
        TIME       total clock time in seconds for all queries \n \
        VTIME      total CPU time in seconds for all queries\n"

BEGIN
{
  start = timestamp;
  total_queries = 0;
  total_nc_queries = 0;
  total_conn = 0;
  total_time = 0;
  total_vtime = 0;
}

syscall::getpeername:entry
/ pid == $1 /
{
  self->in = 1;

  self->arg0 = arg0; /* int s */
  self->arg1 = arg1; /* struct sockaddr * */
  self->arg2 = arg2; /* size_t len */
}

syscall::getpeername:return
/ self->in /
{
  this->len = *(socklen_t *) copyin((uintptr_t)self->arg2, sizeof(socklen_t));
  this->socks = (struct sockaddr *) copyin((uintptr_t)self->arg1, this->len);
  this->hport = (uint_t)(this->socks->sa_data[0]);
  this->lport = (uint_t)(this->socks->sa_data[1]);
  this->hport <<= 8;   this->port = this->hport + this->lport;

  this->a1 = lltostr((uint_t)this->socks->sa_data[2]);
  this->a2 = lltostr((uint_t)this->socks->sa_data[3]);
  this->a3 = lltostr((uint_t)this->socks->sa_data[4]);
  this->a4 = lltostr((uint_t)this->socks->sa_data[5]);
  this->s1 = strjoin(this->a1, ".");
  this->s2 = strjoin(this->s1, this->a2);
  this->s1 = strjoin(this->s2, ".");
  this->s2 = strjoin(this->s1, this->a3);
  this->s1 = strjoin(this->s2, ".");
  self->client_ip = strjoin(this->s1, this->a4);

  @conn[CLIENTS] = count();
  @conn_ps[CLIENTS] = count();

  total_conn++;

  self->arg0 = 0;
  self->arg1 = 0;
  self->arg2 = 0;
}

pid$1::*close_connection*:entry
/ self->in /
{
  self->in = 0;
  self->client_ip = 0;
}

pid$1::*mysql_parse*:entry
/ self->in /
{
  self->t = timestamp;
  self->vt = vtimestamp;

  @query[CLIENTS] = count();
  @query_ps[CLIENTS] = count();

  total_queries++;
}

pid$1::*mysql_parse*:return
/ self->in /
{
  @time[CLIENTS] = sum(timestamp-self->t);
  @vtime[CLIENTS] = sum(vtimestamp-self->vt);

  total_time += (timestamp - self->t);
  total_vtime += (vtimestamp - self->vt);

  self->t = 0;
  self->vt = 0;
}

pid$1::*mysql_execute_command*:entry
/ self-> in /
{
  @nc_query[CLIENTS] = count();
  @nc_query_ps[CLIENTS] = count();

  total_nc_queries++;
}

tick-$2
{
  /* clear the screen and move cursor to top left corner */
  printf("\033[H\033[J");

  this->seconds = (timestamp - start) / 1000000000;

  normalize(@conn_ps, this->seconds);
  normalize(@query_ps, this->seconds);
  normalize(@nc_query_ps, this->seconds);
  normalize(@time, 1000000000);
  normalize(@vtime, 1000000000);

  printf("%-16s          %s   %s     %s   %s    %s  %s   %s   %s\n", \
    "CLIENT IP", "CONN", "CONN/s", "QRS", "QRS/s", "NCQRS", "NCQRS/s", "TIME", "VTIME");
  printa("%-16s        %@6d   %@5d   %@6d   %@5d   %@6d   %@5d   %@5d   %@5d\n", \
    @conn, @conn_ps, @query, @query_ps, @nc_query, @nc_query_ps, @time, @vtime);
  printf("%-16s        %s   %s   %s   %s   %s   %s   %s   %s\n", \
    "", "======", "=====", "======", "=====", "======", "=====", "=====", "=====");
  printf("%-16s        %6d   %5d   %6d   %5d   %6d   %5d   %5d   %5d\n", "", \
    total_conn, total_conn/this->seconds, total_queries, total_queries/this->seconds, \
    total_nc_queries, total_nc_queries/this->seconds, \
    total_time/1000000000, total_vtime/1000000000);

/*
  denormalize(@conn_ps);
  denormalize(@query_ps);
  denormalize(@nc_query_ps);
  denormalize(@total_time);
  denormalize(@total_vtime);
*/

  printf("Running for %d seconds.\n", this->seconds);

  printf(LEGEND);
}

MySQL Query Time Histogram

2010-01-14T16:27:00.003+01:00

When doing MySQL performance tuning on a live server it is often hard to tell what impact there will be on all queries as sometimes by increasing one of the MySQL caches you can make some queries to execute faster but others might get actually slower. However, depending on your environment, it might not necessarily be a bad thing. For example in web serving if most queries would execute within 0.1s but some odd queries need 5s to complete it is generally very bad as user would need to wait at least 5s to get a web page. Now if by some tuning you manage to get these long queries down to below 1s with the cost of getting some sub 0.1s queries taking more time but still less than 1s it would generally be a very good thing to do. Of course in other environments the time requirements might be different but the principle is the same.

Now it is actually very easy to get such a distribution of number of queries being executed by a given MySQL instance within a given time slot if you use DTrace.


 1s resolution                                    
          value  ------------- Distribution ------------- count   
            < 0 |                                         0       
              0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 4700573 
              1 |                                         6366    
              2 |                                         35      
              3 |                                         23      
              4 |                                         39      
              5 |                                         8       
              6 |                                         6       
              7 |                                         5       
              8 |                                         7       
              9 |                                         4       
          >= 10 |                                         9       

Running for 73344 seconds.

The above histogram shows that 4,7mln queries were executed below 1s each, then for another 6366 queries it took between 1-2s for each query to execute, and so on. Now lets do some tuning and see the results again (of course you want to measure for a similar amount of time during similar period of activity - these are just examples):


 1s resolution                                    
          value  ------------- Distribution ------------- count   
            < 0 |                                         0       
              0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 4686051     
              1 |                                         2972       
              2 |                                         0       

Running for 73024 seconds.

That is much better. It is of course very easy to change the resolution of the histogram - but I will leave it for you.

The script requires 2 arguments - PID of a database and how often it should refresh its output, for example in order to get an output every 10s for a database running with PID 12345 run the script as:


./mysql_query_time_distribution.d 12345 10s

The script doesn't distinguish between cached and non-cached queries, it doesn't detect bad (wrong syntax) queries either - however it is relatively easy to extend it to do so (maybe another blog entry one day). It should work fine with all MySQL versions 5.0.x and 5.1.x, possibly with other versions as well.


# cat mysql_query_time_distribution.d
#!/usr/sbin/dtrace -qs


BEGIN
{
 start=timestamp;
}

pid$1::*mysql_parse*:entry
{
 self->t=timestamp;
}

pid$1::*mysql_parse*:return
/ self->t /
{
 @["1s resolution"]=lquantize((timestamp-self->t)/1000000000,0,10);

 self->t=0;
}

tick-$2
{
 printa(@);
 printf("Running for %d seconds.\n", (timestamp-start)/1000000000);
}

Identifying a Full Table Scan in MySQL with Dtrace

2010-01-07T03:05:00.005+01:00

Yesterday I was looking at some performance issues with a mysql database. The database is version 5.1.x so no built-in DTrace SDT probes but still much can be done even without them. What I quickly noticed is that mysql was issuing several hundred thousands syscalls per second and most of them were pread()s and read()s. The databases are using MyISAM engine so mysql does not have a data buffer cache and leaves all the caching to a filesystem. I was interested in how many reads were performed per given query so I wrote a small dtrace script. The script takes as arguments a time after which it will exit and a threshold which represents minimum number of [p]read()s per query to query be printed.

So lets see an example output where we are interested only in queries which causes at least 10000 reads to be issued:

# ./m2.d 60s 10000
### read() count: 64076 ###
SELECT * FROM clip WHERE parent_id=20967 AND type=4 ORDER BY quality ASC

### read() count: 64076 ###
SELECT * FROM clip WHERE parent_id=14319 AND type=4 ORDER BY quality ASC

### read() count: 64076 ###
SELECT * FROM clip WHERE parent_id=20968 AND type=4 ORDER BY quality ASC

There are about 60k entries form parent_id column which suggests that mysql is doing a full table scan when executing above queries. A quick check within mysql revealed that there was no index for parent_id column so mysql was doing full table scans. After the index was created:


# ./m2.d 60s 1
[filtered out all unrelated queries]
### read() count: 6 ###
SELECT * FROM clip WHERE parent_id=22220 AND type=4 ORDER BY quality ASC

### read() count: 8 ###
SELECT * FROM clip WHERE parent_id=8264 AND type=4 ORDER BY quality ASC

### read() count: 4 ###
SELECT * FROM clip WHERE parent_id=21686 AND type=4 ORDER BY quality ASC

### read() count: 4 ###
SELECT * FROM clip WHERE parent_id=21687 AND type=4 ORDER BY quality ASC

So now each query is issuing 5 orders of magnitude less reads()!

Granted, all these reads were satisfied from ZFS ARC cache but still it saves hundreds of thousands unnecessary context switches and memory copying s making the queries *much* more quicker to execute and saving valuable CPU cycles. The real issue I was working on was a little bit more complicated but you get the idea.

The point I'm trying to make here is that although MySQL lacks good tools to analyze its workload you have a very powerful tool called dtrace which allows you to relatively quickly identify what queries are causing an issue and why. And all of that on a running live service without having to reconfigure or restart mysql. I know there is the MySQL Query Analyzer (or whatever it is called) but it requires a mysql proxy to be deployed... In this case it was much quicker and easier to use dtrace.

Below you find the script. Please notice that I had hard-coded the PID of the database and the script could be clean up, etc. - it is the working copy I used. The script can be easily modified to provide lots of additional useful information or it can be limited to only a specific myisam file, etc.


# cat m2.d
#!/usr/sbin/dtrace -qs

#pragma D option strsize=8192


pid13550::*mysql_parse*:entry
{
self->a=1;
self->query=copyinstr(arg1);
self->count=0;

}

pid13550::*mysql_parse*:return
/ self->a && self->count > $2 /
{
printf("### read() count: %d ###\n%s\n\n", self->count, self->query);

self->a=0;
self->query=0;

}

pid13550::*mysql_parse*:return
/ self->a /
{
self->a=0;
self->query=0;
}

syscall::*read*:entry
/ self->a /
{
self->count++;
}

tick-$1
{
exit(0);
}

zpool split

2010-01-05T11:33:00.003+01:00

PSARC/2009/511 zpool split:

OVERVIEW:

Some practices in data centers are built around the use of a volume
manager's ability to clone data. An administrator will attach a set of
disks to mirror an existing configuration, wait for the resilver to
complete, and then physically detach and remove those disks to a new
location.

Currently in zfs, the only way to achieve this is by using zpool offline
to disable a set of disks, zpool detach to permanently remove them after
they've been offlined, move the disks over to a new host, zpool
force-import of the moved disks, and then zpool detach the disks that were
left behind.

This is cumbersome and prone to error, and even then the new pool
cannot be imported on the same host as the original.

PROPOSED SOLUTION:

Introduce a "zpool split" command. This will allow an administrator to
extract one disk from each mirrored top-level vdev and use them to create
a new pool with an exact copy of the data. The new pool can then be
imported on any machine that supports that pool's version.

The new feature should be available in build 131.
See implementation details.

Triple-Parity RAID and Beyond

2009-12-28T15:25:00.002+01:00

Adam Leventhal wrote an interesting article about the future of RAID.

My Presentation at LOSUG

2009-12-17T23:56:00.004+01:00

Yesterday I did a presentation at London Open Solaris User Group on the backup platform I implemented. It utilizes open source technologies like Open Solaris, ZFS, RSYNC and a commodity hardware to effectively offer us a better backup solution than NetBackup and for a fraction of a cost. You can download the slides here. Before you do so it might be worth reading my two previous blog entries: 1 2 which should provide some additional background.

Read-Only Boot from ZFS Snapshot

2009-12-12T22:20:00.001+01:00

PSARC 2009/670

Allow for booting from a ZFS snapshot.  The boot image
will be read-only.  Early in boot a clone of the root
is created and used to provide writable storage for the
system image during its lifetime.  Upon reboot, the
system image will reset to the same previous state.

Sun Cluster 3.2 11/09

2009-12-03T07:39:00.001+01:00

What's New:

VirtualBox 3.1 Released

2009-12-01T14:33:00.003+01:00

This version is a major update. The following major new features were added:

Teleportation (aka live migration); migrate a live VM session from one host to another (see the manual for more information)
VM states can now be restored from arbitrary snapshots instead of only the last one, and new snapshots can be taken from other snapshots as well ("branched snapshots"; see the manual for more information)
2D video acceleration for Windows guests; use the host video hardware for overlay stretching and color conversion (see the manual for more information)
More flexible storage attachments: CD/DVD drives can be attached to an arbitrary IDE controller, and there can be more than one such drive (the manual for more information)
The network attachment type can be changed while a VM is running
Complete rewrite of experimental USB support for OpenSolaris hosts making use of the latest USB enhancements in Solaris Nevada 124 and higher
Significant performance improvements for PAE and AMD64 guests (VT-x and AMD-V only; normal (non-nested) paging)
Experimental support for EFI (Extensible Firmware Interface; see the manual for more information)
Support for paravirtualized network adapters (virtio-net; see the manual for more information)

How to find loose screws with DTrace

2009-11-30T06:53:00.000+01:00

Long ssh logins

2009-11-24T15:51:00.004+01:00

On a couple of our servers running Solaris we noticed that it usually takes more than 10s to login. Once in everything is a snap. I quickly investigated it and this turned out to be interesting. I used truss(1M) to investigate what's going on from the moment I connect to the moment I have a working shell.

# truss -f -o /tmp/a -v all -adDE -p 408

Now I logged in to the system and analyzed /tmp/a file. First I confirmed that it took over 10s to login. From the moment the connection was accepted to the moment I got interactive session it took about 11s as shown below:

[...]
408:     2.6594  0.0007  0.0000 fcntl(4, F_SETFL, (no flags))                   = 0
[...]
12186:  14.0814  0.0001  0.0000 write(4, " | : { b7F S LB7A2  BA13".., 64)      = 64
12196:  read(0, 0x080473DF, 1)          (sleeping...)
[...]

So I checked when it started to go wrong.

[...]
408:     2.6594  0.0007  0.0000 fcntl(4, F_SETFL, (no flags))                   = 0
[...]
12196:   3.7245  0.0003  0.0003 forkx(0)                                        = 12200

The connection started just before the fcntl showed above and everything is executing quick up-to forkx() at 3.7245s. So far it took a little more than 1s. What happens next seems to be a loop of hundreds of entries like:

[...]
12200:   4.5521  0.0000  0.0000 ioctl(3, ZFS_IOC_USERSPACE_ONE, 0x08046790)     Err#48 ENOTSUP
12200:   4.5522  0.0001  0.0000 ioctl(7, MNTIOC_GETMNTENT, 0x08047C1C)          = 0
12200:   4.5917  0.0395  0.0002 ioctl(3, ZFS_IOC_OBJSET_STATS, 0x08046390)      = 0
12200:   4.5918  0.0001  0.0000 getuid()                                        = 35148 [35148]
12200:   4.5919  0.0001  0.0000 getuid()                                        = 35148 [35148]
12200:   4.5919  0.0000  0.0000 door_info(6, 0x08046460)                        = 0
12200:          target=189 proc=0x806FCD0 data=0xDEADBEED
12200:          attributes=DOOR_UNREF|DOOR_NO_CANCEL
12200:          uniquifier=289
12200:   4.5922  0.0003  0.0000 door_call(6, 0x080464D0)                        = 0
12200:          data_ptr=FE430000 data_size=232
12200:          desc_ptr=0x0 desc_num=0
12200:          rbuf=0xFE430000 rsize=16384
12200:   4.5923  0.0001  0.0000 ioctl(3, ZFS_IOC_USERSPACE_ONE, 0x08046790)     Err#48 ENOTSUP
12200:   4.5923  0.0000  0.0000 ioctl(7, MNTIOC_GETMNTENT, 0x08047C1C)          = 0
12200:   4.6095  0.0172  0.0001 ioctl(3, ZFS_IOC_OBJSET_STATS, 0x08046390)      = 0
12200:   4.6096  0.0001  0.0000 getuid()                                        = 35148 [35148]
12200:   4.6096  0.0000  0.0000 getuid()                                        = 35148 [35148]
12200:   4.6097  0.0001  0.0000 door_info(6, 0x08046460)                        = 0
12200:          target=189 proc=0x806FCD0 data=0xDEADBEED
12200:          attributes=DOOR_UNREF|DOOR_NO_CANCEL
12200:          uniquifier=289
12200:   4.6098  0.0001  0.0000 door_call(6, 0x080464D0)                        = 0
12200:          data_ptr=FE430000 data_size=232
12200:          desc_ptr=0x0 desc_num=0
12200:          rbuf=0xFE430000 rsize=16384
12200:   4.6098  0.0000  0.0000 ioctl(3, ZFS_IOC_USERSPACE_ONE, 0x08046790)     Err#48 ENOTSUP
12200:   4.6099  0.0001  0.0000 ioctl(7, MNTIOC_GETMNTENT, 0x08047C1C)          = 0
12200:   4.6201  0.0102  0.0001 ioctl(3, ZFS_IOC_OBJSET_STATS, 0x08046390)      = 0
12200:   4.6202  0.0001  0.0000 getuid()                                        = 35148 [35148]
12200:   4.6203  0.0001  0.0000 getuid()                                        = 35148 [35148]
[...]

The process with PID 12200 was:

12200:   3.8229  0.0947  0.0013 execve("/usr/sbin/quota", 0x0811F9E8, 0x0811D008)  argc = 1
12200:      *** SUID: ruid/euid/suid = 35148 / 0 / 0  ***
12200:   argv: /usr/sbin/quota

By visually looking at couple of pages of these ioctls it looked like most of the total time would be spent in doing ioctl(3, ZFS_IOC_OBJSET_STATS, 0x08046390). Lets check it:

# grep "^12200:" /tmp/a |grep ioctl|grep ZFS_IOC_OBJSET_STATS|awk 'BEGIN{i=0}{i=i+$3}END{print i}'
9.7412

So out of 11s above ioctls along took 9. To have a clear picture lets check how much time the quota command took:

# grep "^12200:" /tmp/a |head -1
12200:   3.7245  3.7245  0.0000 forkx()         (returning as child ...)        = 12196
# grep "^12200:" /tmp/a |tail -1
12200:  13.9854  0.0003  0.0000 _exit(0)

So it took about 10s which means almost 100% of its time was spent doing above ioctls.
There are almost 300 zfs filesystems on this particular server so it all adds up. Sometimes quota completes very quickly sometimes it takes many seconds - I guess depending if requested data from all these zfs filesystems is cached or not. You need to run quota as a non-root user otherwise most checks are skipped and it is always quick.
Since we are not using quota on these systems anyway I commented out quota check in /etc/profile and now a full login takes about 1s on average which is 10-12x improvement.

Xen 3.4

2009-11-20T16:24:00.000+01:00

Xen 3.4 integrated yesterday. It should appear in snv_129.

VirtualBox 3.1.0 Beta 1 released

2009-11-11T17:26:00.003+01:00

VirtualBox 3.1.0 Beta 1 released.

I'm especially interested in "support for OpenSolaris Boomer architecture" which hopefully means a working microphone in a guest on my laptop - that would mean a working Skype on VB/Windows :)

Version 3.1 will be a major update. The following major new features were added:
Teleportation (aka live migration); migrate a live VM session from one machine to another
VM states can now be restored from arbitrary snapshots instead of only the last one, and new snapshots can be taken from other snapshots as well (aka branched snapshots)
2D video acceleration for Windows guests; use the host video hardware for overlay stretching and colour conversion
The network attachment type can be changed while a VM is running
Experimental USB support for OpenSolaris hosts making use of the latest USB enhancements in Solaris Nevada 124 and higher.
Significant performance improvements for PAE and AMD64 guests (VT-x and AMD-V only; normal (non-nested) paging)
Experimental support for EFI (Extended Firmware Interface)
VirtIO network device support

In addition, the following items were fixed and/or added:
VMM: reduced IO-APIC overhead for 32 bits Windows NT/2000/XP/2003 guests; requires 64 bits support (VT-x only; bug #4392)
VMM: fixed double timer interrupt delivery on old Linux kernels using IO-APIC (caused guest time to run at double speed; bug #3135)
VMM: reinit VT-x and AMD-V after host suspend or hibernate; some BIOSes forget this (Windows hosts only; bug #5421)
GUI: prevent starting a VM with a single mouse click (bug #2676)
3D support: major performance improvement in VBO processing
3D support: added GL_EXT_framebuffer_object, GL_EXT_compiled_vertex_array support
3D support: fix crashes in FarCry, SecondLife, Call of Duty, Unreal Tournament, Eve Online (bugs #2801, #2791)
3D support: fix graphics corruption in World of Warcraft (#2816)
iSCSI: support iSCSI targets with more than 2TiB capacity
VRDP: fixed occasional VRDP server crash (bug #5424)
Network: fixed the E1000 emulation for QNX (and probably other) guests (bug #3206)
Network: even if the virtual network cable was disconnected, some guests were able to send / receive packets (E1000; bug #5366)
Network: even if the virtual network cable was disconnected, the PCNet card received some spurious packets which might confuse the guest (bug #4496)
VMDK: fixed handling of split image variants
VHD: fixed incompatibility with Hyper-V
OVF: create manifest files on export and verify the content of an optional manifest file on import
X11 based hosts: allow the user to specify their own scan code layout (bug #2302)
Mac OS X hosts: don't auto show the menu and dock in fullscreen (#bug 4866)
Solaris hosts: combined the kernel interface package into the VirtualBox main package
Solaris hosts: support for OpenSolaris Boomer architecture (with OSS audio backend).
Shared folders: fixed changing case of file names (bug #2520)
Shared folders: VBOXSVR is visible in Network folder (bug #4842)
Windows and Linux Additions: added balloon tip notifier if VirtualBox host version was updated and Additions are out of date
Solaris Additions: fixed as_pagelock() failed errors affecting guest properties (bug #5337)
Windows Additions: added automatic logon support for Windows Vista and Windows 7
Windows Additions: fix crash in seamless mode (contributed by Huihong Luo)
Linux Additions: added support for uninstalling the Linux Guest Additions (bug #4039)
SDK: added object-oriented web service bindings for PHP5

ZFS send dedup

2009-11-10T11:46:00.002+01:00

PSARC/2009/557 ZFS send dedup was integrated yesterday. It allows to dedup zfs send|recv stream regardless if dedup is enabled or not on a sending and/or receiving side. Looks like it both pool and zfs send stream deduplication will be available in snv_128. Of course you can pull the sources and build them yourself if you want to start playing with dedupe now otherwise you will have to wait to the beginning of December when snv_128 should hit /dev Open Solaris repository.

EU officially objects to Sunacle deal

2009-11-10T11:00:00.001+01:00

This is ridiculous and sad... I wonder what would happen now.
The official statement from Oracle.

No need for fsck in ZFS

2009-11-05T10:57:00.003+01:00

Recently there was an article at OSNEWS " Should ZFS Have a fsck Tool?". Well, no it shouldn't. I wanted to write an explanation why it is the case but Joerg was first and there is no point me repeating him. So if you wonder why ZFS doesn't need a fsck tool read Joerg blog entry about it.

ZFS Deduplication Integrated!

2009-11-02T11:41:00.004+01:00

It took more than expected but it has been finally integrated! Read Jeff Bonwick's post on ZFS dedup.

PSARC 2009/571 ZFS Deduplication Properties
6677093 zfs should have dedup capability

You can find code changes here.

Sun/Oracle deal to unravel?

2009-10-30T02:23:00.001+01:00

FAQ from Oracle on Sun Acquisition

2009-10-28T17:02:00.000+01:00

Apple Abandons ZFS

2009-10-25T11:20:00.002+01:00

According to Jeff Bonwick (one of the main developers of ZFS) the real reason is:

> Apple can currently just take the ZFS CDDL code and incorporate it
> (like they did with DTrace), but it may be that they wanted a "private
> license" from Sun (with appropriate technical support and
> indemnification), and the two entities couldn't come to mutually
> agreeable terms.

I cannot disclose details, but that is the essence of it.

Jeff

Solaris 10 on Open Solaris

2009-10-23T13:38:00.006+02:00

Yesterday a new zone brand was integrated into Open Solaris: Solaris 10 Zones. It allows running Solaris 10 in a zone on Open Solaris. The technology will be very useful in some instances for enterprise customers once Solaris 11 is out or when deploying Open Solaris. This is very useful if you want to make use of latest hardware platforms and/or technologies delivered in Open Solaris (and Solaris 11 in the future) and you need to migrate your older environments running Solaris 8, 9 or 10 without introducing changes to them (to mitigate a risk or due to lack of resources to do a more detailed validation of a new environment). This is especially true with Open Solaris and Solaris 11 as they are so much different than older Solaris releases. Branded zones are a very neat solution to provide backward compatibility without putting too much hurdle on innovation or having most of the users who do not need it to pay the price of it.

See Jerry's blog entry about Solaris 10 Zones.

Oracle, MySQL, EU and money

2009-10-23T12:09:00.002+02:00

The whole MySQL issue with Oracle and EU is just silly. I'm not going to repeat arguments here as there are many comments you can find and I believe that all that could be said about it has already been said. If you are interested in what is going on and why then read these two articles which summarize the whole issue very well. Here is one from Groklaw and this one is also very interesting.

It is really sad to see to what extend some people are greedy...

I think that RMS and Monty lost whatever credibility they had left. They definitely lost it completely in my eyes. Not that Monty cares as he is after more money here... He wants to cash-in two times... Call me idealist or whatever but I honestly believe that life and business is not *only* about money.

zfs set dedup=on

2009-10-21T16:12:00.002+02:00

There are two PSARC cases regarding ZFS deduplication which are expected to be approved today.

PSARC 2009/571 ZFS Deduplication Properties
PSARC 2009/557 ZFS send dedup

Hopefully it means that ZFS deduplication will be finally integrated soon.

Sun vs. IBM - Marketing Saga Continues

2009-10-16T12:45:00.003+02:00

So Oracle hasn't even bought Sun yet and still there are better marketing campaigns of Sun than Sun ever did.

IBM's Black Magic

2009-10-14T17:38:00.005+02:00

Somehow I'm not surprised that comments section is disabled in this example of IBM's Black Magic. I won't comment on it myself as there is no point in repeating Joerg arguments - I agree with him in 100%. This is about recent world record in TPC-C benchmark from Oracle/Sun.

However I would like to suggest to Ms. Stahl that there is another big advantage in IBM's setup she's missed - not only IBM did more tpmC/core but they also did:

larger number or racks per tpmC
larger $/tpmC
larger power usage/tpmC

I think that somehow most customers would consider above three as more important than tpmC/core (assuming they would consider TPC-C relevant in the first place).

TPC-C Sun SPARC Enterprise T5440 with Oracle RAC World Record Database Result

2009-10-12T14:00:00.003+02:00

Win 10$ Million

2009-10-09T00:22:00.003+02:00

I already like the Oracle marketing :)

I only hope that while keeping it aggressive and provocative they won't end-up like IBM with their black magic style benchmarks and comparisons... If anything from Sun's culture will survive the acquisition it is rather unlikely to happen.

300k SPC-1 IOPS from IBM

2009-10-08T15:01:00.001+02:00

This is quite impressive result. On the other hand the price per SPC is rather expensive and one wonders why they used a server with 64 cores and used only 48 of them. Is there a technical reason? Anyone knows?

ZFS - What's Coming

2009-10-01T01:17:00.001+02:00

The slides.

Exadata v2

2009-09-18T18:05:00.000+02:00

Improved stat() performance on files on zfs

2009-09-17T15:12:00.003+02:00

Bug ID: 6775100 stat() performance on files on zfs should be improved was fixed in snv_119.
I wanted to do a quick comparison between snv_117 and snv_122 on my workstation to see what kind of improvement there is. I wrote a small C program which does a stat() N times in a loop. This is of course a micro-benchmark. Additionally it doesn't cover a case if doing stat() on not cached entries in DNLC has been improved too.

So I run the program several times on each build after a fresh restart. These were the numbers I was getting on average:


snv_117/ZFS# ptime ./stat_loop test 1000000

real        1.941163151
user        0.219955617
sys         1.707997800

snv_122/ZFS# ptime ./stat_loop test 1000000

real        1.089193770
user        0.199055005
sys         0.889680683

snv_122/UFS# ptime ./stat_loop test 1000000

real        0.905696133
user        0.187513753
sys         0.716955921

This is over 40% improvement in performance of stat() on ZFS - nice.
Still stat() on UFS is faster by about 17%.

The fix could also help some very busy NFS servers :)
AFAIK it has not been backported to Solaris 10 so if you think you need it either go for Open Solaris or open a case with Sun's support and ask for a fix for S10.

Oracle: Sun vs. IBM

2009-08-28T13:16:00.002+02:00

"Oracle and Sun together are hard to match. Just ask IBM. Its fastest server now runs an impressive 6 million TPC-C transactions, but on October 14 at Oracle OpenWorld, we'll reveal the benchmark numbers that prove that even IBM DB2 running on IBM's fastest hardware can't match the speed and performance of Oracle Database on Sun systems. Check back on October 14 as we demonstrate Oracle's commitment to Sun hardware and Sun SPARC."

FCoE Initiator Integrated into b122

2009-08-26T12:17:00.002+02:00

Linux Kernel Development

2009-08-20T09:07:00.001+02:00

Recent Changes to SunSSH

2009-08-13T13:04:00.000+02:00

SXCE To Be EOL'ed

2009-08-11T13:17:00.002+02:00

We all knew it will happen sooner or later. For some people it is probably too soon for others like me it doesn't really matter as they have been deploying Open Solaris distribution instead of SXCE for quite some time. While it might seem like a little bit pre-mature decision I believe that it will allow to better utilize available resources so the open solaris community can focus more on what's in front of us rather then putting their time into SXCE which has no future...

The official announcement:

Sun is announcing the intent to discontinue production of the Solaris Express Community Edition (SXCE) by the end of October time-frame. As we intend to continue on a bi-weekly build schedule, consolidations will move towards producing native Image Packaging System (IPS) packages alongside SVR4 packages and then phase out the latter completely. Technologies such as IPS, Automated Install, Snap Upgrade and the Distribution Constructor will be integrating into a consolidation after following through the established processes including architectural (ARC) review.

We recognize that this transition will require some effort for all members of the OpenSolaris development community, and are committed to working with all of you in making that transition a success. You can expect updated information from us and the communities which manage the consolidations as we further plan the transition schedules.

Questions can be directed to David Comay, Glynn Foster, William Franklin, Stephen Hahn, Dave Miner, Vincent Murphy, or Dan Roberts.

Read or Write Only Process

2009-08-10T12:19:00.001+02:00

PSARC/2009/378:

This project proposes two new "basic" privileges.

FILE_READ
Allows a process to read a file or directory whose
permission or ACL allow the process read permission.

FILE_WRITE
Allows a process to write a file or directory whose
permission or ACL allow the process write permission.

The purpose of these privileges is the ability to create a "read-only" (no FILE_WRITE privilege) and a "write-only" (no FILE_READ privilege) process.

The FILE_WRITE basic privilege is required for any modification to a file or directory: open(2), creat(2), link(2), symlink(2), rename(2), unlink(2), mkdir(2), rmdir(2), mknod(2) etc.

The FILE_READ basic privilege is required for opening a file with O_RDONLY or O_RDWR.

Note: a "basic" privilege is a privilege which is part of the default I, P and E privilege set.

ZFS: logbias

2009-08-10T12:00:00.003+02:00

PSARC/2009/423:

Summary

Provide zfs with the ability to control the use of resources used for synchronous (eg fsync, O_DSYNC) requests. In particular it enables substantially better performance for Oracle and potentially other applications.

Background

Oracle manages two major types of files, the Data Files and the Redo Log files. Writes to Redo Log files are in the path of all transactions and low latency is a requirement. It is critical for good performance to give high priority to these write requests.

Data Files are also the subject of writes from DB writers as a form of scrubbing dirty blocks to insure DB buffer availability. Write latency is much less an issue. Of more more importance is achieving an acceptable level of throughput. These writes are less critical to delivered performance.

Both types of writes are synchronous (using O_DSYNC), and thus treated equally. They compete for the same resources: separate intent log, memory, and normal pool IO. The Data File writes impede the potential performance of the critical Redo Log writers.

Proposal

Create a new "logbias" property for zfs datasets.

If logbias is set to 'latency' (the default) then there is no change from the current implementation. If the logbias property is set to 'throughput' then intent log blocks will be allocated from the main pool instead of any separate intent log devices (if present). Also data will be written immediately to spread the write load thus making for quicker subsequent transaction group commits to the pool.

To change the property, an admin will use the standard zfs set command:

# zfs set logbias=latency {dataset}
# zfs set logbias=throughput {dataset}

Web Stack 1.5

2009-07-30T12:25:00.003+02:00

Radiation-Hardened SPARC

2009-07-27T12:27:00.003+02:00

Atmel Introduces the AT697F Radiation-Hardened SPARC V8 Processor for Space Missions.

Joerg explains on why SPARC?:

"The reason is not Solaris. It´s a different one. Despite general opinion, SPARC isn´t a proprietary architecture. You get go to SPARC International, get the specification for a 100 bucks or so and you can develop and manufacture your own SPARC CPU. This was used by the European Space Agency to develop a radiation hardened version of a SPARC architecture. The development is called LEON, the proc named in the article uses the LEON2-FT design. The LEON2 and LEON3 design are available under the GNU General Public License respectively under the GNU Lesser General Public License. Being able to get such an architecture essentially for cheap money was the essential reason behind the decision for SPARC (besides of other technical reasons)"