SecondLife on Solaris

Sun recently held a conference in SecondLife. Unfortunately there's no Solaris client - only Windows, Mac and Linux. Well, snv+49 has BrandZ, right? I downloaded Linux client logged to Linux zone and just started - it works very good and very fast! So all Solaris users - if you want to join Second Life go BrandZ :)

You can find Conference location here.

ps. Definitely forward over tcp to your X server in global zone rather than ssh x11 forwarding.

BrandZ integrated into snv_49

Hi.

BrandZ project is finally integrated into snv_49. For all of you who don't know it means you can install Linux distribution in a Solaris Zone and run Linux applications. Those Linux applications run under Solaris kernel which means you can use Solaris Resource Manager, DTrace, ZFS, etc. For more details see BrandZ Overview Presentation. BrandZ are expected to be in Solaris 10 Update 4 next year. Right now you can get it with Solaris Express Community Edition and soon with Solaris Express.

Is it hard to install Linux in a zone? Well, below you can see what I did - create a Linux Zone with networking and audio device present for Linux. Basically it's just two commands!

# zonecfg -z linux
linux: No such zone configured

Use 'create' to begin configuring a new zone.

zonecfg:linux> create -t SUNWlx

zonecfg:linux> set zonepath=/home/zones/linux

zonecfg:linux> add net

zonecfg:linux:net> set address=192.168.1.10/24

zonecfg:linux:net> set physical=bge0

zonecfg:linux:net> end

zonecfg:linux> add attr

zonecfg:linux:attr> set name="audio"

zonecfg:linux:attr> set type=boolean

zonecfg:linux:attr> set value=true

zonecfg:linux:attr> end

zonecfg:linux> exit

# zoneadm -z linux install -d /mnt/iso/centos_fs_image.tar.bz2
A ZFS file system has been created for this zone.

Installing zone 'linux' at root directory '/home/zones/linux'

from archive '/mnt/iso/centos_fs_image.tar.bz2'


This process may take several minutes.


Setting up the initial lx brand environment.

System configuration modifications complete!

Setting up the initial lx brand environment.

System configuration modifications complete!

Installation of zone 'linux' completed successfully.

Details saved to log file:

  "/home/zones/linux/root/var/log/linux.install.10064.log"


#

How to fork/exec in an efficient way

I've just found good article on developers.sun.com about "Minimizing Memory Usage for creating Application Subprocesses" - it explains why posix_spawn() interface introduced in Solaris 10 is better than fork/exec approach many developers still use. There's also good section on Memory Overcommiting.

SMF management for normal users

Let's say you have your in-house developed applications under SMF. Now you want to give some users (non-root) ability to restart those applications but only those applications. You do not want to give any other privileges. With SMF it's really easy. You can do it per application instance or for entire group, etc.

1. Add new authorization

# grep ^wp /etc/security/auth_attr wp.applications:::Manage WP applications::

2. Add new property to each SMF service you want to give access to restart/enable/disable

# svccfg -s wpfileback setprop general/action_authorization = astring: wp.applications

With only that property user won't be able to change service status permanently - he/she will be able to
restart or temporarily disable/enable given service (wpfileback in above example). If you want to give
ability to permanently change service status you also need to add:

# svccfg -s wpfileback setprop general/value_authorization = astring: wp.applications

3. Add new authorization to user

# usermod -A wp.applications operator

You can also manually add authorization by editing /etc/user_attr. After above command the fill is:
# grep operator /etc/user_attr operator::::type=normal;auths=wp.applications


Now if you login as user operator you will be able to disable/enable/restart application wpfileback.

Additionally it's useful to give for example developers not only ability to restart their application but also to use dtrace. In order to achieve it add two privileges to user.

# grep operator /etc/user_attr
operator::::type=normal;auths=wp.applications;defaultpriv=basic,dtrace_proc,dtrace_user

Now user operator not only can restart/stop/start its application but also can use dtrace to find problems.

There're also many authorizations and profiles which come with Solaris by default. For example if you add for given user profile 'Service Operator' then you give ability to restart/enable/disable all SMF applications.

All of these possibilities without giving her/him root account.

For more information see smf_security(5) rbac(5) privileges(5)

Thumpers are coming...

or not?

Ehhhh... great technology which you can order and then wait, and wait, and wait, and... nobody can actually tell you for sure how much longer you have to wait. So we're waiting...

ZFS in High Availability Environments

I see that many people are asking about ZFS + Sun Cluster solution. Soon Sun Cluster 3.2 should be released which does support ZFS (among many other new features). Now Solaris 10 is free, Sun Cluster is free. Additionally to install Sun Cluster it's just some clicks in GUI installer and voila! Then some other commands and we have ZFS pool under Sun Cluster management.
Below example (using new SC32 commands, old one are also available for backward compatibility) how to configure 2-node HA-NFS cluster with ZFS - as you can see it's really quick&easy.



Nodes: nfs-1 nfs-2
ZFS pool: files

# clresourcegroup create -n nfs-1,nfs-2 -p Pathprefix=/files/conf/ nfs-files
# clreslogicalhostname create -g nfs-files -h nfs-1 nfs-files-net
# clresourcetype register SUNW.HAStoragePlus
# clresource create -g nfs-files -t SUNW.HAStoragePlus -x Zpools=files nfs-files-hastp
# clresourcegroup online -e -m -M nfs-files
# mkdir /files/conf/SUNW.nfs
# vi /files/conf/SUNW.nfs/dfstab.nfs-files-shares
[put nfs shares here related to pool files]
# clresourcetype register SUNW.nfs
# clresource create -g nfs-files -t SUNW.nfs -p Resource_dependencies=nfs-files-hastp nfs-files-shares


ps. right now it's available as Sun Cluster 3.2 beta - I have already two SC32 beta clusters running with ZFS and must say it just works - there were so minor problems at the beginning but developers from Sun Cluster team helped so fast that I'm still impressed - thank you guys! Right now it works perfectly.

Niagara II internals

Interesting article describing upcoming Niagara II CPU.

Also read here and here.

How much memory does ZFS consume?

When using ZFS standard tools give inaccurate values for free memory as ZFS doesn't use normal page cache and rather allocates directly kernel memory. When low-memory condition occurs ZFS should free its buffer memory. So how to get how much additional memory is possibly free?

bash-3.00# mdb -k
Loading modules: [ unix krtld genunix specfs dtrace cpu.AuthenticAMD.15 ufs md ip sctp usba fcp fctl lofs zfs random nfs crypto fcip cpc logindmux ptm ipc ]
> ::memstat
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                     859062              3355   41%
Anon                       675625              2639   32%
Exec and libs                7994                31    0%
Page cache                  39319               153    2%
Free (cachelist)           110881               433    5%
Free (freelist)            385592              1506   19%

Total                     2078473              8119
Physical                  2049122              8004
>

bash-3.00# echo "::kmastat"|mdb -k|grep zio_buf|awk 'BEGIN {c=0} {c=c+$5} END {print c}'
2923298816

So kernel consumes about 3.2TB of memory and about 2.7GB is allocated to ZFS buffers and basically should be treated as free memory. Approximately free memory on this host is: Free (cachelist) + Free (freelist) + 2923298816.

I guess small script which do all the calculations automatically would be useful.

CrossBow

CrossBow official page.

Read Sunay blog entry - CrossBow: Solaris Network Virtualization & Resource Control

Sun gains market share

Sun recoups server martek share:

"The Santa Clara, Calif.-based company's server revenue rose 15.5 percent to $1.59 billion in the quarter, according to statistics from research firm IDC. The increase outpaced the overall growth of 0.6 percent to $12.29 billion worldwide, with faster gains in x86 servers, blade servers and lower-end models costing less than $25,000.

Sun's three main rivals fared worse. In contrast, IBM's revenue dropped 2.2 percent to $3.42 billion; Hewlett-Packard's dropped 1.7 percent to $3.4 billion; and Dell's dropped 1.3 percent to $1.27 billion."

New servers from Sun

New HW from Sun:

• US IV+ 1.8GHz
• available in v490 an up
  
• looks like it beats latest IBM's POWER5+ CPUs
• X2100 M2 server
• comparing to standard x2100 server it has latest 1200's Opterons, DDR2-667, 4x GbE
  
• X2200 M2 server
• 2x 2000s Opterons (dual-core), 64GB memory supported, 4x GbE, LOM, 2x HDD
  
• Ultra 20 M2 workstation
• comparing to U20 it has latest Opterons, 2x GbE, DDR2-667, better video

Sun's new servers page.
Official Sun announcement.
Related story.

HW RAID vs. ZFS software RAID - part II

This time I tested RAID-5 performance. I used the same hardware as in last RAID-10 benchmark.
I created RAID-5 volume consisting 6 disks on a 3510 head unit with 2 controllers, using random optimization. I also created software RAID-5 (aka RAID-Z) group using ZFS on 6 identical disks in a 3510 JBOD. Both HW and SW RAIDs were connected to the same host (v440). Using filebench's varmail test below are the results.

These tests show that software RAID-5 in ZFS can not only be as fast as hardware RAID-5 it can even be faster. The same is with RAID-10 - ZFS software RAID-10 was faster than hardware RAID-10.

Please note that I tested HW RAID on a 3510 FC array not on some junky PCI RAID card.


1. ZFS on HW RAID5 with 6 disks, atime=off
IO Summary: 444386 ops 7341.7 ops/s, (1129/1130 r/w) 36.1mb/s, 297us cpu/op, 6.6ms latency
IO Summary: 438649 ops 7247.0 ops/s, (1115/1115 r/w) 35.5mb/s, 293us cpu/op, 6.7ms latency

2. ZFS with software RAID-Z with 6 disks, atime=off
IO Summary: 457505 ops 7567.3 ops/s, (1164/1164 r/w) 37.2mb/s, 340us cpu/op, 6.4ms latency
IO Summary: 457767 ops 7567.8 ops/s, (1164/1165 r/w) 36.9mb/s, 340us cpu/op, 6.4ms latency

3. there's some problem in snv_44 with UFS so UFS test is on S10U2 in test #4
4. UFS on HW RAID5 with 6 disks, noatime, S10U2 + patches (the same filesystem mounted as in 3)
IO Summary: 393167 ops 6503.1 ops/s, (1000/1001 r/w) 32.4mb/s, 405us cpu/op, 7.5ms latency
IO Summary: 394525 ops 6521.2 ops/s, (1003/1003 r/w) 32.0mb/s, 407us cpu/op, 7.7ms latency

5. ZFS with software RAID-Z with 6 disks, atime=off, S10U2 + patches (the same disks as in test #2)
IO Summary: 461708 ops 7635.5 ops/s, (1175/1175 r/w) 37.4mb/s, 330us cpu/op, 6.4ms latency
IO Summary: 457649 ops 7562.1 ops/s, (1163/1164 r/w) 37.0mb/s, 328us cpu/op, 6.5ms latency


See my post on zfs-discuss@opensolaris.org list for more details.


I have also found some benchmarks comparing ZFS, UFS, RAISERFS and EXT3 - ZFS was of course the fastest one on the same x86 hardware. See here and here.

DTrace in Mac OS

Thanks to Alan Coopersmith I've just learned that DTrace will be part of MacOS X Leopard.

Mac OS Leopard Xcode:

Track down problems

When you need a bit more help in debugging, Xcode 3.0 offers an extraordinary new program, Xray. Taking its interface cues from timeline editors such as GarageBand, now you can visualize application performance like nothing you’ve seen before. Add different instruments so you can instantly see the results of code analyzers. Truly track read/write actions, UI events, and CPU load at the same time, so you can more easily determine relationships between them. Many such Xray instruments leverage the open source DTrace, now built into Mac OS X Leopard. Xray. Because it’s 2006.

btw: such a GUI tool would be useful for many Solaris admins too

HW RAID vs. ZFS software RAID

I used 3510 head unit with 73GB 15K disks, RAID-10 made of 12 disks in one enclosure.
On the other server (the same server specs) I used 3510 JBODs with the same disk models.

I used filebench to generate workloads. "varmail" workload was used for 60s, two runs for each config.


1. ZFS filesystem on HW lun with atime=off:

IO Summary: 499078 ops 8248.0 ops/s, (1269/1269 r/w) 40.6mb/s, 314us cpu/op, 6.0ms latency
IO Summary: 503112 ops 8320.2 ops/s, (1280/1280 r/w) 41.0mb/s, 296us cpu/op, 5.9ms latency

2. UFS filesystem on HW lun with maxcontig=24 and noatime:

IO Summary: 401671 ops 6638.2 ops/s, (1021/1021 r/w) 32.7mb/s, 404us cpu/op, 7.5ms latency
IO Summary: 403194 ops 6664.5 ops/s, (1025/1025 r/w) 32.5mb/s, 406us cpu/op, 7.5ms latency

3. ZFS filesystem with atime=off with ZFS raid-10 using 12 disks from one enclosure:
IO Summary: 558331 ops 9244.1 ops/s, (1422/1422 r/w) 45.2mb/s, 312us cpu/op, 5.2ms latency
IO Summary: 537542 ops 8899.9 ops/s, (1369/1369 r/w) 43.5mb/s, 307us cpu/op, 5.4ms latency


In other tests HW vs. ZFS software raid show about the same performance.
So it looks like at least in some workloads software ZFS raid can be faster than HW raid.
Also please notice that HW raid was done on real HW array and not some crappy PCI raid card.

For more details see my post on ZFS discuss list.

Solaris Internals

Finally both books of new Solaris Internals are available. It's must buy for everyone seriously using Solaris.
See here and here.