SAM-QFS Open Sourced

I'm really impressed with Sun when it comes to Open Source - Solaris, Sun Cluster, DTace, ZFS, ... and now SAM-QFS. Read More.

I really wonder if such a business model will prove - I guess we have to wait another couple of years to see.

Nevertheless it is not only that all these products are open sourced it is also that they are entirely for free which in many cases is disruptive. For example - I've been installing clusters where it did not make sense before, for example: 2x x86 servers + Solaris (free) + Sun Cluster (free) + MySQL (free). Or instead of spending too much money for NetApp you just go with 2x x86 + Solaris + Sun Cluster + ZFS + Comstar - you will save lot of money. However NetApp story is perhaps worth separate blog entry...

Interview question

During interviews one of the questions I ask is: "What is the basic difference between 32-bit and 64-bit application?". The obvious one is the available address space for an application. The other one is performance - more details here.

btw: it is really scary how few sys admins know the answer to the question, even if just the address space part...

Sun's support

Remember my last post about Sun's support? Well, not about support itself but rather about tools Sun is providing to users which are crap. OSC is still saying it will be upgraded in March 2008 - it's just couple of weeks left so we will see.

Now I got several new boxes and I want to register them using sconadm - with or without support contract and it doesn't work, which means I can't use smpatch. Raising a ticket has not helped so far either.

Sun should definitely put more love to its patching infrastructure - I don't care if it old, we know Indiana is going to change it all, etc. But in a mean time they should assure that we (customers) have a working solution.

Going back to patchdiag tool... feeling disappointed at Sun.
Or rather I will finally try PCA - so far so good and it works!


Update: for whatever reason PCA make patching unusable - patch* commands started to core dump.

I got smpatach working, thanks to Sun's support (thanks Ben!). The problem is bug id: 6643363 which on systems with 64 or more CPUs makes smpatch to fail.

16-core Intel System vs. Niagara-2

Some sysbench results for Niagara-2 (blade), v440 and 16-core Intel box.
For a CPU test Niagara-2 and 16-core Intel box deliver about the same performance with 32 threads, while Niagara is slower for less number of threads.


Sysbench memory test shows up-to 17GB/s for Niagara-2 blade, while only up-to 4GB/s for Intel 16-core box.



Sysbench threads test - again, more threads more performance for Niagara-2. Starting with 32 threads and more Niagara-2 box is the fastest one.


Quick conclusion is that Niagara-2 box can rival 4-CPU 4-core (16 cores in total) Intel boxes if your application can utilize all these cores. Sometimes it's much faster - especially if you need to access lots of memory. Of course it's just sysbench...


Niagara-2 - 1x 1.4GHz 8-core UltraSparc-T2, 32GB RAM
v440 - 4x 1GHz, 8GB RAM
16 core Intel - 4x 4-core Intel CPU (16 cores in total) 2.13GHz, 16GB RAM

MySQL Tuning

Interesting article on basic MySQL tuning.

16x Intel Cores

4x 4-core Intel CPU in one box... why not? The market is so quickly getting multi-core, even my notebook is multi-core these days.

# psrinfo -pv
The physical processor has 4 virtual processors (0 4-6)
 x86 (GenuineIntel 6FB family 6 model 15 step 11 clock 2133 MHz)
       Intel(r) CPU                  @ 2.13GHz
The physical processor has 4 virtual processors (1 7-9)
 x86 (GenuineIntel 6FB family 6 model 15 step 11 clock 2133 MHz)
       Intel(r) CPU                  @ 2.13GHz
The physical processor has 4 virtual processors (2 10-12)
 x86 (GenuineIntel 6FB family 6 model 15 step 11 clock 2133 MHz)
       Intel(r) CPU                  @ 2.13GHz
The physical processor has 4 virtual processors (3 13-15)
 x86 (GenuineIntel 6FB family 6 model 15 step 11 clock 2133 MHz)
       Intel(r) CPU                  @ 2.13GHz
#

FreeBSD 7.0

There is an interview with several FreeBSD developers on what's new in version 7.0

I have mixed feeling about FreeBSD - you look thru what's new and you got a feeling it's just catching up with other OS'es with no innovation on its own. IPMI support? 1:1 threading model? Journaling for a file system? - and not even "real" one... Maybe they did managed to get slightly better performance for a specific workloads but the question is - why would I need FreeBSD?
Then you have little ISV support - try to get Oracle libraries for FreeBSD.

In x86 market you even have already a big market share or you innovate to get one, you add some new value - frankly I can't find any added value in FreeBSD comparing to Solaris or Linux on x86. Sure, one can find some small niche where it makes sense, but in general one will be much more happy with Solaris or Linux.

Sadly (or maybe not?) FreeBSD is becoming more and more an OS for hobbyst - or maybe it has always been....

2530 Array and ZFS

ZFS by default flushes disk cache after each transaction completes or after each synchronous operation completes so it assures all data which is supposed to be on physical disk is actually there. This is good. However if you use ZFS with some external array with battery backed up cache then with some arrays array will flush its entire cache to disks every time ZFS sends a scsi flush command. Because your cache in your array is usually mirrored and battery backed up you don't want that to happen as it usually will affect badly array's performance. Some arrays just ignore these scsi flush commands but some of them not. ZFS team is working on that problem so ZFS sends different scsi flush command which says to flush cache only if it's not protected and not to send scsi flush commands at all to some arrays.

Currently if you are hit with the problem you can configure your array so it ignores cache flushes or you can configure ZFS to not to send scsi flushes at all. Check ZFS Evil tuning guide for more information.

Sun SAS 2530 array is one of this arrays which will flush cache when asked to.

Let's wrtite a simple C program which will create a new file with O_DSYNC flag set, write 255 bytes, close the file then delete it. Then it repeats it N times. Then lets compare what is the time difference when run on ZFS file system on 2530 array with ZFS set to send cache flushes (default) and when ZFS is set to not to send them.

# echo zfs_nocacheflush/D | mdb -k
zfs_nocacheflush:
zfs_nocacheflush:               0             
[default, zfs will send cache flush commands]

# ./filesync-1 /slave/tmp 10000
Time in seconds to create and unlink 10000 files with O_DSYNC: 59.041564

[let's dynamically turn off cache flushes by zfs]
# echo zfs_nocacheflush/W1 | mdb -kw
zfs_nocacheflush:               0               =       0x1

# ./filesync-1 /slave/tmp 10000
Time in seconds to create and unlink 10000 files with O_DSYNC: 7.050389

We get over 8x performance improvement!
With multiple streams we probably would get ever bigger improvement.

To permanently disable ZFS cache flushes put in /etc/system

set zfs:zfs_nocacheflush = 1

Remember that it will disable cache flushes to ALL zfs file systems in your system.

Below is a source code for filesync-1 program. Remember - it's a quick program written in 1 minute to just make a quick test, it's definitely far from beautiful coding.

#include <stdio.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/nameser.h>
#include <resolv.h>
#include <netdb.h>
#include <sys/time.h>
#include <inttypes.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

int main(int argc, char *argv[])
{
  char *path;
  char filename[256];
  int n_req;
  int i, fd;
  hrtime_t stopper_start, stopper_stop;

  if(argc != 3)
  {
    printf("Usage: %s path N\n", argv[0]);
    exit(1);
  }

  path = argv[1];
  n_req = atoi(argv[2]);

  stopper_start = gethrtime();
  i=0;
  while (i++ < n_req)
  {
    strcpy(filename, path);
    strcat(filename, "/filesync_test");
    fd = open(filename, O_CREAT|O_RDWR|O_DSYNC);
    write (fd, filename, 255);
    close(fd);
    unlink(filename);
  }
  stopper_stop = gethrtime();

  printf("Time in seconds to create and unlink %d files with O_DSYNC: %f\n\n", n_req, (float)(stopper_stop - stopper_start)/1000000000);

  exit(0);
}

pNFS 4.1 + RAID-10

That's a good news - pNFS does support now not only RAID-0 but also RAID-10!
Check this presentation on latest update on pNFS (aka NFSv4.1) status in Open Solaris.

Indiana Preview 2

Open Solaris Developer Preview 2 has just been announced. If you want to see how next generation Open Solaris dirsto will look like go for it.

NexentaCP 1.0

Debian Ubuntu/Dapper based on Open Solaris? Why not. I guess a lot of people will be interested.
It is like having Linux but much better one :)

TACC's Ranger Goes Live

I was involved once with building and testing quite a large HPC grid cluster may years ago... Nothing as big as that one :)

MS to buy Yahoo?

That's THE news of the year so far. No point giving any links here - you just couldn't miss it.

New Books on Solaris

Click

Rapid Data Center Deployment