Open Storage Wish List

1. L2ARC should survive reboots

There is already an RFE for this and AFAIK it's been working on.

2. Ability to mirror ARC between cluster nodes

In some workloads it is there important to be able to sustain a provided performance. To warm up a cache it could take even hours during which time the delivered performance could be lower than usual. #1 once implemented should partly fix the issue but still filling-in 128GB of cache could take some time and negatively impact the performance. I think the replication not necessarily would have to be a synchronous one. It probably would be hard to implement and maybe is not worth it...

3. L2ARC and SLOG SSDs shouldn't be included in disk drive IOPS stats.

While I haven't looked into it very closely it seems that when graphing IOPS for disk drives both L2ARC and SLOG numbers are included in totals. I think it is slightly misleading and a separate graph should be provided just for L2ARC and SLOG.

4. Ability to create a storage pool without L2ARC and/or SLOG devices even if they are present

While this is not necessarily important for production deployments it would help with testing/benchmarking so one doesn't have to physically remove SSDs in order to be able to build a pool without them.

Open Storage and Data Caching

I've been playing with Open Storage 7410 recently. Although I've been using its GUI for quite some time thanks to FishWorks beta program it still amazes me how good it is, especially when you compare it to NetApp or Data Domain.

One of the really good things about Open Storage is it allows for quite a lot of Read/Write cache (currently up-to 128GB). If it is still not enough you can put up to ~600GB of additional Read Cache in terms of SSDs. What it means in practice is that many real-life workloads will entirely fit into the cache which in turn will provide excellent performance. In a way this is nothing new except for... economics! Try to find any other NAS product in the market when you can put ~600GB of cache and within the same price range as Open Storage. You won't find anything like this.

I have created a disk pool out of 20x 1TB SATA disk drives which are protected with RAID-DP (akd RAIDZ2 which is an implementation of RAID-6). Now RAIDZ2 is known for a very bad random read performance from multiple streams if data is not cached. Using filebench I run a random read workload for a dataset of 10GB (let's say a small MySQL database) with 16 active streams. The 7410 appliance has been rebooted prior to test so all caches were clean. As you can see on below screenshot at the beginning it was able to sustain ~400 NFSv3 operations per second. After about 50 minutes it delivered ~12,000 NFSv3 operations per second which saturated my 1GbE link. At about the same time the average latency for NFS operations were getting smaller and smaller, same for number of operations to physical disks. At some point all data had been in cache and there were no operations to physical disks at all. B

The appliance could do certainly much more if I would use more GbE links of 10GbE links. Now remember that I used 20x 1TB SATA disk drives in a RAID-DP configuration to get this performance and it could sustain it for workloads of up to ~600GB of a working set size. If you put these numbers into perspective: one 15K FC disk drive can deliver ~250 8KB random reads at most. You would need almost 100 such disk drives configured in RAID-10 to be able to match the performance and still you would get less capacity (even assuming 300GB FC 15K drives).

Open Storage is the game changer for a lot of workloads both in terms of a delivered performance and a cost - currently there isn't really anything in the market which can match it.

Boomer: Next Generation Solaris Audio

See also PSARC/2008/318 for more details and this pdf in specific for a nice overview.

Interview with Larry Ellison

GNU grep vs. grep

Open Solaris b111 x64

$ ptime /usr/gnu/bin/grep mysqld 1_4GB_txt_file

real     1:32.055472017
user     1:30.202692546
sys         0.907308690

$ ptime /usr/bin/grep mysqld

real        8.725173958
user        7.621411130
sys         1.056151347

I guess it's due to GNU version being compiled without optimizations... or maybe it is something else. Once I find some time I will try to investigate it.

Time Slider Enhancements for Open Solaris 2009.06

Solaris 10 5/09

Solaris 10 5/09 aka update 7 is out and available for download. Below are some new features from What's New I found interesting:

Support for Zone Cloning

If the source and the target zonepaths reside on ZFS and both are in the same pool, a snapshot of the source zonepath is taken and the zoneadm clone uses ZFS to clone the zone.

SunSSHWith OpenSSL PKCS#11 Engine Support

This feature enables the SunSSH server and client to use Solaris Cryptographic Framework through the OpenSSL PKCS#11 engine. SunSSH uses cryptographic framework for hardware crypto acceleration of symmetric crypto algorithms which is important to the data transfer speed. This feature is aimed at UltraSPARC® T2 processor platforms with n2cp(7D) crypto driver.

iSCSITarget
Several bug fixes and improvements. See the What's New for more details.

Solaris Power Aware Dispatcher and Deep C-State Support

Event driven CPU power management –On systems that supportDynamic Voltage and Frequency Scaling (DVFS) by Solaris, the kernel scheduler or dispatcher will schedule threads across the system's CPUs in a manner that coalesces load, and frees up other CPUs to be deeply power managed. CPU power state changes are triggered when the dispatcher recognizes that the utilization across a group of power manageable CPUs has changed in a significant way. This eliminates the need to periodically poll CPU utilizations across the system, and enables the system to save more power when CPUs are not used, while driving performance when CPUs are used. Event driven CPU power management is enabled by default on systems that supportDVFS. This feature can be disabled, or the legacy polling-based CPU power management can be used through the cpupm keyword in power.conf(4).

■ Support forDeep Idle CPU PowerManagement or deep C-state support on Intel Nehalem-based systems – The project also adds Solaris support forDeep C-states on Intel Nehalem-based systems. This support enables unused CPU resources to be dynamically placed in a state where they consume a fraction of the power consumed in their normal operating state. This feature also provides Solaris support for the power saving feature, as well as the policy implementation that decides when idle CPUs should request deep idle mode. This feature will be enabled by default where supported, and can be disabled through the cpu-deep-idle keyword in power.conf(4).

■ Observability for Intel's TurboMode feature – IntelNehalem-based systems have the ability to raise the operating frequency of a subset of the available cores when there is enough thermal headroom to do so. This ability temporarily boosts performance, but it is controlled by the hardware and transparent to software. Starting with the Solaris 10 5/09 release, a new kstat module observes when the system is entering the turbo mode and at which frequency it operates.

Some perspective to Suns Q3FY2009 numbers

Upgrading Open Storage 7410

Adam blogged about a new software release for the Open Storage. Because I'm testing 7410 model I went thru an upgrade this morning. First I downloaded the new image and I was impressed - I got 9-10MB/s and it looks like the bottleneck was my 100Mbs link to an office network - I never got such download rates from Sun before, excellent!

Then with just one mouse click I uploaded the new image onto the 7410 and after a short while it was listed as ready to be installed. So I did click on it and after about 25 minutes later it finished. Of course during the upgrade I could still use the appliance.

I really like the end-user experience - couple of mouse-clicks and you're done. That's the way it should be.

I did some testing with filebench before and after the upgrade and I'm really happy to share that I'm getting about 33% performance improvement for varmail workload with the new build. While your mileage may vary I think Sun should have highlighted performance improvements in the Release Notes.

What would really be useful for testing purposes is an ability to create a pool without L2ARC or SLOG devices - this would make life a little bit easier with testing when comparing configurations with and without SSDs. The default behaviour is excellent as it will pick up SSDs and propose most optimal use of them so end-user doesn't have to even understand how it works and how to configure them properly. Still I would like to have an option to not configure L2ARC or SLOG without having to physically pull them out.

Oracle Agrees to Acquire Sun Microsystems

This is a big surprise!

From Oracle's document on the acquisition:

• Protects and extends customers’ investment in Sun technologies
• Accelerate growth of Java as an open industry standard development platform
• Sustain Solaris as an industry standard OS for Oracle software
• Continue Open Storage and Systems focus and innovation
• Ensure continued innovation and investment in Java technology
• Optimize Solaris and Oracle for better performance, reliability, and manageability
• Protects massive customer investment in SPARC
• Open Storage built with industry standard servers and components

Sun's Official Announcement
Wall Street Journal

truss(1M) vs. dtrace(1M)

One of the many benefits of DTrace vs. truss is that dtrace should induce much smaller overhead for tracing applications especially for multi-threaded applications running on multi core/cpu servers. Lets put it to a quick test.

I quickly wrote a small C program which spawns N threads and each thread does stat("/tmp") X times. Then I measured how much time it takes to execute it for 1mln stat()'s in total while running with no tracing at all, running under truss and running under dtrace.

One two-core AMD CPU

# ptime ./threads-2 1 1000000

real        2.662809885
user        0.223471401
sys         2.435895135

# ptime ./threads-2 2 500000

real        1.649542016
user        0.226104849
sys         3.045784378

# ptime truss -t xstat -c ./threads-2 2 500000

syscall               seconds   calls  errors
xstat                   6.966 1000000
stat64                   .000       3       1
             --------  ------   ----
sys totals:             6.966 1000003      1
usr time:                .776
elapsed:               18.520

real       18.533000528
user        5.677239771
sys        16.069020190

# dtrace -n 'syscall::xstat:entry{@=count();}' -c 'ptime ./threads-2 2 500000'
dtrace: description 'syscall::xstat:entry' matched 1 probe

real        1.888294217
user        0.225676973
sys         3.506004575
dtrace: pid 8526 has exited

  1000000

truss made the program to execute about 11x longer while dtrace made program to execute for about 14% longer.


Niagara server:

# ptime ./threads-2 1 1000000

real       10.873
user        1.881
sys         8.992

# ptime ./threads-2 10 100000

real        1.467
user        1.962
sys        12.121

# ptime truss -t xstat -c ./threads-2 1 1000000

syscall               seconds   calls  errors
stat                   26.958 1000004       1
              --------  ------   ----
sys totals:            26.958 1000004      1
usr time:               2.758
elapsed:              214.600

real     3:34.613
user       30.900
sys      2:28.182

# ptime truss -t xstat -c ./threads-2 10 100000

syscall               seconds   calls  errors
stat                   37.259 1000004       1
             --------  ------   ----
sys totals:            37.259 1000004      1
usr time:               3.178
elapsed:              168.010

real     2:48.063
user     1:05.709
sys      3:35.813

# dtrace -n 'syscall::stat:entry{@=count();}' -c 'ptime ./threads-2 1 1000000'
dtrace: description 'syscall::stat:entry' matched 1 probe

real       14.028
user        1.957
sys        12.069
dtrace: pid 12920 has exited

    1000939

# dtrace -n 'syscall::stat:entry{@=count();}' -c 'ptime ./threads-2 10 100000'
dtrace: description 'syscall::stat:entry' matched 1 probe

real        1.858
user        2.142
sys        15.632
dtrace: pid 11679 has exited

  1000083

truss made the program to execute about 20x longer in the single thread case and 115x longer for the multi threaded one while dtrace added no more than 30% to the execution time regardless if the application was running with one or many executing threads. This shows that one has to be especially careful when using truss on a multi CPU/core system on a multi-threaded application. Notice that the performance difference between multi-threaded and single-threaded example for truss shows not that much difference comparing to execution times with no tracing at all which shows the ugly feature of truss - it serializes a multi-threaded application.

Of course the benchmark is the worst-case scenario and in real life you shouldn't get that much overhead from both tools. Still truss in some cases could introduce too much overhead on a production server while dtrace would still be perfectly acceptable allowing you to continue with your investigation.

btw: DTraceToolkit provides a script called dtruss - it's a tool similar to truss but it is using DTrace.

cat threads-2.c


#include <thread.h>
#include <stdlib.h>
#include <pthread.h>
#include <fcntl.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>

void *thread_func(void *arg)
{
  int *N=arg;
  int i;
  struct stat buf;

  for (i=0; i<*N; i++)
    stat("/tmp", &buf);

  return(0);
}

int main(int argc, char **argv)
{
  int N, iter;
  int i;
  int rc;
  pthread_t tid[255];

  if (argc != 3)
  {
    printf("%s number_of_threads number_of_iterations_per_thread\n", argv[0]);
    exit(1);
  }

  N = atoi(argv[1]);
  iter = atoi(argv[2]);

  for (i=0; i<N; i++)
  {
    if (rc = pthread_create(&tid[i], NULL, thread_func, &iter))
      printf("Thread #%d creation failed [%d]\n", i, rc);
  }


  /* wait for all threads to complete */
  for (i=0; i<N; i++)
    pthread_join(tid[i], NULL);

  exit(0);
}

ZFS Deduplicatuion This Summer?

Jeff Bonwick wrote:

"Yes -- dedup is my (and Bill's) current project.  Prototyped in December.
Integration this summer.  I'll blog all the details when we integrate,
but it's what you'd expect of ZFS dedup -- synchronous, no limits, etc."

The CPU Overclocks itself

Joerg reports:

"With the announcement of Intel Nehalem support in Solaris, we pointed to some interesting features, but from my perspective the power-aware dispatcher is the most interesting one. I wrote a while ago about the turbo boost feature of the Nehalem processors. The processor overclocks itself, when there is still head room in the power and thermal budget. It can overclock a core even higher, when other cores are in deep sleep. Otherwise it can make sense not to use a core for a single process, when there is enough compute power available otherwise you could put this core into a deep sleep mode just to save power. The new power-aware dispatcher in Solaris is aware of this side conditions and can dispatch the processes in a System accordingly. You will find more informations at the projects website."

Trying too hard

From time to time I can see people trying to be too clever about some problems. What I mean by that is that sometimes they try too hard to use latest technologies to do something while there is already a solution which does the job. Or sometimes instead of taking a step back and taking a deep breath they dive directly into problem solving coming up with crazy ways to accomplish something. I guess it happens to all of us from time to time. This time it happened to me :) :)

A colleague approached me with a problem he had on some old Solaris 7 server which is stripped and customized and there is no pargs command there. He needed to get a full argument list of a running process but ps truncate it to 80 characters. Well I thought a simple C program should be able to extract the information via /proc. So me trying to be helpful I started to write it right a way. After some time I came up with:

bash-2.05# cat pargs.c

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <procfs.h>
#include <sys/procfs.h>
#include <sys/prsystm.h>



int main(int argc, char *argv[])
{
psinfo_t p;
char *file;
int fd;
int fd_as;
uintptr_t pargv[1024];
char arg[1024];
int i;

if(argc != 3)
{
 printf("Usage: %s /proc/PID/psinfo\n", argv[0]);
 exit(1);
}

file = argv[1];
fd = open(file, O_RDONLY);
if (fd == -1)
{
 printf("Can't open %s file\n", file);
 exit(2);
}

read(fd, &p, sizeof(p));
close(fd);

fd_as = open(argv[2], O_RDONLY);

printf("nlwp: %d\n", p.pr_nlwp);
printf("exec: %s\n", p.pr_fname);
printf("args: %s\n", p.pr_psargs);
printf("argc: %d\n", p.pr_argc);

pread(fd_as, &pargv, p.pr_argc * sizeof (uintptr_t), p.pr_argv);
for (i=0; i<p.pr_argc; i++)
{
 pread(fd_as, &arg, 256, ((uintptr_t *)pargv)[i]);
 printf(" %s\n", arg);
}

close(fd_as);
exit(0);
}

Job done.
Well couple of minutes later I realized that UCB version of ps is able to show long argument list...

bash-2.05# /usr/ucb/ps -axuww  |grep "19179"
XXXX    19179  9.3  2.23998422056 ?        S 11:02:30  0:02 /usr/java/bin/../bin/sparc/native_threads/java -classpath :./classes/packages/jakarta-regexp-1.3.jar:./classes/packages/classes12.zip:./classes/packages/mail.jar:./classes/packages/activation.jar:./classes/ MailSender
bash-2.05#

I had a good laugh at myself.

Library Interposer

Recently I have used Dtrace to change the output of uname() syscall. But if one wants a more permanent and selective approach it is easier to write a small library which would interpose the uname() syscall (well, actually uname() libC function and not a syscall itself). I slightly modified the malloc_interposer example.

After you compiled the library all you have to do is to LD_PRELOAD it in your script so everything started by that script will use it or you can LD_PRELOAD it only for a given binary as shown below. Additionally you have to set a variable uname_release to whatever string you like otherwise the library won't do anything.

# uname -a
SunOS test-server 5.10 Generic_125100-10 sun4u sparc SUNW,Sun-Fire-V440
#
# uname_release="5.7" LD_PRELOAD=./uname_interposer.so uname -a
SunOS test-server 5.7 Generic_125100-10 sun4u sparc SUNW,Sun-Fire-V440



# cat uname_interposer.c
/* Based on http://developers.sun.com/solaris/articles/lib_interposers_code.html#malloc_interposer.c
*/
/* Example of a library interposer: interpose on
* uname().
* Build and use this interposer as following:
* cc -o malloc_interposer.so -G -Kpic malloc_interposer.c
* setenv LD_PRELOAD $cwd/uname_interposer.so
* run the app
* unsetenv LD_PRELOAD
*/

#include <stdio.h>
#include <dlfcn.h>
#include <stdlib.h>

#include <sys/utsname.h>

int uname(struct utsname *name)
{
int rc;
char *release;

static int (*uname_func)(struct utsname *) = NULL;
if(!uname_func)
  uname_func = (int (*)(struct utsname*)) dlsym(RTLD_NEXT, "uname");
rc = uname_func(name);
if (release=getenv("uname_release"))
  strlcpy(name->release, release, _SYS_NMLN);

return(rc);
}
#

# gcc -fPIC -g -o uname_interposer.so -G uname_interposer.c