Memory DeDuplication in Linux

Oracle Solaris on HP ProLiant Servers

Recently there has been lots of confusion regarding running Solaris 10 on non-Sun servers.

HP Oracle Solaris 10 Subscriptions and Support:

"Certifying Oracle Solaris on ProLiant servers since 1996, HP is expanding its relationship with Oracle to include selling Oracle Solaris 10 Operating System Subscriptions and support from HP Technology Services on certified ProLiant servers.

HP will provide the subscriptions and support for the Oracle Solaris 10 Operating System on certified ProLiant servers and Oracle will provide patches and updates directly to HP's customers through Oracle SunSolve.

As part of this expanded relationship HP and Oracle will work together to enhance the customer experience for Oracle Solaris on ProLiant servers and HP increase its participation in the OpenSolaris community."

And of course you can also run Open Solaris on any x86 hardware, including HP servers, entirely for free if you want. I wonder though if it would make sense for HP to also offer support for Open Solaris - more and more customers are deploying Open Solaris instead of Solaris 10 on their servers and Oracle already offers a support for it on their own servers.

ZFS diff

PSARC/2010/105:

        There is a long-standing RFE for zfs to be able to describe
        what has changed between the snapshots of a dataset.
        To provide this capability, we propose a new 'zfs diff'
        sub-command.  When run with appropriate privilege the
        sub-command describes what file system level changes have
        occurred between the requested snapshots.  A diff between the
        current version of the file system and one of its snapshots is
        also supported.

        Five types of change are described:

        o    File/Directory modified
        o    File/Directory present in older snapshot but not newer
        o    File/Directory present in newer snapshot but not older
        o    File/Directory renamed
        o    File link count changed

      zfs diff snapshot  snapshot | filesystem

         Gives a high level description of the differences between a
         snapshot and a descendant dataset.  The descendant may either
         be a later snapshot of the dataset or the current dataset.
         For each file system object that has undergone a change
         between the original snapshot and the descendant, the type of
         change is described along with the name of the file or
         directory.  In the case of a rename, both the old and new
         names are shown.

         The type of change is described with a single character:

         +   Indicates the file/directory was added in the later dataset
         -   Indicates the file/directory was removed in the later dataset
         M   Indicates the file/directory was modified in the later dataset
         R   Indicates the file/directory was renamed in the later dataset

        If the modification involved a change in the link count of a
        file, the change will be expressed as a delta within
        parentheses on the modification line.  Example outputs are
        below:

         M       /myfiles/
         M       /myfiles/link_to_me   (+1)
         R       /myfiles/rename_me -> /myfiles/renamed
         -       /myfiles/delete_me
         +       /myfiles/new_file

Project Brussels Phase II

This project introduces a new CLI utility called ipadm(1M) that can be used to perform:

* IP interfaces management (creation/deletion)
* IP address management (add, delete, show) for static IPv4& IPv6 addresses, DHCP, stateless/stateful IPv6 Address configuration
* protocol (IP/TCP/UDP/SCTP/ICMP) tunable management (set, get, reset) global (ndd(1M)) tunables, as well as per-interface tunables
* provide persistence for all of the three features above so that on reboot the configuration is reapplied

Please see the case materials of PSARC 2010/080 for the latest design document and read ipadm(1M) man page for more information.

This has been integrated into Open Solaris.

CPU/MEM HotPlug on x86 in Open Solaris

The integration of:

 PSARC/2009/104 Hot-Plug Support for ACPI-based Systems
 PSARC/2009/550 PSMI extensions for CPU Hotplug
 PSARC/2009/551 acpihpd ACPI Hotplug Daemon
 PSARC/2009/591 Attachment Points for Hotpluggable x86 systems
 6862510 provide support for cpu hot add on x86
 6874842 provide support for memory hot add on x86
 6883891 cmi interface needs to support dynamic reconfiguration
 6884154 x2APIC and kmdb may not function properly during CPU hotplug event.
 6904971 low priority acpi nexus code review feedback
 6877301 lgrp should support memory hotplug flag in SRAT table

Introduces support for hot-adding cpus and memory to a running Xeon 7500 platform.

ReadyBoost

I didn't know that Windows has a similar technology to ZFS L2ARC which is called ReadyBoost. Nice.

I'm building my new home NAS server and I'm currently seriously considering putting OS on an USB pen drive leaving all sata disks for data only. It looks like with modern USB drives OS should actually boot faster than from a sata disks thanks to much better seek times. I'm planning on doing some experiments first.

ZVOLs - Write Cache

When you create a ZFS volume its write cache is disabled by default meaning that all writes to the volume will be synchronous. Sometimes it might be handy though to be able to enable a write cache for a particular zvol. I wrote a small C program which allows you to check if WC is enabled or not. It also allows you to enable or disable write cache for a specified zvol.

First lets check if write cache is disabled for a zvol rpool/iscsi/vol1

milek@r600:~/progs# ./zvol_wce /dev/zvol/rdsk/rpool/iscsi/vol1
Write Cache: disabled

Now lets issue 1000 writes

milek@r600:~/progs# ptime ./sync_file_create_loop /dev/zvol/rdsk/rpool/iscsi/vol1 1000

real       12.013566363
user        0.003144874
sys         0.104826470

So it took 12s and I also confirmed that writes were actually being issued to a disk drive. Lets enable write cache now and repeat 1000 writes

milek@r600:~/progs# ./zvol_wce /dev/zvol/rdsk/rpool/iscsi/vol1 1
milek@r600:~/progs# ./zvol_wce /dev/zvol/rdsk/rpool/iscsi/vol1
Write Cache: enabled

milek@r600:~/progs# ptime ./sync_file_create_loop /dev/zvol/rdsk/rpool/iscsi/vol1 1000

real        0.239360231
user        0.000949655
sys         0.019019552

Worked fine.

The zvol_wce program is not idiot-proof and it doesn't check if operation succeeded or not. You should be able to compile it by issuing: gcc -o zvol_wce zwol_wce.c

milek@r600:~/progs# cat zvol_wce.c

/* Robert Milkowski
  http://milek.blogspot.com
*/

#include <unistd.h>
#include <stropts.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stropts.h>
#include <sys/dkio.h>


int main(int argc, char **argv)
{
 char *path;
 int wce = 0;
 int rc;
 int fd;

 path = argv[1];

 if ((fd = open(path, O_RDONLY|O_LARGEFILE)) == -1)
   exit(2);

 if (argc>2) {
   wce = atoi(argv[2]) ? 1 : 0;
   rc = ioctl(fd, DKIOCSETWCE, &wce);
 }
 else {
   rc = ioctl(fd, DKIOCGETWCE, &wce);
   printf("Write Cache: %s\n", wce ? "enabled" : "disabled");
 }

 close(fd);
 exit(0);
}

60 Disks in 4U

Chip and PIN is Broken

Dell - No 3rd Party Disk Drives Allowed

Third-party drives not permitted:

"[...]
Is Dell preventing the use of 3rd-party HDDs now?
[....]
Howard_Shoobe at Dell.com:
Thank you very much for your comments and feedback regarding exclusive use of Dell drives. It is common practice in enterprise storage solutions to limit drive support to only those drives which have been qualified by the vendor. In the case of Dell's PERC RAID controllers, we began informing customers when a non-Dell drive was detected with the introduction of PERC5 RAID controllers in early 2006. With the introduction of the PERC H700/H800 controllers, we began enabling only the use of Dell qualified drives. There are a number of benefits for using Dell qualified drives in particular ensuring a positive experience and protecting our data. While SAS and SATA are industry standards there are differences which occur in implementation. An analogy is that English is spoken in the UK, US and Australia. While the language is generally the same, there are subtle differences in word usage which can lead to confusion. This exists in storage subsystems as well. As these subsystems become more capable, faster and more complex, these differences in implementation can have greater impact. Benefits of Dell's Hard Disk and SSD drives are outlined in a white paper on Dell's web site at http://www.dell.com/downloads/global/products/pvaul/en/dell-hard-drives-pov.pdf"

I understand they won't support 3rd party disk drives but blocking a server (a RAID card) from using such disks is something new - an interesting comment here.

IBM 2010: Customers in Revolt

From my own experience their sales people are very aggressive with an attitude of sale first and let someone else worry later. While I always take any vendor claims with a grain of salt I learnt to double or even triple check any IBM's claims.

Power 7

Now lets wait for some benchmarks. I only wish Solaris was running on them as well as right now you need to go the legacy AIX route or not so mature Linux route - not an ideal choice.

Data Corruption - ZFS saves the day, again

We came across an interesting issue with data corruption and I think it might be interesting to some of you. While preparing a new cluster deployment and filling it up with data we suddenly started to see below messages:

XXX cl_runtime: [ID 856360 kern.warning] WARNING: QUORUM_GENERIC: quorum_read_keys error:
 Reading the registration keys failed on quorum device /dev/did/rdsk/d7s2 with error 22.

The d7 quorum device was marked as being offline and we could not bring it online again. There isn't much in documentation about the above message except that it is probably a firmware problem on a disk array and we should contact a vendor. But lets investigate first what is really going on.

By looking at the source code I found that the above message is printed from within quorum_device_generic_impl::quorum_read_keys() and it will only happen if quorum_pgre_key_read() returns with return code 22 (actually any other than 0 or EACCESS but from the syslog message we already suspect that the return code is 22).

The quorum_pgre_key_read() calls quorum_scsi_sector_read() and passes its return code as its own. The quorum_scsi_sector_read() will return with an error only if quorum_ioctl_with_retries() returns with an error or if there is a checksum mismatch.

This is the relevant source code:

406 int
407 quorum_scsi_sector_read(
[...]
449  error = quorum_ioctl_with_retries(vnode_ptr, USCSICMD, (intptr_t)&ucmd,
450      &retval);
451  if (error != 0) {
452   CMM_TRACE(("quorum_scsi_sector_read: ioctl USCSICMD "
453       "returned error (%d).\n", error));
454   kmem_free(ucmd.uscsi_rqbuf, (size_t)SENSE_LENGTH);
455   return (error);
456  }
457
458  //
459  // Calculate and compare the checksum if check_data is true.
460  // Also, validate the pgres_id string at the beg of the sector.
461  //
462  if (check_data) {
463   PGRE_CALCCHKSUM(chksum, sector, iptr);
464
465   // Compare the checksum.
466   if (PGRE_GETCHKSUM(sector) != chksum) {
467    CMM_TRACE(("quorum_scsi_sector_read: "
468        "checksum mismatch.\n"));
469    kmem_free(ucmd.uscsi_rqbuf, (size_t)SENSE_LENGTH);
470    return (EINVAL);
471   }
472
473   //
474   // Validate the PGRE string at the beg of the sector.
475   // It should contain PGRE_ID_LEAD_STRING[1|2].
476   //
477   if ((os::strncmp((char *)sector->pgres_id, PGRE_ID_LEAD_STRING1,
478       strlen(PGRE_ID_LEAD_STRING1)) != 0) &&
479       (os::strncmp((char *)sector->pgres_id, PGRE_ID_LEAD_STRING2,
480       strlen(PGRE_ID_LEAD_STRING2)) != 0)) {
481    CMM_TRACE(("quorum_scsi_sector_read: pgre id "
482        "mismatch. The sector id is %s.\n",
483        sector->pgres_id));
484    kmem_free(ucmd.uscsi_rqbuf, (size_t)SENSE_LENGTH);
485    return (EINVAL);
486   }
487
488  }
489  kmem_free(ucmd.uscsi_rqbuf, (size_t)SENSE_LENGTH);
490
491  return (error);
492 }

With a simple DTrace script I could verify if the quorum_scsi_sector_read() does indeed return with 22 and also I could print what else is going on within the function:

56  -> __1cXquorum_scsi_sector_read6FpnFvnode_LpnLpgre_sector_b_i_ 6308555744942019 enter
56    -> __1cZquorum_ioctl_with_retries6FpnFvnode_ilpi_i_ 6308555744957176 enter
56    <- __1cZquorum_ioctl_with_retries6FpnFvnode_ilpi_i_ 6308555745089857 rc: 0 
56    -> __1cNdbg_print_bufIdbprintf6MpcE_v_ 6308555745108310 enter
56      -> __1cNdbg_print_bufLdbprintf_va6Mbpcrpv_v_ 6308555745120941 enter
56        -> __1cCosHsprintf6FpcpkcE_v_      6308555745134231 enter
56        <- __1cCosHsprintf6FpcpkcE_v_      6308555745148729 rc: 2890607504684 
56      <- __1cNdbg_print_bufLdbprintf_va6Mbpcrpv_v_ 6308555745162898 rc: 1886718112 
56    <- __1cNdbg_print_bufIdbprintf6MpcE_v_ 6308555745175529 rc: 1886718112 
56  <- __1cXquorum_scsi_sector_read6FpnFvnode_LpnLpgre_sector_b_i_ 6308555745188599 rc: 22   

From the above output we know that the quorum_ioctl_with_retries() returns with 0 so it must be a checksum mismatch! As CMM_TRACE() is being called above and there are only three of them in the code lets check with DTrace which one it is:

21  -> __1cNdbg_print_bufIdbprintf6MpcE_v_   6309628794339298 quorum_scsi_sector_read: checksum mismatch.

So now I knew exactly what part of the code is casing the quorum device to be marked offline. The issue might have been caused by many things like: a bug in a disk array firmware, a problem on an SAN, a bug in a HBA's firmware, a bug in a qlc driver or a bug in SC software, or... However because the issue suggests a data corruption and we are loading the cluster with a copy of a database we might have a bigger issue that just an offline quorum device. The configuration is a such that we are using ZFS to mirror between two disks arrays. We have been restoring a couple of TBs of data into and we haven't read almost anything back. Thankfully it is ZFS so we might force a re-check off all data in the pool and I did. ZFS found 14 corrupted blocks and even identified which file is affected. The interesting thing here is that for all blocks both copies on both sides of the mirror were affected. This almost eliminates a possibility of a firmware problem on disk arrays and suggest that the issue was caused by something misbehaving on the host itself. There is still a possibility of an issue on SAN as well. It is very unlikely to be a bug in ZFS as the corruption affected reservation keys as well which has basically nothing to do with ZFS at all. Then we are still writing more and more data into the pool and I'm repeating scrubs and I'm not getting any new corrupted blocks nor quorum is misbehaving (I fixed it by temporarily adding another one, removing the original and re-adding it again while removing the temporary one).

While I still have to find what caused the data corruption the most important thing here is ZFS. Just think about it - what would happen if we were running on any other file system like: UFS, VxFS, ext3, ext4, JFS, XFS, ... Well, almost anything could have happened with them like some data of could be corrupted, some files lost, system could crash, fsck could be forced to run for many hours and still not being able to fix the filesystem and it definitely wouldn't be able to detect any data corruption withing files or everything would be running fine for days, months and then suddenly the system would panic, etc. when application would try to access the corrupted blocks for the first time. Thanks to ZFS what have actually happened? All corrupted blocks were identified, unfortunately both mirrored copies were affected so ZFS can't fix them but it did identified a single file which was affected by all these blocks. We can just remove the file which is only 2GB and restore it again. And all of these while the system was running and we haven't even stopped the restore or didn't have to start from the beginning. Most importantly there is no uncertainty about the state of the filesystem or data within it.

The other important conclusion is that DTrace is a sysadmin's best friend :)

The EC gives a green light

The European Commission clears Oracle's proposed acquisition of Sun Microsystems:

"The European Commission has approved under the EU Merger Regulation the proposed acquisition of US hardware and software vendor Sun Microsystems Inc. by Oracle Corporation, a US enterprise software company. After an in-depth examination, launched in September 2009 (see IP/09/1271 ), the Commission concluded that the transaction would not significantly impede effective competition in the European Economic Area (EEA) or any substantial part of it."

MySQL TOP

This blog entry has been updated:

• added NCQRS, NCQRS/s columns
• fixed issue with dtrace dropping variables if the script was running for extended time periods
• cosmetic changes re output

I need to observe MySQL load from time to time and DTrace is one of the tools to use. Usually I'm using one-liners or I come up with a short script. This time I thought it would be nice to write a script so other people like DBAs could use without having to understand how it actually works. The script prints basic statistics for each client connecting to a database. It gives a nice overview for all clients using a database.

CLIENT IP                 CONN   CONN/s     QRS   QRS/s    NCQRS  NCQRS/s   TIME   VTIME
10.10.10.35                 10       0       61       0       32       0       0       0
10.10.10.30                 17       0       73       0       73       0       0       0
10.10.10.100                52       0       90       0       90       0       0       0
xx-www-11.portal            92       0      249       0       48       0       0       0
xx-cms-1.portal             95       0     1795       5     1669       4      48      48
xx-www-9.portal            198       0      634       1      278       0       0       0
xx-www-13.portal           239       0      986       2      366       1       0       0
xx-www-3.portal            266       0     1028       2      455       1       1       0
xx-www-12.portal           266       0     1070       2      561       1       3       2
xx-www-5.portal            300       0     1431       3      593       1       2       2
xx-www-10.portal           333       0     1221       3      678       1       3       2
xx-www-6.portal            334       0      989       2      446       1       1       0
xx-www-8.portal            358       0     1271       3      497       1       1       0
xx-www-4.portal            395       1     1544       4      744       2       0       0
xx-www-2.portal            445       1     1729       4      764       2       3       2
xx-www-1.portal            962       2     3555       9     1670       4      22      21
xx-www-7.portal           1016       2     3107       8     1643       4     117     115
                        ======   =====   ======   =====   ======   =====   =====   =====
                          5378      14    20833      58    10607      29     207     199
Running for 359 seconds.

         CONN       total number of connections
         CONN/s     average number of connections per second
         QRS        total number of queries
         QRS/s      average number of queries per second
         NCQRS      total number of executed queries
         NCQRS/s    average number of executed queries per second
         TIME       total clock time in seconds for all queries
         VTIME      total CPU time in seconds for all queries

The NCQRS column represents the number of queries which were not served from the MySQL Query Cache while QRS represents all queries issued to MySQL (cached, non-cached or even non-valid queries). If values of VTIME are very close to values of TIME it means that queries are mostly CPU bound. On the other hand the bigger the difference between them the more time is spent on I/O. Another interesting thing to watch is how evenly load is coming from different clients especially in environments where clients are identical www servers behind load balancer and should be generating about the same traffic to a database.

All values are measured since the script was started. There might be some discrepancies with totals in the summary line - this is due to rounding errors. The script should work for MySQL versions 5.0.x, 5.1.x and perhaps for other versions as well. The script doesn't take into account connections made over a socket file - only tcp/ip connections.

The script requires PID of a mysql database as its first argument and a frequency at which output should be refreshed as a second argument, for example to monitor mysql instance with PID 12345 and refresh output every 10s:

       ./mysql_top.d 12345 10s

# cat mysql_top.d
#!/usr/sbin/dtrace -qCs

/*
Robert Milkowski
*/

#define CLIENTS self->client_ip == "10.10.10.11" ? "xx-www-1.portal" : \
    self->client_ip == "10.10.10.12" ? "xx-www-2.portal" : \
    self->client_ip == "10.10.10.13" ? "xx-www-3.portal" : \
    self->client_ip == "10.10.10.14" ? "xx-www-4.portal" : \
    self->client_ip == "10.10.10.15" ? "xx-www-5.portal" : \
    self->client_ip == "10.10.10.16" ? "xx-www-6.portal" : \
    self->client_ip == "10.10.10.17" ? "xx-www-7.portal" : \
    self->client_ip == "10.10.10.18" ? "xx-www-8.portal" : \
    self->client_ip == "10.10.10.19" ? "xx-www-9.portal" : \
    self->client_ip == "10.10.10.20" ? "xx-www-10.portal" : \
    self->client_ip == "10.10.10.21" ? "xx-www-11.portal" : \
    self->client_ip == "10.10.10.22" ? "xx-www-12.portal" : \
    self->client_ip == "10.10.10.23" ? "xx-www-13.portal" : \
    self->client_ip == "10.10.10.29" ? "xx-cms-1.portal" : \
          self->client_ip

#define LEGEND  "\n \
        CONN       total number of connections \n \
        CONN/s     average number of connections per second \n \
        QRS        total number of queries \n \
        QRS/s      average number of queries per second \n \
        NCQRS      total number of executed queries \n \
        NCQRS/s    average number of executed queries per second \n \
        TIME       total clock time in seconds for all queries \n \
        VTIME      total CPU time in seconds for all queries\n"

BEGIN
{
  start = timestamp;
  total_queries = 0;
  total_nc_queries = 0;
  total_conn = 0;
  total_time = 0;
  total_vtime = 0;
}

syscall::getpeername:entry
/ pid == $1 /
{
  self->in = 1;

  self->arg0 = arg0; /* int s */
  self->arg1 = arg1; /* struct sockaddr * */
  self->arg2 = arg2; /* size_t len */
}

syscall::getpeername:return
/ self->in /
{
  this->len = *(socklen_t *) copyin((uintptr_t)self->arg2, sizeof(socklen_t));
  this->socks = (struct sockaddr *) copyin((uintptr_t)self->arg1, this->len);
  this->hport = (uint_t)(this->socks->sa_data[0]);
  this->lport = (uint_t)(this->socks->sa_data[1]);
  this->hport <<= 8;   this->port = this->hport + this->lport;

  this->a1 = lltostr((uint_t)this->socks->sa_data[2]);
  this->a2 = lltostr((uint_t)this->socks->sa_data[3]);
  this->a3 = lltostr((uint_t)this->socks->sa_data[4]);
  this->a4 = lltostr((uint_t)this->socks->sa_data[5]);
  this->s1 = strjoin(this->a1, ".");
  this->s2 = strjoin(this->s1, this->a2);
  this->s1 = strjoin(this->s2, ".");
  this->s2 = strjoin(this->s1, this->a3);
  this->s1 = strjoin(this->s2, ".");
  self->client_ip = strjoin(this->s1, this->a4);

  @conn[CLIENTS] = count();
  @conn_ps[CLIENTS] = count();

  total_conn++;

  self->arg0 = 0;
  self->arg1 = 0;
  self->arg2 = 0;
}

pid$1::*close_connection*:entry
/ self->in /
{
  self->in = 0;
  self->client_ip = 0;
}

pid$1::*mysql_parse*:entry
/ self->in /
{
  self->t = timestamp;
  self->vt = vtimestamp;

  @query[CLIENTS] = count();
  @query_ps[CLIENTS] = count();

  total_queries++;
}

pid$1::*mysql_parse*:return
/ self->in /
{
  @time[CLIENTS] = sum(timestamp-self->t);
  @vtime[CLIENTS] = sum(vtimestamp-self->vt);

  total_time += (timestamp - self->t);
  total_vtime += (vtimestamp - self->vt);

  self->t = 0;
  self->vt = 0;
}

pid$1::*mysql_execute_command*:entry
/ self-> in /
{
  @nc_query[CLIENTS] = count();
  @nc_query_ps[CLIENTS] = count();

  total_nc_queries++;
}

tick-$2
{
  /* clear the screen and move cursor to top left corner */
  printf("\033[H\033[J");

  this->seconds = (timestamp - start) / 1000000000;

  normalize(@conn_ps, this->seconds);
  normalize(@query_ps, this->seconds);
  normalize(@nc_query_ps, this->seconds);
  normalize(@time, 1000000000);
  normalize(@vtime, 1000000000);

  printf("%-16s          %s   %s     %s   %s    %s  %s   %s   %s\n", \
    "CLIENT IP", "CONN", "CONN/s", "QRS", "QRS/s", "NCQRS", "NCQRS/s", "TIME", "VTIME");
  printa("%-16s        %@6d   %@5d   %@6d   %@5d   %@6d   %@5d   %@5d   %@5d\n", \
    @conn, @conn_ps, @query, @query_ps, @nc_query, @nc_query_ps, @time, @vtime);
  printf("%-16s        %s   %s   %s   %s   %s   %s   %s   %s\n", \
    "", "======", "=====", "======", "=====", "======", "=====", "=====", "=====");
  printf("%-16s        %6d   %5d   %6d   %5d   %6d   %5d   %5d   %5d\n", "", \
    total_conn, total_conn/this->seconds, total_queries, total_queries/this->seconds, \
    total_nc_queries, total_nc_queries/this->seconds, \
    total_time/1000000000, total_vtime/1000000000);

/*
  denormalize(@conn_ps);
  denormalize(@query_ps);
  denormalize(@nc_query_ps);
  denormalize(@total_time);
  denormalize(@total_vtime);
*/

  printf("Running for %d seconds.\n", this->seconds);

  printf(LEGEND);
}