Commentary about IP Storage Directions posted by Richard McDougall - really interesting post.
I can only agree with Richard - NFS became worth solution in data centers, mainly because it's cheaper, easier and more flexible. Probably the same will be with iSCSI - the problem with it, as I see it now, is that it's not mature technology (yet) - it's not well tested in many different environments.
In our data center we were skeptical about NFS few years ago - we did some testing with Linux but it wasn't working well - too much kernel panics, etc. Then, some time later, we did try again - this time with Solaris. Well, it's working really well - right now we use NFS with Solaris with many TBs of data. We do use Linux with NFS too - it's working acceptable these days. The problem with Linux NFS client (both 2.4 and 2.6 kernels) is that if NFS server goes somewhat 'crazy' then Linux simply panics while Solaris with the same NFS server (which is not Linux and not Solaris) keeps running. We had to get used to daily Linux panics. Once the problem with NFS server was resolved (unfortunately it took many months) Linux clients started to work properly. On the other hand Linux NFS client with Linux NFS server works quite well these days - I mean stable. Solaris NFS servers just work. The biggest problem with Linux is observability - you can almost say there's none.
Is SAN dead? Definitely not now, and probably not in a next 5-10 years. Why? Because people know it - how to setup HA, tune it, secure etc. And it's very well tested technology. I think that iSCSI will be mostly used in low-end at the beginning, then will steadily go into mid-range and high-ends. What we see now is a combination of SAN and iSCSI. And people won't throw away their current SAN hardware for a long time. But it looks like IP Storage has a bright future and we will see many deployments in low-end in coming years (months?).
ps. What I'm gonna test in coming days is a combination of NFS server with ZFS in a SAN with MPxIO and IPMP :) I'm sure that's gonna be interesting!