[Pc_Support] Re: RAID options for SATA

Fri Jul 15 03:31:34 EDT 2005

From: Damien McKenna <damien at mc-kenna.com>
> The onboard SATA controller on my motherboard, or rather one of them, 
> supports RAID 0, 1, 10 and 5 using a Silicon Image 3114 chip and I'm 
> brainstorming my options for setting up a RAID later on this year.

FRAID (Fake/Free RAID)
It is _always_slower_ than OS-integrated software RAID.

> I read a comparison recently of the SI3114 versus two hardware-based 
> RAID cards in RAID-5 usage and the SI chip came out very well for most 
> operations, in fact it had the consistently good scores for reads & CPU 
> usage but was slow on writes.

Of course!
In RAID-5 reads, it's basically just a read RAID-0 operation (minues 1 stripe).
So you're just directly streaming data directly off of disks.
No duplicate data stream.

That's why I actually recommend _software_ for RAID-0.
E.g., two hardware RAID volumes spread over two PCI-X channels.
It adds *0* overhead, but segments off and essentially doubles the
amount of traffic that can be put to your I/O.
That's for when you really need performance!  ;->

Now once you use s/w RAID-1 writes, you start duplicating data that
goes over your interconnect, but at least you don't bother CPU.
Now do s/w RAID-5 _writes_, you're_killing_ your CPU-memory-I/O
interconnect because every single piece of data written must be
pushed to your CPU just to do the XOR.
The XOR itself doesn't bother your CPU much at all, but the stream
of data that must be pushed up to the CPU is the problem.
You won't see it in CPU usage, but you will notice it if you need to
move lots of other data, because it's contending for the interconnect.

Which is why when it comes to RAID-1 and, better yet, RAID-5 writes,
you should just send your stream to your disk controller, and let it
do any duplication local/direct or, in the case of RAID-5, handle the
XOR operation (with a good size buffer cache).
Software RAID-5, in a nutshell, turns _all_ your writes into
"Programmed I/O" (PIO) through the CPU instead of being "Direct
Memory Access" (DMA) with_out_ bothering your CPU interconnect.

Some new, intelligent RAID cards can do XORs in real-time, or
near-real time.
The 3Ware Escalade 9500S series is now reported to be very
improved in reliability, and has a large DRAM buffer for RAID-5.
I also like the NetCell SR3000 (32-bit) and SR5000 (64-bit) and
it's "RAID-XL" (parallel write RAID, not 32KB block stripes) for
maximum _desktop_ (not heavily queued) performance, but its
Linux drivers are not well proven yet (even though they use the
stock ATA).

> My primary goals are:
> 1. Reliability for when problems arise, i.e. one drive dying not
> taking out the entire RAID.

Either use an intelligent hardware RAID card, or use a disk label
that preserves the OS' meta-data for software RAID.

Always use the OS' integrated s/w RAID before a FRAID card.
FRAID cards are always trouble.

> 2. Ease of use for rebuilding should #1 happen.

Definitely _not_ Linux MD.
It is conservative so you typically have to manually tell it to
rebuild -- i.e., it isn't always sure you've replaced a faulty disk.

> 3. Speed

Speed in what regard?
On a single-user system, s/w RAID-0 is just as fast, sometimes
faster if .
RAID-1 reads and RAID-5 reads can also be fast in s/w.
But once you start doing lots of RAID-1 or RAID-5 writes,
then things change.

> 4. Capacity

Well, that's up to you.
A lot of people do RAID-5 for capacity.
I happen to disagree with the idea.
On a desktop or media server, I'd use more RAID-XL (NetCell)
than RAID-5.
On a server with a good amount of writes, especially lots of
random writes, intelligent hardware RAID-10 is ideal.

> This would primarily be for use with the OS and important data,
> I'll keep a large-ish ATA drive for misc junk / mp3s.  I was also
> thinking of getting four equally sized disks, they're pretty cheap
> and if it gave extra reliability it'd be worth it.

Might even go for a 5th spare.
You can even get a 3U internal bay with 5 SATA hot-swap bays
for around $150, and leave your spare "ready-to-swap" right in
the chassis.

I now typically do this whenever I purchase a 3Ware Escalade
8506-4LP (or, now that people trust them more, 9500S-4LP).
I'm already forking just under $300 out for the card, so I might
as well pay another $150 for the enclosure and make swapping
easy.  SATA is built for hot-swap.

> As I see it four discs would open up two main options:
> * RAID-10
> * RAID-5
> Has anyone used RAID-5 with four discs, if so how well did it
> work?

I'm using a 3Ware Escalade 7800 (8-channel 7000 series, 1MB
SRAM, instead of the standard 2MB in the 7850/7500-8/7506-8),
with (2) RAID-1 system and (6) RAID-5 data.
As long as I'm not killing the measly 1MB SRAM with lots of RAID-5
writes, I'm fairly good.

The 3Ware 7000/8000 series is better at RAID-5 writes than FRAID
or even OS LVM, but it's measly 1-4MB SRAM is a far cry from using
a 3Ware 9500S with 128+MB DRAM added, or an XScale-based
LSI Logic card.
It's really not that much slower at RAID-5 writes than an old i960
33MHz or 66MHz microcontroller-based Promise SuperTrak or
Adaptec 2400A/2800A, or any old i960-based SCSI RAID.

IN A NUTSHELL, unless you want to fork out $500+ for an XScale
SATA or SCSI RAID card, you will _not_ get RAID-5 writes that
perform in excess of 50MBps.
I guess software RAID-5 might be an option that could, if you use
a quality system design (PCI-X channel with BroadCom RAIDCore
PCI-X 4-8 [S]ATA channels), but I don't like the burden it puts
on the CPU interconnect.

--
Bryan J. Smith   mailto:b.j.smith at ieee.org