[Pc_Support] Re: RAID options for SATA

Fri Jul 15 11:50:05 EDT 2005

From: Damien McKenna <dmckenna at thelimucompany.com>
> I see your point, onboard RAID-5 isn't very good.

<anal>
There is *0* "onboard RAID-5."
</anal>

Understand I said that using your main CPU and its interconnect
for "Programmed I/O" (PIO) RAID-5 writes is less-than-ideal.  ;->

> Too bad there aren't any simple cards available that can offload
> the XOR creation without having to have a full device controller too.

Well, think about it.  RAID-5 typically involved massive buffering,
recommended 16MiB DRAM _minimum_.  That's to hold dozens of
32KiB (typical) block sequences for processing.

About the only company I've seen solve it, for desktops and non-
random access server applications, is NetCell with their RAID-XL.
Instead of sequential blocks, it uses a parallel 32-bit or 64-bit
wide bus for _exactly_ 2 drive + 1 parity or 4 drive +1 parity,
respectively, to/from disks.  In that case, you can do XOR in
"real-time" with less need for buffering.

> I've got two high-powered S-ATA controllers on my motherboard,

"High-powered" SATA?  _All_ ATA is just a "dumb bus arbitrator."
It's quite unintelligent, with little more than basic 16-bit BIOS
Int13h disk services for setup/boot.  It's driven 100% by the
integrated drive electronics (IDE) on the drive that knows little
more than how to transfer 32KB blocks to/from system memory
via Direct Memory Access (DMA).

Newer ATA has Native Command Queuing (NCQ) allows the IDE
to queue up operations at the drive, taking far better advantage
of that 2-16MB of DRAM on-board.  All a NCQ ATA controller does
it setup the registers to do it properly, then gets the hell-out-of-
the-way.  Not intelligent at all, and you're still only queuing the
memory-to _specific_ disk transfers, not your entire, RAID
transfer.  That's the big difference between plain'ole ATA and
someone that puts some "intelligence" in front of the ATA.

[ SIDE NOTE:  It's also why Serial Attached SCSI (SAS) is being
introduced.  SAS controllers and SAS drive bays can actually take
SATA drives, as well as SAS drives.  But SAS offers the full,
intelligent SCSI command set atop, from an end-software and
even firmware/middleware standpoint. ]

The only thing that you might be referring to is the fact that
your SATA channels are on a dedicated PCIe channel.  That at
least means it doesn't have to contend with anything else
on its own PCI bus for memory access.

> buying a third would seem like such a waste.

Well, what are you buying?

Your on-board ATA channels cost about $0.10/channel in bus
arbitrator logic.  The cost of the PCB, its traces and pin-outs that
feed it cost far more.

An intelligent ATA controller costs about $30-60 in ASIC/
microcontroller logic, plus another $10-30 for the DRAM, maybe
another $5-30 for SRAM (if not integrated into the ASIC/uC).
You're paying for that "intelligence," not the ATA channels.

From: Damien McKenna <dmckenna at thelimucompany.com>
> Then again, this looks pretty nice:
> http://www.lycom.com.tw/PE105N.htm
> Too bad Google doesn't turn up any sellers.
> So what's a cheap RAID-5 PCI or PCI-e card that does XOR?

I can't tell what it does, but it looks like FRAID.
Because RAID-0, 1 and 0+1 doesn't require any buffering,
and only 32KiB to real-time cache each commit, there are
a few ICs now coming out that compete well with older
3Ware products (circa 5000/6000 series timeframe).
E.g., the IT8212:  
http://lists.centos.org/pipermail/centos/2005-June/007451.html

Until they can fit ~128MB of DRAM into a system-on-a-chip
(SoC) IC for $1, you will _never_ see a "cheap" RAID-5
controller.  At this point, the best I've seen is NetCell's
RAID-XL approach (parallel stripes+parity, instead of 
sequential blocks of stripes+parity), although they still
ship with 128MiB of DRAM outside of its ASIC (for performance
when reading).  The cards cost $150-250.

From: Jason Boxman <jasonb at edseek.com>
> Yeah cheap is definitely the operative word.  You will not find any
> 'cheap'  used 3Ware 8xxx cards on Ebay yet.  I just resold my old
> 7410 in February for 75% of what I paid for it a year earlier.  They
> seem to maintain their value.

The Escalade 5000/6000 use the same ASIC design.
The Escalade 7000/8000 use the same ASIC design.
The Escalade 9000 seems to be an 8000 with a DRAM controller added.
It wouldn't surprise me if there is an Escalade 10000 series in
development that:
A) improves on the ASIC for for split SRAM/DRAM, and
B) adds PCIe bus arbitration to PCI (via new PCI-X support)

Broadcom has produced the first, true 8-channel SATA ASIC for
RAID-5 that supports both PCIe and PCI-X (and can even bridge
between teh two) and up to 768MiB of external DRAM.  It is
designed more for embedded, but you should see PCIe/PCI-X
cards based on it for under $500.  The IC sells for $60 in lots
of 10K, so maybe we'll see it integrated on $400+ server\
mainboards in the near future as well.

From: Jason Boxman <jasonb at edseek.com>
> I recently upgraded to a 7450-4LP.  I like the R5 Fusion stuff.
> I can finally sustain around 40MB/s writes over the 7410's 20MB/s
> or so.  I've found that's more than fast enough for myself as a single
> user.  I guess the real test would be if I had multiple people hitting
> my box locally for media content.  One day, perhaps... ;)

Yes, the small improvements on the 7x50 aka 7500/7506 series,
from the prior 7x00/7x10 series in the ASIC and doubling of the
SRAM (from 1MB to 2MB, as well as a new 4MB in the 12-channels)
did a lot to improve RAID-5 performance.

And the 3Ware 7000/8000 series clearly does well with RAID-5
sustained writes as long as they are sequential bursts.  If they
are random access, the seeks introduce latencies, and the 3Ware
7000/8000 series only has a small amount of 0 wait state SRAM,
_no_ buffering DRAM.  For RAID-5, direct switching with no latency
is typically not a consideration, because you're almost _always_
buffering for XOR operations and the actual commit.

Hence the 3Ware 9000 series which adds DRAM.  I've heard of a
lot of issues with it in the past, but most of the bugs seem to
be worked out in the latest drivers according to most (that's
a 2nd hand testamonial).

--
Bryan J. Smith   mailto:b.j.smith at ieee.org