[Pc_Support] Re: Nforce kernel info

[Bryan J. Smith]

On Wed, 2005-03-16 at 23:05 -0800, coreyfro at coreyfro.com wrote:
> It was regarding RAID 5 and caching.
> That is where I heard about their boards sucking.

Of course, until the 9000 series.

The way 3Ware gets non-blocking I/O is by using SRAM (static RAM).  SRAM
logic is what is used in CPU cache, as well as for ASICs where 0 wait
state is required.  DRAM (dynamic RAM), even SDRAM (synchronous DRAM),
still has latencies of 40-60ns+ (16-25MHz -- yes, _16-25_ MHz) for most
read operations.  Hence why 3Ware kills at RAID-0, 1 and 0+1.  

Unfortunately, SRAM is very large and complex compared to DRAM.  Hence
why 3Ware 7000 and 8000 series cards only have 1-4MB of SRAM.  That
easily overflows with lots of random RAID-5 writes.  Of course, 3Ware
7000/8000 series at RAID-0+1 trashes every RAID-5 card out there -- ATA,
SATA or SCSI -- for servers.  If you can afford the storage
inefficiency, there is nothing _faster_.

Now the new 3Ware 9000 series adds 128MB-1GB SDRAM, removing the RAID-5
buffer overflow issue.  For RAID-5, you don't need non-blocking I/O, you
just need buffer because writes are _rarely_ sync with the burst from
the system (unlike RAID-0, 1, 0+1).  RAID-5 isn't typically worth it
unless you have a lot of reads.

And even then, RAID-3 (desktop) or RAID-4 (server) is argued to be
better by many (including myself, NetApp, NetCell, etc...).

> For mirroring, this is true during a write, but during a read, it is
> nearly identicle to single drive performence.  I don't have a high write
> requirement, and what I would have will be highly suplimented by disk
> cache in system RAM.  In fact, most of might writes would probably never
> make it to disk if I cached aggresively...  Things like compiles or heavy
> VM trashing might only ever write-to-disk 1 out of 10 time.  Because I'll
> need gobs of system RAM for VM's anyway, I think that is where I'll spend
> the $500 bucks I would have otherwise spent on a RAID controller... RAM is
> a more versitile resource for me.

Well, as I said, for $125, you can get a 2-channel 3Ware.  It
interleaves reads just as good as MD does -- but _queues_ in _hardware_
-- all while it removes the write inefficiency.

People are talking about NCQ as the greatest thing since sliced bread,
but it's only helpful when you don't already have an "intelligence" on-
board.  With a 3Ware card, you have "intelligence," so NCQ is
unnecessary.  The 3Ware card roasts most other ATA solutions which don't
have intelligence -- let alone hardware-based using an ASIC+SRAM
solution.

> I have a K7 700 with 6 200GB ATA disks spread across VIA's K7T400's
> onboard controller and two Promise ATA/100 cards,

Ouch, you're shared PCI bus is a major bottleneck there.

> all in one RAID 5, and yes, it fails as often as it sounds like it would...
> headaches are a hobbie of mine.  How many people can say that
> rebuilding a RAID 5 from a two disk failure is a trivial task?

Well, not on production servers IMHO.  ;->

> Also, if I understand correctly, the nVida SATA channels AREN'T PCI
> devices or sharing the PCI BUS.  If this is the case it would seam that
> avoiding a PCI solution would be faster.

Depends on which controllers.  The 4-channel SATA is _should_ to be on a
dedicated 0.25GBps PCIe x1 bus in the nForce4.  The additional 4-
channel, off-chipset SATA is _not_.

> RAID-0 isn't a _R_AID.  I know that is obvious but that is a fundimental
> truth and why I avoid it at all costs.  Besides, I want fast seaks, not
> big throughput.  I am not encoding video, I am planning on running 20 VM's
> on one box.

Which brings back the suggestion of getting an intelligent storage
controller that can queue up I/O requests and handle them efficiently,
interleave reads all while protecting data integrity with RAID-1.  At
$125, it's a small drop in the bucket -- especially with 3Ware's Linux
monitoring tools and other _standard_ Linux software (including GPL
drivers in the stock kernel).

> That bodes well for my super hammer idea.

Yes, it's very ideal.


-- 
Bryan J. Smith                                  b.j.smith at ieee.org 
---------------------------------------------------------------- 
Community software is all about choice, choice of technology.
Unfortunately, too many Linux advocates port over the so-called
"choice" from the commercial software world, brand name marketing.
The result is false assumptions, failure to focus on the real
technical similarities, but loyalty to blind vendor alignments.