[Pc_Support] 3Ware Escalade 8000, 9500S and 9550SX -- WAS: Drives, RAID, backups oh my!

[Bryan J. Smith]

Damien McKenna <dmckenna at thelimucompany.com> wrote:
> What's the real difference between the 8000 and 9000 series
> (64bit PCI varieties)?

I'm going to blog more detail (full tables -- think my GeForce 6/7
blog ;-).

But here's the lineage ...

Escalade 7x00/7x10 series (7200, 7400/7410, 7800/7810)
- 2/4/8-channel
- Ultra DMA Modes 5/100 (and 6/133 with firmware update)
- New 64-bit engine
- 64-bit PCI, 33MHz (except for 7200)
- 7410/7810 shorter versions of 7400/7800
- 1MiB Static RAM (SRAM, only needs capacitance to keep memory)

Escalade 7x50 series (7450, 7850)
- 4/8-channel
- 2MiB Static RAM (SRAM)

Escalade 7000/7500 series
- New 7500-12 channel option w/4MiB Static RAM (SRAM)
- 7200 = 7000-2 (PCI32)
- 7450 = 7500-4, 7850 = 7500-8 (PCI64)

Escalade 8500 series
- SATA, but engine matches 7000/7500 verbatim
- External SATA PHY used to PATA ASIC engine
- 2/4 channel "Low Profile" (LP)
- 8000-2LP (PCI64 -- unlike 7000-2)
- 8500-4LP, 8500-8, 8500-12 (PCI64)

Escalade 7006/7506 and 8006/8506
- Updated with 66MHz PCI64 support
- 7006-2 still PCI32, but now 66MHz

Escalade 9500S
- Adds 128MiB-1GiB of PC133 SDRAM to existing 64-bit ASIC engine
- Better buffer for RAID-5 operations, doubles RAID-5 performance
- Introduce Battery Back up for SDRAM (unlike SRAM, DRAM is "leaky"
and requires 1,000x power/refresh)

Escalade 9550SX
- Adds PowerPC400 microcontroller for RAID-5 off-load
- Doubles RAID-5 performance again over 9500S

> Does the 8000 series support hot-swapping?

Of course!  It's SATA with staggered connectors!

> With the 3Ware 8000 or 9000 cards, is it possible to:
> * Set up multiple RAID-1 sets on the same card,
> e.g. set up one set now and another as needs increase?

Yes!  _All_ cards do -- from 5000 to 9550SX.
You can even _span_ up to 4 cards with newer products.

There was a bug on the 9550S 9.2 firmware where if you had more than
1 RAID-5 array you'd get poor performance.  That was fixed in
9.2.1.1.

> * With a RAID-1 hot-swap one of the drives without the system going
> down, or do you have to take down the server to work on it?

Of course _not_!

This isn't FRAID (fake RAID) or even software RAID.  Software RAID --
_unless_ you use SCSI-2 SCA with disconnect _and_ "hot-plug" support
in Linux 2.6 or Windows NT "Dynamic Discs" (long story) -- has that
same problem.

The 3Ware completely _hides_ the existance of the disk from the
_hardware_ -- not just "trick BIOS/driver" in software.  As long as
your drive tray handles transient -- which is _inherent_ to SATA with
the 15-pin staggered power and data (not if you use Molex though) --
it just works.

> I'm thinking that if one drive fails of replacing it with
> another one, that kinda thing.

3Ware is extremely _well_trusted_ in this regard.

The people who have had problems with 3Ware hot-swap are the ones
using 3Ware with _software_ RAID.  Too many software RAID advocates
are told that 3Ware supports hot-swap -- it can _only_ support
hot-swap _if_ the array is 3Ware hardware RAID managed.  3WARE HAS
NEVER AND WILL NEVER SUPPORT HOT-SWAP FOR "SOFTWARE RAID" -- Smack
anyone who tells you otherwise, they are giving 3Ware a "bad name"
with THEIR IGNORANCE.

Otherwise there are _limited_ATA_standards_ on hot-swap.  The 3Ware
card _might_ use the SCSI subsystem for its "generic block driver"
but it is _not_ provide emulation of an advanced SCSI-2 SCA SAF-TE
chassis (that costs a lot of money).  If you use software RAID you
should _only_ use SCSI-2 with a host adapter that supports SAF-TE,
has full SAF-TE driver for your OS _and_ a SAF-TE chasis with SCA.

Until AHCI and SATA SAF-TE is well supported in the ATA standards,
software RAID is basically _not_reliable_ for hot-swap.  Only
solutions like 3Ware, Areca, etc... where _all_ operations are
handled _in_hardware_ on the on-board ASIC/microcontroller "hides"
such transient from the OS.

Hence why 3Ware and, increasingly, Areca are well trusted.

> What's needed to make SATA hot-swappable?

Physically/electrically, SATA is SCSI SCA.
That means it has staggered data/power.
If you use a Molex power, that negates the power.
But if you use the 15-pin SATA power -- and that means a _true_ SATA
+3.3/+5/+12V (not a Molex converter), it handles transient.

You must then have hardware that can handle a disconnect.
This is _not_ very standardized in ATA, much less the controllers and
their OS drivers.

You must then have the OS handle the disconnect as well.

In Linux 2.6, this is "hot-plug."
In NT, it varies, but NT5+ Dynamic Discs are recommended (long
story).

With a 3Ware or Areca card, the system _never_ sees the drives
_physically_.  That means you can't have an issue with the system/OS
because it thinks the volume is there.  There is _no_ disconnect. 
The on-board ASIC/controller handles _all_ the transient operation --
including drive the ATA controllers.

There's nothing like an ATA controller, an on-board intelligence and
its OS/firmware _all_ from the same company.  I.e., expect *0*
inter-compatibility issues.  That's the _bane_ of ATA -- lack of
following standards between controller + chipset + OS driver.  Gone,
nada, zip with _true_ hardware ATA -- especially 3Ware that uses its
own ATA channels (with flexible FPGA logic).

FRAID hides the drives in software, but the OS/system sees them. 
Software RAID also seems the drives.  In any case, it's up to the OS
to handle the "loss" of a device.  Your "MD" device might be fine --
but the OS can -- and often does PANIC -- because it hangs on the
loss.  Linux 2.6 hot-plug removes this, when properly configured
(with good end-to-end SCSI hardware).  NT5+ with dynamic disks also
help.

> Do you need anything on the inside or does the tray have
> all that's needed?

Again, staggered 15-pin power + 7-pin data makes SATA removals
transient, just like 80-pin SCSI SCA.

> Is there a difference between different trays or would $20-25
> ones at NewEgg be ok?

This isn't PATA.  I guess there _could_ be quality differences
between SATA backplanes.  Or if you go too "cheap" you might get a
tray that does not have a backplane, just "exposes" the SATA
power/data connectors.

> DLT1 - 40/80gb.

Yeah, that's probably not worth upgrading.

If you get new tape, go LTO-1 at least.

> It sure seems a good plan, though I'd prefer to go with something
> that pushes further than our current usage so we won't fill it up
> too quickly.

LTO-1 is already 2.5x more, and likely 10x faster.
If you're going to fork out for LTO-2, might as well go LTO-3.

> One reason for another DL380, besides the fact they're affordable,
> is the six hot-swappable bays.  The only question is whether they
> can be changed/replaced to work with SATA?

Yes, and you'll pay HP an arm and a leg.
Although you can always call them to find out.

> I looked at them and the hot-swap machine started at just under
> $1200 with a single-core CPU, that's without a RAID card and only
> one drive, adding those brings it to over $1700.

$400+ is typical for a true hardware RAID card.

My main point was that HP charges you an arm'n a leg for their
proprietary hot-swap/form-factor.

Personally, I'm now very disappointed with the DL360 after it took
some weight on just the plastic front and it actually was _not_
designed to load it on the metal -- like it should have been.  It not
only damaged the front-board, but it damaged the separate mainboard
12" down.  That's a crappy design -- you should _never_ mount a board
to plastic, but the metal base.

> How much are HP's SATA bays?

I don't know.  You'll need to call them.  It will vary.  I have 1U
DL360s.  I save money by not going with the U320 backplane option.

Don't know for a 2U DL380.

> Yep, they all have dual 100mbit NICs and setting up a second
> network for the servers was going to be part of my plan.

Ouch, no dual GbEs?

> Tempting.  $100 less would give us 2x160gb, which is still loads.

Plan for the future, as well as cross-sync.  What's $100 in your
time?

> The current "RAID" controller is just a SCSI adapter with
> software-based crRpAID.

Are you sure?  They were i960s the last time I checked.  Old, slow,
but still hardware RAID.

> The way I see it is that we're stuck in the middle.  I don't want
> us to upgrade to something I think we're close to filling.

Why?  You get LTO-1 now, which is already a 2.5x increase in
capacity, 10x increase in speed.  Then you upgrade to LTO-3 in a year
or two when you need more -- recycling your LTO-1 to other duties.

LTO-3 can read LTO-1.

> Then again, if I separate the backups into permanent (things that
> never change e.g. conference pictures) vs transient (ever-changing
> documents) and have two backups per routine then it'd work out.

Here's the deal on backup ...

At some point the cost in management and maintainence of wasting time
is _more_ than the cost of a 3-figure drive.  Think about that.

That's how I've regularly and successfully argued for companies to go
LTO -- or LTO-3 when they were going to buy a LTO-1 or LTO-2 --
sometimes with an autoloader no less!

> Yeah, I know, I got bitten that way already.  Need a DLT4000 drive
> that can be easily repaired? ;-)

Again, time is money.  A new LTO-1 drive is 3-figures and has *0*
strings attached, other than some $25 cartridges.

> Nice, but our servers are all 1U or 3U.

Huh?

3U = (3)1.75", 1.75" being the height of a half-height 5.25" drive.
If you have 3U enclosures, you should be able to put one of those
$125 3x5.25" drive enclosures in it and fit five (5) 3.5" drives.

But I seriously doubt the HP DLs take a stock enclosure like that.
But maybe they do?



-- 
Bryan J. Smith   Professional, Technical Annoyance
b.j.smith at ieee.org    http://thebs413.blogspot.com
--------------------------------------------------
I'm a Democrat.  No wait, I'm a Republican.  Hmm,
it seems I'm just whatever someone disagrees with.