[Pc_Support] Bus Bottleneck Crash, No Bandwidth Left?

[Bryan J. Smith]

Jason Boxman <jasonb at edseek.com> wrote:
> With all the talk of bus bottlenecking, I'm curious what exactly
> happens when you hit a bottleneck in the PCI bus, be it legacy
> PCI or PCIe? 

Have you ever gotten "choppy audio" in the old days of ISA?

Or today with cheap PCI audio codecs that need lots of bus access --
especially for 3D spatial?

You've experienced bus saturation.  Especially when raw block I/O,
like disk or (to a lesser extent) network, fights host
software-driven DSP operations like audio, communications, etc...

> Does it simply limit your peak performance or do you actually
> get worse performance when bottlenecking than if you peaked
> at 90% or whatever?  

Depends on your use of the term "peak performance"?

If you're thinking of terms of networking, where you have CSMA/CD --
carrier sense, multiple access, collision avoidance -- aka
"Ethernet," no.  You typically will _not_ get "diminished returns"
like you will due to collisions or lack of flow control.  The shared
PCI peripheral bus is far more "controlled."  You have priorities,
interrupts, etc...

But you can still saturate the bus so both latency and throughput are
limited.

> Should bottlenecking be avoided at all costs, or is it non fatal
> if you can live with the bus being your limiting factor?

Depends.  You should typically try to segment off disk and, to a
lesser extent, network operations from the legacy PCI32 at 33 bus.  PCIe
does this with far, far less traces than another PCI bus like PCI64
or PCI-X.

> A related question:  Can a device be sending on the PCI bus while
> another card receives or is the bus simple utilized or not utilized
> at any given moment?  

Depends on what you are asking.

No, whatever device controls the PCI bus for its access (such as a
Direct Memory Access transfer to system memory), it holds the bus. 
It _could_ possibly deny other devices access to the bus --
especially during a prolonged burst.

Yes, the bus _is_ "there" whether it's being used or not.  The
maximum data transfer rate (DTR) is _only_ if the bus is being used
_continually_ -- not including overhead.  Furthermore, if a transfer
is not buffered, just because you bridge one bus through a higher
speed bus does _not_ mean it's not "wasting" the other busses' time.

This is the #1 reason why Opteron kicks the living crap out of Xeon. 
If you have some low-DTR PCI DMA process going, it can tie up your
_entire_ Xeon's MCH to memory.  With multiple HyperTransport links
and localized memory, and proper processor affinity for I/O support
in the OS, AMD will only use the path the transfer takes. 
Furthermore, the HyperTransport is a _true_ system interconnect --
not a peripheral interconnect, and bursts/buffers as appropriate at
the bridges as capable/designed.

But no, again, it's not CSMA/CD Ethernet.  You're not going to have
"collisions" or "overflows" -- at least not in the Ethernet sense.

> Put another way, since PCIe is often compared to a layer two
> switch,

Huh?  That's an Intel over-simplification.

PCIe allows simple segementation of PCI busses at a very low trace
count -- something that bridged PCI64/PCI-X does not.

PCIe is actually a full, 4-layer stack.

> is the legacy PCI bus like a full duplex or half duplex HUB
> or something else?

It's a full duplex channel, whereas PCI/PCI-X is half duplex.

But unlike Intel's marketing, there _are_ "switched" PCI/PCI-X
designs.  They're called HyperTransport tunnels/bridges.  In fact,
HyperTransport PCIe tunnels are more "true switched" PCIe designs
than Intel's new MCH/ICH which use PCIe channels as a poor-man's
system interconnect.



-- 
Bryan J. Smith     Professional, Technical Annoyance
b.j.smith at ieee.org      http://thebs413.blogspot.com
----------------------------------------------------
*** Speed doesn't kill, difference in speed does ***