Figure 1． Multi-peer System Topology Using PCIe as the System Interconnect

Root Processor

A standard x86-based PC is used as the RP. The RP uses and AMD Athlon64 CPU with the nVidia nForce4 SLI chipset to support the PCIe interface. One PCIe slot is used to connect the multi-port PCIe switch. The system block diagram of the RP is shown in Figure 2.

Figure 2.  RP System Block Diagram

PCIe Switch

The IDT89EBHPES64H16 evaluation board (referred to hereafter as EB64H16) is used as the multi-port PCIe switch module. The system block diagram of an EB64H16 is shown in Figure 3. There is an IDtT 89HPES64H16 PCIe switch (referred to hereafter as PES64H16) on the evaluation board. There are 16 PCIe connectors on the EB64H16 board. A port may be configured as either a x4 or x8 port. When all the ports are configured as x8, only 78 of the PCIe connectors are used to support the 8 ports in x8 configuration. The upstream port is connected to the RP via two x4 infiniband cables. The RP is plugged directly into the PCIe connector.

Figure 3.  Multi-port PCIe Switch Module

The Endpoint Processor

The x86-based EP Processor is an AMD Athlon64 CPU with the nVidia nForce4 SLI chipset to support the PCIe interface. Each x86-based PC connects to one down-stream port of the Multi-port PCIe switch via the Non-transparent Bridge (NTB) port of the IDT PES24NT3 PCIe inter-domain switch. The EP Processor system is shown in Figure 4. Note that the internal endpoint of the NTB is connected to Endpoint Processor while the external endpoints are connected to Multi-port PCIe switch mentioned above.

Figure 4  EP Processor Block Diagram

The software architecture to support PCIe System Interconnect has been presented in this document. This software has been implemented and is working under Linux with the x86 CPU as the Root Processor and the x86 CPU with the IDT PES24NT3 Inter-Domain Switch as the Endpoint Processors. Software source code is available from IDT.

The software is implemented as device drivers and modules running in the Linux Kernel space. There are three layers in the software to separate the different software functions and to allow maximum reuse of the software. The Function Service Layer is the upper layer. It provides the function service that is visible to the Operation System and upper layer application. Multiple function services have been implemented in the current release of the software: the Ethernet Function Service provides a virtual Ethernet interface to the system, the Raw Data Function Service provides transfer of user data between EPs and RPm and the Statistic Function Service provides the function to collect traffic statistics for management and diagnostic purposes. 

The Message Frame Layer contains the Message Frame Service which provides a common message encapsulation and de-capsulation layer to all the function services. It also notifies all other Endpoint Processors whenever a new Endpoint Processors is discovered. The Transport Service layer deals with the actual data transport between Endpoint Processors and Root Processors using the PCIe interface. The Transport service is Endpoint Processor specific. This version of the System Interconnect software supports x86-based Root Processor and Endpoint Processors.

Apart from the inter-processor communication application, this software demonstrates that I/O sharing can now be implemented using a standard PCIe switch. The sharing of a single Ethernet interface by multiple Endpoint Processors and the Root Processor has been implemented and functions properly.

The address translation unit is used to isolate and provide a bridge between different PCIe address domains. The free Q and post Q structures are used as part of the message transport protocol.

This software release lays down the foundation to build more complex systems using the PCIe interface as the System Interconnect. The software follows a modular design which allows the addition of function services and other Endpoint Processors support without making changes to existing software modules. Complex systems, such as embedded computing, blade servers supporting I/O sharing, and communication and storage systems, can be built today using PCIe as the System Interconnect.

For more info about PCI Express and optical fiber communication, please browse our website or contact a Sopto representative by calling 86-755-36946668, or by sending an email to info@sopto.com.