PCIe Hot-Swap Device Driver Part 1

In typical PCIe based systems, PCIe busses are enumerated and resources allocated to each PCIe endpoint device during system initialization. Due to limitations in the enumeration and resource allocation algorithms, once initialized, the PCIe topology is fixed, meaning no new endpoint devices or switches may be connected to the system.

Figure 1  Typical PCIe Topology

The IDT PCIe Hot-Swap driver is a loadable Linux module that employs IDT proprietary enumeration, resource allocation, and device detection algorithms, allowing PCIe endpoint devices and switches to be connected to or disconnected from a system at runtime without compromising the operational state of the other PCIe devices in system. Using Figure 1 as a reference, if the dual GigE controller is disconnected from the system and subsequently reconnected to a port on a different switch, the operational state of the Fibre Channel controller and the SATA/SAS controller will remain unchanged. 

This differs from the Windows Vista implementation of the PCIe Hot Plug which must disable, re-enumerate, and reallocate resources to all devices sharing the same root port before the newly connected device becomes operational. Devices may simply be connected to or disconnected from the system in an arbitrary fashion or in a more graceful fashion by writing to files located under the /sysfs directory. 

Writing 0 to the /sysfs/bus/pci/slots/0000:bb:dd.f/power file (where bb specifies the bus number, dd specifies the device number, and f specifies the function number of a downstream port) will disable the specifies downstream port whilst writing a 1 to the same file will enable the downstream port. 

Gigabit PF Single Port Server Adapter

When a downstream port is disabled, all devices connected to the port are removed from the Linux kernel and the PCI drivers remove () function will be called to perform the required cleanup. By the same token, when a downstream port is enabled, any devices connected to it will be discovered, added to the Linux kernel and the PCI drivers probe () function will be called to perform the device initialization. Downstream ports may also be enabled or disabled by third-party kernel mode drivers by utilizing the Hot-Swap drivers API described in the section entitled Enabling and Disabling Downstream Ports.

Figure 2 below depicts examples of the types of devices that may be connected to or disconnected from a system running the IDT PCIe Hot Swap driver.

Figure 2  Device Removal

The key to achieving this type of Hot Swap functionality is the ability to assign additional unused bus numbers and memory resources to each downstream port in the system. When a new device is connected to the system, these additional resources are utilized to bring the device to an operational state without having to compromise the operational state of the other devices sharing the same root port.

See the Part 2, please click here.

For more info about PCI Express and optical fiber communication, please browse our website or contact a Sopto representative by calling 86-755-36946668, or by sending an email to info@sopto.com.