PCIe Adapter Virtualization with SR-IOV

by | Jul 24, 2015

Within the world of POWER hardware, the PowerVM SEA (Shared Ethernet Adapter) has been the standard when it comes to virtualizing an ethernet adapter for quite some time. With the introduction of SR-IOV (Single Root I/O Virtualization) the additional VIOS layer is removed, allowing for logical hosts to communicate directly with the PCIe adapter instead […]

Within the world of POWER hardware, the PowerVM SEA (Shared Ethernet Adapter) has been the standard when it comes to virtualizing an ethernet adapter for quite some time. With the introduction of SR-IOV (Single Root I/O Virtualization) the additional VIOS layer is removed, allowing for logical hosts to communicate directly with the PCIe adapter instead of through the hypervisor. This direct OS-to-adapter communication is comparable to the LHEA (Logical Host Ethernet Adapter), an IBM proprietary offering included with POWER hardware that many customers utilized for years. Now that PCIe has matured, adapter vendors can offer the same functionality and provide an alternative to LHEA/SEA while reducing the performance overhead associated with passing network traffic through a Virtual I/O Server.

In addition to the lower performance overhead with SR-IOV, users are also able to define Quality-of-Service (QOS), meaning you are able to define a capacity value for each logical port. The QOS feature is what really sets SR-IOV apart from the other ethernet virtualization options, in my opinion. In an SEA configuration, sharing network adapters between production and lower level environments under a single SEA was frowned upon as there was no way to guarantee that production traffic was prioritized. The solution to isolate traffic between environments involved either assigning additional ethernet adapters and configuring additional SEAs within an existing VIO pair, or even creating another VIO Server pair altogether. With SR-IOV, you’re able to guarantee that a logical adapter used for production will, at a minimum, receive a defined percentage (X% out of 100%) of a physical adapter’s capacity. If extra capacity beyond the minimum is available, the logical adapter will use it.

Much like the LHEA adapter, SR-IOV capable adapters can be set to a “dedicated” mode (analogous to “promiscuous” mode in LHEA terms) and assigned to a VIO Server to be used as part of an SEA. I’d argue that using an SR-IOV capable adapter in this fashion defeats the purpose of including it in your design, but it’s certainly nice to have the option if you decide that you aren’t quite ready to roll out SR-IOV into production after your initial testing. A logical port can also be assigned to a VIO Server and used within an SEA if you wish. Finding a use case for all of these options in your environment may take some thought, but the point is that SR-IOV is quite flexible when it comes to deployment options.

Unfortunately, SR-IOV adapters are not LPM capable at this time, but will be in the future. I have yet to see a date tied to LPM support enablement. Any new features, such as LPM enablement, would likely be included in a microcode update, which is a nice change from LHEA enhancements being packaged within system firmware updates as opposed to separate adapter microcode.

IBM recently announced that SR-IOV capability will be available on the entire Power 8 server family as of May 2015. There are few adapter placement requirements that your hardware vendor will need to consider, in addition to the following minimum OS levels:

  • AIX 6.1 TL9 SP5 and APAR IV68443 or later
  • AIX 6.1 TL8 SP7 or later (planned 3Q 2015)
  • AIX 7.1 TL3 SP5 and APAR IV68444, or later
  • AIX 7.1 TL2 SP7 or later (planned availability 3Q 2015)
  • IBM i 7.1 TR10 or later
    (Note: IBM i requires either VIOS or adapter in SR-IOV mode)
  • IBM i 7.2 TR2 or later
    (Note: IBM i requires either VIOS or adapter in SR-IOV mode)
  • Red Hat Enterprise Linux 6.6, or later
  • Red Hat Enterprise Linux 7.1, big endian, or later
  • Red Hat Enterprise Linux 7.1, little endian, or later
  • SUSE Linux Enterprise Server 12, or later
  • Ubuntu 15.04, or later
  • Firmware 830 available June, 2015
  • HMC V8.830

Related Articles