If you are a system administrator, you are probably used to people blaming the hardware whenever an application isn’t performing up to snuff. But it turns out that every once in a while, it is actually the application that is having a problem, and it isn’t hardware-related at all.
Depending on your environment, that may be the case MOST times, but how can you convince the finger-pointers that it isn’t a hardware problem? Today we’ll look at some tools to analyze virtual network utilization to help in some of those situations.
Shared Ethernet Adapters (SEA) have been around for years on Power servers, and are a great way to virtualize your network and share Ethernet adapter hardware resources across many virtual LPARs. But what do you do when someone complains about a “slow network” on one of those LPARs?
Naturally, the first step is to look on the LPAR itself – topas/nmon will allow you to see live updates on the traffic going through the virtual network adapters. Are the values high? Do they seem to hit a “cap” where they can’t go any higher?
What constitutes a high value will depend upon how the SEA is configured on the VIO Server(s) and the physical adapters that are used, as well as the network switches that they are connected to. If they seem to be maxing out at a value near the theoretical max of the underlying adapters, then you may have an application that is using all of the bandwidth that it can get. But what if it is using much less than maximum possible, and there are still complaints of poor network performance? The SEA that it is using may be used by many other LPARs as well, and a bottleneck within the SEA could be limiting the performance of the LPAR in question.
If you have multiple VIO Servers and/or multiple SEAs on your VIO Servers, you’ll have to determine which VIO Server and SEA your LPAR is currently using. I won’t go into detail on that topic today, because there are a lot of different scenarios, but if you aren’t sure, you could always look at the performance of each SEA to see if any of them are having a bottleneck.
Example: SEA Real-Time Analysis
After you log in to your VIO Server, run “oem_setup_env” to switch to a root shell. The lsdev command can tell you which ent devices are your SEAs:
root:/home/padmin # lsdev | grep "Shared" ent12 Available Shared Ethernet Adapter ent13 Available Shared Ethernet Adapter ent22 Available Shared Ethernet Adapter
The “entstat –d ent#” command provides a lot of good information. The output is too much to print here, but I encourage you to look through it – it provides information about all of the physical and virtual adapters that make up the SEA, VLAN IDs, and virtual switch information, link status for the physical adapters and their negotiated speeds, and much more. But the amount of information can be a bit overwhelming, and it doesn’t provide real-time performance information. For that, I like to use “topas –E”, which provides output similar to the following:
root:/home/padmin # topas -E Topas Monitor for host: <VIO_test> Interval: 2 Fri Jul 24 14:53:37 2015 =============================================================================== Network KBPS I-Pack O-Pack KB-In KB-Out ent14 (VLAN VID 210 PRI 0) 25.5 138.3 138.3 12.6 12.9 --ent12 (SEA) 25.5 138.3 138.3 12.6 12.9 |--ent10 (EC PHYS) 12.9 112.9 26.0 10.0 2.9 | |--ent0 (PRIM) 7.8 82.4 7.5 7.1 0.7 | --ent1 (PRIM) 5.1 30.5 18.5 2.9 2.2 --ent8 (VETH) 12.6 25.5 112.4 2.6 9.9 ent15 (VLAN VID 593 PRI 0) 11.4 69.4 69.4 5.7 5.7 --ent13 (SEA) 11.4 69.4 69.4 5.7 5.7 |--ent11 (EC PHYS) 5.7 60.4 9.0 5.1 0.6 | |--ent5 (PRIM) 3.4 31.5 9.0 2.8 0.6 | --ent4 (PRIM) 2.4 29.0 0.0 2.4 0.0 --ent9 (VETH) 5.7 9.0 60.4 0.6 5.1 lo0 0.0 0.0 0.0 0.0 0.0
The first thing to notice about this output is that it is not arranged by SEA, rather by interface adapter (adapters that we have an ip address on). The output from the lsdev command indicated that we have 3 SEAs – ent12, ent13, and ent22; however topas only shows us information related to ent12 and ent13. This is because we don’t have an interface adapter on this VIO Server that uses a VLAN that is trunked by the ent22 SEA. This is a downside to this tool, but in most cases is easy to overcome if the network team is willing to give you another ip address.
I’m always impressed by the layout of the output of this command – they have gotten a lot of great information to fit concisely on one screen, and it is organized in a way that is easy to understand (which isn’t always IBM’s strength). It does a great job of showing you the adapter hierarchy – the relationship between the physical and virtual adapters that make up the SEAs, and the interface adapters that are using them. In this case, we have physical adapters ent0 and ent1 that are combined into Etherchannel ent10 – we know that it is a link aggregation Etherchannel because each adapter is marked as primary. Etherchannel ent10 is combined with virtual Ethernet adapter ent8 to form SEA ent12, which is accessed by interface adapter ent14, using VLAN 210. Likewise, we have ent4 and ent5 link-aggregated into Etherchannel ent11, which is combined with virtual Ethernet adapter ent9 to form SEA ent13, which is utilized by interface adapter ent15, using VLAN 593.
Note that in our case, we are using a SEA Network Interface Backup (SEA-NIB) configuration, as opposed to SEA High Availability (SEA-HA). If we were using SEA-HA, we would also see virtual Ethernet control channel adapters and the SEAs would be labeled Primary or Backup.
The output shows you information about the network traffic going through the SEAs, down to the individual physical adapters. It shows details about the incoming and outgoing traffic in terms of Packets and KB (per second), and overall throughput in KBPS. The topas output updates every 2 seconds by default, so it gives you a real-time view of what is happening on the adapters. And the great part is that this information doesn’t just pertain to the VIO Server’s traffic – it is for all traffic from all virtualized LPARs that are using these SEAs, making it a very powerful and useful tool.
To get back to our original premise, our problem LPAR may not seem like it is hitting a bottleneck, but by looking at topas on the VIO Server, we may find that the SEA is maxing out, which may be limiting the bandwidth that the LPAR can obtain.
Conversely, we may find that the SEA isn’t using anywhere near its maximum, so it doesn’t seem to be a bottleneck. In this case, the problem may be within the application or database, and we now have the proof to show that there isn’t a physical resource problem.
Further Analysis and Historical View
The topas utility on the VIO Server gives you a great real-time view of your SEAs. But what if you want to find out why things ran slowly last night? Or if you have an SEA that is maxing out and you want to find out which LPARs are using it the hardest? You need a more powerful tool, and there is none better than Galileo Performance Explorer.
With Galileo, you can quickly and easily look at the SEA utilization on your VIO Servers. You can drill down to any time frame that you want to see, and show information for any or all of the SEAs within a VIO Server. It is very easy to see if the SEA hit a bottleneck last night, or how the usage has been trending for the past year. Here is an example screenshot below, which shows the SEA utilization for a 12-hour period on the same VIO Server that we looked at above:
Galileo is also great for looking at all of the LPARs (or a subset of LPARs) on a Power server at once. In this case, it can help us determine which LPAR(s) are using the SEA network bandwidth. Below, we see the network usage of all the LPARs on that same server for the same time period:
I’ve removed the LPAR names to protect the innocent, but it is easy to determine which LPARs are using the most bandwidth, and over what period of time. You can use this information to find out what was happening on the LPAR at that time to cause the usage. This information can also give you insight to help tweak the overall design for better performance – for example, if we see a single LPAR becoming the major consumer of an SEA for extended periods of time, it may be necessary to allocate additional adapters to the SEA, or even to provide dedicated adapters to the LPAR (if it does not participate in Live Partition Mobility).
If you are wondering why this chart doesn’t match up exactly to the VIO Server SEA Network Throughput chart, it is because we have a dual VIO Server environment, and we split up our network usage over the two VIO Servers, so the rest of the client LPAR traffic is going through the other VIO Server.
Shared Ethernet Adapters are a great way to implement network virtualization, especially when using the proper tools to view and analyze their performance. If you are thinking of switching from dedicated network adapters to SEA, keep in mind that with SEA there is an additional cost in CPU utilization on the VIO Server(s), and the more clients that are using the SEA, and the higher the bandwidth of the physical adapters, the more CPU cycles will be needed.
We’ve also written an introduction to SR-IOV, which provides an alternative network virtualization technology.