July 2012 Archives

As I mentioned back in March I'm now doing some RIO performance testing with a 10 Gigabit Ethernet link as I couldn't push my test server hard enough with multiple 1 Gigabit links. This involves 2 Intel 10 Gigabit AT2 cards connected directly to each other. One of the cards is in our test server and the other has been tried in various other pieces of test hardware in an attempt to ramp up the bandwidth usage for the RIO tests. Unfortunately driving one of these network cards to its limits takes quite a lot of CPU and also requires that the card is in a PCI Express slot with x8 capabilities; our other test hardware, being slightly older, has plenty of slots capable of taking the card, but only a relatively low spec build machine had any of those slots wired up as x8. This left me in a situation which was fractionally better than teamed multiple 1 Gigabit links but nowhere near the full potential of the 10 Gigabit link.

I've just finished building a new developer workstation with an Asus P9X79 WS Intel X79 motherboard and a 3.20GHz Core i7 3930K processor. We now have the bus bandwidth and the processor power to push the link and stress the server nicely. At 50% link saturation we're running at around 10% CPU usage on the source machine (1 of the 12 cores pegged at 100% by the simple UDP traffic generator that we used in the earlier tests). On the test server we are also using around 10% CPU for my most complicated RIO IOCP design.

Rough results for this 50% loading of the link give 350,000 datagrams per second processed by a traditional IOCP UDP server with 8 threads as against 450,000 datagrams per second from a naive RIO IOCP server design and 920,000 per second from a more complicated design. Pushing the loading to 75%, by running the traffic generator twice, shows no improvement in datagram processing speed from the traditional design but the complicated RIO design moves up to 970,000 per second. These results should be taken with a pinch of salt as I haven't had a chance to check them over and test more thoroughly. I'll present the server designs in the same way that I've presented the previous designs when I publish the full results.

Looks like I'm finally in a position to see what RIO can really do...
The recent changes to 6.5.5 so that JetByteTools::Socket::CReadTimeoutStreamSocketConnectionFilter holds a socket reference when the timer is set causes problems when the socket is shutdown. The filter doesn't see the shutdown and fails to cancel the timer which means that a reference is held on the socket until the timeout expires. This delays socket closure and causes your timeout handling code to be run on a socket which has been cleanly shut down. The fix addresses this by notifying the filter chain of the shutdown.

This change will be included in 6.5.6 which is currently in testing. If you need the fix now, please get in touch.

This bug slipped through testing due to a lapse on my part. A manual step in the release process needs to be automated and made part of the build/test/release cycle that runs on my build machines. All of the servers are tested using the test clients that ship as part of the server examples and many of these test clients should have failed during testing of 6.5.5. The tools that are used as part of the build and test, such as the client tests, are built via a script and use the latest version. They are then checked in as part of the release and are used during the build/test/release cycle on the build machines. For 6.5.5 the script wasn't run and so it was tested with the 6.5.4 versions of the test clients which didn't have this problem.