April 2011 Archives
Version 6.4 of The Server Framework was released today.
This release includes the following, see the release notes, here, for full details of all changes.
- The ability to enable, configurable, automatic, crash dump creation on certain error conditions.
- Two new single reader, multiple writer, lock implementations.
- Full code page support for unicode to multi-byte conversions.
- Rationalisation of output from
- The ability to limit the number of pending file writes to a collection of asynchronous file writers.
- New high performance buffer allocator.
- Improved performance in the connection manager base class's active connection tracking code.
I'm in the process of completing a custom server development project for a client. The server deals with connections from thousands of embedded devices and allows them to download new firmware or configuration data and upload data that they've accumulated in their local memory. To enable maximum scalability the server use asynchronous reads and writes to the file system as well as the network.
One feature of the server is the ability to configure it to create per session log files. This helps in debugging device communication issues by creating a separate log file for each device every time it connects. You can easily see the interaction between a specific device and the server, including full dumps of all data sent and received if desired. Again, the log system uses asynchronous file writes to allow us to scale well.
With some misconfiguration of the server and a heavy load (8000+ connected devices all doing data uploads and file downloads with full logging for all sessions) I managed to put the system into a mode whereby it was using non-paged pool in an uncontrolled manner. Each data transmission generated several log writes, all log write completions went through a single thread and we were writing to over 16000 files at once. Each asynchronous write used a small amount of non-paged pool but I was issuing writes too fast and so I watched the non-paged pool usage grow to over 2gb on my Windows 7 development box and then watched the box fall over as various drivers failed due to lack of non-paged pool. Not a good design.
Such a limiter was pretty easy to add, you simply pass an instance of the limiter in to each file writer and they all share a single limit. Profiling can show how large you can make the limit for a given box and a general purpose "good enough for most machines" limit value is pretty easy to come up with.
Running the badly configured server again with the new limiter showed that everything was working nicely, the server simply slowed down as the limit was reached and the asynchronous logging became, effectively, synchronous. The non-paged pool memory usage stayed reasonable and the server serviced all of the clients without problem.
The changes to the CAsyncFileWriter and the new limiter will be available in the next release, 6.4, which currently doesn't have a release date.