More complex message framing

So far the tutorials have focused on a simple length prefixed message type. This is probably the easiest message in the world to process, the message framing is very simple and there's hardly anything to do in your message framing DLL. Unfortunately not all protocols are this simple to parse. Another common real-world protocol is a line based protocol that is delimited by a terminating character, or characters. One such protocol is the POP3 protocol which works in terms of commands which are delimited by the CR LF sequence.

In this tutorial we'll explore how we can write a message framing plugin for CR LF terminated messages and use per connection user data to track our parsing state.
A POP3 server processes a simple protocol which is line based, see here for the RFC. Since the minimum message length for an empty line is 2 characters we can provide a simple implementation of GetMinimumMessageSize() which looks like this:

The implementation of GetMessageSize() is more complex. When dealing with TCP you must always allow for the fact that you could get any number of bytes in each read that completes. The fact that the remote side might send a string of bytes together as a single send means nothing to you when you're reading data from the network connection; each read could complete with a single byte read from the network, or you could get a single read completion that returns the concatenated results of any number of sends from the remote side. Your message framing code must deal with this. Of course 99% of the time when you're testing your code in the office you'll get complete messages every time, but it's very likely that this will not be the case in production due to other systems using the network, fragmentation, congestion, etc...

WASP's message framing code works by accumulating a message in a buffer until you tell it that the message is a certain size. This allows you to determine the message size in any way that you like and using whatever bytes are required from the message. The only proviso is that all messages fit in a single buffer, but since you're in control of the buffer size that WASP uses that's rarely a problem. In the case of the simple message framing that we have dealt with in earlier tutorials we simply needed to accumulate at least 4 bytes to give us the leading int which contained the message length. WASP made that very easy for us as we could tell it, via our GetMinimumMessageSize() implementation that it shouldn't even bother calling us until it had accumulated at least 4 bytes. For a CR LF terminated protocol things are slightly more complex, we need to scan through every byte in the message until we reach the terminator and then calculate the length of the message.

As mentioned above, we should assume that every message is given to us one byte at a time, so the first call to GetMessageSize() could pass us a buffer containing "A" with a length of 1. The next call could pass us a buffer containing "AP" with a length of 2, etc. Until we receive a buffer which contains "APOP\r\n" with a length of 6 and we have a complete command. To make traversing a message in this way more efficient calls to GetMessageSize() are passed the 'previous length'. So if we were to get "APOP" in one call and "APOP\r\n" in the next the previous length would be 4 and we could continue to scan the input buffer from that offset rather than starting at 0 again each time.

You might think that we could simply look at the end of the buffer and work backwards, that is, simply look at the last character in the buffer and if it's a LF then step back one to see if we have a CR. Whilst that may be an option if the protocol is strictly a one message and one response sequence it's not possible if the client is allowed to send multiple commands whilst waiting for responses from previous commands. Just as a TCP stream may be delivered one byte at a time you can also get any number of "messages" concatenated together. If the client were to send "STAT\r\n" and then "LIST\r\n" without waiting for any responses then the server might receive "STAT\r\nLIST\r\n", or, indeed, any number bytes between 1 and 12. Simply looking at the end of the data wont help in this case and we are forced to sequentially scan forwards for message terminators.

An initial attempt at the scanning code might look something like this:

The problem comes when we need to retain state across calls to GetMessageSize(). After all, we have to allow for the fact that the CR will be present in one call and the LF wont be added until the next. To be able to maintain state from one call to another we need to allocate some per-connection data.

In this simple example the only state that we need to manage is whether the last byte examined was a CR. We'll construct a user data object that's a bit more flexible though. Something like this, perhaps:

Since this is per-connection data we need to allocate it when the connection is created and delete it when the connection is complete.

WASP makes this easy with the OnConnectionEstablished() and OnConnectionComplete() entry points as shown above. The pointer to our user data object will be passed to every call that WASP makes into both our message framing DLL and our message handling DLL as the WaspConnectionUserData. This data is completely opaque to WASP, it does nothing with it and it can be absolutely anything we like.

Our GetMessageSize() implementation can now take advantage of the user data to store the state required for efficiently parsing a CR LF pair that may span two calls to GetMessageSize():

This correctly separates messages based on a CR LF terminator and hold state across calls where required.

Our message handler is now sure to receive complete CR LF terminated messages. Note that we can't simply use them as a character string as they're not guaranteed to be null terminated. We might do something like this to get the inbound message into a form that's easier to use.

You can download the example project for the message framing and message handling DLL from here.

Leave a comment

Follow us on Twitter: @ServerFramework

About this Entry

WASP's config file was the previous entry in this blog.

WASP command line options is the next entry in this blog.

I usually write about the development of The Server Framework, a super scalable, high performance, C++, I/O Completion Port based framework for writing servers and clients on Windows platforms.

Find recent content on the main index or look in the archives to find all content.