Enabling and viewing WASP's performance counters

WASP Tutorial

WASP can expose internal metrics using performance counters. These can then be viewed using the standard Windows performance monitoring tool, perfmon. WASP’s performance counters allow you to see how your server is performing and the counters can be automatically monitored and integrated with the counters from other Windows services, such as SQL server, the base operating system or the CLR, to provide a fully integrated production monitoring system.

Installation of performance counters is done once per machine rather than once per WASP instance and performance counters can be used and viewed both when WASP is running as a Windows Service and from the command line. You install the counters by running WASP from the command line with the /installCounters command line argument. As with installing WASP as a Windows Service, you will be prompted to elevate your credentials if you do not have the appropriate access rights on your account.

Once the counters are installed you can enable them in your WASP config file. Simply add an EnablePerformanceCounters element to the instance node of the config file and set the value to “true”, like so:

<?xml version="1.0" encoding="Windows-1252"?>
<Configuration>
  <WASP
    EnablePerformanceCounters = "true">
    <TCP>
      <Endpoints>
        <EndPoint
          Name="Echo Server"
          Port="5050"
          HandlerDLL="[CONFIG]\EchoServer.dll">
        </EndPoint>
      </Endpoints>
    </TCP>
  </WASP>
</Configuration>

When WASP starts with performance counters enabled it will check to see if the user account that it is running under has the required privileges to be able to create and use the performance counters. Due to the way the counters are implemented the account requires the SE_CREATE_GLOBAL_NAME privilege to create a shared memory segment, WASP simply checks to ensure that the account has admin rights… If the account is capable of creating and using performance counters then WASP will begin to publish its counters if not you’ll get a message in the log file telling you what’s wrong.

To view the counters you need to run perfmon. You can do this by opening a command prompt and typing “perfmon”. This will open the application, note that you may need special rights to be able to monitor performance counters and so we’ll assume that you’re running with administrator rights for the rest of this tutorial.

To be able to view the WASP counters you need to add them to the perfmon display. Select Add Counters and search for the www.ServerFramework.com:WASP counters, they’ll be near the bottom of the list. Since WASP supports multiple concurrent instances you can select counters for each instance separately, or a special _Total instance which accumulates the values from all active instances.

There are a lot of counters but they can be split into several logical groups.

  • Bytes In and Bytes In/sec - these display the amount of data that has been read from the network by WASP.
  • Bytes Out and Bytes Out/sec - these display the amount of data that has been written to the network by WASP.
  • IO Buffers Total, IO Buffers Total +/sec, IO Buffers Total -/sec - most servers developed using The Server Framework use a buffer pool to manage the data buffers used for socket operations. This pool can be configured to cache a certain number of buffers for reuse. Using a buffer from the pool is more efficient than allocating a new buffer. These counters show the total number of buffers that are currently being used by the server, both pooled and currently in use. If the IO Buffers Total value is more than your pool size, or if the +/sec or -/sec values are non zero once your server has been running for a while with the expected number of connections then you could probably do with increasing the pool size…
  • IO Buffers In Use, IO Buffers In Use +/sec, IO Buffers In Use -/sec - these counters show the actual number of buffers that are currently in use by the server for network operations. The IO Buffers In Use counter will be either equal to or smaller than the IO Buffers Total counter with the difference being the number of buffers pooled. Each connection in your server’s listen backlog queue will use one buffer for the read that it has pending, each active connection will use buffers for its active read and write operations.
  • Connections Active - this counter shows the number of active connections to your server.
  • Sockets Total, Sockets Total +/sec, Sockets Total -/sec - as with buffers, sockets are pooled for reuse. Each connection uses a socket object which can be reused for subsequent connections once a connection completes. Using a socket object from the pool is more efficient than allocating one on demand. As with the total buffers counters, these counters can help you see when your pool size is inadequate for the load that your server is experiencing.
  • Sockets In Use, Sockets In Use +/sec, Sockets In Use -/sec - these counters show the actual number of sockets in use. This will closely track the number of active connections plus the size of your listen backlog queue.
  • IO Threads Active, IO Threads Processing, IO Thread Events/sec - all of WASP’s I/O operations are performed on a thread from the I/O pool. The size of this pool can be configured in your config file. The active threads counter will remain constant for the life of the server and will show the number of threads that are configured for the I/O pool. The threads processing counter will fluctuate as I/O operations are handled and will give a rough indication of how busy the server is, but due to how the counter is sampled (see here) it’s only really useful when the server is very busy or as a counter to place a watch on to detect server saturation (watch for the processing counter becoming equal to the active counter and staying there!). The events/sec counter is a better indication of how busy the server’s I/O pool is as it’s not affected by the sample rate of the counter viewer and therefore accurately shows the load on the I/O pool.
  • Business Logic Threads Active, Business Logic Threads Processing, Business Logic Thread Events/sec - WASP can be configured to use two thread pools rather than just one. This is useful if your server has some operations which are blocking in nature and some which are not. By using a two pool design the blocking operations don’t affect the non blocking operations as much since the “business logic pool” can expand and contract as required and the work done on the I/O pool will never block. The counters for this pool are similar in nature to the corresponding counters for the I/O pool.
  • Business Logic Queue, Business Logic Queue +/sec, Business Logic Queue -/sec - when WASP is configured to use a business logic thread pool a queue is used to transfer work items from the I/O pool to the business logic pool. These counters show you how big that queue is getting and how quickly items are being added and removed from it. If the +/sec counter is considerably higher than the -/sec counter then your server is failing to process work items quickly enough and the queue will be growing and operational latency increasing.