TLS 1.2 handshake failure for certificates signed with MD5

| 0 Comments

A while back a client of mine had an issue with a TLS 1.2 connection failing during the TLS handshake. We couldn't see any issues with the code and if we only enabled TLS 1.1 on the server then the connection handshake worked just fine.

Eventually we tracked the issue down to the fact that the certificate in use had been signed with MD5 and that MD5 isn't a valid hash algorithm for TLS 1.2 and so the handshake failed when SChannel attempted to validate the certificate and found that it was unacceptable. There's a Microsoft blog posting about this problem here.

Annoyingly the error message for this is: "The client and server cannot communicate, because they do not possess a common algorithm." which is bang on the money, but doesn't help much until after you've worked out what the problem is. We were already dumping out the enabled algorithms and protocols and couldn't understand why the connection either wasn't working or wasn't downgrading to TLS 1.1.

I've now added some certificate investigation code to the point where my example SChannel servers set up their contexts. This at least allows me to warn and degrade to TLS 1.1 if the certificate is not going to work with TLS 1.2. In production systems I expect we'd just fail to start up.

   CCredentials::Data data;

   data.cCreds = 1;
   data.paCred = &pCertContext;

   // Note that for TLS 1.2 you need a certificate that is signed with SHA1 and NOT MD5.
   // If you have an MD5 certificate then TLS 1.2 will not connect and will NOT downgrade
   // to TLS 1.1 or something else...

   DWORD dataSize = 0;

   if (::CertGetCertificateContextProperty(
      pCertContext,
      CERT_SIGN_HASH_CNG_ALG_PROP_ID,
      0,
      &dataSize))
   {
      ByteBuffer buffer(dataSize);

      BYTE *pData = buffer.GetBuffer();

      if (::CertGetCertificateContextProperty(
         pCertContext,
         CERT_SIGN_HASH_CNG_ALG_PROP_ID,
         pData,
         &dataSize))
      {
         const _tstring signHashDetails(reinterpret_cast<tchar *>(pData), dataSize);

         if (signHashDetails.find(_T("MD5")) != _tstring::npos)
         {
            if (enabledProtocols == 0 || enabledProtocols & SP_PROT_TLS1_2)
            {
               OutputEx(_T("Certificate is signed with: ") + signHashDetails);
               OutputEx(_T("MD5 hashes do not work with TLS 1.2 and connection downgrade does NOT occur"));
               OutputEx(_T("This results in failure to connect. Disabling TLS 1.2"));

               if (enabledProtocols)
               {
                  enabledProtocols &= ~SP_PROT_TLS1_2;
               }

               if (!enabledProtocols)
               {
                  // if enabledProtocols is zero then all protocols are
                  // enabled, so we need to explicitly enable everything
                  // except TLS1.2

                  enabledProtocols = SP_PROT_SSL3TLS1;
              }
            }
         }
      }
   }

   data.grbitEnabledProtocols = enabledProtocols;

Leave a comment

Follow us on Twitter: @ServerFramework

About this Entry

Latest release of The Server Framework: 6.6.4 was the previous entry in this blog.

Latest release of The Server Framework: 6.6.5 is the next entry in this blog.

I usually write about the development of The Server Framework, a super scalable, high performance, C++, I/O Completion Port based framework for writing servers and clients on Windows platforms.

Find recent content on the main index or look in the archives to find all content.