TLS 1.2 handshake failure for certificates signed with MD5

A while back a client of mine had an issue with a TLS 1.2 connection failing during the TLS handshake. We couldn’t see any issues with the code and if we only enabled TLS 1.1 on the server then the connection handshake worked just fine.

Eventually we tracked the issue down to the fact that the certificate in use had been signed with MD5 and that MD5 isn’t a valid hash algorithm for TLS 1.2 and so the handshake failed when SChannel attempted to validate the certificate and found that it was unacceptable. There’s a Microsoft blog posting about this problem here.

Annoyingly the error message for this is: “The client and server cannot communicate, because they do not possess a common algorithm.” which is bang on the money, but doesn’t help much until after you’ve worked out what the problem is. We were already dumping out the enabled algorithms and protocols and couldn’t understand why the connection either wasn’t working or wasn’t downgrading to TLS 1.1.

I’ve now added some certificate investigation code to the point where my example SChannel servers set up their contexts. This at least allows me to warn and degrade to TLS 1.1 if the certificate is not going to work with TLS 1.2. In production systems I expect we’d just fail to start up.

   CCredentials::Data data;

   data.cCreds = 1;
   data.paCred = &pCertContext;

   // Note that for TLS 1.2 you need a certificate that is signed with SHA1 and NOT MD5.
   // If you have an MD5 certificate then TLS 1.2 will not connect and will NOT downgrade
   // to TLS 1.1 or something else...

   DWORD dataSize = 0;

   if (::CertGetCertificateContextProperty(
      pCertContext,
      CERT_SIGN_HASH_CNG_ALG_PROP_ID,
      0,
      &dataSize))
   {
      ByteBuffer buffer(dataSize);

      BYTE *pData = buffer.GetBuffer();

      if (::CertGetCertificateContextProperty(
         pCertContext,
         CERT_SIGN_HASH_CNG_ALG_PROP_ID,
         pData,
         &dataSize))
      {
         const _tstring signHashDetails(reinterpret_cast<tchar *>(pData), dataSize);

         if (signHashDetails.find(_T("MD5")) != _tstring::npos)
         {
            if (enabledProtocols == 0 || enabledProtocols & SP_PROT_TLS1_2)
            {
               OutputEx(_T("Certificate is signed with: ") + signHashDetails);
               OutputEx(_T("MD5 hashes do not work with TLS 1.2 and connection downgrade does NOT occur"));
               OutputEx(_T("This results in failure to connect. Disabling TLS 1.2"));

               if (enabledProtocols)
               {
                  enabledProtocols &= ~SP_PROT_TLS1_2;
               }

               if (!enabledProtocols)
               {
                  // if enabledProtocols is zero then all protocols are
                  // enabled, so we need to explicitly enable everything
                  // except TLS1.2

                  enabledProtocols = SP_PROT_SSL3TLS1;
              }
            }
         }
      }
   }

   data.grbitEnabledProtocols = enabledProtocols;