Part 4: OCSP

I must not be the only one complaining about Certificate Revogation Lists (CRL) usage on Apache, and specially its maturity via mod_ssl module, Don’t know if I am doing something wrong or if there’s any fool-proof way to use this without going to a non-free third-party solution (more on this latter)

hopefully, for helping out all this process of Certificate revogation testing a new protocol was created: The Online Certificate Status Protocol (OCSP)

OCSP is a simple certificate status request and response protocol to a central server. This server is authorised to respond with certificate status information. This process is commonly known as certificate validation. OCSP can provide more timely information regarding the revocation status of a certificate. Use of OCSP reduces network traffic and provides better bandwidth management. Messages using OCSP are Abstract Syntax Notation One (ASN.1) encoded and are communicated over Hypertext Transfer Protocol (HTTP). OCSP servers are normally known as OCSP responders. OCSP can support more than one level of Certificate Authority (CA). OCSP requests could be chained between peer responders to query the issuing certificate authority thus saving client side complexity. An OCSP responder will return a signed response identifying the certificate status.

the principal advantages over CRLs:

OCSP reduces network burden by responding less information than a CRL and it also saves client-side processing by not allowing them to parse CRLs themselves.

In OCSP, if sender has any concerned about the receiver’s private key, then CA’s OCSP checks certificate revocation status in its database and respond accordingly. If OCSP confirms that certificate is OK, sender and receiver can perform secure transaction.

OCSP transmits messages over HTTP, so there may be chances of some delay. If the OCSP server is unavailable, certificate verification will fail.

OCSP can work either in push or pull mode. In push mode, CA server pushes revocation information to the OCSP server or we can configure pull mode so that OCSP server will timely download this information from the CA server.

OCSP supports more than one level of CA.

So this seems the ideal solution we are needing in our server, let put this to work now.
In MOD_SSL documentation we can find the SSLOCSPEnable Directive.
edit /etc/httpd/conf.d/ssl.conf and add this line:

SSLOCSPEnable on

restart Apache with

/etc/ini.t/httpd restart

you should now get this message
Invalid command ‘SSLOCSPEnable’, perhaps misspelled or defined by a module not included in the server configuration

At this point Ive learned once more to give importance to the small lines
In the MOD_SSL documentation there’s this note: “Available in httpd 2.3 and later, if using OpenSSL 0.9.7 or later”
The Apache I am using is httpd-2.2.3-6, so no luck.

The only solution now is grab the most recent httpd source version from the web and compile it. It looks a bit scary idea but it not as hard as it seems.

Download the Unix Source and Unix dependencies Source packages, for example:
# Unix Source: httpd-2.3.8.tar.bz2
# Unix dependencies Source: httpd-2.3.8-deps.tar.bz2 (has the ‘apr’ and ‘apr-util’ sources)

unpack both archives and compile Apache, we are going to install this version on a separate directory (/usr/local/apache2) so we don’t break the existing Apache installation and this way we can choose which Apache to run
Enter on the apache directory and run

./configure --prefix=/usr/local/apache2 --with-apr=/usr/local/apr/ --with-apr-util=/usr/local/apr-util/ --with-mpm=prefork --enable-ssl --disable-charset-lite --enable-mods-shared="proxy proxy_http proxy_connect headers mod_ssl" --disable-include --disable-env --enable-setenvif --disable-status --disable-autoindex --disable-asis --disable-cgi --disable-negotiation --disable-imap --disable-actions --disable-userdir --disable-alias

then generate a self-signed test certificate:

openssl req -new -x509 -days 30 -keyout /usr/local/apache2/conf/ssl.key/server.key -out /usr/local/apache2/conf/ssl.crt/server.crt -subj '/CN=Test-Only Certificate'

To avoid entering the password key each time we restart Apache Ive opted to remove it from the key

cp /usr/local/apache2/conf/ssl.key/server.key /usr/local/apache2/conf/ssl.key/server.key.org
openssl rsa -in /usr/local/apache2/conf/ssl.key/server.key.org -out /usr/local/apache2/conf/ssl.key/server.key

done

for testing the OCSP features its necessary to know the address of the OCSP responder, the server which our Apache will need to communicate to validate the client certificate.
In the best possible case, that address is hard-coded on the client certificate itself, on the “Authority Info Access” (AIA) extension. With this information MOD_SSL OCSP module will know where to find the relevant information just looking into the client Certificate.
The other possible configuration is to add manually the OCSP responder address to the MOD_SSL configuration, using the SSLOCSPDefaultResponder directive
You have to find that address first. Normally that information is on the Authority web pages, if not you have to contact them and ask for it.

I will create a brand new http.conf for testing and merging there the ssl information as well
edit the /usr/local/apache2/conf/http.conf file and add

------------------------------------------------------------------
# ================================================= # Basic settings # =========
========================================
LoadModule unixd_module modules/mod_unixd.so
LoadModule access_compat_module modules/mod_access_compat.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule authn_core_module modules/mod_authn_core.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule headers_module modules/mod_headers.so

User apache
Group apache
ServerAdmin user@xpto.com
ServerName server.name.com
UseCanonicalName Off
ServerSignature Off
HostnameLookups Off
ServerTokens Prod
ServerRoot "/usr/local/apache2"
DocumentRoot "/usr/local/apache2/htdocs"
PidFile /usr/local/apache2/logs/httpd.pid
ScoreBoardFile /usr/local/apache2/logs/httpd.scoreboard 

DirectoryIndex index.html 

 # ======== # HTTP and performance settings # =====================
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 30 

 MinSpareServers 5
 MaxSpareServers 10
 StartServers 5
 MaxClients 150
 MaxRequestsPerChild 0

# =========== # Access control # ================================== 

  Order allow,deny
  Allow from all  

# =========== # MIME encoding # =================================== 

 TypesConfig /usr/local/apache2/conf/mime.types 

#DefaultType text/plain

 AddEncoding x-compress
 AddEncoding x-gzip                  .gz .tgz
 AddType application/x-compress      .Z
 AddType application/x-gzip          .gz .tgz
 AddType application/x-tar           .tgz
 AddType application/x-x509-ca-cert  .crt
 AddType application/x-pkcs7-crl     .crl 

 #============== # Logs # ========================================
LogLevel debug
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
ErrorLog /usr/local/apache2/logs/error_log
CustomLog /usr/local/apache2/logs/access_log combined
CustomLog logs/ssl_request_log "%t %h %{HTTPS}x %{SSL_PROTOCOL}x %{SSL_CIPHER}x
%{SSL_CIPHER_USEKEYSIZE}x %{SSL_CLIENT_VERIFY}x \"%r\" %b"

 # =============== # SSL/TLS settings # ==========================
Listen 0.0.0.0:443
SSLEngine on
SSLOptions +StrictRequire 

 SSLRequireSSL 

SSLProtocol -all +TLSv1 +SSLv3
SSLCipherSuite HIGH:MEDIUM:!aNULL:+SHA1:+MD5:+HIGH:+MEDIUM
Mutex file:/usr/local/apache2/logs/ssl_mutex
SSLRandomSeed startup file:/dev/urandom 1024
SSLRandomSeed connect file:/dev/urandom 1024
SSLSessionCache shm:/usr/local/apache2/logs/ssl_cache_shm
SSLSessionCacheTimeout 600
SSLPassPhraseDialog builtin
SSLCertificateFile /usr/local/apache2/conf/ssl.crt/server.crt
SSLCertificateKeyFile /usr/local/apache2/conf/ssl.key/server.key
SSLCACertificateFile /etc/pki/tls/certs/ca-bundle.crt
SSLVerifyClient require
SSLVerifyDepth 1
SSLProxyEngine off
SSLOCSPEnable on

AddType application/x-x509-ca-cert      .crt
 AddType application/x-pkcs7-crl         .crl 

RequestHeader set SSL_CLIENT_S_DN    ""
RequestHeader set SSL_CLIENT_I_DN    ""
RequestHeader set SSL_SERVER_S_DN_OU ""
RequestHeader set SSL_CLIENT_VERIFY  ""

     # add all the SSL_* you need in the internal web application
     RequestHeader set SSL_CLIENT_S_DN "%{SSL_CLIENT_S_DN}s"
     RequestHeader set SSL_CLIENT_I_DN "%{SSL_CLIENT_I_DN}s"
     RequestHeader set SSL_SERVER_S_DN_OU "%{SSL_SERVER_S_DN_OU}s"
     RequestHeader set SSL_CLIENT_VERIFY "%{SSL_CLIENT_VERIFY}s"

     ProxyPass          http://internal.server.net/
     ProxyPassReverse   http://internal.server.net/

SetEnvIf User-Agent ".*MSIE.*" \     nokeepalive ssl-unclean-shutdown \  downgrade-1.0 force-response-1.0

restart Apache and test it using a web browser to access the server
surprise…no luck. Apache is blocking access to the the web pages

checking the /usr/local/apache2/logs/error_log file I see this errors:

[debug] [pid 10312] ssl_engine_ocsp.c(78):
[client x.y.z.50:43894] no OCSP responder specified in certificate and no default configured
[error] [pid 10312] [client x.y.z.50:43894] Certificate Verification: Error (50): application verification failure
[info] [pid 10312] [client x.y.z.50:43894] SSL library error 1 in handshake (server web.server.net:443)
[info] [pid 10312] SSL Library Error: error:140890B2:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:no certificate returned
[info] [pid 10312] [client x.y.z.50:43894] Connection closed to child 1 with abortive shutdown (server web.server.net:443)

This is a bit strange error as the certificate that is being sent to the server has the proper AIA extensions so nor I don’t understand why the server is not using them. After a lot of Web crawling and posting on forums I couldn’t find any solution for this, so decided, as a workaround, to add the following additional OCSP directive do Apache: (after SSLOCSPEnable on)

SSLOCSPDefaultResponder http://ocsp.auc.cartaodecidadao.pt/publico/ocsp

restart Apache and test again.
Still no luck. This time the OCSP responder is contacted but MOD_SSL is returning “unable to get local issuer certificate” error:

[debug] [pid 10145] ssl_util_ocsp.c(79):  [client x.y.z.50:45551] connecting to OCSP responder 'ocsp.auc.cartaodecidadao.pt'
[debug] [pid 10145] ssl_util_ocsp.c(105): [client x.y.z.50:45551] sending request to OCSP responder
[debug] [pid 10145] ssl_util_ocsp.c(209): [client x.y.z.50:45551] OCSP response header: Date: Fri, 16 Jul 2010 08:31:30 GMT
[debug] [pid 10145] ssl_util_ocsp.c(209): [client x.y.z.50:45551] OCSP response header: Server: Apache
[debug] [pid 10145] ssl_util_ocsp.c(209): [client x.y.z.50:45551] OCSP response header: X-Powered-By: Servlet 2.4; JBoss-4.0.5.GA (build: CVSTag=Branch_4_0 date=200610162339)/Tomcat-5.5
[debug] [pid 10145] ssl_util_ocsp.c(209): [client x.y.z.50:45551] OCSP response header: Expires: Fri, 16 Jul 2010 08:33:30 GMT
[debug] [pid 10145] ssl_util_ocsp.c(209): [client x.y.z.50:45551] OCSP response header: Cache-Control: max-age=120
[debug] [pid 10145] ssl_util_ocsp.c(209): [client x.y.z.50:45551] OCSP response header: Content-Length: 2530
[debug] [pid 10145] ssl_util_ocsp.c(209): [client x.y.z.50:45551] OCSP response header: Connection: close
[debug] [pid 10145] ssl_util_ocsp.c(209): [client x.y.z.50:45551] OCSP response header: Content-Type: application/ocsp-response
[debug] [pid 10145] ssl_util_ocsp.c(252): [client x.y.z.50:45551] OCSP response: got 2530 bytes, 2530 total
[debug] [pid 10145] ssl_util_ocsp.c(235): [client x.y.z.50:45551] OCSP response: got EOF
[error] [pid 10145] SSL Library Error: error:27069065:OCSP routines:OCSP_basic_verify:certificate verify error (Verify error:unable to get local issuer certificate)
[error] [pid 10145] failed to verify the OCSP response
[error] [pid 10145] [client 10.14.148.50:45551] Certificate Verification: Error (50): application verification failure
[info] [pid 10145]  [client 10.14.148.50:45551] SSL library error 1 in handshake (server web.server.net:443)
[info] [pid 10145]  SSL Library Error: error:140890B2:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:no certificate returned
[info] [pid 10145]  [client 10.14.148.50:45551] Connection closed to child 1 with abortive shutdown (server web.server.net:443)

on the open-ssl mailing list Ive got very good tips related to this error:

“For each certificate do this:
openssl x509 -in cert.pem -subject -issuer -noout

The subject of the one you pass to -issuer should match the issuer of the one
you pass to cert. You need a root CA and the rest of the chain passed to
-CApath.”

and

“Your certificate chain needs to be complete. That is it has to include the
root CA (one with issuer and subject the same) and all intermediate
certificates of the responder certificate.”

My Certificate chain was indeed incomplete, there was no ROOT certificate in /etc/pki/tls/certs/ca-bundle.crt, Ive checked the “subject” and “issuer” for each certificate Ive downloaded from the CA web pages and Ive missed the one which had the “issuer” equal to the “subject” one, the ROOT Certificate.

added the missing certificate do ca-bundle.crt, restarted Apache and tried again to access the web server

error again!
checking out on the Apache logs once more:

[debug] [pid 21783] ssl_util_ocsp.c(79):  [client x.y.z.50:54752] connecting to OCSP responder 'ocsp.auc.cartaodecidadao.pt'
[debug] [pid 21783] ssl_util_ocsp.c(105): [client x.y.z.50:54752] sending request to OCSP responder
[debug] [pid 21783] ssl_util_ocsp.c(209): [client x.y.z.50:54752] OCSP response header: Date: Fri, 16 Jul 2010 14:51:24 GMT
[debug] [pid 21783] ssl_util_ocsp.c(209): [client x.y.z.50:54752] OCSP response header: Server: Apache
[debug] [pid 21783] ssl_util_ocsp.c(209): [client x.y.z.50:54752] OCSP response header: X-Powered-By: Servlet 2.4; JBoss-4.0.5.GA
 (build: CVSTag=Branch_4_0 date=200610162339)/Tomcat-5.5
[debug] [pid 21783] ssl_util_ocsp.c(209): [client x.y.z.50:54752] OCSP response header: Expires: Fri, 16 Jul 2010 14:53:24 GMT
[debug] [pid 21783] ssl_util_ocsp.c(209): [client x.y.z.50:54752] OCSP response header: Cache-Control: max-age=120
[debug] [pid 21783] ssl_util_ocsp.c(209): [client x.y.z.50:54752] OCSP response header: Content-Length: 2530
[debug] [pid 21783] ssl_util_ocsp.c(209): [client x.y.z.50:54752] OCSP response header: Connection: close
[debug] [pid 21783] ssl_util_ocsp.c(209): [client x.y.z.50:54752] OCSP response header: Content-Type: application/ocsp-response
[debug] [pid 21783] ssl_util_ocsp.c(252): [client x.y.z.50:54752] OCSP response: got 1127 bytes, 1127 total
[debug] [pid 21783] ssl_util_ocsp.c(252): [client x.y.z.50:54752] OCSP response: got 1403 bytes, 2530 total
[debug] [pid 21783] ssl_util_ocsp.c(235): [client x.y.z.50:54752] OCSP response: got EOF
[error] [pid 21783] SSL Library Error: error:27069070:OCSP routines:OCSP_basic_verify:root ca not trusted
[error] [pid 21783] failed to verify the OCSP response
[error] [pid 21783] [client x.y.z.50:54752] Certificate Verification: Error (50): application verification failure
[info] [pid 21783] [client x.y.z.50:54752] SSL library error 1 in handshake (server web.server.net:443)
[info] [pid 21783] SSL Library Error: error:140890B2:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:no certificate returned
[info] [pid 21783] [client x.y.z.50:54752] Connection closed to child 1 with abortive shutdown (server web.server.net:443)

“root ca not trusted”?
This time I didnt find any clue to how to solve this.

Just for curiosity, instead of using mod_ssl to check certificates using an OCSP responder I am now going to use the openssl command line to do the same check.
I have exported my client cerfificate using the the web browser to lneves.der, uploaded it to the web server and converted to the PEM format using

openssl x509 -in lneves.der -inform DER -out lneves.pem -outform PEM

Now used this command line to make an OCSP query do the specified responder:

openssl ocsp -issuer /etc/pki/tls/certs/CC0003.pem -cert
lneves.pem -url http://ocsp.auc.cartaodecidadao.pt/publico/ocsp
-CAfile /etc/pki/tls/certs/ca-bundle.crt -resp_text

Response verify OK
/home/oracle/lneves.pem: good

success… it works, openssl validates my client certificate without problems.
MOD_SSL not. At this point I was blocked and without any ideia on how to progress on this configuration
I dont know if I was doing something severily wrong here, or if the certificates are kind of broken, or if MOD_SSL OCSP code is not mature enought but the truth is that this is not working as I expected.

What alternatives do I have now?