Part 7: Conclusion

My Conclusion:

OCSP and CRL features is not mature enough in the MOD_SSL Apache module to be used without issues. Maybe I did something that I shouldn’t, if this is the case, I didn’t detected it. At one point on this quest I thought that this was an impossible mission and that the only way to do this configuration was buying some commercial solution. Fortunately some friendly buddies on openssl and mod_nss mailing lists gave very good help on all the issues. And some more carefully searching on the web helped to find more solutions to try besides mod_ssl.
Webcullis was really a surprise as it worked first time and was so easy to install. I was little sceptical on the MOD_NSS usage also because I could not find as many information on the web as with mod_ssl, I thought the project was kind of dead as it hasn’t too much movement and had to few messages on the mailing lists. Maybe the reason for this slow movement is because everyone is happy with the results and don’t have any errors to post!!

After all this work I think for now MOD_NSS is the right way to go, at last at this time of writing
One more thing I didn’t mentioned here.
Ive tried to add support on Apache to authenticate/authorizate the users using an LDAP directory. Tried to use mod_authz_module Apache module and modXLdapAuth.
What I have found after so much time invested is that the mod_ssl code has some BUG dealing with UTF8 characters on the certificate fields (like the “issuer” attribute) MOD_SSL is putting the ‘\x’ characters before the UTF8 data and this is enough to invalidate all the tests I did.

Ive noticed that Webcullis and MOD_NSS didn’t had such BUG so maybe if I do the LDAP tests again using those modules I could have some positive progress on this area.

Unfortunately I couldn’t test Pathfinder as It gave me some errors compiling that I couldn’t resolve without mangling with some system libraries.

This text was done more than six months after the last modification was done on the servers. There’s lot of small things that I didn’t saved at the time, like some Apache logs from the MOD_NSS usage. But nevertheless I hope this information could be so useful for you as It was for me and that it could save you some time, the time I’ve spent can be the time you save! If this is the case than I consider this a double success

Part 6: MOD_NSS

from the MOD_NSS home page:
“mod_nss is an SSL provider derived from the mod_ssl module for the Apache web server that uses the Network Security Services (NSS) libraries. We started with mod_ssl and replaced the OpenSSL calls with NSS calls.

The mod_ssl package was created in April 1998 by Ralf S. Engelschall and was originally derived from the Apache-SSL package developed by Ben Laurie. It is licensed under the Apache 2.0 license. ”

what is very attractive now for us is that MOD_NSS has out-of-the-box  support for Certificate Revocation Lists (CRLs) and OCSP. Not only this but support for another interesting module: mod_revocator
”
mod_revocator is an Apache module that lets an administrator configure remote Certificate Revocation Lists (CRLs) to be downloaded and installed automatically on a regular basis without restarting the server. This helps ensure that the CRLs are kept up-to-date with minimal effort. The module can also bring the server down if the CRL expires and a new one cannot be obtained.”

this module provides:

* Automatic CRL download and installation from an HTTP[S] or LDAP[S] source
* The source can be an executable that retrieves the CRL from another method
* Can specify per CRL source the interval between downloads
* Can specify per CRL source the maximum age of the CRL before requiring a new one
* Can shut down the server if a CRL is not available

this is all very nice indeed! It makes all that automation Ive talked about in previous pages for you. We don’t need to create or use third-party scripts for downloading CRL or delta-CRL, installing them, testing errors, restarting Apache, as all is done for you automatically! Lets see if we can put this to work, sounds to good to be true…

Its possible to install MOD_NSS easily on our system
just type “yum install mod_nss” and you’re done, then tune some mod_nss.conf parameters.
The problem is that then we will want to install the mod_revocator as well, and that module only exists in form of a installation package (RPM) on Redhat based system. On Debian systems it will need to be compiled (or converted to a deb file using the alien tool? I didn’t tested this option)

So Ive decided to be radically different this time
Ive just installed a brand new system with a Fedora 13 distribution

then

yum install mod_nss mod_revocator openssh-server nss-tools
gencert /etc/httpd/alias (generate self-signed certificate for testing)

now import the CA certificates to the NSS database. NSS uses a certificate database rather than discrete files

certutil -d /etc/httpd/alias -A -n "ECCC0001" -t "CT,," -a -i ./EC\  de\ Autenticacao\ do\ Cartao\ de\ Cidadao\ 0001.pem certutil -d  /etc/httpd/alias -A -n "ECCC0002" -t "CT,," -a -i ./EC\ de\  Autenticacao\ do\ Cartao\ de\ Cidadao\ 0002.pem certutil -d  /etc/httpd/alias -A -n "CC0002" -t "CT,," -a -i ./EC\ de\ Autenticacao\  do\ Cartao\ de\ Cidadao\ 0002.pem certutil -d /etc/httpd/alias -A -n  "CC0003" -t "CT,," -a -i ./EC\ de\ Autenticacao\ do\ Cartao\ de\  Cidadao\ 0003.pem certutil -d /etc/httpd/alias -A -n "ECRaiz" -t "CT,,"  -a -i ./ECRaizEstado.pem 
now list the content of the DB
certutil -L -d /etc/httpd/alias/

edit nss.conf

changed Listen 8443 to 443
changed to 443
LogLevel debug
NSSVerifyClient require
NSSOCSP on

bounced Apache But got some error on a log file:

cat /var/log/httpd/error_log
[error] NSS_Initialize failed. Certificate database: /etc/httpd/alias.
[error] SSL Library Error: -8038 SEC_ERROR_NOT_INITIALIZED

The NSS database needs to be readable by the user apache (the default user of httpd). so fix this error changing the owner of /etc/httpd/alias/*.db to root:apache and permissions to mode 0640

bounce apache and test.
It works! I can access the server and can confirm everything in the logs

just to be sure the OCSP responder is being called I’ve captured the network traffic going to/from the responder address using the TCPDUMP tool while using the web site

tcpdump host ocsp.auc.cartaodecidadao.pt

The resulting packets confirmed that the communication is being established

Now to test the CRL functionality, I will begin by installing the other CA server certificate, the one CA which doesn’t allow us to use their OCSP responder without paying a fee. So I will use their CRL lists.

Add the Verisign Server Certificates (the Certificate chain, but one by one) to the NSS database:

certutil -d /etc/httpd/alias -A -n "Verisign" -t "CT,," -a -i ./VeriSignClass2PublicPrimaryCertificationAUthority_G3.pem
certutil -d /etc/httpd/alias -A -n "DigitalSignQualifiedCA" -t "CT,," -a -i ./DigitalSignQualifiedCA.pem
certutil -d /etc/httpd/alias -A -n "VeriSignClass2PublicPrimaryCAG3" -t "CT,," -a -i ./VeriSignClass2PublicPrimaryCAG3.pem
certutil -d /etc/httpd/alias -A -n "BTClass2CAG2" -t "CT,," -a -i ./BTClass2CAG2.pem

alter /etc/httpd/conf.d/revocator.conf file and activate the CRL usage and specifiy the crl file address and the interval between refreshes:

CRLEngine on
CRLFILE "http://onsitecrl.trustwise.com/BritishTelecommunicationsplcBTDigitalSignQualifiedCA/LatestCRL.crl;1440;1440"

Bounce Apache and test.
Voila’! It works again! confirm on the Apache logs that everything is going fine and one more, monitor the communication to the CRL address using TCPDUMP

tcpdump host onsitecrl.trustwise.com

Now to finalise this, its necessary to add the reverse-proxy stuff to the http.conf file

NSSVerifyClient optional

RewriteEngine on
RewriteCond %{SSL:SSL_CLIENT_VERIFY} !=SUCCESS
RewriteRule .* /index.html [L]

     RequestHeader set SSL_CLIENT_S_DN    ""
     RequestHeader set SSL_CLIENT_I_DN    ""
     RequestHeader set SSL_CLIENT_S_DN_O ""
     RequestHeader set SSL_CLIENT_VERIFY  ""

     # add all the SSL_* you need in the internal web application
     RequestHeader set SSL_CLIENT_S_DN "%{SSL_CLIENT_S_DN}s"
     RequestHeader set SSL_CLIENT_I_DN "%{SSL_CLIENT_I_DN}s"
     RequestHeader set SSL_CLIENT_S_DN_O "%{SSL_CLIENT_S_DN_O}s"
     RequestHeader set SSL_CLIENT_VERIFY "%{SSL_CLIENT_VERIFY}s"

     ProxyPass          http://internal.server.net/
     ProxyPassReverse   http://internal.server.net/

Part 5: MOD_SSL alternatives

After some many problems trying to use Apache and MOD_SSL and OCSP I began searching another solutions for setting up this server.
I’ve resumed those to some three free alternatives:

MOD_NSS http://directory.fedoraproject.org/wiki/Mod_nss
Webcullis http://pkif.sourceforge.net/webcullis.html
Pathfinder http://www.carillon.ca/tools/pathfinder.php

After some searching, MOD_NSS looked to me a bit, how can I say, stalled. Theres is not a lot of movement on its mailing lists, and theres not so many hits on MOD_NSS plus OCSP, or at last not as much compared do MOD_SSL and OCSP. If this is a bad thing or a good sign I dont know. Ive decided to test Webcullis first.

Gone to its Web page, the PKIF project and downloaded webcullis-2.1.11-RHEL5-gcc41-dist at the time of this writing
Uncompressed it at the ROOT directory and followed the INSTALL file step by step
(NOTE: Webcullis is expecting to find the CA Certificates on DER format so I’ve converted all the individual certificates that I have downloaded from the CA website to DER format using “openssl x509 -in certificate.crt -out certificate.cer -outform der)

mkdir -p /etc/webcullis/trustroots
mkdir -p /var/log/webcullis
mkdir -p /usr/local/webcullis/lib
cd /root/webcullis-2.1.11-RHEL5-gcc41-dist
cp lib/* /usr/local/webcullis/lib/
cp conf/* /etc/webcullis/
cd /etc/pki/tls/certs
cp *.cer /etc/webcullis/trustroots/
add /usr/local/webcullis/lib to /etc/ld.so.conf.d/webcullis.conf
ldconfig
ln -s /usr/local/webcullis/lib/libmod_auth_webcullis.so /etc/httpd/modules
ln -s /usr/local/webcullis/lib/libmod_auth_webcullis.so /etc/httpd/modules
cp /etc/httpd/conf.d/ssl.conf  /etc/httpd/conf.d/ssl.conf.original
vi /etc/httpd/conf.d/ssl.conf

LoadModule ssl_module modules/mod_ssl.so
LoadModule auth_webcullis_module modules/libmod_auth_webcullis.so
WebcullisIniFile "/etc/webcullis/webcullis.cf"

Listen 443

AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl    .crl

SSLPassPhraseDialog  builtin

SSLSessionCache         shmcb:/var/cache/mod_ssl/scache(512000)
SSLSessionCacheTimeout  300

SSLMutex default

SSLRandomSeed startup file:/dev/urandom  256
SSLRandomSeed connect builtin

SSLCryptoDevice builtin

        SSLOptions +StdEnvVars +ExportCertData

                SetHandler perl-script
                PerlResponseHandler ModPerl::Registry
                PerlOptions +ParseHeaders
                Options +ExecCGI        

        ErrorLog logs/ssl_error_log
        TransferLog logs/ssl_access_log
        LogLevel debug

        SSLEngine on

        SSLProtocol all -SSLv2
        SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW
        SSLCertificateFile /etc/pki/tls/certs/localhost.crt
        SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
        SSLCARevocationFile /etc/httpd/conf/ssl.crl/LatestCRL.pem
        SSLVerifyClient webcullis
        SSLVerifyDepth 4

        RewriteEngine on
        RewriteCond %{SSL:SSL_CLIENT_VERIFY} !^SUCCESS$
        RewriteRule ^/ /index.html [L]

        RequestHeader set SSL_CLIENT_S_DN    ""
        RequestHeader set SSL_CLIENT_I_DN    ""
        RequestHeader set SSL_SERVER_S_DN_OU ""
        RequestHeader set SSL_CLIENT_VERIFY  ""

                RequestHeader set SSL_CLIENT_S_DN "%{SSL_CLIENT_S_DN}s"
                RequestHeader set SSL_CLIENT_I_DN "%{SSL_CLIENT_I_DN}s"
                RequestHeader set SSL_SERVER_S_DN_OU "%{SSL_SERVER_S_DN_OU}s"
                RequestHeader set SSL_CLIENT_VERIFY "%{SSL_CLIENT_VERIFY}s"

                ProxyPass          http://web.server.net/
                ProxyPassReverse   http://web.server.net/

                SSLOptions +StdEnvVars
        SetEnvIf User-Agent ".*MSIE.*" \
                nokeepalive ssl-unclean-shutdown \
                downgrade-1.0 force-response-1.0

        CustomLog logs/ssl_request_log \
                "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

save the /etc/httpd/conf.d/ssl.conf file

cd /etc/httpd/modules
cp mod_ssl.so mod_ssl.so.original
cp /root/webcullis-2.1.11-RHEL5-gcc41-dist/mod_ssl.so .

bounce apache: /etc/init.d/httpd start
test and

SUCCESS! It works, first time. Impressive. Its really a joy seeing this working at last
Just to be sure, check what webcullis module is doing looking at the
/var/log/webcullis/module.log and var/log/webcullis/trace.log log files.

I see on the module.log I see among other messages:

+ Revocation status: NOT REVOKED
+ Revocation source #1
     - Revocation source error code: 0 : Success
     - Revocation source type: 2
     - Revocation source status: NOT_REVOKED
     - OCSP response information:
            + Response status: successful
            + Response type: 1.3.6.1.5.5.7.48.1.1
            + Produced at: 20100720145931Z
            + Responder name: cn=Serviço de Validação on-line do Cartão de Cidadão 000020 - EC de Autenticação do Cidadão,ou=Serviços do Cartão de Cidadão,ou=Validação on-line,o=Cartão de Cidadão,c=PT
            + OCSP single response information:
                    * This update: 20100720145931Z
                    * Cert serial number: 0x5fd933e0f2f95d0f

Fantastic news! I am thrilled

Now, besides accessing a OCSP responder for validating in real-time a client certificate revogation state lets try to use a CRL list instead.
My Apache server will also need to accept certificates from another Certification Authotity, a CA that doesn’t have any OCSP service available. (In this case, its a CA where its OCSP service is not free)
So I will need to use Certification Revogation List instead.
But first I need to install the CA server certificates in Apache. As I couldn’t find them on the CA site Ive contacted them and they kindly mailed them to me

All the Server Certificate Chain was sent me in the a file with a p7b extension.

from the web

PKCS#7/P7B Format:
The PKCS#7 or P7B format is usually stored in Base64 ASCII format and has a file extention of .p7b or .p7c. P7B certificates contain “—–BEGIN PKCS7—–” and “—–END PKCS7—–” statements. A P7B file only contains certificates and chain certificates, not the private key. Several platforms support P7B files including Microsoft Windows and Java Tomcat.

I need to install this chain certificate on the Apache as individual DER certificates (Webcullis needs this way), so used this command line to convert it to a PEM chain first

openssl pkcs7 -in DigitalSignCP.p7b -inform DER -text -print_certs -out certs.pem

then edited the PEM file and saved each certificate data as an individual PEM file (using copy/paste)
then, for each of the three individual certificates created on the last step I’ve converted to the DER format:

openssl x509 -in BTClass2CA_G2.pem -out /etc/webcullis/trustroots/BTClass2CA_G2.cer -outform der
openssl x509 -in BTDigitalSignQualifiedCA.pem -out /etc/webcullis/trustroots/BTDigitalSignQualifiedCA.cer -outform der
openssl x509 -in VeriSignClass2PublicPrimaryCertificationAUthority_G3.pem -out /etc/webcullis/trustroots/VeriSignClass2PublicPrimaryCertificationAUthority.cer -outform der

That’s it. no need to mess with CRL files as Webcullis will do it for you (using AND updating? I’m not sure on this but so far I didn’t found any information confirming or not confirming this feature). Just configure some CRL parameters on the
/etc/webcullis/webcullis.cf

Its nice to have a working configuration, but I will not close this case right now. I will check now how hard and effective is the usage of MOD_NSS.

Part 4: OCSP

I must not be the only one complaining about Certificate Revogation Lists (CRL) usage on Apache, and specially its maturity via mod_ssl module, Don’t know if I am doing something wrong or if there’s any fool-proof way to use this without going to a non-free third-party solution (more on this latter)

hopefully, for helping out all this process of Certificate revogation testing a new protocol was created: The Online Certificate Status Protocol (OCSP)

OCSP is a simple certificate status request and response protocol to a central server. This server is authorised to respond with certificate status information. This process is commonly known as certificate validation. OCSP can provide more timely information regarding the revocation status of a certificate. Use of OCSP reduces network traffic and provides better bandwidth management. Messages using OCSP are Abstract Syntax Notation One (ASN.1) encoded and are communicated over Hypertext Transfer Protocol (HTTP). OCSP servers are normally known as OCSP responders. OCSP can support more than one level of Certificate Authority (CA). OCSP requests could be chained between peer responders to query the issuing certificate authority thus saving client side complexity. An OCSP responder will return a signed response identifying the certificate status.

the principal advantages over CRLs:

OCSP reduces network burden by responding less information than a CRL and it also saves client-side processing by not allowing them to parse CRLs themselves.

In OCSP, if sender has any concerned about the receiver’s private key, then CA’s OCSP checks certificate revocation status in its database and respond accordingly. If OCSP confirms that certificate is OK, sender and receiver can perform secure transaction.

OCSP transmits messages over HTTP, so there may be chances of some delay. If the OCSP server is unavailable, certificate verification will fail.

OCSP can work either in push or pull mode. In push mode, CA server pushes revocation information to the OCSP server or we can configure pull mode so that OCSP server will timely download this information from the CA server.

OCSP supports more than one level of CA.

So this seems the ideal solution we are needing in our server, let put this to work now.
In MOD_SSL documentation we can find the SSLOCSPEnable Directive.
edit /etc/httpd/conf.d/ssl.conf and add this line:

SSLOCSPEnable on

restart Apache with

/etc/ini.t/httpd restart

you should now get this message
Invalid command ‘SSLOCSPEnable’, perhaps misspelled or defined by a module not included in the server configuration

At this point Ive learned once more to give importance to the small lines
In the MOD_SSL documentation there’s this note: “Available in httpd 2.3 and later, if using OpenSSL 0.9.7 or later”
The Apache I am using is httpd-2.2.3-6, so no luck.

The only solution now is grab the most recent httpd source version from the web and compile it. It looks a bit scary idea but it not as hard as it seems.

Download the Unix Source and Unix dependencies Source packages, for example:
# Unix Source: httpd-2.3.8.tar.bz2
# Unix dependencies Source: httpd-2.3.8-deps.tar.bz2 (has the ‘apr’ and ‘apr-util’ sources)

unpack both archives and compile Apache, we are going to install this version on a separate directory (/usr/local/apache2) so we don’t break the existing Apache installation and this way we can choose which Apache to run
Enter on the apache directory and run

./configure --prefix=/usr/local/apache2 --with-apr=/usr/local/apr/ --with-apr-util=/usr/local/apr-util/ --with-mpm=prefork --enable-ssl --disable-charset-lite --enable-mods-shared="proxy proxy_http proxy_connect headers mod_ssl" --disable-include --disable-env --enable-setenvif --disable-status --disable-autoindex --disable-asis --disable-cgi --disable-negotiation --disable-imap --disable-actions --disable-userdir --disable-alias

then generate a self-signed test certificate:

openssl req -new -x509 -days 30 -keyout /usr/local/apache2/conf/ssl.key/server.key -out /usr/local/apache2/conf/ssl.crt/server.crt -subj '/CN=Test-Only Certificate'

To avoid entering the password key each time we restart Apache Ive opted to remove it from the key

cp /usr/local/apache2/conf/ssl.key/server.key /usr/local/apache2/conf/ssl.key/server.key.org
openssl rsa -in /usr/local/apache2/conf/ssl.key/server.key.org -out /usr/local/apache2/conf/ssl.key/server.key

done

for testing the OCSP features its necessary to know the address of the OCSP responder, the server which our Apache will need to communicate to validate the client certificate.
In the best possible case, that address is hard-coded on the client certificate itself, on the “Authority Info Access” (AIA) extension. With this information MOD_SSL OCSP module will know where to find the relevant information just looking into the client Certificate.
The other possible configuration is to add manually the OCSP responder address to the MOD_SSL configuration, using the SSLOCSPDefaultResponder directive
You have to find that address first. Normally that information is on the Authority web pages, if not you have to contact them and ask for it.

I will create a brand new http.conf for testing and merging there the ssl information as well
edit the /usr/local/apache2/conf/http.conf file and add

------------------------------------------------------------------
# ================================================= # Basic settings # =========
========================================
LoadModule unixd_module modules/mod_unixd.so
LoadModule access_compat_module modules/mod_access_compat.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule authn_core_module modules/mod_authn_core.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule headers_module modules/mod_headers.so

User apache
Group apache
ServerAdmin user@xpto.com
ServerName server.name.com
UseCanonicalName Off
ServerSignature Off
HostnameLookups Off
ServerTokens Prod
ServerRoot "/usr/local/apache2"
DocumentRoot "/usr/local/apache2/htdocs"
PidFile /usr/local/apache2/logs/httpd.pid
ScoreBoardFile /usr/local/apache2/logs/httpd.scoreboard 

DirectoryIndex index.html 

 # ======== # HTTP and performance settings # =====================
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 30 

 MinSpareServers 5
 MaxSpareServers 10
 StartServers 5
 MaxClients 150
 MaxRequestsPerChild 0

# =========== # Access control # ================================== 

  Order allow,deny
  Allow from all  

# =========== # MIME encoding # =================================== 

 TypesConfig /usr/local/apache2/conf/mime.types 

#DefaultType text/plain

 AddEncoding x-compress
 AddEncoding x-gzip                  .gz .tgz
 AddType application/x-compress      .Z
 AddType application/x-gzip          .gz .tgz
 AddType application/x-tar           .tgz
 AddType application/x-x509-ca-cert  .crt
 AddType application/x-pkcs7-crl     .crl 

 #============== # Logs # ========================================
LogLevel debug
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
ErrorLog /usr/local/apache2/logs/error_log
CustomLog /usr/local/apache2/logs/access_log combined
CustomLog logs/ssl_request_log "%t %h %{HTTPS}x %{SSL_PROTOCOL}x %{SSL_CIPHER}x
%{SSL_CIPHER_USEKEYSIZE}x %{SSL_CLIENT_VERIFY}x \"%r\" %b"

 # =============== # SSL/TLS settings # ==========================
Listen 0.0.0.0:443
SSLEngine on
SSLOptions +StrictRequire 

 SSLRequireSSL 

SSLProtocol -all +TLSv1 +SSLv3
SSLCipherSuite HIGH:MEDIUM:!aNULL:+SHA1:+MD5:+HIGH:+MEDIUM
Mutex file:/usr/local/apache2/logs/ssl_mutex
SSLRandomSeed startup file:/dev/urandom 1024
SSLRandomSeed connect file:/dev/urandom 1024
SSLSessionCache shm:/usr/local/apache2/logs/ssl_cache_shm
SSLSessionCacheTimeout 600
SSLPassPhraseDialog builtin
SSLCertificateFile /usr/local/apache2/conf/ssl.crt/server.crt
SSLCertificateKeyFile /usr/local/apache2/conf/ssl.key/server.key
SSLCACertificateFile /etc/pki/tls/certs/ca-bundle.crt
SSLVerifyClient require
SSLVerifyDepth 1
SSLProxyEngine off
SSLOCSPEnable on

AddType application/x-x509-ca-cert      .crt
 AddType application/x-pkcs7-crl         .crl 

RequestHeader set SSL_CLIENT_S_DN    ""
RequestHeader set SSL_CLIENT_I_DN    ""
RequestHeader set SSL_SERVER_S_DN_OU ""
RequestHeader set SSL_CLIENT_VERIFY  ""

     # add all the SSL_* you need in the internal web application
     RequestHeader set SSL_CLIENT_S_DN "%{SSL_CLIENT_S_DN}s"
     RequestHeader set SSL_CLIENT_I_DN "%{SSL_CLIENT_I_DN}s"
     RequestHeader set SSL_SERVER_S_DN_OU "%{SSL_SERVER_S_DN_OU}s"
     RequestHeader set SSL_CLIENT_VERIFY "%{SSL_CLIENT_VERIFY}s"

     ProxyPass          http://internal.server.net/
     ProxyPassReverse   http://internal.server.net/

SetEnvIf User-Agent ".*MSIE.*" \     nokeepalive ssl-unclean-shutdown \  downgrade-1.0 force-response-1.0

restart Apache and test it using a web browser to access the server
surprise…no luck. Apache is blocking access to the the web pages

checking the /usr/local/apache2/logs/error_log file I see this errors:

[debug] [pid 10312] ssl_engine_ocsp.c(78):
[client x.y.z.50:43894] no OCSP responder specified in certificate and no default configured
[error] [pid 10312] [client x.y.z.50:43894] Certificate Verification: Error (50): application verification failure
[info] [pid 10312] [client x.y.z.50:43894] SSL library error 1 in handshake (server web.server.net:443)
[info] [pid 10312] SSL Library Error: error:140890B2:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:no certificate returned
[info] [pid 10312] [client x.y.z.50:43894] Connection closed to child 1 with abortive shutdown (server web.server.net:443)

This is a bit strange error as the certificate that is being sent to the server has the proper AIA extensions so nor I don’t understand why the server is not using them. After a lot of Web crawling and posting on forums I couldn’t find any solution for this, so decided, as a workaround, to add the following additional OCSP directive do Apache: (after SSLOCSPEnable on)

SSLOCSPDefaultResponder http://ocsp.auc.cartaodecidadao.pt/publico/ocsp

restart Apache and test again.
Still no luck. This time the OCSP responder is contacted but MOD_SSL is returning “unable to get local issuer certificate” error:

[debug] [pid 10145] ssl_util_ocsp.c(79):  [client x.y.z.50:45551] connecting to OCSP responder 'ocsp.auc.cartaodecidadao.pt'
[debug] [pid 10145] ssl_util_ocsp.c(105): [client x.y.z.50:45551] sending request to OCSP responder
[debug] [pid 10145] ssl_util_ocsp.c(209): [client x.y.z.50:45551] OCSP response header: Date: Fri, 16 Jul 2010 08:31:30 GMT
[debug] [pid 10145] ssl_util_ocsp.c(209): [client x.y.z.50:45551] OCSP response header: Server: Apache
[debug] [pid 10145] ssl_util_ocsp.c(209): [client x.y.z.50:45551] OCSP response header: X-Powered-By: Servlet 2.4; JBoss-4.0.5.GA (build: CVSTag=Branch_4_0 date=200610162339)/Tomcat-5.5
[debug] [pid 10145] ssl_util_ocsp.c(209): [client x.y.z.50:45551] OCSP response header: Expires: Fri, 16 Jul 2010 08:33:30 GMT
[debug] [pid 10145] ssl_util_ocsp.c(209): [client x.y.z.50:45551] OCSP response header: Cache-Control: max-age=120
[debug] [pid 10145] ssl_util_ocsp.c(209): [client x.y.z.50:45551] OCSP response header: Content-Length: 2530
[debug] [pid 10145] ssl_util_ocsp.c(209): [client x.y.z.50:45551] OCSP response header: Connection: close
[debug] [pid 10145] ssl_util_ocsp.c(209): [client x.y.z.50:45551] OCSP response header: Content-Type: application/ocsp-response
[debug] [pid 10145] ssl_util_ocsp.c(252): [client x.y.z.50:45551] OCSP response: got 2530 bytes, 2530 total
[debug] [pid 10145] ssl_util_ocsp.c(235): [client x.y.z.50:45551] OCSP response: got EOF
[error] [pid 10145] SSL Library Error: error:27069065:OCSP routines:OCSP_basic_verify:certificate verify error (Verify error:unable to get local issuer certificate)
[error] [pid 10145] failed to verify the OCSP response
[error] [pid 10145] [client 10.14.148.50:45551] Certificate Verification: Error (50): application verification failure
[info] [pid 10145]  [client 10.14.148.50:45551] SSL library error 1 in handshake (server web.server.net:443)
[info] [pid 10145]  SSL Library Error: error:140890B2:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:no certificate returned
[info] [pid 10145]  [client 10.14.148.50:45551] Connection closed to child 1 with abortive shutdown (server web.server.net:443)

on the open-ssl mailing list Ive got very good tips related to this error:

“For each certificate do this:
openssl x509 -in cert.pem -subject -issuer -noout

The subject of the one you pass to -issuer should match the issuer of the one
you pass to cert. You need a root CA and the rest of the chain passed to
-CApath.”

and

“Your certificate chain needs to be complete. That is it has to include the
root CA (one with issuer and subject the same) and all intermediate
certificates of the responder certificate.”

My Certificate chain was indeed incomplete, there was no ROOT certificate in /etc/pki/tls/certs/ca-bundle.crt, Ive checked the “subject” and “issuer” for each certificate Ive downloaded from the CA web pages and Ive missed the one which had the “issuer” equal to the “subject” one, the ROOT Certificate.

added the missing certificate do ca-bundle.crt, restarted Apache and tried again to access the web server

error again!
checking out on the Apache logs once more:

[debug] [pid 21783] ssl_util_ocsp.c(79):  [client x.y.z.50:54752] connecting to OCSP responder 'ocsp.auc.cartaodecidadao.pt'
[debug] [pid 21783] ssl_util_ocsp.c(105): [client x.y.z.50:54752] sending request to OCSP responder
[debug] [pid 21783] ssl_util_ocsp.c(209): [client x.y.z.50:54752] OCSP response header: Date: Fri, 16 Jul 2010 14:51:24 GMT
[debug] [pid 21783] ssl_util_ocsp.c(209): [client x.y.z.50:54752] OCSP response header: Server: Apache
[debug] [pid 21783] ssl_util_ocsp.c(209): [client x.y.z.50:54752] OCSP response header: X-Powered-By: Servlet 2.4; JBoss-4.0.5.GA
 (build: CVSTag=Branch_4_0 date=200610162339)/Tomcat-5.5
[debug] [pid 21783] ssl_util_ocsp.c(209): [client x.y.z.50:54752] OCSP response header: Expires: Fri, 16 Jul 2010 14:53:24 GMT
[debug] [pid 21783] ssl_util_ocsp.c(209): [client x.y.z.50:54752] OCSP response header: Cache-Control: max-age=120
[debug] [pid 21783] ssl_util_ocsp.c(209): [client x.y.z.50:54752] OCSP response header: Content-Length: 2530
[debug] [pid 21783] ssl_util_ocsp.c(209): [client x.y.z.50:54752] OCSP response header: Connection: close
[debug] [pid 21783] ssl_util_ocsp.c(209): [client x.y.z.50:54752] OCSP response header: Content-Type: application/ocsp-response
[debug] [pid 21783] ssl_util_ocsp.c(252): [client x.y.z.50:54752] OCSP response: got 1127 bytes, 1127 total
[debug] [pid 21783] ssl_util_ocsp.c(252): [client x.y.z.50:54752] OCSP response: got 1403 bytes, 2530 total
[debug] [pid 21783] ssl_util_ocsp.c(235): [client x.y.z.50:54752] OCSP response: got EOF
[error] [pid 21783] SSL Library Error: error:27069070:OCSP routines:OCSP_basic_verify:root ca not trusted
[error] [pid 21783] failed to verify the OCSP response
[error] [pid 21783] [client x.y.z.50:54752] Certificate Verification: Error (50): application verification failure
[info] [pid 21783] [client x.y.z.50:54752] SSL library error 1 in handshake (server web.server.net:443)
[info] [pid 21783] SSL Library Error: error:140890B2:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:no certificate returned
[info] [pid 21783] [client x.y.z.50:54752] Connection closed to child 1 with abortive shutdown (server web.server.net:443)

“root ca not trusted”?
This time I didnt find any clue to how to solve this.

Just for curiosity, instead of using mod_ssl to check certificates using an OCSP responder I am now going to use the openssl command line to do the same check.
I have exported my client cerfificate using the the web browser to lneves.der, uploaded it to the web server and converted to the PEM format using

openssl x509 -in lneves.der -inform DER -out lneves.pem -outform PEM

Now used this command line to make an OCSP query do the specified responder:

openssl ocsp -issuer /etc/pki/tls/certs/CC0003.pem -cert
lneves.pem -url http://ocsp.auc.cartaodecidadao.pt/publico/ocsp
-CAfile /etc/pki/tls/certs/ca-bundle.crt -resp_text

Response verify OK
/home/oracle/lneves.pem: good

success… it works, openssl validates my client certificate without problems.
MOD_SSL not. At this point I was blocked and without any ideia on how to progress on this configuration
I dont know if I was doing something severily wrong here, or if the certificates are kind of broken, or if MOD_SSL OCSP code is not mature enought but the truth is that this is not working as I expected.

What alternatives do I have now?

Part 3: Is the certificate still valid?

Suppose one of our clients have some problems on its certificate and it gets invalidated by its certification authority entity
In our current Web server configuration we are only testing the certificates for expiration dates, so the user with a revogated certificate will still be able to access the server, and this is not correct, we must avoid this someway.

One way to control revogated certificates in the web server is through the usage of Certification Revogation Lists (CRL)
A CRL is a list containing the serial numbers of all certificates issued by a given certification authority (CA) that have been revoked and have not yet expired. A client (relying party) wishing to make use of the information in a certificate must first validate the certificate. In order to validate a certificate, the relying party must, among other things, acquire a recently issued CRL in order to determine whether the certificate has been revoked. Once the client has obtained a CRL, that CRL may be cached for use in future validations. However, after a certain point, a newer CRL must be obtained in order to ensure that validations are based on up-to-date certificate status information

CRL files can get quite big in size, and it can become unpractical to download it periodically to our server, as it grows in size it will take more resources on the network infrastructure, more resources on the server, more time spent downloading, and there’s more probabilities of the file becoming corrupt because of failed transmissions

To try to overcome all this problems there’s another kind of CRL lists, a Delta CRL

A delta-CRL is a CRL that only provides information about certificates whose statuses have changed since the issuance of a specific, previously issued CRL. A client in need of more up-to-date certificate status information, that has already obtained a copy of the previously issued CRL, can download the latest delta-CRL instead of downloading the latest full CRL. Since delta-CRLs tend to be significantly smaller than full CRLs, this will tend to reduce the load on the repository and improve the response time for the client

to use a CRL on our server we need to download it from the Certificate Authority pages, normally the URL where to find the Certificate is a attribute on the certificate itself. Open once more your certificate details (On IE 6.0.2900: Tools -> Internet Options -> Content -> Certificates -> “Personal” tab In Firefox 3.6.8 go to File -> Preferences -> Advanced -> Encryption -> View Certificates) and find the “CRL Distribution Point” attribute. It can exist also another attribute called “freshest CRL” where we can find the URL for downloading the most recent Delta CRL for this certificate

Because CRLs can be published as a text file over HTTP, they can be manually retrieved from the CA using a tool like wget. wget can be used to retrieve any type of published CRL:

*Full CRLs. For example:
wget --no-check-certificate -d https://server.example.com/ca/ee/ca/crl/LatestCRL.crl

*Delta CRLs. For example:
wget --no-check-certificate -N -d https://server.example.com/ca/ee/ca/crl/LatestCRL.crl

The parameters used are the following
no argument     Retrieves the full CRL.
-N     Retrieves the CRL that is newer than the local copy (delta CRL).
-c     Retrieves a partially-downloaded file.
–no-check-certificate     Skips SSL for the connection, so it is not necessary to configure SSL between the host and client.
-d     Prints debug information.

For using CRL lists on Apache we will need to set the SSLCARevocationPath directive on the /etc/httpd/conf.d/ssl.conf file to the filesystem path where to find the CRL files. The SSLCARevocationFile directive can be used as well instead of SSLCARevocationPath but it is expecting a single file where all CRL are joined together in contrast with SSLCARevocationPath which points to a directory where lives individual CRL files.

Example

SSLCARevocationPath /etc/httpd/conf/ssl.crl/

Apache expects to find that files in PEM format so after downloading them they must be converted from the downloaded type (usually DER format) to PEM. We can use this command for this conversion

openssl crl -inform DER -in /etc/httpd/conf/ssl.crl/LatestCRL.crl -outform PEM -out /etc/httpd/conf/ssl.crl/LatestCRL.pem

finally we must create hashes for all the files contained in SSLCARevocationPath, this is a command for a single file:

ln -s /etc/httpd/conf/ssl.crl/LatestCRL.pem `openssl crl -hash -noout -in /etc/httpd/conf/ssl.crl/LatestCRL.pem`.r0

if using SSLCARevocationFile directive instead, we dont need to create hashes for the files but we do need to concatenate all the PEM to a single one, using for example

cat *.pem > allcrls.pem and them
SSLCARevocationPath /etc/httpd/conf/ssl.crl/allcrl.pem

Did we finished? no… after updatind CRL files Apache must be restarted so it can use the most recent changes

In my opinion, this is all a cumbersome process with multiple points of failure. This is a typical scenario to be helped with an automated script. There are an example of such a script at http://wiki.stewingwithmitch.com/wiki/index.php?title=GetCRL.sh

Theres yet another problem, if the CRL distribution points are not fixed, like for instance when using Delta CRLs, we must keep updating the script so it includes the most recent URI for the CRLs. Theres still too much manual intervention involved in this process.

There must be an easier way to deal with certification revogation testing without so many steps.
Yes it is, see the next part where I will try to easy this process… but you will see that it will be not as easy at it seems

Part 2: forwarding to internal server

Now that we can access our test page after a client authentication and  have our communications encrypted via SSL lets take another step. Our goal is to give access to some internal pages via the intermediate machine, so we need to forward all external client requests to the internal server.

We must use the reverse proxy feature of Apache, which is activated using the ProxyPass directive. Add this in your ssl.conf file, before the  </VirtualHost> tag:

ProxyPass        /      http://internal.server.net/
ProxyPassReverse /      http://internal.server.net/

ProxyPassreverse will let Apache adjust the URL in the Location header on HTTP redirect responses. For instance this is essential when Apache is used as a reverse proxy to avoid by-passing the reverse proxy because of HTTP redirects on the backend servers which stay behind the reverse proxy.

Suppose our local server has address http://yourservername/; then our two above lines will not only cause a local request for the <http://yourservername/index.html&gt; to be internally converted into a proxy request to <http://internal.server.net/index.html&gt; (the functionality ProxyPass provides here). It also takes care of redirects the server internal.server sends: when http://internalserver/index.html is redirected by him to http://internal.server.net/something/ Apache adjusts this to http://yourservername/something/ before forwarding the HTTP redirect response to the client.

Now we have created a new problem, as our machine is transforming the client requests and making new ones to our internal server, all the client certificate data that we are willing to use on our internal applications are missing, as for security reasons they are not being forwarded with the proxypass directives.

No worries, we can use RequestHeader directives to control and modify HTTP request and response headers. We can use it to send any variable value, inclusive the SSL related ones

# initialize the special headers to a blank value to avoid http header forgeries
   RequestHeader set SSL_CLIENT_S_DN    ""
   RequestHeader set SSL_CLIENT_I_DN    ""
   RequestHeader set SSL_SERVER_S_DN_OU ""
   RequestHeader set SSL_CLIENT_VERIFY  ""

   # add all the SSL_* you need in the internal web application
   RequestHeader set SSL_CLIENT_S_DN "%{SSL_CLIENT_S_DN}s"
   RequestHeader set SSL_CLIENT_I_DN "%{SSL_CLIENT_I_DN}s"
   RequestHeader set SSL_SERVER_S_DN_OU "%{SSL_SERVER_S_DN_OU}s"
   RequestHeader set SSL_CLIENT_VERIFY "%{SSL_CLIENT_VERIFY}s"

   ProxyPass        /  http://internal.server.net/
   ProxyPassReverse /  http://internal.server.net/

It seems our configuration is now finished, but not yet! Theres something missing here and a very important one. In this actual configuration the server  checks the expiration dates of the client certificates and based on this it accepts or not the certificate. But thats not enough

Part 1

The proxy machine will be a CentOS one that will stay on the DMZ of the network, and we will need just a default Apache installation, so the requisite of being free is easily accomplished

First step (in no particular order):
Install your certificate in the machine you will use for testing. If it comes in a smart-card you should run some software that “registers” the card on the browser

On my IE 6.0.2900 I can confirm it in Tools -> Internet Options -> Content -> Certificates -> “Personal” tab

In Firefox 3.6.8 go to File -> Preferences -> Advanced -> Encryption -> View Certificates

Check there that you can see the registered certificate, select it and view its details

Take a look the Certificate Path tab, this shows the Certificate Authority (CA) that issued your certificate/private Key and all the parent CAs all way up until the root one

All the CAs following the master root CA are called intermediate CAs. My root CA is GTE Cybertrust  and then have more 3 intermediate CAs. Take note of this total (4) as it will be needed soon.

Edit /etc/httpd/conf.d/ssl.conf and uncomment or add  this lines:

ErrorLog logs/ssl_error_log
LogLevel debug

SSLEngine on

SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
SSLCACertificateFile /etc/pki/tls/certs/ca-bundle.crt
SSLVerifyClient require
SSLVerifyDepth 4

CustomLog logs/ssl_request_log \
"%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

The first and last group of commands is for helping on debugging issues, so always keep an eye on /var/log/httpd files when looking for problems (or solutions!)

SSLEngine on

is a requirement to be able to use SSL at all

SSLCertificateFile and SSLCertificateKeyFile must point to the server certificate and private key respectively

If Apache was automatically installed with the OS installation or if it was installed using some package manager then its highly provable that there’s a self-signed test server certificate and his corresponding private key already generated for you to use. So check if /etc/pki/tls/certs/localhost.crt and /etc/pki/tls/private/localhost.key really exists.

If they don’t, or if you want a new server certificate plus key (self-signed, just for testing) you can use Openssl utilities to generate them. Use for example

openssl req -new -x509 -days 30 -keyout  /etc/pki/tls/certs/server.key -out /etc/pki/tls/certs/server.crt -subj  '/CN=Test-Only Certificate'

to create them.
To avoid entering a password each time you restart Apache take the password out of the key using for example

cp /etc/pki/tls/certs/server.key /etc/pki/tls/certs/server.key.org
openssl rsa -in /etc/pki/tls/certs/server.key.org -out /etc/pki/tls/certs/server.key

SSLCACertificateFile

points to the all-in-one file where you can assemble the Certificates of  CAs you deal with. These are used for Remote Server Authentication. Such a file is simply the concatenation of the various PEM (Privacy Enhanced Mail) encoded Certificate files, in order of preference

This individual Certificate files must be downloaded from the CA where you obtained your smart-card. Ask for them if you cant find them on their pages. This files sometimes are encoded in other methods, such as DER format instead of PEM, but this is not problem at all because you can use openssl to convert between all formats.

Some common ones:

• .pem – (Privacy Enhanced Mail) Base64 encoded DER certificate, enclosed between “—–BEGIN CERTIFICATE—–” and “—–END CERTIFICATE—–“
• .cer, .crt, .der – usually in binary DER form, but Base64-encoded certificates are common too (see .pem above)
• .p7b, .p7c – PKCS#7 SignedData structure without data, just certificate(s) or CRL(s)
• .p12 – PKCS#12, may contain certificate(s) (public) and private keys (password protected)

Apache installations comes with a ca-bundle.crt file containing all major Root CA Certificates. You should append at the end of this file the PEM Certificates you downloaded, or even better than that, create a new .crt file (with whichever name you want to give it) just with the necessary PEM Certificates. If you choose to use all of the original file, and if you don’t take any additional precautions, you will be literally giving permissions to access your protected pages to any user with a valid  smart-card from any CA in the world!

SSLVerifyClient require

line is the one who activates the client authentication request.

SSLVerifyDepth

This directive sets how deeply the ssl engine should verify before deciding that the clients don’t have a valid certificate Use the value you stored above (4 in my case). The depth actually is the maximum number of intermediate certificate issuers, i.e. the number of CA certificates which are max allowed to be followed while verifying the client certificate. A depth of 0 means that self-signed client certificates are accepted only, the default depth of 1 means the client certificate can be self-signed or has to be signed by a CA which is directly known to the server

now, lets test what we have so far!

Create some html page on the DocumentRoot of your Apache, for ex:

vi /var/www/html/index.html

<!DOCTYPE html PUBLIC “-//IETF//DTD HTML 2.0//EN”>
<HTML>
<HEAD>
<TITLE>
A Small Hello
</TITLE>
</HEAD>
<BODY>
<H1>Hi</H1>
</BODY>
</HTML>

Now open a browser window and access https://yourservername/index.html
After some moment a window asking for a certificate should appear. Choose the correct one from the list (if you have multiple certificates installed) and hit OK, after that, a PIN window will appear  and you must enter the password for you smart-card private key (You know the PIN don’t you?)

If everything goes fine, you should be granted access to the index.html page, if not, take a look at the apache error files and try to fix it

For guidance, a complete working sample ssl.conf is like this one:

LoadModule ssl_module modules/mod_ssl.so
Listen 443
AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl    .crl
SSLPassPhraseDialog  builtin
SSLSessionCache         shmcb:/var/cache/mod_ssl/scache(512000)
SSLSessionCacheTimeout  300
SSLMutex default
SSLRandomSeed startup file:/dev/urandom  256
SSLRandomSeed connect builtin
SSLCryptoDevice builtin

ErrorLog logs/ssl_error_log
TransferLog logs/ssl_access_log
LogLevel debug
SSLEngine on
SSLProtocol all -SSLv2
SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW
SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
SSLCACertificateFile /etc/pki/tls/certs/ca-bundle.crt
SSLVerifyClient require
SSLVerifyDepth 4

SSLOptions +StdEnvVars

SSLOptions +StdEnvVars

SetEnvIf User-Agent ".*MSIE.*" \
nokeepalive ssl-unclean-shutdown \
downgrade-1.0 force-response-1.0
CustomLog logs/ssl_request_log \
"%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

Use authentication and a reverse proxy to protect your sites

You have some internal sites that you need to give access from the outside, not any kind of access but a secure one. You want to encrypt the communication from/to your site and you want to authenticate the clients as well. Your clients will have to use their smart-card to be able to gain access

You are being asked to use free tools for this, and you don’t have any clue if this is possible or not.