First thing's first. Let's make sure that we have a valid STARTTLS tag listed in the connect banner. Connect to the server and issue an EHLO command as follows :
shell> telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.localdomain (127.0.0.1).
Escape character is '^]'.
220 mail.example.com ESMTP
EHLO testing
250-mail.example.com
250-STARTTLS
250-PIPELINING
250-8BITMIME
250-SIZE 0
250 AUTH LOGIN PLAIN CRAM-MD5
If you see the STARTTLS line, then your server is broadcasting that capability. Next, let's try connecting via openssl and initiate a TLS connection :
shell> openssl s_client -starttls smtp -crlf -connect localhost:25
CONNECTED(00000003)
depth=0 /C=US/ST=MyState/L=MyCity/O=MyCompany/CN=mail.example.com/emailAddress=admin@example.com
verify error:num=18:self signed certificate
verify return:1
depth=0 /C=US/ST=MyState/L=MyCity/O=MyCompany/CN=mail.example.com/emailAddress=admin@example.com
verify return:1
---
Certificate chain
0 s:/C=US/ST=MyState/L=MyCity/O=MyCompany/CN=mail.example.com/emailAddress=admin@example.com
i:/C=US/ST=MyState/L=MyCity/O=MyCompany/CN=mail.example.com/emailAddress=admin@example.com
---
Server certificate
-----BEGIN CERTIFICATE-----
<Certificate snipped for brevity>
-----END CERTIFICATE-----
subject=/C=US/ST=MyState/L=MyCity/O=MyCompany/CN=mail.example.com/emailAddress=admin@example.com
issuer=/C=US/ST=MyState/L=MyCity/O=MyCompany/CN=mail.example.com/emailAddress=admin@example.com
---
No client certificate CA names sent
---
SSL handshake has read 1520 bytes and written 350 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 1024 bit
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1
Cipher : DHE-RSA-AES256-SHA
Session-ID: <snip>
Session-ID-ctx:
Master-Key: <snip>
Key-Arg : None
Krb5 Principal: None
Start Time: 1180580553
Timeout : 300 (sec)
Verify return code: 18 (self signed certificate)
---
220 mail.example.com ESMTP
QUIT
DONE
NOTE : If an excessive amount of time passes between the openssl command being sent and finally seeing the output above, then the temporary certificates are likely the cause. update_tmprsadh needs to be run on a nightly basis to re-generate the temporary certificates. Additionally, they need to have the correct ownership and permissions.