ipa-healthcheck CA errors are often timeouts, not expired certificates

Running ipa-healthcheck --failures-only on my FreeIPA primary (idm.gnali.home, RHEL 10.2, IPA 4.13.1) reported two ERROR results against the Dogtag CA subsystem:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

{
  "source": "pki.server.healthcheck.certs.expiration",
  "check": "CASystemCertExpiryCheck",
  "result": "ERROR",
  "kw": {
    "exception": "Request timed out"
  }
},
{
  "source": "pki.server.healthcheck.certs.trustflags",
  "check": "CASystemCertTrustFlagCheck",
  "result": "ERROR",
  "kw": {
    "key": "ocsp_signing",
    "nssdbDir": "/var/lib/pki/pki-tomcat/conf/alias",
    "msg": "Unable to load cert from NSSDB: Request timed out"
  }
}

The wording sounds alarming — certificate expiry and trust flags are core PKI concerns. In this case, the CA was healthy. The healthcheck simply ran out of time while reading the NSS database.

What the errors actually mean

Both checks inspect system certificates stored in the Dogtag NSS database at /var/lib/pki/pki-tomcat/conf/alias. To do that, ipa-healthcheck shells out to the pki CLI (nss-cert-export, nss-cert-show) for each CA system certificate: signing, OCSP, audit, subsystem, and SSL server.

The keyword in the failure output is not expired or invalid — it is Request timed out.

ipa-healthcheck enforces a per-check timeout. The default is 10 seconds, defined in ipahealthcheck.core.constants and configurable in /etc/ipahealthcheck/ipahealthcheck.conf.

The CA expiry check walks through several certificates in a single plugin run. If the cumulative NSS operations exceed 10 seconds, the alarm fires and the check reports ERROR with a generic timeout exception — even when every certificate is valid.

Why NSS access is slow

While pki-tomcat is running, the Java process holds the NSS softoken database open. Concurrent access from pki or certutil can block on PKCS #11 locks.

On my server, pki-tomcat was active and the NSS directory was in use by the Java process. Under contention, individual pki nss-cert-export calls were slow:

A manual benchmark made the pattern obvious: the first export of Server-Cert cert-pki-ca after a healthcheck run took 87 seconds. Subsequent exports completed in about 1.5 seconds. That is classic lock contention, not corruption.

Meanwhile, direct inspection showed the certificates were fine:

1
2
3
4

certutil -L -d /var/lib/pki/pki-tomcat/conf/alias
pki -d /var/lib/pki/pki-tomcat/conf/alias \
    -f /var/lib/pki/pki-tomcat/conf/password.conf \
    nss-cert-export --format PEM "Server-Cert cert-pki-ca"

All five CA system certs were present with sensible validity dates. pki-server status reported the CA subsystem as enabled and reachable on ports 8080 and 8443.

Confirming with debug output

Running a single check with --debug shows where time is spent:

1
2
3
4

ipa-healthcheck \
  --source pki.server.healthcheck.certs.expiration \
  --check CASystemCertExpiryCheck \
  --debug

In my case, signing and ocsp_signing returned SUCCESS after 5 s and 8.8 s respectively. The next certificate (sslserver / Server-Cert cert-pki-ca) never finished before the 10-second alarm.

The trust-flag check failed on the same bottleneck — loading ocsp_signing from the NSSDB while tomcat held the lock.

Fix: raise the healthcheck timeout

The stock config file only contains a [default] section with no explicit timeout. Add one:

1
2
3

# /etc/ipahealthcheck/ipahealthcheck.conf
[default]
timeout = 120

There is no --timeout CLI flag; the value must live in the config file (or be passed via --config pointing to an alternate file).

After setting timeout = 120, the CA certificate checks passed. The full run took about 66 seconds — well within the new limit, but far beyond the default 10 seconds.

Verify:

1

ipa-healthcheck --failures-only

Takeaways

• ERROR from CASystemCertExpiryCheck does not automatically mean an expiring CA certificate. Read the kw field. exception: Request timed out points to the healthcheck deadline, not PKIX validity.
• The Dogtag NSS database is slow to read while pki-tomcat is running. This is a tooling contention issue, not a sign that the CA is broken.
• Raising timeout in ipahealthcheck.conf is a reasonable fix for production IPA servers where restarting tomcat before every healthcheck is not practical.

If you see the same errors, check the certificates manually first. If they look good and the failures mention timeouts, you are probably in the same situation — not a PKI incident, just a healthcheck that needs more time.