Cassandra TLS certificate verification failure

You're viewing Apigee and Apigee hybrid documentation.
View Apigee Edge documentation.

Symptoms

The TLS certificate verification process in Cassandra pods could fail with an error similar to the following: 

Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:456) at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:323) at sun.security.validator.Validator.validate(Validator.java:271) at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:315) at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:278) at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141) at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:261) at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:698) ... Suppressed: javax.net.ssl.SSLHandshakeException: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1309) at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1270) ...

Possible Causes

Cause Description Troubleshooting instructions applicable for
Apigee CA certificate does not match across Apigee clusters If the Apigee CA certificate does not match across clusters, then TLS certificate verification in Cassandra could fail. Apigee hybrid

Prerequisites

  • kubectl needs to be installed and configured for accessing Apigee clusters.
  • jq is required for formatting JSON content.
  • keytool is required for printing TLS certificates.
  • cmctl is required for reissuing certificates using Cert Manager.

Cause: Apigee CA certificate does not match across Apigee clusters

Diagnosis

  1. Read apigee-ca secret and print Apigee CA certificate of all clusters using the following command: 
    kubectl -n cert-manager get secret apigee-ca -o json | jq -r '.data["tls.crt"]' | base64 -d | keytool -printcert | grep Version -B 10

    An example output:

    kubectl -n cert-manager get secret apigee-ca -o json | jq -r '.data["tls.crt"]' | base64 -d | keytool -printcert | grep Version -B 10 Owner: CN=apigee-hybrid, O=apigee + O=cluster.local Issuer: CN=apigee-hybrid, O=apigee + O=cluster.local Serial number: afcc2ef957cebfd52b118b0b1622021 Valid from: Wed Oct 30 03:09:23 UTC 2024 until: Sat Oct 28 03:09:23 UTC 2034 Certificate fingerprints:  SHA1: 32:D9:77:54:B1:FC:CB:6C:9E:28:C1:04:25:49:0D:F5:7C:88:A5:6C  SHA256: 7C:97:31:3B:56:CD:A3:EF:88:A7:DC:17:AE:D5:F2:A7:6A:48:B3:FC:AA:04:F0:6B:9F:43:C4:5C:6E:15:DE:37 Signature algorithm name: SHA256withECDSA Subject Public Key Algorithm: 256-bit EC (secp256r1) key Version: 3
  2. Read apigee-cassandra-default-tls secret and verify whether the above Apigee CA certificate has been used when generating the Cassandra certificate. The apigee-cassandra-default-tls secret contains the Apigee CA certificate under ca.crt: 
    kubectl -n apigee get secret apigee-cassandra-default-tls -o json | jq -r '.data["ca.crt"]' | base64 -d | keytool -printcert | grep Version -B 10

    An example output:

    kubectl -n apigee get secret apigee-cassandra-default-tls -o json | jq -r '.data["ca.crt"]' | base64 -d | keytool -printcert | grep Version -B 10 Owner: CN=apigee-hybrid, O=apigee + O=cluster.local Issuer: CN=apigee-hybrid, O=apigee + O=cluster.local Serial number: afcc2ef957cebfd52b118b0b1622021 Valid from: Wed Oct 30 03:09:23 UTC 2024 until: Sat Oct 28 03:09:23 UTC 2034 Certificate fingerprints:  SHA1: 32:D9:77:54:B1:FC:CB:6C:9E:28:C1:04:25:49:0D:F5:7C:88:A5:6C  SHA256: 7C:97:31:3B:56:CD:A3:EF:88:A7:DC:17:AE:D5:F2:A7:6A:48:B3:FC:AA:04:F0:6B:9F:43:C4:5C:6E:15:DE:37 Signature algorithm name: SHA256withECDSA Subject Public Key Algorithm: 256-bit EC (secp256r1) key Version: 3
  3. In the above example, the serial number of the Apigee CA certificate found in the apigee-ca secret matches with the Apigee CA certificate found in the apigee-cassandra-default-tls secret: afcc2ef957cebfd52b118b0b1622021. This confirms that the Cassandra certificate has been signed by the same Apigee CA certificate. We could verify this further by following the steps below.
  4. Extract Apigee CA certificate pem file: 
    kubectl -n cert-manager get secret apigee-ca -o json | jq -r '.data["tls.crt"]' | base64 -d > apigee-ca.crt

    An example output:

    kubectl -n cert-manager get secret apigee-ca -o json | jq -r '.data["tls.crt"]' | base64 -d > apigee-ca.crt cat apigee-ca.crt -----BEGIN CERTIFICATE----- MIIBvjCCAWSgAwIBAgIQCvzC75V86/1SsRiwsWIgITAKBggqhkjOPQQDAjA/MSUw DQYDVQQKEwZhcGlnZWUwFAYDVQQKEw1jbHVzdGVyLmxvY2FsMRYwFAYDVQQDEw1h cGlnZWUtaHlicmlkMB4XDTI0MTAzMDAzMDkyM1oXDTM0MTAyODAzMDkyM1owPzEl MA0GA1UEChMGYXBpZ2VlMBQGA1UEChMNY2x1c3Rlci5sb2NhbDEWMBQGA1UEAxMN YXBpZ2VlLWh5YnJpZDBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABNSow7pxNvjj R/jV66nY/w/tn22tu7oXyZS8tAFBnP7D2fFfIdk4tJub3gw/CsoyNa1cKXwAt7Tw SLp1iGJ3CY+jQjBAMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0G A1UdDgQWBBRSjN/cNNbg2kvmddskzdurglxuwTAKBggqhkjOPQQDAgNIADBFAiBp pCgNNC8TVEgF8jR5RK9dXZJRcNY39nFY4DqbH6bUJwIhAPdzx5gee3BIWYwlQAYX CgtCf4blLNq3KlBWTO993XoY -----END CERTIFICATE-----
  5. Extract Cassandra certificate pem file: 
    kubectl -n apigee get secrets apigee-cassandra-default-tls -o json | jq -r '.data["tls.crt"]' | base64 -d > apigee-cassandra-default-tls.crt

    An example output:

    kubectl -n apigee get secrets apigee-cassandra-default-tls -o json | jq -r '.data["tls.crt"]' | base64 -d > apigee-cassandra-default-tls.crt cat apigee-cassandra-default-tls.crt -----BEGIN CERTIFICATE----- MIIDSDCCAu6gAwIBAgIQZcYk/VOfGUQEzpLbAvyyNjAKBggqhkjOPQQDAjA/MSUw DQYDVQQKEwZhcGlnZWUwFAYDVQQKEw1jbHVzdGVyLmxvY2FsMRYwFAYDVQQDEw1h cGlnZWUtaHlicmlkMB4XDTI0MTAzMDAzMTAyMFoXDTM0MTAyODAzMTAyMFowPDE6 MDgGA1UEAxMxYXBpZ2VlLWNhc3NhbmRyYS1kZWZhdWx0LmFwaWdlZS5zdmMuY2x1 c3Rlci5sb2NhbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAM6k8YyB m/AV9cgexU8fZ4OFw8M72oxWEF44sFezZB7NpCqIFBxAM/7iL0tF2qU4S4gpcabD bn30fKKID8651Kytc7KHGT13Nlj9vQRjd0HJD8Qa8YtRcmGKtp+1fbQOcMPxvuNA CzaQyuPwieYKc6D9DpDDkPPCmjVwfaxHmNpdswrt0NQbSecg/xZPXbpzOZ6bUFha 2vTvSTomiDKIPGhWrMnEMJDjFyjpdYND74HnYgw1XGnC4SQNts/kvXligbVmW+Rz oyV7n99eN6cE5J/FHDgiHrBRZUw8ujP2l/p7Y96NcMBnXCsQu6RsCDltXqX1f1pG sIjUAFAZZvM0pDECAwEAAaOCAQIwgf8wDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQW MBQGCCsGAQUFBwMCBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaA FFKM39w01uDaS+Z12yTN26uCXG7BMIGeBgNVHREEgZYwgZOCGGFwaWdlZS1jYXNz YW5kcmEtZGVmYXVsdIIfYXBpZ2VlLWNhc3NhbmRyYS1kZWZhdWx0LmFwaWdlZYIj YXBpZ2VlLWNhc3NhbmRyYS1kZWZhdWx0LmFwaWdlZS5zdmOCMWFwaWdlZS1jYXNz YW5kcmEtZGVmYXVsdC5hcGlnZWUuc3ZjLmNsdXN0ZXIubG9jYWwwCgYIKoZIzj0E AwIDSAAwRQIhANt7WYfSbS4a14Fvf3IXcG+/p3iEGg61suK8jOxtgJMyAiBG3z7Y kgR7SWNzSoom4Oznq9NSub7v75kfQJFKEtP0Mg== -----END CERTIFICATE-----
  6. Verify Cassandra certificate using Apigee CA certificate: 
    openssl verify -CAfile apigee-ca.crt apigee-cassandra-default-tls.crt

    A successful example output:

    openssl verify -CAfile apigee-ca.crt apigee-cassandra-default-tls.crt apigee-cassandra-default-tls.crt: OK

    A failure example output:

    openssl verify -CAfile apigee-ca.crt apigee-cassandra-default-tls.crt CN = apigee-cassandra-default.apigee.svc.cluster.local error 20 at 0 depth lookup: unable to get local issuer certificate error apigee-cassandra-default-tls.crt: verification failed

Resolution

  1. Select an Apigee cluster which has the correct Apigee CA certificate.
  2. Export Apigee CA certificate secret from that cluster to a file: 
    kubectl -n cert-manager get secret apigee-ca -o yaml > apigee-ca.yaml
  3. Apply above Apigee CA certificate secret to all other clusters by selecting one cluster at a time, perform all remaining steps in all clusters: 
    kubectl -n cert-manager apply -f apigee-ca.yaml
  4. Export all existing certificates available in the apigee namespace to a backup file: 
    kubectl -n apigee get certificates --all -o yaml > all-certificates.yaml
  5. Execute following cmctl command for reissuing all certificates found in the apigee namespace: 
    cmctl renew --namespace=apigee --all

    An example output:

    cmctl renew --namespace=apigee --all Manually triggered issuance of Certificate apigee/apigee-cassandra-default Manually triggered issuance of Certificate apigee/apigee-cassandra-schema-setup-demo-hybrid-de-5fdc6d2 Manually triggered issuance of Certificate apigee/apigee-cassandra-schema-val-demo-hybrid-de-5fdc6d2 Manually triggered issuance of Certificate apigee/apigee-cassandra-user-setup-demo-hybrid-de-5fdc6d2 Manually triggered issuance of Certificate apigee/apigee-connect-agent-demo-hybrid-de-5fdc6d2 Manually triggered issuance of Certificate apigee/apigee-datastore-guardrails-tls Manually triggered issuance of Certificate apigee/apigee-istiod Manually triggered issuance of Certificate apigee/apigee-mart-demo-hybrid-de-5fdc6d2 Manually triggered issuance of Certificate apigee/apigee-metrics-adapter-apigee-telemetry Manually triggered issuance of Certificate apigee/apigee-redis-default Manually triggered issuance of Certificate apigee/apigee-redis-envoy-default Manually triggered issuance of Certificate apigee/apigee-runtime-demo-hybrid-de-dev-b276d3f Manually triggered issuance of Certificate apigee/apigee-serving-cert Manually triggered issuance of Certificate apigee/apigee-synchronizer-demo-hybrid-de-dev-b276d3f Manually triggered issuance of Certificate apigee/apigee-udca-demo-hybrid-de-5fdc6d2 Manually triggered issuance of Certificate apigee/apigee-watcher-demo-hybrid-de-5fdc6d2

    This step should reissue all Apigee runtime certificates using the newly imported Apigee CA certificate and that should resolve this problem.

  6. Check the issue date of all certificates against the UTC time and verify whether they have been reissued: 
    kubectl get certificates -n apigee -o json | jq -r '.items[] | "\(.metadata.name): \(.status.notBefore)"' date -u

    An example output:

    kubectl get certificates -n apigee -o json | jq -r '.items[] | "\(.metadata.name): \(.status.notBefore)"' apigee-cassandra-default: 2024-12-16T04:19:58Z apigee-cassandra-schema-setup-demo-hybrid-de-5fdc6d2: 2024-12-16T04:19:58Z apigee-cassandra-schema-val-demo-hybrid-de-5fdc6d2: 2024-12-16T04:19:58Z apigee-cassandra-user-setup-demo-hybrid-de-5fdc6d2: 2024-12-16T04:19:59Z apigee-connect-agent-demo-hybrid-de-5fdc6d2: 2024-12-16T04:20:00Z apigee-datastore-guardrails-tls: 2024-12-16T04:20:01Z apigee-istiod: 2024-12-16T04:20:02Z apigee-mart-demo-hybrid-de-5fdc6d2: 2024-12-16T04:20:02Z apigee-metrics-adapter-apigee-telemetry: 2024-12-16T04:20:03Z apigee-redis-default: 2024-12-16T04:20:04Z apigee-redis-envoy-default: 2024-12-16T04:20:04Z apigee-runtime-demo-hybrid-de-dev-b276d3f: 2024-12-16T04:20:04Z apigee-serving-cert: 2024-12-16T04:20:04Z apigee-synchronizer-demo-hybrid-de-dev-b276d3f: 2024-12-16T04:20:05Z apigee-udca-demo-hybrid-de-5fdc6d2: 2024-12-16T04:20:06Z apigee-watcher-demo-hybrid-de-5fdc6d2: 2024-12-16T04:20:07Z date -u Mon Dec 16 04:23:45 AM UTC 2024
  7. Check the expiry date of all certificates and confirm that has been extended accordingly: 
    kubectl get certificates -n apigee -o json | jq -r '.items[] | "\(.metadata.name): \(.status.notAfter)"'

    An example output:

    kubectl get certificates -n apigee -o json | jq -r '.items[] | "\(.metadata.name): \(.status.notAfter)"' apigee-cassandra-default: 2034-12-14T04:19:58Z apigee-cassandra-schema-setup-demo-hybrid-de-5fdc6d2: 2034-12-14T04:19:58Z apigee-cassandra-schema-val-demo-hybrid-de-5fdc6d2: 2034-12-14T04:19:58Z apigee-cassandra-user-setup-demo-hybrid-de-5fdc6d2: 2034-12-14T04:19:59Z apigee-connect-agent-demo-hybrid-de-5fdc6d2: 2034-12-14T04:20:00Z apigee-datastore-guardrails-tls: 2024-12-16T05:20:01Z apigee-istiod: 2024-12-18T04:20:02Z apigee-mart-demo-hybrid-de-5fdc6d2: 2034-12-14T04:20:02Z apigee-metrics-adapter-apigee-telemetry: 2034-12-14T04:20:03Z apigee-redis-default: 2034-12-14T04:20:04Z apigee-redis-envoy-default: 2034-12-14T04:20:04Z apigee-runtime-demo-hybrid-de-dev-b276d3f: 2034-12-14T04:20:04Z apigee-serving-cert: 2025-03-16T04:20:04Z apigee-synchronizer-demo-hybrid-de-dev-b276d3f: 2034-12-14T04:20:05Z apigee-udca-demo-hybrid-de-5fdc6d2: 2034-12-14T04:20:06Z apigee-watcher-demo-hybrid-de-5fdc6d2: 2034-12-14T04:20:07Z

Must gather diagnostic information

If the problem persists even after following the above instructions, gather the following diagnostic information and then contact Google Cloud Customer Care:

  1. The Google Cloud Project ID.
  2. The Apigee hybrid organization.
  3. The overrides.yaml files from both source and new regions, masking any sensitive information.
  4. The outputs from the commands in Apigee hybrid must-gather.
  5. The outputs from the commands in Apigee hybrid Cassandra must-gather.