Nginx didn't work as expected, all requests failed with 403 #13920
Description
What happened:
All requests failed with 403 for one replica, while there was no such issue for another replica
<html> <head><title>403 Forbidden</title></head> <body> <center><h1>403 Forbidden</h1></center> <hr><center>nginx</center> </body> </html> found following logs from ingress-nginx-controller
1757171414218 I0906 15:10:14.218599 7 controller.go:196] "Configuration changes detected, backend reload required" 1757171414276 I0906 15:10:14.276895 7 controller.go:216] "Backend successfully reloaded" 1757171414277 I0906 15:10:14.277043 7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"test", Name:"nginx-ingress-controller-f454d46db-fl5zk", UID:"c7e7ed06-de43-4330-b3ad-51dc290b9494", APIVersion:"v1", ResourceVersion:"1215888925", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration 1757171414283 I0906 15:10:14.283794 7 store.go:645] "secret was updated and it is used in ingress annotations. Parsing" secret="test/client.ca.truststore" 1757171414284 2025/09/06 15:10:14 [emerg] 27#27: cannot load certificate "/etc/ingress-controller/ssl/test-client-certificate.pem": PEM_read_bio_X509_AUX() failed (SSL: error:0480006C:PEM routines::no start line:Expecting: TRUSTED CERTIFICATE) The issue was resolved after deleting the pod, no such issue for the new pod.
What you expected to happen:
Nginx should handle requests normally same with another replica.
our ingress has following annotations and spec
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true" nginx.ingress.kubernetes.io/auth-tls-secret: test/client.ca.truststore nginx.ingress.kubernetes.io/auth-tls-verify-client: optional name: test-ingress namespace: test spec: rules: - host: <host> http: paths: - backend: service: name: front port: number: 8080 pathType: ImplementationSpecific tls: - hosts: - <host> secretName: client-certificate From my analysis, it seems one of the certificate secret changed and caused all certificate files were wrote again.
while the reload happened due to configuration change from https://github.com/kubernetes/ingress-nginx/blob/main/internal/ingress/controller/controller.go#L213
It's race condition to me. Nginx read empty certificate (truncated when write the file from
ingress-nginx/internal/net/ssl/ssl.go
Line 183 in a031a08
NGINX Ingress controller version (exec into the pod and run /nginx-ingress-controller --version):
/etc/nginx $ /nginx-ingress-controller --version ------------------------------------------------------------------------------- NGINX Ingress controller Release: v1.13.1 Build: c8ce0d146a53fbdb94848548068001909767e2de Repository: https://github.com/kubernetes/ingress-nginx nginx version: nginx/1.27.1 ------------------------------------------------------------------------------- Kubernetes version (use kubectl version):
Server Version: version.Info{Major:"1", Minor:"32", GitVersion:"v1.32.7", GitCommit:"158eee9fac884b429a92465edd0d88a43f81de34", GitTreeState:"clean", BuildDate:"2025-07-15T18:00:33Z", GoVersion:"go1.23.10", Compiler:"gc", Platform:"linux/amd64"} How to reproduce this issue:
It's hard to reproduce. We have applied same change to many clusters (above 200), only two had such issue.
Anything else we need to know: