0

I'm attempting to set up my own bare-metal Kubernetes cluster composed of a control plane (VPS 149.62.186.95) and a worker node (VPS 149.62.189.139). Both my VPS are running Ubuntu 22.04 with 2 vCPUs and 4GB vRAM.

After installing Kubernetes (kubeadm) on both, I ran the following commands on my control plane:

kubeadm init mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config # Weave deamonset kubectl apply -f https://github.com/weaveworks/weave/releases/download/v2.8.1/weave-daemonset-k8s.yaml # baremetal nginx ingress kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/baremetal/deploy.yaml

I edited my DNS records as follows:

  • registry.rottigni.tech -> 149.62.186.95 (control plane)
  • sandbox.rottigni.tech -> 149.62.186.95 (control plane)

I generated SSL certs for these domains on my control plane and stored them in Kubernetes secrets.

Then, I applied my changes using kubectl apply -R -f . inside my repository. Here's the most relevant file, ingress.yml:

apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: name: nginx-ingress-class spec: controller: k8s.io/ingress-nginx --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: lr-labs-ingress namespace: lr-labs annotations: # cert-manager.io/issuer: "letsencrypt-prod" nginx.ingress.kubernetes.io/rewrite-target: /$1 spec: ingressClassName: nginx-ingress-class tls: - hosts: - registry.rottigni.tech secretName: registry-ssl - hosts: - sandbox.rottigni.tech secretName: sandbox-ssl rules: - host: registry.rottigni.tech http: paths: - path: / pathType: Prefix backend: service: name: registry-rottigni port: number: 5000 - host: sandbox.rottigni.tech http: paths: - path: / pathType: Prefix backend: service: name: sandbox-rottigni port: number: 80

At this point, everything seems to be working on the kubectl side, according to the following output:

Every 2.0s: kubectl get pods,deploy,svc,ing,endpoints,node -A NAMESPACE NAME READY STATUS RESTARTS AGE ingress-nginx pod/ingress-nginx-admission-create-wn7xr 0/1 Completed 0 7h58m ingress-nginx pod/ingress-nginx-admission-patch-dtzxf 0/1 Completed 0 7h58m ingress-nginx pod/ingress-nginx-controller-86968d59df-vdv9f 1/1 Running 0 6h52m kube-system pod/coredns-5dd5756b68-h4jwb 1/1 Running 0 7h58m kube-system pod/coredns-5dd5756b68-jrsdq 1/1 Running 0 7h58m kube-system pod/etcd-epsilon-rottigni-tech 1/1 Running 0 7h58m kube-system pod/kube-apiserver-epsilon-rottigni-tech 1/1 Running 0 7h58m kube-system pod/kube-controller-manager-epsilon-rottigni-tech 1/1 Running 0 7h58m kube-system pod/kube-proxy-4v8bc 1/1 Running 0 7h58m kube-system pod/kube-proxy-8hpjj 1/1 Running 0 7h57m kube-system pod/kube-scheduler-epsilon-rottigni-tech 1/1 Running 0 7h58m kube-system pod/weave-net-gvzk5 2/2 Running 1 (7h58m ago) 7h58m kube-system pod/weave-net-tbr92 2/2 Running 0 7h57m lr-labs pod/registry-rottigni-5c6cfd877f-qqxsw 1/1 Running 0 7h54m lr-labs pod/sandbox-rottigni-9668f7447-dv7js 1/1 Running 0 7h54m NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE ingress-nginx deployment.apps/ingress-nginx-controller 1/1 1 1 7h58m kube-system deployment.apps/coredns 2/2 2 2 7h58m lr-labs deployment.apps/registry-rottigni 1/1 1 1 7h54m lr-labs deployment.apps/sandbox-rottigni 1/1 1 1 7h54m NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 7h58m ingress-nginx service/ingress-nginx-controller NodePort 10.108.71.135 <none> 80:32095/TCP,443:30221/TCP 7h58m ingress-nginx service/ingress-nginx-controller-admission ClusterIP 10.107.203.218 <none> 443/TCP 7h58m kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 7h58m lr-labs service/registry-rottigni NodePort 10.106.231.17 <none> 5000:31673/TCP 7h54m lr-labs service/sandbox-rottigni NodePort 10.111.173.239 <none> 80:31607/TCP 7h54m NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE lr-labs ingress.networking.k8s.io/lr-labs-ingress nginx-ingress-class registry.rottigni.tech,sandbox.rottigni.tech 149.62.189.139 80, 443 7h54m NAMESPACE NAME ENDPOINTS AGE default endpoints/kubernetes 149.62.186.95:6443 7h58m ingress-nginx endpoints/ingress-nginx-controller 10.44.0.2:443,10.44.0.2:80 7h58m ingress-nginx endpoints/ingress-nginx-controller-admission 10.44.0.2:8443 7h58m kube-system endpoints/kube-dns 10.32.0.2:53,10.32.0.3:53,10.32.0.2:53 + 3 more... 7h58m lr-labs endpoints/registry-rottigni 10.44.0.3:5000 7h54m lr-labs endpoints/sandbox-rottigni 10.44.0.1:80 7h54m NAMESPACE NAME STATUS ROLES AGE VERSION node/epsilon-rottigni-tech Ready control-plane 7h58m v1.28.2 node/zeta-rottigni-tech Ready <none> 7h57m v1.28.2

One strange thing is that my ingress has IP 149.62.189.139, which is my worker node IP (not control plane IP). Is this the correct behavior?

Despite this, I'm unable to reach my services from the public DNS through my nginx ingress controller. I can, however, correctly reach both services using NodePort from my control plane and worker node SSH sessions, eliminating any application errors.

It appears that my ingress isn't facing the internet on the correct IP. I also checked my ingress-nginx-controller logs, which seem fine:

------------------------------------------------------------------------------- NGINX Ingress controller Release: v1.9.5 Build: f503c4bb5fa7d857ad29e94970eb550c2bc00b7c Repository: https://github.com/kubernetes/ingress-nginx nginx version: nginx/1.21.6 ------------------------------------------------------------------------------- W0105 14:11:47.972914 7 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0105 14:11:47.973187 7 main.go:205] "Creating API client" host="https://10.96.0.1:443" I0105 14:11:47.983447 7 main.go:249] "Running in Kubernetes cluster" major="1" minor="28" git="v1.28.5" state="clean" commit="506050d61cf291218dfbd41ac93913945c9aa0da" platform="linux/amd64" I0105 14:11:48.470371 7 main.go:101] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem" I0105 14:11:48.511293 7 ssl.go:536] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key" I0105 14:11:48.529226 7 nginx.go:260] "Starting NGINX Ingress controller" I0105 14:11:48.539963 7 event.go:298] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"a5edeb27-12a0-4576-ab70-19d8cffb4dfe", APIVersion:"v1", ResourceVersion:"338", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller I0105 14:11:49.635029 7 store.go:440] "Found valid IngressClass" ingress="lr-labs/lr-labs-ingress" ingressclass="nginx-ingress-class" I0105 14:11:49.636090 7 event.go:298] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"lr-labs", Name:"lr-labs-ingress", UID:"5663167b-9b94-4cb6-8670-5e5e1076bfaa", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1159", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync I0105 14:11:49.638242 7 backend_ssl.go:67] "Adding secret to local store" name="lr-labs/registry-ssl" I0105 14:11:49.641278 7 backend_ssl.go:67] "Adding secret to local store" name="lr-labs/sandbox-ssl" I0105 14:11:49.731218 7 nginx.go:303] "Starting NGINX process" I0105 14:11:49.731681 7 leaderelection.go:245] attempting to acquire leader lease ingress-nginx/ingress-nginx-leader... I0105 14:11:49.732611 7 nginx.go:323] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key" I0105 14:11:49.733509 7 controller.go:190] "Configuration changes detected, backend reload required" I0105 14:11:49.736538 7 status.go:84] "New leader elected" identity="ingress-nginx-controller-685dff585b-2s8jp" I0105 14:11:49.849528 7 controller.go:210] "Backend successfully reloaded" I0105 14:11:49.849813 7 controller.go:221] "Initial sync, sleeping for 1 second" I0105 14:11:49.850019 7 event.go:298] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-86968d59df-vdv9f", UID:"7ac8215f-175f-4496-90a1-dd63d6acad4e", APIVersion:"v1", ResourceVersion:"6819", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration I0105 14:12:41.662980 7 status.go:84] "New leader elected" identity="ingress-nginx-controller-86968d59df-vdv9f" I0105 14:12:41.663007 7 leaderelection.go:255] successfully acquired lease ingress-nginx/ingress-nginx-leader 10.32.0.1 - - [05/Jan/2024:14:36:35 +0000] "GET / HTTP/1.1" 400 248 "-" "curl/7.81.0" 81 0.000 [] [] - - - - 479859864f23ba0523fb9c29c1932d56 10.32.0.1 - - [05/Jan/2024:14:38:30 +0000] "GET / HTTP/1.1" 308 164 "-" "curl/7.81.0" 85 0.000 [lr-labs-sandbox-rottigni-80] [] - - - - e262c16d9495ef00f412f0ba73dbb453 10.32.0.1 - - [05/Jan/2024:15:25:27 +0000] "GET / HTTP/1.1" 308 164 "-" "curl/7.81.0" 85 0.000 [lr-labs-sandbox-rottigni-80] [] - - - - 9b662b33bb59be1846863af658f251ab 10.32.0.1 - - [05/Jan/2024:16:08:32 +0000] "GET / HTTP/1.1" 308 164 "-" "curl/7.81.0" 85 0.000 [lr-labs-sandbox-rottigni-80] [] - - - - a54eda51703800c9a1726a9e1187988d

I even see some manually initiated curl requests for debugging directly to the nginx service using the Host HTTP header.

Additionally here my working curl commands to my cluster service resources:

root@epsilon-rottigni-tech:~/k8s.rottigni.tech/k8s# kubectl get svc -A NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 8h ingress-nginx ingress-nginx-controller NodePort 10.108.71.135 <none> 80:32095/TCP,443:30221/TCP 8h ingress-nginx ingress-nginx-controller-admission ClusterIP 10.107.203.218 <none> 443/TCP 8h kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 8h lr-labs registry-rottigni NodePort 10.106.231.17 <none> 5000:31673/TCP 8h lr-labs sandbox-rottigni NodePort 10.111.173.239 <none> 80:31607/TCP 8h root@epsilon-rottigni-tech:~/k8s.rottigni.tech/k8s# curl 10.106.231.17:5000 root@epsilon-rottigni-tech:~/k8s.rottigni.tech/k8s# curl 10.108.71.135:80 <html> <head><title>404 Not Found</title></head> <body> <center><h1>404 Not Found</h1></center> <hr><center>nginx</center> </body> </html> root@epsilon-rottigni-tech:~/k8s.rottigni.tech/k8s# curl 10.108.71.135:443 <html> <head><title>400 The plain HTTP request was sent to HTTPS port</title></head> <body> <center><h1>400 Bad Request</h1></center> <center>The plain HTTP request was sent to HTTPS port</center> <hr><center>nginx</center> </body> </html> root@epsilon-rottigni-tech:~/k8s.rottigni.tech/k8s# curl 10.111.173.239:80 <html> <head> <style> html { background: url(./Cheddar.jpg) no-repeat center center fixed; -webkit-background-size: cover; -moz-background-size: cover; -o-background-size: cover; background-size: cover; } h1 { font-family: Arial, Helvetica, sans-serif; background: rgba(187, 187, 187, 0.5); width: 6em; padding: 0.5em 1em; margin: 1em; } </style> </head> <body> <h1>Cheddar</h1> </body> </html>root@epsilon-rottigni-tech:~/k8s.rottigni.tech/k8s#

Could someone please help me debug this problem? Am I missing an important step for this kind of production deployment?

Thank you everybody!

UPDATE:

Given the following situation:

kubectl get svc,ing -A NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 23h ingress-nginx service/ingress-nginx-controller NodePort 10.108.71.135 <none> 80:32095/TCP,443:30221/TCP 23h ingress-nginx service/ingress-nginx-controller-admission ClusterIP 10.107.203.218 <none> 443/TCP 23h kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 23h lr-labs service/registry-rottigni NodePort 10.106.231.17 <none> 5000:31673/TCP 23h lr-labs service/sandbox-rottigni NodePort 10.111.173.239 <none> 80:31607/TCP 23h NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE lr-labs ingress.networking.k8s.io/lr-labs-ingress nginx-ingress-class registry.rottigni.tech,sandbox.rottigni.tech 149.62.189.139 80, 443 23h

I am able to reach nginx from internet like this:

But I can't like this:

Improve this question
0

1 Answer 1

Reset to default
0

Basically you did everything correctly. The chosen way to make your service reachable from the outside does not allow you to use specific, low, ports in order to keep isolation between services. You have two options:

  1. use a service of type LoadBalancer which will allocate a separate IP for the service, providing isolation between services and allowing you a specific port like 80 or 443
  2. modify the ingress deployment and use a hostPort for the nginx containers, you can then use the node IPs where a pod is running. This can lead to a lot of headaches (f.e. when updating the deployment) but might be an easy solution to get started
Improve this answer
2
  • Thanks for you answer, I fixed my problem using MetalLB with a super basic config. I misunderstood the ingress resource behaviour because of his IP address… I still don’t understand why the ingress resource has an IP assigned since its only purpose is to provide an nginx configuration to the ingress-nginx-controller service. Everything is now working as expected, thanks anyway for your help! Commented Jan 9, 2024 at 7:48
  • If you are using MetalLB and It is configured correctly it should provide you with a dedicated IP as it is a LoadBalancer implementation. Please accept the answer if it solved your issue. Commented Jan 9, 2024 at 19:58

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.