0

We are trying to setup a basic kubernetes (v1.25) cluster in a set of VMs (centos 7) which are offline (internet blocked). We have downloaded the images of the kubernetes server components (i.e. apiserver, controller, etcd, kube-proxy, scheduler, coredns etc.) in another server through docker pull where the internet is active, saed those images as tar, transferred them into the offline server, imported into the containerd image repository (ctr -n=k8s.io image import kube-apiserver-v1.25.0.tar). However, if we try to pull the images (kubeadm config images pull) or do the kubeadm init to start the master node, we are getting error as it is trying to connect to registry.k8s.io domain and subsequently to domain storage.googleapis.com domain. We took a temporarily special approval to get connectivity with the above 2 domains to run the kubeadm init command to create the kubernetes containers. Could you please provide a solution so that the initial HEAD call also goes to the local image registry rather than going to the online registry? Eventhough we took permission for connectivity with the kubernetes specific domains, while we are trying to install CALICO as network plugin, we are facing similar issues as it has its own domains to connect. And is a similar way, it would happen for any image from an online public registry. So effectively we need a solution to this.

Below is the error which we got while running kubeadm init before we got the kubernetes domains connected to our server.

$ kubeadm init --pod-network-cidr=10.184.48.0/24 --kubernetes-version=1.25.0 --apiserver-advertiseddress=$(hostname -i) [init] Using Kubernetes version: v1.25.0 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR ImagePull]: failed to pull image registry.k8s.io/kube-apiserver:v1.25.0: output: E0908 08:30:08.405405 19080 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"registry.k8s.io/kube-apiserver:v1.25.0\": failed to resolve reference \"registry.k8s.io/kube-apiserver:v1.25.0\": failed to do request: Head \"https://registry.k8s.io/v2/kube-apiserver/manifests/v1.25.0\": dial tcp: lookup registry.k8s.io on [::1]:53: read udp [::1]:51094->[::1]:53: read: connection refused" image="registry.k8s.io/kube-apiserver:v1.25.0" time="2022-09-08T08:30:08Z" level=fatal msg="pulling image: rpc error: code = Unknown desc = failed to pull and unpack image \"registry.k8s.io/kube-apiserver:v1.25.0\": failed to resolve reference \"registry.k8s.io/kube-apiserver:v1.25.0\": failed to do request: Head \"https://registry.k8s.io/v2/kube-apiserver/manifests/v1.25.0\": dial tcp: lookup registry.k8s.io on [::1]:53: read udp [::1]:51094->[::1]:53: read: connection refused" , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.k8s.io/kube-controller-manager:v1.25.0: output: E0908 08:30:08.473579 19110 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"registry.k8s.io/kube-controller-manager:v1.25.0\": failed to resolve reference \"registry.k8s.io/kube-controller-manager:v1.25.0\": failed to do request: Head \"https://registry.k8s.io/v2/kube-controller-manager/manifests/v1.25.0\": dial tcp: lookup registry.k8s.io on [::1]:53: read udp [::1]:54055->[::1]:53: read: connection refused" image="registry.k8s.io/kube-controller-manager:v1.25.0" time="2022-09-08T08:30:08Z" level=fatal msg="pulling image: rpc error: code = Unknown desc = failed to pull and unpack image \"registry.k8s.io/kube-controller-manager:v1.25.0\": failed to resolve reference \"registry.k8s.io/kube-controller-manager:v1.25.0\": failed to do request: Head \"https://registry.k8s.io/v2/kube-controller-manager/manifests/v1.25.0\": dial tcp: lookup registry.k8s.io on [::1]:53: read udp [::1]:54055->[::1]:53: read: connection refused" , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.k8s.io/kube-scheduler:v1.25.0: output: E0908 08:30:08.540054 19140 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"registry.k8s.io/kube-scheduler:v1.25.0\": failed to resolve reference \"registry.k8s.io/kube-scheduler:v1.25.0\": failed to do request: Head \"https://registry.k8s.io/v2/kube-scheduler/manifests/v1.25.0\": dial tcp: lookup registry.k8s.io on [::1]:53: read udp [::1]:60665->[::1]:53: read: connection refused" image="registry.k8s.io/kube-scheduler:v1.25.0" time="2022-09-08T08:30:08Z" level=fatal msg="pulling image: rpc error: code = Unknown desc = failed to pull and unpack image \"registry.k8s.io/kube-scheduler:v1.25.0\": failed to resolve reference \"registry.k8s.io/kube-scheduler:v1.25.0\": failed to do request: Head \"https://registry.k8s.io/v2/kube-scheduler/manifests/v1.25.0\": dial tcp: lookup registry.k8s.io on [::1]:53: read udp [::1]:60665->[::1]:53: read: connection refused" , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.k8s.io/kube-proxy:v1.25.0: output: E0908 08:30:08.604658 19172 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"registry.k8s.io/kube-proxy:v1.25.0\": failed to resolve reference \"registry.k8s.io/kube-proxy:v1.25.0\": failed to do request: Head \"https://registry.k8s.io/v2/kube-proxy/manifests/v1.25.0\": dial tcp: lookup registry.k8s.io on [::1]:53: read udp [::1]:57387->[::1]:53: read: connection refused" image="registry.k8s.io/kube-proxy:v1.25.0" time="2022-09-08T08:30:08Z" level=fatal msg="pulling image: rpc error: code = Unknown desc = failed to pull and unpack image \"registry.k8s.io/kube-proxy:v1.25.0\": failed to resolve reference \"registry.k8s.io/kube-proxy:v1.25.0\": failed to do request: Head \"https://registry.k8s.io/v2/kube-proxy/manifests/v1.25.0\": dial tcp: lookup registry.k8s.io on [::1]:53: read udp [::1]:57387->[::1]:53: read: connection refused" , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.k8s.io/pause:3.8: output: E0908 08:30:08.664003 19202 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"registry.k8s.io/pause:3.8\": failed to resolve reference \"registry.k8s.io/pause:3.8\": failed to do request: Head \"https://registry.k8s.io/v2/pause/manifests/3.8\": dial tcp: lookup registry.k8s.io on [::1]:53: read udp [::1]:33765->[::1]:53: read: connection refused" image="registry.k8s.io/pause:3.8" time="2022-09-08T08:30:08Z" level=fatal msg="pulling image: rpc error: code = Unknown desc = failed to pull and unpack image \"registry.k8s.io/pause:3.8\": failed to resolve reference \"registry.k8s.io/pause:3.8\": failed to do request: Head \"https://registry.k8s.io/v2/pause/manifests/3.8\": dial tcp: lookup registry.k8s.io on [::1]:53: read udp [::1]:33765->[::1]:53: read: connection refused" , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.k8s.io/etcd:3.5.4-0: output: E0908 08:30:08.724547 19233 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"registry.k8s.io/etcd:3.5.4-0\": failed to resolve reference \"registry.k8s.io/etcd:3.5.4-0\": failed to do request: Head \"https://registry.k8s.io/v2/etcd/manifests/3.5.4-0\": dial tcp: lookup registry.k8s.io on [::1]:53: read udp [::1]:50537->[::1]:53: read: connection refused" image="registry.k8s.io/etcd:3.5.4-0" time="2022-09-08T08:30:08Z" level=fatal msg="pulling image: rpc error: code = Unknown desc = failed to pull and unpack image \"registry.k8s.io/etcd:3.5.4-0\": failed to resolve reference \"registry.k8s.io/etcd:3.5.4-0\": failed to do request: Head \"https://registry.k8s.io/v2/etcd/manifests/3.5.4-0\": dial tcp: lookup registry.k8s.io on [::1]:53: read udp [::1]:50537->[::1]:53: read: connection refused" , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.k8s.io/coredns/coredns:v1.9.3: output: E0908 08:30:08.783023 19264 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"registry.k8s.io/coredns/coredns:v1.9.3\": failed to resolve reference \"registry.k8s.io/coredns/coredns:v1.9.3\": failed to do request: Head \"https://registry.k8s.io/v2/coredns/coredns/manifests/v1.9.3\": dial tcp: lookup registry.k8s.io on [::1]:53: read udp [::1]:33298->[::1]:53: read: connection refused" image="registry.k8s.io/coredns/coredns:v1.9.3" time="2022-09-08T08:30:08Z" level=fatal msg="pulling image: rpc error: code = Unknown desc = failed to pull and unpack image \"registry.k8s.io/coredns/coredns:v1.9.3\": failed to resolve reference \"registry.k8s.io/coredns/coredns:v1.9.3\": failed to do request: Head \"https://registry.k8s.io/v2/coredns/coredns/manifests/v1.9.3\": dial tcp: lookup registry.k8s.io on [::1]:53: read udp [::1]:33298->[::1]:53: read: connection refused" , error: exit status 1 [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...` To see the stack trace of this error execute with --v=5 or higher
Improve this question
2
  • Are you using containerd as a CRI? Did you download your images from the official registry (registry.k8s.io)? Commented Oct 3, 2022 at 2:47
  • Yes, I am using containerd as CRI. I downloaded the images from official registry and imported into my repo server. I try the kubeadm commands from the master node, expecting it to pull the images from my repo server. Commented Oct 14, 2022 at 6:24

1 Answer 1

Reset to default
0

You have several options here, the easiest one is to tag the images that you downloaded with registry.k8s.io in each node (depending on your setup, control-plane/worker):

ctr --namespace=k8s.io image tag <yourimage> registry.k8s.io/<yourimage>

Like

ctr --namespace=k8s.io image tag etcd:3.5.4-0 registry.k8s.io/etcd:3.5.4-0

If you retag all images that you need you can install it offline without issues.

I hope it helps.

Improve this answer
2
  • Thanks for the response. I tried the same, however, ended up with the same error though. Let me explain the setup. I have one VM operating as my repo server, another VM is acting as my master node. In repo VM, I downloaded the images for kubernetes server components, installed containerd, imported all the images into containerd with namespace k8s.io. I can see the images with "ctr -n k8s.io images ls". However, when I try "kubeadm config images pull", it gives me same error irrespective of I do/do not tag the images in the repo VM with or without "registry.k8s.io". Commented Oct 14, 2022 at 6:23
  • Debanjan, you have to pull all images needed manually to each node, retaging in the repo is not to make pull the images locally (kubernetes will go to inet). So if you have a master node and few workers you need to pull all images that you need in each of them and retag to make it work. Commented Oct 15, 2022 at 11:36

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.