Problem pulling images when running private docker registry inside of Kubernetes
3
I migrated our docker registry that was running on an external dedicated server into our Kubernetes cluster
Now I can still push and pull images to the registry from every external machine but when I try to deploy images from the registry to the Kubernetes cluster itself it is not able to pull it. I get the following error log:
Warning Failed 47s (x3 over 1m) kubelet, gke-kube-1-default-pool-c5e11d0f-zxm8 Failed to pull image "myregistry.example.com/appimage:1": rpc error: code = Unknown desc = Error response from daemon: Get https://myregistry.example.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Warning Failed 47s (x3 over 1m) kubelet, gke-kube-1-default-pool-c5e11d0f-zxm8 Error: ErrImagePull
The registry is configured to be accessible via https://myregistry.example.com by a traefik ingress controller and it looks like Kubernetes is internally trying to take a different route?
docker
kubernetes edited Nov 17 '18 at 0:53
Rico
29.3k95370
asked Nov 16 '18 at 11:21
Thomas EinwallerThomas Einwaller
4,26943246
|
show 1 more comment
3
I migrated our docker registry that was running on an external dedicated server into our Kubernetes cluster
Now I can still push and pull images to the registry from every external machine but when I try to deploy images from the registry to the Kubernetes cluster itself it is not able to pull it. I get the following error log:
Warning Failed 47s (x3 over 1m) kubelet, gke-kube-1-default-pool-c5e11d0f-zxm8 Failed to pull image "myregistry.example.com/appimage:1": rpc error: code = Unknown desc = Error response from daemon: Get https://myregistry.example.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Warning Failed 47s (x3 over 1m) kubelet, gke-kube-1-default-pool-c5e11d0f-zxm8 Error: ErrImagePull
The registry is configured to be accessible via https://myregistry.example.com by a traefik ingress controller and it looks like Kubernetes is internally trying to take a different route?
docker
kubernetes edited Nov 17 '18 at 0:53
Rico
29.3k95370
asked Nov 16 '18 at 11:21
Thomas EinwallerThomas Einwaller
4,26943246
Did you addimagePullSecretin your manifest file?
– Shudipta Sharma
Nov 16 '18 at 11:43
Why not just use theClusterIPor internal K8S DNS to reach the registry ? In your current way, you are exiting the cluster and retuning back in. Kind of sounds like a longer path to take.
– Jason Stanley
Nov 17 '18 at 0:59
theimagePullSecretis ok, using theClusterIPwould be strange because the configuration would have to be different than on any other server or cluster pulling the same image
– Thomas Einwaller
Nov 17 '18 at 21:04
Which version of k8s? Did you solve this? I have exactly same issue running k8s 1.11.6 on Azure. Moreover, some times I succeed to create pod with image from my in-cluster registry, but some times not (for the same image). I monitored logs of the pod running my registry (kubectl logs -n docker-registry docker-registry-5c6998f89f-hx96l -f) and found that it doesn't receive any requests whenImagePullBackOffhappens. Found this issue: github.com/kubernetes/kubernetes/issues/63874 which recommends upgrading to the latest k8s, will give it a try...
– Stanislav Poslavsky
Jan 30 at 22:35
Also this seems to be relevant: digitalocean.com/community/questions/…
– Stanislav Poslavsky
Jan 30 at 22:40
|
show 1 more comment
3
3
3
1
I migrated our docker registry that was running on an external dedicated server into our Kubernetes cluster
Now I can still push and pull images to the registry from every external machine but when I try to deploy images from the registry to the Kubernetes cluster itself it is not able to pull it. I get the following error log:
Warning Failed 47s (x3 over 1m) kubelet, gke-kube-1-default-pool-c5e11d0f-zxm8 Failed to pull image "myregistry.example.com/appimage:1": rpc error: code = Unknown desc = Error response from daemon: Get https://myregistry.example.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Warning Failed 47s (x3 over 1m) kubelet, gke-kube-1-default-pool-c5e11d0f-zxm8 Error: ErrImagePull
The registry is configured to be accessible via https://myregistry.example.com by a traefik ingress controller and it looks like Kubernetes is internally trying to take a different route?
docker
kubernetes edited Nov 17 '18 at 0:53
Rico
29.3k95370
asked Nov 16 '18 at 11:21
Thomas EinwallerThomas Einwaller
4,26943246
I migrated our docker registry that was running on an external dedicated server into our Kubernetes cluster
Now I can still push and pull images to the registry from every external machine but when I try to deploy images from the registry to the Kubernetes cluster itself it is not able to pull it. I get the following error log:
Warning Failed 47s (x3 over 1m) kubelet, gke-kube-1-default-pool-c5e11d0f-zxm8 Failed to pull image "myregistry.example.com/appimage:1": rpc error: code = Unknown desc = Error response from daemon: Get https://myregistry.example.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Warning Failed 47s (x3 over 1m) kubelet, gke-kube-1-default-pool-c5e11d0f-zxm8 Error: ErrImagePull
The registry is configured to be accessible via https://myregistry.example.com by a traefik ingress controller and it looks like Kubernetes is internally trying to take a different route?
docker
kubernetes docker
kubernetes edited Nov 17 '18 at 0:53
Rico
29.3k95370
asked Nov 16 '18 at 11:21
Thomas EinwallerThomas Einwaller
4,26943246
edited Nov 17 '18 at 0:53
Rico
29.3k95370
asked Nov 16 '18 at 11:21
Thomas EinwallerThomas Einwaller
4,26943246
edited Nov 17 '18 at 0:53
Rico
29.3k95370
edited Nov 17 '18 at 0:53
Rico
29.3k95370
edited Nov 17 '18 at 0:53
Rico
29.3k95370
29.3k95370
asked Nov 16 '18 at 11:21
Thomas EinwallerThomas Einwaller
4,26943246
asked Nov 16 '18 at 11:21
Thomas EinwallerThomas Einwaller
4,26943246
asked Nov 16 '18 at 11:21
Thomas EinwallerThomas Einwaller
4,26943246
4,26943246
Did you addimagePullSecretin your manifest file?
– Shudipta Sharma
Nov 16 '18 at 11:43
Why not just use theClusterIPor internal K8S DNS to reach the registry ? In your current way, you are exiting the cluster and retuning back in. Kind of sounds like a longer path to take.
– Jason Stanley
Nov 17 '18 at 0:59
theimagePullSecretis ok, using theClusterIPwould be strange because the configuration would have to be different than on any other server or cluster pulling the same image
– Thomas Einwaller
Nov 17 '18 at 21:04
Which version of k8s? Did you solve this? I have exactly same issue running k8s 1.11.6 on Azure. Moreover, some times I succeed to create pod with image from my in-cluster registry, but some times not (for the same image). I monitored logs of the pod running my registry (kubectl logs -n docker-registry docker-registry-5c6998f89f-hx96l -f) and found that it doesn't receive any requests whenImagePullBackOffhappens. Found this issue: github.com/kubernetes/kubernetes/issues/63874 which recommends upgrading to the latest k8s, will give it a try...
– Stanislav Poslavsky
Jan 30 at 22:35
Also this seems to be relevant: digitalocean.com/community/questions/…
– Stanislav Poslavsky
Jan 30 at 22:40
|
show 1 more comment
Did you addimagePullSecretin your manifest file?
– Shudipta Sharma
Nov 16 '18 at 11:43
Why not just use theClusterIPor internal K8S DNS to reach the registry ? In your current way, you are exiting the cluster and retuning back in. Kind of sounds like a longer path to take.
– Jason Stanley
Nov 17 '18 at 0:59
theimagePullSecretis ok, using theClusterIPwould be strange because the configuration would have to be different than on any other server or cluster pulling the same image
– Thomas Einwaller
Nov 17 '18 at 21:04
Which version of k8s? Did you solve this? I have exactly same issue running k8s 1.11.6 on Azure. Moreover, some times I succeed to create pod with image from my in-cluster registry, but some times not (for the same image). I monitored logs of the pod running my registry (kubectl logs -n docker-registry docker-registry-5c6998f89f-hx96l -f) and found that it doesn't receive any requests whenImagePullBackOffhappens. Found this issue: github.com/kubernetes/kubernetes/issues/63874 which recommends upgrading to the latest k8s, will give it a try...
– Stanislav Poslavsky
Jan 30 at 22:35
Also this seems to be relevant: digitalocean.com/community/questions/…
– Stanislav Poslavsky
Jan 30 at 22:40
Did you add
imagePullSecret in your manifest file?– Shudipta Sharma
Nov 16 '18 at 11:43
Did you add
imagePullSecret in your manifest file?– Shudipta Sharma
Nov 16 '18 at 11:43
Why not just use the
ClusterIP or internal K8S DNS to reach the registry ? In your current way, you are exiting the cluster and retuning back in. Kind of sounds like a longer path to take.– Jason Stanley
Nov 17 '18 at 0:59
Why not just use the
ClusterIP or internal K8S DNS to reach the registry ? In your current way, you are exiting the cluster and retuning back in. Kind of sounds like a longer path to take.– Jason Stanley
Nov 17 '18 at 0:59
the
imagePullSecret is ok, using the ClusterIP would be strange because the configuration would have to be different than on any other server or cluster pulling the same image– Thomas Einwaller
Nov 17 '18 at 21:04
the
imagePullSecret is ok, using the ClusterIP would be strange because the configuration would have to be different than on any other server or cluster pulling the same image– Thomas Einwaller
Nov 17 '18 at 21:04
Which version of k8s? Did you solve this? I have exactly same issue running k8s 1.11.6 on Azure. Moreover, some times I succeed to create pod with image from my in-cluster registry, but some times not (for the same image). I monitored logs of the pod running my registry (
kubectl logs -n docker-registry docker-registry-5c6998f89f-hx96l -f) and found that it doesn't receive any requests when ImagePullBackOff happens. Found this issue: github.com/kubernetes/kubernetes/issues/63874 which recommends upgrading to the latest k8s, will give it a try...– Stanislav Poslavsky
Jan 30 at 22:35
Which version of k8s? Did you solve this? I have exactly same issue running k8s 1.11.6 on Azure. Moreover, some times I succeed to create pod with image from my in-cluster registry, but some times not (for the same image). I monitored logs of the pod running my registry (
kubectl logs -n docker-registry docker-registry-5c6998f89f-hx96l -f) and found that it doesn't receive any requests when ImagePullBackOff happens. Found this issue: github.com/kubernetes/kubernetes/issues/63874 which recommends upgrading to the latest k8s, will give it a try...– Stanislav Poslavsky
Jan 30 at 22:35
Also this seems to be relevant: digitalocean.com/community/questions/…
– Stanislav Poslavsky
Jan 30 at 22:40
Also this seems to be relevant: digitalocean.com/community/questions/…
– Stanislav Poslavsky
Jan 30 at 22:40
|
show 1 more comment
1 Answer
1
active
oldest
votes
1
It's an odd path, but it should work, but taking a wild guess it looks like a DNS issue. (It works for me connecting to an externally facing service). Some things to look at:
- Can you resolve
myregistry.example.comfrom any other running pod? - What does the
/etc/resolv.conflook like? - What about your K8s nodes
/etc/resolv.conf
- Can you resolve
myregistry.example.comfrom your nodes?
answered Nov 17 '18 at 2:02
RicoRico
29.3k95370
It seems like it is not a DNS issue because it gets a connection timeout
– Thomas Einwaller
Nov 17 '18 at 21:07
Did you confirm?
– Rico
Nov 17 '18 at 21:22
I can resolve the registry form any running pod, the resolv.conf on the pod hasnameserver 10.31.240.10in it, I can resolv the registry from the kube nodes, though the resolv.conf looks strange:nameserver 169.254.169.254
– Thomas Einwaller
Nov 18 '18 at 14:20
That looks odd...
– Rico
Nov 19 '18 at 5:11
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53336855%2fproblem-pulling-images-when-running-private-docker-registry-inside-of-kubernetes%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
1
It's an odd path, but it should work, but taking a wild guess it looks like a DNS issue. (It works for me connecting to an externally facing service). Some things to look at:
- Can you resolve
myregistry.example.comfrom any other running pod? - What does the
/etc/resolv.conflook like? - What about your K8s nodes
/etc/resolv.conf
- Can you resolve
myregistry.example.comfrom your nodes?
answered Nov 17 '18 at 2:02
RicoRico
29.3k95370
It seems like it is not a DNS issue because it gets a connection timeout
– Thomas Einwaller
Nov 17 '18 at 21:07
Did you confirm?
– Rico
Nov 17 '18 at 21:22
I can resolve the registry form any running pod, the resolv.conf on the pod hasnameserver 10.31.240.10in it, I can resolv the registry from the kube nodes, though the resolv.conf looks strange:nameserver 169.254.169.254
– Thomas Einwaller
Nov 18 '18 at 14:20
That looks odd...
– Rico
Nov 19 '18 at 5:11
add a comment |
1
It's an odd path, but it should work, but taking a wild guess it looks like a DNS issue. (It works for me connecting to an externally facing service). Some things to look at:
- Can you resolve
myregistry.example.comfrom any other running pod? - What does the
/etc/resolv.conflook like? - What about your K8s nodes
/etc/resolv.conf
- Can you resolve
myregistry.example.comfrom your nodes?
answered Nov 17 '18 at 2:02
RicoRico
29.3k95370
It seems like it is not a DNS issue because it gets a connection timeout
– Thomas Einwaller
Nov 17 '18 at 21:07
Did you confirm?
– Rico
Nov 17 '18 at 21:22
I can resolve the registry form any running pod, the resolv.conf on the pod hasnameserver 10.31.240.10in it, I can resolv the registry from the kube nodes, though the resolv.conf looks strange:nameserver 169.254.169.254
– Thomas Einwaller
Nov 18 '18 at 14:20
That looks odd...
– Rico
Nov 19 '18 at 5:11
add a comment |
1
1
1
It's an odd path, but it should work, but taking a wild guess it looks like a DNS issue. (It works for me connecting to an externally facing service). Some things to look at:
- Can you resolve
myregistry.example.comfrom any other running pod? - What does the
/etc/resolv.conflook like? - What about your K8s nodes
/etc/resolv.conf
- Can you resolve
myregistry.example.comfrom your nodes?
answered Nov 17 '18 at 2:02
RicoRico
29.3k95370
It's an odd path, but it should work, but taking a wild guess it looks like a DNS issue. (It works for me connecting to an externally facing service). Some things to look at:
- Can you resolve
myregistry.example.comfrom any other running pod? - What does the
/etc/resolv.conflook like? - What about your K8s nodes
/etc/resolv.conf
- Can you resolve
myregistry.example.comfrom your nodes?
answered Nov 17 '18 at 2:02
RicoRico
29.3k95370
answered Nov 17 '18 at 2:02
RicoRico
29.3k95370
answered Nov 17 '18 at 2:02
RicoRico
29.3k95370
answered Nov 17 '18 at 2:02
RicoRico
29.3k95370
29.3k95370
It seems like it is not a DNS issue because it gets a connection timeout
– Thomas Einwaller
Nov 17 '18 at 21:07
Did you confirm?
– Rico
Nov 17 '18 at 21:22
I can resolve the registry form any running pod, the resolv.conf on the pod hasnameserver 10.31.240.10in it, I can resolv the registry from the kube nodes, though the resolv.conf looks strange:nameserver 169.254.169.254
– Thomas Einwaller
Nov 18 '18 at 14:20
That looks odd...
– Rico
Nov 19 '18 at 5:11
add a comment |
It seems like it is not a DNS issue because it gets a connection timeout
– Thomas Einwaller
Nov 17 '18 at 21:07
Did you confirm?
– Rico
Nov 17 '18 at 21:22
I can resolve the registry form any running pod, the resolv.conf on the pod hasnameserver 10.31.240.10in it, I can resolv the registry from the kube nodes, though the resolv.conf looks strange:nameserver 169.254.169.254
– Thomas Einwaller
Nov 18 '18 at 14:20
That looks odd...
– Rico
Nov 19 '18 at 5:11
It seems like it is not a DNS issue because it gets a connection timeout
– Thomas Einwaller
Nov 17 '18 at 21:07
It seems like it is not a DNS issue because it gets a connection timeout
– Thomas Einwaller
Nov 17 '18 at 21:07
Did you confirm?
– Rico
Nov 17 '18 at 21:22
Did you confirm?
– Rico
Nov 17 '18 at 21:22
I can resolve the registry form any running pod, the resolv.conf on the pod has
nameserver 10.31.240.10 in it, I can resolv the registry from the kube nodes, though the resolv.conf looks strange: nameserver 169.254.169.254– Thomas Einwaller
Nov 18 '18 at 14:20
I can resolve the registry form any running pod, the resolv.conf on the pod has
nameserver 10.31.240.10 in it, I can resolv the registry from the kube nodes, though the resolv.conf looks strange: nameserver 169.254.169.254– Thomas Einwaller
Nov 18 '18 at 14:20
That looks odd...
– Rico
Nov 19 '18 at 5:11
That looks odd...
– Rico
Nov 19 '18 at 5:11
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53336855%2fproblem-pulling-images-when-running-private-docker-registry-inside-of-kubernetes%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Did you add
imagePullSecretin your manifest file?– Shudipta Sharma
Nov 16 '18 at 11:43
Why not just use the
ClusterIPor internal K8S DNS to reach the registry ? In your current way, you are exiting the cluster and retuning back in. Kind of sounds like a longer path to take.– Jason Stanley
Nov 17 '18 at 0:59
the
imagePullSecretis ok, using theClusterIPwould be strange because the configuration would have to be different than on any other server or cluster pulling the same image– Thomas Einwaller
Nov 17 '18 at 21:04
Which version of k8s? Did you solve this? I have exactly same issue running k8s 1.11.6 on Azure. Moreover, some times I succeed to create pod with image from my in-cluster registry, but some times not (for the same image). I monitored logs of the pod running my registry (
kubectl logs -n docker-registry docker-registry-5c6998f89f-hx96l -f) and found that it doesn't receive any requests whenImagePullBackOffhappens. Found this issue: github.com/kubernetes/kubernetes/issues/63874 which recommends upgrading to the latest k8s, will give it a try...– Stanislav Poslavsky
Jan 30 at 22:35
Also this seems to be relevant: digitalocean.com/community/questions/…
– Stanislav Poslavsky
Jan 30 at 22:40