1

Where are the possible places to configure K8S using a proxy as it repeats "proxyconnect tcp: dial tcp 127.0.0.1:1082: connect: connection refused" during setting up Ray Cluster?

Follow instruction Ray CLuster quickstart:

helm repo add kuberay https://ray-project.github.io/kuberay-helm/ helm repo update # Install both CRDs and KubeRay operator v1.1.1. helm install kuberay-operator kuberay/kuberay-operator --version 1.1.1 # Confirm that the operator is running in the namespace `default`. kubectl get pods # NAME READY STATUS RESTARTS AGE # kuberay-operator-7fbdbf8c89-pt8bk 1/1 Running 0 27s

in the step2. I get aErrImagePull status pod, and the real output:

(base) ➜ ~ helm install kuberay-operator kuberay/kuberay-operator --version 1.0.0 [36/197] NAME: kuberay-operator LAST DEPLOYED: Fri Jul 26 08:56:30 2024 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None (base) ➜ ~ kubectl get pods NAME READY STATUS RESTARTS AGE kuberay-operator-5d64d88fdb-shrkv 0/1 ErrImagePull 0 10s (base) ➜ ~ kubectl describe pod kuberay-operator-5d64d88fdb-shrkv Name: kuberay-operator-5d64d88fdb-shrkv Namespace: default Priority: 0 Service Account: kuberay-operator Node: kind-control-plane/172.23.0.2 Start Time: Fri, 26 Jul 2024 08:56:31 +0800 Labels: app.kubernetes.io/component=kuberay-operator app.kubernetes.io/instance=kuberay-operator app.kubernetes.io/name=kuberay-operator pod-template-hash=5d64d88fdb ..... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 22s default-scheduler Successfully assigned default/kuberay-operator-5d64d88fdb-shrkv to kind-control-plane Normal BackOff 21s kubelet Back-off pulling image "kuberay/operator:v1.0.0" Warning Failed 21s kubelet Error: ImagePullBackOff Normal Pulling 6s (x2 over 21s) kubelet Pulling image "kuberay/operator:v1.0.0" Warning Failed 6s (x2 over 21s) kubelet Failed to pull image "kuberay/operator:v1.0.0": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/kuberay/operator:v1.0.0" : failed to resolve reference "docker.io/kuberay/operator:v1.0.0": failed to do request: Head "https://registry-1.docker.io/v2/kuberay/operator/manifests/v1.0.0": proxyconnect tcp: dial tcp 127.0.0.1:1082: connect: connection refused Warning Failed 6s (x2 over 21s) kubelet Error: ErrImagePull

The puzzle question is the message: proxyconnect tcp: dial tcp 127.0.0.1:1082: connect: connection refused

I tried following ways but did not find any proxy configuration:

(base) ➜ ~ echo $HTTP_PROXY (base) ➜ ~ echo $HTTPS_PROXY (base) ➜ ~ cat /etc/environment PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin" (base) ➜ ~ cat /etc/docker/daemon.json { "registry-mirrors": [ "https://5wxalzzb.mirror.aliyuncs.com", "https://hub-mirror.c.163.com", "https://mirror.iscas.ac.cn", "https://docker.m.daocloud.io" ], "runtimes": { "nvidia": { "args": [], "path": "nvidia-container-runtime" } } }

To mitigate the pod issue, I started local port 1082, which is a HTTP proxy without AuthZ & AuthN, and then re-install the kuberay/operator again, but the same error event message for proxy.

Improve this question
3
  • The error indicates that k8s cluster is attempting to use a proxy server on your local machine at port 1082 to pull the image but it is unable to connect. It might be a configuration issue with the proxy. You might not have a proxy server configured or running on your machine at port 1082.If you have a proxy server, its configurations might be wrong Commented Jul 26, 2024 at 17:41
  • @SaiChandiniRouthu Do you have idea where is possible places that confogures these 1082 port as a proxy? Commented Jul 27, 2024 at 5:11
  • @ Xu Shijie Have you gone through my solution & the answer helps to resolve your issue, if not let me know, I am happy to assist you further. – Commented Aug 5, 2024 at 8:40

1 Answer 1

Reset to default
0

It might be that the proxy got configured wrongly or there is no existing proxy server running on port 1082.

Do you have an idea where possible places that configure these 1082 ports as a proxy?

  • Did you look up your environment’s Proxy settings including HTTP PROXY, HTTPS PROXY and NO_PROXY environment variables.Other files like curl, wget or docker configuration can as well be examined for possible declaration of a proxy.
  • Let's find out if there are firewalls or network security policies redirecting traffic to port 1082.
  • Consider the Kubernetes configuration to find out whether it has anything on proxies. Check also kube proxy,configuration like env vars.
  • For example, if you are using Docker as a container runtime, ensure their configurations do not contain any abnormal network configurations and proxy settings.
  • There may be some applications or tools having particular ports for internal communication. So what processes or services are running on port 1082

NOTE:

Temporarily turn off all proxies in your environment to see if this is what causes the problem.

Look for any error messages related to the network connectivity or proxy issues in the k8s logs.

If you are unable to resolve the proxy issue, try pulling the image from a different registry to rule out registry specific problems.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.