On an on-prem GitLab server hosted on Kubernetes, after an upgrade of GitLab to the latest version 17.1.0, I notice that the webservice fails to connect to the registry service:
{"component": "gitlab","subcomponent":"exceptions_json","severity":"ERROR","time":"2024-06-23T19:24:45.983Z","correlation_id":"01J14RZ0P76QJCW9CBD0XHAY23","meta.caller_id":"Projects::Registry::RepositoriesController#index","meta.remote_ip":"10.42.3.47","meta.feature_category":"container_registry","meta.user":"paul","meta.user_id":2,"meta.project":"personal/my-project","meta.root_namespace":"personal","meta.client_id":"user/2","exception.class":"Faraday::ConnectionFailed","exception.message":"Failed to open TCP connection to gitlab-registry.gitlab.svc.cluster.local:5000 (execution expired)","exception.backtrace":["gems/gitlab-http/lib/net_http/connect_patch.rb:52:in `initialize'","gems/gitlab-http/lib/net_http/connect_patch.rb:52:in `open'","gems/gitlab-http/lib/net_http/connect_patch.rb:52:in `block in connect'","timeout (0.3.2) lib/timeout.rb:189:in `block in timeout'","timeout (0.3.2) lib/timeout.rb:196:in `timeout'","gems/gitlab-http/lib/net_http/connect_patch.rb:50:in `connect'","net-http (0.4.1) lib/net/http.rb:1580:in `do_start'","net-http (0.4.1) lib/net/http.rb:1569:in `start'","faraday-net_http (1.0.1) lib/faraday/adapter/net_http.rb:138:in `request_via_get_method'","faraday-net_http (1.0.1) lib/faraday/adapter/net_http.rb:129:in `request_with_wrapped_block'","faraday-net_http (1.0.1) lib/faraday/adapter/net_http.rb:122:in `perform_request'","faraday-net_http (1.0.1) lib/faraday/adapter/net_http.rb:66:in `block in call'","faraday (1.10.0) lib/faraday/adapter.rb:50:in `connection'","faraday-net_http (1.0.1) lib/faraday/adapter/net_http.rb:64:in `call'","lib/gitlab/faraday/error_callback.rb:30:in `call'","faraday-retry (1.0.3) lib/faraday/retry/middleware.rb:140:in `call'","faraday_middleware (1.2.0) lib/faraday_middleware/response_middleware.rb:36:in `call'","faraday_middleware (1.2.0) lib/faraday_middleware/response_middleware.rb:36:in `call'","faraday_middleware (1.2.0) lib/faraday_middleware/response_middleware.rb:36:in `call'","faraday_middleware (1.2.0) lib/faraday_middleware/response_middleware.rb:36:in `call'","faraday_middleware (1.2.0) lib/faraday_middleware/response_middleware.rb:36:in `call'","faraday_middleware (1.2.0) lib/faraday_middleware/response_middleware.rb:36:in `call'","faraday (1.10.0) lib/faraday/middleware.rb:18:in `call'","faraday_middleware (1.2.0) lib/faraday_middleware/request/encode_json.rb:26:in `call'","faraday (1.10.0) lib/faraday/rack_builder.rb:154:in `build_response'","faraday (1.10.0) lib/faraday/connection.rb:516:in `run_request'","faraday (1.10.0) lib/faraday/connection.rb:200:in `get'","lib/container_registry/client.rb:45:in `registry_info'","lib/container_registry/client.rb:40:in `block in registry_info'","lib/container_registry/base_client.rb:44:in `with_dummy_client'","lib/container_registry/client.rb:39:in `registry_info'","app/controllers/concerns/registry/connection_errors_handler.rb:35:in `ping_container_registry'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:400:in `block in make_lambda'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:199:in `block (2 levels) in halting'","actionpack (7.0.8.4) lib/abstract_controller/callbacks.rb:34:in `block (2 levels) in \u003cmodule:Callbacks\u003e'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:200:in `block in halting'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:595:in `block in invoke_before'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:595:in `each'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:595:in `invoke_before'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:116:in `block in run_callbacks'","app/controllers/application_controller.rb:468:in `set_current_admin'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:459:in `set_session_storage'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","lib/gitlab/i18n.rb:114:in `with_locale'","lib/gitlab/i18n.rb:120:in `with_user_locale'","app/controllers/application_controller.rb:450:in `set_locale'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","marginalia (1.11.1) lib/marginalia.rb:109:in `record_query_comment'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","app/controllers/application_controller.rb:443:in `set_current_context'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","sentry-rails (5.17.3) lib/sentry/rails/controller_transaction.rb:28:in `block in sentry_around_action'","sentry-ruby (5.17.3) lib/sentry/hub.rb:102:in `with_child_span'","sentry-ruby (5.17.3) lib/sentry-ruby.rb:490:in `with_child_span'","sentry-rails (5.17.3) lib/sentry/rails/controller_transaction.rb:14:in `sentry_around_action'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:138:in `run_callbacks'","actionpack (7.0.8.4) lib/abstract_controller/callbacks.rb:233:in `process_action'","actionpack (7.0.8.4) lib/action_controller/metal/rescue.rb:23:in `process_action'","actionpack (7.0.8.4) lib/action_controller/metal/instrumentation.rb:67:in `block in process_action'","activesupport (7.0.8.4) lib/active_support/notifications.rb:206:in `block in instrument'","activesupport (7.0.8.4) lib/active_support/notifications/instrumenter.rb:24:in `instrument'","activesupport (7.0.8.4) lib/active_support/notifications.rb:206:in `instrument'","actionpack (7.0.8.4) lib/action_controller/metal/instrumentation.rb:66:in `process_action'","actionpack (7.0.8.4) lib/action_controller/metal/params_wrapper.rb:259:in `process_action'","activerecord (7.0.8.4) lib/active_record/railties/controller_runtime.rb:27:in `process_action'","actionpack (7.0.8.4) lib/abstract_controller/base.rb:151:in `process'","actionview (7.0.8.4) lib/action_view/rendering.rb:39:in `process'","actionpack (7.0.8.4) lib/action_controller/metal.rb:188:in `dispatch'","actionpack (7.0.8.4) lib/action_controller/metal.rb:251:in `dispatch'","actionpack (7.0.8.4) lib/action_dispatch/routing/route_set.rb:49:in `dispatch'","actionpack (7.0.8.4) lib/action_dispatch/routing/route_set.rb:32:in `serve'","actionpack (7.0.8.4) lib/action_dispatch/routing/mapper.rb:18:in `block in \u003cclass:Constraints\u003e'","actionpack (7.0.8.4) lib/action_dispatch/routing/mapper.rb:48:in `serve'","actionpack (7.0.8.4) lib/action_dispatch/journey/router.rb:50:in `block in serve'","actionpack (7.0.8.4) lib/action_dispatch/journey/router.rb:32:in `each'","actionpack (7.0.8.4) lib/action_dispatch/journey/router.rb:32:in `serve'","actionpack (7.0.8.4) lib/action_dispatch/routing/route_set.rb:852:in `call'","gitlab-experiment (0.9.1) lib/gitlab/experiment/middleware.rb:19:in `call'","omniauth (2.1.0) lib/omniauth/strategy.rb:202:in `call!'","omniauth (2.1.0) lib/omniauth/strategy.rb:169:in `call'","flipper (0.26.2) lib/flipper/middleware/memoizer.rb:72:in `memoized_call'","flipper (0.26.2) lib/flipper/middleware/memoizer.rb:37:in `call'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `block in call'","lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled'","lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `call'","lib/gitlab/middleware/memory_report.rb:13:in `call'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/web_transaction.rb:46:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'","lib/gitlab/database/query_analyzer.rb:40:in `within'","lib/gitlab/middleware/query_analyzer.rb:11:in `call'","lib/gitlab/middleware/organizations/current.rb:20:in `call'","batch-loader (2.0.5) lib/batch_loader/middleware.rb:11:in `call'","rack-attack (6.7.0) lib/rack/attack.rb:103:in `call'","apollo_upload_server (2.1.6) lib/apollo_upload_server/middleware.rb:19:in `call'","lib/gitlab/middleware/multipart.rb:173:in `call'","rack-attack (6.7.0) lib/rack/attack.rb:127:in `call'","warden (1.2.9) lib/warden/manager.rb:36:in `block in call'","warden (1.2.9) lib/warden/manager.rb:34:in `catch'","warden (1.2.9) lib/warden/manager.rb:34:in `call'","rack-cors (2.0.1) lib/rack/cors.rb:102:in `call'","rack (2.2.8.1) lib/rack/tempfile_reaper.rb:15:in `call'","rack (2.2.8.1) lib/rack/etag.rb:27:in `call'","rack (2.2.8.1) lib/rack/conditional_get.rb:27:in `call'","rack (2.2.8.1) lib/rack/head.rb:12:in `call'","actionpack (7.0.8.4) lib/action_dispatch/http/permissions_policy.rb:38:in `call'","actionpack (7.0.8.4) lib/action_dispatch/http/content_security_policy.rb:36:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'","rack (2.2.8.1) lib/rack/session/abstract/id.rb:266:in `context'","rack (2.2.8.1) lib/rack/session/abstract/id.rb:260:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/cookies.rb:704:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:99:in `run_callbacks'","actionpack (7.0.8.4) lib/action_dispatch/middleware/callbacks.rb:26:in `call'","sentry-rails (5.17.3) lib/sentry/rails/rescued_exception_interceptor.rb:12:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/debug_exceptions.rb:28:in `call'","lib/gitlab/middleware/path_traversal_check.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","sentry-ruby (5.17.3) lib/sentry/rack/capture_exceptions.rb:29:in `block (2 levels) in call'","sentry-ruby (5.17.3) lib/sentry/hub.rb:251:in `with_session_tracking'","sentry-ruby (5.17.3) lib/sentry-ruby.rb:403:in `with_session_tracking'","sentry-ruby (5.17.3) lib/sentry/rack/capture_exceptions.rb:20:in `block in call'","sentry-ruby (5.17.3) lib/sentry/hub.rb:59:in `with_scope'","sentry-ruby (5.17.3) lib/sentry-ruby.rb:383:in `with_scope'","sentry-ruby (5.17.3) lib/sentry/rack/capture_exceptions.rb:19:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/show_exceptions.rb:29:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lograge (0.11.2) lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'","railties (7.0.8.4) lib/rails/rack/logger.rb:25:in `block in call'","activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:99:in `block in tagged'","activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:37:in `tagged'","activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:99:in `tagged'","railties (7.0.8.4) lib/rails/rack/logger.rb:25:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/remote_ip.rb:93:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:15:in `call'","lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'","request_store (1.5.1) lib/request_store/middleware.rb:19:in `call'","rack (2.2.8.1) lib/rack/method_override.rb:24:in `call'","rack (2.2.8.1) lib/rack/runtime.rb:22:in `call'","rack-timeout (0.6.3) lib/rack/timeout/core.rb:148:in `block in call'","rack-timeout (0.6.3) lib/rack/timeout/support/timeout.rb:19:in `timeout'","rack-timeout (0.6.3) lib/rack/timeout/core.rb:147:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/compressed_json.rb:44:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/executor.rb:14:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","rack (2.2.8.1) lib/rack/sendfile.rb:110:in `call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'","gitlab-labkit (0.36.0) lib/labkit/middleware/rack.rb:22:in `block in call'","gitlab-labkit (0.36.0) lib/labkit/context.rb:35:in `with_context'","gitlab-labkit (0.36.0) lib/labkit/middleware/rack.rb:21:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/request_id.rb:26:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/host_authorization.rb:131:in `call'","railties (7.0.8.4) lib/rails/engine.rb:530:in `call'","railties (7.0.8.4) lib/rails/railtie.rb:226:in `public_send'","railties (7.0.8.4) lib/rails/railtie.rb:226:in `method_missing'","lib/gitlab/middleware/release_env.rb:13:in `call'","rack (2.2.8.1) lib/rack/urlmap.rb:74:in `block in call'","rack (2.2.8.1) lib/rack/urlmap.rb:58:in `each'","rack (2.2.8.1) lib/rack/urlmap.rb:58:in `call'","puma (6.4.0) lib/puma/configuration.rb:272:in `call'","puma (6.4.0) lib/puma/request.rb:100:in `block in handle_request'","puma (6.4.0) lib/puma/thread_pool.rb:378:in `with_force_shutdown'","puma (6.4.0) lib/puma/request.rb:99:in `handle_request'","puma (6.4.0) lib/puma/server.rb:443:in `process_client'","puma (6.4.0) lib/puma/server.rb:241:in `block in run'","puma (6.4.0) lib/puma/thread_pool.rb:155:in `block in spawn_thread'"],"exception.cause_class":"Net::OpenTimeout","user.username":"paul","tags.program":"web","tags.locale":"en","tags.feature_category":"container_registry","tags.correlation_id":"01J14RZ0P76QJCW9CBD0XHAY23","extra.class":"ContainerRegistry::BaseClient","extra.url":"http://gitlab-registry.gitlab.svc.cluster.local:5000/v2/"} From the gitlab-runner pod, I can resolve the hostname and I can also cURL the URL from the error above.
gitlab-gitlab-runner-5dff88bf84-b7hpg:/$ curl -v http://gitlab-registry.gitlab.svc.cluster.local:5000 * Host gitlab-registry.gitlab.svc.cluster.local:5000 was resolved. * IPv6: (none) * IPv4: 10.43.168.151 * Trying 10.43.168.151:5000... * Connected to gitlab-registry.gitlab.svc.cluster.local (10.43.168.151) port 5000 > GET / HTTP/1.1 > Host: gitlab-registry.gitlab.svc.cluster.local:5000 > User-Agent: curl/8.5.0 > Accept: */* > < HTTP/1.1 200 OK < Cache-Control: no-cache < Date: Thu, 27 Jun 2024 11:27:52 GMT < Content-Length: 0 < * Connection #0 to host gitlab-registry.gitlab.svc.cluster.local left intact Also from the web service pod I can reach the port 5000 with cURL:
git@gitlab-webservice-default-695d898db4-6gtk4:/$ curl -v http://gitlab-registry.gitlab.svc.cluster.local:5000 * Trying 10.43.168.151:5000... * Connected to gitlab-registry.gitlab.svc.cluster.local (10.43.168.151) port 5000 (#0) > GET / HTTP/1.1 > Host: gitlab-registry.gitlab.svc.cluster.local:5000 > User-Agent: curl/7.88.1 > Accept: */* > < HTTP/1.1 200 OK < Cache-Control: no-cache < Date: Thu, 27 Jun 2024 11:30:29 GMT < Content-Length: 0 < * Connection #0 to host gitlab-registry.gitlab.svc.cluster.local left intact I can also reach the service if I forward the port or access it through a reverse proxy. There are no network policies configured.
I have changed the config map for the web service to update the url from http://gitlab-registry.gitlab.svc:5000 to http://gitlab-registry.gitlab.svc.cluster.local:5000 as the first hostname was not being resolved by the runner pod.
The service exists, the namespace is gitlab, I don't see any connection attempts in the registry pod logs from the web service.
