[9.0] (backport #304) Proxy support

[mergify]

Description

Introduces proxy config to accept proxy URI to connect to Elasticsearch.

• Closes [Feature]: Add proxy support for the Elastic Integration Filter #303

Test

• For the local test, used SquidMan
• Config

// generator ... filter { elastic_integration { # cloud id => "es_integ_1" cloud_id => "my-cloud-id" api_key => "api-key" proxy => "http://127.0.0.1:8083" } // output 

• When proxy is stopped, cannot connect to ES

[2025-06-05T17:06:16,068][ERROR][logstash.javapipeline ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: Fetching Elasticsearch version information failed: Connection refused>, :backtrace=>["/Users/mashhur/Dev/elastic/ls-plugins/logstash-filter-elastic_integration/lib/logstash/filters/elastic_integration.rb:306:in `raise_config_error!'", "/Users/mashhur/Dev/elastic/ls-plugins/logstash-filter-elastic_integration/lib/logstash/filters/elastic_integration.rb:395:in `connected_es_version_info'", "/Users/mashhur/Dev/elastic/ls-plugins/logstash-filter-elastic_integration/lib/logstash/filters/elastic_integration.rb:428:in `serverless?'", "/Users/mashhur/Dev/elastic/ls-plugins/logstash-filter-elastic_integration/lib/logstash/filters/elastic_integration.rb:351:in `initialize_elasticsearch_rest_client!'", "/Users/mashhur/Dev/elastic/ls-plugins/logstash-filter-elastic_integration/lib/logstash/filters/elastic_integration.rb:136:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in `register'", "/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/java_pipeline.rb:245:in `block in register_plugins'", "org/jruby/RubyArray.java:1981:in `each'", "/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/java_pipeline.rb:244:in `register_plugins'", "/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/java_pipeline.rb:623:in `maybe_setup_out_plugins'", "/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/java_pipeline.rb:257:in `start_workers'", "/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/java_pipeline.rb:198:in `run'", "/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/java_pipeline.rb:150:in `block in start'"], "pipeline.sources"=>["/Users/mashhur/Dev/elastic/logstash/config/elastic_integration_simple.conf"], :thread=>"#<Thread:0x5bda8081 /Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/java_pipeline.rb:138 run>"} 

• When connected, successfully processes the data stream pipeline executions

╰─➤ bin/logstash -f config/elastic_integration_simple.conf --enable-local-plugin-development Using system java: /Users/mashhur/.sdkman/candidates/java/current/bin/java Sending Logstash logs to /Users/mashhur/Dev/elastic/logstash/logs which is now configured via log4j2.properties [2025-06-05T17:14:24,266][INFO ][logstash.runner ] Log4j configuration path used is: /Users/mashhur/Dev/elastic/logstash/config/log4j2.properties [2025-06-05T17:14:24,270][WARN ][logstash.runner ] The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead. [2025-06-05T17:14:24,271][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"9.0.2", "jruby.version"=>"jruby 9.4.9.0 (3.1.4) 2024-11-04 547c6b150e OpenJDK 64-Bit Server VM 21.0.5+11-LTS on 21.0.5+11-LTS +indy +jit [arm64-darwin]"} [2025-06-05T17:14:24,272][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11] [2025-06-05T17:14:24,293][INFO ][org.logstash.jackson.StreamReadConstraintsUtil] Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000` (logstash default) [2025-06-05T17:14:24,293][INFO ][org.logstash.jackson.StreamReadConstraintsUtil] Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000` (logstash default) [2025-06-05T17:14:24,293][INFO ][org.logstash.jackson.StreamReadConstraintsUtil] Jackson default value override `logstash.jackson.stream-read-constraints.max-nesting-depth` configured to `1000` (logstash default) [2025-06-05T17:14:24,307][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because command line options are specified [2025-06-05T17:14:24,560][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false} [2025-06-05T17:14:24,817][INFO ][org.reflections.Reflections] Reflections took 48 ms to scan 1 urls, producing 149 keys and 521 values [2025-06-05T17:14:24,863][INFO ][logstash.codecs.json ] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message) [2025-06-05T17:14:25,008][INFO ][logstash.javapipeline ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise. [2025-06-05T17:14:25,013][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//127.0.0.1"]} [2025-06-05T17:14:25,073][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://{cloud-host}.us-west-2.aws.found.io:443/]}} [2025-06-05T17:14:25,278][INFO ][logstash.outputs.elasticsearch][main] Connected to ES instance {:url=>"https://{outpus-es-host}.us-west-2.aws.found.io:443/"} [2025-06-05T17:14:25,279][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.17.0) {:es_version=>8} [2025-06-05T17:14:25,430][INFO ][co.elastic.logstash.filters.elasticintegration.PreflightCheck][main] Connected to Elasticsearch version: 8.17.0 [2025-06-05T17:14:25,431][INFO ][co.elastic.logstash.filters.elasticintegration.PreflightCheck][main] Elasticsearch build_flavor: default [2025-06-05T17:14:25,433][INFO ][logstash.filters.elasticintegration][main] by not manually configuring self-managed databases with `geoip_database_directory => ...` you accept and agree to the MaxMind EULA, which allows Elastic Integrations to use Logstash's Geoip Database Management service. For more details please visit https://www.maxmind.com/en/geolite2/eula [2025-06-05T17:14:26,268][INFO ][co.elastic.logstash.filters.elasticintegration.PreflightCheck][main] Elasticsearch license OK (active enterprise) [2025-06-05T17:14:26,268][INFO ][logstash.filters.elasticintegration][main] This 9.0.0 version of plugin embedded Ingest node components from Elasticsearch 9.0 [2025-06-05T17:14:26,268][WARN ][logstash.filters.elasticintegration][main] This plugin v9.0.0 is connected to an older MAJOR version of Elasticsearch v8.17.0, and may have trouble loading or running pipelines that use features that were deprecated before Elasticsearch v9.0; for the best experience, align major/minor versions across the Elastic Stack. [2025-06-05T17:14:26,274][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>10, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1250, "pipeline.sources"=>["/Users/mashhur/Dev/elastic/logstash/config/elastic_integration_simple.conf"], :thread=>"#<Thread:0xf0a96db /Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/java_pipeline.rb:138 run>"} [2025-06-05T17:14:26,608][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.33} [2025-06-05T17:14:26,625][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"} [2025-06-05T17:14:26,636][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} { "sequence" => 0, "@timestamp" => 2025-06-06T00:14:26.634535Z, "data_stream" => { "namespace" => "default", "type" => "logs", "dataset" => "citrix_waf.log" }, "@metadata" => { "target_ingest_pipeline" => "_none", "_ingest_document" => { "timestamp" => "2025-06-06T00:14:26.634535Z", "index" => "logs-citrix_waf.log-default" } }, "host" => "Mashhurs-MacBook-Pro.local", "@version" => "1", "message" => "Dec 18 21:46:17 <local0.info>{my-message} spt=1 method=GET", "event" => { "original" => "{\n \"data_stream\": {\n \"type\": \"logs\",\n \"dataset\": \"citrix_waf.log\",\n \"namespace\": \"default\"\n },\n \"message\": \"Dec 18 21:46:17 <local0.info> {redacted}\"\n }" } } { "sequence" => 1, "@timestamp" => 2025-06-06T00:14:26.635851Z, "data_stream" => { "namespace" => "default", "type" => "logs", "dataset" => "citrix_waf.log" }, "@metadata" => { "target_ingest_pipeline" => "_none", "_ingest_document" => { "timestamp" => "2025-06-06T00:14:26.635851Z", "index" => "logs-citrix_waf.log-default" } }, "host" => "Mashhurs-MacBook-Pro.local", "@version" => "1", "message" => "{redacted}", "event" => { "original" => "{\n \"data_stream\": {\n \"type\": \"logs\",\n \"dataset\": \"citrix_waf.log\",\n \"namespace\": \"default\"\n },\n \"message\": \"Dec 18 21:46:17 <local0.info> {redacted}\"\n }" } } [2025-06-05T17:14:27,315][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"} [2025-06-05T17:14:27,653][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main} [2025-06-05T17:14:27,661][INFO ][logstash.runner ] Logstash shut down. ╭─me ~/Dev/elastic/logstash ‹upstream-9.0*› ╰─➤ 

and we can check in apache-http debug logs:

[2025-06-05T17:21:12,843][DEBUG][org.apache.http.impl.nio.client.InternalHttpAsyncClient][main] [exchange: 1] Connection allocated: CPoolProxy{http-outgoing-0 [ACTIVE]} [2025-06-05T17:21:12,843][DEBUG][org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl][main] http-outgoing-0 127.0.0.1:51598<->127.0.0.1:8083[ACTIVE][r:]: Set attribute http.nio.exchange-handler [2025-06-05T17:21:12,843][DEBUG][org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl][main] http-outgoing-0 127.0.0.1:51598<->127.0.0.1:8083[ACTIVE][rw:]: Event set [w] [2025-06-05T17:21:12,843][DEBUG][org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl][main] http-outgoing-0 127.0.0.1:51598<->127.0.0.1:8083[ACTIVE][rw:]: Set timeout 0 [2025-06-05T17:21:12,843][DEBUG][org.apache.http.impl.nio.client.InternalIODispatch][main] http-outgoing-0 [ACTIVE]: Connected [2025-06-05T17:21:12,844][DEBUG][org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl][main] http-outgoing-0 127.0.0.1:51598<->127.0.0.1:8083[ACTIVE][rw:]: Set attribute http.nio.http-exchange-state **** ```<hr>This is an automatic backport of pull request #304 done by [Mergify](https://mergify.com). 

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 8ab17b1

Failed CI Steps

• Unit test for 9.0.4-SNAPSHOT, snapshot: true
• Integration test for 9.0.4-SNAPSHOT, snapshot: true

History

cc @mashhurs

[mashhurs]

clean backport!
LGTM!