Skip to content

Conversation

cmacknz
Copy link
Member

@cmacknz cmacknz commented Jun 30, 2025

https://github.com/open-telemetry/opentelemetry-collector/tree/main/connector/forwardconnector

Include the forwardconnector, which we are highly likely to need for OTel native integrations to allow for unique processors per receiver without having to have unique pipelines with duplicate exporters.

To test this PR I used the following configuration showing how to have 2 filelog receivers that could have separate processors applied but export via the same exporter.

receivers: filelog: include: [ /var/log/system.log ] start_at: beginning filelog/2: include: [ /var/log/wifi.log ] start_at: beginning processors: resource: attributes: - key: service.name action: insert value: elastic-otel-test exporters: debug: verbosity: detailed sampling_initial: 10000 sampling_thereafter: 10000 connectors: forward: extensions: memory_limiter: check_interval: 1s limit_mib: 700 spike_limit_mib: 180 health_check: endpoint: "localhost:13133" path: "/health/status" service: extensions: [health_check, memory_limiter] pipelines: logs/system: receivers: [filelog] processors: [resource] exporters: [forward] logs/wifi: receivers: [filelog/2] processors: [resource] exporters: [forward] logs/export: receivers: [forward] exporters: [debug]
@cmacknz cmacknz self-assigned this Jun 30, 2025
@cmacknz cmacknz added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Jun 30, 2025
@cmacknz cmacknz requested a review from a team as a code owner June 30, 2025 21:06
@cmacknz cmacknz added the backport-8.19 Automated backport to the 8.19 branch label Jun 30, 2025
@cmacknz cmacknz requested a review from ycombinator June 30, 2025 21:06
@cmacknz cmacknz added the backport-9.1 Automated backport to the 9.1 branch label Jun 30, 2025
@cmacknz cmacknz requested a review from pchila June 30, 2025 21:06
@elasticmachine
Copy link
Collaborator

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

Copy link
Contributor

@pkoutsovasilis pkoutsovasilis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed forwardconnector seems that it does the trick, so LGTM. Only one minor thingie @cmacknz please run mage fmt and repush 😄

Copy link
Member

@pchila pchila left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks ok, I wonder if the new component should be listed in internal/pkg/otel/core-components.yaml as introduced by PR #8530

Holding off approval till CI is green

@cmacknz
Copy link
Member Author

cmacknz commented Jul 2, 2025

Core components means users can open SDHs against them and we'll try to fix them. I don't think we we are at the point where we need that for the foward connector as we don't depend on it for anything yet.

@swiatekm
Copy link
Contributor

swiatekm commented Jul 2, 2025

@cmacknz could you add this component to this file? That way we can be confident it's actually possible to start a collector with it included in a pipeline.

jsoriano
jsoriano previously approved these changes Jul 2, 2025
Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@cmacknz
Copy link
Member Author

cmacknz commented Jul 2, 2025

@cmacknz could you add this component to this file? That way we can be confident it's actually possible to start a collector with it included in a pipeline.

Done in 1d856a2

@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

History

cc @cmacknz

Copy link
Member

@pchila pchila left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@cmacknz cmacknz merged commit 393e2ed into elastic:main Jul 3, 2025
19 checks passed
mergify bot pushed a commit that referenced this pull request Jul 3, 2025
* EDOT collector: include the forward connector. https://github.com/open-telemetry/opentelemetry-collector/tree/main/connector/forwardconnector * Add changelog. * mage fmt. * Add forward connector to all-components.yml (cherry picked from commit 393e2ed)
mergify bot pushed a commit that referenced this pull request Jul 3, 2025
* EDOT collector: include the forward connector. https://github.com/open-telemetry/opentelemetry-collector/tree/main/connector/forwardconnector * Add changelog. * mage fmt. * Add forward connector to all-components.yml (cherry picked from commit 393e2ed)
cmacknz added a commit that referenced this pull request Jul 3, 2025
* EDOT collector: include the forward connector. https://github.com/open-telemetry/opentelemetry-collector/tree/main/connector/forwardconnector * Add changelog. * mage fmt. * Add forward connector to all-components.yml (cherry picked from commit 393e2ed) Co-authored-by: Craig MacKenzie <craig.mackenzie@elastic.co>
cmacknz added a commit that referenced this pull request Jul 3, 2025
* EDOT collector: include the forward connector. https://github.com/open-telemetry/opentelemetry-collector/tree/main/connector/forwardconnector * Add changelog. * mage fmt. * Add forward connector to all-components.yml (cherry picked from commit 393e2ed) Co-authored-by: Craig MacKenzie <craig.mackenzie@elastic.co>
v1v added a commit to v1v/elastic-agent that referenced this pull request Jul 6, 2025
…ng-oblt-cli * upstream: (37 commits) [AutoOps] Add `autoops-es.yml` to Packages (elastic#8728) EDOT collector: include the forward connector. (elastic#8753) Revert "ci: pin elastic-agent version (elastic#8736)" (elastic#8754) bk: retry Start ESS stack for integration tests (elastic#8553) Re-enable TestStandaloneUpgradeRollbackOnRestarts on windows (elastic#8718) removed reviewers from dependabot.yml (elastic#8709) Pass `--header` enrollment option to fleet-server (elastic#8071) Add ability for local output configuration to add to policy configuration (elastic#8766) Bump up github.com/go-viper/mapstructure/v2 dependency (elastic#8764) [Synthetics] Upgrade node to latest lts v20 (elastic#8712) [CI] BK Vault plugin for EC access (elastic#8377) feat: singleTest mage target for each integration test package (elastic#8691) ci: always include 8.19 LTS release branch in snapshots of test versions (elastic#8761) build(deps): bump github.com/elastic/mito from 1.19.0 to 1.20.0 (elastic#8755) chore: fix elastic-agent helm chart examples (elastic#8765) feat: support onboarding-id for kubernetes (elastic#8692) [main][Automation] Bump VM Image version to 1751072471 (elastic#8734) ci: revert deployment_csp_configuration.yaml to create_deployment_csp_configuration.yaml (elastic#8746) kustomize: pin kube-state-metrics version (elastic#8739) ci: pin elastic-agent version (elastic#8736) ...
v1v added a commit that referenced this pull request Jul 6, 2025
* upstream: (39 commits) Fix otel extension status reporting (#8696) Refactor user change on service (#8347) [AutoOps] Add `autoops-es.yml` to Packages (#8728) EDOT collector: include the forward connector. (#8753) Revert "ci: pin elastic-agent version (#8736)" (#8754) bk: retry Start ESS stack for integration tests (#8553) Re-enable TestStandaloneUpgradeRollbackOnRestarts on windows (#8718) removed reviewers from dependabot.yml (#8709) Pass `--header` enrollment option to fleet-server (#8071) Add ability for local output configuration to add to policy configuration (#8766) Bump up github.com/go-viper/mapstructure/v2 dependency (#8764) [Synthetics] Upgrade node to latest lts v20 (#8712) [CI] BK Vault plugin for EC access (#8377) feat: singleTest mage target for each integration test package (#8691) ci: always include 8.19 LTS release branch in snapshots of test versions (#8761) build(deps): bump github.com/elastic/mito from 1.19.0 to 1.20.0 (#8755) chore: fix elastic-agent helm chart examples (#8765) feat: support onboarding-id for kubernetes (#8692) [main][Automation] Bump VM Image version to 1751072471 (#8734) ci: revert deployment_csp_configuration.yaml to create_deployment_csp_configuration.yaml (#8746) ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-8.19 Automated backport to the 8.19 branch backport-9.1 Automated backport to the 9.1 branch Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team

6 participants