Skip to content

add logs only edot kube-stack configuration #8785

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Jul 8, 2025
Merged

add logs only edot kube-stack configuration #8785

merged 6 commits into from
Jul 8, 2025

Conversation

rogercoll
Copy link
Contributor

@rogercoll rogercoll commented Jul 2, 2025 •
edited
Loading

What does this PR do?

Adds a configuration for the kube-stack Helm Chart that focuses exclusively on collecting file-based logs. All metrics and traces receivers are disabled.

  • Cluster Stats Collector: Disabled because it primarily collects metrics. While the k8sobjects receiver can gather Kubernetes API events as logs, it has also been disabled due to the lack of meaningful structured data and visualization support.
  • APM: Both auto-instrumentation and OTLP receivers are disabled.

Why is it important?

Logs Essential onboarding flow.

cc @gbamparop @chrisdistasio

Checklist

  • I have read and understood the pull request guidelines of this project.
  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in ./changelog/fragments using the changelog tool
  • I have added an integration test or an E2E test

Disruptive User Impact

How to test this PR locally

Related issues

Questions to ask yourself

  • How are we going to support this in production?
  • How are we going to measure its adoption?
  • How are we going to debug this?
  • What are the metrics I should take care of?
  • ...
@mergify Mergify
Copy link
Contributor

mergify bot commented Jul 2, 2025

This pull request does not have a backport label. Could you fix it @rogercoll? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
  • backport-active-all is the label that automatically backports to all active branches.
  • backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
  • backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.
@rogercoll rogercoll marked this pull request as ready for review July 4, 2025 13:00
@rogercoll rogercoll requested review from a team as code owners July 4, 2025 13:00
@pierrehilbert pierrehilbert added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Jul 4, 2025
@elasticmachine
Copy link
Collaborator

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)
@pierrehilbert pierrehilbert requested review from swiatekm and removed request for kaanyalti July 4, 2025 13:03
@gbamparop
Copy link

cc @mykolaharmash
@rogercoll rogercoll mentioned this pull request Jul 4, 2025
Open
@swiatekm
Copy link
Contributor

swiatekm commented Jul 8, 2025

Could we additionally check that we actually send data to ES in the integration test?
@rogercoll rogercoll requested a review from ChrsMark July 8, 2025 13:47
@rogercoll
Copy link
Contributor Author

Could we additionally check that we actually send data to ES in the integration test?

@swiatekm That configuration will be shared in serverless Kibana only, relying on the managed OTLP endpoint. Please correct me if I am wrong, but I am not aware of having a serverless ES available in the elastic-agent testing infrastructure. That's why the other mOTLP only configuration is not testing ingested data either: https://github.com/elastic/elastic-agent/pull/8785/files#diff-5a1bb04237df0e807fe36f344425d6d89a7e0ddcf7a373499dbe695becb68d79R97
ChrsMark
ChrsMark reviewed Jul 8, 2025
View reviewed changes
@swiatekm
Copy link
Contributor

swiatekm commented Jul 8, 2025

Could we additionally check that we actually send data to ES in the integration test?

@swiatekm That configuration will be shared in serverless Kibana only, relying on the managed OTLP endpoint. Please correct me if I am wrong, but I am not aware of having a serverless ES available in the elastic-agent testing infrastructure. That's why the other mOTLP only configuration is not testing ingested data either: https://github.com/elastic/elastic-agent/pull/8785/files#diff-5a1bb04237df0e807fe36f344425d6d89a7e0ddcf7a373499dbe695becb68d79R97

We do in fact have some tests running against serverless, though there isn't great support for this in the agent testing framework itself. A test can check that it's running in serverless (and skip itself if not), but provisioning the stack has to be done manually in a bash script.

I'm fine not doing that just for this PR, but when we eventually make it more convenient to use serverless in agent integration tests, we should revisit these configurations.
swiatekm
swiatekm previously approved these changes Jul 8, 2025
View reviewed changes
@elasticmachine
Copy link
Collaborator

⏳ Build in-progress, with failures

Failed CI Steps

History

cc @rogercoll
@rogercoll rogercoll requested a review from swiatekm July 8, 2025 16:07
@rogercoll rogercoll added the backport-9.0 Automated backport to the 9.0 branch label Jul 8, 2025
@rogercoll rogercoll enabled auto-merge (squash) July 8, 2025 16:55
@elastic-sonarqube Elastic SonarQube
Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube
@rogercoll rogercoll merged commit 502cd5c into elastic:main Jul 8, 2025
20 checks passed
mergify bot pushed a commit that referenced this pull request Jul 8, 2025
@rogercoll @mergify
add logs only edot kube-stack configuration (#8785)
1f187c2 
* add logs only edot kube-stack configuration * rename logs values file * disable default pipelines * remove resource/k8s processor and use k8sattributes processor for service attributes (cherry picked from commit 502cd5c) # Conflicts: #	magefile.go
@mergify mergify bot mentioned this pull request Jul 8, 2025
Merged
8 tasks
rogercoll added a commit that referenced this pull request Jul 8, 2025
@mergify @rogercoll
[9.0](backport #8785) add logs only edot kube-stack configuration (#8890
cfc02b6 
) * add logs only edot kube-stack configuration (#8785) * add logs only edot kube-stack configuration * rename logs values file * disable default pipelines * remove resource/k8s processor and use k8sattributes processor for service attributes (cherry picked from commit 502cd5c) # Conflicts: #	magefile.go * fix: merge conflicts * rollback k8sattributes config * fix: labels indentation --------- Co-authored-by: Roger Coll <rogercoll@protonmail.com>
@rogercoll rogercoll added the backport-9.1 Automated backport to the 9.1 branch label Jul 21, 2025
mergify bot pushed a commit that referenced this pull request Jul 21, 2025
@rogercoll @mergify
add logs only edot kube-stack configuration (#8785)
0f42e7d 
* add logs only edot kube-stack configuration * rename logs values file * disable default pipelines * remove resource/k8s processor and use k8sattributes processor for service attributes (cherry picked from commit 502cd5c)
@mergify mergify bot mentioned this pull request Jul 21, 2025
Merged
8 tasks
rogercoll added a commit that referenced this pull request Jul 21, 2025
@mergify @rogercoll
[9.1] (backport #8785) add logs only edot kube-stack configuration (#…
2bcf7b5 
…9089) * add logs only edot kube-stack configuration (#8785) * add logs only edot kube-stack configuration * rename logs values file * disable default pipelines * remove resource/k8s processor and use k8sattributes processor for service attributes (cherry picked from commit 502cd5c) * fix: downgrade elastic-agent tag to 9.1.0 --------- Co-authored-by: Roger Coll <rogercoll@protonmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-9.0 Automated backport to the 9.0 branch backport-9.1 Automated backport to the 9.1 branch Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team

6 participants

@rogercoll @elasticmachine @gbamparop @swiatekm @ChrsMark @pierrehilbert