Skip to content

Conversation

@aleksmaus
Copy link
Contributor

@aleksmaus aleksmaus commented Sep 17, 2020

What does this PR do?

OSQuery package ported from POC PR elastic/package-storage#98

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all datasets collect metrics or logs.

The latter requires osquerybeat that is currently is not a part of the agent distribution by default.

Screenshots

Screen Shot 2020-09-17 at 10 30 46 AM

This doesn't need to be merged right away, the development is still pending on product input and the shape of the package is expected to change.

@aleksmaus aleksmaus added the enhancement New feature or request label Sep 17, 2020
@elasticmachine
Copy link

elasticmachine commented Sep 17, 2020

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Branch indexing]

  • Start Time: 2020-11-24T22:28:10.203+0000

  • Duration: 2 min 30 sec

Steps errors 1

Expand to view the steps failures

Check integration: osquery

  • Took 0 min 0 sec . View more details on here

Log output

Expand to view the last 100 lines of log output

[2020-11-24T22:30:38.133Z] 2020/11/24 22:30:37 DEBUG Write README.md file (package: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/nginx) [2020-11-24T22:30:38.133Z] 2020/11/24 22:30:37 DEBUG Create directories: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/nginx/docs [2020-11-24T22:30:38.133Z] 2020/11/24 22:30:37 DEBUG Write README.md file to: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/nginx/docs/README.md [2020-11-24T22:30:38.133Z] README.md file rendered: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/nginx/docs/README.md [2020-11-24T22:30:38.133Z] 2020/11/24 22:30:37 DEBUG Build directory: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/build/integrations/nginx/0.3.7 [2020-11-24T22:30:38.134Z] 2020/11/24 22:30:37 DEBUG Clear target directory (path: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/build/integrations/nginx/0.3.7) [2020-11-24T22:30:38.134Z] 2020/11/24 22:30:37 DEBUG Copy package content (source: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/nginx) [2020-11-24T22:30:38.134Z] 2020/11/24 22:30:37 DEBUG Encode dashboards [2020-11-24T22:30:38.134Z] Package built: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/build/integrations/nginx/0.3.7 [2020-11-24T22:30:38.134Z] Done [2020-11-24T22:30:38.154Z] Running in /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/o365 [2020-11-24T22:30:38.441Z] + ../../build/elastic-package check -v [2020-11-24T22:30:38.441Z] 2020/11/24 22:30:38 DEBUG Enable verbose logging [2020-11-24T22:30:38.441Z] Format the package [2020-11-24T22:30:38.441Z] Done [2020-11-24T22:30:38.441Z] Lint the package [2020-11-24T22:30:38.441Z] 2020/11/24 22:30:38 DEBUG Check if README.md is up-to-date [2020-11-24T22:30:38.441Z] 2020/11/24 22:30:38 DEBUG Generate README.md file (package: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/o365) [2020-11-24T22:30:38.441Z] 2020/11/24 22:30:38 DEBUG Template file for README.md found: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/o365/_dev/build/docs/README.md [2020-11-24T22:30:38.441Z] 2020/11/24 22:30:38 DEBUG Render README.md file (package: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/o365, templatePath: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/o365/_dev/build/docs/README.md) [2020-11-24T22:30:38.441Z] 2020/11/24 22:30:38 DEBUG Read existing README.md file (package: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/o365) [2020-11-24T22:30:38.441Z] Done [2020-11-24T22:30:38.441Z] Build the package [2020-11-24T22:30:38.441Z] 2020/11/24 22:30:38 DEBUG Update the README.md file [2020-11-24T22:30:38.441Z] 2020/11/24 22:30:38 DEBUG Generate README.md file (package: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/o365) [2020-11-24T22:30:38.441Z] 2020/11/24 22:30:38 DEBUG Template file for README.md found: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/o365/_dev/build/docs/README.md [2020-11-24T22:30:38.442Z] 2020/11/24 22:30:38 DEBUG Render README.md file (package: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/o365, templatePath: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/o365/_dev/build/docs/README.md) [2020-11-24T22:30:38.442Z] 2020/11/24 22:30:38 DEBUG Write README.md file (package: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/o365) [2020-11-24T22:30:38.442Z] 2020/11/24 22:30:38 DEBUG Create directories: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/o365/docs [2020-11-24T22:30:38.442Z] 2020/11/24 22:30:38 DEBUG Write README.md file to: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/o365/docs/README.md [2020-11-24T22:30:38.442Z] README.md file rendered: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/o365/docs/README.md [2020-11-24T22:30:38.442Z] 2020/11/24 22:30:38 DEBUG Build directory: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/build/integrations/o365/0.2.5 [2020-11-24T22:30:38.442Z] 2020/11/24 22:30:38 DEBUG Clear target directory (path: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/build/integrations/o365/0.2.5) [2020-11-24T22:30:38.442Z] 2020/11/24 22:30:38 DEBUG Copy package content (source: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/o365) [2020-11-24T22:30:38.442Z] 2020/11/24 22:30:38 DEBUG Encode dashboards [2020-11-24T22:30:38.442Z] Package built: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/build/integrations/o365/0.2.5 [2020-11-24T22:30:38.442Z] Done [2020-11-24T22:30:38.461Z] Running in /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/okta [2020-11-24T22:30:38.747Z] + ../../build/elastic-package check -v [2020-11-24T22:30:38.747Z] 2020/11/24 22:30:38 DEBUG Enable verbose logging [2020-11-24T22:30:38.747Z] Format the package [2020-11-24T22:30:38.747Z] Done [2020-11-24T22:30:38.747Z] Lint the package [2020-11-24T22:30:38.747Z] 2020/11/24 22:30:38 DEBUG Check if README.md is up-to-date [2020-11-24T22:30:38.747Z] 2020/11/24 22:30:38 DEBUG Generate README.md file (package: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/okta) [2020-11-24T22:30:38.747Z] 2020/11/24 22:30:38 DEBUG Template file for README.md found: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/okta/_dev/build/docs/README.md [2020-11-24T22:30:38.747Z] 2020/11/24 22:30:38 DEBUG Render README.md file (package: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/okta, templatePath: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/okta/_dev/build/docs/README.md) [2020-11-24T22:30:38.747Z] 2020/11/24 22:30:38 DEBUG Read existing README.md file (package: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/okta) [2020-11-24T22:30:38.747Z] Done [2020-11-24T22:30:38.747Z] Build the package [2020-11-24T22:30:38.747Z] 2020/11/24 22:30:38 DEBUG Update the README.md file [2020-11-24T22:30:38.747Z] 2020/11/24 22:30:38 DEBUG Generate README.md file (package: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/okta) [2020-11-24T22:30:38.747Z] 2020/11/24 22:30:38 DEBUG Template file for README.md found: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/okta/_dev/build/docs/README.md [2020-11-24T22:30:38.747Z] 2020/11/24 22:30:38 DEBUG Render README.md file (package: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/okta, templatePath: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/okta/_dev/build/docs/README.md) [2020-11-24T22:30:38.747Z] 2020/11/24 22:30:38 DEBUG Write README.md file (package: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/okta) [2020-11-24T22:30:38.747Z] 2020/11/24 22:30:38 DEBUG Create directories: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/okta/docs [2020-11-24T22:30:38.747Z] 2020/11/24 22:30:38 DEBUG Write README.md file to: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/okta/docs/README.md [2020-11-24T22:30:38.747Z] README.md file rendered: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/okta/docs/README.md [2020-11-24T22:30:38.747Z] 2020/11/24 22:30:38 DEBUG Build directory: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/build/integrations/okta/0.2.4 [2020-11-24T22:30:38.747Z] 2020/11/24 22:30:38 DEBUG Clear target directory (path: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/build/integrations/okta/0.2.4) [2020-11-24T22:30:38.747Z] 2020/11/24 22:30:38 DEBUG Copy package content (source: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/okta) [2020-11-24T22:30:38.747Z] 2020/11/24 22:30:38 DEBUG Encode dashboards [2020-11-24T22:30:38.747Z] Package built: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/build/integrations/okta/0.2.4 [2020-11-24T22:30:38.747Z] Done [2020-11-24T22:30:38.767Z] Running in /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/osquery [2020-11-24T22:30:39.054Z] + ../../build/elastic-package check -v [2020-11-24T22:30:39.054Z] 2020/11/24 22:30:38 DEBUG Enable verbose logging [2020-11-24T22:30:39.054Z] Format the package [2020-11-24T22:30:39.054Z] Done [2020-11-24T22:30:39.054Z] Lint the package [2020-11-24T22:30:39.054Z] 2020/11/24 22:30:38 DEBUG Check if README.md is up-to-date [2020-11-24T22:30:39.054Z] 2020/11/24 22:30:38 DEBUG Generate README.md file (package: /var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/osquery) [2020-11-24T22:30:39.054Z] 2020/11/24 22:30:38 DEBUG README file is static, can't be generated from the template file [2020-11-24T22:30:39.054Z] Error: checking package failed: linting package failed: found 12 validation errors: [2020-11-24T22:30:39.054Z] 1. item [2c5565b0-bd32-11ea-9023-9f2ec2bc3202.json] is not allowed in folder [/var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/osquery/kibana/dashboard] [2020-11-24T22:30:39.054Z] 2. item [50c10930-bf6d-11ea-aeb5-6b987e54d041.json] is not allowed in folder [/var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/osquery/kibana/dashboard] [2020-11-24T22:30:39.054Z] 3. item [04df1ce0-bf6e-11ea-aeb5-6b987e54d041.json] is not allowed in folder [/var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/osquery/kibana/search] [2020-11-24T22:30:39.054Z] 4. item [2e25d300-bf6e-11ea-aeb5-6b987e54d041.json] is not allowed in folder [/var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/osquery/kibana/search] [2020-11-24T22:30:39.054Z] 5. item [f6b55eb0-bd31-11ea-9023-9f2ec2bc3202.json] is not allowed in folder [/var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/osquery/kibana/search] [2020-11-24T22:30:39.054Z] 6. item [14068c70-bf6f-11ea-aeb5-6b987e54d041.json] is not allowed in folder [/var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/osquery/kibana/visualization] [2020-11-24T22:30:39.054Z] 7. item [20d2d880-bf71-11ea-aeb5-6b987e54d041.json] is not allowed in folder [/var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/osquery/kibana/visualization] [2020-11-24T22:30:39.054Z] 8. item [6384d030-bd3b-11ea-a4c2-23fe3946f872.json] is not allowed in folder [/var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/osquery/kibana/visualization] [2020-11-24T22:30:39.055Z] 9. item [7cfacb20-bf6e-11ea-aeb5-6b987e54d041.json] is not allowed in folder [/var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/osquery/kibana/visualization] [2020-11-24T22:30:39.055Z] 10. item [b130a710-bd3a-11ea-a4c2-23fe3946f872.json] is not allowed in folder [/var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/osquery/kibana/visualization] [2020-11-24T22:30:39.055Z] 11. item [b35f4080-bd32-11ea-9023-9f2ec2bc3202.json] is not allowed in folder [/var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/osquery/kibana/visualization] [2020-11-24T22:30:39.055Z] 12. item [f98a8a70-bf70-11ea-aeb5-6b987e54d041.json] is not allowed in folder [/var/lib/jenkins/workspace/gest-manager_integrations_PR-273/src/github.com/elastic/integrations/packages/osquery/kibana/visualization] [2020-11-24T22:30:39.055Z] [2020-11-24T22:30:39.126Z] Stage "Test integrations" skipped due to earlier failure(s) [2020-11-24T22:30:39.146Z] Stage "Update Package Storage" skipped due to earlier failure(s) [2020-11-24T22:30:39.392Z] Running on Jenkins in /var/lib/jenkins/workspace/gest-manager_integrations_PR-273 [2020-11-24T22:30:39.586Z] [INFO] getVaultSecret: Getting secrets [2020-11-24T22:30:39.642Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID [2020-11-24T22:30:40.238Z] + chmod 755 generate-build-data.sh [2020-11-24T22:30:40.238Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/integrations/PR-273/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/integrations/PR-273/runs/7 FAILURE 149773 [2020-11-24T22:30:40.238Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/integrations/PR-273/runs/7/steps/?limit=10000 -o steps-info.json [2020-11-24T22:30:40.488Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/integrations/PR-273/runs/7/tests/?status=FAILED -o tests-errors.json [2020-11-24T22:30:40.739Z] Retry 1/3 exited 22, retrying in 1 seconds... [2020-11-24T22:30:42.082Z] Retry 2/3 exited 22, retrying in 2 seconds... [2020-11-24T22:30:43.944Z] Retry 3/3 exited 22, no more retries left. [2020-11-24T22:30:43.944Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/integrations/PR-273/runs/7/log/ -o pipeline-log.txt 

@andresrc andresrc added Team:Integrations Label for the Integrations team Team:SIEM (Deprecated) labels Sep 17, 2020
@elasticmachine
Copy link

Pinging @elastic/integrations (Team:Integrations)

@elasticmachine
Copy link

Pinging @elastic/siem (Team:SIEM)

@mtojek
Copy link
Contributor

mtojek commented Sep 21, 2020

The CI is unhappy at the moment. Please run mage check locally and commit modified files. Most likely it's about formatting.

@mtojek mtojek self-requested a review September 21, 2020 09:59
Copy link
Contributor

@mtojek mtojek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI notified about issues.

@aleksmaus
Copy link
Contributor Author

@mtojek mage check ran cleanly before as still finishes cleanly on the same branch locally, just checked

@mtojek
Copy link
Contributor

mtojek commented Sep 21, 2020

Hm.. it's weird. Could you please push-force changes to restart CI job? I suppose you merged potential changes from master.

EDIT:

I see you're missing some changes from master branch. Please rebase it first and we can check then.

Copy link

@jonathan-buttner jonathan-buttner Sep 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think these json blobs might need to be decoded 🤔 . I think the scripts in this repo handle encoding them before they are copied to the package-storage repo.

Example decoded: https://github.com/elastic/integrations/blob/master/packages/aws/kibana/dashboard/aws-0eb5a6a0-694f-11ea-b0ac-95d4ecb1fecd.json#L18

Example encoded in package storage: https://github.com/elastic/package-storage/blob/production/packages/aws/0.2.4/kibana/dashboard/aws-0eb5a6a0-694f-11ea-b0ac-95d4ecb1fecd.json#L9

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are from POC and sure will change as well. Will update to have it decoded.

@aleksmaus
Copy link
Contributor Author

Rebased, so had to force push.

@mtojek
Copy link
Contributor

mtojek commented Oct 2, 2020

Please adjust the PR accordingly to changes introduced in #280 .

Main changes:

  1. Rebase against master.
  2. Rename "dataset" folder to "data_stream".
  3. Rename "config_templates" to "policy_templates" in package manifest file.
@aleksmaus aleksmaus requested a review from mtojek October 12, 2020 14:17
@mtojek
Copy link
Contributor

mtojek commented Nov 23, 2020

@aleksmaus I'm afraid this PR is not compliant with latest changes in the Integrations. Would you like to re-adjust it or prefer to close it?

@aleksmaus
Copy link
Contributor Author

will check what's up and update as needed.

@elasticmachine
Copy link

elasticmachine commented Nov 30, 2020

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #273 updated

    • Start Time: 2021-01-27T19:35:55.962+0000
  • Duration: 9 min 34 sec

  • Commit: dba46ac

Test stats 🧪

Test Results
Failed 0
Passed 1
Skipped 0
Total 1

@andrewkroh andrewkroh mentioned this pull request Dec 10, 2020
2 tasks
@aleksmaus
Copy link
Contributor Author

Updated the package naming in order to resolve the conflict with existing osquery filebeat package

Screen Shot 2021-01-26 at 4 00 05 PM

Screen Shot 2021-01-26 at 4 00 20 PM

Screen Shot 2021-01-26 at 4 19 05 PM

@aleksmaus
Copy link
Contributor Author

Removed all the kibana files for now, these were for POC.
@mtojek

@aleksmaus aleksmaus changed the title OSQuery POC package ported from package-storage OSQuery Elastic Managed package Jan 26, 2021
@aleksmaus
Copy link
Contributor Author

Screen Shot 2021-01-27 at 2 34 48 PM

This makes the input_type more cleaner and concise for the actions that will target osquery.
@aleksmaus aleksmaus merged commit d393a8d into elastic:master Jan 27, 2021
@andrewkroh andrewkroh added the New Integration Issue or pull request for creating a new integration package. label Aug 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request New Integration Issue or pull request for creating a new integration package. Team:Integrations Label for the Integrations team Team:SIEM (Deprecated)

7 participants