Convert logs integration to type: input

[hop-dev]

Closes #5162

Move the custom logs integration to be an input package.

Input packages can only move to GA if they specify a kibana version greater than or equal to 8.8

I have removed the default value for dataset so that dataset is not hidden in the "advanced" tab, the UI does still default it to generic. This is because I believe dataset should be more prominent for an input package.

Test Scenarios

to test, use the elastic-package install command from the logs directory.

• Upgrade an existing logs policy to the input package version
  • There should be no conflicts in the upgrade
  • Logs should not be re-ingested
  • Logs should continue to be ingested to the same index
  • no new index template should be created
• Create a new logs policy
  • on policy creation the templates for the specified dataset should be created, e.g helloworld dataset should create logs-helloworld index template

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-03-28T15:14:37.026+0000

• Duration: 15 min 43 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[BenB196]

Hi, I saw this PR and was wondering (and this could be the wrong place to ask this). But I believe that the logfile input is deprecated. With this PR introducing a major version increase and a minimum version of 8.8.0. Would it make sense to migrate this to the filestream input and leverage the new take_over functionality added in 8.7.0?

[hop-dev]

Hi @BenB196 sorry about the delay in replying.

I spoke to @rdner last week about this last week and his view was that as the takeover mode is still considered beta and it isn't ready for us to move the custom logs integration over just yet.

[rdner]

We're going to release a beta package with the filestream+take_over first and let people to try it out.

[hop-dev]

OK this is ready to merge, I have re-run my tests. Going to wait until Monday morning though 😅

[elasticmachine]

Package log - 2.0.0 containing this change is available at https://epr.elastic.co/search?package=log