Add contract testing for policies (#1847)

Add a new policy test runner that can be used to test different sets of configuration values. With this test runner, a set of package or data stream variables can be used to create a policy, and the policy can be compared with a pre-generated one. Examples are provided for the sql_input and apache test packages. --------- Co-authored-by: Mario Rodriguez Molins <marrodmo@gmail.com>

Author: jsoriano

README.md

@@ -568,7 +568,12 @@ For details on how to run static tests for a package, see the [HOWTO guide](http
 #### System Tests
 These tests allow you to test a package's ability to ingest data end-to-end.
 
-For details on how to configure amd run system tests, review the [HOWTO guide](https://github.com/elastic/elastic-package/blob/main/docs/howto/system_testing.md).
+For details on how to configure and run system tests, review the [HOWTO guide](https://github.com/elastic/elastic-package/blob/main/docs/howto/system_testing.md).
+
+#### Policy Tests
+These tests allow you to test different configuration options and the policies they generate, without needing to run a full scenario.
+
+For details on how to configure and run policy tests, review the [HOWTO guide](https://github.com/elastic/elastic-package/blob/main/docs/howto/policy_testing.md).
 
 ### `elastic-package test asset`
 
@@ -582,6 +587,12 @@ _Context: package_
 
 Run pipeline tests for the package.
 
+### `elastic-package test policy`
+
+_Context: package_
+
+Run policy tests for the package.
+
 ### `elastic-package test static`
 
 _Context: package_

cmd/testrunner.go

@@ -50,7 +50,12 @@ For details on how to run static tests for a package, see the [HOWTO guide](http
 #### System Tests
 These tests allow you to test a package's ability to ingest data end-to-end.
 
-For details on how to configure amd run system tests, review the [HOWTO guide](https://github.com/elastic/elastic-package/blob/main/docs/howto/system_testing.md).`
+For details on how to configure and run system tests, review the [HOWTO guide](https://github.com/elastic/elastic-package/blob/main/docs/howto/system_testing.md).
+
+#### Policy Tests
+These tests allow you to test different configuration options and the policies they generate, without needing to run a full scenario.
+
+For details on how to configure and run policy tests, review the [HOWTO guide](https://github.com/elastic/elastic-package/blob/main/docs/howto/policy_testing.md).`
 
 func setupTestCommand() *cobraext.Command {
 var testTypeCmdActions []cobraext.CommandAction
@@ -314,7 +319,7 @@ func testTypeCommandActionFactory(runner testrunner.TestRunner) cobraext.Command
 }
 
 var kibanaClient *kibana.Client
-if testType == "system" || testType == "asset" {
+if testType == "system" || testType == "asset" || testType == "policy" {
 // pipeline and static tests do not require a kibana client to perform their required operations
 kibanaClient, err = stack.NewKibanaClientFromProfile(profile)
 if err != nil {

docs/howto/policy_testing.md

@@ -0,0 +1,114 @@
+# HOWTO: Writing policy tests for a package
+
+## Introduction
+Elastic Packages support a variety of configuration variables as defined in
+their manifest files. Policy tests allow to check that specific sets of
+variables are accepted by Fleet, and they produce an expected agent policy.
+
+## Defining policy tests
+
+Policy tests can be defined in data streams in integration packages, or at the
+package level in input packages.
+
+Each test is composed of two files, one for the configuration of the policy, and
+another one for the expected result.
+
+When defining the tests at the data stream level, they must be defined following
+this structure.
+```
+<package root>/
+ data_stream/
+ <data stream>/
+ _dev/
+ test/
+ policy/
+ test-<test name>.yml
+ test-<test name>.expected
+```
+
+When defining the tests at the package level, in input packages, they must be
+defined following this structure:
+```
+<package root>/
+ _dev/
+ test/
+ policy/
+ test-<test name>.yml
+ test-<test name>.expected
+```
+
+It is possible, and encouraged, to define multiple policy tests for each package
+or data stream.
+
+
+### Defining the configuration of the policy
+
+Test configuration for the policy is defined in a YAML file prefixed with
+`test-`.
+
+In these configuration files it is possible to define:
+- Values for the variables of the package manifest.
+- Values for the variables of the data stream manifests (only used for
+ integration packages).
+- Input to use, for packages supporting multiple input types.
+
+For example, the following configuration tells Fleet to create a policy using an
+specific input, and some variables at the package level:
+```
+input: httpjson
+vars:
+ url: http://localhost:1234/api/v1/logs
+ username: test
+ password: test
+```
+
+The following configuration would set a value for a variable defined at the data
+stream level:
+```
+data_stream:
+ vars:
+ paths:
+ - "/var/logs/apache/access.log*"
+```
+
+This configuration may look familiar to you if you are used to define system
+tests. The main difference is that policy tests are not going to be executed, so
+anything can be configured there, without expectations on having running
+services or reachable services. Also, no placeholders are expected to be found
+in policy tests.
+
+
+### Defining the expected policy
+
+Once you have decided the policy settings you would like to test, you should
+define the expected resulting policy. In principle it is possible to define it
+manually given that most of the information is included in the package, but it
+can be quite cumbersome. `elastic-package` is able to generate this file for
+you, using the `--generate` flag.
+
+If you run the policy tests with the `--generate` flag, `elastic-package` will
+write the found policy in the expected place.
+```
+$ elastic-package test policy --generate
+```
+
+Then check that the generated content is what you would expect to have.
+
+
+## Running policy tests
+
+You can run policy tests with the `elastic-package test` command. If a package
+includes policy tests, they will be executed if no test type is specified. You
+can also run the policy tests only indicating its type:
+```
+$ elastic-package test policy
+```
+
+With integration packages, you can run the policy tests for a single data
+stream, for example:
+```
+$ elastic-package test policy --data-streams access
+```
+
+Results are displayed using the usual format options. When the test fail,
+`elastic-package` shows the differences between the expected and found policy.

internal/kibana/policies.go

@@ -25,6 +25,9 @@ type Policy struct {
 DataOutputID string `json:"data_output_id,omitempty"`
 }
 
+// DownloadedPolicy represents a policy as returned by the download policy API.
+type DownloadedPolicy json.RawMessage
+
 // CreatePolicy persists the given Policy in Fleet.
 func (c *Client) CreatePolicy(ctx context.Context, p Policy) (*Policy, error) {
 reqBody, err := json.Marshal(p)
@@ -84,6 +87,22 @@ func (c *Client) GetPolicy(ctx context.Context, policyID string) (*Policy, error
 return &resp.Item, nil
 }
 
+// DownloadPolicy fetches the agent Policy as would be downloaded by an agent.
+func (c *Client) DownloadPolicy(ctx context.Context, policyID string) (DownloadedPolicy, error) {
+statusCode, respBody, err := c.get(ctx, fmt.Sprintf("%s/agent_policies/%s/download", FleetAPI, policyID))
+if err != nil {
+return nil, fmt.Errorf("could not get policy: %w", err)
+}
+if statusCode == http.StatusNotFound {
+return nil, &ErrPolicyNotFound{id: policyID}
+}
+if statusCode != http.StatusOK {
+return nil, fmt.Errorf("could not get policy; API status code = %d; response body = %s", statusCode, respBody)
+}
+
+return respBody, nil
+}
+
 // GetRawPolicy fetches the given Policy with all the fields in Fleet.
 func (c *Client) GetRawPolicy(ctx context.Context, policyID string) (json.RawMessage, error) {
 statusCode, respBody, err := c.get(ctx, fmt.Sprintf("%s/agent_policies/%s", FleetAPI, policyID))