[SSI integrations] Set response.split.ignore_empty_value: true

[chrisberkhout]

Proposed commit message

[SSI integrations] Set `response.split.ignore_empty_value: true` (#) For integrations that use `response.split` options to split on a list, don't explicitly set the `ignore_empty_value` option, and don't keep parent fields, we usually don't want to index a document if the target list is empty. However, the default functionality is to index the whole document if the split target is empty, so this change sets `ignore_empty_value: true` explicitly. The [`response.split` documentation][1] says: > If the split target is empty the parent document will be kept. > If documents with empty splits should be dropped, the > `ignore_empty_value` option should be set to `true`. [1]: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-httpjson.html#response-split 

Discussion

The effects of the somewhat counterintuitive default split behavior are sometimes reported by users, as was the case in #9705.

I used the following Python script to identify and update relevant cases.

fix.py

#!/usr/bin/env python3 import os import fnmatch import re import yaml # define functions to generate and transform items def yml_hbs_files(root=os.getcwd()): files = [] for dirpath, dirnames, filenames in os.walk(root): for filename in fnmatch.filter(filenames, "*.yml.hbs"): files.append({ "file": os.path.join(dirpath, filename) }) return files def add_split_config(item): config = [] with open(item["file"], 'r') as file: is_split_config = False for line in file.readlines(): if line.startswith('response.split:'): is_split_config = True config = [] # one case has two configs and it's the second one that matters elif not line.startswith(' ') and not line.startswith('\t'): is_split_config = False if is_split_config: config.append(line) result = item.copy() result["split_config"] = config return result def add_not_templated(item): templated = False for line in item["split_config"]: if re.match(r".*{{.*}}.*", line): templated = True result = item.copy() result["not_templated"] = not templated return result def add_yaml(item): parsed = None if len(item["split_config"]) > 0 and item["not_templated"]: parsed = yaml.safe_load("".join(item["split_config"])) result = item.copy() result["yaml"] = parsed return result def add_is_list_type(item): is_list_type = False if item["yaml"]: is_list_type = item["yaml"]["response.split"].get("type", "list") == "list" result = item.copy() result["is_list_type"] = is_list_type return result def add_no_top_level_ignore_empty_value(item): absent = True if item["yaml"]: absent = not "ignore_empty_value" in item["yaml"]["response.split"] result = item.copy() result["no_top_level_ignore_empty_value"] = absent return result def add_false_top_level_keep_parent(item): is_set = False if item["yaml"]: is_set = item["yaml"]["response.split"].get("keep_parent", False) result = item.copy() result["false_top_level_keep_parent"] = not is_set return result def add_new_config_if_suitable(item): suitable = \ len(item["split_config"]) > 0 and \ item["not_templated"] and \ item["is_list_type"] and \ item["no_top_level_ignore_empty_value"] and \ item["false_top_level_keep_parent"] if suitable: new_yaml = item["yaml"].copy() new_yaml["response.split"] = {} for k, v in item["yaml"]["response.split"].items(): new_yaml["response.split"][k] = v if k == "target": # insert after top-level target param new_yaml["response.split"]["ignore_empty_value"] = True new_config = yaml.dump(new_yaml, sort_keys=False) result = item.copy() result["new_config"] = new_config return result else: return item # generate and transform items items = yml_hbs_files() items = map(add_split_config, items) items = map(add_not_templated, items) items = map(add_yaml, items) items = map(add_is_list_type, items) items = map(add_no_top_level_ignore_empty_value, items) items = map(add_false_top_level_keep_parent, items) items = map(add_new_config_if_suitable, items) items = list(items) # print summary selected = items.copy() print(" total selected") print(" --------- ---------") print("*.yml.hbs %9d %9d" % ( len(items), len(selected) )) selected = [i for i in selected if len(i["split_config"]) > 0] print("has split_config: %9d %9d" % ( sum(len(i["split_config"]) > 0 for i in items), len(selected) )) selected = [i for i in selected if i["not_templated"]] print("not_templated: %9d %9d" % ( sum(i["not_templated"] for i in items), len(selected) )) selected = [i for i in selected if i["is_list_type"]] print("is_list_type: %9d %9d" % ( sum(i["is_list_type"] for i in items), len(selected) )) selected = [i for i in selected if i["no_top_level_ignore_empty_value"]] print("no_top_level_ignore_empty_value: %9d %9d" % ( sum(i["no_top_level_ignore_empty_value"] for i in items), len(selected) )) selected = [i for i in selected if i["false_top_level_keep_parent"]] print("false_top_level_keep_parent: %9d %9d" % ( sum(i["false_top_level_keep_parent"] for i in items), len(selected) )) selected = [i for i in selected if "new_config" in i] print("new_config: %9d %9d" % ( sum("new_config" in i for i in items), len(selected) )) print() # # print details # for item in items: # if ( # len(item["split_config"]) > 0 and \ # item["not_templated"] and \ # item["is_list_type"] and \ # item["no_top_level_ignore_empty_value"] and \ # item["false_top_level_keep_parent"] and \ # "new_config" in item and \ # True # ): # print(item["file"]) # print() # print("".join(item["split_config"])) # print(item.get("new_config")) # print() # # apply changes # for item in items: # if "new_config" in item: # with open(item["file"], 'r', encoding='utf-8') as f: # content = f.read() # updated_content = content.replace("".join(item["split_config"]), item["new_config"]) # with open(item["file"], 'w', encoding='utf-8') as f: # f.write(updated_content) # print("%s updated" % item["file"])

It's summary was:

 total selected --------- --------- *.yml.hbs 1061 1061 has split_config: 186 186 not_templated: 1060 185 is_list_type: 133 133 no_top_level_ignore_empty_value: 1032 105 false_top_level_keep_parent: 1057 101 new_config: 101 101 

I left out the AWS Security Hub changes already handled in #9705.

Some minor formatting changes are included.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Related issues

• Relates [aws] Don't index empty AWS Security Hub responses #9705
• Supersedes [many integrations] Set response.split.ignore_empty_value: true #9815
• Closes [Proofpoint TAP] Empty array response should not generate an event #9965

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[chrisberkhout]

@efd6 This is the same as #9815, except only for SSI-only integrations and with version bumps and changelog updates. The commits separate out and revert the non-SSI changes.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: bcc5991

History

• 💚 Build #11863 succeeded 7e6f561072f022e688a0d70f6472649a9be6da11
• 💚 Build #11860 succeeded ad005d2

cc @chrisberkhout

[elasticmachine]

Package 1password - 1.28.0 containing this change is available at https://epr.elastic.co/search?package=1password

[elasticmachine]

Package atlassian_jira - 1.25.0 containing this change is available at https://epr.elastic.co/search?package=atlassian_jira

[elasticmachine]

Package bitwarden - 1.12.0 containing this change is available at https://epr.elastic.co/search?package=bitwarden

[elasticmachine]

Package carbon_black_cloud - 2.1.0 containing this change is available at https://epr.elastic.co/search?package=carbon_black_cloud

[elasticmachine]

Package cisa_kevs - 1.1.0 containing this change is available at https://epr.elastic.co/search?package=cisa_kevs

[elasticmachine]

Package cloudflare - 2.26.0 containing this change is available at https://epr.elastic.co/search?package=cloudflare

[elasticmachine]

Package forgerock - 1.16.0 containing this change is available at https://epr.elastic.co/search?package=forgerock

[elasticmachine]

Package google_scc - 1.3.0 containing this change is available at https://epr.elastic.co/search?package=google_scc

[elasticmachine]

Package ti_cybersixgill - 1.28.0 containing this change is available at https://epr.elastic.co/search?package=ti_cybersixgill

[elasticmachine]

Package ti_eset - 1.1.0 containing this change is available at https://epr.elastic.co/search?package=ti_eset

[elasticmachine]

Package ti_mandiant_advantage - 1.2.0 containing this change is available at https://epr.elastic.co/search?package=ti_mandiant_advantage

[elasticmachine]

Package ti_misp - 1.33.0 containing this change is available at https://epr.elastic.co/search?package=ti_misp

[elasticmachine]

Package ti_rapid7_threat_command - 1.16.0 containing this change is available at https://epr.elastic.co/search?package=ti_rapid7_threat_command

[elasticmachine]

Package ti_threatq - 1.27.0 containing this change is available at https://epr.elastic.co/search?package=ti_threatq

[elasticmachine]

Package trend_micro_vision_one - 1.18.0 containing this change is available at https://epr.elastic.co/search?package=trend_micro_vision_one

[elasticmachine]

Package zerofox - 1.24.0 containing this change is available at https://epr.elastic.co/search?package=zerofox

[elasticmachine]

Package zeronetworks - 1.14.0 containing this change is available at https://epr.elastic.co/search?package=zeronetworks