Too many packages failing in daily job summary

There are a bunch of packages that are failing daily build jobs. These issues can be found here.

The chronology of these shows that there are two versions that are showing the failures still (I assume that the other version are no longer being tested having passed into the past).

8.19.0: last 2025-08-11
8.19.1: last 2025-08-26
8.19.3: last 2025-09-02
8.19.4: last 2025-09-22
8.19.5: last 2025-09-25 (active (stack) and active (logsdb))
9.1.0: last 2025-07-02
9.2.0: last 2025-09-25 (active)

The causes of the failures are briefly detailed below.

8.19.5

(stack or logsdb dailies)

• 1password: empty template elastic/integrations#15415
• akamai: empty template
• atlassian_bitbucket (stack):
  • api: template programmer error: "failed to execute template : template: :1:17: executing \"\" at <.last_response.body.pagingInfo.nextPageLink>: map has no entry for key \"nextPageLink\""
  • api-cloud: empty template
• aws (ssi only)
  • cloudtrail: infrastructure? "Failed processing SQS message. Message was deleted. Processing error: non-retryable error: the message is an invalid S3 notification: missing Records field"
  • firewall_logs: infrastructure? "Failed processing SQS message. Message was deleted. Processing error: non-retryable error: the message is an invalid S3 notification: missing Records field"
  • vpcflow: infrastructure? "Failed processing SQS message. Message was deleted. Processing error: non-retryable error: the message is an invalid S3 notification: missing Records field"
  • waf: infrastructure? "Failed processing SQS message. Message was deleted. Processing error: non-retryable error: the message is an invalid S3 notification: missing Records field"
• aws_bedrock: infrastructure? "Failed processing SQS message. Message was deleted. Processing error: non-retryable error: the message is an invalid S3 notification: missing Records field"
• bitwarden:
  • event: template programmer error: "failed to execute template last_update_at: template: :1:23: executing \"\" at <.last_response.body.continuationToken>: map has no entry for key \"continuationToken\""
  • policy: empty template
• canva: infrastructure? "Failed processing SQS message. Message was deleted. Processing error: non-retryable error: the message is an invalid S3 notification: missing Records field"
• cisco_secure_endpoint: empty template
• cisco_umbrella: infrastructure? "Failed processing SQS message. Message was deleted. Processing error: non-retryable error: the message is an invalid S3 notification: missing Records field"
• forgerock: template programmer error: "failed to execute template rate-limit_remaining: template: :1:16: executing \"\" at <.last_response.headers.Get>: map has no entry for key \"headers\""
• github: empty template
• google_cloud_storage: ? "found unsupported content-type: text/plain; charset=utf-8"
• google_scc: empty template elastic/integrations#15415
• google_workspace: empty template
• httpjson: template programmer error: "failed to execute template page: template: :1:16: executing \"\" at <.last_response.body.page>: map has no entry for key \"page\""
• imperva_cloud_waf: infrastructure? "Failed processing SQS message. Message was deleted. Processing error: non-retryable error: the message is an invalid S3 notification: missing Records field"
• lumos: template programmer error: "failed to execute template since: template: :1:13: executing \"\" at <.last_event.created>: map has no entry for key \"created\""
• microsoft_exchange_online_message_trace (logsdb): template programmer error: "failed to execute template $skiptoken: template: :1:69: executing \"\" at <.last_response.terminate_pagination>: map has no entry for key \"terminate_pagination\""
• mimecast: only v1 (httpjson input): intentional template failure: "failed to execute template x-mc-app-id: the template execution failed"
• netskope: infrastructure? "Failed processing SQS message. Message was deleted. Processing error: non-retryable error: the message is an invalid S3 notification: missing Records field"
• okta: empty template
• proofpoint_tap (logsdb): empty template
• sentinel_one_cloud_funnel: infrastructure? "Failed processing SQS message. Message was deleted. Processing error: non-retryable error: the message is an invalid S3 notification: missing Records field"
• slack: empty template
• sophos_central: template programmer error: "failed to execute template from_date: template: :1:24: executing \"\" at <.last_response.body.has_more>: map has no entry for key \"has_more\""
• sublime_security: infrastructure? "Failed processing SQS message. Message was deleted. Processing error: non-retryable error: the message is an invalid S3 notification: missing Records field"
• symantec_endpoint_security: infrastructure? "Failed processing SQS message. Message was deleted. Processing error: non-retryable error: the message is an invalid S3 notification: missing Records field"
• tenable_sc
  • asset: template programmer error: "failed to execute template last_event_ts: template: :1:30: executing \"\" at <.last_response.body.response.endOffset>: map has no entry for key \"endOffset\""
  • plugin: ? "error processing response: server responded with status code 404 Not Found"
  • vulnerability: template programmer error: "failed to execute template last_event_ts: template: :1:30: executing \"\" at <.last_response.body.response.endOffset>: map has no entry for key \"endOffset\""
• ti_eset: template programmer error: "failed to execute template added_after: template: :1:20: executing \"\" at <.last_response.body.more>: map has no entry for key \"more\""
• ti_misp
  • threat: template programmer error: "failed to execute template timestamp: template: :1:13: executing \"\" at <.last_event.Event.timestamp>: map has no entry for key \"Event\""
  • threat_attributes: empty template
• ti_rapid7_threat_command
  • alert: empty template
  • ioc: template programmer error: "failed to execute template offset: template: :1:16: executing \"\" at <.last_response.body.nextOffset>: map has no entry for key \"nextOffset\""
  • vulnerability: template programmer error: "failed to execute template offset: template: :1:16: executing \"\" at <.last_response.body.nextOffset>: map has no entry for key \"nextOffset\""
• tines (stack): empty template
• trellix_edr_cloud: infrastructure? "Failed processing SQS message. Message was deleted. Processing error: non-retryable error: the message is an invalid S3 notification: missing Records field"
• trend_micro_vision_one (stack): mock error? "error processing response: server responded with status code 401 Unauthorized: {\"error\":{\"message\":\"Invalid token, Authorization token invaild for payload\",\"innererror\":{\"service\":\"svp\",\"code\":\"InvalidToken\"},\"code\":\"InvalidCredentials\"}}"
• zerofox (logsdb): template programmer error: "failed to execute template : template: :1:16: executing \"\" at <.last_response.body.next>: map has no entry for key \"next\""

9.2.0

(stack daily only)

Essentially as above, but also including:

• citrix_waf: grok pattern issue "[Provided Grok expressions do not match field value: [fo> 81.2.69.144 10/06/2014:14:03:30 GMT ns1 0-PPE-0 : TCP CONN_TERMINATE 4474 0 : Source 127.0.0.1:80 - Destination 127.0.0.2:39771 - Start Time 10/06/2014:14:02:46 GMT - End Time 10/06/2014:14:03:30 GMT - Total_bytes_send 1 - Total_bytes_recv 1]]
  [1] found error.message in event: [Text '2 Dec 19 00:38:09' could not be parsed at index 0]"
• darktrace
  • ai_analyst_alert: ? "found error.message in event: [Unexpected end-of-input within/between Object entries
    at [Source: (String)"{"summariser":"ScanSummary","acknowledged":false,"pinned":false,"createdAt":1657749437781,"attackPhases":[4],"title":"Port Scanning","id":"eabcdef0-1234-1234-1234-cabcdefghij9","incidentEventUrl":"https://www.example.com/#aiaincidentevent/eabcdef0-1234-1234-1234-cabcdefghij9","children":["eabcdef0-1234-1234-1234-cabcdefghij9"],"category":"suspicious","currentGroup":"eab12345-1234-1234-1234-cabcdef12345","groupCategory":"suspicious","groupScore":6.857722547303857,"groupPreviousGroups":[],"activit"[truncated 1712 chars]; line: 1, column: 2213]]"
  • model_breach_alert: ? no hits
• hpe_aruba_cx: field "log.file.fingerprint" is undefined
• zeronetworks: template programmer error: "failed to execute template _cursor: template: :1:16: executing \"\" at <.last_response.body.scrollCursor>: map has no entry for key \"scrollCursor\""

Common problems in this set

These two classes of failures are appropriate for assignment to Crest.

Some others either need more investigation or appear to be due to infrastructure failure (but need more investigation to confirm).

Empty template

Fix is essentially as in elastic/integrations#15415

• akamai
• bitwarden
• cisco_secure_endpoint
• github
• google_workspace
• okta
• proofpoint_tap
• slack
• ti_misp
• ti_rapid7_threat_command
• tines

Template programmer error

It is not valid to dot index into a missing field in the Go templating system. It is necessary to use something like [[if index .last_response.body "<possibly_absent_field>"]][[.last_response.body.<possibly_absent_field>]][[end]].

• atlassian_bitbucket
• bitwarden
• forgerock
• httpjson
• lumos
• microsoft_exchange_online_message_trace
• sophos_central
• tenable_sc
• ti_eset
• ti_misp
• ti_rapid7_threat_command
• zerofox
• zeronetworks