[syslog_router] Add syslog router integration

.github/CODEOWNERS

@@ -349,6 +349,7 @@
 /packages/synthetics @elastic/obs-ux-infra_services-team
 /packages/synthetics_dashboards @elastic/obs-ux-infra_services-team
 /packages/sysdig @elastic/security-service-integrations
+/packages/syslog_router @elastic/sec-deployment-and-devices
 /packages/sysmon_linux @elastic/sec-linux-platform
 /packages/system @elastic/obs-infraobs-integrations
 /packages/system/changelog.yml @elastic/obs-infraobs-integrations @elastic/sec-linux-platform @elastic/sec-windows-platform

packages/syslog_router/_dev/build/build.yml

@@ -0,0 +1,4 @@
+dependencies:
+ ecs:
+ reference: "git@v8.16.0"
+ import_mappings: true

packages/syslog_router/_dev/build/docs/README.md

@@ -0,0 +1,208 @@
+# Syslog Router Integration
+
+The Syslog Router integration can be used on a stream of syslog events to
+identify which integrations they belong to and forward to the appropriate
+data stream.
+
+## Data streams
+
+Syslog events will be routed to the data stream provided in the pattern
+definition. In the event a match cannot be made, an event will be placed
+into the `log` data stream. See the **Setup** section in this document for
+further explanation on how to configure data streams.
+
+## Requirements
+
+Elasticsearch for storing and searching your data and Kibana for visualizing
+and managing it. We recommend using our hosted Elasticsearch Service on
+Elastic Cloud, or self-manage the Elastic Stack on your own hardware.
+Additionally, to route events to other data streams, the corresponding
+Elastic Integration assets will need to be installed.
+
+## Setup
+
+Install the relevant integration assets in Kibana.
+
+1. In order for the forwarded event to be properly handled, the target integration's assets (data stream, ingest pipeline, index template, etc.) need to be installed. In Kibana, navigate to **Management** > **Integrations** in the sidebar.
+
+2. Find the relevant integration(s) by searching or browsing the catalog. For example, the Cisco ASA integration. 
+
+![Cisco ASA Integration](../img/catalog-cisco-asa.png)
+
+3. Navigate to the **Settings** tab and click **Install Cisco ASA assets**. Confirm by clicking **Install Cisco ASA** in the popup.
+
+![Install Cisco ASA assets](../img/install-assets.png)
+
+## Configuration
+
+### Overview
+
+The integration comes preconfigured with a number of pattern definitions. The
+pattern definitions are used in the order given. Care must be taken to ensure
+the patterns are executed in the correct order. Regular expressions which are
+more relaxed and could potentially match against multiple integrations should be
+run last and stricter patterns should be run first. The next priority should be
+given to integrations which will see the most traffic.
+
+Pattern definitions may be reordered by moving the entire `if/then` block up or
+down in the list. For example, moving **Imperva SecureSphere** above **Cisco ASA**:
+
+**Before:**
+
+```yaml
+- if:
+ and:
+ - not.has_fields: _conf.dataset
+ - regexp.message: "%ASA-"
+ then:
+ - add_fields:
+ target: ''
+ fields:
+ _conf.dataset: "cisco_asa.log"
+ _conf.tz_offset: "UTC"
+ _temp_.internal_zones: ['trust']
+ _temp_.external_zones: ['untrust']
+- if:
+ and:
+ - not.has_fields: _conf.dataset
+ - regexp.message: "CEF:0\\|Imperva Inc.\\|SecureSphere"
+ then:
+ - add_fields:
+ target: ''
+ fields:
+ _conf.dataset: "imperva.securesphere"
+ - decode_cef:
+ field: message
+```
+
+**After:**
+
+```yaml
+- if:
+ and:
+ - not.has_fields: _conf.dataset
+ - regexp.message: "CEF:0\\|Imperva Inc.\\|SecureSphere"
+ then:
+ - add_fields:
+ target: ''
+ fields:
+ _conf.dataset: "imperva.securesphere"
+ - decode_cef:
+ field: message
+- if:
+ and:
+ - not.has_fields: _conf.dataset
+ - regexp.message: "%ASA-"
+ then:
+ - add_fields:
+ target: ''
+ fields:
+ _conf.dataset: "cisco_asa.log"
+ _conf.tz_offset: "UTC"
+ _temp_.internal_zones: ['trust']
+ _temp_.external_zones: ['untrust']
+```
+
+Individual pattern definitions may be disabled by removing the definition
+entirely or by inserting comment characters (`#`) in front of the appropriate lines:
+
+```yaml
+# - if:
+# and:
+# - not.has_fields: _conf.dataset
+# - regexp.message: "%ASA-"
+# then:
+# - add_fields:
+# target: ''
+# fields:
+# _conf.dataset: "cisco_asa.log"
+# _conf.tz_offset: "UTC"
+# _temp_.internal_zones: ['trust']
+# _temp_.external_zones: ['untrust']
+```
+
+### Adding New Patterns
+
+Example configuration:
+
+```yaml
+- if:
+ and:
+ - not.has_fields: _conf.dataset
+ - regexp.message: "CEF:0\\|Imperva Inc.\\|SecureSphere"
+ then:
+ - add_fields:
+ target: ''
+ fields:
+ _conf.dataset: "imperva.securesphere"
+ - decode_cef:
+ field: message
+```
+
+At its core, the Syslog Router integration utilizes the [built-in conditionals and processors](https://www.elastic.co/guide/en/beats/filebeat/current/defining-processors.html)
+provided within Beats. While there are certain requirements that need to be
+maintained, additional conditions and processors may be added, if required.
+
+The top level of each configuration contains an `if`/`else` condition. In the
+`if` statement, an `and` combines two conditions. The first ensures that another
+match has not already occurred, while the second condition is a `regex`, or regular
+expression, which performs the actual match. If the regular expression
+matches the `message` field, then the processors in the `then` statement of the
+configuration will run.
+
+If multiple patterns are required, they may be combined with an `or` condition:
+
+```yaml
+- if:
+ and:
+ - not.has_fields: _conf.dataset
+ - or:
+ - regexp.message: <PATTERN_1>
+ - regexp.message: <PATTERN_2>
+```
+
+In the `then` statement, a list of processors can be given. At minimum, an 
+`add_fields` processor needs to be added with the following fields:
+
+**Required fields:**
+
+- `_conf.dataset`: The dataset (`integration.data_stream`) to forward to. This field is used by the routing rules in the integration to route documents to the correct pipeline.
+
+Additional processors, such as `decode_cef` or `syslog`, may be provided if
+additional processing is required.
+
+## Compatibility
+
+Out of the box, the Syslog Router integration supports matching events from a
+number of integrations. Assets from these integrations must still be installed
+for events to be properly indexed (see **Setup** above).
+
+**DISCLAIMER**: Due to subtle differences in how devices can emit syslog events,
+the patterns provided by default with the Syslog Router integration may not work
+in all cases. Some integrations may not be listed here, even though they support
+syslog events. In these cases, patterns would either be too complex or could
+overlap with patterns from other integrations, resulting in negative impacts on
+performance or accuracy in matching events to integrations. Custom patterns will
+need to be created for these cases.
+
+- Arista NG Firewall
+- Check Point
+- Cisco ASA
+- Cisco FTD
+- Cisco ISE
+- Cisco Secure Email Gateway
+- Citrix WAF (CEF format only)
+- Fortinet FortiEDR
+- Fortinet FortiGate
+- Fortinet FortiMail
+- Fortinet FortiManager
+- Fortinet FortiProxy
+- Imperva SecureSphere (CEF format only)
+- Iptables
+- Juniper SRX
+- Palo Alto Next-Gen Firewall
+- QNAP NAS
+- Snort
+- Sonicwall Firewall
+- Sophos XG
+- Stormshield

packages/syslog_router/_dev/deploy/docker/docker-compose.yml

@@ -0,0 +1,23 @@
+version: '2.3'
+services:
+ syslog-filestream:
+ image: alpine
+ volumes:
+ - ./sample_logs:/sample_logs:ro
+ - ${SERVICE_LOGS_DIR}:/var/log
+ command: /bin/sh -c "cp /sample_logs/* /var/log/"
+ syslog-udp:
+ image: docker.elastic.co/observability/stream:v0.15.0
+ volumes:
+ - ./sample_logs:/sample_logs:ro
+ command: log --start-signal=SIGHUP --delay=5s --addr elastic-agent:9515 -p=udp /sample_logs/test.log
+ syslog-tcp:
+ image: docker.elastic.co/observability/stream:v0.15.0
+ volumes:
+ - ./sample_logs:/sample_logs:ro
+ command: log --start-signal=SIGHUP --delay=5s --addr elastic-agent:9515 -p=tcp /sample_logs/test.log
+ syslog-tls:
+ image: docker.elastic.co/observability/stream:v0.15.0
+ volumes:
+ - ./sample_logs:/sample_logs:ro
+ command: log --start-signal=SIGHUP --delay=5s --addr elastic-agent:9516 -p=tls --insecure /sample_logs/test.log

packages/syslog_router/_dev/deploy/docker/sample_logs/test.log

@@ -0,0 +1 @@
+<13>Oct 11 22:14:15 test-host testd[1024]: Custom log

packages/syslog_router/changelog.yml

@@ -0,0 +1,6 @@
+# newer versions go on top
+- version: "0.1.0"
+ changes:
+ - description: Initial draft of the package
+ type: enhancement
+ link: https://github.com/elastic/integrations/pull/11727

packages/syslog_router/data_stream/log/_dev/test/pipeline/test-common-config.yml

@@ -0,0 +1,3 @@
+fields:
+ tags:
+ - preserve_original_event

packages/syslog_router/data_stream/log/_dev/test/pipeline/test-syslog-events.json

@@ -0,0 +1,15 @@
+{
+ "events": [
+ {
+ "@timestamp": "2024-04-23T09:16:40.645Z",
+ "message": "Oct 10 2018 12:34:56 localhost CiscoASA[999]: %ASA-4-106023: Deny tcp src outside:192.168.19.254/80 dst inside:172.31.98.44/8277 by access-group \"inbound\" [0x0, 0x0]",
+ "_conf": {
+ "dataset": "cisco_asa.log"
+ }
+ },
+ {
+ "@timestamp": "2024-04-23T09:16:40.645Z",
+ "message": "Custom log"
+ }
+ ]
+}

packages/syslog_router/data_stream/log/_dev/test/pipeline/test-syslog-events.json-expected.json

@@ -0,0 +1,32 @@
+{
+ "expected": [
+ {
+ "@timestamp": "2024-04-23T09:16:40.645Z",
+ "_conf": {
+ "dataset": "cisco_asa.log"
+ },
+ "data_stream": {
+ "dataset": "cisco_asa.log",
+ "namespace": "default",
+ "type": "logs"
+ },
+ "ecs": {
+ "version": "8.16.0"
+ },
+ "message": "Oct 10 2018 12:34:56 localhost CiscoASA[999]: %ASA-4-106023: Deny tcp src outside:192.168.19.254/80 dst inside:172.31.98.44/8277 by access-group \"inbound\" [0x0, 0x0]",
+ "tags": [
+ "preserve_original_event"
+ ]
+ },
+ {
+ "@timestamp": "2024-04-23T09:16:40.645Z",
+ "ecs": {
+ "version": "8.16.0"
+ },
+ "message": "Custom log",
+ "tags": [
+ "preserve_original_event"
+ ]
+ }
+ ]
+}

packages/syslog_router/data_stream/log/_dev/test/system/test-filestream-config.yml

@@ -0,0 +1,10 @@
+service: syslog-filestream
+service_notify_signal: SIGHUP
+input: filestream
+data_stream:
+ vars:
+ paths:
+ - "{{SERVICE_LOGS_DIR}}/*.log"
+ preserve_original_event: true
+assert:
+ hit_count: 1

packages/syslog_router/data_stream/log/_dev/test/system/test-tcp-config.yml

@@ -0,0 +1,10 @@
+service: syslog-tcp
+service_notify_signal: SIGHUP
+input: tcp
+data_stream:
+ vars:
+ listen_address: 0.0.0.0
+ listen_port: 9515
+ preserve_original_event: true
+assert:
+ hit_count: 1

packages/syslog_router/data_stream/log/_dev/test/system/test-tls-config.yml

@@ -0,0 +1,62 @@
+service: syslog-tls
+service_notify_signal: SIGHUP
+input: tcp
+data_stream:
+ vars:
+ listen_address: 0.0.0.0
+ listen_port: 9516
+ preserve_original_event: true
+ ssl: |
+ key: |
+ -----BEGIN PRIVATE KEY-----
+ MIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQDhCLvLsQAHufsN
+ U+u1x/CequAUphfXZqLhDo2Eo/holfBS0+ey4bnzPL6lS9NFL5JkLQA2gYESqsXU
+ /Ru8E76Az1egzMwT3TVAPLVU8NbrxBqeNiQa2m9wC37HQy4qC9OxL28LUoKtFjxS
+ cD1sa0oikXCJN1a3BSoAf9iiZ/dxz4WVfrNhrzq2JFXjravY84n5ujkZOg45Pg70
+ 4vHOeg0rBbIoSNfjDUVZWjwC95K1BMN3msOTL9juv/EDa6BujqCxl+G1nY7JPFDL
+ SHWis65p+1AAa5xieYDb47vyJ0SSR7lEURTXZOkkM6k5JWfgkATEmGzRxPkOloIT
+ Xg9ag1OlAgMBAAECggEAEHfPJmzhj68wjB0kFr13AmWG2Hv/Kqg8KzQhbx+AwkaW
+ u7j+L70NGpvLZ9VQtLNyhxoz9cksZO1SZO/Q48aeHlcOFppmJN3/U6AdtQWa9M35
+ FLLpmX16wjxVHsfvzOvopgLOoYl8PqZt66qDFDgVyMnT7na6RdJ+7GJuvBPXq+Bc
+ vgThvAZitHSAOhnBFYmTMlBi6AzOMMsaFlgE3Xf9v3M0pAKItPRKMhXlC3MyvA/v
+ jgbra4Ib+0ryohggHheHB3bn3Jgv7iFKoW9OQSePVxacJ+kfr9H+No5g495URzqR
+ mx/96WCiv3rAh3ct8Sk/C4/3zMC8fUueDJIVjhgw0QKBgQD8NufLINNkIpBrLoCS
+ 972oFEjZB2u6EusQ7X9raROqpaw26ZSu+zSHeIKCGQ93M3aRb3FpdGeOxgZ095MV
+ 8a+nlh4stOvHj2Mm5YhTBDUavTC7o9aVR3Od5eTXUpHnaJpNI/uyIcKupeK1UJnV
+ UlBLeIwo/vJ1gsVrKMMAJkuKbwKBgQDkaWRRd0w2gUIbCTGf203BqXft0VdIiOW7
+ +gnkeaNHAf09XljzxMcQzrB8kG63aKVGbJffphEfzxtiJ+HRQVH+7QpKRhU/GHmu
+ +6OKkxTcxJm5zhoRFxcSi2wG4PWmUGJvc7ss1OJGcaOUxwocCepO7N/jfdDz9Uke
+ KnA+YWOdKwKBgQDteZkYlojT0QOgF8HyH5gQyUCqMKWLJ0LzxltiPCbLV4Dml1pq
+ w5Z7M8nWS1hXiTpLx93GSFc1hFkSCwYP9GfK6Lryp0sVtHnMZvTMDbseuSJImwRx
+ vDwtYQfugg1lEQWwOoBEAiu3m/PxernNtNprpU57T0nlwUK3GkM5QdWAuwKBgQCZ
+ ZF3GiANapzupxGbbH//8Cr9LqsafI7CEqMpz8WxBh4h16iJ6sq+tDeFgBe8UpOY5
+ gTwNKg1d+0w8guQYD3HtbWr3rlEeamVtqfiOW3ArQqyqJ0tCJuuLvK3zgKf35Qv2
+ JRaSaPT8sdxVUcXsRoxgLJu+vwPQke1koMN4YRbwuQKBgQDJiZ/WSeqa5oIqkXbn
+ hjm7RXKaf2oE1U/bNjdSFtdEP7T4vUvvr7Hq2f/jiBLtCE7w16PJjKx9iIq2+jMl
+ qIY43Sk9bdi5FxtYTHda0hwrbH274P+QVcVs5PXCT0TGktOleHGBlXaaPrxl9iCh
+ 8tmmxZZYa5aQxEO/lxB9xQKaiQ==
+ -----END PRIVATE KEY-----
+ certificate: |
+ -----BEGIN CERTIFICATE-----
+ MIIDazCCAlOgAwIBAgIUW5TDu1tJMY2Oa7PsL+BQSmeWqz0wDQYJKoZIhvcNAQEL
+ BQAwRTELMAkGA1UEBhMCVVMxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM
+ GEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDAeFw0yMTEwMDEwNTAwMjNaFw0yMTEw
+ MDIwNTAwMjNaMEUxCzAJBgNVBAYTAlVTMRMwEQYDVQQIDApTb21lLVN0YXRlMSEw
+ HwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQwggEiMA0GCSqGSIb3DQEB
+ AQUAA4IBDwAwggEKAoIBAQDhCLvLsQAHufsNU+u1x/CequAUphfXZqLhDo2Eo/ho
+ lfBS0+ey4bnzPL6lS9NFL5JkLQA2gYESqsXU/Ru8E76Az1egzMwT3TVAPLVU8Nbr
+ xBqeNiQa2m9wC37HQy4qC9OxL28LUoKtFjxScD1sa0oikXCJN1a3BSoAf9iiZ/dx
+ z4WVfrNhrzq2JFXjravY84n5ujkZOg45Pg704vHOeg0rBbIoSNfjDUVZWjwC95K1
+ BMN3msOTL9juv/EDa6BujqCxl+G1nY7JPFDLSHWis65p+1AAa5xieYDb47vyJ0SS
+ R7lEURTXZOkkM6k5JWfgkATEmGzRxPkOloITXg9ag1OlAgMBAAGjUzBRMB0GA1Ud
+ DgQWBBRYUSKDHBBE9Q6fTeTqogicCxcXwDAfBgNVHSMEGDAWgBRYUSKDHBBE9Q6f
+ TeTqogicCxcXwDAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBc
+ T8B+GpvPy9NQ700LsywRPY0L9IJCKiu6j3TP1tqqSPjAC/cg9ac+bFXuWOu7V+KJ
+ s09Q/pItq9SLX6UvnfRzTxu5lCBwwGX9cL131mTIu5SmFo7Eks+sorbiIarWDMoC
+ e+9An3GFpagW+YhOt4BdIM5lTqoeodzganDBsOUZI9aDAj2Yo5h2O7r6Wd12cb6T
+ mz8vMfB2eG8BxU20ZMfkdERWjiyXHOSBQqeqfkV8d9370gMu5RcJNcIgnbmTRdho
+ X3HJFiimZVaNjXATqmC/y2A1KXvJdamPLy3mGXkW2cFLoPCdK2OZFUHqiuc1bigA
+ qEf55SihFqErRMeURPPF
+ -----END CERTIFICATE-----
+assert:
+ hit_count: 1