3.5.0 Feature release

[sebix]

This release introduces new fields requested by many community members, implementing IEP008: IntelMQ Data Format: Constituency field and IEP009: IntelMQ Data Format: Describe product and mark vulnerabilities.

Requirements

Python >=3.9 is now required, which is available on all platforms supported by IntelMQ.

Data Format

To save new fields from IntelMQ Data Format in existing PostgreSQL instances, the following schema
update is necessary:

CREATE TYPE severity_enum AS ENUM ( 'critical', 'high', 'medium', 'low', 'info', 'undefined' ); ALTER TABLE events ADD "product.full_name" text; ALTER TABLE events ADD "product.name" text; ALTER TABLE events ADD "product.vendor" text; ALTER TABLE events ADD "product.version" text; ALTER TABLE events ADD "product.vulnerabilities" text; ALTER TABLE events ADD severity varchar(10); ALTER TABLE events ADD "constituency" text; UPDATE events SET severity = (extra ->> 'severity')::severity_enum;

Optionally remove the severity field from the extra fields in existing entries:

UPDATE events SET extra = extra - 'severity';

Postgres databases

To switch to the more efficient data type jsonb instead of json, use the following SQL statement. Data is preserved. JSONB also has more query and data manipulation features than plain JSON.

ALTER TABLE events ALTER COLUMN "extra" SET DATA TYPE jsonb;

Configuration

• New parameter stop_retry_limit (PR#2598 by Lukas Heindl).

Core

• Drop support for Python 3.8 (fixes tests: pymssql not installable on 3.8 #2616, PR#2617 by Sebastian Wagner).
• intelmq.lib.splitreports: Handle bot parameter chunk_size values empty string, due to missing parameter typing checks (PR#2604 by Sebastian Wagner).
• intelmq.lib.mixins.sql Add Support for MySQL (PR#2625 by Karl-Johan Karlsson).
• New parameter stop_retry_limit to gracefully handle stopping bots which take longer to shutdown (PR#2598 by Lukas Heindl, fixes intelmqctl stop bots are still running #2595).
• intelmq.lib.datatypes: Remove unneeded Dict39 alias (PR#2639 by Nakul Rajpal, fixes Remove Dict39 compatibility shim #2635)
• intelmq.lib.mixins.http: Only set HTTP header 'Authorization' if username or password are set and are not both empty string as they are by default in the Manager (fixes HTTP: If username and password are empty strings (default), Authorization is still used #2590, PR#2634 by Sebastian Wagner).
• intelmq.lib.message.Message.from_dict: Do not modify the dict parameter by adding the __type field and raise an error when type is not determinable (PR#2545 by Sebastian Wagner).
• intelmq.lib.mixins.cache.CacheMixin was extended to support temporary storing messages in a cache queue
  (PR#2509 by Kamil Mankowski).

Data Format

• Implementing IEP009 introducing fields to
  identify products and vulnerabilities: product.full_name, product.name, product.vendor,
  product.version, product.vulnerabilities. To store in existing PostgreSQL instances, a schema update is necessary, see first section.
• added severity field to help with triaging received events (PR#2575 by Kamil Mańkowski).
  To allow saving the field in PostgreSQL database in existing installations, the following schema update shown in the first section is necessary
• Implementing IEP008 introducing the constituency field for easier identification in
  multi-constituency setups. (PR#2573 by Kamil Mańkowski)
  To use in current PostgreSQL installations, a schema update is
  necessary, see first section

Bots

Collectors

• intelmq.bots.collectors.mail.collector_mail_attach: Decrypt GPG attachments (PR#2623 by Edvard Rejthar).
• intelmq.bots.collectors.mail.collector_mail_attach: Allow empty attachments (PR#2647 by Edvard Rejthar).
• intelmq.bots.collectors.shodan.collector_alert: Added a new collector to query the Shodan Alert API (PR#2618 by Sebastian Wagner and Malawi CERT).
• Remove intelmq.bots.collectors.blueliv as it uses an unmaintained library, does not work any more and breaks other CI tests (fixes Blueliv dependendcy breaks full-test in CI #2593, PR#2632 by Sebastian Wagner).

Parsers

• intelmq.bots.parsers.cymru.parser_cap_program: Add mapping for TOR and ipv6-icmp protocol (PR#2621 by Mikk Margus Möll).
• Remove intelmq.bots.parsers.blueliv as it is obsolete with the removed collector (PR#2632 by Sebastian Wagner).
• intelmq.bots.parser.json.parser:
  • Support data containing lists of JSON Events (PR#2545 by Tim de Boer).
  • Add default classification.type with value undetermined if input data has now classification itself (PR#2545 by Sebastian Wagner).

Experts

• intelmq.bots.experts.asn_lookup.expert:
  • Print URLs to stdout only in verbose mode (PR#2591 by Sebastian Wagner).
  • Check for database file existence and writability (fixes ASN Lookup expert: Missing sane default value for database path #2566).
  • Use database path matching to installation type (PR#2606 by Sebastian Wagner).
• intelmq.bots.experts.fake.expert:
  • Use database path matching to installation type (PR#2606 by Sebastian Wagner).
  • Add new mode random_single_value (PR#2601 by Sebastian Wagner).
• intelmq.bots.experts.sieve.expert: Test for textX dependency in self-check (PR#2605 by Sebastian Wagner).
• intelmq.bots.experts.trusted_introducer_lookup.expert: Change to new TI database URL (fixes Trusted Introducer lookup bot: data URL is 404 #2620, PR#2633 by Sebastian Wagner).

Outputs

• intelmq.bots.outputs.smtp_batch.output:
  • Add new parameter additional_grouping_keys for an enhanced email batching feature.
  • Add new parameter templating for additional template variables.
  • Add new parameter allowed_fieldnames for csv field specification.
  • Add new parameter fieldnames_translation for naming csv headers (PR#2610 by Lukas Heindl, fixes [smtp_batch] Additional field for grouping #2586).
• intelmq.bots.outputs.sql.output: Add Support for MySQL (PR#2625 by Karl-Johan Karlsson).

Documentation

• Fix and refresh links to mailing lists (PR#2609 by Kamil Mańkowski)
• Aggregate Bot: Add illustration graphics (PR#2612 by Sebastian Wagner).

Packaging

• Replace /opt/intelmq example paths in bots with variable VAR_STATE_PATH for correct paths in LSB-path setups like with packages (PR#2587 by Sebastian Wagner).
• New deb-package intelmq-contrib with all contrib/ scripts and documentation (PR#2614 by Sebastian Wagner).
• New deb-package intelmq-autostart containing systemd services and timers to start all enabled IntelMQ bots at boot and periodically (PR#2638 by Sebastian Wagner).

Tests

• intelmq.tests.lib.test_pipeline.TestAmqp.test_acknowledge: Skip on all Python versions when running on CI (PR#2602 by Sebastian Wagner).
• .github/workflows/codespell.yml, debian-package.yml, regexploit.yml: Upgrade to ubuntu-latest runners (PR#2602 by Sebastian Wagner).
• intelmq.test.test_conf: With changed behaviour in ruamel.yaml on line wrapping since version 0.18.13, only test the parsabilty of runtime.yaml (PR#2619 by Sebastian Wagner).
• intelmq.test.BotTestCase.test_static_bot_check_method: Remove debugging stub raising for all non-empty checks (PR#2622 by Sebastian Wagner).

Tools

• intelmq.bin.intelmq_psql_initdb: Use JSONB type by default, Postgres supports it since version 9 (PR#2597 by Sebastian Wagner).
• intelmq.bin.rewrite_config_files: Removed obsolete JSON configuration file rewriter (PR#2613 by Sebastian Wagner).
• intelmq/lib/bot_debugger.py: Fix overwriting the runtime logging level by command line parameter (PR#2603 by Sebastian Wagner, fixes ctl: interactive run ignores custom log level #2563).
• intelmq.bin.intelmqctl: Fix bot log level filtering (PR#2607 by Sebastian Wagner, fixes "intelmqctl log" level selection does not work #2596).

Contrib

• Bash Completion: Adapt to YAML-style runtime configuration (PR#2642 by Sebastian Wagner, fixes Bash completion scripts depend on old JSON-based configuration files #2094).
• Remove prettyprint script, use jq instead (PR#2551 by Sebastian Wagner).

Known issues

This is short list of the most important known issues. The full list can be retrieved from GitHub.

• stomp.py 8.2.0+ breaks the version check in stomp bots (stomp.py 8.2.0+ breaks the version check in stomp bots #2600).
• Traceback when calling intelmqdump without write access to the log file (Traceback when calling intelmqdump without write access to the log file #2529).
• pyyaml PendingDeprecationWarning: you should no longer specify 'unsafe' -> test failure (pyyaml PendingDeprecationWarning: you should no longer specify 'unsafe' -> test failure #2489).
• intelmq.parsers.html_table may not process invalid URLs in patched Python version due to changes in urllib (Regression on parsing invalid URLs #2382).
• Breaking changes in 'rt' 3.0 library (Breaking changes in 'rt' 3.0 library #2367).
• Type error with SQL output bot's prepare_values returning list instead of tuple (Type error with SQL output bot's prepare_values returning list instead of tuple #2255).
• intelmq_psql_initdb does not work for SQLite (intelmq_psql_initdb does not work for SQLite #2202).
• intelmqsetup: should install a default state file (intelmqsetup: should install a default state file #2175).
• Misp Expert - Crash if misp event already exist (Misp Expert - Crash if misp event already exist #2170).
• Spamhaus CERT parser uses wrong field (Spamhaus CERT parser uses wrong field #2165).
• Custom headers ignored in HTTPCollectorBot (Custom headers ignored in HTTPCollectorBot  #2150).
• intelmqctl log: parsing syslog does not work (intelmqctl log: parsing syslog does not work #2097).
• Bots started with IntelMQ-API/Manager stop when the webserver is restarted (Bots started with IntelMQ-API/Manager stop when the webserver is restarted #952).
• Corrupt dump files when interrupted during writing (Corrupt dump files when interrupted during writing #870).

---

This discussion was created from the release 3.5.0 Feature release.