3.2.0 Feature release: Running IntelMQ bots as Python Library

[sebix]

IEP007: Running IntelMQ bots as Python Library is implemented.

Installation: https://intelmq.readthedocs.io/en/develop/user/installation.html
Upgrade: https://intelmq.readthedocs.io/en/develop/user/upgrade.html

The accompanying 3.2.0 release of intelmq-api switches it's backend from the library hug to fastapi.
Deb-packages of intelmq-api 3.2.0 are delayed for some distributions because of necessary changes in packaging.

Core

• intelmq.lib.utils:
  • resolve_dns: Deprecate dnspython versions pre-2.0.0 and disable search domains (PR#2352)
• Fixed not resetting destination path statistics in the stats cache after restarting bot (Fixes #2331)
• Force flushing statistics if bot will sleep longer than flushing delay (Fixes #2336)
• intelmq.lib.upgrages: Fix a bug in the upgrade function for version 3.1.0 which caused an exception if a generic csv parser instance had no parameter type (PR#2319 by Filip Pokorný).
• intelmq.lib.datatypes: Adds TimeFormat class to be used for the time_format bot parameter (PR#2329 by Filip Pokorný).
• intelmq.lib.exceptions: Fixes a bug in InvalidArgument exception (PR#2329 by Filip Pokorný).
• intelmq.lib.harmonization:
  • Changes signature and names of DateTime conversion functions for consistency, backwards compatible (PR#2329 by Filip Pokorný).
  • Ensure rejecting URLs with leading whitespaces after changes in CPython (fixes #2377)
• intelmq.lib.bot.Bot: Allow setting the parameters via parameter on bot initialization.

Development

• CI: pin the Codespell version to omit troubles caused by its new releases (PR FIX: Pin codespell version #2379).

Bots

Collectors

• intelmq.bots.collector.rt:
  • restrict python-rt to be below version 3.0 due to introduced breaking changes,
  • added support for Subject NOT LIKE queries,
  • added support for multiple values in ticket subject queries.
• intelmq.bots.collectors.rsync: Support for optional private key, relative time parsing for the source path, extra rsync parameters and strict host key checking (PR#2241 by Mateo Durante).

Parsers

• intelmq.bots.parsers.shadowserver._config:
  • Reset detected feedname at shutdown to re-detect the feedname on reloads (PR#2361 by @elsif2, fixes Shadowserver parser may fail to autodetect report type after reload #2360).
• intelmq.bots.parsers.shadowserver._config:
  • Added 'IPv6-Vulnerable-Exchange' alias and 'Accessible-WS-Discovery-Service' report. (PR#2338)
  • Removed unused p0f_genre and p0f_detail from the 'DNS-Open-Resolvers' report. (PR#2338)
  • Added 'Accessible-SIP' report. (PR#2348)
  • Added 'IPv6-Open-HTTP-Proxy' and 'IPv6-Accessible-HTTP-Proxy' aliases. (PR#2348)
  • Removed duplicate mappings from the 'Spam-URL' report. (PR#2348)
• intelmq.bots.parsers.generic.parser_csv: Changes time_format parameter to use new TimeFormat class (PR#2329 by Filip Pokorný).
• intelmq.bots.parsers.html_table.parser: Changes time_format parameter to use new TimeFormat class (PR#2329 by Filip Pokorný).
• intelmq.bots.parsers.turris.parser.py Updated to the latest data format (issue Turris greylist has been updated #2167). (PR#2373 by Filip Pokorný).

Experts

• intelmq.bots.experts.sieve:
  • Allow empty lists in sieve rule files (PR#2341 by Mikk Margus Möll).
• intelmq.bots.experts.cymru_whois:
  • Ignore AS names with unexpected unicode characters (PR#2352, fixes Cymru whois expert: handle strange ASN symbols better #2132)
  • Avoid extraneous search domain-based queries on NXDOMAIN result (PR#2352)
• intelmq.bots.experts.sieve:
  • Added :before and :after keywords (PR#2374)

Outputs

• intelmq.bots.outputs.cif3.output: Added (PR#2244 by Michael Davis).
• intelmq.bots.outputs.sql.output: New parameter fail_on_errors (PR#2362 by Sebastian Wagner).
• intelmq.bots.outputs.smtp_batch.output: Added a bot to gathering the events and sending them by e-mails at a stroke as CSV files (PR#2253 by Edvard Rejthar)

Documentation

• API: update API installation to be aligned with the rewritten API, and clarify some missing steps.

Tests

• New decorator skip_installation and environment variable INTELMQ_TEST_INSTALLATION to skip tests requiring an IntelMQ installation on the test host by default (PR#2370 by Sebastian Wagner, fixes intelmqctl and intelmqdump tests require an installed IntelMQ instance #2369)

Tools

• intelmqsetup:
  • SECURITY: fixed a low-risk bug causing the tool to change owner of / if run with the INTELMQ_PATHS_NO_OPT environment variable set. This affects only the PIP package as the DEB/RPM packages don't contain this tool. (PR#2355 by Kamil Mańkowski, fixes intelmqsetup changes the root directory ownership #2354)
• contrib.eventdb.separate-raws-table.sql: Added the missing commas to complete the sql syntax. (PR#2386, fixes Missing commas in SQL query for separate Events table #2125 by Sebastian Kufner)
• intelmq_psql_initdb:
  • Added parameter -o to set the output file destination. (by Sebastian Kufner)
• intelmqctl:
  • Increased the performance through removing unnecessary reads. (by Sebastian Kufner)

Known Issues

This is short list of the most important known issues. The full list can be retrieved from GitHub.

• intelmq.parsers.html_table may not process invalid URLs in patched Python version due to changes in urllib (Regression on parsing invalid URLs #2382).
• Breaking changes in 'rt' library (Breaking changes in 'rt' 3.0 library #2367).
• Stomp collector failed (Stomp collector failed #2342).
• Type error with SQL output bot's prepare_values returning list instead of tuple (Type error with SQL output bot's prepare_values returning list instead of tuple #2255).
• intelmq_psql_initdb does not work for SQLite (intelmq_psql_initdb does not work for SQLite #2202).
• intelmqsetup: should install a default state file (intelmqsetup: should install a default state file #2175).
• Misp Expert - Crash if misp event already exist (Misp Expert - Crash if misp event already exist #2170).
• Turris greylist has been updated (Turris greylist has been updated #2167).
• Spamhaus CERT parser uses wrong field (Spamhaus CERT parser uses wrong field #2165).
• Custom headers ignored in HTTPCollectorBot (Custom headers ignored in HTTPCollectorBot  #2150).
• Missing commas in SQL query for separate Events table (Missing commas in SQL query for separate Events table #2125).
• intelmqctl log: parsing syslog does not work (intelmqctl log: parsing syslog does not work #2097).
• Bash completion scripts depend on old JSON-based configuration files (Bash completion scripts depend on old JSON-based configuration files #2094).
• Bot configuration examples use JSON instead of YAML (Bot configuration examples use JSON instead of YAML #2066).
• Bots started with IntelMQ-API/Manager stop when the webserver is restarted (Bots started with IntelMQ-API/Manager stop when the webserver is restarted #952).
• Corrupt dump files when interrupted during writing (Corrupt dump files when interrupted during writing #870).

---

This discussion was created from the release 3.2.0 Feature release: Running IntelMQ bots as Python Library.