3.5.0 Feature release

This release introduces new fields requested by many community members, implementing IEP008: IntelMQ Data Format: Constituency field and IEP009: IntelMQ Data Format: Describe product and mark vulnerabilities.

Requirements

Python >=3.9 is now required, which is available on all platforms supported by IntelMQ.

Data Format

To save new fields from IntelMQ Data Format in existing PostgreSQL instances, the following schema
update is necessary:

CREATE TYPE severity_enum AS ENUM (
    'critical',
    'high',
    'medium',
    'low',
    'info',
    'undefined'
);
ALTER TABLE events ADD "product.full_name" text;
ALTER TABLE events ADD "product.name" text;
ALTER TABLE events ADD "product.vendor" text;
ALTER TABLE events ADD "product.version" text;
ALTER TABLE events ADD "product.vulnerabilities" text;
ALTER TABLE events ADD severity varchar(10);
ALTER TABLE events ADD "constituency" text;
UPDATE events SET severity = (extra ->> 'severity')::severity_enum;

Optionally remove the severity field from the extra fields in existing entries:

UPDATE events SET extra = extra - 'severity';

Postgres databases

To switch to the more efficient data type jsonb instead of json, use the following SQL statement. Data is preserved. JSONB also has more query and data manipulation features than plain JSON.

ALTER TABLE events
   ALTER COLUMN "extra" SET DATA TYPE jsonb;

Configuration

• New parameter stop_retry_limit (PR#2598 by Lukas Heindl).

Core

• Drop support for Python 3.8 (fixes #2616, PR#2617 by Sebastian Wagner).
• intelmq.lib.splitreports: Handle bot parameter chunk_size values empty string, due to missing parameter typing checks (PR#2604 by Sebastian Wagner).
• intelmq.lib.mixins.sql Add Support for MySQL (PR#2625 by Karl-Johan Karlsson).
• New parameter stop_retry_limit to gracefully handle stopping bots which take longer to shutdown (PR#2598 by Lukas Heindl, fixes #2595).
• intelmq.lib.datatypes: Remove unneeded Dict39 alias (PR#2639 by Nakul Rajpal, fixes #2635)
• intelmq.lib.mixins.http: Only set HTTP header 'Authorization' if username or password are set and are not both empty string as they are by default in the Manager (fixes #2590, PR#2634 by Sebastian Wagner).
• intelmq.lib.message.Message.from_dict: Do not modify the dict parameter by adding the __type field and raise an error when type is not determinable (PR#2545 by Sebastian Wagner).
• intelmq.lib.mixins.cache.CacheMixin was extended to support temporary storing messages in a cache queue
  (PR#2509 by Kamil Mankowski).

Data Format

• Implementing IEP009 introducing fields to
  identify products and vulnerabilities: product.full_name, product.name, product.vendor,
  product.version, product.vulnerabilities. To store in existing PostgreSQL instances, a schema update is necessary, see first section.
• added severity field to help with triaging received events (PR#2575 by Kamil Mańkowski).
  To allow saving the field in PostgreSQL database in existing installations, the following schema update shown in the first section is necessary
• Implementing IEP008 introducing the constituency field for easier identification in
  multi-constituency setups. (PR#2573 by Kamil Mańkowski)
  To use in current PostgreSQL installations, a schema update is
  necessary, see first section

Bots

Collectors

• intelmq.bots.collectors.mail.collector_mail_attach: Decrypt GPG attachments (PR#2623 by Edvard Rejthar).
• intelmq.bots.collectors.mail.collector_mail_attach: Allow empty attachments (PR#2647 by Edvard Rejthar).
• intelmq.bots.collectors.shodan.collector_alert: Added a new collector to query the Shodan Alert API (PR#2618 by Sebastian Wagner and Malawi CERT).
• Remove intelmq.bots.collectors.blueliv as it uses an unmaintained library, does not work any more and breaks other CI tests (fixes #2593, PR#2632 by Sebastian Wagner).

Parsers

• intelmq.bots.parsers.cymru.parser_cap_program: Add mapping for TOR and ipv6-icmp protocol (PR#2621 by Mikk Margus Möll).
• Remove intelmq.bots.parsers.blueliv as it is obsolete with the removed collector (PR#2632 by Sebastian Wagner).
• intelmq.bots.parser.json.parser:
  • Support data containing lists of JSON Events (PR#2545 by Tim de Boer).
  • Add default classification.type with value undetermined if input data has now classification itself (PR#2545 by Sebastian Wagner).

Experts

• intelmq.bots.experts.asn_lookup.expert:
  • Print URLs to stdout only in verbose mode (PR#2591 by Sebastian Wagner).
  • Check for database file existence and writability (fixes #2566).
  • Use database path matching to installation type (PR#2606 by Sebastian Wagner).
• intelmq.bots.experts.fake.expert:
  • Use database path matching to installation type (PR#2606 by Sebastian Wagner).
  • Add new mode random_single_value (PR#2601 by Sebastian Wagner).
• intelmq.bots.experts.sieve.expert: Test for textX dependency in self-check (PR#2605 by Sebastian Wagner).
• intelmq.bots.experts.trusted_introducer_lookup.expert: Change to new TI database URL (fixes #2620, PR#2633 by Sebastian Wagner).

Outputs

• intelmq.bots.outputs.smtp_batch.output:
  • Add new parameter additional_grouping_keys for an enhanced email batching feature.
  • Add new parameter templating for additional template variables.
  • Add new parameter allowed_fieldnames for csv field specification.
  • Add new parameter fieldnames_translation for naming csv headers (PR#2610 by Lukas Heindl, fixes #2586).
• intelmq.bots.outputs.sql.output: Add Support for MySQL (PR#2625 by Karl-Johan Karlsson).

Documentation

• Fix and refresh links to mailing lists (PR#2609 by Kamil Mańkowski)
• Aggregate Bot: Add illustration graphics (PR#2612 by Sebastian Wagner).

Packaging

• Replace /opt/intelmq example paths in bots with variable VAR_STATE_PATH for correct paths in LSB-path setups like with packages (PR#2587 by Sebastian Wagner).
• New deb-package intelmq-contrib with all contrib/ scripts and documentation (PR#2614 by Sebastian Wagner).
• New deb-package intelmq-autostart containing systemd services and timers to start all enabled IntelMQ bots at boot and periodically (PR#2638 by Sebastian Wagner).

Tests

• intelmq.tests.lib.test_pipeline.TestAmqp.test_acknowledge: Skip on all Python versions when running on CI (PR#2602 by Sebastian Wagner).
• .github/workflows/codespell.yml, debian-package.yml, regexploit.yml: Upgrade to ubuntu-latest runners (PR#2602 by Sebastian Wagner).
• intelmq.test.test_conf: With changed behaviour in ruamel.yaml on line wrapping since version 0.18.13, only test the parsabilty of runtime.yaml (PR#2619 by Sebastian Wagner).
• intelmq.test.BotTestCase.test_static_bot_check_method: Remove debugging stub raising for all non-empty checks (PR#2622 by Sebastian Wagner).

Tools

• intelmq.bin.intelmq_psql_initdb: Use JSONB type by default, Postgres supports it since version 9 (PR#2597 by Sebastian Wagner).
• intelmq.bin.rewrite_config_files: Removed obsolete JSON configuration file rewriter (PR#2613 by Sebastian Wagner).
• intelmq/lib/bot_debugger.py: Fix overwriting the runtime logging level by command line parameter (PR#2603 by Sebastian Wagner, fixes #2563).
• intelmq.bin.intelmqctl: Fix bot log level filtering (PR#2607 by Sebastian Wagner, fixes #2596).

Contrib

• Bash Completion: Adapt to YAML-style runtime configuration (PR#2642 by Sebastian Wagner, fixes #2094).
• Remove prettyprint script, use jq instead (PR#2551 by Sebastian Wagner).

Known issues

This is short list of the most important known issues. The full list can be retrieved from GitHub.

• stomp.py 8.2.0+ breaks the version check in stomp bots (#2600).
• Traceback when calling intelmqdump without write access to the log file (#2529).
• pyyaml PendingDeprecationWarning: you should no longer specify 'unsafe' -> test failure (#2489).
• intelmq.parsers.html_table may not process invalid URLs in patched Python version due to changes in urllib (#2382).
• Breaking changes in 'rt' 3.0 library (#2367).
• Type error with SQL output bot's prepare_values returning list instead of tuple (#2255).
• intelmq_psql_initdb does not work for SQLite (#2202).
• intelmqsetup: should install a default state file (#2175).
• Misp Expert - Crash if misp event already exist (#2170).
• Spamhaus CERT parser uses wrong field (#2165).
• Custom headers ignored in HTTPCollectorBot (#2150).
• intelmqctl log: parsing syslog does not work (#2097).
• Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952).
• Corrupt dump files when interrupted during writing (#870).

3.4.0 Feature release

Documentation: https://docs.intelmq.org/latest/

NEWS

Most important changes potentially requiring administration attention

Requirements

Python 3.8 or newer is required.

Bots

CIF 3 API Output deprecation

The CIF 3 API Output bot is not compatible with Python version greater or equal to 3.12 and will be removed in the future due to lack of maintenance.
See https://lists.cert.at/pipermail/intelmq-users/2024-December/000474.html for more information.

Twitter Collector removal

As the bot does not work anymore and uses an unmaintained library, it is removed from IntelMQ.
Please remove if from your setup.

intelmqctl check and intelmqctl upgrade-config command warns if you have the bot in use.

Twitter Parser renaming

The Twitter parser is renamed to IoC Extractor Parser (intelmq.bots.parsers.ioc_extractor).
intelmqctl upgrade-config will automatically adapt the configuration.

The previous module name is left as a stub to load the IoC Extractor parser for backwards-compatibility.

Packaging

Packages are now also available for Ubuntu 24.04.
To upgrade an Ubuntu 22.04 installation to 24.04 please refer to the Ubuntu documentation: https://documentation.ubuntu.com/server/how-to/software/upgrade-your-release/index.html

Full changelog

Configuration

Core

• AMQP: Fix maintaining pipeline connection when during interrupted connections (PR#2533 by Kamil Mankowski).
• Python 3.8 or newer is required (PR#2541 by Sebastian Wagner).
• intelmq.lib.utils.list_all_bots/intelmqctl check: Fix check for bot executable in $PATH by using the bot name instead of the import path (fixes #2559, PR#2564 by Sebastian Wagner).

Bots

Collectors

• intelmq.bots.collectors.shadowserver.collector_reports_api.py:
  • Fixed behaviour if parameter types value is empty string, behave the same way as not set, not like no type.
• intelmq.bots.collectors.misp: Use PyMISP class instead of deprecated ExpandedPyMISP (PR#2532 by Radek Vyhnal)
• intelmq.bots.collectors.http.collector_http: Log the downloaded size in bytes to ease troubleshooting (PR#2554 by Sebastian Wagner).
• intelmq.bots.collectors.mail.collector_mail_url:
  • Log the downloaded size in bytes to ease troubleshooting (PR#2554 by Sebastian Wagner).
  • Fix import for Timeout exception preventing another exception (fixes #2555, PR#2556 by Sebastian Wagner).
• Remove intelmq.bots.collectors.twitter as it uses an unmaintained library and does not work any more (fixes #2346, #2441, PR#2568 by Sebastian Wagner).

Parsers

• intelmq.bots.parsers.shadowserver._config:
  • fix error message formatting if schema file is absent (PR#2528 by Sebastian Wagner).
• intelmq.bots.parsers.shadowserver.parser:
  • Fix to avoid schema download if not configured #2530.
• intelmq.bots.parsers.misp.parser: Replace deprecated datetime function utcfromtimestamp for Ubuntu 24.04 compatibility (PR#2577 by Sebastian Wagner, fixes #2576, #2571).
• intelmq.bots.parsers.cleanmx.parser: Replace deprecated datetime function utcfromtimestamp for Ubuntu 24.04 compatibility (PR#2577 by Sebastian Wagner, fixes #2576, #2571).
• Renamed intelmq.bots.parsers.twitter to intelmq.bots.parser.ioc_extractor (PR#2568 by Sebastian Wagner).
  • Added intelmq.bots.parsers.twitter as a stub to load the IoC Extractor parser.

Experts

• intelmq.bots.experts.securitytxt:
  • Added new bot (PR#2538 by Frank Westers and Sebastian Wagner).
• intelmq.bots.experts.misp: Use PyMISP class instead of deprecated ExpandedPyMISP (PR#2532 by Radek Vyhnal).
• intelmq.bots.experts.fake.expert: New expert to fake data (PR#2567 by Sebastian Wagner).

Outputs

• intelmq.bots.outputs.cif3.output:
  • The requirement can only be installed on Python version < 3.12.
  • Add a check on the Python version and exit if incompatible.
  • Add a deprecation warning (PR#2544 by Sebastian Wagner).
• intelmq.bots.outputs.sql.output:
  • Treat an empty string fields parameter as unset parameter, fixing a crash in default configuration (PR#2548 by Sebastian Wagner, fixes #2548).

Documentation

• docs/admin/installation/linux-packages: Add [signed-by=] options, add wget command as alternative to curl (PR#2547 by Sebastian Wagner).
• Add documentation on the Redis pipeline (databases, configuration), fix generic pipeline documentation and add missing information on parameters, add unlinked intelmqctl docs to the index and TOC (PR#2560 by Sebastian Wagner).
• Remove empty page tutorials/intelmq-manager (PR#2562 by Sebastian Wagner).

Packaging

• Packages for Ubuntu 24.04 (by Sebastian Wagner, fixes #2571).

Tests

• Install build dependencies for pymssql on Python 3.8 as there are no wheels available for this Python version (PR#2542 by Sebastian Wagner).
• Install psql explicitly for workflow support on other platforms such as act (PR#2542 by Sebastian Wagner).
• Create intelmq user & group if running privileged to allow dropping privileges (PR#2542 by Sebastian Wagner).
• intelmq.tests.lib.test_pipeline.TestAmqp.test_acknowledge: Also skip on Python 3.11 and 3.12 besides on 3.8 when running on CI (PR#2542 by Sebastian Wagner).
• Full pytest workflow: Version-independent install of postgres client, for Ubuntu 24.04 (default on GitHub now) test environment compatibility (PR#2557 by Sebastian Wagner).
• Debian package build workflow: Use artifact upload v4 instead of v3 (PR#2565 by Sebastian Wagner).

Known issues

This is short list of the most important known issues. The full list can be retrieved from GitHub.

• intelmqctl: interactive run ignores custom log level (#2563).
• intelmq.parsers.html_table may not process invalid URLs in patched Python version due to changes in urllib (#2382).
• Breaking changes in 'rt' 3.0 library (#2367).
• Type error with SQL output bot's prepare_values returning list instead of tuple (#2255).
• intelmq_psql_initdb does not work for SQLite (#2202).
• intelmqsetup: should install a default state file (#2175).
• Misp Expert - Crash if misp event already exist (#2170).
• Spamhaus CERT parser uses wrong field (#2165).
• Custom headers ignored in HTTPCollectorBot (#2150).
• intelmqctl log: parsing syslog does not work (#2097).
• Bash completion scripts depend on old JSON-based configuration files (#2094).
• Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952).
• Corrupt dump files when interrupted during writing (#870).

3.3.1 Bugfix release

Documentation: https://docs.intelmq.org/latest/

Core

• intelmq.lib.utils.drop_privileges: When IntelMQ is called as root and dropping the privileges to user intelmq, also set the non-primary groups associated with the intelmq user. Makes the behaviour of running intelmqctl as root closer to the behaviour of sudo -u intelmq ... (PR#2507 by Mikk Margus Möll).
• intelmq.lib.utils.unzip: Ignore directories themselves when extracting data to prevent the extraction of empty data for a directory entries (PR#2512 by Kamil Mankowski).

Bots

Collectors

• intelmq.bots.collectors.shadowserver.collector_reports_api.py:
  • Added support for the types parameter to be either a string or a list (PR#2495 by elsif2).
  • Refactored to utilize the type field returned by the API to match the requested types instead of a sub-string match on the filename.
  • Fixed timezone issue for collecting reports (PR#2506 by elsif2).
  • Fixed behaviour if parameter reports value is empty string, behave the same way as not set, not like no report (PR#2523 by Sebastian Wagner).
• intelmq.bots.collectors.shodan.collector_stream (PR#2492 by Mikk Margus Möll):
  • Add alert parameter to Shodan stream collector to allow fetching streams by configured alert ID
• intelmq.bots.collectors.mail._lib: Remove deprecated parameter attach_unzip from default parameters (PR#2511 by Sebastian Wagner).

Parsers

• intelmq.bots.parsers.shadowserver._config:
  • Fetch schema before first run (PR#2482 by elsif2, fixes #2480).
• intelmq.bots.parsers.dataplane.parser: Use | as field delimiter, fix parsing of AS names including | (PR#2488 by DigitalTrustCenter).
• all parsers: add copy_collector_provided_fields parameter allowing copying additional fields from the report, e.g. extra.file_name.
  (PR#2513 by Kamil Mankowski).

Experts

• intelmq.bots.experts.sieve.expert:
  • For :contains, =~ and !~, convert the value to string before matching avoiding an exception. If the value is a dict, convert the value to JSON (PR#2500 by Sebastian Wagner).
  • Add support for variables in Sieve scripts (PR#2514 by Mikk Margus Möll, fixes #2486).
• intelmq.bots.experts.filter.expert:
  • Treat value false for parameter filter_regex as false (PR#2499 by Sebastian Wagner).

Outputs

• intelmq.bots.outputs.misp.output_feed: Handle failures if saved current event wasn't saved or is incorrect (PR by Kamil Mankowski).
• intelmq.bots.outputs.smtp_batch.output: Documentation on multiple recipients added (PR#2501 by Edvard Rejthar).

Documentation

• Bots: Clarify some section of Mail collectors and the Generic CSV Parser (PR#2510 by Sebastian Wagner).

Known Issues

This is short list of the most important known issues. The full list can be retrieved from GitHub.

• intelmq.parsers.html_table may not process invalid URLs in patched Python version due to changes in urllib (#2382).
• Breaking changes in 'rt' 3.0 library (#2367).
• Type error with SQL output bot's prepare_values returning list instead of tuple (#2255).
• intelmq_psql_initdb does not work for SQLite (#2202).
• intelmqsetup: should install a default state file (#2175).
• Misp Expert - Crash if misp event already exist (#2170).
• Spamhaus CERT parser uses wrong field (#2165).
• Custom headers ignored in HTTPCollectorBot (#2150).
• intelmqctl log: parsing syslog does not work (#2097).
• Bash completion scripts depend on old JSON-based configuration files (#2094).
• Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952).
• Corrupt dump files when interrupted during writing (#870).

3.3.0

3.3.0 (2024-03-01)

Configuration

• Add new optional configuration parameters for intelmq.bots.collectors.stomp.collector
  and intelmq.bots.outputs.stomp.output (PR#2408 by Jan Kaliszewski):
  • auth_by_ssl_client_certificate (Boolean, default: true; if false then
    ssl_client_certificate and ssl_client_certificate_key will be ignored);
  • username (STOMP authentication login, default: "guest"; to be used only
    if auth_by_ssl_client_certificate is false);
  • password (STOMP authentication passcode, default: "guest"; to be used only
    if auth_by_ssl_client_certificate is false).
• Add the possibility to set the ssl_ca_certificate configuration parameter for
  intelmq.bots.collectors.stomp.collector and/or intelmq.bots.outputs.stomp.output
  to an empty string - which means that the SSL machinery used for STOMP communication
  will attempt to load the system’s default CA certificates (PR#2414 by Jan Kaliszewski).

Core

• intelmq.lib.message: For invalid message keys, add a hint on the failure to the exception: not allowed by configuration or not matching regular expression (PR#2398 by Sebastian Wagner).
• intelmq.lib.exceptions.InvalidKey: Add optional parameter additional_text (PR#2398 by Sebastian Wagner).
• Change the way we discover bots to allow easy extending based on the entry point name. (PR#2413 by Kamil Mankowski)
• intelmq.lib.mixins: Add a new class, StompMixin (defined in a new submodule: stomp),
  which provides certain common STOMP-bot-specific operations, factored out from
  intelmq.bots.collectors.stomp.collector and intelmq.bots.outputs.stomp.output
  (PR#2408 and PR#2414 by Jan Kaliszewski).
• intelmq.lib.upgrades: Replace deprecated instances of url2fqdn experts by the new url expert in runtime configuration (PR#2432 by Sebastian Wagner).
• intelmq.lib.bot: Ensure closing log files on reloading (PR#2435 by Kamil Mankowski).
• AMQP Pipeline: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).
• Only load the config once when starting intelmqctl (which makes IntelMQ API calls take less time) (PR#2444 by DigitalTrustCenter).

Development

• Makefile: Add codespell and test commands (PR#2425 by Sebastian Wagner).

Data Format

Bots

Collectors

• intelmq.bots.collectors.stomp.collector (PR#2408 and PR#2414 by Jan Kaliszewski):
  • Drop support for versions of stomp.py older than 4.1.12.
  • Update the code to support new versions of stomp.py, including the latest (8.1.0);
    fixes #2342.
  • Add support for authentication based on STOMP login and passcode, introducing three
    new configuration parameters (see above: Configuration).
  • Add support for loading the system’s default CA certificates, as an alternative to
    specifying the CA certificate(s) file path explicitly (see above: Configuration).
  • Fix (by carefully targeted monkey patching) certain security problems caused by
    SSL-related weaknesses that some versions of stomp.py suffer from.
  • Fix the reconnection behavior: do not attempt to reconnect after shutdown. Also,
    never attempt to reconnect if the version of stomp.py is older than 4.1.21 (it
    did not work properly anyway).
  • Add coercion of the port config parameter to int.
  • Add implementation of the check hook (verifying, in particular, accessibility
    of necessary file(s)).
  • Remove undocumented and unused attributes of StompCollectorBot instances:
    ssl_ca_cert, ssl_cl_cert, ssl_cl_cert_key.
    • Minor fixes/improvements and some refactoring (see also above: Core...).
• intelmq.bots.collectors.amqp: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).
• intelmq.bots.collectors.shadowserver.collector_reports_api:
  • The 'json' option is no longer supported as the 'csv' option provides better performance (PR#2372 by elsif2).
• intelmq.bots.collectors.alienvault_otx.collector (PR#2449 by qux-bbb):
  • Fix modified_pulses_only is always False.

Parsers

• intelmq.bots.parsers.netlab_360.parser: Removed as the feed is discontinued. (#2442 by Filip Pokorný)
• intelmq.bots.parsers.webinspektor.parser: Removed as the feed is discontinued. (#2442 by Filip Pokorný)
• intelmq.bots.parsers.sucuri.parser: Removed as the feed is discontinued. (#2442 by Filip Pokorný)
• intelmq.bots.parsers.shadowserver._config:
  • Switch to dynamic configuration to decouple report schema changes from IntelMQ releases by regularly downloading them from the Shadowserver server (PR#2372 by elsif2).
• intelmq.bots.parsers.cymru: Save current line. (PR by Kamil Mankowski)

Experts

• intelmq.bots.experts.jinja (PR#2417 by Mikk Margus Möll):
  • Add optional socket_perms and socket_group parameters to change
    file permissions on socket file, if it is in use.
• intelmq.bots.experts.ripe (PR#2461 by Mikk Margus Möll):
  • Handle "No abuse contact found for" messages for non-ASN resources

Outputs

• intelmq.bots.outputs.stomp.output (PR#2408 and PR#2414 by Jan Kaliszewski):
  • Drop support for versions of stomp.py older than 4.1.12.
  • Update the code to support new versions of stomp.py, including the latest (8.1.0).
  • Add support for authentication based on STOMP login and passcode, introducing three
    new configuration parameters (see above: Configuration).
  • Add support for loading the system’s default CA certificates, as an alternative to
    specifying the CA certificate(s) file path explicitly (see above: Configuration).
  • Fix (by carefully targeted monkey patching) certain security problems caused by
    SSL-related weaknesses that some versions of stomp.py suffer from.
  • Fix AttributeError caused by attempts to get unset attributes of StompOutputBot
    (ssl_ca_cert et consortes).
  • Add coercion of the port config parameter to int.
  • Add implementation of the check hook (verifying, in particular, accessibility
    of necessary file(s)).
  • Add stomp.py version check (raise MissingDependencyError if not >=4.1.12).
  • Minor fixes/improvements and some refactoring (see also above: Core...).
• intelmq.bots.outputs.stomp.output (PR#2423 by Kamil Mankowski):
  • Try to reconnect on NotConnectedException.
• intelmq.bots.outputs.smtp_batch.output (PR #2439 by Edvard Rejthar):
  • Fix ability to send with the default bcc
• intelmq.bots.outputs.amqp: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).

Documentation

• Add a readthedocs configuration file to fix the build fail (PR#2403 by Sebastian Wagner).
• Add a guide of developing extensions packages (PR#2413 by Kamil Mankowski)
• Update/fix/improve the stuff related to the STOMP bots and integration with the n6's
  Stream API (PR#2408 and PR#2414 by Jan Kaliszewski).
• Complete documentation overhaul. Change to markdown format. Uses the mkdocs-material (PR#2419 by Filip Pokorný).
• Adds warning banner if not browsing the latest version of the docs (PR#2445 by Filip Pokorný).
• Fix logo path in index.md when building the docs (PR#2445 by Filip Pokorný).

Packaging

• Add pendulum to suggested packages, as it is required for the sieve bot (PR#2424 by Sebastian Wagner).
• debian/control: in Suggests field, replace python3-stomp.py (>= 4.1.9) with
  python3-stomp (>= 4.1.12), i.e., fix the package name by removing the .py
  suffix and bump the minimum version to 4.1.12 (PR#2414 by Jan Kaliszewski).

Tests

Tools

• intelmq_psql_initdb:
  • got support for providing custom harmonization file, generating view for storing raw fields separately, and adding IF NOT EXISTS/OR REPLACE clauses (PR#2404 by Kamil Mankowski).
  • got support for generating JSONB fields for PostgreSQL schema (PR#2436 by Kamil Mankowski).

Contrib

Known issues

3.2.1 Important Bugfixes

Installation: https://intelmq.readthedocs.io/en/develop/user/installation.html
Upgrade: https://intelmq.readthedocs.io/en/develop/user/upgrade.html

All Bots

Fixes an issue which prevented bots from stopping gracefully after reloading.
As logrotate reloads all bots regularly, this bug affects most IntelMQ installations.

Reverse DNS Expert

Until IntelMQ version 3.2.0, the bot incorrectly cached and re-used results for /24 networks instead of single IP addresses.
If the bot retrieved the PTR for 192.0.43.7, it was cached for 192.0.43.0/24 and used for all IP addresses in this range, for example for 192.0.43.8.
IntelMQ version 3.2.1 fixes this issue.

The bugfix will correctly increase the cache sizes and decrease the performance, as less (incorrect) data is re-used.

3.2.0 Feature release: Running IntelMQ bots as Python Library

IEP007: Running IntelMQ bots as Python Library is implemented.

Installation: https://intelmq.readthedocs.io/en/develop/user/installation.html
Upgrade: https://intelmq.readthedocs.io/en/develop/user/upgrade.html

The accompanying 3.2.0 release of intelmq-api switches it's backend from the library hug to fastapi.
Deb-packages of intelmq-api 3.2.0 are delayed for some distributions because of necessary changes in packaging.

Core

• intelmq.lib.utils:
  • resolve_dns: Deprecate dnspython versions pre-2.0.0 and disable search domains (PR#2352)
• Fixed not resetting destination path statistics in the stats cache after restarting bot (Fixes #2331)
• Force flushing statistics if bot will sleep longer than flushing delay (Fixes #2336)
• intelmq.lib.upgrages: Fix a bug in the upgrade function for version 3.1.0 which caused an exception if a generic csv parser instance had no parameter type (PR#2319 by Filip Pokorný).
• intelmq.lib.datatypes: Adds TimeFormat class to be used for the time_format bot parameter (PR#2329 by Filip Pokorný).
• intelmq.lib.exceptions: Fixes a bug in InvalidArgument exception (PR#2329 by Filip Pokorný).
• intelmq.lib.harmonization:
  • Changes signature and names of DateTime conversion functions for consistency, backwards compatible (PR#2329 by Filip Pokorný).
  • Ensure rejecting URLs with leading whitespaces after changes in CPython (fixes #2377)
• intelmq.lib.bot.Bot: Allow setting the parameters via parameter on bot initialization.

Development

• CI: pin the Codespell version to omit troubles caused by its new releases (PR #2379).

Bots

Collectors

• intelmq.bots.collector.rt:
  • restrict python-rt to be below version 3.0 due to introduced breaking changes,
  • added support for Subject NOT LIKE queries,
  • added support for multiple values in ticket subject queries.
• intelmq.bots.collectors.rsync: Support for optional private key, relative time parsing for the source path, extra rsync parameters and strict host key checking (PR#2241 by Mateo Durante).

Parsers

• intelmq.bots.parsers.shadowserver._config:
  • Reset detected feedname at shutdown to re-detect the feedname on reloads (PR#2361 by @elsif2, fixes #2360).
• intelmq.bots.parsers.shadowserver._config:
  • Added 'IPv6-Vulnerable-Exchange' alias and 'Accessible-WS-Discovery-Service' report. (PR#2338)
  • Removed unused p0f_genre and p0f_detail from the 'DNS-Open-Resolvers' report. (PR#2338)
  • Added 'Accessible-SIP' report. (PR#2348)
  • Added 'IPv6-Open-HTTP-Proxy' and 'IPv6-Accessible-HTTP-Proxy' aliases. (PR#2348)
  • Removed duplicate mappings from the 'Spam-URL' report. (PR#2348)
• intelmq.bots.parsers.generic.parser_csv: Changes time_format parameter to use new TimeFormat class (PR#2329 by Filip Pokorný).
• intelmq.bots.parsers.html_table.parser: Changes time_format parameter to use new TimeFormat class (PR#2329 by Filip Pokorný).
• intelmq.bots.parsers.turris.parser.py Updated to the latest data format (issue #2167). (PR#2373 by Filip Pokorný).

Experts

• intelmq.bots.experts.sieve:
  • Allow empty lists in sieve rule files (PR#2341 by Mikk Margus Möll).
• intelmq.bots.experts.cymru_whois:
  • Ignore AS names with unexpected unicode characters (PR#2352, fixes #2132)
  • Avoid extraneous search domain-based queries on NXDOMAIN result (PR#2352)
• intelmq.bots.experts.sieve:
  • Added :before and :after keywords (PR#2374)

Outputs

• intelmq.bots.outputs.cif3.output: Added (PR#2244 by Michael Davis).
• intelmq.bots.outputs.sql.output: New parameter fail_on_errors (PR#2362 by Sebastian Wagner).
• intelmq.bots.outputs.smtp_batch.output: Added a bot to gathering the events and sending them by e-mails at a stroke as CSV files (PR#2253 by Edvard Rejthar)

Documentation

• API: update API installation to be aligned with the rewritten API, and clarify some missing steps.

Tests

• New decorator skip_installation and environment variable INTELMQ_TEST_INSTALLATION to skip tests requiring an IntelMQ installation on the test host by default (PR#2370 by Sebastian Wagner, fixes #2369)

Tools

• intelmqsetup:
  • SECURITY: fixed a low-risk bug causing the tool to change owner of / if run with the INTELMQ_PATHS_NO_OPT environment variable set. This affects only the PIP package as the DEB/RPM packages don't contain this tool. (PR#2355 by Kamil Mańkowski, fixes #2354)
• contrib.eventdb.separate-raws-table.sql: Added the missing commas to complete the sql syntax. (PR#2386, fixes #2125 by Sebastian Kufner)
• intelmq_psql_initdb:
  • Added parameter -o to set the output file destination. (by Sebastian Kufner)
• intelmqctl:
  • Increased the performance through removing unnecessary reads. (by Sebastian Kufner)

Known Issues

This is short list of the most important known issues. The full list can be retrieved from GitHub.

• intelmq.parsers.html_table may not process invalid URLs in patched Python version due to changes in urllib (#2382).
• Breaking changes in 'rt' library (#2367).
• Stomp collector failed (#2342).
• Type error with SQL output bot's prepare_values returning list instead of tuple (#2255).
• intelmq_psql_initdb does not work for SQLite (#2202).
• intelmqsetup: should install a default state file (#2175).
• Misp Expert - Crash if misp event already exist (#2170).
• Turris greylist has been updated (#2167).
• Spamhaus CERT parser uses wrong field (#2165).
• Custom headers ignored in HTTPCollectorBot (#2150).
• intelmqctl log: parsing syslog does not work (#2097).
• Bash completion scripts depend on old JSON-based configuration files (#2094).
• Bot configuration examples use JSON instead of YAML (#2066).
• Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952).
• Corrupt dump files when interrupted during writing (#870).

3.1.0 Feature Release

Installation documentation:
https://intelmq.readthedocs.io/en/maintenance/user/installation.html
Upgrade documentation:
https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html

• Upgraded syntax to Python 3.6 (mostly Format-Strings) using pyuprade (PR#2136 by Sebastian Wagner).

Core

• intelmq.lib.upgrades:
  • Refactor upgrade functions global configuration handling removing the old-style defaults configuration (PR#2058 by Sebastian Wagner).
  • Pass version history as parameter to upgrade functions (PR#2058 by Sebastian Wagner).
• intelmq.lib.message:
  • Fix and pre-compile the regular expression for harmonization key names and also check keys in the extra. namespace (PR#2059 by Sebastian Wagner, fixes #1807).
• intelmq.lib.bot.SQLBot was replaced by an SQLMixin in intelmq.lib.mixins.SQLMixin. The Generic DB Lookup Expert bot and the SQLOutput bot were updated accordingly.
  • Added support for MSSQL (PR#2171 by Karl-Johan Karlsson).
  • Added optional reconnect delay parameter (PR#2171 by Karl-Johan Karlsson).
• Added an ExpertBot class - it should be used by all expert bots as a parent class
• Introduced a module for IntelMQ related datatypes intelmq.lib.datatypes which for now only contains an Enum listing the four bot types
• Added a bottype attribute to CollectorBot, ParserBot, ExpertBot, OutputBot
• Introduces a module for IntelMQ processmanagers. The processmanagers were up until now part of the intelmqct script.
  They now reside in intelmq.lib.processmanager which also contains an interface definition the processmanager implementations must adhere to.
  Both the processmanagers and the intelmqctl script were cleaned up a bit.
  The LogLevel and ReturnType Enums were added to intelmq.lib.datatypes.
• intelmq.lib.bot:
  • Enhance behaviour if an unconfigured bot is started (PR#2054 by Sebastian Wagner).
  • Fix line recovery and message dumping of the ParserBot (PR#2192 by Sebastian Wagner).
    • Previously the dumped message was always the last message of a report if the report contained multiple lines leading to data-loss.
  • Fix crashing at start in multithreaded bots (PR#2236 by DigitalTrustCenter).
  • Added default_fields parameter to ParserBot (PR#2293 by Filip Pokorný)
• intelmq.lib.pipeline:
  • Changed BRPOPLPUSH to BLMOVE, because BRPOPLPUSH has been marked as deprecated by redis in favor of BLMOVE (PR#2149 and PR#2240 by Sebastian Waldbauer and Sebastian Wagner, fixes #1827, #2233).
• intelmq.lib.utils:
  • Added wrapper resolve_dns for querying DNS, with the support for recommended methods from dnspython package in versions 1 and 2.
  • Moved line filtering inside RewindableFileHandle for easier handling and limiting number of temporary objects.
• intelmq.lib.harmonization:
  • Fixed DateTime handling of naive time strings (previously assumed local timezone, now assumes UTC) (PR#2279 by Filip Pokorný, fixes #2278)
  • Removes tzone argument from DateTime.from_timestamp and DateTime.from_epoch_millis
  • DateTime.from_timstamp now also allows string argument
• Removes pytz global dependency
• Removed support for Python 3.6, including removing conditional dependencies and updating syntax to use features from newest versions. (fixes #2272)

Development

• Removed Python 3.6 from CI.
• Enabled tests against Python 3.11.

Bots

• Set the parent class of all bots to the correct bot class

Collectors

• intelmq.bots.collectors.mail._lib:
  • Add support for unverified SSL/STARTTLS connections (PR#2055 by Sebastian Wagner).
  • Fix exception handling for aborted IMAP connections (PR#2187 by Sebastian Wagner).
• intelmq.bots.collectors.blueliv: Fix Blueliv collector requirements (PR#2161 by Gethvi).
• intelmq.bots.collectors.github_api._collector_github_api: Added personal access token support (PR#2145 by Sebastian Waldbauer, fixes #1549).
• intelmq.bots.collectors.file.collector_file: Added file lock support, no more race conditions (PR#2147 by Sebastian Waldbauer, fixes #2128)
• intelmq.bots.collectors.shadowserver.collector_reports_api.py: Added file_format option to download reports in CSV format for better performance (PR#2246 by elsif2)

Parsers

• intelmq.bots.parsers.alienvault.parser_otx: Save CVE data in extra.cve instead of extra.CVE due to the field name restriction on lower-case characters (PR#2059 by Sebastian Wagner).

• intelmq.bots.parsers.anubisnetworks.parser: Changed field name format from extra.communication.http.x_forwarded_for_#1 to extra.communication.http.x_forwarded_for_1 due to the field name restriction on alphanumeric characters (PR#2059 by Sebastian Wagner).

• intelmq.bots.parsers.dataplane.parser:

  • Add support for additional feeds (PR#2102 by Mikk Margus Möll).
    • DNS Recursion Desired
    • DNS Recursion Desired ANY
    • DNS Version
    • Protocol 41
    • SMTP Greet
    • SMTP Data
    • Telnet Login
    • VNC/RFB Login
  • Fix event object creation (PR#2298 by DigitalTrustCenter).

• Removed intelmq.bots.parsers.malc0de: this bot was marked as deprecated and removed from feed due to offline status (PR#2184 by Tamas Gutsohn, fixes #2178).

• intelmq.bots.parsers.microsoft.parser_ctip:

  • New parameter overwrite (PR#2112 by Sebastian Wagner, fixes #2022).
  • Fix handling of field Payload.domain if it contains the same IP address as Payload.serverIp (PR#2144 by Mikk Margus Möll and Sebastian Wagner).
  • Handle Payload field with non-base64-encoded JSON content and numbered dictionaries (PR#2193 by Sebastian Wagner)

• intelmq.bots.parsers.shodan.parser (PR#2117 by Mikk Margus Möll):

  • Instead of keeping track of extra.ftp.<something>.parameters, FTP parameters are collected together into extra.ftp.features as a list of said features, reducing field count.
  • Shodan field rsync.modules is collected.
  • Conversion functions can raise NoValueException with a string argument to signify that the conversion would not succeed, such as in the case of a single IP address being given in hostnames, which would then be passed into source.reverse_dns and fail to validate as a FQDN.
  • Variable _common_keys is moved out of the class.
  • _dict_dict_to_obj_list is introduced, for converting a string-to-dict mapping into a list of dicts with the previous key as an attribute of the dict; this can be useful for preventing issues where, when feeding the data into aggregating tools, you'd end up with many more fields than necessary, e.g vulns.CVE-2010-0001.cvss, CVE-2010-0002.cvss etc.
  • _get_first to get the first item from a list, with NoValueException raised on empty lists.
  • _get_first_hostname to handle the first valid FQDN from a list of hostnames for hostnames in the Shodan banner, if there is one, and gives NoValueException otherwise.
  • ssl.cert.serial and ssl.dhparams.generator, which may return both integers and strings, are converted to strings.
  • Changes to method apply_mapping, such as reducing needless loop iterations, removing a big try-except, and adding the NoValueException handling described above.
  • Stops falsy values (False, 0) besides None from being filtered out.

• intelmq.bots.parsers.shadowserver._config:

  • Added support for Accessible AMQP, Device Identification Report (IPv4 and IPv6) (PR#2134 by Mateo Durante).
  • Added file name mapping for SSL-POODLE-Vulnerable-Servers IPv6 (file name scan6_ssl_poodle) (PR#2134 by Mateo Durante).
  • Added Malware-URL, Sandbox-Connection, Sandbox-DNS, Accessible-AMQP, Open-AnonymouIs-MQTT, Accessible-QUIC, Accessible-SSH, SYNful-Knock, and Special (PR#2227 by elsif2)
  • Removed legacy reports Amplification-DDoS-Victim, CAIDA-IP-Spoofer, Darknet, Drone, Drone-Brute-Force, IPv6-Sinkhole-HTTP-Drone, Microsoft-Sinkhole, and Sinkhole-HTTP-Drone (PR#2227 by elsif2).
  • Users storing events in a database should be aware that field names and types have been updated (PR#2227 by elsif2).
  • Corrected "Accessible-AMQP" message_length type (int) and added "STUN" support (PR#2235 by elsif2).
  • Added amplification factor to UDP scan reports (PR#2238 by elsif2).
  • Added version and build_date to "Vulnerable-HTTP" report (PR#2238 by elsif2).
  • The following field types have been standardized across all Shadowserver reports (PR#2246 by elsif2):
    destination.fqdn (validate_fqdn)
    destination.url (convert_http_host_and_url)
    extra.browser_trusted (convert_bool)
    extra.duration (convert_int)
    extra.end_time (convert_date_utc)
    extra.freak_vulnerable (convert_bool)
    extra.ok (convert_bool)
    extra.password (validate_to_none)
    extra.ssl_poodle (convert_bool)
    extra.status (convert_int)
    extra.uptime (convert_int)
    extra.version (convert_to_none)
    source.network (validate_network)
  • The following report field names have changed to better represent their values:
    scan_rsync:extra.password renamed to extra.has_password
    scan_elasticsearch:status renamed to http_code
  • Added Accessible-HTTP-proxy and Open-HTTP-proxy (PR#2246 by elsif2).
  • Added http_agent to the Honeypot-DDoS report and added the DDoS-Participant report (PR#2303 by elsif2)
  • Added Accessible-SLP, IPv6 Accesssible-SLP, IPv6-DNS-Open-Resolvers, and IPv6-Open-LDAP-TCP reports (PR#2311 by elsif2)
  • Standardized response_length to response_size in Accessible-ICS and Open-MSSQL (PR#2311 by elsif2)

• intelmq.bots.parsers.cymru.parser_cap_program: The parser mapped the hostname into source.fqdn which is not allowed by the IntelMQ Data Format. Added a check (PR#2215 by Sebastian Waldbauer, fixes #2169)

• intelmq.bots.parsers.generic.parser_csv:

  • Use RewindableFileHandle to u...

3.1.0 Release candidate

What's Changed

• BUG: utils/load_configuration: fix loading JSON with tabs in #2010
• CTIP parser: Handle TLP value "unknown" in #2009
• FIX: s/self.parameters/self/ in Shodan collector by @monoidic in #2020
• BUG: provide filename when reading a non-archived file from mail by @kapalex in #2021
• Add new bot: cut string by delimeter, max length by @mariuskarotkis in #1967
• Core fixes by @monoidic in #2027
• DOC: Fix typo by @schacht-certat in #2029
• Make update script less verbose by default by @schacht-certat in #2033
• Remove malwardomains parser by @schacht-certat in #2026
• Add new bot: domain valid by @mariuskarotkis in #1966
• Support Shadowserver Vulnerable SMTP server feed, fix #1984 by @monoidic in #2037
• Shadowserver feed updates by @schacht-certat in #2036
• Remove postgresql output bot by @schacht-certat in #2045
• DOC: remove mention of intelmqctl list bot json output by @schacht-certat in #2030
• Remove workarounds for XMPP bots by @schacht-certat in #2048
• Use the YAML unsafe loader instead of the safe loader by @schacht-certat in #2041
• Make the harmonization attribute of the bot private by @schacht-certat in #2049
• ENH: imap collector: support unverified connections in #2055
• ENH: utils: list_bots: strip descriptions in #2056
• lib/harm: minor change to allowed classification.types list in #2057
• ENH: small refactor of upgrade functions in #2058
• Add Honeypot HTTP Scan feed to Shadowserver parser by @monoidic in #2047
• IDF: Fix and enforce field name restrictions in #2059
• JinjaExpert - modify message fields using jinja2 by @schacht-certat in #2040
• Implement a clear class hierarchy by @schacht-certat in #2042
• DOCS: Docker-compose commands updated by @waldbauer-certat in #2035
• Refactoring intelmqctl by @schacht-certat in #2067
• ENH: lib/bot: fix behavior for unconfigured bots in #2054
• BUG: fix v300_pipeline_file_removal by @schacht-certat in #2083
• Adapt ansible workflow to ansible tests by @schacht-certat in #2081
• BUG: use correct private parameter variable by @schacht-certat in #2087
• DOCS: fix some sphinx warnings by @schacht-certat in #2065
• ENH: make json output indented to make it better readable by humans by @schacht-certat in #2079
• ENH: use the upstream ansible tasks playbook by @schacht-certat in #2089
• Return an INFO string if log file does not yet exist by @schacht-certat in #2100
• MAINT: update the url for the vagrant+ansible test repository by @schacht-certat in #2061
• FIX: statistics_host returns if no host value is given by @waldbauer-certat in #2104
• DOCS: Updated docker installation instructions by @waldbauer-certat in #2069
• BUG: fix logrotate ownership options in #2111
• Add support for new feeds to Dataplane parser by @monoidic in #2102
• Add RPZ file output bot by @mariuskarotkis in #1962
• Add new bot: bro output file by @mariuskarotkis in #1963
• DOC: restructure installation documentation in #2113
• ENH: ctip parser: added parameter overwrite in #2112
• Add new bot: cut string from string by @mariuskarotkis in #1965
• ENH: intelmqsetup: revise installation of manager in #2114
• DOC: enhance misp integration document in #2115
• ENH: Shodan parser: handle invalid hostnames, move _common_keys out of class, adjust apply_mapping by @monoidic in #2117
• Add JSON parser function to Jinja2 templates by @creideiki in #2120
• DOC: feeds.yaml: fix HIBP nginx config by @sebix in #2126
• FIX: Fixes update-database script on the last few days of a month. by @gethvi in #2121
• DOC: add changelog for PR#2121 by @sebix in #2129
• Docs: Remove dead IRC channel by @sebix in #2130
• Added new shadowserver files to _config.py by @mateodurante in #2134
• DOC: Update changelog and docs for shadowserver PR#2134 by @sebix in #2135
• doc: feeds whishlist: add misp warning lists by @sebix in #2141
• docs: feeds whishlist: add avast ioc repo by @sebix in #2139
• ENH: upgrade Python syntax to 3.6+ by @sebix in #2136
• MS CTIP Azure feed invalid hostname filtering by @monoidic in #2144
• TST: also test on python 3.10, switch to pytest by @sebix in #2142
• FIX: CodeQL exponential backtracking on strings by @waldbauer-certat in #2148
• FIX: Added vscode to gitignore & automated docker build with ACT by @waldbauer-certat in #2153
• FIX: threshold expert: configure timeout using Redis cache mixin by @creideiki in #2155
• FIX: Fixes Blueliv collector requirements by @gethvi in #2161
• Add fix Blueliv collector requirements to CHANGELOG.md by @gutsohnCERT in #2173
• FIX: Moved SQLBot to SQLMixin namespace in generic-db-lookup by @waldbauer-certat in #2146
• DOC: installation: remove EOL Fedora versions by @sebix in #2186
• BUG: mail collectors: fix IMAP abort handling by @wagner-intevation in #2187
• bug: fix ctl process manager initialization non-interactive by @wagner-intevation in #2189
• doc: ms azure collector: more precise requirement name by @wagner-intevation in #2190
• intelmqsetup: fix typo in message by @sebix in #2181
• gitignore: add coverage file by @sebix in #2182
• ENH: new parameter autoupdate_cached_database by @sebix in #2180
• tst: fix reverse_dns test by @wagner-intevation in #2208
• pkg: add requests_mock as dev dependency by @sebix in #2210
• intelmqsetup: Fix #2197 by @wagner-intevation in #2198
• Fix reconnection in SQL Mixin by @wagner-intevation in #2201
• ParserBot: Fix line recovery and message dumping by @wagner-intevation in #2192
• FIX: Added check if cymru hostname is an ip address by @waldbauer-certat in #2215
• bug: fix ctip parser handling JSON Payload content by @wagner-intevation in #2193
• docs: workaround for sphinx-doc/sphinx#10701 by @wagner-intevation in #2225
• FIX: Remove AutoShun from IntelMQ as the feed is discontinued by @waldbauer-certat in #2214
• SQL: add MSSQL support and allow storing only a subset of event fields by @creideiki in #2172
• ENH: Added personal access token support to GitHub Collector by @waldbauer-certat in #2145
• FIX: Use redis BLMOVE instead of BRPOPLPUSH (deprecated) by @waldbauer-certat in #2149
• postgresql output: escape all null bytes by @wagner-intevation in #2223
• Remove malc0de feed due to offline by @gutsohnCERT in #2184
• TST: Fixed regex to hold gcc version by @waldbauer-certat in #2216
• Shadowserver 202207 by @elsif2 in https://github.com/certtools/intelmq/p...

3.0.2 Bugfix release

Installation documentation:
https://intelmq.readthedocs.io/en/maintenance/user/installation.html
Upgrade documentation:
https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html

Core

• intelmq.lib.bot.CollectorBot: Fixed an issue with within the new_report function, which re-loads the harmonization file after a new incoming dataset, which leads to CPU drain and decreased performance (PR#2106 by Sebastian Waldbauer, fixes #2098).
• intelmq.lib.bot.Bot: Make private members __is_multithreadable and __collector_empty_process protected members _is_multithreadable and _collector_empty_process to make them easily modifiable by Bot classes (PR#2109 by Sebastian Wagner, fixes #2108).
  Also affected and adapted bots by this change are:
  • intelmq.bots.collectors.api.collector_api
  • intelmq.bots.collectors.stomp.collector
  • intelmq.bots.experts.splunk_saved_search.expert
  • intelmq.bots.experts.threshold.expert
  • intelmq.bots.outputs.file.output
  • intelmq.bots.outputs.misp.output_api
  • intelmq.bots.outputs.misp.output_feed
  • intelmq.bots.outputs.tcp.output
  • intelmq.bots.outputs.udp.output
• intelmq.lib.cache: Do not create the Cache class if the host is null, allows deactivating the bot statistics (PR#2104 by Sebastian Waldbauer, fixes #2103).

Bots

Experts

• intelmq.bots.experts.domain_suffix.expert: Only print skipped database update message if verbose mode is active (PR#2107 by Sebastian Wagner, fixes #2016).

Documentation

• Add configuration upgrade steps for 3.0 to NEWS (PR#2101 by Sebastian Wagner).

Known issues

See open bug reports for a more detailed list.

• ParserBot: erroneous raw line recovery in error handling (#1850).

3.0.1 Bugfix Release

Installation documentation:
https://intelmq.readthedocs.io/en/maintenance/user/installation.html
Upgrade documentation:
https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html

Core

• intelmq.lib.bot_debugger: Fix accessing the bot's destination queues (PR#2027 by Mikk Margus Möll).
• intelmq.lib.pipeline: Fix handling of load_balance parameter (PR#2027 by Mikk Margus Möll).
• intelmq.lib.bot: Fix handling of parameter destination_queues if value is an empty dictionary (PR#2051 by Sebastian Wagner, fixes #2034).

Bots

Collectors

• intelmq.bots.collectors.shodan.collector_stream: Fix access to parameters, the bot wrongly used self.parameters (PR#2020 by Mikk Margus Möll).
• intelmq.bots.collectors.mail.collector_mail_attach: Add attachment file name as extra.file_name also if the attachment is not compressed (PR#2021 by Alex Kaplan).
• intelmq.bots.collectors.http.collector_http_stream: Fix access to parameters, the bot wrongly used self.parameters (by Sebastian Wagner).

Parsers

• intelmq.bots.parsers.microsoft.parser_ctip: Map Payload.domain to destination.fqdn instead of extra.payload.domain as it matches to destination.ip from DestinationIp (PR#2023 by Sebastian Wagner).
• Removed intelmq.bots.parsers.malwaredomains because the upstream data source (malwaredomains.com) does not exist anymore (PR#2026 by Birger Schacht, fixes #2024).
• intelmq.bots.parsers.shadowserver.config:
  • Add support for feed "Vulnerable SMTP Server" (PR#2037 by Mikk Margus Möll).
  • Fix differentiation between feeds "Accessible HTTP" and "Vulnerable HTTP" (PR#2037 by Mikk Margus Möll, fixes #1984).
  • Add support for the new feeds Microsoft Sinkhole Events Report, Microsoft Sinkhole HTTP Events Report (PR#2036 by Birger Schacht).
  • Complement feed mappings and documentation for feeds with IPv4 and IPv6 variants (PR#2046 by Mikk Margus Möll and Sebastian Wagner).
  • Feed names with and without the optional IPv4/IPv6 postfix can be used now consistently.
  • Add support for feed "Honeypot HTTP Scan" (PR#2047 by Mikk Margus Möll).
  • Update filename mapping for changed filename of feed "Accessible-MSRDPUDP" (PR#2060 by abr4xc).

Experts

• intelmq.bots.experts.gethostbyname.expert: Handle numeric values for the gaierrors_to_ignore parameter (PR#2073 by Sebastian Wagner, fixes #2072).
• intelmq.bots.experts.filter.expert: Fix handling of empty-string parameters not_after and not_before (PR#2075 by Sebastian Wagner, fixes #2074).

Outputs

• intelmq.bots.outputs.mcafee.output_esm_ip: Fix access to parameters, the bot wrongly used self.parameters (by Sebastian Wagner).
• intelmq.bots.outputs.misp.output_api: Fix access to parameters, the bot wrongly used self.parameters (by Sebastian Wagner).
• intelmq.bots.outputs.smtp.output: Add Content-Disposition-header to the attachment, fixing the display in Mail Clients as actual attachment (PR#2052 by Sebastian Wagner, fixes #2018).

Documentation

• Various formatting fixes (by Sebastian Wagner).
• Removed the malwaredomains feed from the feeds list because the upstream data source (malwaredomains.com) does not exist anymore (PR#2026 by Birger Schacht, fixes #2024).
• Update Docker installation instructions (PR#2035 by Sebastian Waldbauer).

Packaging

• intelmq-update-database crontab: Add missing recordedfuture_iprisk update call (by Sebastian Wagner).

Tests

• Replace calls to deprecated/undocumented logging.warn with logging.warning (by Sebastian Wagner, fixes #2013).
• intelmq.tests.bots.experts.rdap.test_expert: Declare cache use, fixes build failures (by Sebastian Wagner, fixes #2014).
• intelmq.tests.bots.collectors.mail.test_collector_attach: Test text attachment (by Sebastian Wagner).

Tools

• intelmqctl:
  • Also honour parameters from environment variables (PR#2068 by Sebastian Wagner, fixes #2063).
  • Fix management actions (start/stop/status/reload/restart) for groups (PR#2086 by Sebastian Wagner, fixes #2085).
  • Do not use hardcoded logging path in /opt/intelmq, use the internal default instead (PR#2092 by Sebastian Wagner, fixes #2091).

Known issues

See open bug reports for a more detailed list.

• ParserBot: erroneous raw line recovery in error handling (#1850).