Releases · certtools/intelmq

30 Jul 13:45

2.2.1

4ecb114

Bugfix release

Installation documentation:
https://github.com/certtools/intelmq/blob/2.2.1/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.2.1/docs/UPGRADING.md

Core

intelmq.lib.upgrades:
- Add upgrade function for changed configuration of the feed "Abuse.ch URLHaus" (#1571, PR#1572 by Filip Pokorný).
- Add upgrade function for removal of HPHosts Hosts file feed and intelmq.bots.parsers.hphosts parser (#1559).
- intelmq.lib.harmonization:
  - For IP Addresses, explicitly reject IPv6 addresses with scope ID (due to changed behavior in Python 3.9, #1550).

Development

Ignore line length (E501) in code-style checks altogether.

Bots

Collectors

intelmq.bots.collectors.misp: Fix access to actual MISP object (PR#1548 by Tomas Bellus @tomas321)
intelmq.bots.collectors.stomp: Remove empty client.pem file.

Parsers

intelmq.bots.parsers.shadowserver.config:
- Add support for Accessible-CoAP feed (PR #1555 by Thomas Hungenberg).
- Add support for Accessible-ARD feed (PR #1584 by Tomas Bellus @tomas321).
intelmq.bots.parser.anubisnetworks.parser: Ignore "TestSinkholingLoss" events, these are not intended to be sent out at all.
intelmq.bots.parsers.generic.parser_csv: Allow values of type dictionary for parameter type_translation.
intelmq.bots.parsers.hphosts: Removed, feed is unavailable (#1559).
intelmq.bots.parsers.cymru.parser_cap_program: Add support for comment "username" for "scanner" category.
intelmq.bots.parsers.malwareurl.parser: Check for valid FQDN and IP address in URL and IP address columns (PR#1585 by Marius Urkis).

Experts

intelmq.bots.experts.maxmind_geoip: On Python < 3.6, require maxminddb < 2, as that version does no longer support Python 3.5.

Outputs

intelmq.bot.outputs.udp: Fix error handling on sending, had a bug itself.

Documentation

Feeds:
- Update documentation of feed "Abuse.ch URLHaus" (#1571, PR#1572 by Filip Pokorný).
Bots:
- Overhaul of all bots' description fields (#1570).
User-Guide:
- Overhaul pipeline configuration section and explain named queues better (#1577).

Tests

intelmq.tests.bots.experts.cymru: Adapt test_empty_result, remove test_unicode_as_name and test_country_question_mark (#1576).

Tools

intelmq.bin.intelmq_gen_docs: Format parameters of types lists with double quotes around values to produce conform JSON, ready to copy and paste the value into the IntelMQ Manager's bot parameter form.
intelmq.bin.intelmqctl:
- debug: In JSON mode, use dictionaries instead of lists.
- debug: Add PATH to the paths shown.
- check: Show $PATH environment variable if executable cannot be found.

Contrib

malware_name_mapping: Change MISP Threat Actors URL to new URL (branch master -> main) in download script.

Known issues

Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
Corrupt dump files when interrupted during writing (#870).
Bash completion scripts search in wrong directory in packages (#1561).
Cymru Expert: Wrong Cache-Key Calculation (#1592).

Assets 2

18 Jun 08:26

2.2.0

91a0d40

Feature Release

Installation documentation:
https://github.com/certtools/intelmq/blob/2.2.0/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.2.0/docs/UPGRADING.md

Dropped support for Python 3.4.

Core

__init__: Changes to the path-handling, see User Guide, section /opt and LSB paths for more information
- The environment variable INTELMQ_ROOT_DIR can be used to set custom root directories instead of /opt/intelmq/ (#805) in case of non LSB-path installations.
- The environment variable ROOT_DIR can be used to set custom root directories instead of / (#805) in case of LSB-path installations.
intelmq.lib.exceptions: Added MissingDependencyError for show error messages about a missing library and how to install it (#1471).
- Added optional parameter installed to show the installed version.
- Added optional parameter additional_text to show arbitrary text.
Adding more type annotations for core libraries.
intelmq.lib.pipeline.Pythonlist.sleep: Drop deprecated method.
intelmq.lib.utils: write_configuration: Append a newline at end of configuration/file to allow proper comparisons & diffs.
intelmq.lib.test: BotTestCase drops privileges upon initialization (#1489).
intelmq.lib.bot:
- New class OutputBot:
  - Method export_event to format/export events according to the parameters given by the user.
- ParserBot: New methods parse_json_stream and recover_line_json_stream.
- ParserBot.recover_line_json: Fix format by adding a list around the line data.
- Bot.send_message: In debugging log level, the path to which the message is sent is now logged too.

Bots

Bots with dependencies: Use of intelmq.lib.exceptions.MissingDependencyError.

Collectors

intelmq.bots.collectors.misp.collector: Deprecate parameter misp_verify in favor of generic parameter http_verify_cert.
intelmq.bots.collectors.tcp.collector: Drop compatibility with Python 3.4.
intelmq.bots.collectors.stomp.collector:
- Check the stomp.py version and show an error message if it does not match.
- For stomp.py versions >= 5.0.0 redirect the stomp.PrintingListener output to debug logging.
intelmq.bots.collectors.microsoft.collector_azure: Support current Python library azure-storage-blob>= 12.0.0, configuration is incompatible and needs manual change. See NEWS file and bot's documentation for more details.
intelmq.bots.collectors.amqp.collector_amqp: Require pika minimum version 1.0.
intelmq.bots.collectors.github_api.collector_github_contents_api: Added (PR#1481).

Parsers

intelmq.bots.parsers.autoshun.parser: Drop compatibility with Python 3.4.
intelmq.bots.parsers.html_table.parser: Drop compatibility with Python 3.4.
intelmq.bots.parsers.shadowserver.parser: Add support for MQTT and Open-IPP feeds (PR#1512, PR#1544).
intelmq.bots.parsers.taichung.parser:
- Migrate to ParserBot.
- Also parse geolocation information if available.
intelmq.bots.parsers.cymru.parser_full_bogons:
- Migrate to ParserBot.
- Add last updated information in raw.
intelmq.bots.parsers.anubisnetworks.parser: Add new parameter use_malware_familiy_as_classification_identifier.
intelmq.bots.parsers.microsoft.parser_ctip: Compatibility for new CTIP data format used provided by the Azure interface.
intelmq.bots.parsers.cymru.parser_cap_program: Support for openresolver type.
intelmq.bots.parsers.github_feed.parser: Added (PR#1481).
intelmq.bots.parsers.urlvir.parser: Removed, as the feed is discontinued (#1537).

Experts

intelmq.bots.experts.csv_converter: Added as converter to CSV.
intelmq.bots.experts.misp: Added (PR#1475).
intelmq.bots.experts.modify: New parameter maximum_matches.

Outputs

intelmq.bots.outputs.amqptopic:
- Use OutputBot and export_event.
- Allow formatting the routing key with event data by the new parameter format_routing_key (boolean).
intelmq.bots.outputs.file: Use OutputBot and export_event.
intelmq.bots.outputs.files: Use OutputBot and export_event.
intelmq.bots.outputs.misp.output_feed: Added, creates a MISP Feed (PR#1473).
intelmq.bots.outputs.misp.output_api: Added, pushes to MISP via the API (PR#1506, PR#1536).
intelmq.bots.outputs.elasticsearch.output: Dropped ElasticSearch version 5 compatibility, added version 7 compatibility (#1513).

Documentation

Document usage of the INTELMQ_ROOT_DIR environment variable.
Added document on MISP integration possibilities.
Feeds:
- Added "Full Bogons IPv6" feed.
- Remove discontinued URLVir Feeds (#1537).

Packaging

setup.py do not try to install any data to /opt/intelmq/ as the behavior is inconsistent on various systems and with intelmqsetup we have a tool to create the structure and files anyway.
debian/rules:
- Provide a blank state file in the package.
Patches:
- Updated fix-intelmq-paths.patch.

Tests

Travis: Use intelmqsetup here too.
- Install required build dependencies for the Debian package build test.
- This version is no longer automatically tested on Python < 3.5.
- Also run the tests on Python 3.8.
- Run the Debian packaging tests on Python 3.5 and the code-style test on 3.8.
Added tests for the new bot intelmq.bots.outputs.misp.output_feed (#1473).
Added tests for the new bot intelmq.bots.experts.misp.expert (#1473).
Added tests for intelmq.lib.exceptions.
Added tests for intelmq.lib.bot.OutputBot and intelmq.lib.bot.OutputBot.export_event.
Added IPv6 tests for intelmq.bots.parsers.cymru.parser_full_bogons.
Added tests for intelmq.lib.bot.ParserBot's new methods parse_json_stream and recover_line_json_stream.
intelmq.tests.test_conf: Set encoding to UTF-8 for reading the feeds.yaml file.

Tools

intelmqctl:
- upgrade-config:
  - Allow setting the state file location with the --state-file parameter.
  - Do not require a second run anymore, if the state file is newly created (#1491).
  - New parameter no_backup/--no-backup to skip creation of .bak files for state and configuration files.
- Only require psutil for the IntelMQProcessManager, not for process manager independent calls like upgrade-config or check.
- Add new command debug to output some information for debugging. Currently implemented:
  - paths
  - environment variables
- IntelMQController: New argument --no-file-logging to disable logging to file.
- If dropping privileges does not work, intelmqctl will now abort (#1489).
intelmqsetup:
- Add argument parsing and an option to skip setting file ownership, possibly not requiring root permissions.
- Call intelmqctl upgrade-config and add argument for the state file path (#1491).
intelmq_generate_misp_objects_templates.py: Tool to create a MISP object template (#1470).
intelmqdump: New parameter -t or --truncate to optionally give the maximum length of raw data to show, 0 for no truncating.

Contrib

Added development-tools.
ElasticSearch: Dropped version 5 compatibility, added version 7 compatibility (#1513).
Malware Name Mapping Downloader:
- New parameter --mwnmp-ignore-adware.
- The parameter --add-default supports an optional parameter to define the default value.

Known issues

Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
Corrupt dump files when interrupted during writing (#870).

Assets 2

26 May 10:58

2.1.3

32fb175

Bugfix release

Installation documentation:
https://github.com/certtools/intelmq/blob/2.1.3/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.3/docs/UPGRADING.md

Requirements

The python library requests is (again) listed as dependency of the core (#1519).

Core

intelmq.lib.upgrades:
- Harmonization upgrade: Also check and update regular expressions.
- Add function to migrate the deprecated parameter attach_unzip to extract_files for the mail attachment collector.
- Add function to migrate changed Taichung URL feed.
- Check for discontinued Abuse.CH Zeus Tracker feed.
intelmq.lib.bot:
- ParserBot.recover_line: Parameter line needs to be optional, fix usage of fallback value self.current_line.
- start: Handle decoding errors in the pipeline different so that the bot is not stuck in an endless loop (#1494).
- start: Only acknowledge a message in case of errors, if we actually had a message to dump, which is not the case for collectors.
- _dump_message: Dump messages with encoding errors base64 encoded, not in JSON format as it's not possible to decode them (#1494).
intelmq.lib.test:
- BotTestCase.run_bot: Add parameters allowed_error_count and allowed_warning_count to allow set the number per run, not per test class.
- Set source_pipeline_broker and destination_pipeline_broker to pythonlist instead of the old broker, fixes intelmq.tests.lib.test_bot.TestBot.test_pipeline_raising.
- Fix test for (allowed) errors and warnings.
intelmq.lib.exceptions:
- InvalidKey: Add KeyError as parent class.
- DecodingError: Added, string representation has all relevant information on the decoding error, including encoding, reason and the affected string (#1494).
intelmq.lib.pipeline:
- Decode messages in Pipeline.receive not in the implementation's _receive so that the internal counter is correct in case of decoding errors (#1494).
intelmq.lib.utils:
- decode: Raise new DecodingError if decoding fails.

Harmonization

protocol.transport: Adapt regular expression to allow the value nvp-ii (protocol 11).

Bots

Collectors

intelmq.bots.collectors.mail.collector_mail_attach:
- Fix handling of deprecated parameter name attach_unzip.
- Fix handling of attachments without filenames (#1538).
intelmq.bots.collectors.stomp.collector: Fix compatibility with stomp.py versions > 4.1.20 and catch errors on shutdown.
intelmq.bots.collectors.microsoft:
- Update REQUIREMENTS.txt temporarily fixing deprecated Azure library (#1530, PR#1532).
- intelmq.bots.collectors.microsoft.collector_interflow: Add method for printing the file list.

Parsers

intelmq.bots.parsers.cymru.parser_cap_program: Support for protocol 11 (nvp-ii) and conficker type.
intelmq.bots.parsers.taichung.parser: Support more types/classifications:
- Application Compromise: Apache vulnerability & SQL injections
- Brute-force: MSSQL & SSH password guess attacks; Office 365, SSH & SIP attacks
- C2 Sever: Attack controller
- DDoS
- DoS: DNS, DoS, Excess connection
- IDS Alert / known vulnerability exploitation: backdoor
- Malware: Malware Proxy
- Warn on new unknown types.
intelmq.bots.parsers.bitcash.parser: Removed as feed is discontinued.
intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc and intelmq.bots.parsers.fraunhofer.parser_ddosattack_target: Removed as feed is discontinued.
intelmq.bots.parsers.malwaredomains.parser: Correctly classify C&C and phishing events.
intelmq.bots.parsers.shadowserver.parser: More verbose error message for missing report specification (#1507).
intelmq.bots.parsers.n6.parser_n6stomp: Always add n6 field name as malware.name independent of category.
intelmq.bots.parsers.anubisnetworks: Update parser with new data format.
intelmq.bots.parsers.bambenek: Add new feed URLs with Host faf.bambenekconsulting.com (#1525, PR#1526).
intelmq.bots.parsers.abusech.parser_ransomware: Removed, as the feed is discontinued (#1537).
intelmq.bots.parsers.nothink.parser: Removed, as the feed is discontinued (#1537).
intelmq.bots.parsers.n6.parser: Remove not allowed characters in the name field for malware.name and write original value to event_description.text instead.

Experts

intelmq.bots.experts.cymru_whois.lib: Fix parsing of AS names with Unicode characters.

Outputs

intelmq.bots.outputs.mongodb:
- Set default port 27017.
- Use different authentication mechanisms per MongoDB server version to fix compatibility with server version >= 3.4 (#1439).

Documentation

Feeds:
- Remove unavailable feed Abuse.CH Zeus Tracker.
- Remove the field status, offline feeds should be removed.
- Add a new field public to differentiate between private and public feeds.
- Adding documentation URLs to nearly all feeds.
- Remove unavailable Bitcash.cz feed.
- Remove unavailable Fraunhofer DDos Attack feeds.
- Remove unavailable feed Abuse.CH Ransomware Tracker (#1537).
- Update information on Bambenek Feeds, many require a license now (#1525).
- Remove discontinued Nothink Honeypot Feeds (#1537).
Developers Guide: Fix the instructions for /opt/intelmq file permissions.

Packaging

Patches: fix-logrotate-path.patch: also include path to rotated file in patch.
Fix paths from /opt to LSB for setup.py and contrib/logrotate/intelmq in build process (#1500).
Add runtime dependency debianutils for the program which, which is required for intelmqctl.

Tests

Dropping Travis tests for 3.4 as required libraries dropped 3.4 support.
intelmq.tests.bots.experts.cymru_whois:
- Drop missing ASN test, does not work anymore.
- IPv6 to IPv4 test: Test for two possible results.
intelmq.lib.test: Fix compatibility of logging capture with Python >= 3.7 by reworking the whole process (#1342).
intelmq.bots.collectors.tcp.test_collector: Removing custom mocking and bot starting, not necessary anymore.
Added tests for intelmq.bin.intelmqctl.IntelMQProcessManager._interpret_commandline.
Fix and split tests.bots.experts.ripe.test_expert.test_ripe_stat_error_json.
Added tests for invalid encodings in input messages in intelmq.tests.lib.test_bot and intelmq.tests.lib.test_pipeline (#1494).
Travis: Explicitly enable RabbitMQ management plugin.
intelmq.tests.lib.test_message: Fix usage of the parameter blacklist for Message hash tests (#1539).

Tools

intelmqsetup: Copy missing BOTS file to IntelMQ's root directory (#1498).
intelmq_gen_docs: Feed documentation generation: Handle missing/empty parameters.
intelmqctl:
- IntelMQProcessManager: For the status of running bots also check the bot ID of the commandline and ignore the path of the executable (#1492).
- IntelMQController: Fix exit codes of check command for JSON output (now 0 on success and 1 on error, was swapped, #1520).
intelmqdump:
- Handle base64-type messages for show, editor and recovery actions.

Contrib

intelmq/bots/experts/asn_lookup/update-asn-data: Use pyasn_util_download.py to download the data instead from RIPE, which cannot be parsed currently (#1517, PR#1518, hadiasghari/pyasn#62).

Known issues

HTTP stream collector: retry on regular connection problems? (#1435).
Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
Reverse DNS: Only first record is used (#877).
Corrupt dump files when interrupted during writing (#870).

Assets 2

28 Jan 15:48

2.1.2

84e1bf8

Bugfix release

Install documentation:
https://github.com/certtools/intelmq/blob/2.1.2/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.2/docs/UPGRADING.md

Core

__init__: Resolve absolute path for STATE_FILE_PATH variable (resolves ..).
intelmq.lib.utils:
- log: Do not raise an exception if logging to neither file nor syslog is requested.
- logging StreamHandler: Colorize all warning and error messages red.
- logging FileHandler: Strip all shell colorizations from the messages (#1436).
intelmq.lib.message:
- Message.to_json: Set sort_keys=True to get reproducible results.
- drop_privileges: Handle situations where the user or group intelmq does not exist.
intelmq.lib.pipeline:
- Amqp._send and Amqp._acknowledge: Log traceback in debug mode in case of errors and necessary re-connections.
- Amqp._acknowledge: Reset delivery tag if acknowledge was successful.

Bots

Collectors

intelmq.bots.collectors.misp.collector:
- Add compatibility with current pymisp versions and versions released after January 2020 (PR #1468).

Parsers

intelmq.bots.parsers.shadowserver.config: Add some missing fields for the feed accessible-rdp (#1463).
intelmq.bots.parsers.shadowserver.parser:
- Feed-detection based on file names: The prefixed date is optional now.
- Feed-detection based on file names: Re-detect feed for every report received (#1493).

Experts

intelmq.bots.experts.national_cert_contact_certat: Handle empty responses by server (#1467).
intelmq.bots.experts.maxmind_geoip: The script update-geoip-data now requires a license key as second parameter because of upstream changes (#1484)).

Outputs

intelmq.bots.outputs.restapi.output: Fix logging of response body if response status code was not ok.

Documentation

Remove some hardcoded /opt/intelmq/ paths from code comments and program outputs.

Packaging

debian/rules: Only replace /opt/intelmq/ with LSB-paths in some certain files, not the whole tree, avoiding wrong replacements.
debian/rules and debian/intelmq.install: Do install the examples configuration directly instead of working around the abandoned examples directory.

Tests

lib/test_utils: Skip some tests on Python 3.4 because contextlib.redirect_stdout and contextlib.redirect_sterr are not supported on this version.
Travis: Stop running tests with all optional dependencies on Python 3.4, as more and more libraries are dropping support for it. Tests on the core and code without non-optional requirements are not affected.
tests.bots.parsers.html_table: Make tests independent of current year.

Tools

intelmqctl upgrade-config: Fix missing substitution in error message "State file %r is not writable.".

Known issues

bots trapped in endless loop if decoding of raw message fails (#1494)
intelmqctl status of processes: need to check bot id too (#1492)
MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439)
ctl: shell colorizations are logged (#1436)
http stream collector: retry on regular connection problems? (#1435)
tests: capture logging with context manager (#1342)
Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
n6 parser: mapping is modified within each run (#905)
reverse DNS: Only first record is used (#877)
Corrupt dump files when interrupted during writing (#870)

Assets 2

11 Nov 15:48

2.1.1

db82a82

2.1.1

Install documentation:
https://github.com/certtools/intelmq/blob/2.1.1/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.1/docs/UPGRADING.md

Configuration

Default configuration:
- Remove discontinued feed "Feodo Tracker Domains" from default configuration.
- Add "Feodo Tracker Browse" feed to default configuration.

Core

intelmq.lib.pipeline: AMQP: using port 15672 as default (like RabbitMQ's defaults) for the monitoring interface for getting statistical data (intelmqctl_rabbitmq_monitoring_url).
intelmq.lib.upgrades: Added a generic upgrade function for harmonization, checking of all message types, it's fields and their types.
intelmq.lib.utils:
- TimeoutHTTPAdapter: A subclass of requests.adapters.HTTPAdapter with the possibility to set the timeout per adapter.
- create_request_session_from_bot: Use the TimeoutHTTPAdapter with the user-defined timeout. Previously the timeout was not functional.

Bots

Parsers

intelmq.bots.parsers.shadowserver.parser: Fix logging message if the parameter feedname is not present.
intelmq.bots.parsers.shodan.parser: Also add field classification.identifier ('network-scan') in minimal mode.
intelmq.bots.parsers.spamhaus.parser_cert: Add support for category 'misc'.
intelmq.bots.parsers.cymru.parser_cap_program:
- Add support for phishing events without URL.
- Add support for protocols >= 143 (unassigned, experiments, testing, reserved), saving the number to extra, as the data would be bogus.
intelmq.bots.parsers.microsoft.parser_bingmurls:
- Save the Tags data as source.geolocation.cc.

Experts

intelmq.bots.experts.modify.expert: Fix bug with setting non-string values (#1460).

Outputs

intelmq.bots.outputs.smtp:
- Allow non-existent field in text formatting by using a default value None instead of throwing errors.
- Fix Authentication (#1464).
- Fix sending to multiple recipients (#1464).

Documentation

Feeds:
- Fix configuration of Feodo Tracker Browse feed.
Bots:
- Sieve expert: Document behavior of != with lists.

Tests

Adaption and extension of the test cases to the changes.

Tools

intelmq.bin.intelmqctl:
- check: Check if running the upgrade function for harmonization is necessary.
- upgrade-config: Run the upgrade function for harmonization.
- intelmqctl restart did throw an error as the message for restarting was not defined (#1465).

Known issues

MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439)
ctl: shell colorizations are logged (#1436)
http stream collector: retry on regular connection problems? (#1435)
tests: capture logging with context manager (#1342)
Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
n6 parser: mapping is modified within each run (#905)
reverse DNS: Only first record is used (#877)
Corrupt dump files when interrupted during writing (#870)

Assets 2

15 Oct 11:10

2.1.0

ced415d

2.1.0

Install documentation:
https://github.com/certtools/intelmq/blob/2.1.0/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.0/docs/UPGRADING.md

Core

intelmq.lib.harmonization:
- Use correct parent classes.
- Add DateTime.convert as interface for all existing conversion functions.
- add DateTime.convert_from_format.
- add DateTime.convert_from_format_midnight.
- add DateTime.convert_fuzzy.
intelmq.lib.pipeline:
- Redis: Use single connection client if calling bot is not multithreaded. Gives a small speed advantage.
- Require the bot instance as parameter for all pipeline classes.
- New internal variable _has_message to keep the state of the pipeline.
- Split receive and acknowledge into public-facing and private methods.
- Add reject_message method to the Pipeline class for explicit requeue of messages.
- AMQP:
  - Make exchange configurable.
  - If exchange is set, the queues are not declared, the queue name is for routing used by exchanges.
intelmq.lib.bot:
- Log message after successful bot initialization, no log message anymore for ready pipeline.
- Use existing current message if receive is called and the current message still exists.
- Fix handling of received messaged after a sighup that happend during a blocking receving connection using explicit rejection (#1438).
- New method _parse_common_parameters called before init to parse commonly used argument. Currently supported: extract_files.
intelmq.lib.test:
- Fix the tests broker by providing the testing pipeline.
intelmq.lib.utils:
- unzip:
  - new parameter return_names to optionally return the file names.
  - support for zip
  - new parameters try_zip, try_gzip and try_tar to control which compressions are tried.
  - rewritten to an iterative approach
- add file_name_from_response to extract a file name from a Response object for downloaded files.
intelmq.lib.upgrades: Added v210_deprecations for deprecated parameters.

Harmonization

Add extra to reports.

Bots

Collectors

intelmq.bots.collectors.http.collector_http:
- More extensive usage of intelmq.lib.utils.unzip.
- Save the file names in the report if files have been extracted form an archive.
intelmq.bots.collectors.rt.collector_rt:
- Save ticket information/metadata in the extra fields of the report.
- Support for RT 3.8 and RT 4.4.
- New parameters extract_attachment and extract_download for generic archive extraction and consistency. The parameter unzip_attachment is deprecated.
intelmq.bots.collectors.mail.*: Save email information/metadata in the extra fields of the report. See the bots documentation for a complete list of provided data.
- intelmq.bots.collectors.mail.collector_mail_attach: Check for existence/validity of the attach_regex parameter.
- Use the lib's unzip function for uncompressing attachments and use the .
- intelmq.bots.collectors.mail.collector_mail_url: Save the file name of the downloaded file as extra.file_name.
intelmq.bots.collectors.amqp.collector_amqp: New collector to collect data from (remote) AMQP servers, for bot IntelMQ as well as external data.
- use default SSL context for client purposes, fixes compatibility with python < 3.6 if TLS is used.

Parsers

intelmq.bot.parsers.html_table.parser:
- New parameter "html_parser".
- Use time conversion functions directly from intelmq.lib.harmonization.DateTime.convert.
- Limit lxml dependency on 3.4 to < 4.4.0 (incompatibility).
intelmq.bots.parsers.netlab_360.parser: Add support for hajime scanners.
intelmq.bots.parsers.hibp.parser_callback: A new parser to parse data retrieved from a HIBP Enterprise Subscription.
intelmq.bots.parsers.shadowserver.parser:
- Ability to detect the feed base on the reports's field extra.file_name, so the parameter feedname is no longer required and one configured parser can parse any feed (#1442).

Experts

Add geohash expert.
intelmq.bot.experts.generic_db_lookup.expert
- new optional parameter engine with postgresql (default) and sqlite (new) as possible values.

Outputs

Add intelmq.bots.outputs.touch.output.
intelmq.bot.outputs.postgresql.output:
- deprecated in favor of intelmq.bot.outputs.sql.output
- Compatibility shim will be available in the 2.x series.
intelmq.bot.outputs.sql.output added generic SQL output bot. Comparted to
- new optional parameter engine with postgresql (default) and sqlite (new) as possible values.
intelmq.bots.outputs.stomp.output: New parameters message_hierarchical_output, message_jsondict_as_string, message_with_type, single_key.

Documentation

Feeds:
- Add ViriBack feed.
- Add Have I Been Pwned Enterprise Callback.
intelmq.tests.bots.outputs.amqptopic.test_output: Added.
Move the documentation of most bots from separate README files to the central Bots.md and feeds.yaml files.

Tests

Travis:
- Use UTC timezone.
Tests for utils.unzip.
Add a new asset: Zip archive with two files, same as with tar.gz archive.
Added tests for the Mail Attachment & Mail URL collectors.
Ignore logging-tests on Python 3.7 temporarily (#1342).

Tools

intelmqctl:
- Use green and red text color for some interactive output to indicate obvious errors or the absence of them.
intelmqdump:
- New edit action v to modify a message saved in the dump (#1284).

Contrib

malware name mapping:
- Add support for MISP treat actors data, see it's README for more information.
  - And handle empty synonyms in misp's galxies data.
- Move apply-Script to the new EventDB directory
EventDB: Scripts for applying malware name mapping and domain suffixes to an EventDB.

Known issues

MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439)
ctl: shell colorizations are logged (#1436)
http stream collector: retry on regular connection problems? (#1435)
tests: capture logging with context manager (#1342)
Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
n6 parser: mapping is modified within each run (#905)
reverse DNS: Only first record is used (#877)
Corrupt dump files when interrupted during writing (#870)

Assets 2

14 Oct 19:06

2.0.2

ff598e9

2.0.2

Install documentation:
https://github.com/certtools/intelmq/blob/2.0.2/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.0.2/docs/UPGRADING.md

As always: read the NEWS file, upgrade according to the documentation
and have fun! If you get any errors, please report them here or in the
bug tracker.

Core

intelmq.lib.bot.CollectorBot: Support the deprecated parameter feed until version 2.2 as the documentation was not properly updated (#1445).
intelmq.lib.bot.Bot:
- _dump_message: Wait for up to 60 seconds instead of 50 if the dump file is locked (the log message was said 60, but the code was for only 50).
intelmq.lib.upgrades.v202_fixes
- Migration of deprecated parameter feed for Collectors.
- Ripe expert parameter query_ripe_stat_ip was not correctly configured in v110_deprecations, now use query_ripe_stat_asn as default if it does not exist.
intelmq.lib.upgrades.v110_deprecations: Fix upgrade of ripe expert configuration.
intelmq.lib.bot_debugger:
- Fix handling of empty messages generated by parser when user wanted to show the result by "--show-sent" flag.
- Fix handling of sent messages for bots using the path_permissive paramter (#1453).
intelmq.lib.pipeline.Amqp:
- use default SSL context for client purposes, fixes compatibility with python < 3.6 if TLS is used.
- Reconnect once on sending messages if disconnect detected.

Bots

Collectors

intelmq.bots.collectors.api.collector_api:
- Handle non-existing IO loop in shutdown.
- Close socket on shutdown, fixes reloading.
- Marked as non-threadable.
intelmq.bots.collectors.rt.collector_rt: Check for matching URLs if no attachment_regex is given.
intelmq.bots.collectors.stomp.collector_stomp: Handle disconnects by actively reconnecting.

Parsers

intelmq.bots.cymru.parser_cap_program: Fix parsing of the new $certname_$date.txt report format (#1443):
- Support protocol ICMP.
- Fix error message for unsupported protocols.
- Support fields destination_port_numbers, port.
- Support for all proxy types without ports.
- Use Country Code of AS as source.geolocation.cc.
- Support for 'scanner' and 'spam' categories.
- Handle bogus lines with missing separator.
- Fix bug preventing use of old format after using the new format.
- Handle postfix (total_count:..) for destination port numbers.

Experts

intelmq.bots.experts.cymru_whois.expert: Add optional parameter overwrite, current behavior was True, default if not given is True now, will change to False in 3.0.0 (#1452, #1455).
intelmq.bots.experts.modify.expert: Add optional parameter overwrite, current behavior was True, default if not given is True now, will change to False in 3.0.0 (#1452, #1455).
intelmq.bots.experts.reverse_dns.expert: Add optional parameter overwrite, current behavior was True, default if not given is True now, will change to False in 3.0.0 (#1452, #1455).

Outputs

intelmq.bots.outputs.amqptopic.output: use default SSL context for client purposes, fixes compatibility with python < 3.6 if TLS is used.

Packaging

Rules:
- Exclude intelmqsetup tool in packages
- Include update-rfiprisk-data in packages

Tests

Tests for intelmq.lib.upgrades.v202_fixes.
Tests for intelmq.lib.upgrades.v110_deprecations.
Extended tests for intelmq.bots.parser.cymru.parser_cap_program.

Tools

intelmqctl:
- More and more precise logging messages for botnet starting and restarting, enable and disable.
- No error message for disabled bots on botnet reload.
- Fix upgrade-conf is state file is empty or not existing.
- Use arpgarse's store_true action for flags instead of store_const.
- If the loading of the defaults configuration failed, a variable definition was missing and causing an exception (#1456).

Contrib

Check MK Statistics Cronjob:
- Use statistics_* parameters.
- Make file executable
- Handle None values in *.temporary.* keys and treat them as 0.
systemd:
- Add PIDFile parameter to service file.

Known issues

MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439)
ctl: shell colorizations are logged (#1436)
http stream collector: retry on regular connection problems? (#1435)
tests: capture logging with context manager (#1342)
Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
n6 parser: mapping is modified within each run (#905)
reverse DNS: Only first record is used (#877)
Corrupt dump files when interrupted during writing (#870)

Assets 2

14 Oct 19:07

2.0.1

5cfe2ad

2.0.1

Install documentation:
https://github.com/certtools/intelmq/blob/2.0.1/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.0.1/docs/UPGRADING.md

As always: read the NEWS file, upgrade according to the documentation
and have fun! If you get any errors, please report them here or in the
bug tracker.

Core

intelmq.lib.harmonization:
- IPAddress: Remove Scope/Zone IDs for IPv6 addresses in sanitation.
- All types: Handle None for validation and sanitation gracefully.
intelmq.lib.bot:
- fix parameters of ParserBot and CollectorBot constructors, allowing intelmqctl run with these bots again (#1414).
- Also run rate_limit after retry counter reset (#1431).
__version_info__:
- is now available in the top level module.
- uses integer values now instead of strings for numerical version parts
Also provide (empty) ROOT_DIR for non-pip installations.
intelmq.lib.upgrades: New library file upgrades with upgrade functions.
intelmq.lib.utils:
- New function setup_list_logging for intelmqctl check an possibly others.
  - Fix return values (#1423).
- New function version_smaller for version comparisons.
- New function lazy_int for version conversions.
- parse_logline: Handle thread IDs.
- log takes a new argument logging_level_stream for the logging level of the console handler.
- New constant LOG_FORMAT_SIMPLE, used by intelmqctl.
- New function write_configuration to write dicts to files in the correct json formatting.
- New function create_request_session_from_bot.
intelmq.lib.pipeline:
- AMQP:
  - Actually use source/destination_pipeline_amqp_virtual_host parameter.
  - Support for SSL with source/destination_pipeline_ssl parameter.
- pipeline base class: add missing dummy methods.
- Add missing return types.
- Redis: Evaluate return parameter of queue/key deletion.
Variable STATE_FILE_PATH added.

Development

intelmq.bin.intelmq_gen_docs: For yaml use safe_load instead of unsafe load.

Harmonization

IPAddress type: Remove Scope/Zone IDs for IPv6 addresses in sanitation.
TLP: Sanitation handles now more cases: case-insensitive prefixes and arbitrary whitespace between the prefix and the value (#1420).

Bots

Collectors

intelmq.bots.collectors.http.collector_http: Use utils.create_request_session_from_bot.
intelmq.bots.collectors.http.collector_http_stream: Use utils.create_request_session_from_bot and thus fix some retries on connection timeouts.
intelmq.bots.collectors.mail.collector_mail_url: Use utils.create_request_session_from_bot.
intelmq.bots.collectors.microsoft.collector_interflow: Use utils.create_request_session_from_bot and thus fix retries on connection timeouts.
intelmq.bots.collectors.rt.collector_rt: Use utils.create_request_session_from_bot and thus fix retries on connection timeouts.
intelmq.bots.collectors.twitter.collector_twitter: Use utils.create_request_session_from_bot and thus fix retries on connection timeouts for non-twitter connections.

Parsers

intelmq.bots.parsers.n6.parser_n6stomp: use malware-generic instead of generic-n6-drone for unknown infected system events.
intelmq.bots.parsers.abusech.parser_ip: Support LastOnline column in feodo feed (#1400) and use it for time.source if available.
- Use lower case malware names as default, should not make a difference in practice.
- Fix handling of CSV header for feodotracker (#1417, #1418).
intelmq.bots.parsers.netlab_360.parser: Detect feeds with https:// too.

Experts

intelmq.bots.experts.generic_db_lookup: Recommend psycopg2-binary package.
intelmq.bots.experts.modify.expert:
- Compile regular expressions (all string rules) at initialization, improves the speed.
- Warn about old configuration style deprecation.
intelmq.bots.experts.do_portal.expert:
- Use utils.create_request_session_from_bot and thus fix retries on connection timeouts (#1432).
- Treat "502 Bad Gateway" as timeout which can be retried.
intelmq.bots.experts.ripe.expert: Use utils.create_request_session_from_bot and thus fix retries on connection timeouts.
intelmq.bots.experts.url2fqdn.expert: Support for IP addresses in hostnames (#1416).
intelmq.bots.experts.national_cert_contact_certat.expert: Use utils.create_request_session_from_bot and thus fix retries on connection timeouts.

Outputs

intelmq.bots.outputs.postgresql: Recommend psycopg2-binary package.
intelmq.bots.outputs.amqptopic:
- Shutdown: Close connection only if connection exists.
- Add support for pika > 1. Pika changed the way it indicates (Non-)Acknowledgments of sent messages.
- Gracefully handle unroutable messages and give advice.
- Support for connections without authentication.
- Replace deprecated parameter type with exchange_type for exchange_declare, supporting pika >= 0.11 (#1425).
- New parameters message_hierarchical_output, message_with_type, message_jsondict_as_string.
- New parameter use_ssl for SSL connections.
- New parameter single_key for sending single fields instead of the full event.
intelmq.bots.outputs.mongodb.output: Support for pymongo >= 3.0.0 (#1063, PR#1421).
intelmq.bots.outputs.file: time.* field serialization: support for microseconds.
intelmq.bots.outputs.mongodb.output: Support for authentication in pymongo >= 3.5 (#1062).
intelmq.bots.outputs.restapi.output: Use utils.create_request_session_from_bot and thus fix retries on connection timeouts.

Documentation

Add certbund-contact to the ecosystem document.
Rename the IDEA expert to "IDEA Converter".
Add the new configuration upgrade function to the docs.
User Guide:
- Clarify on Uninstallation

Packaging

Do not execute the tcp collector tests during debian and ubuntu builds as they fail there.

Tests

intelmq.lib.test: Disable statistics for test runs of bots.
contrib.malware_name_mapping: Added tests.
Travis: Also run tests of contrib.

Tools

intelmqsetup: Only change directory ownerships if necessary.
intelmqctl:
- Provide new command upgrade-conf to uprade configuration to a newer version.
  - Makes backups of configurations files on its own.
  - Also checks for previously skipped or new functions of older versions and catches up.
- Provides logging level on class layer.
- Fix -q flag for intelmqctl list queues by renaming its alternative name to --non-zero to avoid a name collision with the global --quiet parameter.
- For console output the string intelmqctl: at the beginning of each line is no longer present.
- check: Support for the state file added. Checks if it exists and all upgrade functions have been executed successfully.
- Waits for up to 2 seconds when stopping a bot (#1434).
- Exits early on restart when stopping a bot did not work (#1434).
- intelmqctl run process -m debugging: Mock acknowledge method if incoming message is mocked too, otherwise a different message is acknowledged.
- Queue listing for AMQP: Support non-default monitoring URLs, see User-Guide.

Contrib

logcheck rules: Adapt ignore rule to cover the instance IDs of bot names.
malware name mapping:
- Ignore lines in mapping starting with '#'.
- Optionally include malpedia data.
- Fix command line parsing for not arguments (#1427).

bash-completion: Support for intelmqctl upgrade-config added.

Known issues

http stream collector: retry on regular connection problems? (#1435)
tests: capture logging with context manager (#1342)
Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
n6 parser: mapping is modified within each run (#905)
reverse DNS: Only first record is used (#877)
Corrupt dump files when interrupted during writing (#870)

Assets 2

22 May 13:10

2.0.0

14bde43

2.0.0

Installation instructions:
https://github.com/certtools/intelmq/blob/2.0.0/docs/INSTALL.md
Upgrade instructions:
https://github.com/certtools/intelmq/blob/2.0.0/docs/UPGRADING.md

There are some features considered as beta and marked as such in the documentation, do not use them in production yet.

See also the changelog for 2.0.0.beta1 below.

Configurations

Defaults: New parameters statistics_host, statistics_port, statistics_databasae, statistics_password for statistics redis database (#1402).

Core

Add more and fix some existing type annotations.
intelmq.lib.bot:
- Use statistics_* parameters for bot's statistics (#1402).
- Introduce collector_empty_process for collectors with an empty process() method, hardcoded 1s minimum sleep time, preventing endless loops, causing high load (#1364).
- Allow to disable multithreading by initialization parameter, used by intelmqctl / the bot debugger (#1403).
intelmq.lib.pipeline: redis: OOM can also be low memory, add this to log message (#1405).
intelmq.lib.harmonization: ClassificationType: Update RSIT mapping (#1380):
- replace botnet drone with infected-system
- replace infected system with infected-system
- replace ids alert with ids-alert
- replace c&c with c2server
- replace malware configuration with malware-configuration
- sanitize replaces these values on the fly
Allow using non-opt/ (LSB) paths with environment variable INTELMQ_PATHS_NO_OPT.
Disable/disallow threading for all collectors and some other bots.

Development

Applied isort to all core files and core-related test files, sorting the imports there (every thing except bots and bots' tests).

Harmonization

See the Core section for the changes in the allowed values for classification.type.

Bots

Use the new RSIT types in several types, see above

Parsers

intelmq.bots.parsers.spamhaus.parser_cert: Added support for extortion events.

Experts

added intelmq.bots.experts.do_portal.expert.

Outputs

intelmq.bots.outputs.elasticsearch.output: Support for TLS added (#1406).
intelmq.bots.outputs.tcp.output: Support non-intelmq counterparts again. New parameter counterpart_is_intelmq, see NEWS.md for more information (#1385).

Packaging

Update IntelMQ path fix patch after INTELMQ_PATHS_NO_OPT introduction, provide INTELMQ_PATHS_OPT environment variable for packaged instances.

Tests

test_conf: For yaml use safe_load instead of unsafe load.
Travis: Switch distribution from trusty to xenial, adapt scripts.
- Add Python 3.7 to tests.
Don't use Cerberus 1.3 because of pyeve/cerberus#489

Tools

intelmqdump: Fix creation of pipeline object by providing a logger.
intelmqctl: Disable multithreading for interactive runs / the bot debugger (#1403).

Known issues

tests: capture logging with context manager (#1342)
pymongo 3.0 deprecates used insert method (#1063)
pymongo >= 3.5: authentication changes (#1062)
Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
n6 parser: mapping is modified within each run (#905)
reverse DNS: Only first record is used (#877)
Corrupt dump files when interrupted during writing (#870)

Assets 2

10 Apr 12:50

2.0.0.beta1

c1e57d4

2.0.0 Beta 1 Pre-release

Pre-release

Installation instructions:
https://github.com/certtools/intelmq/blob/2.0.0.beta1/docs/INSTALL.md
Upgrade instructions:
https://github.com/certtools/intelmq/blob/2.0.0.beta1/docs/UPGRADING.md

There are some features considered as beta and marked as such in the documentation, do not use them in production yet.

upgraded all files to python3-only syntax, e.g. use super() instead of super(..., ...) in all files. Migration from old to new string formatting has not been applied if the resulting code would be longer.

Removals of deprecated code:

Removed compatibility shim intelmq.bots.collectors.n6.collector_stomp, use intelmq.bots.collectors.stomp.collector instead (see #1124).
Removed compatibility shim intelmq.bots.parsers.cymru_full_bogons.parser, use intelmq.bots.parsers.cymru.parser_full_bogons instead.
Removed compatibility shim handing deprecated parameter feed for collectors. Use name instead.
Removed deprecated and unused method intelmq.lib.pipeline.Pipeline.sleep.
Removed support for deprecated parameter query_ripe_stat in intelmq.bots.experts.ripe.expert, use query_ripe_stat_asn and query_ripe_stat_ip instead (#1291).
Removed deprecated and unused function intelmq.lib.utils.extract_tar.

Core

lib/pipeline:
- Allow setting the broker of source and destination independently.
- Support for a new AMQP broker. See User Guide for configuration. (#1179)
lib/bot:
- Dump messages locks the dump file using unix file locks (#574).
- Print idle/rate limit time also in human readable format (#1332).
- set_request_parameters: Use {} as default proxy value instead of None. Allows updating of existing proxy dictionaries.
- Bots drop privileges if they run as root.
- Save statistics on successfully and failed processed messages in the redis database 3.
lib/utils
- Function unzip to extract files from gzipped and/or tar-archives.
- New class ListHandler: new handler for logging purpose which saves the messages in a list.
- Add function seconds_to_human.
- Add function drop_privileges.
- parse_relative: Strip string before parsing.
- parse_logline: Do not convert the timestamps to UTC, leave them as is.
lib/cache:
- Allow ttl to be None explicitly.
- Overwrite existing cache keys in the database instead of discarding the new data.
lib/bot:
- Basic, but easy-to-configure multi-threading using python's threading library. See the User-Guide for more information (#111, #186).
bin/intelmqctl:
- Support for Supervisor as process manager (#693, #1360).

Harmonization

Bots

Collectors

added intelmq.bots.parsers.opendxl.collector (#1265).
added intelmq.bots.collectors.api: collecting data using an HTTP API (#123, #1187).
added intelmq.bots.collectors.rsync (#1286).
intelmq.bots.collectors.http.collector_http:
- Add support for uncompressing of gzipped-files (#1270).
- Add time-delta support for time formatted URLs (#1366).
intelmq.collectors.blueliv.collector_crimeserver: Allow setting the API URL by parameter (#1336).
intelmq.collectors.mail:
- Use internal lib for functionality.
- Add intelmq.bots.collectors.mail.collector_mail_body.
- Support for ssl_ca_certificate parameter (#1362).

Parsers

added intelmq.bots.parsers.mcafee.parser_atd (#1265).
intelmq.bots.parsers.generic.parser_csv:
- New parameter columns_required to optionally ignore parse errors for columns.
added intelmq.bots.parsers.cert_eu.parser_csv (#1287).
- Do not overwrite the local time.observation with the data from the feed. The feed's field 'observation time' is now saved in the field extra.cert_eu_time_observation.
- Fix parsing of asn (renamed to source asn, source.asn internally) and handle existing feed.accuracy for parsing confidence.
- Update columns and mapping to current (2019-04-02) data.
added intelmq.bots.parsers.surbl.surbl
added intelmq.bots.parsers.html_table (#1381).
intelmq.bot.parsers.netlab_360.parser: Handle empty lines containing blank characters (#1393).
intelmq.bots.parsers.n6.parser_n6stomp: Handle events without IP addresses.
intelmq.bots.parsers.cymru.parser_cap_program: Handle new feed format.
intelmq.bots.parsers.shadowserver:
- Add support for the Accessible-FTP feed (#1391).
intelmq.bots.parsers.dataplane.parser:
- Fix parse errors and log more context (#1396).
added intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc.py and intelmq.bots.parsers.fraunhofer.parser_ddosattack_target.py (#1373).

Experts

added intelmq.bots.experts.recordedfuture_iprisk (#1267).
added intelmq.bots.experts.mcafee.expert_mar (1265).
renamed intelmq.bots.experts.ripencc_abuse_contact.expert to intelmq.bots.experts.ripe.expert, compatibility shim will be removed in version 3.0.
- Added support for geolocation information in ripe expert with a new parameter query_ripe_stat_geolocation (#1317).
- Restructurize the expert and de-duplicataion (#1384).
- Handle '?' in geolocation country data (#1384).
intelmq.bots.experts.ripe.expert:
- Use a requests session (#1363).
- Set the requests parameters once per session.
intelmq.bots.experts.maxmind_geoip.expert: New parameter use_registered to use the registered country (#1344).
intelmq.bots.experts.filter.expert: Support for paths (#1208).

Outputs

added intelmq.bots.experts.mcafee.output_esm (1265).
added intelmq.bots.outputs.blackhole (#1279).
intelmq.bots.outputs.restapi.expert:
- Set the requests parameters once per session.
intelmq.bots.outputs.redis:
- New parameter hierarchichal_output (#1388).
- New parameter with_type.
intelmq.bots.outputs.amqptopic.output: Compatibility with pika 1.0.0 (#1084, #1394).

Documentation

added documentation for feeds
- CyberCrime Tracker
- Feodo Tracker Latest
Feeds: Document abuse.ch URLhaus feed (#1379).
Install and Upgrading: Use intelmqsetup tool.

Packaging

Tests

Add tests of AMQP broker.
Travis: Change the ownership of /opt/intelmq to the current user.

Tools

intelmqctl check: Now uses the new ListHandler from utils to handle the logging in JSON output mode.
intelmqctl run: The message that a running bot has been stopped, is not longer a warning, but an informational message. No need to inform sysadmins about this intended behaviour.
intelmqdump: Inspecting dumps locks the dump file using unix file locks (#574).
intelmqctl:
- After the check if the program runs as root, it tries to drop privileges. Only if this does not work, a warning is shown.
intelmqsetup: New tool for initialize an IntelMQ environment.

Contrib

malware_name_mapping:
- Added the script apply_mapping_eventdb.py to apply the mapping to an eventdb.
- Possibility to add local rules using the download tool.
check_mk:
- Added scripts for monitoring queues and statistics.

Known issues

Multi-threaded bots require multiple SIGTERMs (#1403)
Stats can't be saved with AMQP if redis is password-protected (#1402)
Update taxonomies to current RSIT and vice-versa (#1380)
stomp collector bot constantly uses 100% of CPU (#1364)
tests: capture logging with context manager (#1342)
Consistent message counter log messages for all kind of bots (#1278)
pymongo 3.0 deprecates used insert method (#1063)
pymongo >= 3.5: authentication changes (#1062)
Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
n6 parser: mapping is modified within each run (#905)
reverse DNS: Only first record is used (#877)
Corrupt dump files when interrupted during writing (#870)

Assets 2

Releases: certtools/intelmq

Bugfix release

Core

Development

Bots

Collectors

Parsers

Experts

Outputs

Documentation

Tests

Tools

Contrib

Known issues

Uh oh!

Feature Release

Core

Bots

Collectors

Parsers

Experts

Outputs

Documentation

Packaging

Tests

Tools

Contrib

Known issues

Uh oh!

Bugfix release

Requirements

Core

Harmonization

Bots

Collectors

Parsers

Experts

Outputs

Documentation

Packaging

Tests

Tools

Contrib

Known issues

Uh oh!

Bugfix release

Core

Bots

Collectors

Parsers

Experts

Outputs

Documentation

Packaging

Tests

Tools

Known issues

Uh oh!

2.1.1

Configuration

Core

Bots

Parsers

Experts

Outputs

Documentation

Tests

Tools

Known issues

Uh oh!

2.1.0

Core

Harmonization

Bots

Collectors

Parsers

Experts

Outputs

Documentation

Tests