Skip to content

Releases: certtools/intelmq

Bugfix release 1.2.0

25 Mar 14:41
1.1.2
6abc7ac

Choose a tag to compare

View all tags
Bugfix release 1.2.0

Installation instructions:
https://github.com/certtools/intelmq/blob/1.1.2/docs/INSTALL.md
Upgrade instructions:
https://github.com/certtools/intelmq/blob/1.1.2/docs/UPGRADING.md

Core

  • intelmq.lib.bot:
    • Bot.__handle_sighup: Handle exceptions in shutdown method of bots.

Harmonization

  • FQDN: Disallow : in FQDN values to prevent values like '10.0.0.1:8080' (#1235).

Bots

Collectors

  • intelmq.bots.collectors.stomp.collector
    • Fix name of shutdown method, was ineffective in the past.
    • Ignore NotConnectedException errors on disconnect during shutdown.
  • intelmq.bots.collectors.mail.collector_mail_url: Decode body if it is bytes (#1367).
  • intelmq.bots.collectors.tcp.collector: Timeout added. More stable version.

Parsers

  • intelmq.bots.parsers.shadowserver:
    • Add support for the Amplification-DDoS-Victim, HTTP-Scanners, ICS-Scanners and Accessible-Ubiquiti-Discovery-Service feeds (#1368, #1383)
  • intelmq.bots.parsers.microsoft.parser_ctip:
    • Workaround for mis-formatted data in networkdestinationipv4 field (since 2019-03-14).
    • Ignore "hostname" ("destination.fqdn") if it contains invalid data.
  • intelmq.bots.parsers.shodan.parser:
    • In minimal_mode:
      • Fix the parsing, previously only source.geolocation.cc and extra.shodan was correctly filled with information.
      • Add a classification.type = 'other' to all events.
      • Added tests for this mode.
    • Normal mode:
      • Fix the parsing of timestamp to `time.source in the normal mode, previously no timezone information has been added and thus every event raised an exception.
      • ISAKMP: Ignore isakmp.aggressive, as the content is same as isakmp or less.
  • intelmq.bots.parsers.abusech.parser_ip: Re-structure the bot and support new format of the changed "Feodo Tracker Domains" feed.
  • intelmq.bots.parsers.n6.parser:
    • Add parsing for fields "confidence", "expires" and "source".
    • Add support for type "bl-other" (category "other").

Experts

  • intelmq.bots.experts.sieve.expert: Fix key definition to allow field names with numbers (malware.hash.md5/sha1, #1371).

Outputs

  • intelmq.bots.outputs.tcp.output: Timeout added. When no separator used, awaits that every message is acknowledged by a simple "Ok" string to ensure more stability.

Documentation

  • Install: Update operating system versions
  • Sieve Expert: Fix elsif -> elif.
  • Rephrase the description of time.* fields.
  • Feeds: New URL and format of the "Feodo Tracker IPs" feed. "Feodo Tracker Domains" has been discontinued.

Packaging

Tests

  • Add missing __init__.py files in 4 bot's test directories. Previously these tests have never been executed.
  • intelmq.lib.test: Allow bot test class names with an arbitrary postfix separated by an underscore. E.g. TestShodanParserBot_minimal.

Tools

  • intelmqctl:
    • status: Show commandline differences if a program with the expected PID could be found, but they do not match (previous output was None).
    • Use logging level from defauls configuration if possible, otherwise intelmq's internal default. Previously, DEBUG was used unconditionally.

Known issues

  • Bots started with IntelMQ-Manager stop when the webserver is restarted (#952).
  • stomp collector bot constantly uses 100% of CPU (#1364).
Loading

Bugfix release 1.1.1

15 Jan 16:18
1.1.1
22b635f

Choose a tag to compare

View all tags
Bugfix release 1.1.1

Installation instructions:
https://github.com/certtools/intelmq/blob/1.1.1/docs/INSTALL.md
Upgrade instructions:
https://github.com/certtools/intelmq/blob/1.1.1/docs/UPGRADING.md

Core

  • lib/harmonization.py: Change parse_utc_isoformat of DateTime class from private to public (related to #1322).
  • lib/utils.py: Add new function object_pair_hook_bots.
  • lib.bot.py:
    • ParserBot's method recover_line_csv now also handles given tempdata.
    • Bot.acknowledge_message() deletes __current_message to free the memory, saves memory in idling parsers with big reports.
    • start(): Warn once per run if error_dump_message is set to false.
    • Bot.start(), ParserBot.process(): If errors happen on bots without destination pipeline, the on_error path has been queried and lead to an exception being raised.
    • start(): If error_procedure is pass and on pipeline errors, the bot retries forever (#1333).
  • lib/message.py:
    • Fix add('extra', ..., overwrite=True): old extra fields have not been deleted previously (#1335).
    • Do not ignore empty or ignored (as defined in _IGNORED_VALUES) values of extra.* fields for backwards compatibility (#1335).
  • lib/pipeline.py (Redis.receive): Wait in 1s steps if redis is busy loading its snapshot from disk (#1334).

Default configuration

  • Set error_dump_message to true by default in defaults.conf.
  • Fixed typo in defaults.conf: proccess_manager -> process_manager

Development

  • bin/rewrite_config_files.py: Fix ordering of BOTS file (#1327).

Harmonization

Update to 2018-09-26 version. New values are per taxonomy:

  • Taxonomy 'intrusions':
    • "application-compromise"
    • "burglary"
    • "privileged-account-compromise"
    • "unprivileged-account-compromise"
  • Taxonomy 'fraud':
    • "copyright"
    • "masquerade"
    • "unauthorized-use-of-resources"
  • Taxonomy 'information content security':
    • "data-loss"
  • Taxonomy 'vulnerable':
    • "ddos-amplifier"
    • "information-disclosure"
    • "potentially-unwanted-accessible"
    • "vulnerable-system"
    • "weak-crypto"
  • Taxonomy 'availability':
    • "dos"
    • "outage"
    • "sabotage"
  • Taxonomy 'abusive-content':
    • "harmful-speech"
    • "violence"
  • Taxonomy 'malicious code':
    • "malware-distribution"
  • Taxonomy 'information-gathering':
    • "social-engineering"
    • "sniffing"
  • Taxonomy 'information content security':
    • "Unauthorised-information-access"
    • "Unauthorised-information-modification"

Bots

Collectors

  • intelmq.bots.collectors.http.collector_http:
    • Fix parameter name extract_files in BOTS (#1331).
    • Fix handling of extract_files parameter if the value is an empty string.
    • Handle not installed dependency library requests gracefully.
    • Explain extract_files parameter in docs and use a sane default in BOTS file.
  • intelmq.bots.collectors.mail.collector_mail_url:
    • Handle HTTP status codes != 2xx the same as HTTP timeouts: No exception, but graceful handling.
    • Handle HTTP errors (bad status code and timeouts) with error_procedure == 'pass' but marking the mail as read and logging the error.
    • Handle not installed dependency library requests gracefully.
  • intelmq.bots.collectors.http.collector_http_stream:
    • Handle not installed dependency library requests gracefully.
  • intelmq.bots.collectors.microsoft.collector_interflow:
    • Handle not installed dependency library requests gracefully.
  • intelmq.bots.collectors.rt.collector_rt:
    • Handle not installed dependency library requests gracefully.
  • added intelmq.bots.collectors.shodan.collector_stream for collecting shodan stream data (#1096).
    • Correctly check the version of the shodan library, it resulted in wrong comparisons with two digit numbers.
  • intelmq.bots.collectors.microsoft.collector_interflow:
    • Add check if Cache's TTL is big enough compared to not_older_than and throw an error otherwise.

Parsers

  • intelmq.bots.parsers.misp: Fix Object attribute (#1318).
  • intelmq.bots.parsers.cymru.parser_cap_program:
    • Add support for new format (extra data about botnet of 'bots').
    • Handle AS number 0.
  • intelmq.bots.parsers.shadowserver:
    • Spam URL reports: remove src_naics, src_sic columns.
    • fix parsing of 'spam' events in ShadowServer's 'Botnet Drone Hadoop' Report (#1271).
    • Add support in parser to ignore some columns in config file by using False as intelmq key.
    • Add support for the Outdated-DNSSEC-Key and Outdated-DNSSEC-Key-IPv6 feeds.
    • Add support for the Accessible-Rsync feed.
    • Document support for the Open-LDAP-TCP feed.
    • Add support for Accessible-HTTP and Open-DB2-Discovery-Service (#1349).
    • Add support for Accessible-AFP (#1351).
    • Add support for Darknet (#1353).
  • intelmq.bots.parsers.generic.parser_csv: If the skip_header parameter was set to True, the header was not part of the raw field as returned by the recover_line method. The header is now saved and handled correctly by the fixed recovery method.
  • intelmq.bots.parsers.cleanmx.parser: Use field first instead of firsttime for time.source (#1329, #1348).
  • intelmq.bots.parsers.twitter.parser: Support for url-normalize >= 1.4.1 and recommend it. Added new optional parameter default_scheme, passed to url-normalize (#1356).

Experts

  • intelmq.bots.experts.national_cert_contact_certat.expert:
    • Handle not installed dependency library requests gracefully.
  • intelmq.bots.experts.ripencc_abuse_contact.expert:
    • Handle not installed dependency library requests gracefully.
  • intelmq.bots.experts.sieve.expert:
    • check method: Add missing of the harmonization for the check, caused an error for every check.
    • Add text and more context to error messages.
    • README: Fix 'modify' to 'update' (#1340).
    • Handle empty rules file (#1343).
  • intelmq.bots.experts.idea.expert: Add mappings for new harmonization classification.type values, see above.

Outputs

  • intelmq.bots.outputs.redis:
    • Fix sending password to redis server.
    • Fix for redis-py >= 3.0.0: Convert Event to string explicitly (#1354).
    • Use Redis class instead of deprecated StrictRedis for redis-py >= 3.0.0 (#1355).
  • intelmq.bots.outputs.mongodb:
    • New parameter replacement_char (default: '_') for non-hierarchical output as dots in key names are not allowed (#1324, #1322).
    • Save value of fields time.observation and time.source as native datetime object, not as string (#1322).
  • intelmq.bots.outputs.restapi.output:
    • Handle not installed dependency library requests gracefully.

Documentation

  • FAQ
    • Explanation and solution on orphaned queues.
    • Section on how and why to remove raw data.
  • Add or fix the tables of contents for all documentation files.
  • Feeds:
    • Fix Autoshun Feed URL (#1325).
    • Add parameters name and provider to intelmq/etc/feeds.yaml, docs/Feeds.md and intelmq/bots/BOTS (#1321).
  • Add SECURITY.md file.

Packaging

  • Change the maintainer from Sasche Wilde to Sebastian Wagner (#1320).

Tests

  • intelmq.tests.lib.test_bot: Skip test_logging_level_other on python 3.7 because of unclear behavior related to copies of loggers (#1269).
  • intelmq.tests.bots.collectors.rt.test_collector: Remove test because the REST interface of the instance has been closed (see also python-rt/python-rt#28).

Tools

  • intelmqctl check: Shows more detailed information on orphaned queues.
  • intelmqctl:
    • Correctly determine the status of bots started with intelmqctl run.
    • Fix output of errors during bot status determination, making it compatible to IntelMQ Manager.
    • check subcommand: Show bot ID for messages also in JSON output.
    • run [bot-id] process -m [message] works also with bots without a configured source pipeline (#1307).

Contrib

  • elasticsearch/elasticmapper: Add tlp field (#1308).
  • feeds-config-generator/intelmq_gen_feeds_conf:
    • Add parameters to write resulting configuration directly to files (#1321).
    • Handle collector's feed.name and feed.provider (#1314).

Known issues

  • Bots started with IntelMQ-Manager stop when the webserver is restarted (#952).
  • Tests: capture logging with context manager (#1342).
  • stomp collector bot constantly uses 100% of CPU (#1364).
Loading

Feature release 1.1.0

05 Sep 12:53
1.1.0
56ca81b

Choose a tag to compare

View all tags
Feature release 1.1.0

Installation instructions:
https://github.com/certtools/intelmq/blob/1.1.0/docs/INSTALL.md
Upgrade instructions:
https://github.com/certtools/intelmq/blob/1.1.0/docs/UPGRADING.md

  • Support for Python 3.3 has been dropped in IntelMQ and some dependencies of it. Python 3.3 reached its end of life and Python 3.4 or newer is a hard requirement now.
  • The list of feeds docs/Feeds.md has now a machine-readable equivalent YAML file in intelmq/etc/feeds.yaml
    A tool to convert from yaml to md has been added.

Tools

  • intelmq_gen_feeds_docs addded to bin directory, allows generating the Feeds.md documentation file from feeds.yaml
  • intelmq_gen_docs merges both intelmq_gen_feeds_docs and intelmq_gen_harm_docs in one file and automatically updates the documentation files.

intelmqctl

  • intelmqctl start prints the bot's last error messages if the bot failed to start (#1021).
  • intelmqctl start message "is running" is printed every time. (Until now, it wasn't said when a bot was just starting.)
  • intelmqctl start/stop/restart/reload/status now has a "--group" flag which allows you to specify the group of the bots that should be influenced by the command.
  • intelmqctl check checks for defaults.conf completeness if the shipped file from the package can be found.
  • intelmqctl check shows errors for non-importable bots.
  • intelmqctl list bots -q only prints the IDs of enabled bots.
  • intelmqctl list queues-and-status prints both queues and bots statuses (so that it can be used in eg. intelmq-manager).
  • intelmqctl run parameter for showing a sent message.
  • intelmqctl run if message is sent to a non-default path, it is printed out.
  • intelmqctl restart bug fix; returned some half-nonsense, now returns return state of start and stop operation in a list (#1226).
  • intelmqctl check: New parameter --no-connections to prevent the command from making connections e.g. to the redis pipeline.s
  • intelmqctl list queues: don't display named paths amongst standard queues.
  • The process status test failed if the PATH did not include the bot executables and the which command failed. Then the proccess's command line could not be compared correctly. The fix warns of this and adds a new status 'unknown' (#1297).

Contrib

  • tool feeds-config-generator to automatically generate the collector and parser runtime and pipeline configurations.
  • malware_name_mapping: Download and convert tool for malware family name mapping has been added.
  • Added a systemd script which creates systemd units for bots (#953).
  • contrib/cron-jobs/update-asn-data, contrib/cron-jobs/update-geoip-data, contrib/cron-jobs/update-tor-nodes: Errors produce proper output.

Core

  • lib/bot
    • use SIGTERM instead of SIGINT to stop bots (#981).
    • Bots can specify a static method check(parameters) which can perform individual checks specific to the bot.
      These functions will be called by intelmqctl check if the bot is configured with the given parameters
    • top level bot parameters (description, group, module, name) are exposed as members of the class.
    • The parameter feed for collectors is deprecated for 2.0 and has been replaced by the more consistent name (#1144).
    • bug: allow path parameter for CollectorBot class.
    • Handle errors better when the logger could not be initialized.
    • ParserBot:
      • For the csv parsing methods, ParserBot.csv_params is now used for all these methods.
      • ParserBot.parse_csv_dict now saves the field names in ParserBot.csv_fieldnames.
      • ParserBot.parse_csv_dict now saves the raw current line in ParserBot.current_line.
      • ParserBot.recover_line_csv_dict now uses the raw current line.
  • lib/message:
    • Subitems in fields of type JSONDict (see below) can be accessed directly. E.g. you can do:
      event['extra.foo'] = 'bar'
      event['extra.foo'] # gives 'bar'
      It is still possible to set and get the field as whole, however this may be removed or changed in the future:
      event['extra'] = '{"foo": "bar"}'
      event['extra'] # gives '{"foo": "bar"}'
      "Old" bots and configurations compatible with 1.0.x do still work.
      Also, the extra field is now properly exploded when exporting events, analogous to all other fields.
      The in operator works now for both - the old and the new - behavior.
    • Message.add: The parameter overwrite accepts now three different values: True, False and None (new).
      True: An existing value will be overwritten
      False: An existing value will not be overwritten (previously an exception has been raised when the value was given).
      None (default): If the value exists an KeyExists exception is thrown (previously the same as False).
      This allows shorter code in the bots, as an 'overwrite' configuration parameter can be directly passed to the function.
    • The message class has now the possibility to return a default value for non-exisiting fields, see Message.set_default_value.
    • Message.get behaves the same like Message.__getitem__ (#1305).
  • Add RewindableFileHandle to utils making handling of CSV files more easy (optionally)
  • lib/pipeline:
    • you may now define more than one destination queues path the bot should pass the message to, see Pipelines (#1088, #1190).
    • the special path "_on_error" can be used to pass messages to different queues in case of processing errors (#1133).
  • lib/harmonization: Accept AS prefix for ASN values (automatically stripped).

Bots

  • Removed print statements from various bots.
  • Replaced various occurences of self.logger.error() + self.stop() with raise ValueError.

Collectors

  • bots.collectors.mail:
    • New parameters; sent_from: filter messages by sender, sent_to: filter messages by recipient
    • More debug logs
  • bots.collectors.n6.collector_stomp: renamed to bots.collectors.stomp.collector (#716)
  • bots.collectors.rt:
    • New parameter search_requestor to search for field Requestor.
    • Empty strings and null as value for search parameters are ignored.
    • Empty parameters attachment_regex and url_regex handled.
  • bots.collectors.http.collector_http: Ability to optionally use the current time in parameter http_url, added parameter http_url_formatting.
  • bots.collectors.stomp.collector: Heartbeat timeout is now logged with log level info instead of warning.
  • added intelmq.bots.collectors.twitter.collector_twitter
  • added intelmq.bots.collectors.tcp.collector that can be bound to another IntelMQ instance by a TCP output
  • bots.collectors.microsoft.collector_interflow: added for MS interflow API
    • Automatic ungzipping for .gz files.
  • added intelmq.bots.collectors.calidog.collector_certstream for collecting certstream data (#1120).
  • added intelmq.bots.collectors.shodan.collector_stream for collecting shodan stream data (#1096).
    • Add proxy support.
    • Fix handling of parameter countries.

Parsers

  • bots.parsers.shadowserver:
    • changed feednames . Please refer to it's README for the exact changes.
    • If the conversion function fails for a line, an error is raised and the offending line will be handled according to the error handling configuration.
      Previously errors like these were only logged and ignored otherwise.
    • add support for the feeds
      • Accessible-Hadoop (#1231)
      • Accessible ADB (#1285)
    • Remove deprecated parameter override, use overwrite instead (#1071).
    • The raw values now are exactly the input with quotes unchanged, the ParserBot methods are now used directly (#1011).
  • The Generic CSV Parser bots.parsers.generic.parser_csv:
    • It is possible to filter the data before processing them using the new parameters filter_type and filter_text.
    • It is possible to specify multiple columns using | character in parameter columns.
    • The parameter time_format now supports 'epoch_millis' for seconds since the Epoch, milliseconds are supported but not used.
  • renamed bots.parsers.cymru_full_bogons.parser to bots.parsers.cymru.parser_full_bogons, compatibility shim will be removed in version 2.0
  • added bots.parsers.cymru.parser_cap_program
  • added intelmq.bots.parsers.zoneh.parser for ZoneH feeds
  • added intelmq.bots.parsers.sucuri.parser
  • added intelmq.bots.parsers.malwareurl.parser
  • added intelmq.bots.parsers.threatminer.parser
  • added intelmq.bots.parsers.webinspektor.parser
  • added intelmq.bots.parsers.twitter.parser
  • added intelmq.bots.parsers.microsoft.parser_ctip
    • ignore the invalid IP '0.0.0.0' for the destination
    • fix the raw/dumped messages, did not contain the paling list previously.
    • use the new harmonization field tlp instead of extra.tlp.
  • bots.parsers.alienvault.parser_otx: Save TLP data in the new harmonization field tlp.
  • added intelmq.bots.parsers.openphish.parser_commercial
  • added intelmq.bots.parsers.microsoft.parser_bingmurls
  • added intelmq.bots.parsers.calidog.parser_certstream for parsing certstream data (#1120).
  • added intelmq.bots.parsers.shodan.parser for parsing shodan data (#1096).
  • change the classification type from 'botnet drone' to infected system' in various parses.
  • intelmq.bots.parsers.spamhaus.parser_cert: Added support for all known bot types.

Experts

  • Added sieve expert for filtering and modifying events (#1083)
  • capable of distributing the event to appropriate named queues
  • bots.experts.modify
    • default rulesets: all malware name mappings have been migrated to the Malware Name Mapping repository ruleset. See the new added contrib tool for download and conversion.
    • new parameter case_sensitive (default: True)
  • Added wait expert for sleeping
  • A...
Read more
Loading

Maintenance release 1.0.6

03 Sep 08:59
1.0.6
4ca8aea

Choose a tag to compare

View all tags
Maintenance release 1.0.6

Installation instructions:
https://github.com/certtools/intelmq/blob/1.0.6/docs/INSTALL.md
Upgrade instructions:
https://github.com/certtools/intelmq/blob/1.0.6/docs/UPGRADING.md

Bots

Collectors

  • bots.collectors.rt.collector_rt: Log ticket id for downloaded reports.

Parsers

  • bots.parsers.shadowserver:
    • if required fields do not exist in data, an exception is raised, so the line will be dumped and not further processed.
    • fix a bug in the parsing of column cipher_suite in ssl poodle reports (#1288).

Experts

  • Reverse DNS Expert: ignore all invalid results and use first valid one (#1264).
  • intelmq/bots/experts/tor_nodes/update-tor-nodes: Use check.torproject.org as source as internet2.us is down (#1289).

Outputs

  • bots.output.amqptopic:
    • The default exchange must not be declared (#1295).
    • Unencodable characters are prepended by backslashes by default. Otherwise Unicode characters can't be encoded and sent (#1296).
    • Gracefully close AMQP connection on shutdown of bot.

Documentation

  • Bots: document redis cache parameters.
  • Installation documentation: Ubuntu needs universe repositories.

Packaging

  • Dropped support for Ubuntu 17.10, it reached its End of Life as of 2018-07-19.

Tests

  • Drop tests for Python 3.3 for the mode with all requirements, as some optional dependencies do not support Python 3.3 anymore.
  • lib.test: Add parameter compare_raw (default: True) to assertMessageEqual, to optionally skip the comparison of the raw field.
  • Add tests for RT collector.
  • Add tests for Shadowserver Parser:
    • SSL Poodle Reports.
    • Helper functions.

Tools

  • intelmqctl list now sorts the output of bots and queues (#1262).
  • intelmqctl: Correctly handle the corner cases with collectors and outputs for getting/sending messages in the bot debugger (#1263).
  • intelmqdump: fix ordering of dumps in a file in runtime. All operations are applied to a sorted list (#1280).

Contrib

  • cron-jobs/update-tor-nodes: Use check.torproject.org as source as internet2.us is down (#1289).
Loading

1.1.0 Release Candidate 2

14 Aug 10:48
1.1.0rc2
93b3b18

Choose a tag to compare

View all tags
1.1.0 Release Candidate 2 Pre-release
Pre-release
  • Support for Python 3.3 has been dropped in IntelMQ and some dependencies of it. Python 3.3 reached its end of life and Python 3.4 or newer is a hard requirement now.
  • The list of feeds docs/Feeds.md has now a machine-readable equivalent YAML file in intelmq/etc/feeds.yaml
    A tool to convert from yaml to md has been added.

Tools

  • intelmq_gen_feeds_docs addded to bin directory, allows generating the Feeds.md documentation file from feeds.yaml
  • intelmq_gen_docs merges both intelmq_gen_feeds_docs and intelmq_gen_harm_docs in one file and automatically updates the documentation files.

intelmqctl

  • intelmqctl start prints the bot's last error messages if the bot failed to start (#1021).
  • intelmqctl start message "is running" is printed every time. (Until now, it wasn't said when a bot was just starting.)
  • intelmqctl start/stop/restart/reload/status now has a "--group" flag which allows you to specify the group of the bots that should be influenced by the command.
  • intelmqctl check checks for defaults.conf completeness if the shipped file from the package can be found.
  • intelmqctl check shows errors for non-importable bots.
  • intelmqctl list bots -q only prints the IDs of enabled bots.
  • intelmqctl list queues-and-status prints both queues and bots statuses (so that it can be used in eg. intelmq-manager).
  • intelmqctl run parameter for showing a sent message.
  • intelmqctl run if message is sent to a non-default path, it is printed out.
  • intelmqctl restart bug fix; returned some half-nonsense, now returns return state of start and stop operation in a list (#1226).
  • intelmqctl check: New parameter --no-connections to prevent the command from making connections e.g. to the redis pipeline.s
  • intelmqctl list queues: don't display named paths amongst standard queues.

Contrib

  • tool feeds-config-generator to automatically generate the collector and parser runtime and pipeline configurations.
  • malware_name_mapping: Download and convert tool for malware family name mapping has been added.
  • Added a systemd script which creates systemd units for bots (#953).
  • contrib/cron-jobs/update-asn-data, contrib/cron-jobs/update-geoip-data, contrib/cron-jobs/update-tor-nodes: Errors produce proper output.

Core

  • lib/bot
    • use SIGTERM instead of SIGINT to stop bots (#981).
    • Bots can specify a static method check(parameters) which can perform individual checks specific to the bot.
      These functions will be called by intelmqctl check if the bot is configured with the given parameters
    • top level bot parameters (description, group, module, name) are exposed as members of the class.
    • The parameter feed for collectors is deprecated for 2.0 and has been replaced by the more consistent name (#1144).
    • bug: allow path parameter for CollectorBot class.
    • Handle errors better when the logger could not be initialized.
    • ParserBot:
      • For the csv parsing methods, ParserBot.csv_params is now used for all these methods.
      • ParserBot.parse_csv_dict now saves the field names in ParserBot.csv_fieldnames.
      • ParserBot.parse_csv_dict now saves the raw current line in ParserBot.current_line.
      • ParserBot.recover_line_csv_dict now uses the raw current line.
  • lib/message:
    • Subitems in fields of type JSONDict (see below) can be accessed directly. E.g. you can do:
      event['extra.foo'] = 'bar'
      event['extra.foo'] # gives 'bar'
      It is still possible to set and get the field as whole, however this may be removed or changed in the future:
      event['extra'] = '{"foo": "bar"}'
      event['extra'] # gives '{"foo": "bar"}'
      "Old" bots and configurations compatible with 1.0.x do still work.
      Also, the extra field is now properly exploded when exporting events, analogous to all other fields.
      The in operator works now for both - the old and the new - behavior.
    • Message.add: The parameter overwrite accepts now three different values: True, False and None (new).
      True: An existing value will be overwritten
      False: An existing value will not be overwritten (previously an exception has been raised when the value was given).
      None (default): If the value exists an KeyExists exception is thrown (previously the same as False).
      This allows shorter code in the bots, as an 'overwrite' configuration parameter can be directly passed to the function.
    • The message class has now the possibility to return a default value for non-exisiting fields, see Message.set_default_value.
  • Add RewindableFileHandle to utils making handling of CSV files more easy (optionally)
  • lib/pipeline:
    • you may now define more than one destination queues path the bot should pass the message to, see Pipelines (#1088, #1190).
    • the special path "_on_error" can be used to pass messages to different queues in case of processing errors (#1133).
  • lib/harmonization: Accept AS prefix for ASN values (automatically stripped).

Bots

  • Removed print statements from various bots.
  • Replaced various occurences of self.logger.error() + self.stop() with raise ValueError.

Collectors

  • bots.collectors.mail:
    • New parameters; sent_from: filter messages by sender, sent_to: filter messages by recipient
    • More debug logs
  • bots.collectors.n6.collector_stomp: renamed to bots.collectors.stomp.collector (#716)
  • bots.collectors.rt:
    • New parameter search_requestor to search for field Requestor.
    • Empty strings and null as value for search parameters are ignored.
    • Empty parameters attachment_regex and url_regex handled.
  • bots.collectors.http.collector_http: Ability to optionally use the current time in parameter http_url, added parameter http_url_formatting.
  • bots.collectors.stomp.collector: Heartbeat timeout is now logged with log level info instead of warning.
  • added intelmq.bots.collectors.twitter.collector_twitter
  • added intelmq.bots.collectors.tcp.collector that can be bound to another IntelMQ instance by a TCP output
  • bots.collectors.microsoft.collector_interflow: added for MS interflow API
    • Automatic ungzipping for .gz files.
  • added intelmq.bots.collectors.calidog.collector_certstream for collecting certstream data (#1120).
  • added intelmq.bots.collectors.shodan.collector_stream for collecting shodan stream data (#1096).
    • Add proxy support.
    • Fix handling of parameter countries.

Parsers

  • bots.parsers.shadowserver:
    • changed feednames . Please refer to it's README for the exact changes.
    • If the conversion function fails for a line, an error is raised and the offending line will be handled according to the error handling configuration.
      Previously errors like these were only logged and ignored otherwise.
    • add support for the feeds
      • Accessible-Hadoop (#1231)
      • Accessible ADB (#1285)
    • Remove deprecated parameter override, use overwrite instead (#1071).
    • The raw values now are exactly the input with quotes unchanged, the ParserBot methods are now used directly (#1011).
  • The Generic CSV Parser bots.parsers.generic.parser_csv:
    • It is possible to filter the data before processing them using the new parameters filter_type and filter_text.
    • It is possible to specify multiple columns using | character in parameter columns.
    • The parameter time_format now supports 'epoch_millis' for seconds since the Epoch, milliseconds are supported but not used.
  • renamed bots.parsers.cymru_full_bogons.parser to bots.parsers.cymru.parser_full_bogons, compatibility shim will be removed in version 2.0
  • added bots.parsers.cymru.parser_cap_program
  • added intelmq.bots.parsers.zoneh.parser for ZoneH feeds
  • added intelmq.bots.parsers.sucuri.parser
  • added intelmq.bots.parsers.malwareurl.parser
  • added intelmq.bots.parsers.threatminer.parser
  • added intelmq.bots.parsers.webinspektor.parser
  • added intelmq.bots.parsers.twitter.parser
  • added intelmq.bots.parsers.microsoft.parser_ctip
    • ignore the invalid IP '0.0.0.0' for the destination
    • fix the raw/dumped messages, did not contain the paling list previously.
    • use the new harmonization field tlp instead of extra.tlp.
  • bots.parsers.alienvault.parser_otx: Save TLP data in the new harmonization field tlp.
  • added intelmq.bots.parsers.openphish.parser_commercial
  • added intelmq.bots.parsers.microsoft.parser_bingmurls
  • added intelmq.bots.parsers.calidog.parser_certstream for parsing certstream data (#1120).
  • added intelmq.bots.parsers.shodan.parser for parsing shodan data (#1096).
  • change the classification type from 'botnet drone' to infected system' in various parses.
  • intelmq.bots.parsers.spamhaus.parser_cert: Added support for all known bot types.

Experts

  • Added sieve expert for filtering and modifying events (#1083)
  • capable of distributing the event to appropriate named queues
  • bots.experts.modify
    • default rulesets: all malware name mappings have been migrated to the Malware Name Mapping repository ruleset. See the new added contrib tool for download and conversion.
    • new parameter case_sensitive (default: True)
  • Added wait expert for sleeping
  • Added domain suffix expert to extract the TLD/Suffix from a domain name.
  • bots.experts.maxmind_geoip: New (optional) parameter overwrite, by default false. The current default was to overwrite!
  • intelmq.bots.experts.ripencc_abuse_contact: Extend deprecated parameter compatibility query_ripe_stat until 2.0 because of a logic bug in the compatibility code, use query_ripe_stat_asn and query_ripe_stat_ip instead (#1071, #1291).
  • intelmq/bots/experts/asn_lookup/update-asn-data: E...
Read more
Loading

1.1.0 Release candidate 1

28 Jun 14:28
1.1.0rc1
4ed19dc

Choose a tag to compare

View all tags
1.1.0 Release candidate 1 Pre-release
Pre-release

Installation instructions:
https://github.com/certtools/intelmq/blob/1.1.0rc1/docs/INSTALL.md
Upgrade instructions:
https://github.com/certtools/intelmq/blob/1.1.0rc1/docs/UPGRADING.md

  • Support for Python 3.3 has been dropped, it reached its end of life.
  • The list of feeds docs/Feeds.md has now a machine-readable equivalent YAML file in intelmq/etc/feeds.yaml
    A tool to convert from yaml to md has been added.

Tools

  • intelmq_gen_feeds_docs add to bin directory, allows generating the Feeds.md documentation file from feeds.yaml
  • intelmq_gen_docs merges both intelmq_gen_feeds_docs and intelmq_gen_harm_docs in one file and automatically updates the documentation files.

intelmqctl

  • intelmqctl start prints the bot's last error messages if the bot failed to start (#1021).
  • intelmqctl start message "is running" is printed every time. (Until now, it wasn't said when a bot was just starting.)
  • intelmqctl start/stop/restart/reload/status now have a "--group" flag which allows you to specify the group of the bots that should be influenced by the command.
  • intelmqctl check checks for defaults.conf completeness if the shipped file from the package can be found.
  • intelmqctl check shows errors for non-importable bots.
  • intelmqctl list bots -q only prints the IDs of enabled bots.
  • intelmqctl list queues-and-status prints both queues and bots statuses (so that it can be used in eg. intelmq-manager).
  • intelmqctl run parameter for showing a sent message.
  • intelmqctl run if message is sent to a non-default path, it is printed out.
  • intelmqctl restart bug fix; returned some half-nonsense, now returns return state of start and stop operation in a list (#1226).
  • intelmqctl check: New parameter --no-connections to prevent the command from making connections e.g. to the redis pipeline.s

Contrib

  • tool feeds-config-generator to automatically generate the collector and parser runtime and pipeline configurations.
  • malware_name_mapping: Download and convert tool for malware family name mapping has been added.
  • Added a systemd script which creates systemd units for bots (#953).

Core

  • lib/bot
    • use SIGTERM instead of SIGINT to stop bots (#981).
    • Bots can specify a static method check(parameters) which can perform individual checks specific to the bot.
      These functions will be called by intelmqctl check if the bot is configured with the given parameters
    • top level bot parameters (description, group, module, name) are exposed as members of the class.
    • The parameter feed for collectors is deprecated for 2.0 and has been replaced by the more consistent name (#1144).
  • lib/message:
    • Subitems in fields of type JSONDict (see below) can be accessed directly. E.g. you can do:
      event['extra.foo'] = 'bar'
      event['extra.foo'] # gives 'bar'
      It is still possible to set and get the field as whole, however this may be removed or changed in the future:
      event['extra'] = '{"foo": "bar"}'
      event['extra'] # gives '{"foo": "bar"}'
      "Old" bots and configurations compatible with 1.0.x do still work.
      Also, the extra field is now properly exploded when exporting events, analogous to all other fields.
    • Message.add: The parameter overwrite accepts now three different values: True, False and None (new).
      True: An existing value will be overwritten
      False: An existing value will not be overwritten (previously an exception has been raised when the value was given).
      None (default): If the value exists an KeyExists exception is thrown (previously the same as False).
      This allows shorter code in the bots, as an 'overwrite' configuration parameter can be directly passed to the function.
    • The message class has now the possibility to return a default value for non-exisiting fields, see Message.set_default_value.
  • Add RewindableFileHandle to utils making handling of CSV files more easy (optionally)
  • lib/pipeline:
    • you may now define more than one destination queues path the bot should pass the message to, see Pipelines (#1088, #1190).
    • the special path "_on_error" can be used to pass messages to different queues in case of processing errors (#1133).
  • lib/harmonization: Accept AS prefix for ASN values (automatically stripped).

Bots

Collectors

  • bots.collectors.mail:
    • New parameters; sent_from: filter messages by sender, sent_to: filter messages by recipient
    • More debug logs
  • bots.collectors.n6.collector_stomp: renamed to bots.collectors.stomp.collector (#716)
  • bots.collectors.rt:
    • New parameter search_requestor to search for field Requestor.
    • Empty strings and null as value for search parameters are ignored.
    • Empty parameters attachment_regex and url_regex handled.
  • bots.collectors.http.collector_http: Ability to optionally use the current time in parameter http_url, added parameter http_url_formatting.
  • bots.collectors.stomp.collector: Heartbeat timeout is now logged with log level info instead of warning.
  • added intelmq.bots.collectors.twitter.collector_twitter
  • added intelmq.bots.collectors.tcp.collector that can be bound to another IntelMQ instance by a TCP output
  • bots.collectors.microsoft.collector_interflow: added for MS interflow API
    • Automatic ungzipping for .gz files.
  • added intelmq.bots.collectors.calidog.collector_certstream for collecting certstream data (#1120).
  • added intelmq.bots.collectors.shodan.collector_stream for collecting shodan stream data (#1096).

Parsers

  • bots.parsers.shadowserver:
    • changed feednames . Please refer to it's README for the exact changes.
    • If the conversion function fails for a line, an error is raised and the offending line will be handled according to the error handling configuration.
      Previously errors like these were only logged and ignored otherwise.
    • add support for the feed Accessible-Hadoop
  • The Generic CSV Parser bots.parsers.generic.parser_csv:
    • It is possible to filter the data before processing them using the new parameters filter_type and filter_text.
    • It is possible to specify multiple columns using | character in parameter columns.
    • The parameter time_format now supports 'epoch_millis' for seconds since the Epoch, milliseconds are supported but not used.
  • renamed bots.parsers.cymru_full_bogons.parser to bots.parsers.cymru.parser_full_bogons, compatibility shim will be removed in version 2.0
  • added bots.parsers.cymru.parser_cap_program
  • added intelmq.bots.parsers.zoneh.parser for ZoneH feeds
  • added intelmq.bots.parsers.sucuri.parser
  • added intelmq.bots.parsers.malwareurl.parser
  • added intelmq.bots.parsers.threatminer.parser
  • added intelmq.bots.parsers.webinspektor.parser
  • added intelmq.bots.parsers.twitter.parser
  • added intelmq.bots.parsers.microsoft.parser_ctip
    • ignore the invalid IP '0.0.0.0' for the destination
    • fix the raw/dumped messages, did not contain the paling list previously.
    • use the new harmonization field tlp instead of extra.tlp.
  • bots.parsers.alienvault.parser_otx: Save TLP data in the new harmonization field tlp.
  • added intelmq.bots.parsers.openphish.parser_commercial
  • added intelmq.bots.parsers.microsoft.parser_bingmurls
  • added intelmq.bots.parsers.calidog.parser_certstream for parsing certstream data (#1120).
  • added intelmq.bots.parsers.shodan.parser for parsing shodan data (#1096).

Experts

  • Added sieve expert for filtering and modifying events (#1083)
  • capable of distributing the event to appropriate named queues
  • bots.experts.modify
    • default rulesets: all malware name mappings have been migrated to the Malware Name Mapping repository ruleset. See the new added contrib tool for download and conversion.
    • new parameter case_sensitive (default: True)
  • Added wait expert for sleeping
  • Added domain suffix expert to extract the TLD/Suffix from a domain name.
  • bots.experts.maxmind_geoip: New (optional) parameter overwrite, by default false. The current default was to overwrite!

Outputs

  • bots.outputs.file:
    • String formatting can be used for file names.
    • New parameter single_key to only save one field.

Harmonization

  • Renamed JSON to JSONDict and added a new type JSON. JSONDict saves data internally as JSON, but acts like a dictionary. JSON accepts any valid JSON.
  • fixed regex for protocol.transport it previously allowed more values than it should have.
  • New ASN type. Like integer but checks the range.
  • added destination.urlpath and source.urlpath to harmonization.
  • New field tlp for tlp level specification.
    • New TLP type. Allows all four tlp levels, removes 'TLP:' prefix and converts to upper case.
  • Added new classification.type 'vulnerable client'
  • Added (destination|source).domain_suffix to hold the TLD/domain suffix.
  • New allowed value for classification.type: infected system for taxonomy malicious code (#1197).

Requirements

  • Requests is no longer listed as dependency of the core. For depending bots the requirement is noted in their REQUIREMENTS.txt file.

Documentation

  • Use Markdown for README again, as pypi now supports it.

Tests

  • Travis now correctly stops if a requirement could not be installed (#1257).
  • New tests for validating etc/feeds.yaml and bots/BOTS using cerberus and schemes are added (#1166).
  • New test for checking if docs/Feeds.md is up to date with etc/feeds.yaml.

Known bugs

  • bots.experts.sieve does not support textX (#1246).
  • performance degradation for extra fields (#1117).
  • Postgres output: support condensed JSONDicts (#1107).
  • Bots started with IntelMQ-M...
Read more
Loading

1.0.5

22 Jun 08:19
1.0.5
b2fa783

Choose a tag to compare

View all tags
1.0.5

Installation instructions:
https://github.com/certtools/intelmq/blob/1.0.5/docs/INSTALL.md
Upgrade instructions:
https://github.com/certtools/intelmq/blob/1.0.5/docs/UPGRADING.md

Core

  • lib/message: Report() can now create a Report instance from Event instances (#1225).
  • lib/bot:
    • The first word in the log line Processed ... messages since last logging. is now adaptible and set to Forwarded in the existing filtering bots (#1237).
    • Kills oneself again after proper shutdown if the bot is XMPP collector or output (#970). Previously these two bots needed two stop commands to get actually stopped.
  • lib/utils: log: set the name of the py.warnings logger to the bot name (#1184).

Bots

Collectors

  • bots.collectors.mail.collector_mail_url: handle empty downloaded reports (#988).
  • bots.collectos.file.collector_file: handle empty files (#1244).

Parsers

  • Shadowserver parser:
    • SSL FREAK: Remove optional column device_serial and add several new ones.
    • Fixed HTTP URL parsing for multiple feeds (#1243).
  • Spamhaus CERT parser:
    • add support for smtpauth, l_spamlink, pop, imap, rdp, smb, iotscan, proxyget, iotmicrosoftds, automatedtest, ioturl, iotmirai, iotcmd, iotlogin and iotuser (#1254).
    • fix extra.destination.local_port -> extra.source.local_port.

Experts

  • bots.experts.filter: Pre-compile regex at bot initialization.

Tests

  • Ensure that the bots did process all messages (#291).

Tools

  • intelmqctl:
    • intelmqctl run has a new parameter -l --loglevel to overwrite the log level for the run (#1075).
    • intelmqctl run [bot-id] mesage send can now send report messages (#1077).
  • intelmqdump:
    • has now command completion for bot names, actions and queue names in interacive console.
    • automatically converts messages from events to reports if the queue the message is being restored to is the source queue of a parser (#1225).
    • is now capable to read messages in dumps that are dictionaries as opposed to serialized dicts as strings and does not convert them in the show command (#1256).
    • truncated messages are no longer used/saved to the file after being shown (#1255).
    • now again denies recovery of dumps if the corresponding bot is running. The check was broken (#1258).
    • now sorts the dump by the time of the dump. Previously, the list was in random order (#1020).

Known issues

no known issues

Loading

1.0.4

20 Apr 14:35
1.0.4
70f8098

Choose a tag to compare

View all tags
1.0.4

Installation instructions:
https://github.com/certtools/intelmq/blob/1.0.4/docs/INSTALL.md
Upgrade instructions:
https://github.com/certtools/intelmq/blob/1.0.4/docs/UPGRADING.md

General

  • make code style compatible to pycodestyle 2.4.0
  • fixed permissions of some files (they were executable but shouldn't be)

Core

  • lib/harmonization:
    • FQDN validation now handles None correctly (raised an Exception).
    • Fixed several sanitize() methods, the generic sanitation method were called by is_valid, not the sanitize methods (#1219).

Harmonization

Bots

Parsers

  • Shadowserver parser:
    • The fields url and http_url now handle HTTP URL paths and HTTP requests for all feeds (#1204).
    • The conversion function validate_fqdn now handles empty strings correctly.
    • Feed 'drone (hadoop)':
      • Correct validation of field cc_dns, will now only be added as destination.fqdn if correct FQDN, otherwise ignored. Previously this field could be saved in extra containing an IP address.
      • Adding more mappings for added columns.
    • A lot of newly added fields and fixed conversions.
    • Add newly added columns of Ssl-Scan feed to parser
  • Spamhaus CERT parser:
  • fix parsing and classification for bot names 'openrelay', 'iotrdp', 'sshauth', 'telnetauth', 'iotcmd', 'iotuser', 'wpscanner', 'w_wplogin', 'iotscan'
    see the NEWS file - Postgresql section - for all changes.
  • CleanMX phishing parser: handle FQDNs in IP column (#1162).

Experts

  • bots.experts.ripencc_abuse_contact: Add existing parameter mode to BOTS file.

Tools

  • intelmqctl check: Fixed and extended message for 'run_mode' check.
  • intelmqctl start botnet. When using --type json, no non-json information about wrong bots are output because that would confuse eg. intelmq-manager

Tests

  • lib/bot: No dumps will be written during tests (#934).
  • lib/test: Expand regular expression on python version to match pre-releases (debian testing).

Packaging

  • Static data is now included in source tarballs, development files are excluded

Known issues

  • bots.collectors/outputs.xmpp must be killed two times (#970).
  • When running bots with intelmqctl run [bot-id] the log level is always INFO (#1075).
  • intelmqctl run [bot-id] message send [msg] does only support Events, not Reports (#1077).
  • A warning issued by the python warnings module is logged without the bot-id (#1184).
Loading

1.0.3

05 Feb 14:11
1.0.3
cbb63ce

Choose a tag to compare

View all tags
1.0.3

Installation
Upgrade

Contrib

  • logrotate: use sudo for postrotate script
  • cron-jobs: use the scripts in the bots' directories and link them (#1056, #1142)

Core

  • lib.harmonization: Handle idna encoding error in FQDN sanitation (#1175, #1176).
  • lib.bot:
    • Bots stop when redis gives the error "OOM command not allowed when used memory > 'maxmemory'." (#1138).
    • warnings of bots are catched by the logger (#1074, #1113).
    • Fixed exitcodes 0 for graceful shutdowns .
    • better handling of problems with pipeline and especially it's initialization (#1178).
    • All parsers using ParserBot's methods now log the sum of successfully parsed and failed lines at the end of each run (#1161).

Harmonization

  • Rule for harmonization keys is enforced (#1104, #1141).
  • New allowed values for classification.type: tor & leak (see n6 parser below ).

Bots

Collectors

  • bots.collectors.mail.collector_mail_attach: Support attachment file parsing for imbox versions newer than 0.9.5 (#1134).
  • bots.outputs.smtp.output: Fix STARTTLS, threw an exception (#1152, #1153).

Parsers

  • All CSV parsers ignore NULL-bytes now, because the csv-library cannot handle it (#967, #1114).
  • bots.experts.modify default ruleset: changed conficker rule to catch more spellings.
  • bots.parsers.shadowserver.parser: Add Accessible Cisco Smart Install (#1122).
  • bots.parsers.cleanmx.parser: Handle new columns first and last, rewritten for XML feed. See NEWS.md for upgrade instructions (#1131, #1136, #1163).
  • bots.parsers.n6.parser: Fix classification mappings. See NEWS file for changes values (#738, #1127).

Documentation

  • Release.md add release procedure documentation
  • Bots.md: fix example configuration for modify expert

Tools

  • intelmqctl now exits with exit codes > 0 when errors happened or the operation was not successful. Also, the status operation exits with 1, if bots are stopped, but enabled. (#977, #1143)
  • intelmctl check checks for valid run_mode in runtime configuration (#1140).

Tests

  • tests.lib.test_pipeline: Redis tests clear all queues before and after tests (#1086).
  • Repaired debian package build on travis (#1169).
  • Warnings are not allowed by default, an allowed count can be specified (#1129).
  • tests.bots.experts.cymru_whois/abusix: Skipped on travis because of ongoing problems.

Packaging

  • cron jobs: fix paths of executables

Known issues

  • bots.collectors/outputs.xmpp must be killed two times (#970).
  • When running bots with intelmqctl run [bot-id] the log level is always INFO (#1075).
  • intelmqctl run [bot-id] message send [msg] does only support Events, not Reports (#1077).
  • python3 setup.py sdist does not include static files in the resulting tarballs (#1146).
  • bots.parsers.cleanmx.parser: The cleanMX feed may have FQDNs as IPs in rare cases, such lines are dumped (#1162).
Loading

1.0.2

09 Nov 12:14
1.0.2
0dc128f

Choose a tag to compare

View all tags
1.0.2

Installation
Upgrade

Core

  • lib.message.add: parameter force has finally been removed, should have been gone in 1.0.0.rc1 already

Bots

  • collectors.mail.collector_mail_url: Fix bug which prevented marking emails seen due to disconnects from server (#852).
  • parsers.spamhaus.parser_cert: Handle/ignore 'AS?' in feed (#1111)

Packaging

  • The following changes have been in effect for the built packages already since version 1.0.0
  • Support building for more distributions, now supported: CentOS 7, Debian 8 and 9, Fedora 25 and 26, RHEL 7, openSUSE Leap 42.2 and 42.3 and Tumbleweed, Ubuntu 14.04 and 16.04
  • Use LSB-paths for created packages (/etc/intelmq/, /var/lib/intelmq/, /run/intelmq/) (#470). Does does not affect installations with setuptools/pip.
  • Change the debian package format from native to quilt
  • Fix problems in postint and postrm scripts
  • Use systemd-tmpfile for creation of /run/intelmq/

Documentation

  • Add disclaimer on maxmind database in bot documentation and code and the cron-job (#1110)
Loading