chore: Removing RTAs (#4437)

* Delete RTAs

* Delete RTA-related orchestration code

* Drop RTAs from tests

* Remove RTAs from README

* Further cleanup

* Readme update

* Version bump and no more RTAs

* Styling fixes

* Drop RTAs from config files

* Drop `rule-mapping.yaml`

* Bring back event collector / normalizer

* Drop rta mention

* Cleanup rta leftovers

* Style fix

---------

Co-authored-by: Mika Ayenson, PhD <[email protected]>

Author: traut

.github/CODEOWNERS

@@ -5,7 +5,6 @@ tests/**/*.py     @mikaayenson @eric-forte-elastic @terrancedejesus
 detection_rules/  @mikaayenson @eric-forte-elastic @terrancedejesus
 tests/            @mikaayenson @eric-forte-elastic @terrancedejesus
 lib/              @mikaayenson @eric-forte-elastic @terrancedejesus
-rta/              @mikaayenson @eric-forte-elastic @terrancedejesus
 hunting/          @mikaayenson @eric-forte-elastic @terrancedejesus
 
 # skip rta-mapping to avoid the spam

.github/paths-labeller.yml

@@ -12,8 +12,6 @@
   - "detection_rules/**/*.py"
   - "kibana/**/*.py"
   - "kql/**/*.py"
-- "RTA":
-  - "rta/**/*"
 - "Hunting":
   - "hunting/**/*"
 

.pre-commit-config.yaml

@@ -6,21 +6,21 @@ repos:
     hooks:
       - id: flake8
         args: ['--ignore=D203,C901,E501,W503', '--max-line-length=120','--max-complexity=10', '--statistics']
-        exclude: '^rta|^kql'
+        exclude: '^kql'
   - repo: https://github.com/PyCQA/bandit
     rev: 1.7.4
     hooks:
       - id: bandit
         args: ['-s', 'B101,B603,B404,B607']
-        exclude: '^rta|^kql'
+        exclude: '^kql'
   # Potential future rigor
   # - repo: https://github.com/PyCQA/pylint
   #   rev: v2.15.6
   #   hooks:
   #     - id: pylint
   #       language: system
-  #       exclude: '^rta|^kql'
+  #       exclude: '^kql'
   # - repo: https://github.com/PyCQA/isort
   #   rev: 5.10.1
   #   hooks:
-  #     - id: isort
+  #     - id: isort

README.md

@@ -16,6 +16,7 @@ This repository was first announced on Elastic's blog post, [Elastic Security op
   - [Overview of this repository](#overview-of-this-repository)
   - [Getting started](#getting-started)
   - [How to contribute](#how-to-contribute)
+  - [RTAs](#rtas)
   - [Licensing](#licensing)
   - [Questions? Problems? Suggestions?](#questions-problems-suggestions)
 
@@ -31,7 +32,6 @@ Detection Rules contains more than just static rule files. This repository also
 | [`hunting/`](./hunting/)                        | Root directory where threat hunting package and queries are stored                   |
 | [`kibana/`](lib/kibana)                         | Python library for handling the API calls to Kibana and the Detection Engine        |
 | [`kql/`](lib/kql)                               | Python library for parsing and validating Kibana Query Language                     |
-| [`rta/`](rta)                                   | Red Team Automation code used to emulate attacker techniques, used for rule testing |
 | [`rules/`](rules)                               | Root directory where rules are stored                                               |
 | [`rules_building_block/`](rules_building_block) | Root directory where building block rules are stored                                |
 | [`tests/`](tests)                               | Python code for unit testing rules                                                  |
@@ -133,9 +133,14 @@ For more advanced command line interface (CLI) usage, refer to the [CLI guide](C
 
 We welcome your contributions to Detection Rules! Before contributing, please familiarize yourself with this repository, its [directory structure](#overview-of-this-repository), and our [philosophy](PHILOSOPHY.md) about rule creation. When you're ready to contribute, read the [contribution guide](CONTRIBUTING.md) to learn how we turn detection ideas into production rules and validate with testing.
 
+## RTAs
+
+Red Team Automations (RTAs) used to emulate attacker techniques and verify the rules can be found in dedicated
+repository - [Cortado](https://github.com/elastic/cortado).
+
 ## Licensing
 
-Everything in this repository — rules, code, RTA, etc. — is licensed under the [Elastic License v2](LICENSE.txt). These rules are designed to be used in the context of the Detection Engine within the Elastic Security application. If you’re using our [Elastic Cloud managed service](https://www.elastic.co/cloud/) or the default distribution of the Elastic Stack software that includes the [full set of free features](https://www.elastic.co/subscriptions), you’ll get the latest rules the first time you navigate to the detection engine.
+Everything in this repository — rules, code, etc. — is licensed under the [Elastic License v2](LICENSE.txt). These rules are designed to be used in the context of the Detection Engine within the Elastic Security application. If you’re using our [Elastic Cloud managed service](https://www.elastic.co/cloud/) or the default distribution of the Elastic Stack software that includes the [full set of free features](https://www.elastic.co/subscriptions), you’ll get the latest rules the first time you navigate to the detection engine.
 
 Occasionally, we may want to import rules from another repository that already have a license, such as MIT or Apache 2.0. This is welcome, as long as the license permits sublicensing under the Elastic License v2. We keep those license notices in `NOTICE.txt` and sublicense as the Elastic License v2 with all other rules. We also require contributors to sign a [Contributor License Agreement](https://www.elastic.co/contributor-agreement) before contributing code to any Elastic repositories.
 

detection_rules/__init__.py

@@ -19,7 +19,6 @@
     ghwrap,
     kbwrap,
     main,
-    mappings,
     ml,
     misc,
     navigator,
@@ -37,7 +36,6 @@
     'eswrap',
     'ghwrap',
     'kbwrap',
-    'mappings',
     "main",
     'misc',
     'ml',

detection_rules/cli_utils.py

@@ -265,7 +265,4 @@ def rule_prompt(path=None, rule_type=None, required_only=True, save=True, verbos
         print('Did not set the following values because they are un-required when set to the default value')
         print(' - {}'.format('\n - '.join(skipped)))
 
-    # rta_mappings.add_rule_to_mapping_file(rule)
-    # click.echo('Placeholder added to rule-mapping.yaml')
-
     return rule

detection_rules/docs.py

@@ -77,7 +77,6 @@ def populate(self):
         self.add_summary()
         self.add_rule_details()
         self.add_attack_matrix()
-        self.add_rta_mapping()
         self.add_rule_details(self.deprecated_rules, 'Deprecated Rules')
 
     def add_summary(self):
@@ -172,27 +171,6 @@ def add_rule_details(self, rules: Optional[Union[DeprecatedCollection, RuleColle
 
         worksheet.autofilter(0, 0, len(rules) + 1, len(headers) - 1)
 
-    def add_rta_mapping(self):
-        """Add a worksheet for the RTA/Rule RTA mapping."""
-        from .rule_loader import rta_mappings
-
-        worksheet = self.add_worksheet('RTA Mapping')
-        worksheet.freeze_panes(1, 0)
-        headers = ('Rule ID', 'Rule Name', 'RTA')
-        for column, header in enumerate(headers):
-            worksheet.write(0, column, header, self.default_header_format)
-
-        row = 1
-        for rule_id, mapping in rta_mappings.get_rta_mapping().items():
-            worksheet.write(row, 0, rule_id)
-            worksheet.write(row, 1, mapping['rule_name'])
-            worksheet.write(row, 2, mapping['rta_name'])
-            row += 1
-
-        worksheet.set_column(0, 0, 35)
-        worksheet.set_column(1, 1, 50)
-        worksheet.set_column(2, 2, 35)
-
     def add_attack_matrix(self):
         """Add a worksheet for ATT&CK coverage."""
         worksheet = self.add_worksheet(attack_tm + ' Coverage')

detection_rules/eswrap.py

@@ -21,9 +21,9 @@
 from .main import root
 from .misc import add_params, client_error, elasticsearch_options, get_elasticsearch_client, nested_get
 from .rule import TOMLRule
-from .rule_loader import rta_mappings, RuleCollection
-from .utils import format_command_options, normalize_timing_and_sort, unix_time_to_formatted, get_path
 
+from .rule_loader import RuleCollection
+from .utils import format_command_options, normalize_timing_and_sort, unix_time_to_formatted, get_path
 
 COLLECTION_DIR = get_path('collections')
 MATCH_ALL = {'bool': {'filter': [{'match_all': {}}]}}
@@ -60,7 +60,7 @@ def parse_unique_field_results(rule_type: str, unique_fields: List[str], search_
     return {'results': parsed_results} if parsed_results else {}
 
 
-class RtaEvents:
+class Events:
     """Events collected from Elasticsearch."""
 
     def __init__(self, events):
@@ -87,7 +87,7 @@ def _get_dump_dir(rta_name=None, host_id=None, host_os_family=None):
             os.makedirs(dump_dir, exist_ok=True)
             return dump_dir
 
-    def evaluate_against_rule_and_update_mapping(self, rule_id, rta_name, verbose=True):
+    def evaluate_against_rule(self, rule_id, verbose=True):
         """Evaluate a rule against collected events and update mapping."""
         from .utils import combine_sources, evaluate
 
@@ -96,15 +96,10 @@ def evaluate_against_rule_and_update_mapping(self, rule_id, rta_name, verbose=Tr
         merged_events = combine_sources(*self.events.values())
         filtered = evaluate(rule, merged_events, normalize_kql_keywords=RULES_CONFIG.normalize_kql_keywords)
 
-        if filtered:
-            sources = [e['agent']['type'] for e in filtered]
-            mapping_update = rta_mappings.add_rule_to_mapping_file(rule, len(filtered), rta_name, *sources)
+        if verbose:
+            click.echo('Matching results found')
 
-            if verbose:
-                click.echo('Updated rule-mapping file with: \n{}'.format(json.dumps(mapping_update, indent=2)))
-        else:
-            if verbose:
-                click.echo('No updates to rule-mapping file; No matching results')
+        return filtered
 
     def echo_events(self, pager=False, pretty=True):
         """Print events to stdout."""
@@ -322,8 +317,8 @@ def count_from_rule(self, rules: RuleCollection, start_time=None, end_time='now'
         return survey_results
 
 
-class CollectRtaEvents(CollectEvents):
-    """Collect RTA events from elasticsearch."""
+class CollectEventsWithDSL(CollectEvents):
+    """Collect events from elasticsearch."""
 
     @staticmethod
     def _group_events_by_type(events):
@@ -340,15 +335,15 @@ def run(self, dsl, indexes, start_time):
         results = self.search(dsl, language='dsl', index=indexes, start_time=start_time, end_time='now', size=5000,
                               sort=[{'@timestamp': {'order': 'asc'}}])
         events = self._group_events_by_type(results)
-        return RtaEvents(events)
+        return Events(events)
 
 
 @root.command('normalize-data')
 @click.argument('events-file', type=click.File('r'))
 def normalize_data(events_file):
     """Normalize Elasticsearch data timestamps and sort."""
     file_name = os.path.splitext(os.path.basename(events_file.name))[0]
-    events = RtaEvents({file_name: [json.loads(e) for e in events_file.readlines()]})
+    events = Events({file_name: [json.loads(e) for e in events_file.readlines()]})
     events.save(dump_dir=os.path.dirname(events_file.name))
 
 
@@ -383,15 +378,15 @@ def collect_events(ctx, host_id, query, index, rta_name, rule_id, view_events):
     dsl['bool'].setdefault('filter', []).append({'bool': {'should': [{'match_phrase': {'host.id': host_id}}]}})
 
     try:
-        collector = CollectRtaEvents(client)
+        collector = CollectEventsWithDSL(client)
         start = time.time()
         click.pause('Press any key once detonation is complete ...')
         start_time = f'now-{round(time.time() - start) + 5}s'
         events = collector.run(dsl, index or '*', start_time)
         events.save(rta_name=rta_name, host_id=host_id)
 
         if rta_name and rule_id:
-            events.evaluate_against_rule_and_update_mapping(rule_id, rta_name)
+            events.evaluate_against_rule(rule_id)
 
         if view_events and events.events:
             events.echo_events(pager=True)

detection_rules/etc/rule-mapping.yaml

@@ -27,7 +27,6 @@
 from .generic_loader import GenericCollection
 from .exception import (TOMLExceptionContents,
                         build_exception_objects, parse_exceptions_results_from_api)
-from .mappings import build_coverage_map, get_triggered_rules, print_converage_summary
 from .misc import (
     add_client, client_error, nested_set, parse_user_config
 )
@@ -696,29 +695,3 @@ def prep_rule(author: str):
     updated_rule.write_text(json.dumps(template_rule, sort_keys=True))
     click.echo(f'Rule saved to: {updated_rule}. Import this to Kibana to create alerts on all dnstwist-* indexes')
     click.echo('Note: you only need to import and enable this rule one time for all dnstwist-* indexes')
-
-
-@root.group('rta')
-def rta_group():
-    """Commands related to Red Team Automation (RTA) scripts."""
-
-
-# create command to show rule-rta coverage
-@rta_group.command('coverage')
-@click.option("-o", "--os-filter", default="all",
-              help="Filter rule coverage summary by OS. (E.g. windows) Default: all")
-def rta_coverage(os_filter: str):
-    """Show coverage of RTA / rules by os type."""
-
-    # get all rules
-    all_rules = RuleCollection.default()
-
-    # get rules triggered by RTA
-    triggered_rules = get_triggered_rules()
-
-    # build coverage map
-    coverage_map = build_coverage_map(triggered_rules, all_rules)
-
-    # # print summary
-    all_rule_count = len(all_rules.rules)
-    print_converage_summary(coverage_map, all_rule_count, os_filter)