[DaC] Beta Release (#3889)

Co-authored-by: Justin Ibarra <[email protected]>
Co-authored-by: brokensound77 <[email protected]>
Co-authored-by: Mika Ayenson <[email protected]>
Co-authored-by: Mika Ayenson <[email protected]>

Author: 3 people

CLI.md

@@ -5,7 +5,7 @@ the [README](README.md). Basic use of the CLI such as [creating a rule](CONTRIBU
 [testing](CONTRIBUTING.md#testing-a-rule-with-the-cli) are referenced in the [contribution guide](CONTRIBUTING.md).
 
 
-## Using a config file or environment variables
+## Using a user config file or environment variables
 
 CLI commands which are tied to Kibana and Elasticsearch are capable of parsing auth-related keyword args from a config
 file or environment variables.
@@ -17,9 +17,9 @@ follows:
 * config values
 * prompt (this only applies to certain values)
 
-#### Setup a config file
+#### Setup a user config file
 
-In the root directory of this repo, create the file `.detection-rules-cfg.json` and add relevant values
+In the root directory of this repo, create the file `.detection-rules-cfg.json` (or `.yaml`) and add relevant values
 
 Currently supported arguments:
 * elasticsearch_url
@@ -42,6 +42,8 @@ on the building block rules.
 
 Using the environment variable `DR_BYPASS_TAGS_VALIDATION` will bypass the Detection Rules Unit Tests on the `tags` field in toml files.
 
+Using the environment variable `DR_BYPASS_TIMELINE_TEMPLATE_VALIDATION` will bypass the timeline template id and title validation for rules. 
+
 ## Importing rules into the repo
 
 You can import rules into the repo using the `create-rule` or `import-rules-to-repo` commands. Both of these commands will
@@ -85,9 +87,19 @@ Usage: detection_rules import-rules-to-repo [OPTIONS] [INPUT_FILE]...
   Import rules from json, toml, yaml, or Kibana exported rule file(s).
 
 Options:
-  --required-only            Only prompt for required fields
-  -d, --directory DIRECTORY  Load files from a directory
-  -h, --help                 Show this message and exit.
+  -ac, --action-connector-import  Include action connectors in export
+  -e, --exceptions-import         Include exceptions in export
+  --required-only                 Only prompt for required fields
+  -d, --directory DIRECTORY       Load files from a directory
+  -s, --save-directory DIRECTORY  Save imported rules to a directory
+  -se, --exceptions-directory DIRECTORY
+                                  Save imported exceptions to a directory
+  -sa, --action-connectors-directory DIRECTORY
+                                  Save imported actions to a directory
+  -ske, --skip-errors             Skip rule import errors
+  -da, --default-author TEXT      Default author for rules missing one
+  -snv, --strip-none-values       Strip None values from the rule
+  -h, --help                      Show this message and exit.
 ```
 
 The primary advantage of using this command is the ability to import multiple rules at once. Multiple rule paths can be
@@ -97,10 +109,14 @@ a combination of both.
 In addition to the formats mentioned using `create-rule`, this will also accept an `.ndjson`/`jsonl` file
 containing multiple rules (as would be the case with a bulk export).
 
+The `-s/--save-directory` is an optional parameter to specify a non default directory to place imported rules. If it is not specified, the first directory specified in the rules config will be used.
+
 This will also strip additional fields and prompt for missing required fields.
 
 <a id="note-3">\* Note</a>: This will attempt to parse ALL files recursively within a specified directory.
 
+Additionally, the `-e` flag can be used to import exceptions in addition to rules from the export file.
+
 
 ## Commands using Elasticsearch and Kibana clients
 
@@ -165,6 +181,8 @@ Options:
   -h, --help                   Show this message and exit.
 
 Commands:
+  export-rules   Export custom rules from Kibana.
+  import-rules   Import custom rules into Kibana.
   search-alerts  Search detection engine alerts with KQL.
   upload-rule    Upload a list of rule .toml files to Kibana.
 ```
@@ -272,21 +290,24 @@ directly.
 ```console
 Usage: detection_rules export-rules-from-repo [OPTIONS]
 
-  Export rule(s) into an importable ndjson file.
+  Export rule(s) and exception(s) into an importable ndjson file.
 
 Options:
   -f, --rule-file FILE
   -d, --directory DIRECTORY       Recursively load rules from a directory
   -id, --rule-id TEXT
   -o, --outfile PATH              Name of file for exported rules
   -r, --replace-id                Replace rule IDs with new IDs before export
-  --stack-version [7.10|7.11|7.12|7.13|7.14|7.15|7.16|7.8|7.9|8.0|8.1|8.10|8.11|8.12|8.13|8.2|8.3|8.4|8.5|8.6|8.7|8.8|8.9]
+  --stack-version [7.8|7.9|7.10|7.11|7.12|7.13|7.14|7.15|7.16|8.0|8.1|8.2|8.3|8.4|8.5|8.6|8.7|8.8|8.9|8.10|8.11|8.12|8.13|8.14]
                                   Downgrade a rule version to be compatible
                                   with older instances of Kibana
   -s, --skip-unsupported          If `--stack-version` is passed, skip rule
                                   types which are unsupported (an error will
                                   be raised otherwise)
   --include-metadata              Add metadata to the exported rules
+  -ac, --include-action-connectors
+                                  Include Action Connectors in export
+  -e, --include-exceptions        Include Exceptions Lists in export
   -h, --help                      Show this message and exit.
 ```
 
@@ -317,6 +338,7 @@ Options:
   --kibana-url TEXT
   -kp, --kibana-password TEXT
   -kc, --kibana-cookie TEXT    Cookie from an authed session
+  --api-key TEXT
   --cloud-id TEXT              ID of the cloud instance.
 
 Usage: detection_rules kibana import-rules [OPTIONS]
@@ -329,7 +351,7 @@ Options:
   -id, --rule-id TEXT
   -o, --overwrite                 Overwrite existing rules
   -e, --overwrite-exceptions      Overwrite exceptions in existing rules
-  -a, --overwrite-action-connectors
+  -ac, --overwrite-action-connectors
                                   Overwrite action connectors in existing
                                   rules
   -h, --help                      Show this message and exit.
@@ -476,6 +498,51 @@ python -m detection_rules kibana import-rules -d test-export-rules -o
 
 ### Exporting rules
 
+This command should be run with the `CUSTOM_RULES_DIR` envvar set, that way proper validation is applied to versioning when the rules are downloaded. See the [custom rules docs](docs/custom-rules.md) for more information.
+
+```
+python -m detection_rules kibana export-rules -h
+
+█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
+█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
+█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█
+
+Kibana client:
+Options:
+  --ignore-ssl-errors TEXT
+  --space TEXT                 Kibana space
+  --provider-name TEXT         Elastic Cloud providers: cloud-basic and cloud-
+                               saml (for SSO)
+  --provider-type TEXT         Elastic Cloud providers: basic and saml (for
+                               SSO)
+  -ku, --kibana-user TEXT
+  --kibana-url TEXT
+  -kp, --kibana-password TEXT
+  -kc, --kibana-cookie TEXT    Cookie from an authed session
+  --api-key TEXT
+  --cloud-id TEXT              ID of the cloud instance.
+
+Usage: detection_rules kibana export-rules [OPTIONS]
+
+  Export custom rules from Kibana.
+
+Options:
+  -d, --directory PATH            Directory to export rules to  [required]
+  -acd, --action-connectors-directory PATH
+                                  Directory to export action connectors to
+  -ed, --exceptions-directory PATH
+                                  Directory to export exceptions to
+  -da, --default-author TEXT      Default author for rules missing one
+  -r, --rule-id TEXT              Optional Rule IDs to restrict export to
+  -ac, --export-action-connectors
+                                  Include action connectors in export
+  -e, --export-exceptions         Include exceptions in export
+  -s, --skip-errors               Skip errors when exporting rules
+  -sv, --strip-version            Strip the version fields from all rules
+  -h, --help                      Show this message and exit.
+
+```
+
 Example of a rule exporting, with errors skipped
 
 ```
@@ -648,4 +715,4 @@ value = "fast"
 ```
 
 The easiest way to _update_ a rule with existing transform entries is to use `guide-plugin-convert` and manually add it
-to the rule.
+to the rule.

detection_rules/__init__.py

@@ -11,6 +11,8 @@
 assert (3, 12) <= sys.version_info < (4, 0), "Only Python 3.12+ supported"
 
 from . import (  # noqa: E402
+    custom_schemas,
+    custom_rules,
     devtools,
     docs,
     eswrap,
@@ -28,6 +30,8 @@
 )
 
 __all__ = (
+    'custom_rules',
+    'custom_schemas',
     'devtools',
     'docs',
     'eswrap',

detection_rules/action.py

@@ -0,0 +1,64 @@
+# Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+# or more contributor license agreements. Licensed under the Elastic License
+# 2.0; you may not use this file except in compliance with the Elastic License
+# 2.0.
+
+"""Dataclasses for Action."""
+from dataclasses import dataclass
+from pathlib import Path
+from typing import List, Optional
+
+from .mixins import MarshmallowDataclassMixin
+from .schemas import definitions
+
+
+@dataclass(frozen=True)
+class ActionMeta(MarshmallowDataclassMixin):
+    """Data stored in an exception's [metadata] section of TOML."""
+    creation_date: definitions.Date
+    rule_id: List[definitions.UUIDString]
+    rule_name: str
+    updated_date: definitions.Date
+
+    # Optional fields
+    deprecation_date: Optional[definitions.Date]
+    comments: Optional[str]
+    maturity: Optional[definitions.Maturity]
+
+
+@dataclass
+class Action(MarshmallowDataclassMixin):
+    """Data object for rule Action."""
+    @dataclass
+    class ActionParams:
+        """Data object for rule Action params."""
+        body: str
+
+    action_type_id: definitions.ActionTypeId
+    group: str
+    params: ActionParams
+    id: Optional[str]
+    frequency: Optional[dict]
+    alerts_filter: Optional[dict]
+
+
+@dataclass(frozen=True)
+class TOMLActionContents(MarshmallowDataclassMixin):
+    """Object for action from TOML file."""
+    metadata: ActionMeta
+    actions: List[Action]
+
+
+@dataclass(frozen=True)
+class TOMLAction:
+    """Object for action from TOML file."""
+    contents: TOMLActionContents
+    path: Path
+
+    @property
+    def name(self):
+        return self.contents.metadata.rule_name
+
+    @property
+    def id(self):
+        return self.contents.metadata.rule_id