Merge pull request certtools#2560 from sebix/docs

sebix · web-flow · commit c00c35545810 · 2025-01-30T11:34:45.000+01:00
Docs: Add redis docs, fix existing pipeline docs, add unlinked file
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -49,6 +49,7 @@
 
 ### Documentation
 - `docs/admin/installation/linux-packages`: Add `[signed-by=]` options, add wget command as alternative to curl (PR#2547 by Sebastian Wagner).
+- Add documentation on the Redis pipeline (databases, configuration), fix generic pipeline documentation and add missing information on parameters, add unlinked intelmqctl docs to the index and TOC (PR#2560 by Sebastian Wagner).
 
 ### Packaging
 
diff --git a/docs/admin/beta-features.md b/docs/admin/beta-features.md
@@ -56,78 +56,84 @@ only RabbitMQ as broker has been tested.
 You can change the broker for single bots (set the parameters in the runtime configuration per bot) or for the whole
 botnet (using the global configuration).
 
+### Settings
+
 You need to set the parameter
 `source_pipeline_broker`/`destination_pipeline_broker` to `amqp`. There are more parameters available:
 
-!!! bug
-    This section of the documentation is currently incomplete and will be updated later.
-
 **`destination_pipeline_broker`**
 
 (required, string) `"amqp"`
 
-
 **`destination_pipeline_host`**
 
-()  (default: `'127.0.0.1'`)
+(optional, string)  (default: `'127.0.0.1'`)
 
 **`destination_pipeline_port`**
 
-() (default: 5672)
+(optional, integer) (default: 5672)
 
 **`destination_pipeline_username`**
 
-()
+(optional, string)
 
 **`destination_pipeline_password`**
 
-()
+(optional, string)
+
+**`destination_pipeline_ssl`**
+
+(optional, boolean) (default: false)
 
 **`destination_pipeline_socket_timeout`**
 
-() (default: no timeout)
+(optional, integer) (default: no timeout)
 
 **`destination_pipeline_amqp_exchange`**
 
-() Only change/set this if you know what you do. If set, the destination queues are not declared as queues, but used as routing key. (default: `''`).
+(optional, string) Only change/set this if you know what you do. If set, the destination queues are not declared as queues, but used as routing key. (default: `''`).
 
 **`destination_pipeline_amqp_virtual_host`**
 
-() (default: `'/'`)
+(optional, string) (default: `'/'`)
 
 **`source_pipeline_host`**
 
-() (default: `'127.0.0.1'`)
-
+(optional, string) (default: `'127.0.0.1'`)
 
 **`source_pipeline_port`**
 
-() (default: 5672)
+(optional, port) (default: 5672)
 
 **`source_pipeline_username`**
 
-()
+(optional, string)
 
 **`source_pipeline_password`**
 
-()
+(optional, string)
+
+**`source_pipeline_ssl`**
+
+(optional, boolean) (default: false)
 
 **`source_pipeline_socket_timeout`**
 
-() (default: no timeout)
+(optional, string) (default: no timeout)
 
 **`source_pipeline_amqp_exchange`**
 
-() Only change/set this if you know what you do. If set, the destination queues are not declared as queues, but used as routing key. (default: ['']).
-
+(optional, string) Only change/set this if you know what you do. If set, the destination queues are not declared as queues, but used as routing key. (default: `""`]).
 
 **`source_pipeline_amqp_virtual_host`**
 
-() (default: `'/'`)
+(optional, string) (default: `'/'`)
 
 **`intelmqctl_rabbitmq_monitoring_url`**
 
-() string, see below (default: `"http://{host}:15672"`)
+(optional, string) string, see below (default: `"http://{host}:15672"`)
+
+### Monitoring queues
 
 For getting the queue sizes, `intelmqctl` needs to connect to the monitoring interface of RabbitMQ. If the monitoring
 interface is not available under `http://{host}:15672` you can manually set using the
@@ -137,10 +143,11 @@ user account, make sure to add the tag "monitoring" to it, otherwise IntelMQ can
 
 ![](../static/images/rabbitmq-user-monitoring.png)
 
+### Statistics
+
 Setting the statistics (and cache) parameters is necessary when the local redis is running under a non-default
 host/port. If this is the case, you can set them explicitly:
 
-
 **`statistics_database`**
 
 () `3`
diff --git a/docs/admin/configuration/intelmq.md b/docs/admin/configuration/intelmq.md
@@ -156,30 +156,14 @@ Some information can as well be found in Python's documentation on the used
 
 If the path `_on_error` exists for a bot, the message is also sent to this queue, instead of (only) dumping the file if
 configured to do so.
-
-##### Miscellaneous
-
-**`load_balance`**
-
-(required, boolean) this option allows you to choose the behavior of the queue. Use the following values:
-
-  - **true** - splits the messages into several queues without duplication
-  - **false** - duplicates the messages into each queue - When using AMQP as message broker, take a look at the `multithreading`{.interpreted-text role="ref"} section and the `instances_threads` parameter.
-
-**`rate_limit`**
-
-(required, integer) time interval (in seconds) between messages processing. int value.
-
-**`ssl_ca_certificate`**
-
-(optional, string) trusted CA certificate for IMAP connections (supported by some bots).
+##### Pipeline
 
 **`source_pipeline_broker`**
 
 (optional, string) Allowed values are `redis` and `amqp`. Selects the message broker IntelMQ should use. As this parameter can be overridden by each bot, this allows usage of different broker systems and hosts, as well as switching between them on the same IntelMQ instance. Defaults to `redis`.
 
   - **redis** - Please note that persistence has to be [manually activated](http://redis.io/topics/persistence).
-  - **amqp** - [Using the AMQP broker]() is currently beta but there are no known issues. A popular AMQP broker is [RabbitMQ](https://www.rabbitmq.com/).
+  - **amqp** - [Using the AMQP broker](../beta-features.md#using-amqp-message-broker) is currently beta but there are no known issues. A popular AMQP broker is [RabbitMQ](https://www.rabbitmq.com/).
 
 **`destination_pipeline_broker`**
 
@@ -220,6 +204,22 @@ configured to do so.
 
 (required, integer) broker database that the bot will use to connect and send messages (requirement from
   redis broker).
+##### Miscellaneous
+
+**`load_balance`**
+
+(required, boolean) this option allows you to choose the behavior of the queue. Use the following values:
+
+  - **true** - splits the messages into several queues without duplication
+  - **false** - duplicates the messages into each queue - When using AMQP as message broker, take a look at the `multithreading`{.interpreted-text role="ref"} section and the `instances_threads` parameter.
+
+**`rate_limit`**
+
+(required, integer) time interval (in seconds) between messages processing. int value.
+
+**`ssl_ca_certificate`**
+
+(optional, string) trusted CA certificate for IMAP connections (supported by some bots).
 
 **`http_proxy`**
 
diff --git a/docs/admin/configuration/redis.md b/docs/admin/configuration/redis.md
@@ -0,0 +1,44 @@
+# Redis Pipeline (Message broker)
+<!-- comment
+   SPDX-FileCopyrightText: 2025 Sebastian Wagner, Intevation GmbH <sebix@sebix.at>
+   SPDX-License-Identifier: AGPL-3.0-or-later
+-->
+
+The default IntelMQ Pipeline (the message exchange between bots) is [Redis](https://redis.io/) or its OpenSource successor [Valkey](https://valkey.io/).
+
+For AMQP (RabbitMQ) see [Using AMQP Message Broker](../beta-features.md#using-amqp-message-broker) in the section about Beta features.
+
+## Usage of databases
+
+You can use any redis database for any purpose. There are no hardcoded defaults or other requirements in IntelMQ.
+
+It's also possible to use the same database for two different bots. Just make sure, that their data doesn't collide.
+
+These are some of the usages of redis databases using the IntelMQ default values:
+
+- 2: Pipeline (Queues)
+- 4: IntelMQ Tests
+- 6: Deduplicator Expert
+- 7: Reverse DNS Expert
+- 8: RDAP Expert, Aggregate Expert
+- 10: RIPE Expert
+- 12: Shadowserver Reports API Collector
+- 15: SMTP Batch Output
+
+By default, Redis/valkey have a maximum of 16 databases (0-15). In it the Redis/valkey server configuration file, this value can be increased.
+While the number of maximum databases [is unlimited](https://redis.io/docs/latest/embeds/how-many-databases-software/), the practical limit is INT_32.
+
+## Setup tips
+
+SWAP space should be bigger or equal to your memory. See also [Hardware requirements](../hardware-requirements.md).
+
+### Redis memory overcommitting
+
+It is [recommended](https://redis.io/docs/latest/operate/oss_and_stack/management/admin/#linux) to enable memory overcommitting for Redis.
+
+Run `sysctl vm.overcommit_memory=1` to set it for the current session.
+To enable it permanently, create a file with `vm.overcommit_memory = 1` in `/etc/sysctl.d/intelmq.conf`.
+
+### Maxmemory
+
+To limit the maximum memory used by Redis and also to mitigate possible downsides of memory overcommitting, you can set a maximum memory usage in the Redis/Valkey server configuration with setting `maxmemory <bytes>`.
diff --git a/docs/admin/management/intelmqctl-more.md b/docs/admin/management/intelmqctl-more.md
diff --git a/intelmq/lib/pipeline.py b/intelmq/lib/pipeline.py
@@ -488,23 +488,23 @@ def receive(self) -> str:
 
 class Amqp(Pipeline):
     queue_args = {'x-queue-mode': 'lazy'}
-    source_pipeline_host = '127.0.0.1'
-    destination_pipeline_host = '127.0.0.1'
-    source_pipeline_db = 2
-    destination_pipeline_db = 2
-    source_pipeline_username = None
-    destination_pipeline_username = None
-    source_pipeline_password = None
-    destination_pipeline_password = None
-    source_pipeline_socket_timeout = None
-    destination_pipeline_socket_timeout = None
-    source_pipeline_amqp_virtual_host = '/'
-    destination_pipeline_amqp_virtual_host = '/'
-    source_pipeline_ssl = False
-    destination_pipeline_ssl = False
-    source_pipeline_amqp_exchange = ""
-    destination_pipeline_amqp_exchange = ""
-    intelmqctl_rabbitmq_monitoring_url = None
+    source_pipeline_host: str = '127.0.0.1'
+    destination_pipeline_host: str = '127.0.0.1'
+    source_pipeline_db: int = 2
+    destination_pipeline_db: int = 2
+    source_pipeline_username: Optional[str] = None
+    destination_pipeline_username: Optional[str] = None
+    source_pipeline_password: Optional[str] = None
+    destination_pipeline_password: Optional[str] = None
+    source_pipeline_socket_timeout: Optional[int] = None
+    destination_pipeline_socket_timeout: Optional[int] = None
+    source_pipeline_amqp_virtual_host: str = '/'
+    destination_pipeline_amqp_virtual_host: str = '/'
+    source_pipeline_ssl: bool = False
+    destination_pipeline_ssl: bool = False
+    source_pipeline_amqp_exchange: str = ""
+    destination_pipeline_amqp_exchange: str = ""
+    intelmqctl_rabbitmq_monitoring_url: Optional[str] = None
 
     def __init__(self, logger, pipeline_args: dict = None, load_balance=False, is_multithreaded=False):
         super().__init__(logger, pipeline_args, load_balance, is_multithreaded)
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -116,9 +116,11 @@ nav:
           - IntelMQ: 'admin/configuration/intelmq.md'
           - IntelMQ API: 'admin/configuration/intelmq-api.md'
           - IntelMQ Manager: 'admin/configuration/intelmq-manager.md'
+          - Redis: 'admin/configuration/redis.md'
       - Management:
           - IntelMQ: 'admin/management/intelmq.md'
           - IntelMQ API: 'admin/management/intelmq-api.md'
+          - intelmqctl: 'admin/management/intelmqctl-more.md'
       - Database:
           - PostgreSQL: 'admin/database/postgresql.md'
           - Elasticsearch: 'admin/database/elasticsearch.md'
@@ -152,4 +154,4 @@ nav:
   - Changelog: 'changelog.md'
   - Security: 'security.md'
   - Community: 'community.md'
-  - Help: 'help.md'
+  - Help: 'help.md'
