update eventdb and postgresql documentation

update to current version
notes on distributed setups: networking, permissions
installation: refer to upstream
database: template and encoding

Author: sebix

CHANGELOG.md

@@ -42,7 +42,7 @@
   - Add a deprecation warning (PR#2544 by Sebastian Wagner)
 
 ### Documentation
-- `docs/admin/installation/linux-packages`: Add `[signed-by=]` options, add wget command as alternative to curl
+- `docs/admin/installation/linux-packages`: Add `[signed-by=]` options, add wget command as alternative to curl (PR#2547 by Sebastian Wagner).
 
 ### Packaging
 

docs/admin/database/postgresql.md

@@ -15,20 +15,15 @@ You have two basic choices to run PostgreSQL:
 
 1. on the same machine as intelmq, then you could use Unix sockets if available on your platform
 2. on a different machine. In which case you would need to use a TCP connection and make sure you give the right
-   connection parameters to each psql or client call.
+   connection parameters to each psql or client call. On the networking side, make sure the connections are allowed, postgresql-server is listening on the correct interface and the user is allowed to connect (`pg_hba.conf`).
 
-Make sure to consult your PostgreSQL documentation about how to allow network connections and authentication in case 2.
+### PostgreSQL Server Version
 
-### PostgreSQL Version
-
-Any supported version of PostgreSQL should work (v>=9.2 as of Oct 2016) [[1]](https://www.postgresql.org/support/versioning/).
-
-If you use PostgreSQL server v >= 9.4, it gives you the possibility to use the time-zone [formatting string](https://www.postgresql.org/docs/9.4/static/images/functions-formatting.html) "OF" for date-times and the [GiST index for the CIDR type](https://www.postgresql.org/docs/9.4/static/images/release-9-4.html#AEN120769). This may be useful depending on how you plan to use the events that this bot writes into the database.
-
-### intelmq_psql_initdb
+Any supported version of PostgreSQL should work (v>=13 as of January 2025) [[1]](https://www.postgresql.org/support/versioning/).
+### events table definition (`intelmq_psql_initdb`)
 
 IntelMQ comes with the `intelmq_psql_initdb` command line tool designed to help with creating the
-EventDB. It creates in the first line:
+EventDB tabls. It generates these SQL commands for you:
 
 - A `CREATE TABLE events` statement with all valid IntelMQ fields as columns and correct types
 - Several indexes as examples for a good read & search performance
@@ -37,22 +32,21 @@ Having an `events` table as outlined in the SQL file, IntelMQ's SQL Output Bot c
 
 In addition, the script supports some additional features supporting use cases described later in this document:
 
-- `--partition-key` - for generating schema aligned with TimescaleDB or partitioned tables,
-- `--separate-raws` - for generating views and triggers needed to `eventdb_raws_table` (works also together with adjustments for partitioning).
+- `--partition-key` - for generating schema aligned with TimescaleDB or partitioned tables
+- `--separate-raws` - for generating views and triggers needed to `eventdb_raws_table` (works also together with adjustments for partitioning). This can increase the performance by separating data in two tables. See section [Separating raw values in PostgreSQL using view and trigger](#separating-raw-values-in-postgresql-using-view-and-trigger) below for more information.
 
 For a full list of supported parameters, call the script help using `-h` parameter.
 
 All elements of the generated SQL file can be adapted and extended before running the SQL file against a database, especially the indexes. Please review the generated script before applying.
 
 Be aware that if you create tables using another DB user that is used later by the output bot, you may need to adjust ownership or privileges in the database. If you have problems with database permissions,
-refer to `PostgreSQL documentation <https://www.postgresql.org/docs/current/ddl-priv.html>`.
+refer to [PostgreSQL documentation](https://www.postgresql.org/docs/current/ddl-priv.html).
 
 ### Installation
 
-Use `intelmq_psql_initdb` to create initial SQL statements from `harmonization.conf`. The script will create the
-required table layout and save it as `/tmp/initdb.sql`
+First, install the PostgreSQL on your server using system packages or using any installation variant supported by the project: [PostgreSQL server installation](https://www.postgresql.org/download/)
 
-You need a PostgreSQL database-user to own the result database. The recommendation is to use the name `intelmq`
+You need a PostgreSQL database-user to own the resulting database. The recommendation is to use the name `intelmq`
 . There may already be such a user for the PostgreSQL database-cluster to be used by other bots. (For example from
 setting up the expert/certbund_contact bot.)
 
@@ -66,16 +60,16 @@ createuser --no-superuser --no-createrole --no-createdb --encrypted --pwprompt i
 Create the new database:
 
 ```bash
-createdb --encoding='utf-8' --owner=intelmq intelmq-events
+createdb --encoding='utf-8' --owner=intelmq intelmq-events --template template0
 ```
 
-(The encoding parameter should ensure the right encoding on platform where this is not the default.)
+The encoding parameter should ensure the right encoding on platform where this is not the default. Template `template1` has encoding `SQL_ASCII`, so in order to use UTF8, `template0` is required.
 
 Now initialize it as database-user `intelmq` (in this example a network connection to localhost is used, so you would
 get to test if the user `intelmq` can authenticate):
 
 ```bash
-psql -h localhost intelmq-events intelmq </tmp/initdb.sql
+psql -h localhost intelmq-events intelmq < /tmp/initdb.sql
 ```
 ## EventDB Utilities