[Epic] 3.0 config - Part 1 #3544

veod32 · 2023-06-26T11:13:32Z

The following box.cfg options were described in instance_config with defaults similar to ones in box.cfg: * auth_type * auth_delay * disable_guest * password_lifetime_days * password_min_length * password_enforce_uppercase * password_enforce_lowercase * password_enforce_digits * password_enforce_specialchars * password_history_length Part of tarantool#8861 NO_DOC=tarantool/doc#3544 links the most actual schema, no need to update the issue.

The usual environment configuration source is useful for parametrized run: ``` TT_MEMTX_MEMORY=<...> tarantool --name <...> --config <...> ``` However, sometimes a user may need to set a default value, which doesn't rewrite one that is provided in the configuration. Now, it is possible to do using the environment variables with the `_DEFAULT` suffix. ``` TT_MEMTX_MEMORY_DEFAULT=<...> tarantool --name <...> --config <...> ``` This feature may be especially useful for wrappers that run tarantool internally with some default paths for data directories, socket files, pid file. We likely will use it in the `tt` tool. Part of tarantool#8862 NO_DOC=added into tarantool/doc#3544 manually

The following box.cfg options were described in instance_config with defaults similar to ones in box.cfg: * auth_type * auth_delay * disable_guest * password_lifetime_days * password_min_length * password_enforce_uppercase * password_enforce_lowercase * password_enforce_digits * password_enforce_specialchars * password_history_length Part of tarantool#8861 NO_DOC=tarantool/doc#3544 links the most actual schema, no need to update the issue.

The following box.cfg options were described in instance_config with defaults similar to ones in box.cfg: * auth_type * auth_delay * disable_guest * password_lifetime_days * password_min_length * password_enforce_uppercase * password_enforce_lowercase * password_enforce_digits * password_enforce_specialchars * password_history_length Part of #8861 NO_DOC=tarantool/doc#3544 links the most actual schema, no need to update the issue.

The usual environment configuration source is useful for parametrized run: ``` TT_MEMTX_MEMORY=<...> tarantool --name <...> --config <...> ``` However, sometimes a user may need to set a default value, which doesn't rewrite one that is provided in the configuration. Now, it is possible to do using the environment variables with the `_DEFAULT` suffix. ``` TT_MEMTX_MEMORY_DEFAULT=<...> tarantool --name <...> --config <...> ``` This feature may be especially useful for wrappers that run tarantool internally with some default paths for data directories, socket files, pid file. We likely will use it in the `tt` tool. Part of tarantool#8862 NO_DOC=added into tarantool/doc#3544 manually

It is convenient to access configuration using `config:get()` from the application script (`app.file` or `app.module`). However, before this commit, it was not possible, because the configuration was not considered as applied before the application script is loaded. Now, the script loading is moved into the post-apply phase. The resulting sequence of steps on startup/reload is the following. * collect configuration information (from all the sources) * <if the previous step is failed, set `check_errors` status and break> * apply the configuration (call all the appliers) * <if the previous step is failed, set `check_errors` status and break> * <set the new successful status: `ready` or `check_warnings`> * call post-apply hooks (including application script loading) * <if the previous step is failed, set `check_errors` status and break> * <set the new successful status: `ready` or `check_warnings`> I would like to briefly comment the changes in the tests. * `app_test.lua`: added the check for the new behavior (call config:get() from the app script) * `appliers_test.lua`: fixed the applier call (`.apply` -> `.post_apply`) * `config_test.lua`: fixed status observed in the app script (`*_in_progress` -> `ready`) NO_DOC=reflected in tarantool/doc#3544

It is convenient to access configuration using `config:get()` from the application script (`app.file` or `app.module`). However, before this commit, it was not possible, because the configuration was not considered as applied before the application script is loaded. Now, the script loading is moved into the post-apply phase. The resulting sequence of steps on startup/reload is the following. * collect configuration information (from all the sources) * <if the previous step is failed, set `check_errors` status and break> * apply the configuration (call all the appliers) * <if the previous step is failed, set `check_errors` status and break> * <set the new successful status: `ready` or `check_warnings`> * call post-apply hooks (including application script loading) * <if the previous step is failed, set `check_errors` status and break> * <set the new successful status: `ready` or `check_warnings`> I would like to briefly comment the changes in the tests. * `app_test.lua`: added the check for the new behavior (call config:get() from the app script) * `appliers_test.lua`: fixed the applier call (`.apply` -> `.post_apply`) * `config_test.lua`: fixed status observed in the app script (`*_in_progress` -> `ready`) Part of tarantool#8862 NO_DOC=reflected in tarantool/doc#3544

The usual environment configuration source is useful for parametrized run: ``` TT_MEMTX_MEMORY=<...> tarantool --name <...> --config <...> ``` However, sometimes a user may need to set a default value, which doesn't rewrite one that is provided in the configuration. Now, it is possible to do using the environment variables with the `_DEFAULT` suffix. ``` TT_MEMTX_MEMORY_DEFAULT=<...> tarantool --name <...> --config <...> ``` This feature may be especially useful for wrappers that run tarantool internally with some default paths for data directories, socket files, pid file. We likely will use it in the `tt` tool. Part of tarantool#8862 NO_DOC=added into tarantool/doc#3544 manually

It lists the environment variables that are taken into account by the config module. TBD: Part of #xxxx NO_DOC=reflected in tarantool/doc#3544

The usual environment configuration source is useful for parametrized run: ``` TT_MEMTX_MEMORY=<...> tarantool --name <...> --config <...> ``` However, sometimes a user may need to set a default value, which doesn't rewrite one that is provided in the configuration. Now, it is possible to do using the environment variables with the `_DEFAULT` suffix. ``` TT_MEMTX_MEMORY_DEFAULT=<...> tarantool --name <...> --config <...> ``` This feature may be especially useful for wrappers that run tarantool internally with some default paths for data directories, socket files, pid file. We likely will use it in the `tt` tool. Part of tarantool#8862 NO_DOC=added into tarantool/doc#3544 manually

The usual environment configuration source is useful for parametrized run: ``` TT_MEMTX_MEMORY=<...> tarantool --name <...> --config <...> ``` However, sometimes a user may need to set a default value, which doesn't rewrite one that is provided in the configuration. Now, it is possible to do using the environment variables with the `_DEFAULT` suffix. ``` TT_MEMTX_MEMORY_DEFAULT=<...> tarantool --name <...> --config <...> ``` This feature may be especially useful for wrappers that run tarantool internally with some default paths for data directories, socket files, pid file. We likely will use it in the `tt` tool. Part of #8862 NO_DOC=added into tarantool/doc#3544 manually

It is convenient to access configuration using `config:get()` from the application script (`app.file` or `app.module`). However, before this commit, it was not possible, because the configuration was not considered as applied before the application script is loaded. Now, the script loading is moved into the post-apply phase. The resulting sequence of steps on startup/reload is the following. * collect configuration information (from all the sources) * <if the previous step is failed, set `check_errors` status and break> * apply the configuration (call all the appliers) * <if the previous step is failed, set `check_errors` status and break> * <set the new successful status: `ready` or `check_warnings`> * call post-apply hooks (including application script loading) * <if the previous step is failed, set `check_errors` status and break> * <set the new successful status: `ready` or `check_warnings`> I would like to briefly comment the changes in the tests. * `app_test.lua`: added the check for the new behavior (call config:get() from the app script) * `appliers_test.lua`: fixed the applier call (`.apply` -> `.post_apply`) * `config_test.lua`: fixed status observed in the app script (`*_in_progress` -> `ready`) Part of tarantool#8862 NO_DOC=reflected in tarantool/doc#3544

It is convenient to access configuration using `config:get()` from the application script (`app.file` or `app.module`). However, before this commit, it was not possible, because the configuration was not considered as applied before the application script is loaded. Now, the script loading is moved into the post-apply phase. The resulting sequence of steps on startup/reload is the following. * collect configuration information (from all the sources) * <if the previous step is failed, set `check_errors` status and break> * apply the configuration (call all the appliers) * <if the previous step is failed, set `check_errors` status and break> * <set the new successful status: `ready` or `check_warnings`> * call post-apply hooks (including application script loading) * <if the previous step is failed, set `check_errors` status and break> * <set the new successful status: `ready` or `check_warnings`> I would like to briefly comment the changes in the tests. * `app_test.lua`: added the check for the new behavior (call config:get() from the app script) * `appliers_test.lua`: fixed the applier call (`.apply` -> `.post_apply`) * `config_test.lua`: fixed status observed in the app script (`*_in_progress` -> `ready`) Part of #8862 NO_DOC=reflected in tarantool/doc#3544

This patch introduces all metrics configuration. Part of tarantool#8861 NO_DOC=tarantool/doc#3544 links the most actual schema, no need to update the issue.

It is convenient to access configuration using `config:get()` from the application script (`app.file` or `app.module`). However, before this commit, it was not possible, because the configuration was not considered as applied before the application script is loaded. Now, the script loading is moved into the post-apply phase. The resulting sequence of steps on startup/reload is the following. * collect configuration information (from all the sources) * <if the previous step is failed, set `check_errors` status and break> * apply the configuration (call all the appliers) * <if the previous step is failed, set `check_errors` status and break> * <set the new successful status: `ready` or `check_warnings`> * call post-apply hooks (including application script loading) * <if the previous step is failed, set `check_errors` status and break> * <set the new successful status: `ready` or `check_warnings`> I would like to briefly comment the changes in the tests. * `app_test.lua`: added the check for the new behavior (call config:get() from the app script) * `appliers_test.lua`: fixed the applier call (`.apply` -> `.post_apply`) * `config_test.lua`: fixed status observed in the app script (`*_in_progress` -> `ready`) Part of tarantool#8862 NO_DOC=reflected in tarantool/doc#3544

This patch introduces all metrics configuration. Part of tarantool#8861 NO_DOC=tarantool/doc#3544 links the most actual schema, no need to update the issue.

It is convenient to access configuration using `config:get()` from the application script (`app.file` or `app.module`). However, before this commit, it was not possible, because the configuration was not considered as applied before the application script is loaded. Now, the script loading is moved into the post-apply phase. The resulting sequence of steps on startup/reload is the following. * collect configuration information (from all the sources) * <if the previous step is failed, set `check_errors` status and break> * apply the configuration (call all the appliers) * <if the previous step is failed, set `check_errors` status and break> * <set the new successful status: `ready` or `check_warnings`> * call post-apply hooks (including application script loading) * <if the previous step is failed, set `check_errors` status and break> * <set the new successful status: `ready` or `check_warnings`> I would like to briefly comment the changes in the tests. * `app_test.lua`: added the check for the new behavior (call config:get() from the app script) * `appliers_test.lua`: fixed the applier call (`.apply` -> `.post_apply`) * `config_test.lua`: fixed status observed in the app script (`*_in_progress` -> `ready`) Part of tarantool#8862 NO_DOC=reflected in tarantool/doc#3544

It is convenient for development environments, when the configuration file and the application sources reside in the same directory. The same logic was recently implemented for the main script, see #8182. The same problems appears in context of startup from a configuration file, so it seems meaningful to adjust module search paths in this case too. Part of #8862 NO_DOC=This change is too minor to describe in the documentation issue tarantool/doc#3544. I'll work with the documentation team regarding details of startup/reload flow and we'll determine what should go to the user documentation and what shouldn't.

All user-defined users and roles are not being removed and their privileges are not being revoked when this user or role is removed from config. This is done to prevent extreme repercussions of misconfiguration, e.g. empty config is provided to cluster and it breaks up. Default users and roles are not supposed to be changed, so this rule does not apply to them. Now all of non-default privileges will be revoked if such user or role is removed from config. Default users: * guest * admin Default roles: * super * public * replication Part of tarantool#8967 NO_DOC=tarantool/doc#3544 links the most actual schema, no need to update the issue.

andreyaksenov · 2023-12-21T10:42:20Z

Details

Demo:

Recording: https://cloud.mail.ru/public/3KEi/gnVzVELJL
Notes: https://www.notion.so/tarantool/Getting-Started-with-3-0-03d461efc01c43dd9e09f2bc4903717b

Demo 2024-08-24 (3.0.0-alpha2)

Recording: https://cloud.mail.ru/public/ANBh/WPUNQk8Wg
Slides: https://docs.google.com/presentation/d/1nGGOLzVik4vGppN9jpbLXsXCMl-M4j9--ZC_nEkyHb0/edit?usp=sharing
Sharding example code: https://gist.github.com/Totktonada/6339517a32cd7df4ac7df700fc3fa6b8

State diagram

stateDiagram-v2
    pre_ready: ready
    pre_check_warnings: check_warnings
    pre_check_errors: check_errors
    wait_for_reload: wait for config#colon;reload()
    note left of pre_check_errors: waits for config#colon;reload()

    [*] --> startup_in_progress: broadcast
    startup_in_progress --> _collect

    wait_for_reload --> reload_in_progress: broadcast
    reload_in_progress --> _collect

    _collect --> _apply

    state _apply {
        [*] --> pre_ready: broadcast
        [*] --> pre_check_warnings: broadcast
        [*] --> pre_check_errors: broadcast
    }

    pre_ready --> _post_apply
    pre_check_warnings --> _post_apply
    _apply --> wait_for_reload

    state _post_apply {
        [*] --> ready: broadcast
        [*] --> check_warnings: broadcast
        [*] --> check_errors: broadcast
    }

    _post_apply --> wait_for_reload

veod32 added epic 3.0 labels Jun 26, 2023

Totktonada mentioned this issue Jul 20, 2023

config: add security section to schema tarantool/tarantool#8883

Merged

Totktonada mentioned this issue Jul 28, 2023

config: allow to call config:get() from app script tarantool/tarantool#8929

Merged

Totktonada added a commit to Totktonada/tarantool that referenced this issue Jul 31, 2023

config: add --help-env-list CLI option

63f3642

It lists the environment variables that are taken into account by the config module. TBD: Part of #xxxx NO_DOC=reflected in tarantool/doc#3544

Totktonada added a commit to Totktonada/tarantool that referenced this issue Jul 31, 2023

config: add --help-env-list CLI option

0d1fae7

It lists the environment variables that are taken into account by the config module. TBD: Part of #xxxx NO_DOC=reflected in tarantool/doc#3544

Lord-KA added a commit to Lord-KA/tarantool that referenced this issue Aug 3, 2023

config: add metrics section to schema

fc2478a

This patch introduces all metrics configuration. Part of tarantool#8861 NO_DOC=tarantool/doc#3544 links the most actual schema, no need to update the issue.

Lord-KA mentioned this issue Aug 3, 2023

config: add metrics section to schema tarantool/tarantool#8943

Merged

Lord-KA added a commit to Lord-KA/tarantool that referenced this issue Aug 3, 2023

config: add metrics section to schema

827a8d7

This patch introduces all metrics configuration. Part of tarantool#8861 NO_DOC=tarantool/doc#3544 links the most actual schema, no need to update the issue.

veod32 added the config label Aug 28, 2023

This was referenced Aug 31, 2023

feedback: Launching an application | Tarantool #3627

Closed

[Config] Configuration in a file #3651

Closed

[Config] Configuration in code (legacy) #3652

Closed

This was referenced Sep 6, 2023

config: revoke privs for default users and roles tarantool/tarantool#9091

Merged

Roles #3702

Closed

tarantool deleted a comment from andreyaksenov Sep 8, 2023

locker mentioned this issue Sep 19, 2023

config: add security.auth_retries option tarantool/tarantool#9160

Merged

veod32 removed their assignment Sep 21, 2023

andreyaksenov mentioned this issue Jun 23, 2023

feedback: Настройка синхронной репликации | Tarantool #3536

Open

andreyaksenov changed the title ~~[Epic] 3.0 config~~ [Epic] 3.0 config - Part 1 Dec 21, 2023

andreyaksenov mentioned this issue Dec 21, 2023

[Epic] 3.0 config - Part 2 #3957

Closed

14 tasks

andreyaksenov closed this as completed Dec 28, 2023

TarantoolBot mentioned this issue Aug 15, 2024

feedback: Module config | Tarantool #4450

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Epic] 3.0 config - Part 1 #3544

[Epic] 3.0 config - Part 1 #3544

veod32 commented Jun 26, 2023 •

edited by andreyaksenov

Loading

andreyaksenov commented Dec 21, 2023 •

edited

Loading

[Epic] 3.0 config - Part 1 #3544

[Epic] 3.0 config - Part 1 #3544

Comments

veod32 commented Jun 26, 2023 • edited by andreyaksenov Loading

Doc issues

Getting Started

Concepts

How-to

Administration

Reference

Misc

andreyaksenov commented Dec 21, 2023 • edited Loading

Details

Demo:

Demo 2024-08-24 (3.0.0-alpha2)

State diagram

veod32 commented Jun 26, 2023 •

edited by andreyaksenov

Loading

andreyaksenov commented Dec 21, 2023 •

edited

Loading