Skip to content

Commit 93f606d

Browse files
committed
replication.synchro_queue_max_size configuration option added
Fixes #4749
1 parent 52cbbf7 commit 93f606d

File tree

1 file changed

+177
-12
lines changed

1 file changed

+177
-12
lines changed

doc/reference/configuration/configuration_reference.rst

+177-12
Original file line numberDiff line numberDiff line change
@@ -3544,6 +3544,7 @@ The ``replication`` section defines configuration parameters related to :ref:`re
35443544

35453545
- :ref:`replication.anon <configuration_reference_replication_anon>`
35463546
- :ref:`replication.autoexpel <configuration_reference_replication_autoexpel>`
3547+
- :ref:`replication.anon <configuration_reference_replication_autoexpel>`
35473548
- :ref:`replication.bootstrap_strategy <configuration_reference_replication_bootstrap_strategy>`
35483549
- :ref:`replication.connect_timeout <configuration_reference_replication_connect_timeout>`
35493550
- :ref:`replication.election_mode <configuration_reference_replication_election_mode>`
@@ -3554,6 +3555,7 @@ The ``replication`` section defines configuration parameters related to :ref:`re
35543555
- :ref:`replication.skip_conflict <configuration_reference_replication_skip_conflict>`
35553556
- :ref:`replication.sync_lag <configuration_reference_replication_sync_lag>`
35563557
- :ref:`replication.sync_timeout <configuration_reference_replication_sync_timeout>`
3558+
- :ref:`replication.synchro_queue_max_size <configuration_reference_replication_synchro_queue_max_size`>
35573559
- :ref:`replication.synchro_quorum <configuration_reference_replication_synchro_quorum>`
35583560
- :ref:`replication.synchro_timeout <configuration_reference_replication_synchro_timeout>`
35593561
- :ref:`replication.threads <configuration_reference_replication_threads>`
@@ -3618,6 +3620,13 @@ The ``replication`` section defines configuration parameters related to :ref:`re
36183620

36193621
The expulsion process follows the standard procedure, involving the removal of the instance from the ``_cluster`` system space.
36203622

3623+
The ``autoexpel`` logic is activated during specific events:
3624+
3625+
- **Startup**. When the cluster starts, ``autoexpel`` checks and removes instances not matching the updated configuration.
3626+
- **Reconfiguration**. When the YAML configuration is reloaded, ``autoexpel`` compares the current state to the updated configuration and performs necessary expulsions.
3627+
- ``box.status`` **watcher event**. Changes detected by the ``box.status`` watcher also trigger the ``autoexpel`` mechanism.
3628+
3629+
36213630
``autoexpel`` does not take any actions on newly joined instances unless one of the triggering events occurs.
36223631
This means that an instance meeting the ``autoexpel`` criterion can still join the cluster, but it may be removed
36233632
later during reconfiguration or on subsequent triggering events.
@@ -3627,8 +3636,7 @@ The ``replication`` section defines configuration parameters related to :ref:`re
36273636
global levels. It is not applicable at the instance level.
36283637

36293638

3630-
**Configuration fields**
3631-
3639+
Configuration fields
36323640

36333641
- ``by`` (string, default: ``nil``): specifies the ``autoexpel`` criterion. Currently, only ``prefix`` is supported and must be explicitly set.
36343642

@@ -3638,7 +3646,7 @@ The ``replication`` section defines configuration parameters related to :ref:`re
36383646

36393647

36403648

3641-
replication.autoexpel.by
3649+
replication.autoexpel_by.*
36423650
~~~~~~~~~~~~~
36433651

36443652
``replication.autoexpel_by`` purpose is to define the criterion used for determining which instances in a cluster are
@@ -3648,15 +3656,15 @@ replication.autoexpel.by
36483656

36493657
- Instances that are part of the cluster and should adhere to the YAML configuration.
36503658

3651-
- Instances or tools (e.g., CDC tools) that use the replication channel but are not part of the cluster configuration.
3659+
- Instances or tools (e.g., CDC tools) that use the replication channel but are not part of the cluster configuration.
36523660

36533661

3654-
The default value of ``by`` is ``nil``, meaning no ``autoexpel`` criterion is applied unless explicitly set.
3662+
The default value of by is ``nil``, meaning no ``autoexpel`` criterion is applied unless explicitly set.
36553663

36563664
Currently, the only supported value for by is ``prefix``. The ``prefix`` value instructs the system to identify instances
36573665
based on their names, matching them against a prefix pattern defined in the configuration.
36583666

3659-
If the ``autoexpel`` feature is enabled, the ``by`` field must be explicitly set to ``prefix``.
3667+
If the ``autoexpel`` feature is enabled (``enabled: true``), the ``by`` field must be explicitly set to ``prefix``.
36603668

36613669
The absence of this field or an unsupported value will result in configuration errors.
36623670

@@ -3675,7 +3683,7 @@ replication.autoexpel.by
36753683
36763684

36773685

3678-
replication.autoexpel.enabled
3686+
replication.autoexpel_enabled.*
36793687
~~~~~~~~~~~~~
36803688

36813689
The ``replication.autoexpel_enabled`` field is a boolean configuration option that determines whether the autoexpel logic is active for the cluster.
@@ -3685,7 +3693,7 @@ replication.autoexpel.enabled
36853693

36863694
By default, the ``enabled`` field is set to ``false``, meaning the ``autoexpel`` logic is turned off. This ensures that no instances are automatically removed unless explicitly configured.
36873695

3688-
**Enabling ``autoexpel`` logic**
3696+
Enabling ``autoexpel`` logic
36893697

36903698
To enable ``autoexpel``, you should set enabled to true in the ``replication.autoexpel`` section of your YAML configuration:
36913699

@@ -3701,8 +3709,7 @@ replication.autoexpel.enabled
37013709
To disable ``autoexpel``, set enabled to ``false``.
37023710

37033711

3704-
**Dependencies**
3705-
3712+
Dependencies
37063713

37073714
If ``enabled`` is set to ``true``, the following fields are required:
37083715

@@ -3717,7 +3724,7 @@ replication.autoexpel.enabled
37173724
| Environment variable: TT_REPLICATION_AUTOEXPEL_ENABLED
37183725
37193726

3720-
replication.autoexpel.prefix
3727+
replication.autoexpel_prefix.*
37213728
~~~~~~~~~~~~~
37223729

37233730
The ``prefix`` field filters instances for expulsion by differentiating cluster instances (from the YAML configuration) from external services (e.g., CDC tools). Only instances matching the prefix are considered.
@@ -3748,7 +3755,7 @@ replication.autoexpel.prefix
37483755
37493756
In this setup:
37503757

3751-
- Instances are prefixed with a replicaset name (e.g., ``r-001-i-001`` for replicaset ``r-001``).
3758+
- Instances are grouped by replicaset names (e.g., ``r-001-i-001`` for ``replicaset r-001``).
37523759
- The prefix ensures that only instances with names matching the replicaset name are auto expelled when removed from the configuration.
37533760

37543761

@@ -3773,6 +3780,111 @@ replication.autoexpel.prefix
37733780
| Environment variable: TT_REPLICATION_AUTOEXPEL_PREFIX
37743781
37753782

3783+
3784+
autoexpel full example
3785+
~~~~~~~~~~~~~
3786+
3787+
1. Create a ``config.yaml`` file with the following content:
3788+
3789+
.. code-block:: yaml
3790+
3791+
credentials:
3792+
users:
3793+
guest:
3794+
roles: [super]
3795+
3796+
replication:
3797+
failover: manual
3798+
autoexpel:
3799+
enabled: true
3800+
by: prefix
3801+
prefix: '{{ replicaset_name }}'
3802+
3803+
iproto:
3804+
listen:
3805+
- uri: 'unix/:./var/run/{{ instance_name }}.iproto'
3806+
3807+
groups:
3808+
g-001:
3809+
replicasets:
3810+
r-001:
3811+
leader: r-001-i-001
3812+
instances:
3813+
r-001-i-001: {}
3814+
r-001-i-002: {}
3815+
r-001-i-003: {}
3816+
3817+
3818+
This configuration:
3819+
- Sets up authentication with a guest user assigned the super role.
3820+
- Enables the ``autoexpel`` option to automatically expel instances not present in the YAML file.
3821+
- Defines instance names based on a prefix pattern: ``{{ replicaset_name }}``.
3822+
- Lists three instances: ``r-001-i-001``, ``r-001-i-002``, and ``r-001-i-003``.
3823+
3824+
3825+
2. Open terminal window and start three instances using the following commands:
3826+
3827+
.. code-block:: lua
3828+
3829+
tarantool --name r-001-i-001 --config config.yaml -i
3830+
3831+
3832+
.. code-block:: lua
3833+
3834+
tarantool --name r-001-i-002 --config config.yaml -i
3835+
3836+
3837+
.. code-block:: lua
3838+
3839+
tarantool --name r-001-i-003 --config config.yaml -i
3840+
3841+
3. Edit ``config.yaml`` and remove the following entry for ``r-001-i-003``:
3842+
3843+
.. code-block:: lua
3844+
r-001-i-003: {}
3845+
3846+
3847+
The updated ``config.yaml`` should look like this:
3848+
3849+
.. code-block:: yaml
3850+
3851+
groups:
3852+
g-001:
3853+
replicasets:
3854+
r-001:
3855+
leader: r-001-i-001
3856+
instances:
3857+
r-001-i-001: {}
3858+
r-001-i-002: {}
3859+
3860+
Save the file.
3861+
3862+
4. For the leader instance (``r-001-i-001``), check the ``_cluster`` space:
3863+
3864+
.. hint::
3865+
3866+
The ``_cluster`` system space in Tarantool stores metadata about all instances currently recognized as part of the cluster.
3867+
It shows which instances are registered and active.
3868+
3869+
You should see ``r-001-i-003`` still listed in the ``_cluster`` system space.
3870+
3871+
5. Reload the configuration:
3872+
3873+
.. code-block:: lua
3874+
3875+
config = require('config')
3876+
config:reload()
3877+
3878+
6. Verify the changes:
3879+
3880+
.. code-block:: lua
3881+
3882+
box.space._cluster:fselect()
3883+
3884+
After the reload, ``r-001-i-003`` should no longer appear in the ``_cluster`` system space.
3885+
3886+
3887+
37763888
.. _configuration_reference_replication_bootstrap_strategy:
37773889

37783890
.. confval:: replication.bootstrap_strategy
@@ -4034,6 +4146,59 @@ replication.autoexpel.prefix
40344146
| Default: 0
40354147
| Environment variable: TT_REPLICATION_SYNC_TIMEOUT
40364148
4149+
4150+
.. _configuration_reference_replication_synchro_queue_max_size:
4151+
4152+
.. confval:: replication.synchro_queue_max_size
4153+
4154+
**Since:** :doc:`3.3.0 </release/3.3.0>`
4155+
4156+
The maximum size of the synchronous transaction queue on a master node, in bytes. The size limit isn't strict, i.e. if there's at least one free byte, the whole write request fits and no blocking is involved.
4157+
This parameter ensures that the queue does not grow indefinitely, potentially impacting performance and resource usage, and applies only to the master node.
4158+
4159+
The ``0`` value disables the limit.
4160+
4161+
4162+
If the synchronous queue reaches the configured size limit, new transactions attempting to enter the queue are discarded.
4163+
In such cases, the system returns an error to the user:
4164+
``The synchronous transaction queue is full.``
4165+
4166+
This size limitation does not apply during the recovery process. Transactions processed during recovery are unaffected by the queue size limit.
4167+
4168+
Use the following command to view the current size of the synchronous queue:
4169+
4170+
.. code-block:: lua
4171+
4172+
box.info.synchro.queue.size
4173+
4174+
4175+
.. code-block:: command line
4176+
4177+
tarantool> box.info.synchro
4178+
---
4179+
- queue:
4180+
owner: 1
4181+
size: 60
4182+
busy: false
4183+
len: 1
4184+
term: 2
4185+
quorum: 2
4186+
...
4187+
4188+
Set the synchronous queue size limit in the configuration file:
4189+
4190+
.. code-block:: yaml
4191+
4192+
replication:
4193+
synchro_queue_max_size: 33554432 # Limit set to 32 MB
4194+
4195+
|
4196+
| Type: integer
4197+
| Default: 16777216 (16 MB)
4198+
| Environment variable: TT_REPLICATION_SYNCHRO_QUEUE_MAX_SIZE
4199+
4200+
4201+
40374202
.. _configuration_reference_replication_synchro_quorum:
40384203

40394204
.. confval:: replication.synchro_quorum

0 commit comments

Comments
 (0)