Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pbckp 690 update readme #38

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions AUTHORS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Authors

This list contains project contributors: developers, QA or tech writers.

Alexey Kondratov [@ololobus](https://github.com/ololobus)
Yura Sokolov [@funny-falcon](https://github.com/funny-falcon)
Anastasia Lubennikova [@alubennikova](https://github.com/alubennikova)
Mikhail Kulagin [@kulaginm](https://github.com/kulaginm)
Daniel Shelepanov [@daniel-95](https://github.com/daniel-95)
Maxim Orlov [@ziva777](https://github.com/ziva777)
Dmitry Koval [@kovdb75](https://github.com/kovdb75)
Marina Polyakova [@MarinaPolyakova](https://github.com/MarinaPolyakova)
Roman Zharkov [@rzharkov](https://github.com/rzharkov)
Victor Wagner [@vbwagner](https://github.com/vbwagner)
vegebird [@vegebird](https://github.com/vegebird)
waaeer [@waaeer](https://github.com/waaeer)
61 changes: 33 additions & 28 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,22 @@
[![Build Status](https://travis-ci.com/postgrespro/ptrack.svg?branch=master)](https://travis-ci.com/postgrespro/ptrack)
[![codecov](https://codecov.io/gh/postgrespro/ptrack/branch/master/graph/badge.svg)](https://codecov.io/gh/postgrespro/ptrack)
[![GitHub release](https://img.shields.io/github/v/release/postgrespro/ptrack?include_prereleases)](https://github.com/postgrespro/ptrack/releases/latest)

# ptrack
## PTRACK

## Overview
## Overview

Ptrack is a block-level incremental backup engine for PostgreSQL. You can [effectively use](https://postgrespro.github.io/pg_probackup/#pbk-setting-up-ptrack-backups) `ptrack` engine for taking incremental backups with [pg_probackup](https://github.com/postgrespro/pg_probackup) backup and recovery manager for PostgreSQL.
PTRACK allows speed up incremental backups for the huge PostgreSQL databases. PTRACK store changes of physical blocks in the memory. You can [effectively use](https://postgrespro.github.io/pg_probackup/#pbk-setting-up-ptrack-backups) `PTRACK` engine for taking incremental backups by [pg_probackup](https://github.com/postgrespro/pg_probackup).

It is designed to allow false positives (i.e. block/page is marked in the `ptrack` map, but actually has not been changed), but to never allow false negatives (i.e. loosing any `PGDATA` changes, excepting hint-bits).
Current patch are available for PostgreSQL [11](https://github.com/postgrespro/ptrack/blob/master/patches/REL_11_STABLE-ptrack-core.diff), [12](https://github.com/postgrespro/ptrack/blob/master/patches/REL_12_STABLE-ptrack-core.diff), [13](https://github.com/postgrespro/ptrack/blob/master/patches/REL_13_STABLE-ptrack-core.diff), [14](https://github.com/postgrespro/ptrack/blob/master/patches/REL_14_STABLE-ptrack-core.diff), [15](https://github.com/postgrespro/ptrack/blob/master/patches/REL_15_STABLE-ptrack-core.diff)

Currently, `ptrack` codebase is split between small PostgreSQL core patch and extension. All public SQL API methods and main engine are placed in the `ptrack` extension, while the core patch contains only certain hooks and modifies binary utilities to ignore `ptrack.map.*` files.
## Enterprise edition

Enterprise PTRACK are part of [Postgres Pro Enterprise](https://postgrespro.ru/products/postgrespro/enterprise) and offers the capability to track more than 100 000 tables and indexes concurrently without any degradation in speed [CFS (compressed file system)](https://postgrespro.ru/docs/enterprise/current/cfs).
According to benchmarks, it operated up to 5 times faster and useful for ERP and DWH with huge amount of tables and relations between them.

This extension is compatible with PostgreSQL [11](https://github.com/postgrespro/ptrack/blob/master/patches/REL_11_STABLE-ptrack-core.diff), [12](https://github.com/postgrespro/ptrack/blob/master/patches/REL_12_STABLE-ptrack-core.diff), [13](https://github.com/postgrespro/ptrack/blob/master/patches/REL_13_STABLE-ptrack-core.diff), [14](https://github.com/postgrespro/ptrack/blob/master/patches/REL_14_STABLE-ptrack-core.diff).

## Installation

1) Get latest `ptrack` sources:
1) Get latest `PTRACK` sources:

```shell
git clone https://github.com/postgrespro/ptrack.git
@@ -43,23 +43,23 @@ echo "shared_preload_libraries = 'ptrack'" >> postgres_data/postgresql.conf
echo "ptrack.map_size = 64" >> postgres_data/postgresql.conf
```

6) Compile and install `ptrack` extension
6) Compile and install `PTRACK` extension

```shell
USE_PGXS=1 make -C /path/to/ptrack/ install
```

7) Run PostgreSQL and create `ptrack` extension
7) Run PostgreSQL and create `PTRACK` extension

```sql
postgres=# CREATE EXTENSION ptrack;
```

## Configuration

The only one configurable option is `ptrack.map_size` (in MB). Default is `0`, which means `ptrack` is turned off. In order to reduce number of false positives it is recommended to set `ptrack.map_size` to `1 / 1000` of expected `PGDATA` size (i.e. `1000` for a 1 TB database).
The only one configurable option is `ptrack.map_size` (in MB). Default is `0`, which means `PTRACK` is turned off. In order to reduce number of false positives it is recommended to set `ptrack.map_size` to `1 / 1000` of expected `PGDATA` size (i.e. `1000` for a 1 TB database).

To disable `ptrack` and clean up all remaining service files set `ptrack.map_size` to `0`.
To disable `PTRACK` and clean up all remaining service files set `ptrack.map_size` to `0`.

## Public SQL API

@@ -102,7 +102,7 @@ postgres=# SELECT * FROM ptrack_get_change_stat('0/285C8C8');

## Upgrading

Usually, you have to only install new version of `ptrack` and do `ALTER EXTENSION 'ptrack' UPDATE;`. However, some specific actions may be required as well:
Usually, you have to only install new version of `PTRACK` and do `ALTER EXTENSION 'ptrack' UPDATE;`. However, some specific actions may be required as well:

#### Upgrading from 2.0.0 to 2.1.*:

@@ -113,7 +113,7 @@ Usually, you have to only install new version of `ptrack` and do `ALTER EXTENSIO

#### Upgrading from 2.1.* to 2.2.*:

Since version 2.2 we use a different algorithm for tracking changed pages. Thus, data recorded in the `ptrack.map` using pre 2.2 versions of `ptrack` is incompatible with newer versions. After extension upgrade and server restart old `ptrack.map` will be discarded with `WARNING` and initialized from the scratch.
Since version 2.2 we use a different algorithm for tracking changed pages. Thus, data recorded in the `ptrack.map` using pre 2.2 versions of `PTRACK` is incompatible with newer versions. After extension upgrade and server restart old `ptrack.map` will be discarded with `WARNING` and initialized from the scratch.

#### Upgrading from 2.2.* to 2.3.*:

@@ -126,29 +126,33 @@ Since version 2.2 we use a different algorithm for tracking changed pages. Thus,
#### Upgrading from 2.3.* to 2.4.*:

* Stop your server
* Update ptrack binaries
* Update `PTRACK` binaries
* Start server
* Do `ALTER EXTENSION 'ptrack' UPDATE;`.

## Limitations

1. You can only use `ptrack` safely with `wal_level >= 'replica'`. Otherwise, you can lose tracking of some changes if crash-recovery occurs, since [certain commands are designed not to write WAL at all if wal_level is minimal](https://www.postgresql.org/docs/12/populate.html#POPULATE-PITR), but we only durably flush `ptrack` map at checkpoint time.
1. You can only use `PTRACK` safely with `wal_level >= 'replica'`. Otherwise, you can lose tracking of some changes if crash-recovery occurs, since [certain commands are designed not to write WAL at all if wal_level is minimal](https://www.postgresql.org/docs/12/populate.html#POPULATE-PITR), but we only durably flush `PTRACK` map at checkpoint time.

2. The only one production-ready backup utility, that fully supports `ptrack` is [pg_probackup](https://github.com/postgrespro/pg_probackup).
2. The only one production-ready backup utility, that fully supports `PTRACK` is [pg_probackup](https://github.com/postgrespro/pg_probackup).

3. You cannot resize `ptrack` map in runtime, only on postmaster start. Also, you will loose all tracked changes, so it is recommended to do so in the maintainance window and accompany this operation with full backup.
3. You cannot resize `PTRACK` map in runtime, only on postmaster start. Also, you will lose all tracked changes, so it is recommended to do so in the maintainance window and accompany this operation with full backup.

4. You will need up to `ptrack.map_size * 2` of additional disk space, since `ptrack` uses additional temporary file for durability purpose. See [Architecture section](#Architecture) for details.
4. You will need up to `ptrack.map_size * 2` of additional disk space, since `PTRACK` uses additional temporary file for durability purpose. See [Architecture section](#Architecture) for details.

## Benchmarks

Briefly, an overhead of using `ptrack` on TPS usually does not exceed a couple of percent (~1-3%) for a database of dozens to hundreds of gigabytes in size, while the backup time scales down linearly with backup size with a coefficient ~1. It means that an incremental `ptrack` backup of a database with only 20% of changed pages will be 5 times faster than a full backup. More details [here](benchmarks).
Briefly, an overhead of using `PTRACK` on TPS usually does not exceed a couple of percent (~1-3%) for a database of dozens to hundreds of gigabytes in size, while the backup time scales down linearly with backup size with a coefficient ~1. It means that an incremental `PTRACK` backup of a database with only 20% of changed pages will be 5 times faster than a full backup. More details [here](benchmarks).

## Architecture

We use a single shared hash table in `ptrack`. Due to the fixed size of the map there may be false positives (when some block is marked as changed without being actually modified), but not false negative results. However, these false postives may be completely eliminated by setting a high enough `ptrack.map_size`.
It is designed to permit false positives (i.e., block/page is marked as altered in the `PTRACK` map when it hasn't actually been changed), but it never tolerates false negatives (i.e., it never loses any PGDATA modifications, barring hint-bits).

At present, the PTRACK codebase is divided between a small PostgreSQL core patch and an extension. The public SQL API methods and the main engine are housed in the `PTRACK` extension, whereas the core patch only includes specific hooks and modifies binary utilities to disregard ptrack.map.* files.

All reads/writes are made using atomic operations on `uint64` entries, so the map is completely lockless during the normal PostgreSQL operation. Because we do not use locks for read/write access, `ptrack` keeps a map (`ptrack.map`) since the last checkpoint intact and uses up to 1 additional temporary file:
In `PTRACK`, we use a single shared hash table. Due to the fixed size of the map, there can be false positives (when a block is marked as changed without actual modification), but false negatives are not allowed. Nevertheless, these false positives can be completely removed by setting a sufficiently high ptrack.map_size.

All reads/writes are performed using atomic operations on uint64 entries, making the map completely lockless during standard PostgreSQL operation. Since we do not utilize locks for read/write access, `PTRACK` maintains a map (ptrack.map) from the last checkpoint unaltered and uses a maximum of one additional temporary file.

* temporary file `ptrack.map.tmp` to durably replace `ptrack.map` during checkpoint.

@@ -158,7 +162,8 @@ To gather the whole changeset of modified blocks in `ptrack_get_pagemapset()` we

## Contribution

Feel free to [send pull requests](https://github.com/postgrespro/ptrack/compare), [fill up issues](https://github.com/postgrespro/ptrack/issues/new), or just reach one of us directly (e.g. <[Alexey Kondratov](mailto:a.kondratov@postgrespro.ru?subject=[GitHub]%20Ptrack), [@ololobus](https://github.com/ololobus)>) if you are interested in `ptrack`.
Feel free to [send pull requests](https://github.com/postgrespro/ptrack/compare), [fill up issues](https://github.com/postgrespro/ptrack/issues/new).
See also the list of [authors](AUTHORS.md) who participated in this project.

### Tests

@@ -177,8 +182,8 @@ docker-compose run tests

Available test modes (`MODE`) are `basic` (default) and `paranoia` (per-block checksum comparison of `PGDATA` content before and after backup-restore process). Available test cases (`TEST_CASE`) are `tap` (minimalistic PostgreSQL [tap test](https://github.com/postgrespro/ptrack/blob/master/t/001_basic.pl)), `all` or any specific [pg_probackup test](https://github.com/postgrespro/pg_probackup/blob/master/tests/ptrack.py), e.g. `test_ptrack_simple`.

### TODO
## Acknowledgments

PTRACK development is supported by Postgres Professional
You can ask any question about contributing or usage in [Russian chat](https://t.me/pg_probackup) or [Internation chat](https://t.me/pg_probackup_eng) of [pg_probackup](https://github.com/postgrespro/pg_probackup).

* Should we introduce `ptrack.map_path` to allow `ptrack` service files storage outside of `PGDATA`? Doing that we will avoid patching PostgreSQL binary utilities to ignore `ptrack.map.*` files.
* Can we resize `ptrack` map on restart but keep the previously tracked changes?
* Can we write a formal proof, that we never loose any modified page with `ptrack`? With TLA+?