docs

Author: amyasnikov

README.md

@@ -15,6 +15,7 @@ Validity is the [NetBox](https://netboxlabs.com/oss/netbox/) plugin to write "au
 
 * **Configuration compliance**. You can make sure your devices are provisioned properly, and their config follows the rules you have defined via tests.
 * **Pre- / post-configuration checks**. You can make sure your network is in the expected state before or/and after configuration changes have been made. You can use Validity API to include these checks into your automation pipelines.
+* **Config Backup**. Save your device configs to **Git** or **AWS S3**
 
 **Validity usage workflow:**
 
@@ -40,7 +41,6 @@ Validity completely separates compliance test code from all the other things lik
 * Truly vendor-agnostic. You can easily integrate any vendor config format using [TTP](https://github.com/dmulyalin/ttp) or a bunch of other [serialization options](https://validity.readthedocs.io/en/latest/entities/serializers/)
 * Writing compliance tests using Python expressions and [JQ](https://stedolan.github.io/jq/manual/)
 * Gathering configuration or state info directly from the devices via **SSH**, **Telnet**, **Netconf** or **REST API**.
-* Flexible selector system to apply the tests only to a specific subset of devices
 * Concept of **dynamic pairs**. With dynamic pair you can compare 2 different devices between each other (e.g. compare the configuration of 2 MC-LAG members).
 * **Test result explanation**. When some test fails, you can get the **explanation** of the calculation process step by step. It helps to identify the cause of the failure.
 * **ORM access** inside the test. You have full access to the **device** properties. For instance, you may leverage [Configuration Contexts](https://docs.netbox.dev/en/stable/features/context-data/) NetBox feature to store your desired configuration and compare it with the config collected from the device.

docs/entities/backuppoints.md

@@ -0,0 +1,53 @@
+# Backup Points
+
+If a [Data Source](https://netboxlabs.com/docs/netbox/en/stable/models/core/datasource/) is used to download data to NetBox, **Backup Point** solves the opposite task, it can **upload** data source contents to some external storage. This is mostly suitable for performing [device configuration backup](../features/config_backup.md).
+
+Currently **Git** and **AWS S3** backup methods are available.
+
+## Fields
+
+#### Name
+The Name of the Backup Point. Must be unique.
+
+#### Data Source
+Data Source which is going to be backed up. Only Data Sources with  the type [device_polling](datasources.md) are allowed to be used here.
+
+#### Back Up After Sync
+If this param is `True`, backup will be performed each time respective Data Source is synced. For instance, you press "Sync" button on the Data Source page, sync is performed and after that Data Source contents are uploaded to the destination(s) specified in all bound Backup Points.
+
+#### Method
+Defines protocol for uploading the Data Source.
+
+Available options are:
+* Git (over http(s))
+* Amazon S3 (or any other S3-compatible service)
+
+#### URL
+URL of the endpoint where the Data Source contents will be uploaded.
+
+Examples of the URLs:
+
+| Method        | Example URL                                                |
+|---------------|------------------------------------------------------------|
+| Git           | `https://github.com/amyasnikov/my_awesome_repo`            |
+| S3 no-archive | `https://s3.eu-north-1.amazonaws.com/bucket_name/folder`   |
+| S3 archive    | `https://s3.eu-north-1.amazonaws.com/bucket_name/arch.zip` |
+
+
+#### Ignore Rules
+A set of rules (one per line) identifying filenames to ignore during Data Source backup. Some examples are provided below. See Python's [fnmatch()](https://docs.python.org/3/library/fnmatch.html) documentation for a complete reference.
+
+| Rule               | Description                                                  |
+|--------------------|--------------------------------------------------------------|
+| `folder/file.json` | Do not back up one specific file                             |
+| `*.txt`            | Do not back up all files with **txt** extension              |
+| `router-??.txt`    | Do not back up files like `router-11.txt` and `router-ab.txt` |
+
+
+#### Git-specific parameters
+* **Username**, **Password** - credentials to perform HTTP authentication, and access the repository
+* **Branch** - git branch. Default repository branch will be used if the field is empty
+
+#### S3-specific parameters
+* **AWS access key ID**, **AWS secret access key** - credentials to perform authentication in S3 service
+* **archive** - if this field is True, Data Source contents will be packed into ZIP archive and then uploaded

docs/entities/datasources.md

@@ -5,7 +5,7 @@ Data Source can be used in the following ways:
 
 * Store configuration or state data for all or particular devices
 * Be referenced by another entity containing any text information. By this moment, these entities are **Tests**, **Name Sets** and **Serializers**.
-* Data Source called **Validity Polling** is used to poll network devices and collect gathered information
+* Data Source called **Validity Polling** (with type **device_polling**) is used to poll network devices and collect gathered information
 
 ## Custom Fields
 

docs/features/config_backup.md

@@ -0,0 +1,68 @@
+# Configuration Backup
+
+Validity supports device configurations backups to remote **Git** or **S3** server(s).
+
+## Backup provisioning
+
+1. Follow the steps described in the [Quickstart: Polling](../quickstart_polling.md) article to set up all the entities required for device polling.
+
+2. Polling is done when the Data Source is synced. In most cases it will be the default **Validity Polling** Data Source, but you are free to create your own one with the type *device_polling*.
+
+3. Create a [Backup Point](../entities/backuppoints.md) which describes a place where the Data Source with the configs have to be uploaded.
+
+Example using pynetbox:
+
+```python
+import pynetbox
+
+nb = pynetbox.api('http://localhost:8000', token=nb_token)
+
+validity_polling = nb.core.data_sources.get(name="Validity Polling")
+
+nb.plugins.validity.backup_points.create(
+    name='my_github_repo',
+    data_source=data_source.id,
+    upload_url='https://github.com/amyasnikov/my_repo',
+    method='git',
+    parameters={"username": "amyasnikov", "password": github_token}
+)
+```
+
+## Backing Up
+
+There are several options to trigger the backup process:
+
+* Press "Back Up" button on the Backup Point page.
+
+* Specify `Backup After Sync: True` in the Backup Point settings. This will trigger the backup process each time the respective data source is being synced (no matter via button, API or somehow else).
+
+* Execute [RunTests](../entities/scripts.md#run-tests) script with `Sync Data Sources: True` option (`Backup After Sync: True` is required as well).
+
+
+### Options for periodic backup
+
+There are several options to provision periodic configuration backup process (e.g. daily, each 6 hours, etc):
+
+1. Provision [RunTests](../entities/scripts.md#run-tests) script to execute on a regular basis using **Interval** script parameter. This will run everything at once:
+    * poll the devices (perform data source sync)
+    * execute compliance tests
+    * back up the data source
+
+
+2. If you don't want to run the tests via scheduler and want backup only, you can create tiny [custom script](https://netboxlabs.com/docs/netbox/en/stable/customization/custom-scripts/) inside your NetBox and configure it to run periodically.
+
+```python
+from core.models import DataSource
+from extras.scripts import Script, ObjectVar
+
+class SyncDataSource(Script):
+    data_source = ObjectVar(model=DataSource)
+
+    def run(self, data, commit):
+        data['data_source'].sync()
+
+```
+
+Execution of this script triggers the sync of the DataSource (and hence the backup process for bound Backup points with `Backup After Sync: True`).
+
+3. Wait till periodic Data Source sync is implemented in the core NetBox. There is an [issue](https://github.com/netbox-community/netbox/issues/18287) for it, upvotes are appreciated.

mkdocs.yml

@@ -19,12 +19,14 @@ nav:
     - Results & Reports: entities/results_and_reports.md
     - Pollers: entities/pollers.md
     - Commands: entities/commands.md
+    - Backup Points: entities/backuppoints.md
     - Scripts: entities/scripts.md
   - Features & Tips:
     - How to write the Tests: features/howto_write_tests.md
     - How to work with Device State: features/state.md
     - Dynamic Pairs: features/dynamic_pairs.md
     - Webhooks: features/webhooks.md
+    - Config backup: features/config_backup.md
 
 markdown_extensions:
   - pymdownx.highlight: