Auto-backup to coco (feature)

[matt-deboer]

Feature Request
It would be useful if the app could perform an automated backup on an env-configurable interval.

Ideally, the format could just be coco.json, one per dataset. They could possibly all be combined into a single uber-coco.json (array) so that all the important data could be restored from a single file--or alternately, just allow multiple coco.json files to be imported together from a folder or something.

[jsbroks]

Currently the best option is to us docker to backup the volume.

I like the idea of having a option though the web client in which a admin user can control backup intervals, restore, etc.

An uber-coco file could have many issues. If users have large datasets the file can become to large and not be able to read into memory. Lazy load could be implemented but I feel that would be to much work when simpler alternatives exist.

Before a method can be determine we first must decide what features are key since this will be a big feature update.

• How should the backup, and restore process be controlled?
  • Using the web client with options in the admin/panel would be cleaner then setting environment variables (imo).
• Should images be backed up as well? Since the path is saved, if you try to restore from a backup and a image file has changes name or directory, the annotation wont be able to load.
  • The backup process will be larger (more complicated) if images are include
  • Maybe a toggle option to include images
• How should the data be backup?
  • Using docker for backups?
    • We could mount the docker.sock inside the container to control backup and restore a volume. But, I am not sure if it could handle restoring well the container is using the volume.
  • Using Mongo options to backup and restore?
    • Might be the simplistic process for handling annotation backups
  • Create coco files per dataset?
    • Worry about handling large files

Any insight from the community would be nice

[matt-deboer]

If made available via API method, it can be used in the UI and via automation tools.

A toggle option makes sense for including images.

I think a zip file containing a coco.json per dataset (and optionally, any included images) makes sense. It should compress well (at least for the coco json files), and the more that it looks like a standard format (COCO), the easier it is to use outside of this tool (as in training networks).

Also, if the backup/restore happens via COCO format files, it means that backup/restore and import/export are very close to the same process.

This could also get rid of the need for "INITIALIZE_FROM_FILE" option, if a user could just construct their own "backup" zip file with images and one or more coco.json files to bootstrap the server.

Note: my perspective is subjective based on a workflow where I don't want to maintain a long-running mongo instance, and I plan to bring-up/tear-down the annotator on an as-needed basis on my desktop. Once I'm done annotating, I want to export the annotated datasets to a cloud bucket in COCO format, and stop the annotator tool.

[jsbroks]

For your us case you can disable user authentication. I say can because I haven't tested it extensively.

[matt-deboer]

I will be your guinea pig for that feature :)

[matt-deboer]

Seems to work as expected; thanks.