PersistentVolume with external storage support for in-cluster databases

[mathieu-benoit]

Today, the in-cluster accounts-db and ledger-db databases are StatefulSets, using an emptyDir volume. As a best practice for such in-cluster scenario it would be great to have the PersistentVolume with an external storage. Maybe with Filestore like illustrated here? Or maybe another storage? That will also illustrate yet another great story with multiple Google Cloud services while showing best practices. This will allow Bank of Anthos's users (Googlers, etc.) to have a ready to use solution for such stateful workloads scenarios.

We maybe don't want to have this as default, and we may want to keep this as emptyDir volume by default (?), but maybe to start with a doc or Kustomize overlay would be a great?

Tagging @bourgeoisor @ahrarmonsur @charlieyu1996 as I'm sure they could provide more insights around this and maybe planning this for 2023?

[bourgeoisor]

We do have an "external to the cluster" database solution: https://github.com/GoogleCloudPlatform/bank-of-anthos/tree/main/extras/cloudsql

But it may be interesting to dig into the idea of "internal database" (maximum amount of customizability and control) coupled with external underlying storage.

[bourgeoisor]

This idea is definitely on the back-burner now, but it could still be an interesting alternative.

[NimJay]

Commenting since this is near out-of-SLO:

• Net-new samples is not a team priority today. Keeping this issue at p3.
• We should keep this open because it's a great idea!
• Something similar was also requested in Add database persistent storage #189.