Skip to content

backing-store-crd.md

Latest commit

 

History

History
521 lines (470 loc) · 18.4 KB

backing-store-crd.md

File metadata and controls

521 lines (470 loc) · 18.4 KB
Raw

NooBaa Operator /

BackingStore CRD

The BackingStore CRD represents a storage target to be used as underlying storage for the data in NooBaa buckets. These storage targets are used to store deduplicated, compressed and encrypted chunks of data. The encryption keys are stored separately. BackingStores are referred to by name when defining a BucketClass.

Supported BackingStore types:

  • aws-s3
  • s3-compatible
  • ibm-cos
  • google-cloud-storage
  • azure-blob
  • pv-pool

It is also possible to add new backing store types by providing a GET/PUT key-value store, see backing stores supported by NooBaa.

Definitions

Reconciliation

It is possible to create backingstores by using the NooBaa CLI tool (from here on referred to in shell commands as noobaa), or by applying YAMLs.

From here on out, fields that depend on the user's choice and input will be marked with <>. In cases where the value might not be inferrable from context, an additional explanation might be included. Note that the commands will apply to the namespace that's currently active (can be checked with kubectl config current-context). A command-specific namespace can be set by adding the -n/--namespace <NAMESPACE> flag. Also note that, if possible and applicable, omitting the access-key and secret-key flags is recommended in order to avoid leakage of secrets. When both flags are ommitted, the CLI will prompt the user to enter the keys interactively.

If the user opts to use the CLI's --secret-name option, or to apply a YAML, please see Store Secret Creation. In case of YAML application, the value under secret.namespace needs to point to the secret's namespace.

Cloud Storage Permissions

To successfully initialize a BackingStore, the provided cloud credentials must have sufficient permissions to manage the target bucket and list all buckets within the account. This listing is required for NooBaa’s external connection validation.

If permissions are insufficient, the BackingStore will remain in the Creating phase with a TemporaryError. After several retry attempts, it will ultimately transition to the Rejected phase.

AWS S3

Uses the S3 API for storing encrypted chunks of data in AWS buckets

noobaa backingstore create aws-s3 <BACKINGSTORE NAME> --access-key <> --secret-key <> --target-bucket <>
apiVersion: noobaa.io/v1alpha1
kind: BackingStore
metadata:
  finalizers:
  - noobaa.io/finalizer
  name: <>
  namespace: <>
spec:
  awsS3:
    secret:
      name: <>
      namespace: <>
    targetBucket: <>
  type: aws-s3

Permissions: the user need to have permissions in AWS account for AWS S3 service:

  1. Account level: s3:ListAllMyBuckets
  2. Bucket level: grant access on both arn:aws:s3:::<target-bucket> and arn:aws:s3:::<target-bucket>/* (bucket and object ARNs are both required).

AWS-S3 Security Token Service (STS)

Similarly to AWS-S3 this backingstore uses the S3 API for storing encrypted chunks of data in AWS buckets. However, the difference between the two backingstore types lies in the authentication method:

  • AWS S3 uses long-lived credentials - a pair of static, user-provided access keys.
  • AWS S3 STS uses short-lived credentials - the user provides an Amazon Resource Name (ARN) of a role. Under the hood, for every single interaction with AWS, we use AssumeRuleWithWebIdentity, which requests temporary security credentials. The role ARN structure is arn:aws:iam::<aws-account-id>:role/<name>.

This type of backingstore is useful in cases where the user wishes to limit access to their AWS cloud for a specific amount of time, and for easier management of the cloud's security.

Before using this backingstore, an OpenIDConnect provider needs to be set up, which is outside the scope of these docs.

noobaa backingstore create aws-sts-s3 <BACKINGSTORE NAME> --target-bucket <> --aws-sts-arn <>
apiVersion: noobaa.io/v1alpha1
kind: BackingStore
metadata:
  finalizers:
  - noobaa.io/finalizer
  name: <>
  namespace: <>
spec:
  awsS3:
    awsSTSRoleARN: <>
    targetBucket: <>
    secret:
      name: <>
      namespace: <>
  type: aws-s3

S3 Compatible

Uses the S3 API for storing encrypted chunks of data in buckets that can be interacted with via an S3-compatible API

noobaa backingstore create s3-compatible <BACKINGSTORE NAME> --endpoint <> --signature-version <> --access-key <> --secret-key <> --target-bucket <>
apiVersion: noobaa.io/v1alpha1
kind: BackingStore
metadata:
  finalizers:
  - noobaa.io/finalizer
  name: <>
  namespace: <>
spec:
  s3Compatible:
    endpoint: <>
    secret:
      name: <>
      namespace: <>
    signatureVersion: <>
    targetBucket: <>
  type: s3-compatible

IBM COS

Uses the IBM COS API for storing encrypted chunks of data in IBM COS buckets

noobaa backingstore create ibm-cos bs --endpoint <> --access-key <> --secret-key <> --target-bucket <>
apiVersion: noobaa.io/v1alpha1
kind: BackingStore
metadata:
  finalizers:
  - noobaa.io/finalizer
  name: <>
  namespace: <>
spec:
  s3Compatible:
    endpoint: <>
    secret:
      name: <>
      namespace: <>
    signatureVersion: <>
    targetBucket: <>
  type: ibm-cos

Google Cloud Storage (GCP)

Uses the Google Cloud Storage API for storing encrypted chunks of data in Google Cloud buckets

noobaa backingstore create google-cloud-storage <BACKINGSTORE NAME> --private-key-json-file <PATH TO credentials.json> --target-bucket <>
apiVersion: noobaa.io/v1alpha1
kind: BackingStore
metadata:
  finalizers:
  - noobaa.io/finalizer
  name: <>
  namespace: <>
spec:
  googleCloudStorage:
    secret:
      name: <>
      namespace: <>
    targetBucket: <>
  type: google-cloud-storage

The credentials.json is an example path to the Google Cloud service account JSON key. Service account keys are used to authenticate a service account to Google Cloud APIs. When you create a key, you download the private key, which you can then use to authenticate your application.

Permissions: the service account should have the storage.buckets.list permission. You can grant it by assigning project-level roles, for example:

  • Combination of the Browser and Storage Admin roles.
    • Browser role: provides the storage.buckets.list permission.
    • Storage Admin role: provides full control over buckets and objects.
  • The Owner role (for development; not recommended for production environments).

Google Cloud Storage Workload Identity Federation (GCP WIF, STS)

Similarly to Google Cloud Storage this backing store uses the Google Cloud Storage API for storing encrypted chunks of data in GCP buckets. However, the difference between the two backingstore types lies in the authentication method:

  • google-cloud-storage uses long-lived service_account JSON keys.
  • google-cloud-storage-sts uses a short-lived token via GCP Workload Identity Federation: the JSON type is external_account, which is built from parameters (PROJECT_NUMBER, POOL_ID, PROVIDER_ID, SERVICE_ACCOUNT_EMAIL).

This type of backingstore is useful in cases where the user wishes to limit access to their GCP cloud for a specific amount of time, and for easier management of the cloud's security.

Before using this backingstore, there are configurations that need to be done on the GCP account and on the cluster (for example, an OpenIDConnect provider needs to be set up) which is outside the scope of these docs.

noobaa backingstore create google-cloud-storage-sts <BACKINGSTORE NAME> --target-bucket <> --project-number <> --pool-id <> --provider-id <> --service-account-email <>
apiVersion: noobaa.io/v1alpha1
kind: BackingStore
metadata:
  finalizers:
  - noobaa.io/finalizer
  name: <>
  namespace: <>
spec:
  googleCloudStorage:
    secret:
      name: <>
      namespace: <>
    targetBucket: <>
  type: google-cloud-storage

Azure Blob Storage

Uses the Azure Blob API for storing encrypted chunks of data in an Azure container

noobaa backingstore create azure-blob <BACKINGSTORE NAME> --account-key <> --account-name <> --target-blob-container <>
apiVersion: noobaa.io/v1alpha1
kind: BackingStore
metadata:
  finalizers:
  - noobaa.io/finalizer
  name: <>
  namespace: <>
spec:
  azureBlob:
    secret:
      name: <>
      namespace: <>
    targetBlobContainer: <>
  type: azure-blob

Azure STS

Similarly to Azure Blob Storage this backingstore uses the Azure Blob API for storing encrypted chunks of data in Azure container. However, the difference between the two backingstore types lies in the authentication method:

  • Azure Blob Storage uses long-lived credentials - a pair of static, user-provided access keys.
  • Azure STS uses short-lived credentials . Applications will utilize client_id, tenant_id to create federated JWT token based backing store. The system will verify the Azure connection using the provided client_id, tenant_id, and federated token

This type of backingstore is useful in cases where the user wishes to limit access to their Azure cloud for a specific amount of time, and for easier management of the cloud's security.

noobaa backingstore create azure-sts-blob <BACKINGSTORE NAME> --account-name <> --target-blob-container <> --client-id <> --tenant-id <>
apiVersion: noobaa.io/v1alpha1
kind: BackingStore
metadata:
  finalizers:
  - noobaa.io/finalizer
  name: <>
  namespace: <>
  labels:
    app: noobaa
spec:
  azureBlob:
    clientId: <>
    secret:
      name: <>
      namespace: <>
    targetBlobContainer: <>
  type: azure-blob

Persistent Volume Pool

A unique kind of backingstore that uses local storage resources instead of ones provided by cloud services. Creates a StatefulSet with a PVC mounted in each pod. Each resource will connect to the NooBaa core and provide the PV filesystem storage to be used for storing encrypted chunks of data. It is possible to configure the number of pods to be used and their PV size.

Each PV needs at least 17 Gibibytes to function properly. The bare minimum defined in the source code is 16 Gibibytes, but it is better to take some extra space.

*In the YAML, the value of storage should adhere to the K8s Memory Resource Units format.

noobaa backingstore create pv-pool <BACKINGSTORE NAME> --num-volumes <> --pv-size-gb <INT >= 17> --storage-class <THE SC THAT WILL BE USED FOR PV PROVISIONING>
apiVersion: noobaa.io/v1alpha1
kind: BackingStore
metadata:
  finalizers:
  - noobaa.io/finalizer
  name: <>
  namespace: <>
spec:
  pvPool:
    numVolumes: <>
    resources:
      requests:
        storage: <SEE NOTE ABOVE*>
    storageClass: STORAGE-CLASS-NAME
  type: pv-pool

Examples

AWS S3

noobaa backingstore create aws-s3 aws-backingstore --namespace app-namespace --access-key AKIAIOSFODNN7EXAMPLE --secret-key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY --target-bucket personal-bucket
apiVersion: noobaa.io/v1alpha1
kind: BackingStore
metadata:
  finalizers:
  - noobaa.io/finalizer
  name: aws-backingstore
  namespace: app-namespace
spec:
  awsS3:
    secret:
      name: user-created-aws-s3-secret
      namespace: secret-namespace
    targetBucket: personal-bucket
  type: aws-s3

AWS S3 STS

noobaa backingstore create aws-sts-s3 aws-sts-backingstore --target-bucket personal-bucket --aws-sts-arn arn:aws:iam::111111111111:role/noobaa_sts
apiVersion: noobaa.io/v1alpha1
kind: BackingStore
metadata:
  finalizers:
  - noobaa.io/finalizer
  name: aws-sts-backingstore
  namespace: app-namespace
spec:
  awsS3:
    awsSTSRoleARN: arn:aws:iam::111111111111:role/noobaa_sts
    targetBucket: personal-bucket
    secret: {}
  type: aws-s3

AZURE STS

noobaa backingstore create azure-sts-blob azure-sts-backingstore --account-name azureSTSAccount --target-blob-container azure-sts-bs --client-id 8feb6304-7cee-40ea-b501-ef0d08520874 --tenant-id 486b6304-7cee-40qw-n507-hy7d520123
apiVersion: noobaa.io/v1alpha1
kind: BackingStore
metadata:
  finalizers:
  - noobaa.io/finalizer
  name: azure-sts-backingstore
  namespace: app-namespace
  labels:
    app: noobaa
spec:
  azureBlob:
    clientId: 8feb6304-7cee-40ea-b501-ef0d08520874
    secret:
      name: noobaa-azure-container-creds
      namespace: noobaa
      targetBlobContainer: azure-sts-bs
type: azure-blob

GCP

noobaa backingstore create google-cloud-storage gcp-bs --target-bucket personal-bucket --private-key-json-file '/example/path'
apiVersion: noobaa.io/v1alpha1
kind: BackingStore
metadata:
  finalizers:
  - noobaa.io/finalizer
  labels:
    app: noobaa
  name: gcp-bs
  namespace: app-namespace
spec:
  googleCloudStorage:
    secret:
      name: backing-store-google-cloud-storage-gcp-bs
      namespace: secret-namespace
    targetBucket: personal-bucket
  type: google-cloud-storage

GCP WIF (STS)

noobaa backingstore create google-cloud-storage-sts gcp-bs-sts --project-number='123456789' --pool-id='my-pool' --provider-id='my-provider' --service-account-email='noobaa-wif-sa@my-project.iam.gserviceaccount.com' --target-bucket personal-bucket
apiVersion: noobaa.io/v1alpha1
kind: BackingStore
metadata:
  finalizers:
  - noobaa.io/finalizer
  labels:
    app: noobaa
  name: gcp-bs-sts
  namespace: app-namespace
spec:
  googleCloudStorage:
    secret:
      name: backing-store-google-cloud-storage-gcp-bs-sts
      namespace: secret-namespace
    targetBucket: personal-bucket
  type: google-cloud-storage

Persistent Volume Pool

noobaa backingstore create pv-pool pv-backingstore --num-volumes 3 --pv-size-gb 17 --storage-class local-path
apiVersion: noobaa.io/v1alpha1
kind: BackingStore
metadata:
  finalizers:
  - noobaa.io/finalizer
  name: pv-backingstore
  namespace: app-namespace
spec:
  pvPool:
    numVolumes: 3
    resources:
      requests:
        storage: 17Gi
    storageClass: local-path
  type: pv-pool

Modifying a Backing Store's Credentials

If a user wishes to change a backing store's credentials, the appropriate secret should be updated by the user, and the operator will propagate the new credentials to the system server.

A backing store's secret can be found in the CR's YAML's spec field, under a key named secret.

Resource Status

It is possible to check a resource's status in several ways, including:

  • kubectl get backingstore <NAME> -o yaml
  • kubectl describe backingstore <NAME>
  • noobaa backingstore status <NAME>

Below is an example of a healthy backing store's status, as retrieved with the first command:

apiVersion: noobaa.io/v1alpha1
kind: BackingStore
metadata:
  name: aws-s3
  namespace: noobaa
spec:
  ...
status:
  conditions:
  - lastHeartbeatTime: "2019-11-05T13:50:50Z"
    lastTransitionTime: "2019-11-06T07:03:46Z"
    message: noobaa operator completed reconcile - backing store is ready
    reason: BackingStorePhaseReady
    status: "True"
    type: Available
  - lastHeartbeatTime: "2019-11-05T13:50:50Z"
    lastTransitionTime: "2019-11-06T07:03:46Z"
    message: noobaa operator completed reconcile - backing store is ready
    reason: BackingStorePhaseReady
    status: "False"
    type: Progressing
  - lastHeartbeatTime: "2019-11-05T13:50:50Z"
    lastTransitionTime: "2019-11-05T13:50:50Z"
    message: noobaa operator completed reconcile - backing store is ready
    reason: BackingStorePhaseReady
    status: "False"
    type: Degraded
  - lastHeartbeatTime: "2019-11-05T13:50:50Z"
    lastTransitionTime: "2019-11-06T07:03:46Z"
    message: noobaa operator completed reconcile - backing store is ready
    reason: BackingStorePhaseReady
    status: "True"
    type: Upgradeable
    mode:
      modeCode: OPTIMAL
      ...
    phase: Ready

An example for an unhealthy backing store can be seen at the bottom of the next section.

Deletion

Backing stores are used for data persistency; therefore, a cleanup process must run before they may be deleted. The operator will use the finalizer pattern, and set a finalizer on every backing store to mark that external cleanup is needed before it can be deleted.

After marking a backingstore for deletion, the operator will notify the NooBaa server of the deletion, which will enter a decommissioning state, in which NooBaa will attempt to rebuild the data to a new backing store location. Once the decommissioning process completes the operator will remove the finalizer and allow the CR to be deleted.

There are cases where the decommissioning cannot be completed due to an inability to read the data from the backing store. For example, in cases where the target bucket was already deleted, the credentials were invalidated, or the system cannot connect to the target storage. In these cases the system status will be used to report the issue and suggest a manual resolution. For example:

apiVersion: noobaa.io/v1alpha1
kind: BackingStore
metadata:
  name: aws-s3
  namespace: noobaa
  finalizers:
    - noobaa.io/finalizer
spec:
  ...
status:
    conditions:
  - lastHeartbeatTime: "2019-11-06T14:06:35Z"
    lastTransitionTime: "2019-11-06T14:11:36Z"
    message: BackingStore "bs" invalid external connection "INVALID_CREDENTIALS"
    reason: INVALID_CREDENTIALS
    status: Unknown
    type: Available
  - lastHeartbeatTime: "2019-11-06T14:06:35Z"
    lastTransitionTime: "2019-11-06T14:11:36Z"
    message: BackingStore "bs" invalid external connection "INVALID_CREDENTIALS"
    reason: INVALID_CREDENTIALS
    status: "False"
    type: Progressing
  - lastHeartbeatTime: "2019-11-06T14:06:35Z"
    lastTransitionTime: "2019-11-06T14:11:36Z"
    message: BackingStore "bs" invalid external connection "INVALID_CREDENTIALS"
    reason: INVALID_CREDENTIALS
    status: "True"
    type: Degraded
  - lastHeartbeatTime: "2019-11-06T14:06:35Z"
    lastTransitionTime: "2019-11-06T14:11:36Z"
    message: BackingStore "bs" invalid external connection "INVALID_CREDENTIALS"
    reason: INVALID_CREDENTIALS
    status: Unknown
    type: Upgradeable
  phase: Rejected