Amazon S3 - Firebolt Documentation

Configuring object storage is the recommended setup for all production deployments of Firebolt Core. With object storage as the backing store for table data, durability does not depend on the persistent volumes mounted to each node — even a complete loss of all volumes does not cause data loss. See Data durability for a full explanation. To enable object storage, configure the storage scheme via managed_table_storage_api_scheme and set the bucket name via managed_table_bucket_name_override.

Prerequisites

Before you begin, ensure that you have the following installed and configured:

A Kubernetes cluster (v1.19+)
kubectl command-line tool configured to access your cluster
helm (v3+) installed on your local machine
An AWS account with permissions to create S3 buckets, IAM roles, and IAM policies

Use Amazon S3

The following examples use an S3 bucket named firebolt-core-demo-data but you can choose any name you like.

Create an S3 bucket

export AWS_DEFAULT_REGION=us-east-1
export FIREBOLT_CORE_MANAGED_BUCKET_NAME=firebolt-core-demo-data

aws s3api create-bucket \
  --bucket "${FIREBOLT_CORE_MANAGED_BUCKET_NAME}"

aws s3api put-public-access-block \
  --bucket "${FIREBOLT_CORE_MANAGED_BUCKET_NAME}" \
  --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

aws s3api put-bucket-encryption \
  --bucket "${FIREBOLT_CORE_MANAGED_BUCKET_NAME}" \
  --server-side-encryption-configuration '{
    "Rules": [
      {
        "ApplyServerSideEncryptionByDefault": {
          "SSEAlgorithm": "AES256"
        }
      }
    ]
  }'

aws s3api put-bucket-lifecycle-configuration \
  --bucket "${FIREBOLT_CORE_MANAGED_BUCKET_NAME}" \
  --lifecycle-configuration '{
    "Rules": [
      {
        "ID": "expire_soft_deleted_objects",
        "Status": "Enabled",
        "Filter": {
          "Tag": {
            "Key": "IsDeleted",
            "Value": "true"
          }
        },
        "Expiration": {
          "Date": "2016-01-12T00:00:00+00:00"
        }
      },
      {
        "ID": "abort_incomplete_multipart_upload",
        "Status": "Enabled",
        "Filter": {
          "Prefix": ""
        },
        "AbortIncompleteMultipartUpload": {
          "DaysAfterInitiation": 1
        }
      }
    ]
  }'

IAM role and IAM policy

Create an IAM role with the following IAM policy that grants Firebolt Core permissions to manage objects in this bucket. Use a mechanism like AWS IRSA or AWS Pod Identity to assign an IAM role with these permissions to your Firebolt Core workload.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::{{FIREBOLT_CORE_MANAGED_BUCKET_NAME}}"
            ],
            "Sid": "StorageBuckets"
        },
        {
            "Action": [
                "s3:GetObject*",
                "s3:PutObject*"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::{{FIREBOLT_CORE_MANAGED_BUCKET_NAME}}/*"
            ],
            "Sid": "ObjectAccess"
        }
    ]
}

Configure Core to use S3

Set aws_region, managed_table_storage_api_scheme and managed_table_bucket_name_override in your Firebolt Core node configuration.

cat <<EOF > firebolt-core-config.json
{
  "config": {
    "aws_region": "${AWS_DEFAULT_REGION}"
    "managed_table_storage_api_scheme": "s3://",
    "managed_table_bucket_name_override": "${FIREBOLT_CORE_MANAGED_BUCKET_NAME}"
  }
}
EOF

helm install core-demo ../firebolt-core/helm/ -n firebolt-core \
  --set-json customNodeConfig="$(cat firebolt-core-config.json)"

Custom Kubernetes ServiceAccount

If you use AWS IRSA and created your own Kubernetes ServiceAccount, you can set it in the Helm chart via:

helm install core-demo ../firebolt-core/helm/ -n firebolt-core \
  --set-json customNodeConfig="$(cat firebolt-core-config.json)" \
  --set serviceAccount=core-demo

Confirm that object storage is working

To confirm that your managed storage works, you can create a table and see if new prefixes are created in your bucket. If the queries hang, make sure to check the logs of Firebolt Core and look out for any AWS IAM access denied error logs.

kubectl port-forward pod/core-demo-firebolt-core-0 3473:3473 -n firebolt-core

curl -s "http://localhost:3473" --data-binary "create table test (val int);";
curl -s "http://localhost:3473" --data-binary "insert into test values (1);";

aws s3 ls firebolt-core-demo-data
# expected output similar to:
#    PRE SRd8FBoIadUX_Jd-pxV9qQ~31~all~0/
#    PRE drU1S3fjduVWesJyToDXDQ~33~all~0/

​Prerequisites

​Use Amazon S3

​Create an S3 bucket

​IAM role and IAM policy

​Configure Core to use S3

​Custom Kubernetes ServiceAccount

​Confirm that object storage is working