Creating AWS S3 Bucket for Backup

By Paul Heinlein | Feb 5, 2019 (updated Feb 6, 2019)

I needed to create for a client several AWS S3 buckets that would be used for system backups.

The basic rule was one bucket per host. Each bucket would be flat in the sense that all files would live in the top-level namespace; there would be no “subdirectories.”

The backup software would consolidate files to be backed up into tar archives. The bulk of the data would live in files named dat-*.tar.gz, but there would be some other archives named without the dat- prefix. To minimize costs, all dat-* files should cycle into Glacier-class storage after one day.

Backups would be performed by host-specific IAM accounts with limited rights, and all files would be encrypted with an AWS KMS key, which would provide an additional layer of access control.

I first created the IAM user accounts, some for staff members and one per host to be backed up. The staff accounts had wider access privileges, while the host accounts were given no default rights.

Then I created one KMS encryption key per host. Staff were given encrypt/decrypt rights to all the keys, while the host accounts were only given access to the host-specific key.

After that, I needed to

create each bucket,
apply a bucket policy to allow the appropriate host account access to the bucket,
apply a policy to ensure that all files named dat-* would be cycled into Glacer after one day, and
apply a policy to ensure all files would be encrypted on the server side with the appropriate host key.

Bucket Access Policy

Assuming the host in question is called “myhost01,” I have a per-host user named myhost01_svc and a per-host bucket called myhost01. This policy allows the host user basic read/write access to the its bucket.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123412341234:user/myhost01_svc"
      },
      "Action": [
        "s3:GetLifecycleConfiguration",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads"
       ],
      "Resource": "arn:aws:s3:::myhost01"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123412341234:user/myhost01_svc"
      },
      "Action": [
        "s3:DeleteObject",
        "s3:GetObject",
        "s3:ListMultipartUploadParts",
        "s3:PutObject",
        "s3:RestoreObject"
      ],
      "Resource": "arn:aws:s3:::myhost01/*"
    }
  ]
}

Lifecycle Policy

All files named dat-* should transition to Glacier after one day.

{
  "Rules": [
    {
      "Filter": {
        "Prefix": "dat-"
      },
      "Status": "Enabled",
      "Transitions": [
        {
          "Days": 1,
          "StorageClass": "GLACIER"
        }
      ],
      "ID": "Data to Glacier"
    }
  ]
}

Encryption Policy

By default, all uploaded files should be encrypted with the host’s KMS key.

{
  "Rules": [
    {
      "ApplyServerSideEncryptionByDefault": {
        "KMSMasterKeyID": "arn:aws:kms:us-west-2:123412341234:key/856223bc-634e-48dc-8bb3-862a50b72209",
        "SSEAlgorithm": "aws:kms"
      }
    }
  ]
}

Putting It All Together

Here’s the shell script that creates the bucket and applies the policies.

#!/bin/bash

BUCKET="myhost01"
BUCKETURL="s3://${BUCKET}"

ACLPOLICY="file://access.json"
LCPOLICY="file://lifecycle.json"
SSEPOLICY="file://encryption.json"

# many S3 AWS operations will return quickly at at the command line
# but require a few seconds to complete. this routine allows us to
# add a time countdown between operations
function countdown {
  local CNT=$1
  local MSG=${2:-waiting}

  echo -n "$MSG ... "
  for N in $(seq $CNT 1); do
    echo -n "$N "
    sleep 1
  done
  echo
}

# shared exit routine for failures
function exitonfail {
  local RET=$1
  local MSG=${2:-operation failed.}

  if test "$RET" -ne 0; then
    echo "$MSG exiting now." >&2
    exit 1
  fi
}

echo -n "testing for existence of $BUCKET ... "
aws s3 ls $BUCKETURL >/dev/null 2>&1
if test $? -eq 0; then
  echo "$BUCKET already exists; exiting now"
  exit 0
else
  echo "doesn't exist"

  echo "making S3 bucket $BUCKET"
  aws s3 mb $BUCKETURL
  exitonfail $? "bucket creation failed."
  countdown 5 "pausing for bucket creation"

  echo "adding access policy to $BUCKET"
  aws s3api put-bucket-policy \
    --bucket $BUCKET \
    --policy $ACLPOLICY
  exitonfail $? "access-policy configuration failed."
  countdown 8 "pausing for access-policy creation"

  echo "adding lifecycle configuration to $BUCKET"
  aws s3api put-bucket-lifecycle-configuration \
    --bucket $BUCKET \
    --lifecycle-configuration $LCPOLICY
  exitonfail $? "lifecycle configuration failed."
  countdown 8 "pausing for lifecycle creation"

  echo "adding server-side encryption policy to $BUCKET"
  aws s3api put-bucket-encryption \
    --bucket $BUCKET \
    --server-side-encryption-configuration $SSEPOLICY
  exitonfail $? "encryption configuration failed."
  countdown 8 "pausing for encyption configuration"
fi

echo "$BUCKET fully created."

Aws