AWS Filtering in bash

By Paul Heinlein | Dec 8, 2017

I often use the AWS Command Line Interface to create or modify AWS assets, typically in the context of a bash shell script. I’ve found that I frequently need to extract information from one aws operation in order to complete another one.

What follows are some of what I’ve learned in that filtering operation.

JSON Return Values

Unless you specify otherwise, the aws utility uses JSON to express return values, e.g.,

[bash]$ aws ec2 describe-key-pairs
{
  "KeyPairs": [
    {
      "KeyName": "carol-rsa",
      "KeyFingerprint": "aa:bb:cc:dd:..."
    },
    {
      "KeyName": "ted-rsa",
      "KeyFingerprint": "ff:dd:cc:bb:..."
    }
  ]
}

Similarly, when you create an object:

[bash]$ aws ec2 create-security-group \
>> --description 'My Security Group' \
>> --group-name 'my-security-group'
{
  "GroupId": "sg-cf1db7b3"
}

Instead of JSON, you can use the ’text’ or ’table’ formatting styles, and I’ll show uses for the ’text’ style below, but JSON has the best query tools so that’s what I use most of the time.

My Basic Rule

Here’s my basic rule. When using aws to create objects, capture the raw JSON and query it later. When using aws to describe objects, filter for the important data and output it as text.

Using JMESPath

Sometimes, as shown above, the aws utility provides filters for extracting data. Other times, you want a custom query. In that case, JMESPath is that you need. It’s not a particularly easy language to master, partly because it relies on strict data types to an extent I wouldn’t expect, but it’s a powerful JSON query tool.

Here’s a query for listing the four most recent fully virtualized Amazon Linux AMIs that are optimized for SSD (gp2) storage; the most recent image is listed first:

aws ec2 describe-images \
  --query \
    '
      reverse(sort_by(Images, &CreationDate)) |
      [?Name != `null`] |
      [?starts_with(Name, `amzn-ami-hvm`) == `true`] |
      [?BlockDeviceMappings[0].Ebs.VolumeType == `gp2`] |
      [0:4].[ImageId, Description]
    ' \
  --output text

aws ec2 describe-images \
  --filter Name=root-device-type,Values=ebs \
  --query \
    '
      Images[?Name != `null`]                            |
      [?starts_with(Name, `amzn-ami-hvm`) == `true`]     |
      [?BlockDeviceMappings[0].Ebs.VolumeType == `gp2`]  |
      reverse(sort_by(@, &CreationDate))                 |
      [0:4].[ImageId, Description]
    ' \
  --output text

# similar, but for Official CentOS 7 images, where we know the
# owner ID.
aws ec2 describe-images \
  --owners 410186602215 \
  --output text \
  --query 'reverse(sort_by(Images, &CreationDate))[?starts_with(Description, `CentOS Linux 7`)].[ImageId, CreationDate, Description]'

Aws  Cloud  Howto