AWS CLI & scripting cloud operations

The AWS CLI as a scripting engine: profiles, —query, —output, pagination

Section 1 of 5 · ~3 min

The AWS CLI v2 is not just a click-free way through the console — it is the tool you script cloud operations with. Every command returns JSON, and four levers turn that raw JSON into something a bash script can drive: profiles pick who and where, --query picks what, --output picks the shape, and pagination controls how much.

Profiles answer “which account and identity?” Instead of one set of keys, ~/.aws/config and ~/.aws/credentials hold named profiles — dev, prod, a client’s account. aws s3 ls --profile prod runs as that identity; export AWS_PROFILE=prod sets it for the whole shell. This is the guardrail that stops you running a destructive command against production when you meant staging.

--query filters the JSON before it reaches you, using JMESPath — a query language for JSON. aws ec2 describe-instances --query 'Reservations[].Instances[].InstanceId' reaches into the nested response and returns just the instance IDs, no jq required. You can project several fields at once and filter with a predicate like [?State.Name=='running']. Learning JMESPath is the single biggest force-multiplier on the CLI.

--output picks the shape: json (default, for machines and jq), table (aligned columns, for human eyes), yaml, and — the scripting workhorse — text. --output text prints bare values with no quotes or brackets, so IID=$(aws ... --output text) drops a clean value straight into a shell variable.

Pagination is the trap. The CLI auto-paginates by default: it silently makes repeated calls until it has every result, which on a large account can be slow or hit rate limits. --max-items caps how many you get, --no-paginate returns only the first page, and --page-size tunes the request size without changing the total. Know it exists so a “hanging” describe call doesn’t surprise you.

Real world: Think of the CLI as a librarian. The profile is which branch you walked into. Pagination is the librarian fetching books one cartload at a time until the shelf is empty. --query is you saying “just the titles, only the ones published after 2020,” and --output text is getting them read aloud as a plain list instead of handed over as a bound catalogue.

The --query engine is JMESPath, the same JSON query library that powers Ansible’s json_query filter — so the expressions you learn here transfer straight into infrastructure-as-code tooling.

Idempotent scripting patterns

A script you can run twice without harm is idempotent, and it is the difference between a toy and something you trust in automation. Two habits get you most of the way. First, describe before you create: query for the resource and only create it if the query comes back empty, so a re-run doesn’t make a duplicate or error out. Second, prefer operations that are naturally idempotent — tagging is the classic case, because create-tags with the same key simply overwrites, so re-tagging is always safe to repeat. Today’s lab leans on read-only describe calls and one tagging script for exactly this reason: both are safe to run again and again.

Hands-On Lab

Section 2 of 5 · ~3 min

Budget about 25 minutes. Drive this from the AWS CLI v2 as your Day 47 IAM user (not root), in us-east-1 — swap in your own region but stay consistent. Every command today is read-only or a tag, so nothing bills you. Type each one and read the output before moving on.

What this costs: ₹0. Every command in this lab is either a read-only describe/get call or a create-tags/delete-tags call — none of them create a billable resource, and tags themselves are always free. You tag your existing default VPC (which already exists and costs nothing), never a new instance or volume. There is nothing to terminate; the only cleanup is deleting the tags you added, which is optional and also free.

# 1. Confirm you're the IAM user from Day 47 (NOT root) and set a default region.
aws sts get-caller-identity
export AWS_DEFAULT_REGION=us-east-1
# Output (your Account and Arn will differ):
# {
#     "UserId": "AIDA...EXAMPLE",
#     "Account": "123456789012",
#     "Arn": "arn:aws:iam::123456789012:user/devops-you"
# }

# 2. List the named profiles the CLI knows about (from ~/.aws/config and credentials).
aws configure list-profiles
# Output (yours will differ — you may only have 'default' today):
# default

# 3. Report on instances with --query, formatted as a table for human eyes.
aws ec2 describe-instances \
  --query 'Reservations[].Instances[].[InstanceId,InstanceType,State.Name]' \
  --output table
# Output (an empty result prints nothing at all — the table box only renders when there is at least one row):
# ---------------------------------------------------
# |                DescribeInstances                |
# +----------------------+------------+-------------+
# |  i-0123456789abcdef0 |  t2.micro  |  running    |
# +----------------------+------------+-------------+

# 4. Same data, filtered to only running instances, as bare text for scripting.
aws ec2 describe-instances \
  --query "Reservations[].Instances[?State.Name=='running'].InstanceId" \
  --output text
# Output (tab/newline separated bare IDs — empty if none are running):
# i-0123456789abcdef0

# 5. Pagination: the CLI auto-fetches ALL pages. --max-items 5 caps the result.
aws ec2 describe-instance-types \
  --query 'InstanceTypes[].InstanceType' --output text --max-items 5
# Output (5 of the hundreds of types; a NextToken is offered to continue):
# t2.micro  t2.small  t3.micro  m5.large  c5.large

# 6. Capture the default VPC's ID into a shell variable with --output text.
VPC=$(aws ec2 describe-vpcs --filters Name=isDefault,Values=true \
  --query 'Vpcs[0].VpcId' --output text)
echo "$VPC"
# Output (your default VPC ID will differ — this is what we'll tag):
# vpc-0aa11bb22cc33dd44

# 7. Idempotency check: does this VPC already carry an Owner tag?
aws ec2 describe-tags \
  --filters "Name=resource-id,Values=$VPC" "Name=key,Values=Owner" \
  --query 'Tags[0].Value' --output text
# Output (None means the tag doesn't exist yet — safe to create):
# None

# 8. Write a small idempotent tagging script — tags every default VPC in the region.
cat > ~/tag-vpcs.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
OWNER="devops-you"
for vpc in $(aws ec2 describe-vpcs --filters Name=isDefault,Values=true \
    --query 'Vpcs[].VpcId' --output text); do
  # create-tags OVERWRITES an existing key, so re-running is safe (idempotent).
  aws ec2 create-tags --resources "$vpc" \
    --tags Key=Owner,Value="$OWNER" Key=ManagedBy,Value=m90
  echo "tagged $vpc"
done
EOF
echo "script written"
# Output:
# script written

# 9. Run the script. It reports each VPC it tagged.
bash ~/tag-vpcs.sh
# Output (your VPC ID will differ):
# tagged vpc-0aa11bb22cc33dd44

# 10. Verify the tags landed by reading them back with --query.
aws ec2 describe-tags --filters "Name=resource-id,Values=$VPC" \
  --query 'Tags[].[Key,Value]' --output text
# Output (yours will differ, but Owner and ManagedBy should appear):
# ManagedBy  m90
# Owner      devops-you

# 11. Prove idempotency: run the script AGAIN. Same result — no duplicate tags.
bash ~/tag-vpcs.sh
aws ec2 describe-tags --filters "Name=resource-id,Values=$VPC" \
  --query 'length(Tags)' --output text
# Output (still exactly 2 tags — create-tags overwrote, it didn't duplicate):
# 2

# 12. Tidy up (optional — tags are free): remove the two tags you added.
aws ec2 delete-tags --resources "$VPC" \
  --tags Key=Owner Key=ManagedBy
echo "tags removed"
# Output:
# tags removed

Read the last blocks back: step 11 is the one that matters — running the script twice left exactly two tags, not four. That is idempotency, and it is what lets you schedule a script without fear of it multiplying its own effects.

Common Errors & Fixes

Section 3 of 5 · ~3 min

These three catch almost everyone scripting the CLI for the first time — and note that the nastiest one throws no error at all, just silent empty output. Read each carefully; recognising the failure is the actual skill.

Common error: Passing --profile prod when no such profile exists in your config:

The config profile (prod) could not be found

Why: --profile looks up a named block in ~/.aws/config / ~/.aws/credentials. If you never created a [profile prod] block — or you have a typo, or you’re on a different machine — the CLI has no credentials to load and stops before making any call.

Fix: List what actually exists with aws configure list-profiles, then create the missing one with aws configure --profile prod (or add an sso/role_arn block for assumed roles). Match the name exactly — profiles are case-sensitive.

How you’d spot it in prod: A pipeline that runs locally but fails in CI with “profile could not be found” is almost always relying on a developer’s ~/.aws that the runner doesn’t have. CI should use an assumed role or environment credentials, not a named profile baked into someone’s laptop.

Common pitfall: Writing a JMESPath filter with an unquoted string literal — --query 'Reservations[].Instances[?State.Name==running]' — and getting nothing back with no error at all:

Why: [?State.Name==running] does not error — it compares State.Name to a nonexistent field named running, so it silently returns nothing. In JMESPath a bare word like running is read as an identifier (a field reference), not the text “running”. The two fields never match, the filter drops every element, and you get an empty result that looks exactly like “no running instances” — the most dangerous kind of bug, because there’s no error text to tip you off.

Fix: Quote the literal: [?State.Name=='running']. Then wrap the whole --query value in double quotes so the shell preserves the single quotes instead of stripping them before the CLI sees them: --query "Reservations[].Instances[?State.Name=='running'].InstanceId".

How you’d spot it in prod: A describe/filter script that keeps reporting “0 matching resources” when you know some exist is the tell — suspect an unquoted literal in the JMESPath before you suspect the account. Test the expression interactively against one saved JSON response, and compare the filtered count against the unfiltered one to confirm the predicate actually matches.

Common error: Forgetting --output text, so a captured variable carries JSON quotes into the next command:

An error occurred (InvalidInstanceID.Malformed) when calling the DescribeInstances operation: Invalid id: "\"i-0123456789abcdef0\""

Why: With the default json output, IID=$(aws ... --query 'Instances[0].InstanceId') stores the value with its surrounding quotes — literally "i-0123...". The next command passes those quote characters as part of the ID, and AWS rejects the malformed string.

Fix: Add --output text to the capturing command so the value comes back bare. As a rule, any aws call whose result you assign to a variable should carry --query to pick one field and --output text to strip the packaging.

How you’d spot it in prod: A Malformed or Invalid id error where the ID in the message is wrapped in extra quotes or brackets is the tell — the upstream command emitted JSON where the downstream one wanted a bare string. Fix the output format, not the ID.

AWS CLI Interview Questions

Section 4 of 5 · ~1 min

Cover the answers below and say your own version out loud first — name what --query, --output text and a profile each do before you reveal each answer. Recalling before revealing is what makes these stick when an interviewer asks them cold. The four questions and answers render right after this note.

Go Deeper

Section 5 of 5 · ~1 min

Optional extras if you have ~30 more minutes today:

• 10 min — Work through the interactive tutorial at jmespath.org, then re-run today’s describe-instances with a projection that renames fields — JMESPath multiselect hashes are the trick that turns a describe call into a clean report.
• 10 min — Read the AWS CLI configuration and named profiles docs, then set up a second profile that assumes a role — the real-world pattern for switching accounts without long-lived keys.
• 5 min — Compare --output table (readable) with --output text (scriptable) on the same command, and try --cli-auto-prompt to have the CLI walk you through a command’s options interactively.
• 5 min — Extend tag-vpcs.sh to emit a CSV report instead: loop your instances and echo InstanceId,State,Type per line — a two-minute change that turns a tagging script into an inventory report.