CloudWatch — metrics, logs, alarms

CloudWatch: metrics, logs, and the alarms that page you

Section 1 of 5 · ~2 min

Amazon CloudWatch is AWS’s built-in monitoring service — the place every other AWS service quietly sends its telemetry. You don’t install it or switch it on: the moment you launch an EC2 instance, run a Lambda, or create a load balancer, that resource is already publishing data to CloudWatch. Your job is to read it, and to wire an alarm so a machine watches the numbers while you sleep.

CloudWatch has three pillars.

Metrics are time-ordered numbers — CPU utilisation, request count, queue depth — one data point per timestamp. Every metric lives in a namespace (a folder, e.g. AWS/EC2 for EC2’s built-in metrics), and is pinned down by dimensions: the name/value pairs that say which thing this number describes — InstanceId=i-0abc… narrows CPUUtilization to one specific instance. Namespace plus metric name plus dimensions uniquely identify a line on a graph. EC2 publishes basic metrics every five minutes for free; detailed one-minute monitoring costs extra.

Logs are the text side. A log group is a named bucket of logs for one application or resource (e.g. /aws/ec2/my-app); inside it, each source writes to its own log stream — usually one per instance or container. EC2 does not ship your system logs by default: you install the CloudWatch agent on the instance to push /var/log/syslog, app logs, and even memory usage (which the hypervisor can’t see from outside) up to a log group.

Alarms watch one metric and change state — OK, ALARM, or INSUFFICIENT_DATA — when it crosses a threshold you set for a number of periods. An alarm on its own just changes colour; it earns its keep when its action points at an SNS topic (Simple Notification Service), a fan-out channel that then emails you, texts you, or triggers auto scaling. Metric → alarm → SNS → your inbox is the canonical AWS alerting chain.

Real world: CloudWatch is a building’s facilities system. Metrics are the gauges — temperature, power draw — sampled every minute; logs are the maintenance logbook each room writes into; an alarm is the smoke detector wired to a threshold; and SNS is the intercom that pages the on-call engineer, the fire brigade, and the lobby screen at once. The detector alone just blinks red — it’s the intercom wiring that actually gets someone out of bed.

Airbnb runs on this exact pattern at scale: its EC2 and ECS services publish metrics and logs to CloudWatch, alarms watch the golden signals, and SNS fans the ALARM state out to PagerDuty and Slack. The alarm you build by hand today — CPU over a threshold, notify an SNS topic — is that same primitive, just multiplied across thousands of resources and routed to a pager instead of your inbox.

Today you create an SNS topic, subscribe your email, set a metric alarm on an EC2 instance’s CPUUtilization, watch it evaluate, push and read a CloudWatch log, then delete the alarm and topic so nothing lingers.

Hands-On Lab

Section 2 of 5 · ~4 min

Budget about 25 minutes. Drive this from the AWS CLI v2 as your Day 47 IAM user (not root), in us-east-1 — swap in your own region if you prefer, but stay consistent. Type every command and read the output before moving on.

What this costs: ₹0 if you stay on the free tier and finish the teardown. CloudWatch’s always-free allowance covers 10 alarms and plenty of API calls, the first 5 GB of log ingestion, and SNS gives 1,000 free email notifications a month — so the metric, alarm, log group and email in this lab are effectively free. Accounts on the legacy 12-month Free Tier and the credit-based Free Tier for accounts created since mid-2025 both cover this comfortably. The one real charge is the t2.micro you launch to alarm on: it’s free-tier eligible, but a t2.micro left running is roughly ₹700–900/month, plus ~₹300/month for its public IPv4 address once the free tier’s 750 IPv4 hours are used up, so Step 10 terminates it. Alarms and SNS cost effectively nothing at this volume — the bill only grows if you forget the instance.

# 1. Confirm you're your Day 47 IAM user (NOT root) and set a default region.
aws sts get-caller-identity
export AWS_DEFAULT_REGION=us-east-1
# Output (your Account and Arn will differ):
# {
#     "UserId": "AIDA...EXAMPLE",
#     "Account": "123456789012",
#     "Arn": "arn:aws:iam::123456789012:user/devops-you"
# }

# 2. Launch one free-tier t2.micro to alarm on (latest Ubuntu 24.04 AMI from SSM).
AMI=$(aws ssm get-parameters \
  --names /aws/service/canonical/ubuntu/server/24.04/stable/current/amd64/hvm/ebs-gp3/ami-id \
  --query 'Parameters[0].Value' --output text)
IID=$(aws ec2 run-instances --image-id "$AMI" --instance-type t2.micro \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=m90-cw}]' \
  --query 'Instances[0].InstanceId' --output text)
aws ec2 wait instance-running --instance-ids "$IID"; echo "$IID"
# Output (your instance ID will differ):
# i-0123456789abcdef0

# 3. Create an SNS topic — the fan-out channel the alarm will publish to.
TOPIC=$(aws sns create-topic --name m90-cw-alerts --query 'TopicArn' --output text)
echo "$TOPIC"
# Output (your account ID will differ):
# arn:aws:sns:us-east-1:123456789012:m90-cw-alerts

# 4. Subscribe your email. AWS emails a confirmation link you MUST click.
aws sns subscribe --topic-arn "$TOPIC" \
  --protocol email --notification-endpoint you@example.com
# Output — stays "pending confirmation" until you click the link in your inbox:
# {
#     "SubscriptionArn": "pending confirmation"
# }

# 5. Alarm when this instance's average CPU tops 70% for one 5-min period.
aws cloudwatch put-metric-alarm --alarm-name m90-cpu-high \
  --namespace AWS/EC2 --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value="$IID" \
  --statistic Average --period 300 --evaluation-periods 1 \
  --threshold 70 --comparison-operator GreaterThanThreshold \
  --alarm-actions "$TOPIC"
# Output: put-metric-alarm returns nothing on success — no news is good news.

# 6. Read the alarm's state. Fresh alarms start INSUFFICIENT_DATA, then settle to OK.
aws cloudwatch describe-alarms --alarm-names m90-cpu-high \
  --query 'MetricAlarms[0].{Name:AlarmName,State:StateValue}'
# Output (yours will differ — expect INSUFFICIENT_DATA for the first ~5 min):
# {
#     "Name": "m90-cpu-high",
#     "State": "INSUFFICIENT_DATA"
# }

# 7. Now the Logs side. Create a log group — the named bucket for one app's logs.
aws logs create-log-group --log-group-name /m90/demo
aws logs describe-log-groups --log-group-name-prefix /m90/demo \
  --query 'logGroups[0].logGroupName' --output text
# Output:
# /m90/demo

# 8. Create a log stream inside it, then push one log event (timestamp is in milliseconds).
aws logs create-log-stream --log-group-name /m90/demo --log-stream-name day57
aws logs put-log-events --log-group-name /m90/demo --log-stream-name day57 \
  --log-events "timestamp=$(date +%s)000,message=hello from day 57"
# Output (a sequence token — modern accounts no longer require it on the next put):
# {
#     "nextSequenceToken": "4957...EXAMPLE"
# }

# 9. Read the events back out of the stream — this is what the CloudWatch console shows.
aws logs get-log-events --log-group-name /m90/demo --log-stream-name day57 \
  --query 'events[].message' --output text
# Output:
# hello from day 57

# 10. TEARDOWN. Terminate the instance — the only part of this lab that bills if left running.
aws ec2 terminate-instances --instance-ids "$IID" \
  --query 'TerminatingInstances[0].CurrentState.Name' --output text
# Output:
# shutting-down

# 11. Delete the alarm and the log group.
aws cloudwatch delete-alarms --alarm-names m90-cpu-high
aws logs delete-log-group --log-group-name /m90/demo
echo "alarm + log group deleted"
# Output:
# alarm + log group deleted

# 12. Delete the SNS topic (and its subscription) so nothing is left behind.
aws sns delete-topic --topic-arn "$TOPIC"
echo "topic deleted — teardown complete"
# Output:
# topic deleted — teardown complete

Read the last outputs back: terminated compute is what keeps the bill at ₹0, and a deleted alarm, log group and topic mean nothing lingers to bill or clutter the console next month. CloudWatch itself costs almost nothing at this volume — the discipline that matters is killing the EC2 instance you launched to feed it.

Common Errors & Fixes

Section 3 of 5 · ~2 min

These three catch almost everyone wiring their first alarm. Read the error text slowly — parsing it is the actual skill.

Common error: The alarm flips to ALARM but no email ever arrives, because the SNS subscription was never confirmed:

$ aws sns list-subscriptions-by-topic --topic-arn "$TOPIC"
"SubscriptionArn": "PendingConfirmation"

Why: aws sns subscribe for the email protocol only requests a subscription; AWS sends a confirmation link to that address and the subscription stays in PendingConfirmation until someone clicks it. An unconfirmed subscriber receives nothing when the topic publishes.

Fix: Open the “AWS Notification - Subscription Confirmation” email and click Confirm subscription, then re-run list-subscriptions-by-topic — the SubscriptionArn changes from PendingConfirmation to a real ARN. Check spam if it hasn’t arrived.

How you’d spot it in prod: Alarms turn red in the console but nobody is paged — the first thing to check is that every SNS subscriber is confirmed, not pending. A newly added email or a rotated address is the usual culprit.

Common error: A new CPU alarm never leaves INSUFFICIENT_DATA, because its period is finer than the metric’s publish rate:

"StateValue": "INSUFFICIENT_DATA",
"StateReason": "Insufficient Data: 1 datapoint were unknown."

Why: EC2 basic monitoring publishes CPUUtilization once every five minutes. Set --period 60 (one minute) and most evaluation windows contain no data point at all, so the alarm can never gather enough data to decide OK or ALARM.

Fix: Match the period to the metric — use --period 300 for basic monitoring — or enable detailed monitoring on the instance (aws ec2 monitor-instances) if you genuinely need one-minute periods, and expect to pay for it. Also confirm the InstanceId dimension exactly matches a running instance.

How you’d spot it in prod: An alarm stuck in INSUFFICIENT_DATA forever is almost always a period/granularity mismatch, or a dimension that names a resource that no longer exists — not a broken metric.

Common error: Pushing a log event before creating the stream:

An error occurred (ResourceNotFoundException) when calling the PutLogEvents operation: The specified log stream does not exist.

Why: put-log-events writes to a log stream that must already exist inside the log group. Creating the group alone isn’t enough — the stream is a separate resource, and unlike some SDKs the CLI won’t auto-create it.

Fix: Run aws logs create-log-stream --log-group-name /m90/demo --log-stream-name day57 first, then put-log-events. If the group is missing you get the same error naming the log group instead — create it with create-log-group.

How you’d spot it in prod: A logging agent or Lambda that suddenly throws ResourceNotFoundException on PutLogEvents usually means a retention policy deleted the group, or a new per-day/per-host stream name wasn’t created — the writer must create the stream, not assume it.

CloudWatch Interview Questions

Section 4 of 5 · ~1 min

Cover each answer, then say your own version out loud first — name what a namespace and a dimension are, and trace the metric → alarm → SNS chain, before you reveal each answer. Recalling before revealing is what makes these stick when an interviewer asks them cold. The four questions and answers render right after this note.

Go Deeper

Section 5 of 5 · ~1 min

Optional extras if you have ~30 more minutes today:

• 10 min — Relaunch a t2.micro, install the CloudWatch agent, and configure it to push memory usage and /var/log/syslog to a log group — the piece EC2 can’t report on its own. Terminate the instance when you’re done.
• 10 min — Read the CloudWatch alarm states and metric-math docs, then build a composite alarm that fires only when two conditions are true at once — how real teams cut alert noise.
• 5 min — Create a metric filter on a log group that counts lines matching ERROR, turning free-text logs into a metric you can alarm on — the bridge between the Logs and Metrics pillars.
• 5 min — Skim the CloudWatch pricing page and the always-free tier so you know exactly which alarms, dashboards and log volumes stay free and which quietly bill.