Google Cloud Platform
Intermediate 15 min read Google Cloud Platform

Pub/Sub & Observability

Pub/Sub is GCP's asynchronous messaging service with at-least-once delivery; Cloud Logging, Cloud Monitoring and Cloud Trace are the three pillars of observability. Learn topics, subscriptions and acks, then how to query logs, build alerts and trace requests across services.

Pub/Subtopicsubscriptionpublishersubscriberat-least-oncedead-letterCloud LoggingCloud MonitoringCloud Trace
Prerequisites: GCP Platform Overview Compute Engine & IAM
Introduction

What is this?

Pub/Sub is GCP's asynchronous messaging service: publishers send messages to topics, and subscribers receive them from subscriptions, with at-least-once delivery and independent scaling of producers and consumers. Cloud Logging collects, stores and queries logs; Cloud Monitoring provides metrics, dashboards and alerting; Cloud Trace records the path of a request across services. Together they form the messaging and observability backbone of GCP applications.

Motivation

Why do you need it?

  • Asynchronous messaging decouples producers from consumers so systems tolerate bursts and partial failures.
  • Pub/Sub is the default glue for event-driven architectures, connecting services, Cloud Functions, GKE and BigQuery.
  • Observability is how you answer 'is it working?' — logs, metrics and traces are the three pillars.
  • Alerts and dashboards turn raw telemetry into something an on-call engineer can act on.
  • Every DevOps interview includes at-least-once delivery, retries and observability questions.
Think about it

Mental model

Imagine

A post office with tracking

Think of

Pub/Sub and observability

Pub/Sub is the mail system: senders (publishers) drop letters into a route (topic) without caring who reads them, and the post office delivers copies to every subscribed postbox (subscription). If a reader is slow, letters queue up — nothing is lost, but delivery is at-least-once, so readers must handle duplicates. Cloud Logging is the archive of every package scanned; Cloud Monitoring is the dashboard of post office throughput; Cloud Trace is the tracking number that follows a single package through every sorting station.

Mechanics

How it works

  • Publishers send messages to a topic; Pub/Sub stores and replicates them and fans them out to each subscription.
  • Subscribers pull or receive pushed messages; Pub/Sub delivers at-least-once, so consumers must be idempotent.
  • Subscriptions can be configured with retention, ack deadlines and dead-letter topics to handle failures.
  • Cloud Logging ingests logs from services via agents or APIs and stores them queryable by resource and labels.
  • Cloud Monitoring collects metrics such as CPU, latency and uptime, builds dashboards and fires alerts; Cloud Trace records distributed request spans.

A message flowing through Pub/Sub with observability alongside.

Publisher sends a message
Pub/Sub replicates and retains
Subscription delivers
Ack or dead-letter
Telemetry flows
Alerts and dashboards
Reference

Key commands

gcloud pubsub topics create orders

Create a topic that messages are published to.

gcloud pubsub subscriptions create orders-sub --topic=orders

Create a subscription to receive messages from a topic.

gcloud pubsub topics publish orders --message="order-12345 placed"

Publish a message to a topic.

gcloud pubsub subscriptions pull orders-sub --limit=1

Pull pending messages from a subscription.

gcloud logging read "resource.type=gce_instance" --limit=10

Query recent logs by resource type.

gcloud monitoring dashboards list

List the monitoring dashboards in the project.

Hands-on

Examples

Publish and pull a Pub/Sub message

Publish and pull a Pub/Sub message
$ gcloud pubsub topics create orders
Created topic [projects/my-project/topics/orders].
 
$ gcloud pubsub subscriptions create orders-sub --topic=orders
Created subscription [projects/my-project/subscriptions/orders-sub].
 
$ gcloud pubsub topics publish orders --message="order-12345 placed"
messageIds:
  '1203597848487128'
 
$ gcloud pubsub subscriptions pull orders-sub --limit=1
DATA                 MESSAGE_ID
order-12345 placed   1203597848487128

The topic is the category, the subscription is the delivery agreement, and publish drops the message into the system. Pull retrieves one pending message; once your code processes it, it sends an ack so Pub/Sub can drop the message from the backlog.

Real world

In production

  • Decouple web tiers from background processors with Pub/Sub so bursts do not drop work.
  • Use Cloud Functions or Cloud Run subscribers to process messages with automatic retries.
  • Centralize logs in Cloud Logging with log-based metrics and retention policies.
  • Set up uptime checks and alerting policies in Cloud Monitoring for the services that matter.
  • Adopt distributed tracing with Cloud Trace on Cloud Run and GKE to find slow paths.
Guidelines

Best practices

  • Make subscribers idempotent because Pub/Sub delivers at-least-once and messages may repeat.
  • Use dead-letter topics to isolate poison messages instead of blocking the whole subscription.
  • Set ack deadlines and use pull with batching to tune throughput and latency.
  • Structure logs with JSON fields so queries and log-based metrics are reliable.
  • Add alerting thresholds and notification channels for the top metrics only.
Watch out

Common mistakes

  • Building consumers that are not idempotent and processing the same message twice.
  • Ignoring the subscription backlog and discovering large lag during an incident.
  • Letting a poison message block the subscription while it retries forever.
  • Storing everything in logs without retention and paying for years of junk.
  • Alerting on everything, then ignoring all alerts because they never stop firing.
Root cause analysis

Common issues

Error
Messages accumulate on the subscription
Root cause

The subscriber is slow, down, or failing to ack messages.

Fix

Check subscriber logs and the ack deadline, then scale the subscriber or fix the crash; the backlog drains once acks succeed.

Error
Duplicate processing observed
Root cause

Pub/Sub delivers at-least-once, and messages can arrive more than once.

Fix

Make the consumer idempotent using a business key or message ID, and store processed IDs for deduplication.

Error
No logs appearing in Cloud Logging
Root cause

The log is disabled, the agent is missing, or the query filters it out.

Fix

Verify the log name and resource label, confirm the logging agent is running, and check the retention window.

When things break

Debugging tips

  • gcloud pubsub subscriptions describe shows backlog and ack state per subscription.
  • Check the message flow by publishing test messages and watching the backlog.
  • For logs, scope queries by resource.type and timestamp ranges to keep them fast.
  • Use the Cloud Monitoring metrics explorer before writing alerting policies.
  • For latency, check Cloud Trace spans for the slow service before guessing at the database.
Interviews

Interview checkpoint

Practice

Hands-on lab

Simulate a Message Queue and Analyze Logs Locally
25 min · intermediate
In progress
Objectives
  • Model a topic and subscription as directories
  • Publish and deliver messages like Pub/Sub
  • Acknowledge and deduplicate a backlog
  • Generate and analyze timestamped log lines

Simulate a Message Queue and Analyze Logs Locally

25 min · intermediate

0/6 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/3
1

Which delivery guarantee does Pub/Sub provide by default?

2

A Pub/Sub subscription can have a dead-letter topic that catches messages that repeatedly fail to ack.

3

A subscriber crashed overnight and the backlog grew. When it restarts, what should it do first?

Quick reference

Cheat sheet

Pub/Sub essentials
TopicNamed category for messages
SubscriptionDelivery agreement with ack state
PublisherSends messages to a topic
SubscriberReceives and acks messages
Dead-letter topicCatch messages that fail repeatedly
Observability
Cloud LoggingQueryable logs with labels
Cloud MonitoringMetrics, dashboards, alerting
Cloud TraceDistributed request spans
Uptime checksSynthetic probes of public endpoints
Go deeper

Official documentation

Pub/Sub Overview
docs
Pub/Sub Best Practices
docs
Cloud Logging Overview
docs
Cloud Monitoring Overview
docs
Cloud Trace Overview
docs