Projects
ProjectsObservabilityadvancedObservability

Production Kubernetes Observability Platform

Prometheus, Grafana, logs, alerts and service health.

~22 hours 9 phases Capstone
PrometheusGrafanaLokiKubernetes

Overview

Build a production Kubernetes observability platform with Prometheus metrics, Grafana dashboards, log aggregation and alerting.

The problem

Observe a production Kubernetes platform end to end.

Why it matters

This is a real, in-demand scenario: Observe a production Kubernetes platform end to end.. Teams that can ship and run a observability like this solve an actual business need and are hired for that ability.

What you'll build

  • An observability platform
  • Dashboards
  • Alerting

What you'll learn

  • Observability at scale
  • Alerting
  • Kubernetes ops

Business problem

Operators need unified signals to run production safely.

Technical problem

Collect, store and visualise metrics, logs and alerts at scale.

When you're done: A working observability project — code committed, tests passing, and documented in your portfolio.

Requirements

User requirements

  • Metric collection
  • Dashboards
  • Log search
  • Alerts

System requirements

  • Prometheus
  • Grafana
  • Loki
  • Alerts

Functional requirements

  • Scrape config
  • Dashboards
  • Log collection
  • Alert rules
  • Service monitors

Non-functional requirements

  • Scalable
  • Reliable retention

Technical requirements

  • Prometheus Operator
  • Grafana
  • Loki
  • Alertmanager

Security requirements

  • Restrict dashboards
  • Secure pipeline

Operational requirements

  • Retention policy

Constraints

  • Kubernetes-based stack

Success criteria

  • Full signal coverage
  • Alerts route correctly

Flows

Request Flow

  1. 1

    Browser

    User requests the application over HTTPS.

  2. 2

    CDN / DNS

    Edge routes the request to the origin.

  3. 3

    Load Balancer

    Distributes traffic across healthy app instances.

  4. 4

    Application

    Handles the request, applies business logic.

  5. 5

    Cache

    Serves fast reads for hot data.

  6. 6

    Database

    Persists data and returns results.

Project Structure

Root of the client application source.

Setup

  1. 1

    Prerequisites

    You need the tools required by Prometheus, Grafana, Loki installed and available on your PATH.

  2. 2

    Clone project

    Clone the repository locally.

    1. Clone the repository
    git clone <your-repo-url> <project-folder>
  3. 3

    Install dependencies

    Install the project dependencies.

    1. Initialize a Node.js project
    npm init -y

    Expected

    Creates a package.json with defaults.

    If it fails

    command not found: npm

    Fix

    Install Node.js LTS from the official site, then reopen your terminal.

  4. 4

    Configure environment

    Set the environment variables documented for this project. Never commit secrets.

    1. Create the environment file
    cp .env.example .env   # then edit values
  5. 5

    Start project

    Run the development command and confirm it starts.

    1. Start the development server
    npm run dev

    Expected

    Dev server starts; watch terminal for the local URL.

    If it fails

    Port 3000 is already in use

    Fix

    Run on another port: npm run dev -- --port 3001, or stop the conflicting process.

  6. 6

    Verify

    Open the printed local URL and confirm the primary flow works.

Build Phases

0 of 9 phases complete

  1. Objective: Establish the next milestone of the observability stack.

    What changes: The project advances to the next coherent state, keeping the previous milestones working.

    Files touched

    Relevant source and configuration files for this milestone.

    Run

    1. Initialize a Node.js project
    npm init -y

    Expected

    Creates a package.json with defaults.

    If it fails

    command not found: npm

    Fix

    Install Node.js LTS from the official site, then reopen your terminal.

    Tasks

    • Implement the specific behaviour required for this milestone.
    • Verify the previous milestone still works.

    Validate: The milestone behaves as expected and does not break existing functionality.

    Common mistakes

    • • Skipping validation and moving on before confirming the milestone works.

  2. Objective: Establish the next milestone of the observability stack.

    What changes: The project advances to the next coherent state, keeping the previous milestones working.

    Files touched

    Relevant source and configuration files for this milestone.

    Run

    1. Install the prometheus package
    npm install prometheus

    Expected

    Adds prometheus to node_modules and package.json.

    If it fails

    npm ERR! network / EACCES

    Fix

    Check your network or run with sudo on Linux/macOS; on Windows run the terminal as Administrator.

    Tasks

    • Implement the specific behaviour required for this milestone.
    • Verify the previous milestone still works.

    Validate: The milestone behaves as expected and does not break existing functionality.

    Common mistakes

    • • Skipping validation and moving on before confirming the milestone works.

  3. Objective: Establish the next milestone of the observability stack.

    What changes: The project advances to the next coherent state, keeping the previous milestones working.

    Files touched

    Relevant source and configuration files for this milestone.

    Run

    1. Start the development server
    npm run dev

    Expected

    Dev server starts; watch terminal for the local URL.

    If it fails

    Port 3000 is already in use

    Fix

    Run on another port: npm run dev -- --port 3001, or stop the conflicting process.

    Tasks

    • Implement the specific behaviour required for this milestone.
    • Verify the previous milestone still works.

    Validate: The milestone behaves as expected and does not break existing functionality.

    Common mistakes

    • • Skipping validation and moving on before confirming the milestone works.

  4. Objective: Establish the next milestone of the observability stack.

    What changes: The project advances to the next coherent state, keeping the previous milestones working.

    Files touched

    Relevant source and configuration files for this milestone.

    Run

    1. Build the application for production
    npm run build

    Expected

    A production bundle is emitted to the output directory.

    If it fails

    Build fails with a type or syntax error

    Fix

    Read the error message, fix the referenced file, and re-run the build.

    Tasks

    • Implement the specific behaviour required for this milestone.
    • Verify the previous milestone still works.

    Validate: The milestone behaves as expected and does not break existing functionality.

    Common mistakes

    • • Skipping validation and moving on before confirming the milestone works.

  5. Objective: Establish the next milestone of the observability stack.

    What changes: The project advances to the next coherent state, keeping the previous milestones working.

    Files touched

    Relevant source and configuration files for this milestone.

    Tasks

    • Implement the specific behaviour required for this milestone.
    • Verify the previous milestone still works.

    Validate: The milestone behaves as expected and does not break existing functionality.

    Common mistakes

    • • Skipping validation and moving on before confirming the milestone works.

  6. Objective: Establish the next milestone of the observability stack.

    What changes: The project advances to the next coherent state, keeping the previous milestones working.

    Files touched

    Relevant source and configuration files for this milestone.

    Tasks

    • Implement the specific behaviour required for this milestone.
    • Verify the previous milestone still works.

    Validate: The milestone behaves as expected and does not break existing functionality.

    Common mistakes

    • • Skipping validation and moving on before confirming the milestone works.

  7. Objective: Establish the next milestone of the observability stack.

    What changes: The project advances to the next coherent state, keeping the previous milestones working.

    Files touched

    Relevant source and configuration files for this milestone.

    Tasks

    • Implement the specific behaviour required for this milestone.
    • Verify the previous milestone still works.

    Validate: The milestone behaves as expected and does not break existing functionality.

    Common mistakes

    • • Skipping validation and moving on before confirming the milestone works.

  8. Objective: Establish the next milestone of the observability stack.

    What changes: The project advances to the next coherent state, keeping the previous milestones working.

    Files touched

    Relevant source and configuration files for this milestone.

    Tasks

    • Implement the specific behaviour required for this milestone.
    • Verify the previous milestone still works.

    Validate: The milestone behaves as expected and does not break existing functionality.

    Common mistakes

    • • Skipping validation and moving on before confirming the milestone works.

  9. Objective: Establish the next milestone of the observability stack.

    What changes: The project advances to the next coherent state, keeping the previous milestones working.

    Files touched

    Relevant source and configuration files for this milestone.

    Tasks

    • Implement the specific behaviour required for this milestone.
    • Verify the previous milestone still works.

    Validate: The milestone behaves as expected and does not break existing functionality.

    Common mistakes

    • • Skipping validation and moving on before confirming the milestone works.

Commands

1. Initialize a Node.js project
npm init -y

Expected

Creates a package.json with defaults.

If it fails

command not found: npm

Fix

Install Node.js LTS from the official site, then reopen your terminal.

2. Install the prometheus package
npm install prometheus

Expected

Adds prometheus to node_modules and package.json.

If it fails

npm ERR! network / EACCES

Fix

Check your network or run with sudo on Linux/macOS; on Windows run the terminal as Administrator.

3. Start the development server
npm run dev

Expected

Dev server starts; watch terminal for the local URL.

If it fails

Port 3000 is already in use

Fix

Run on another port: npm run dev -- --port 3001, or stop the conflicting process.

4. Build the application for production
npm run build

Expected

A production bundle is emitted to the output directory.

If it fails

Build fails with a type or syntax error

Fix

Read the error message, fix the referenced file, and re-run the build.

Testing

Unit tests

  • Core logic and pure functions behave correctly.

    npm init -y

    Success: All unit tests pass without errors.

Integration tests

  • Components integrate correctly with data sources and services.

    npm test

    Success: Integration tests pass for the primary flows.

API tests

  • Endpoints respond to valid and invalid input correctly.

    npm test

    Success: API tests cover happy and failure paths.

End-to-end tests

  • A user can complete the primary journey end to end.

    npm run test:e2e

    Success: The main user journey completes successfully.

Security tests

  • No obvious security regressions are introduced.

    npm audit

    Success: No high or critical vulnerabilities.

Debugging & Troubleshooting

ImagePullBackOff

Why: Kubernetes cannot pull the container image.

Check: Inspect the pod events and image name.

kubectl describe pod <pod> | grep -A3 Events

Fix: Fix the image tag, registry auth, or availability.

Prevent: Verify images exist and pull policy is correct.

CrashLoopBackOff

Why: The container keeps starting and crashing.

Check: Read the container logs and previous logs.

kubectl logs <pod> --tail=30 && kubectl logs <pod> --previous

Fix: Fix the application error, missing config, or failing dependency.

Prevent: Add readiness/liveness probes to surface failures early.

Pod stuck Pending

Why: No node can fit the pod — resources, taints, or scheduling.

Check: Inspect events for scheduling failures.

kubectl describe pod <pod> | grep -A3 Events

Fix: Add capacity, remove taints, or reduce resource requests.

Prevent: Request resources your nodes can actually satisfy.

OOMKilled

Why: The container exceeded its memory limit.

Check: Inspect the container state and memory limits.

kubectl describe pod <pod> | grep -i memory

Fix: Raise the limit or reduce the application memory footprint.

Prevent: Set realistic limits and profile memory usage.

Security

Security is treated as a core requirement: input is validated, secrets are protected, and dependencies are kept up to date.

  • Input validation

    Validate and sanitize all user input before use.

  • Secrets management

    Store secrets in environment variables, never in source.

  • Dependency hygiene

    Regularly audit dependencies for known vulnerabilities.

Observability

Logs

  • Structured logs for every service
  • Centralized log storage with retention

Metrics

  • Service, infrastructure and business metrics
  • RED and USE coverage

Traces

  • Distributed traces across services and infrastructure

Health checks

  • Per-service health probes
  • External uptime checks

Alerts

  • Alerting on SLO burn and critical signals
  • On-call escalation paths

Dashboards

  • Service, infra and business dashboards
  • SLO dashboards

SLIs

  • Availability — Proportion of valid requests answered successfully
  • Latency — Distribution of response times
  • Error rate — Proportion of failed requests

SLOs

  • Availability — 99.9% over 30 days
  • Latency — P95 under 300ms

Deployment

  1. 1

    Prepare for deployment

    Build the production artifact.

    1. Build for production
    npm run build
  2. 2

    Deploy

    Deploy to the project target platform following its documented steps.

    1. Deploy
    npm run deploy   # or the platform-specific command
  3. 3

    Verify

    Confirm the deployed application responds and its health check passes.

    1. Check health
    curl -I https://<your-deployed-url>

Documentation

Project Overview

Build a production Kubernetes observability platform with Prometheus metrics, Grafana dashboards, log aggregation and alerting. This is a advanced observability project in the observability stack space, built to demonstrate real engineering, not just copied code.

What You Will Build

- An observability platform - Dashboards - Alerting

Problem Statement

Operators need unified signals to run production safely. Collect, store and visualise metrics, logs and alerts at scale.

Technology Stack

Prometheus, Grafana, Loki, Kubernetes

Installation

Clone the repository and install dependencies with the documented commands.

Local Development

Run the development command and open the printed local URL.

Testing

Run the test commands documented in the Testing section and confirm they pass.

Deployment

Follow the documented deployment steps for this project's target platform.

Interview Explanation

Use the Interview section to explain the architecture, technology choices and trade-offs.

Build Process

Phases guide the build in 9 checkpoints. Complete each and validate before moving on.

Interview Prep

30-second pitch

I built Production Kubernetes Observability Platform — observe a production kubernetes platform end to end. It uses Prometheus, Grafana, Loki and solves operators need unified signals to run production safely.

2-minute walkthrough

Build a production Kubernetes observability platform with Prometheus metrics, Grafana dashboards, log aggregation and alerting. The core requirements were Scrape config, Dashboards, Log collection. I structured the work around Operate a full observability stack, Define alerts, Link metrics and logs and validated the result with the documented tests.

Architecture

Because this is a observability project, the architecture is concentrated in observability stack rather than a distributed system.

Why these technologies

I chose Prometheus, Grafana, Loki, Kubernetes because they are the pragmatic, well-supported tools best suited to building a observability of this kind.

Why this architecture

The structure exists to keep concerns separated, make each piece testable, and scale the project cleanly as requirements grow.

At scale

At scale I would introduce caching, move to a service boundary structure, add proper observability, and enforce stricter contracts between components.

What can fail

The data source becomes unavailable, A dependency introduces a breaking change, Performance degrades under load, Configuration drift between environments

Monitoring

Monitoring is covered in the Observability section, focusing on distributed metrics, logs, traces and SLOs.

Security

Security was handled by validating input, protecting secrets, and following the practices listed in the Security section.

Deployment strategy

Deployment follows the steps in the Deployment section for the target platform, with a health check to confirm success.

What I'd do differently

With more time I would add deeper test coverage, stronger observability, and automated deployment.

Practice Q&A

Why did you build Production Kubernetes Observability Platform?

To demonstrate Operate a full observability stack and Define alerts on a real, portfolio-worthy project.

What was the hardest part?

Coordinating the pieces so each stays correct as the project grows, and validating end to end.

How do you handle failure and scale?

By designing for correctness first and adding resilience once the core works.

Extensions

  • Add deeper test coverage
  • Add observability and alerting
  • Harden security
  • Automate deployment

Completion Checklist

Prerequisite Projects

Next Steps

Recommended next

SRE-Style Production Reliability Platform

Turns observability into SLO-driven reliability.