Projects
ProjectsReliability / Production EngineeringadvancedReliability

SRE-Style Production Reliability Platform

SLI, SLO, alerting, incident response and error budgets.

~22 hours 9 phases Capstone
PrometheusGrafanaAlertingIncident tooling

Overview

Implement an SRE reliability platform with SLIs and SLOs, error budgets, alerting and a documented incident response process.

The problem

Run a service with SLOs, error budgets and incident response.

Why it matters

This is a real, in-demand scenario: Run a service with SLOs, error budgets and incident response.. Teams that can ship and run a reliability like this solve an actual business need and are hired for that ability.

What you'll build

  • SLI/SLO platform
  • Error budgets
  • Reliable alerting

What you'll learn

  • SRE
  • SLO engineering
  • Incident management

Business problem

Reliability must be measured, budgeted and protected.

Technical problem

Turn signals into SLOs, budgets and alerting.

When you're done: A working reliability project — code committed, tests passing, and documented in your portfolio.

Requirements

User requirements

  • Measured SLIs
  • SLO windows
  • Controlled alerting

System requirements

  • Metrics
  • SLO computation
  • Alerting

Functional requirements

  • SLI definition
  • SLO windows
  • Burn rate alerts
  • Error budget view

Non-functional requirements

  • Actionable
  • Bounded

Technical requirements

  • Prometheus rules
  • dashboards
  • alertmanager

Security requirements

  • Restrict dashboards

Operational requirements

  • Incident runbook

Constraints

  • Existing stack

Success criteria

  • SLOs computed
  • Error budgets tracked

Flows

Request Flow

  1. 1

    Browser

    User requests the application over HTTPS.

  2. 2

    CDN / DNS

    Edge routes the request to the origin.

  3. 3

    Load Balancer

    Distributes traffic across healthy app instances.

  4. 4

    Application

    Handles the request, applies business logic.

  5. 5

    Cache

    Serves fast reads for hot data.

  6. 6

    Database

    Persists data and returns results.

Project Structure

Root of the client application source.

Setup

  1. 1

    Prerequisites

    You need the tools required by Prometheus, Grafana, Alerting installed and available on your PATH.

  2. 2

    Clone project

    Clone the repository locally.

    1. Clone the repository
    git clone <your-repo-url> <project-folder>
  3. 3

    Install dependencies

    Install the project dependencies.

    1. Initialize a Node.js project
    npm init -y

    Expected

    Creates a package.json with defaults.

    If it fails

    command not found: npm

    Fix

    Install Node.js LTS from the official site, then reopen your terminal.

  4. 4

    Configure environment

    Set the environment variables documented for this project. Never commit secrets.

    1. Create the environment file
    cp .env.example .env   # then edit values
  5. 5

    Start project

    Run the development command and confirm it starts.

    1. Start the development server
    npm run dev

    Expected

    Dev server starts; watch terminal for the local URL.

    If it fails

    Port 3000 is already in use

    Fix

    Run on another port: npm run dev -- --port 3001, or stop the conflicting process.

  6. 6

    Verify

    Open the printed local URL and confirm the primary flow works.

Build Phases

0 of 9 phases complete

  1. Objective: Establish the next milestone of the reliable production system.

    What changes: The project advances to the next coherent state, keeping the previous milestones working.

    Files touched

    Relevant source and configuration files for this milestone.

    Run

    1. Initialize a Node.js project
    npm init -y

    Expected

    Creates a package.json with defaults.

    If it fails

    command not found: npm

    Fix

    Install Node.js LTS from the official site, then reopen your terminal.

    Tasks

    • Implement the specific behaviour required for this milestone.
    • Verify the previous milestone still works.

    Validate: The milestone behaves as expected and does not break existing functionality.

    Common mistakes

    • • Skipping validation and moving on before confirming the milestone works.

  2. Objective: Establish the next milestone of the reliable production system.

    What changes: The project advances to the next coherent state, keeping the previous milestones working.

    Files touched

    Relevant source and configuration files for this milestone.

    Run

    1. Install the prometheus package
    npm install prometheus

    Expected

    Adds prometheus to node_modules and package.json.

    If it fails

    npm ERR! network / EACCES

    Fix

    Check your network or run with sudo on Linux/macOS; on Windows run the terminal as Administrator.

    Tasks

    • Implement the specific behaviour required for this milestone.
    • Verify the previous milestone still works.

    Validate: The milestone behaves as expected and does not break existing functionality.

    Common mistakes

    • • Skipping validation and moving on before confirming the milestone works.

  3. Objective: Establish the next milestone of the reliable production system.

    What changes: The project advances to the next coherent state, keeping the previous milestones working.

    Files touched

    Relevant source and configuration files for this milestone.

    Run

    1. Start the development server
    npm run dev

    Expected

    Dev server starts; watch terminal for the local URL.

    If it fails

    Port 3000 is already in use

    Fix

    Run on another port: npm run dev -- --port 3001, or stop the conflicting process.

    Tasks

    • Implement the specific behaviour required for this milestone.
    • Verify the previous milestone still works.

    Validate: The milestone behaves as expected and does not break existing functionality.

    Common mistakes

    • • Skipping validation and moving on before confirming the milestone works.

  4. Objective: Establish the next milestone of the reliable production system.

    What changes: The project advances to the next coherent state, keeping the previous milestones working.

    Files touched

    Relevant source and configuration files for this milestone.

    Run

    1. Build the application for production
    npm run build

    Expected

    A production bundle is emitted to the output directory.

    If it fails

    Build fails with a type or syntax error

    Fix

    Read the error message, fix the referenced file, and re-run the build.

    Tasks

    • Implement the specific behaviour required for this milestone.
    • Verify the previous milestone still works.

    Validate: The milestone behaves as expected and does not break existing functionality.

    Common mistakes

    • • Skipping validation and moving on before confirming the milestone works.

  5. Objective: Establish the next milestone of the reliable production system.

    What changes: The project advances to the next coherent state, keeping the previous milestones working.

    Files touched

    Relevant source and configuration files for this milestone.

    Tasks

    • Implement the specific behaviour required for this milestone.
    • Verify the previous milestone still works.

    Validate: The milestone behaves as expected and does not break existing functionality.

    Common mistakes

    • • Skipping validation and moving on before confirming the milestone works.

  6. Objective: Establish the next milestone of the reliable production system.

    What changes: The project advances to the next coherent state, keeping the previous milestones working.

    Files touched

    Relevant source and configuration files for this milestone.

    Tasks

    • Implement the specific behaviour required for this milestone.
    • Verify the previous milestone still works.

    Validate: The milestone behaves as expected and does not break existing functionality.

    Common mistakes

    • • Skipping validation and moving on before confirming the milestone works.

  7. Objective: Establish the next milestone of the reliable production system.

    What changes: The project advances to the next coherent state, keeping the previous milestones working.

    Files touched

    Relevant source and configuration files for this milestone.

    Tasks

    • Implement the specific behaviour required for this milestone.
    • Verify the previous milestone still works.

    Validate: The milestone behaves as expected and does not break existing functionality.

    Common mistakes

    • • Skipping validation and moving on before confirming the milestone works.

  8. Objective: Establish the next milestone of the reliable production system.

    What changes: The project advances to the next coherent state, keeping the previous milestones working.

    Files touched

    Relevant source and configuration files for this milestone.

    Tasks

    • Implement the specific behaviour required for this milestone.
    • Verify the previous milestone still works.

    Validate: The milestone behaves as expected and does not break existing functionality.

    Common mistakes

    • • Skipping validation and moving on before confirming the milestone works.

  9. Objective: Establish the next milestone of the reliable production system.

    What changes: The project advances to the next coherent state, keeping the previous milestones working.

    Files touched

    Relevant source and configuration files for this milestone.

    Tasks

    • Implement the specific behaviour required for this milestone.
    • Verify the previous milestone still works.

    Validate: The milestone behaves as expected and does not break existing functionality.

    Common mistakes

    • • Skipping validation and moving on before confirming the milestone works.

Commands

1. Initialize a Node.js project
npm init -y

Expected

Creates a package.json with defaults.

If it fails

command not found: npm

Fix

Install Node.js LTS from the official site, then reopen your terminal.

2. Install the prometheus package
npm install prometheus

Expected

Adds prometheus to node_modules and package.json.

If it fails

npm ERR! network / EACCES

Fix

Check your network or run with sudo on Linux/macOS; on Windows run the terminal as Administrator.

3. Start the development server
npm run dev

Expected

Dev server starts; watch terminal for the local URL.

If it fails

Port 3000 is already in use

Fix

Run on another port: npm run dev -- --port 3001, or stop the conflicting process.

4. Build the application for production
npm run build

Expected

A production bundle is emitted to the output directory.

If it fails

Build fails with a type or syntax error

Fix

Read the error message, fix the referenced file, and re-run the build.

Testing

Unit tests

  • Core logic and pure functions behave correctly.

    npm init -y

    Success: All unit tests pass without errors.

Integration tests

  • Components integrate correctly with data sources and services.

    npm test

    Success: Integration tests pass for the primary flows.

API tests

  • Endpoints respond to valid and invalid input correctly.

    npm test

    Success: API tests cover happy and failure paths.

End-to-end tests

  • A user can complete the primary journey end to end.

    npm run test:e2e

    Success: The main user journey completes successfully.

Security tests

  • No obvious security regressions are introduced.

    npm audit

    Success: No high or critical vulnerabilities.

Debugging & Troubleshooting

ImagePullBackOff

Why: Kubernetes cannot pull the container image.

Check: Inspect the pod events and image name.

kubectl describe pod <pod> | grep -A3 Events

Fix: Fix the image tag, registry auth, or availability.

Prevent: Verify images exist and pull policy is correct.

CrashLoopBackOff

Why: The container keeps starting and crashing.

Check: Read the container logs and previous logs.

kubectl logs <pod> --tail=30 && kubectl logs <pod> --previous

Fix: Fix the application error, missing config, or failing dependency.

Prevent: Add readiness/liveness probes to surface failures early.

Pod stuck Pending

Why: No node can fit the pod — resources, taints, or scheduling.

Check: Inspect events for scheduling failures.

kubectl describe pod <pod> | grep -A3 Events

Fix: Add capacity, remove taints, or reduce resource requests.

Prevent: Request resources your nodes can actually satisfy.

OOMKilled

Why: The container exceeded its memory limit.

Check: Inspect the container state and memory limits.

kubectl describe pod <pod> | grep -i memory

Fix: Raise the limit or reduce the application memory footprint.

Prevent: Set realistic limits and profile memory usage.

Security

Security is treated as a core requirement: input is validated, secrets are protected, and dependencies are kept up to date.

  • Input validation

    Validate and sanitize all user input before use.

  • Secrets management

    Store secrets in environment variables, never in source.

  • Dependency hygiene

    Regularly audit dependencies for known vulnerabilities.

Observability

Logs

  • Structured logs for every service
  • Centralized log storage with retention

Metrics

  • Service, infrastructure and business metrics
  • RED and USE coverage

Traces

  • Distributed traces across services and infrastructure

Health checks

  • Per-service health probes
  • External uptime checks

Alerts

  • Alerting on SLO burn and critical signals
  • On-call escalation paths

Dashboards

  • Service, infra and business dashboards
  • SLO dashboards

SLIs

  • Availability — Proportion of valid requests answered successfully
  • Latency — Distribution of response times
  • Error rate — Proportion of failed requests

SLOs

  • Availability — 99.9% over 30 days
  • Latency — P95 under 300ms

Deployment

  1. 1

    Prepare for deployment

    Build the production artifact.

    1. Build for production
    npm run build
  2. 2

    Deploy

    Deploy to the project target platform following its documented steps.

    1. Deploy
    npm run deploy   # or the platform-specific command
  3. 3

    Verify

    Confirm the deployed application responds and its health check passes.

    1. Check health
    curl -I https://<your-deployed-url>

Documentation

Project Overview

Implement an SRE reliability platform with SLIs and SLOs, error budgets, alerting and a documented incident response process. This is a advanced reliability project in the reliable production system space, built to demonstrate real engineering, not just copied code.

What You Will Build

- SLI/SLO platform - Error budgets - Reliable alerting

Problem Statement

Reliability must be measured, budgeted and protected. Turn signals into SLOs, budgets and alerting.

Technology Stack

Prometheus, Grafana, Alerting, Incident tooling

Installation

Clone the repository and install dependencies with the documented commands.

Local Development

Run the development command and open the printed local URL.

Testing

Run the test commands documented in the Testing section and confirm they pass.

Deployment

Follow the documented deployment steps for this project's target platform.

Interview Explanation

Use the Interview section to explain the architecture, technology choices and trade-offs.

Build Process

Phases guide the build in 9 checkpoints. Complete each and validate before moving on.

Interview Prep

30-second pitch

I built SRE-Style Production Reliability Platform — run a service with slos, error budgets and incident response. It uses Prometheus, Grafana, Alerting and solves reliability must be measured, budgeted and protected.

2-minute walkthrough

Implement an SRE reliability platform with SLIs and SLOs, error budgets, alerting and a documented incident response process. The core requirements were SLI definition, SLO windows, Burn rate alerts. I structured the work around Define SLIs/SLOs, Burn-rate alerting, Manage error budgets and validated the result with the documented tests.

Architecture

Because this is a reliability-production project, the architecture is concentrated in reliable production system rather than a distributed system.

Why these technologies

I chose Prometheus, Grafana, Alerting, Incident tooling because they are the pragmatic, well-supported tools best suited to building a reliability of this kind.

Why this architecture

The structure exists to keep concerns separated, make each piece testable, and scale the project cleanly as requirements grow.

At scale

At scale I would introduce caching, move to a service boundary structure, add proper observability, and enforce stricter contracts between components.

What can fail

The data source becomes unavailable, A dependency introduces a breaking change, Performance degrades under load, Configuration drift between environments

Monitoring

Monitoring is covered in the Observability section, focusing on distributed metrics, logs, traces and SLOs.

Security

Security was handled by validating input, protecting secrets, and following the practices listed in the Security section.

Deployment strategy

Deployment follows the steps in the Deployment section for the target platform, with a health check to confirm success.

What I'd do differently

With more time I would add deeper test coverage, stronger observability, and automated deployment.

Practice Q&A

Why did you build SRE-Style Production Reliability Platform?

To demonstrate Define SLIs/SLOs and Burn-rate alerting on a real, portfolio-worthy project.

What was the hardest part?

Coordinating the pieces so each stays correct as the project grows, and validating end to end.

How do you handle failure and scale?

By designing for correctness first and adding resilience once the core works.

Extensions

  • Add deeper test coverage
  • Add observability and alerting
  • Harden security
  • Automate deployment

Completion Checklist

Prerequisite Projects

Next Steps

Recommended next

Microservices Deployment Platform

Combines reliability with multi-service delivery.