DevOps Stack

Git tracks history as a graph of commits, and a branch is just a movable pointer into it. The mechanics are simple; the disagreements are all about workflow — trunk-based development with short-lived branches, or long-lived release branches merged periodically.

The choice matters more than it looks. Long-lived branches accumulate merge risk in proportion to how long they live, so teams that integrate daily spend less time resolving conflicts than teams that integrate monthly, even though each individual merge feels smaller. Trunk-based development is the modern default for exactly this reason, and it is what makes continuous integration possible at all.

Every change is merged into the mainline frequently, and each merge triggers an automated build and test run. The point is not the automation — it is the frequency. CI without frequent merging is just a build server.

What it buys is a short distance between introducing a defect and discovering it. A test that fails within ten minutes of your commit is a small problem; the same failure found three weeks later, in someone else's branch, is an investigation. The discipline that makes it work is keeping the build green — a pipeline that is habitually red teaches everyone to ignore it.

Continuous delivery means every change that passes the pipeline is releasable, and a human decides when to release. Continuous deployment removes that human — anything that passes goes to production automatically. The distinction is one approval gate, and people use the terms interchangeably to their cost.

Deployment frequency turns out to correlate with stability rather than against it, which is counterintuitive until you notice that small frequent changes are easier to diagnose and reverse than large infrequent ones. The prerequisite for either is confidence in the test suite, since the pipeline is only as trustworthy as what it checks.

An artefact is the immutable output of a build — a container image, a wheel, a jar — stored in a registry and identified by a version or content hash. The rule that makes deployment predictable is build once, deploy everywhere: the exact bytes tested in staging are the bytes that reach production.

Rebuilding per environment breaks that guarantee silently, because a dependency resolved at build time may resolve differently an hour later. Pinning versions and recording the commit that produced each artefact is what makes "which code is running in production" an answerable question.

Development, staging, production — the same artefact promoted through progressively more realistic environments, with configuration injected rather than baked in. Config comes from environment variables or a config service; secrets come from a secrets manager.

The recurring failure is environment drift: staging diverges from production in data volume, hardware, network topology, or third-party integrations, and stops predicting anything. Staging that differs materially from production gives false confidence, which is worse than having no staging at all because it is trusted.

A container packages an application with its dependencies and runs as an isolated process on a shared kernel — much lighter than a virtual machine, which carries a whole operating system. An image is the immutable filesystem it starts from, built in layers that are cached and shared between images.

Layer ordering is the practical skill: put the things that change rarely — base image, system packages, dependencies — early, and your own code last, so a code change rebuilds one small layer instead of everything. Multi-stage builds let you compile in a fat image and ship a thin one, which cuts both image size and attack surface.

Where images live: Docker Hub, ECR, GCR, GHCR, or a self-hosted one. Images are addressed by tag or by digest, and the difference matters — a tag is a mutable pointer, so :latest can change underneath you between two deployments that look identical.

Referencing by digest rather than tag is what makes a deployment reproducible. Registries are also where scanning and signing happen, so they are a natural enforcement point for supply chain policy.

A declarative orchestrator: you describe the desired state — this many replicas of this image with these resources — and controllers continuously reconcile reality toward it. Pods are the unit of scheduling, Deployments manage replica sets, Services provide stable networking, and Ingress handles external traffic.

The mental shift is from imperative to declarative. You do not start containers; you declare what should exist and the control loop makes it so, including after a node dies. The cost is real operational complexity, and the honest question is whether you need it — a managed container service is sufficient for a great many workloads that reach for Kubernetes because it is the default answer.

Kubernetes manifests are YAML, and YAML does not template itself. Helm packages a set of manifests as a chart with parameterised values, so the same application can be deployed to different environments by changing a values file rather than by copying and editing manifests.

Alternatives worth knowing: Kustomize does overlays without templating, which some teams prefer because the output stays readable YAML. The failure mode of both is the same — configuration sprawl, where the values files become the real complexity and nobody can say what a given environment is actually running.

A layer that moves cross-cutting network concerns — mutual TLS, retries, timeouts, traffic splitting, per-request telemetry — out of application code and into sidecar proxies alongside each service. Istio and Linkerd are the common implementations.

It is genuinely useful at a certain scale and considerable overhead below it. The trade is a uniform policy and observability layer against another distributed system to operate and debug, so the sensible sequence is to reach for it when the problems it solves are actually hurting, not in anticipation.

Infrastructure defined in files, version-controlled and applied by a tool rather than clicked together in a console. Terraform is the common choice because it is declarative and cloud-agnostic; CloudFormation, Pulumi, and CDK occupy the same space with different trade-offs.

What it actually buys is reviewability and repeatability — infrastructure changes go through pull request like code, and an environment can be recreated from scratch. The discipline required is that nobody changes anything by hand, because a single console edit puts reality out of step with the code and the next apply will either revert it or fail.

Terraform records what it believes exists in a state file, and compares that to your configuration to compute a plan. The state file is therefore critical infrastructure: it must be stored remotely, locked during writes so two people cannot apply simultaneously, and never edited casually.

Drift is reality diverging from state — a resource changed manually, or deleted, or created by another process. Detecting it early is the point of running plan regularly rather than only before an apply, because drift discovered during an incident is drift discovered at the worst possible time.

Ansible, Chef, and Puppet configure the inside of machines — packages, files, services — as opposed to provisioning the machines themselves. Ansible is agentless and runs over SSH, which is why it survived into a container-dominated world where the others largely did not.

Containers absorbed most of this job: if the image is immutable and complete, there is nothing left to configure at runtime. Configuration management remains relevant for the machines underneath — bare metal, VMs, network appliances, and anything that cannot be thrown away and rebuilt.

The git repository is the single source of truth for what should be running, and an in-cluster agent — Argo CD or Flux — continuously pulls and reconciles the cluster toward it. Deployment becomes a merge rather than a pipeline step that pushes.

Two properties follow. The cluster self-heals toward the declared state, so a manual change is reverted automatically. And the audit trail is the git history, which answers "who changed what and when" without a separate system — valuable enough that regulated environments often adopt it for that alone.

Replace instances a few at a time, waiting for each batch to become healthy before continuing. The default in Kubernetes, and the cheapest safe option because it needs no extra capacity beyond the surge allowance.

Its weakness is that both versions serve traffic during the roll, so the new version must be backward compatible with the old one's data and API contract. Rollback is another roll in the opposite direction, which is not instant — that is the argument for blue-green when speed of reversal matters.

Run two complete environments. Blue serves production; green receives the new version and is tested; then traffic switches at the load balancer. Rollback is switching back, which takes seconds.

The cost is double the infrastructure during the cutover, and the complication is anything with state — databases, caches, in-flight sessions do not switch cleanly with the traffic. Which is why schema changes are usually decoupled from deployments entirely and rolled out as separately compatible steps.

Send a small fraction of traffic — 1%, then 5%, then more — to the new version while watching error rates and latency, and abort if they degrade. It limits the blast radius of a bad release to the fraction exposed.

What makes a canary work is not the traffic split but the automated decision to abort. A canary nobody is monitoring is just a slow deployment. The subtlety is that low-frequency failures need a large enough sample or a long enough soak to show up at all.

A runtime switch that decouples deploying code from releasing behaviour. Code ships dark, is enabled for internal users, then a percentage, then everyone — and is turned off instantly if something is wrong, without a deployment.

They are the fastest rollback mechanism available and they accumulate debt relentlessly. Every flag is a branch in the runtime, and the number of possible states doubles with each one, so flags need removal dates and someone accountable for deleting them once a rollout is complete.

Three signals answering different questions. Metrics are cheap numeric aggregates over time — is something wrong. Logs are discrete events with detail — what exactly happened. Traces follow one request across services — where the time went.

Cost profiles differ sharply: metrics are cheap and bounded, logs are expensive at volume, traces are usually sampled. The distinction that matters is monitoring versus observability — monitoring answers questions you knew to ask in advance, observability lets you ask new ones about a failure nobody anticipated.

Four signals cover most of what a service needs: latency, traffic, errors, and saturation. Start there rather than instrumenting everything, because a dashboard with sixty panels is one nobody reads during an incident.

Report latency as percentiles, not averages — a mean hides the tail that users actually experience, and averaging percentiles across instances is mathematically meaningless. Dashboards earn their place by being useful at 3am, which is a much higher bar than being comprehensive.

An SLI is a measured indicator, such as the proportion of requests served under 300 ms. An SLO is the target for it — 99.9% over 30 days. The gap between the target and 100% is the error budget: the amount of failure you have agreed is acceptable.

The budget is what makes the framework useful, because it converts reliability from an argument into arithmetic. Budget remaining means you can ship aggressively; budget exhausted means reliability work takes priority. It also makes explicit that 100% is the wrong target — the cost curve goes vertical, and users cannot tell the difference through their own network.

An alert should mean a human needs to act now. Anything else belongs on a dashboard or in a ticket. The standard that works is alert on symptoms, not causes — page when users are affected, not when CPU is high, because high CPU is often fine and users being affected never is.

Alert fatigue is the dominant failure mode and it is a design problem, not a discipline problem: an on-call rotation that fires forty times a night trains people to acknowledge without reading. Every page should be reviewed afterwards for whether it was actionable, and the unactionable ones deleted.

During an incident, roles beat heroics: someone commands, someone communicates, someone investigates. Mitigate before diagnosing — roll back, fail over, shed load — because understanding the cause is a slower path to restoring service than reversing the change.

Afterwards, a blameless postmortem asks how the system allowed the failure rather than who caused it. This is not politeness; it is the only way to get accurate information, since people who expect blame report less. The output is a small number of tracked actions — a postmortem with twenty action items and no owners produces nothing.

Distributing traffic across instances, at layer 4 on connections or layer 7 on requests. Layer 7 costs more and buys routing on path, header, or weight, which is what canary deployments and A/B routing depend on.

Health checks decide who receives traffic, and the distinction worth getting right is liveness versus readiness: liveness failing means restart me, readiness failing means stop sending traffic but leave me alone. Conflating them produces restart loops during a slow dependency outage, turning a degradation into an outage.

Adding and removing capacity against a signal — CPU, request rate, or queue depth. Horizontal scaling adds instances; vertical scaling makes them bigger and eventually hits a ceiling.

The practical problems are timing. Scaling reacts after load arrives, so a slow start-up means you are always behind a spike — which is why pre-warming and predictive scaling exist for known traffic patterns. Scale-in needs to be more conservative than scale-out, or you get flapping, and queue depth is usually a better signal than CPU because it measures the backlog directly.

Every network call needs a timeout, because the default is to wait forever and one slow dependency will then exhaust your connection pool. Retries need exponential backoff with jitter — synchronised retries from many clients turn a brief blip into a stampede that keeps the dependency down.

A circuit breaker stops calling a failing dependency entirely for a period, failing fast instead of queuing, then probes to see if it has recovered. Together with bulkheads — isolating resource pools per dependency — this is what stops one failure cascading into the whole system.

Deliberately injecting failure — killing instances, adding latency, partitioning networks — to verify that the resilience you designed actually works. The premise is that untested failover is not failover, it is an assumption.

It only makes sense once you have observability good enough to see what happens and a blast radius you can contain. Start in staging with a hypothesis, do it during working hours with everyone watching, and have the abort ready first.

Credentials belong in a dedicated store — Vault, AWS Secrets Manager, or the cloud equivalent — injected at runtime, never in the repository or the image. Committed secrets are permanent: git history keeps them after deletion, so the only real remedy is rotation.

The more durable answer is to avoid long-lived credentials entirely using workload identity, where a service assumes a role and receives short-lived tokens automatically. A secret that expires in an hour is a much smaller problem than one that never does.

Every identity — human, service, pipeline — gets the narrowest set of permissions that lets it do its job, and no more. It is the control that limits how far a compromise or a mistake can travel.

It erodes by default, because broad permissions are the fast way to unblock someone at 5pm and nobody narrows them afterwards. The counters are time-bound elevation rather than standing access, and periodic review of what permissions are actually being exercised versus merely granted.

Most of what you ship is somebody else's code. That makes dependencies an attack surface: typosquatted packages, compromised maintainer accounts, and malicious updates to legitimate libraries are all real and all observed in the wild.

The controls are lockfiles so builds are reproducible, an SBOM recording what is actually in an artefact, automated scanning of dependencies and images, and signing so you can verify an artefact came from your pipeline. The SBOM matters most on the day a CVE lands, when the first question is simply which of your systems contain the affected version.

Moving checks earlier in the cycle — static analysis, dependency scanning, IaC policy checks, and tests running on every commit rather than in a review before release. A defect caught in the editor costs a fraction of the same defect caught in production.

The failure mode is gates that everyone learns to bypass: a scanner producing hundreds of findings, mostly noise, gets waived as a matter of routine. Fewer checks that block reliably beat comprehensive checks that are habitually overridden, which makes tuning the signal the actual work.