Terraform state that survives your team

Most Terraform problems we get called in for are not Terraform problems. They are state problems that went unnoticed while exactly one person was running apply.

The failure mode is consistent. A single engineer builds the initial infrastructure with local state, everything works, and the repo gets shared. The second engineer runs apply, Terraform sees an empty state file, and proposes creating resources that already exist. If someone confirms that plan against production, the recovery is measured in hours.

Put state in a backend on day one

Local state is only appropriate for a scratch project you intend to destroy. Everything else starts with a remote backend and locking:

terraform {
  backend "s3" {
    bucket       = "acme-tfstate-prod"
    key          = "platform/network/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true
  }
}

Three properties matter here and each one prevents a specific incident:

• encrypt - state contains resource attributes, and depending on your providers, secrets. It is a sensitive artifact, not a build output.
• Locking - without it, two concurrent applies interleave writes and corrupt state. S3 native locking via use_lockfile replaced the older DynamoDB table approach; if you are on an older provider version, the dynamodb_table argument does the same job.
• A deliberate key - the state path is your blast radius boundary. Decide it consciously.

Split state by blast radius, not by tidiness

The instinct is to organize state files to mirror the directory structure. The better question is: what do I want to be able to destroy without thinking hard?

Networking, shared data stores, and per-service compute have very different change frequencies and very different consequences when something goes wrong. A single monolithic state file means a typo in a service definition puts the plan for your VPC on screen at the same time.

We generally land on three tiers:

1. Foundation - accounts, VPCs, subnets, transit gateways. Changes rarely, reviewed carefully.
2. Platform - clusters, shared databases, DNS zones, IAM baseline. Changes monthly.
3. Services - per-application resources. Changes daily, owned by the team that ships the service.

Cross-tier references go through remote state data sources or, better, through published outputs in a registry. Resist the urge to reach across tiers with hardcoded ARNs.

Make drift visible before it bites

Drift is not prevented by policy, it is caught by a schedule. A nightly terraform plan in CI that fails loudly on a non-empty diff turns "someone clicked something in the console three weeks ago" into a same-day notification.

terraform plan -detailed-exitcode -lock=false
# exit 0 = no changes, 2 = drift detected, 1 = error

Wire exit code 2 to a notification rather than a build failure. Drift is information, not necessarily an emergency, and teams that get paged for it quickly learn to ignore the page.

The handover test

The question we ask at the end of every infrastructure engagement is simple: if the person who wrote this left tomorrow, could someone else rebuild the environment from the repository alone?

If the answer requires a caveat - a manual step, a console setting, a credential that lives in someone's password manager - that caveat is the next piece of work.