Terraform in Production: A No-Nonsense Guide to GitOps, Remote State, and CI/CD

So much Infrastructure as Code (IaC) documentation stops at terraform apply. You've provisioned a VPC and an EC2 instance from your laptop. It works. Ship it, right?

Wrong. Production isn't a laptop. It's a shared, living system where state is the source of truth, manual applies are a ticking time bomb, and the human running the command is the single biggest risk. If you've ever spent a Sunday afternoon fighting a corrupted state file or untangling a drift nightmare, you know the pain.

This guide isn't theory. It's the exact workflow I use and have debugged for managing multi-cloud infrastructure safely. We'll bridge the gap between a simple main.tf and a production-grade setup. We'll cover the non-negotiables: a secure remote backend, a strict GitOps flow, and a CI/CD pipeline that applies changes without direct human access to cloud secrets.

Who Are the Actors in This Setup?

Before we touch code, we need to understand the roles. In a secure setup, no single person has god-mode access. Permissions are split between systems and human reviewers.

The goal is simple: Humans propose; machines enforce. The CI/CD runner is the only actor with write access to the cloud. You, the engineer, don't even need cloud credentials stored locally.

Part 1: The Foundation - Production-Grade Remote State

Running Terraform locally means your state file sits on your machine. That's a single point of failure for collaboration, security, and file corruption. A remote backend is non-negotiable. The big three options are AWS S3, Azure Storage, and Terraform Cloud.

Let's look at a real-world Azure setup, which is a perfect example of the "belt and suspenders" approach.

Azure Remote Backend: Locking & Encryption by Default

This configuration solves three critical problems:

1. Shared State: Everyone accesses the same source of truth.
2. State Locking: Uses Azure Blob Storage leases to prevent two engineers from running apply simultaneously.
3. Encryption: Data is encrypted at rest and in transit.

terraform {
  backend "azurerm" {
    resource_group_name  = "rg-terraform-state-prod"
    storage_account_name = "stterraformstateprod"
    container_name       = "tfstate"
    key                  = "production/network.terraform.tfstate"
    
    # Locking is enabled by default
    # use_azuread_auth = true # Use this instead of access keys for auth!
  }
}

The Real Fix: I've seen a team locked out of their cloud because the person who ran the last apply was on vacation, and the state file lived on their locked laptop. The hard-earned lesson? If a single person leaving stops you from deploying, it's not production-ready. Remote backends fix this.

Part 2: The Workflow - GitOps, Not ClickOps

GitOps means your Git repository is the single source of truth. Not the Terraform Cloud GUI, not the AWS Console, and definitely not a manual CLI command. The desired state of your infrastructure is declared in .tf files on the main branch. The system continuously reconciles the actual state to match.

Our Git branching strategy is dead simple but strictly enforced:

• main branch: The sacred, protected branch. Merges here trigger an automatic apply to production. No direct commits are allowed.
• Feature branches: All changes, from new resources to variable tweaks, happen here. They trigger a terraform plan to be posted as a comment on the Pull Request.

Part 3: The Pipeline - CI/CD with Secure Authentication

The CI/CD pipeline is the engine room. Its job is to authenticate, validate, plan, and apply infrastructure changes securely. The most critical part is how the pipeline authenticates to the cloud provider. Under no circumstances should you store long-lived access keys as CI/CD secrets. Use OpenID Connect (OIDC) federation.

The Critical Fix: OIDC Authentication (AWS & Azure)

This was a pain point I debugged for hours. We had an incident where a leaked CI variable containing AWS keys was used to spin up crypto miners. The exact fix? Rip out all static keys and use OIDC, which generates short-lived tokens for each pipeline run.

Here's how the trust is set up so your CI runner can assume a role in your cloud account.

AWS OIDC Provider Setup

resource "aws_iam_openid_connect_provider" "github" {
  url = "https://token.actions.githubusercontent.com"

  client_id_list = ["sts.amazonaws.com"]

  thumbprint_list = ["6938fd4d98bab03faadb97b34396831e3780aea1"]
}

resource "aws_iam_role" "terraform_ci" {
  name = "terraform-ci-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = { Federated = aws_iam_openid_connect_provider.github.arn }
      Action = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "token.actions.githubusercontent.com:aud" : "sts.amazonaws.com"
        },
        StringLike = {
          "token.actions.githubusercontent.com:sub" : "repo:your-org/your-repo:*"
        }
      }
    }]
  })
}

With the trust established, the pipeline can be secure and stateless. Here's a generic, production-ready pipeline definition. This isn't just a plan and apply; it's a gated process.

Pipeline Stages Diagram

Production CI/CD Configuration (GitHub Actions)

This workflow translates the diagram into code. Notice the "Re-plan" step before apply. This is a crucial safety net to ensure nothing changed between the PR merge and the apply.

name: 'Terraform GitOps'
on:
  pull_request:
    branches: [ main ]
  push:
    branches: [ main ]

permissions:
  id-token: write # Required for OIDC
  contents: read

jobs:
  terraform:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3

      - name: Terraform Init
        run: terraform init
        # No credentials needed! Auth is handled by the 'configure' step.

      # --- Plan Stage (For both PRs and Merges) ---
      - name: Terraform Plan
        id: plan
        run: terraform plan -no-color -out=tfplan
        continue-on-error: true # Don't fail on plan changes

      - name: Update Pull Request
        uses: actions/github-script@v7
        if: github.event_name == 'pull_request'
        with:
          script: |
            const output = `#### Terraform Plan \`${{ steps.plan.outcome }}\`
            \`\`\`hcl\n${{ steps.plan.outputs.stdout }}\n\`\`\`
            *Pusher: @${{ github.actor }}*`;
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: output
            });

      # --- Apply Stage (Only on merge to main) ---
      - name: Terraform Apply
        if: github.event_name == 'push' && steps.plan.outcome == 'success'
        run: terraform apply -auto-approve tfplan

Hard-Won Production Lessons (Real Debugging Notes)

A guide isn't complete without the scars to prove it. Here are the exact fixes to problems that wasted hours of my time.

1. The "State Locking is Not Optional" Incident: The issue: Two engineers ran apply on different PRs at the same time. They were working on the same state file but didn't know it. Result: a corrupted state file that terraform force-unlock couldn't fix. The fix was a manual state recovery from a backup. The permanent fix: setting prevent_destroy = true on life-critical state resources like databases and using DynamoDB/Blob lease locking religiously.

2. The "Too Many Permissions" Drift: We gave our CI role AdministratorAccess because it was easy. A wrong variable in a PR deleted a production security group. The exact fix was writing a scoped IAM policy that only allows what Terraform needs to manage—no more "*" on delete actions.

3. The "Mystery Drift" Nightmare: We had engineers manually tweaking auto-scaling group sizes during incidents from the AWS console. The next Terraform run reset them, causing a mini-outage. The fix? A CI/CD scheduled task that runs terraform plan -detailed-exitcode every hour. If it detects drift, it sends an alert and creates an automatic ticket. Your infrastructure shouldn't have secrets.

Final Word

This is the setup we use every day. It's not glamorous, but it's solid. It removes the heroics from infrastructure management and replaces them with a boring, predictable pipeline. A laptop is for writing code, not for applying it.

The goal is to make terraform apply an event so mundane that nobody bats an eye when it happens. That's when you've truly bridged the gap to production.