YahyaTechnical writing on cloud infrastructure, distributed systems, and reliability engineering.https://yahyaoncloud.com/blog2026-09-13T01:46:58.832ZYahyahello@yahyaoncloud.comArchitecting Observable & Resilient Cloud Infrastructure with GitOpshttps://yahyaoncloud.com/blog/building-observable-resilient-cloud-infrastructure2026-09-10T00:00:00.000Z2026-09-10T00:00:00.000ZLocal LLM Setup Baits: Common Pitfalls, Fake Benchmark Claims, and Reality Checkshttps://yahyaoncloud.com/blog/local-llm-setup-baits-pitfalls-and-reality-checks2026-06-25T00:00:00.000Z2026-06-25T00:00:00.000ZAgentic Monitoring in Production: Replacing Static Dashboards with Autonomous Diagnostic Agentshttps://yahyaoncloud.com/blog/agentic-monitoring-production-autonomous-diagnostic-agents2026-06-04T00:00:00.000Z2026-06-04T00:00:00.000Z 85%` or `HTTP 500 rate > 2%`) produce alert fatigue and fail to diagnose complex, multi-service cascading failures. Agentic monitoring shifts observability from passive alerting to active, autonomous diagnostics.
Instead of waking up on-call engineers at 3 AM with raw log dumps, autonomous agents execute diagnostic routines, isolate root causes, and propose or trigger remediation workflows.
## How an Agentic Monitoring Loop Works
1. **Trigger Phase**: Prometheus or vector log collectors detect an anomaly trace.
2. **Investigation Loop**: An LLM agent (equipped with tool-calling capabilities) executes CLI diagnostic commands (`kubectl logs`, `kubectl describe pod`, querying Jaeger distributed traces).
3. **Synthesis & Action**: The agent compiles a root-cause summary, posts it to Slack/Opsgenie, and executes pre-approved remediation scripts (e.g., restarting broken worker pods or scaling deployment replicas).
```typescript
// Example Agentic Diagnostic Function in TypeScript
interface DiagnosticTool {
name: string;
execute: (params: Record) => Promise;
}
const kubectlLogTool: DiagnosticTool = {
name: "get_pod_logs",
execute: async ({ podName, namespace }) => {
return await execCommand(`kubectl logs ${podName} -n ${namespace} --tail=50`);
}
};
```
## Practical Guardrails for Production Agents
- **Read-Only First**: Restrict autonomous agents to read-only diagnostic commands during initial rollout.
- **Human-in-the-Loop Approval**: Require a single-click Slack button approval before executing destructive remediation tasks (e.g., database failovers or pod deletions).]]>Terraform in Production: A No-Nonsense Guide to GitOps, Remote State, and CI/CDhttps://yahyaoncloud.com/blog/terraform-in-production-gitops-remote-state-cicd2026-05-28T00:00:00.000Z2026-05-28T00:00:00.000Z|git push| GitRepo
Rev[Senior Reviewer] -->|Approves PR| GitRepo
end
subgraph System_Actors [System Actors]
GitRepo[Git Repository]
CI[CI/CD Runner] -->|AssumeRole via OIDC| Cloud[Cloud Provider]
State[Remote Backend]
end
Dev-- Proposes change to -->GitRepo
Rev-- Triggers merge on -->GitRepo
GitRepo-- Triggers pipeline on merge -->CI
CI-- 1. Authenticates -->Cloud
CI-- 2. Plans & Applies -->Cloud
Cloud-- 3. Reads/Writes state -->State
```
The goal is simple: **Humans propose; machines enforce.** The CI/CD runner is the only actor with write access to the cloud. You, the engineer, don't even need cloud credentials stored locally.
---
### Part 1: The Foundation - Production-Grade Remote State
Running Terraform locally means your state file sits on your machine. That's a single point of failure for collaboration, security, and file corruption. A remote backend is non-negotiable. The big three options are AWS S3, Azure Storage, and Terraform Cloud.
Let's look at a real-world Azure setup, which is a perfect example of the "belt and suspenders" approach.
#### Azure Remote Backend: Locking & Encryption by Default
This configuration solves three critical problems:
1. **Shared State:** Everyone accesses the same source of truth.
2. **State Locking:** Uses Azure Blob Storage leases to prevent two engineers from running `apply` simultaneously.
3. **Encryption:** Data is encrypted at rest and in transit.
```hcl
# backend.tf
terraform {
backend "azurerm" {
resource_group_name = "rg-terraform-state-prod"
storage_account_name = "stterraformstateprod"
container_name = "tfstate"
key = "production/network.terraform.tfstate"
# Locking is enabled by default
# use_azuread_auth = true # Use this instead of access keys for auth!
}
}
```
**The Real Fix:** I've seen a team locked out of their cloud because the person who ran the last `apply` was on vacation, and the state file lived on their locked laptop. The hard-earned lesson? If a single person leaving stops you from deploying, it's not production-ready. Remote backends fix this.
---
### Part 2: The Workflow - GitOps, Not ClickOps
GitOps means your Git repository is the single source of truth. Not the Terraform Cloud GUI, not the AWS Console, and definitely not a manual CLI command. The desired state of your infrastructure is declared in `.tf` files on the `main` branch. The system continuously reconciles the actual state to match.
Our Git branching strategy is dead simple but strictly enforced:
```mermaid
gitGraph
commit id: "initial commit"
branch feature/add-postgres-server
checkout feature/add-postgres-server
commit id: "feat: add postgres config"
commit id: "fix: correct sku name"
checkout main
merge feature/add-postgres-server tag: "v1.2.0"
branch hotfix/wrong-nsg-rule
checkout hotfix/wrong-nsg-rule
commit id: "hotfix: restrict SSH to VPN"
checkout main
merge hotfix/wrong-nsg-rule tag: "v1.2.1"
```
- **`main` branch:** The sacred, protected branch. Merges here trigger an automatic `apply` to production. No direct commits are allowed.
- **Feature branches:** All changes, from new resources to variable tweaks, happen here. They trigger a `terraform plan` to be posted as a comment on the Pull Request.
---
### Part 3: The Pipeline - CI/CD with Secure Authentication
The CI/CD pipeline is the engine room. Its job is to authenticate, validate, plan, and apply infrastructure changes securely. The most critical part is how the pipeline authenticates to the cloud provider. **Under no circumstances should you store long-lived access keys as CI/CD secrets.** Use OpenID Connect (OIDC) federation.
#### The Critical Fix: OIDC Authentication (AWS & Azure)
This was a pain point I debugged for hours. We had an incident where a leaked CI variable containing AWS keys was used to spin up crypto miners. The exact fix? Rip out all static keys and use OIDC, which generates short-lived tokens for each pipeline run.
Here's how the trust is set up so your CI runner can assume a role in your cloud account.
**AWS OIDC Provider Setup**
```hcl
resource "aws_iam_openid_connect_provider" "github" {
url = "https://token.actions.githubusercontent.com"
client_id_list = ["sts.amazonaws.com"]
thumbprint_list = ["6938fd4d98bab03faadb97b34396831e3780aea1"]
}
resource "aws_iam_role" "terraform_ci" {
name = "terraform-ci-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Principal = { Federated = aws_iam_openid_connect_provider.github.arn }
Action = "sts:AssumeRoleWithWebIdentity"
Condition = {
StringEquals = {
"token.actions.githubusercontent.com:aud" : "sts.amazonaws.com"
},
StringLike = {
"token.actions.githubusercontent.com:sub" : "repo:your-org/your-repo:*"
}
}
}]
})
}
```
With the trust established, the pipeline can be secure and stateless. Here's a generic, production-ready pipeline definition. This isn't just a `plan` and `apply`; it's a gated process.
**Pipeline Stages Diagram**
```mermaid
sequenceDiagram
participant PR as Pull Request
participant CI as CI/CD System
participant Cloud as Cloud Provider
participant Repo as VCS
Note over PR, Repo: Developer opens a PR from feature-branch
PR->>CI: Triggers "Terraform Plan" job
CI->>Cloud: Authenticate via OIDC
CI->>Cloud: terraform init & plan
Cloud-->>CI: Return plan output
CI->>Repo: Post plan as PR comment
Repo->>PR: Senior reviewer reviews plan & merges
Note over PR, Repo: Merge to main branch
Repo->>CI: Triggers "Terraform Apply" job
CI->>Cloud: Authenticate via OIDC
CI->>Cloud: terraform init & plan (again)
CI-->>CI: Manual approval gate (optional but critical)
CI->>Cloud: terraform apply -auto-approve
Cloud-->>CI: Apply success log
CI->>Repo: Report deployment status
```
#### Production CI/CD Configuration (GitHub Actions)
This workflow translates the diagram into code. Notice the "Re-plan" step before apply. This is a crucial safety net to ensure nothing changed between the PR merge and the apply.
```yaml
# .github/workflows/terraform.yml
name: 'Terraform GitOps'
on:
pull_request:
branches: [ main ]
push:
branches: [ main ]
permissions:
id-token: write # Required for OIDC
contents: read
jobs:
terraform:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Terraform
uses: hashicorp/setup-terraform@v3
- name: Terraform Init
run: terraform init
# No credentials needed! Auth is handled by the 'configure' step.
# --- Plan Stage (For both PRs and Merges) ---
- name: Terraform Plan
id: plan
run: terraform plan -no-color -out=tfplan
continue-on-error: true # Don't fail on plan changes
- name: Update Pull Request
uses: actions/github-script@v7
if: github.event_name == 'pull_request'
with:
script: |
const output = `#### Terraform Plan \`${{ steps.plan.outcome }}\`
\`\`\`hcl\n${{ steps.plan.outputs.stdout }}\n\`\`\`
*Pusher: @${{ github.actor }}*`;
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: output
});
# --- Apply Stage (Only on merge to main) ---
- name: Terraform Apply
if: github.event_name == 'push' && steps.plan.outcome == 'success'
run: terraform apply -auto-approve tfplan
```
---
### Hard-Won Production Lessons (Real Debugging Notes)
A guide isn't complete without the scars to prove it. Here are the exact fixes to problems that wasted hours of my time.
1. **The "State Locking is Not Optional" Incident:**
The issue: Two engineers ran `apply` on different PRs at the same time. They were working on the same state file but didn't know it. Result: a corrupted state file that `terraform force-unlock` couldn't fix. The fix was a manual state recovery from a backup. The permanent fix: setting `prevent_destroy = true` on life-critical state resources like databases and using DynamoDB/Blob lease locking religiously.
2. **The "Too Many Permissions" Drift:**
We gave our CI role `AdministratorAccess` because it was easy. A wrong variable in a PR deleted a production security group. The exact fix was writing a scoped IAM policy that only allows what Terraform needs to manage—no more `"*"` on delete actions.
3. **The "Mystery Drift" Nightmare:**
We had engineers manually tweaking auto-scaling group sizes during incidents from the AWS console. The next Terraform run reset them, causing a mini-outage. The fix? A CI/CD scheduled task that runs `terraform plan -detailed-exitcode` every hour. If it detects drift, it sends an alert and creates an automatic ticket. Your infrastructure shouldn't have secrets.
### Final Word
This is the setup we use every day. It's not glamorous, but it's solid. It removes the heroics from infrastructure management and replaces them with a boring, predictable pipeline. A laptop is for writing code, not for applying it.
The goal is to make `terraform apply` an event so mundane that nobody bats an eye when it happens. That's when you've truly bridged the gap to production.]]>The Rise of AMD AI Halo Chips: APUs, ROCm, and Unified Memory Workloadshttps://yahyaoncloud.com/blog/amd-ai-halo-chips-apus-rocm-unified-memory2026-05-12T00:00:00.000Z2026-05-12T00:00:00.000ZCloud GPU Infrastructure: Navigating AWS H100s, RunPod, and Serverless GPUshttps://yahyaoncloud.com/blog/cloud-gpu-infrastructure-aws-runpod-serverless2026-04-22T00:00:00.000Z2026-04-22T00:00:00.000ZRunning Open-Weights LLMs on Budget GPUs: Benchmarking RX 6500 XT, GTX 1660, Arc A750, and RX 6600https://yahyaoncloud.com/blog/testing-cheap-llms-on-amd-rx-6500-xt2026-04-05T00:00:00.000Z2026-04-05T00:00:00.000ZMLOps in Production: Building Automated Pipeline Infrastructure with Kubeflow & MLflowhttps://yahyaoncloud.com/blog/mlops-in-production-automated-pipeline-infrastructure2026-03-18T00:00:00.000Z2026-03-18T00:00:00.000ZOptimizing LLM Inference: Quantization, Speculative Decoding, and vLLMhttps://yahyaoncloud.com/blog/optimizing-llm-inference-vllm-quantization2026-02-28T00:00:00.000Z2026-02-28T00:00:00.000ZBuilding Production-Grade RAG Systems: From Vector Indexing to Hybrid Searchhttps://yahyaoncloud.com/blog/building-production-grade-rag-systems-hybrid-search2026-02-14T00:00:00.000Z2026-02-14T00:00:00.000ZWhat the heck is Ansible anyways?https://yahyaoncloud.com/blog/what-the-heck-is-ansible-anyways2026-01-19T00:00:00.000Z2026-01-19T00:00:00.000ZWelcome to YahyaOnCloudhttps://yahyaoncloud.com/blog/welcome-to-yahyaoncloud2026-01-19T00:00:00.000Z2026-01-19T00:00:00.000ZStarting with Go Programminghttps://yahyaoncloud.com/blog/starting-with-go-programming2024-04-11T00:00:00.000Z2024-04-11T00:00:00.000Z