Six Misconfigurations, One Night: Bringing Up a Dev Environment Before a Client Demo

Summary

At 11pm the night before Moncton CRM's first client demo, the dev environment on AWS ECS Fargate was unreachable. The team lead who normally owns cloud and infrastructure configuration wasn't available at that hour. Cloud deployment wasn't my usual scope on the project, but the demo was the next morning and someone had to get the environment healthy, so I took it on. It took until roughly 3am, because the first thing I fixed wasn't the problem — it was the first of six.

Problem

The visible symptom was small: after Microsoft SSO login, the app redirected to 0.0.0.0:4000/dashboard instead of the real domain. That looked like a single misconfigured PORT environment variable, and correcting it was the obvious first move. It also wasn't the fix — it just changed which layer failed next. Between that first symptom and a healthy environment, five more misconfigurations surfaced, each one hidden behind the last: an ALB target group still pointed at the old port, a health check path that returned a redirect the load balancer didn't follow, a container health check command hardcoded to the old port, a security group that had never been opened for the new one, and — unrelated to any of it — a frontend API_URL pointing at a subdomain that didn't resolve.

Discussion

The port change that wasn't one fix

The frontend's task definition had PORT=4000, the backend's port, so Next.js constructed every redirect against 0.0.0.0:4000 instead of the real domain. Setting PORT=3000 and deploying a new task revision was correct — and immediately broke the ECS health check, because nothing was listening on 4000 anymore for a target group that was still registered on it.

Every layer that had memorized the old port

From there it was one dependency at a time:

• The ALB target group (development-tg-web) had the task registered on port 4000. It had to be deregistered and re-registered on 3000, and the ECS service's load-balancer container-port setting updated to match.
• The ELB health check path was /health, which doesn't exist in this Next.js app and returned a 307 redirect — which ALB health checks don't follow by default. Fixed by pointing the check at / and adding 307 to the accepted success codes alongside 200.
• The container health check command, defined separately inside the task definition, was still curl -sf http://localhost:4000/. ELB health checks and container health checks are independent systems that both had to be told about the new port; fixing one didn't fix the other.
• The security group on the ECS task only allowed inbound traffic on 4000 from the ALB. Port 3000 was silently dropped until an explicit inbound rule was added for it.

None of these four were caused by each other. They were four separate places that had been told, at some point, that the app lived on port 4000, and updating the environment variable didn't propagate to any of them.

A second, unrelated root cause

With routing and health checks finally green, login still failed — ERR_TOO_MANY_REDIRECTS, with the frontend logging getMeAction failed — fetch failed. This one had nothing to do with ports: the frontend's API_URL was set to dev-api.yqgpros.com, a subdomain that didn't resolve. Locally this had always worked because API_URL pointed at localhost:4000 — but in ECS, frontend and backend are separate containers, and localhost in one doesn't reach the other. Updating the SSM parameter to the correct routed domain and forcing a redeploy (SSM values only apply at task startup) was the last fix.

Key Takeaways

• A load balancer's listed protocol/port (HTTP:4000) is a label — the port that actually receives traffic is the registered target port, and it doesn't update itself when application config changes.
• ELB health checks and container health checks are two independent systems pointed at the same app; both have to be corrected, and passing one says nothing about the other.
• In ECS Fargate, containers don't share a network namespace with each other the way localhost might suggest locally — cross-service calls need the real routed domain, not localhost.
• SSM Parameter Store values are read once at task startup; a changed value does nothing until the service is force-redeployed.
• When a fix "resolves" an incident but a new, differently-shaped symptom appears immediately after, that's a sign the root cause is still upstream — not a reason to start a fresh investigation.
• The fix only helps once. Writing the trace down — which is what got shared with the team the next day — is what makes the same chain recognizable in minutes instead of hours the next time.