# My Idle Background Worker Was the Most Expensive Thing I Owned


I opened the Cloud Run billing breakdown expecting the database to be the villain. Failing that, the AI generation, which calls out to a model and streams tokens back and *feels* expensive, in the way that things which visibly do work feel expensive.

Neither. Sitting at the top of the list, by a comfortable margin, was a service that had spent the month doing almost nothing at all.

That service was the background worker, a Celery consumer. Its job is to pick up background tasks: moderate a story when it's published, send a verification email at signup, deliver a webhook, escalate a support ticket. On a personal project with a handful of users, that queue is empty essentially all of the time.

Every other component scaled to zero and cost nothing while idle. This one had been quietly billing me around the clock to sit and wait for work that mostly never arrived. It's a side project, so the bill is mine. I do a regular pass over billing, traffic and error rates for exactly that reason, and this was the week the billing half of it earned its keep.

## TL;DR

A Celery worker is a polling consumer. Cloud Run only allocates CPU in response to requests. Those two facts are structurally incompatible, and the incompatibility is invisible because everything works perfectly the entire time.

The fix was one environment variable. The interesting part is the four things I had to check about my own codebase before that one variable was safe, and the fact that the obvious cheaper option is broken rather than cheap.

Nothing was broken before I started, and nothing is broken now. What changed is what I'm willing to pay for, and what I gave up to stop paying it.

Most of this project was vibe coded on purpose, as a test of how far that approach goes without close architectural steering. What it produced was a conventional architecture that worked correctly and carried an operational cost nothing in the repository could have told you about.

## What is a worker actually doing when there's nothing to do?

This is the question I hadn't thought carefully about, and the answer is the entire post.

A web service is easy to reason about on a serverless platform. A request arrives, the platform starts an instance if one isn't running, the instance does the work, and when the traffic stops the platform scales back to zero. You pay for the work. No traffic, no bill. The mental model is clean and it's why scale-to-zero is such a pleasant default.

A Celery worker does not fit that model at all, and it doesn't fit it in a way that is easy to miss.

A worker has no HTTP surface. Nothing can send it a request, because it isn't listening for one. What it does instead is sit in a loop, blocking on `BRPOP` against Redis, asking the broker over and over whether anything has shown up. Kombu, the transport layer underneath Celery, does this roughly once a second. Forever. That is not a bug or a misconfiguration, it is what a pull-based consumer *is*.

Now put those two things side by side. Cloud Run allocates CPU in response to requests, and scales up in response to requests. A worker never receives a request. So to keep it alive you have to tell the platform to stop doing the thing it's good at:

```
--min-instances=1 --max-instances=1 --no-cpu-throttling
```

Read those flags as a sentence. Always keep exactly one instance running. Never scale it down. Never throttle its CPU when it looks idle, because it always looks idle, right up until the moment it has something to do.

That's one CPU, allocated and billed continuously, whose entire job is to ask an empty queue if anything has happened yet.

>![Three tracks across one day: a few brief task executions, continuous once-a-second broker polling, and a solid unbroken bar of billed CPU](https://cdn.hashnode.com/uploads/covers/62db053c38759e6b49828665/46045d71-591e-497f-b2e3-c1a63cdecf31.svg)

And on the other side of the connection, the same loop is generating traffic against Redis all day long. An idle system, producing a steady stream of commands, burning through a free-tier quota to confirm repeatedly that there is nothing to confirm.

The reason this survives so long in a project is that there is no symptom. Nothing is slow. Nothing errors. Tasks run correctly the instant they're enqueued. Every test passes, every dashboard is green, and the only place the problem is visible at all is a billing page you have no particular reason to open.

## The experiment underneath this

I should say plainly where this architecture came from, because it's the part of the story I find most interesting now.

I deliberately vibe coded most of this project. I wanted to test the claim that coding agents can build a full-stack app almost end to end, and more importantly to find where that approach failed when nobody was carefully steering the architecture. It's a personal project, which made it a safe place to let the experiment break things.

So: an agent wrote the Celery configuration, wired the task modules, set up the broker connection, and deployed the worker as a second service. I reviewed it, it looked right, and it was right.

Ask any competent engineer for background jobs in a Python web app and you'll get Celery, a Redis broker, and a separate worker process. Ask an agent and you'll get the same thing faster, because that is overwhelmingly what the documentation says, what the tutorials say, and what thousands of correct production systems do. It's the right answer on a VM. It's the right answer on ECS, and on a Kubernetes deployment, and on anything with a machine that stays switched on.

It's the wrong answer on Cloud Run, and it's wrong for a reason that does not appear anywhere in the code.

Which is the result the experiment actually returned, and it wasn't the one I expected. The interesting failure wasn't bad code. The generated architecture was conventional, it worked correctly, and left alone it would have kept working correctly for years. The constraint it missed was operational cost, because that information lived outside the repository.

Nothing in that repository is a clue. There's no smell to catch in review, no test that could fail, no lint rule, no type error. Every file is idiomatic. The information you need in order to notice the problem lives in a billing console, in a different browser tab, behind a login the agent doesn't have, describing a pricing model that isn't in the source tree.

An agent optimises for *is this the conventional architecture for this stack*. That's a good objective and it's usually the one you want. It's just not the same question as *what does this cost me every month on this specific platform*, and the second question is the one that mattered.

The part I find genuinely instructive is what happened next. When I brought the agent the constraint, once it knew about request-scoped CPU allocation and a personal budget, it reasoned about it perfectly well. The min-instances trap below, the Cloud Tasks alternative, the four safety conditions: that analysis came out of the collaboration, and it's good analysis. The model was never incapable of the reasoning.

It just never raised the question. Nothing prompted it to weigh a running cost, because nothing in the request mentioned one, and an agent will not go looking for a constraint you haven't told it about. That's the same finding as an earlier post of mine about tool loops, arriving from a completely different direction: the model does what the surrounding structure makes it do, and everything outside that structure is invisible.

>![Six steps from a request for background jobs to a continuously billed CPU. The first three sit inside the repository and are all correct; the last three exist only on the billing page](https://cdn.hashnode.com/uploads/covers/62db053c38759e6b49828665/a751bb44-65cd-4bd6-a36d-55eec0aab83f.svg)

So this is roughly where I've landed on what the human contributes in a vibe-coded project. Not the code. The code was fine. The contribution is knowing which questions the code cannot answer, and having a habit that puts you in front of the answers anyway.

## Why the obvious fix is broken, not cheap

The first idea everyone has, including me, is: fine, set `--min-instances=0` on the worker and let it scale to zero like everything else.

It works. Your bill goes to zero. Your queue silently never drains again.

Think about what would have to happen for that to work. The instance count is zero, so there is no worker running. For a worker to start, something would have to trigger a scale-up. The only thing that triggers a scale-up on Cloud Run is an incoming request. Nothing sends the worker a request, because it has no HTTP surface, because it's a polling consumer. There is no path from "a task was enqueued" to "an instance exists to consume it."

So tasks pile up in Redis. The API keeps accepting work and handing it to the broker perfectly happily. Stories sit in `pending` and never publish. Verification emails are never sent. Nothing throws, nothing alerts, and every one of those failures looks like a UI problem to whoever reports it.

Guard against this one before it happens, or spend an afternoon staring at a queue depth graph wondering which part of your code is dropping tasks. The answer is none of them.

This is the beat worth taking away from the whole post, and it generalises well past this stack: when you make a pull-based component cheaper, check that anything is still pulling.

## What else was on the table

Four other options got considered properly before the one I picked.

**Cloud Scheduler triggering a Cloud Run Job that drains the queue.** Works, costs about nothing. Keeps Celery, keeps the broker polling, and adds up to N minutes of latency depending on your schedule. More moving parts than the problem deserved.

**An e2-micro Always Free VM running the worker.** Viable, and the closest thing to a real alternative. Rejected because it puts a server back in my life, one I'd have to patch and monitor, and because external egress from a VM out to a managed Redis and a managed Postgres is not the kind of free that stays free.

**Cloud Tasks pushing to an HTTP endpoint.** This is the right answer. I'll come back to it, because it deserves its own section rather than a table row.

**Keeping the worker.** Which is the problem, stated as an option, for completeness. It's worth writing this row down every time you do this exercise. Sometimes the honest conclusion is that the status quo is fine and you should go do something else.

## The one-line fix

```
CELERY_TASK_ALWAYS_EAGER=true
```

In eager mode, `.delay()` stops being an enqueue. It executes the task immediately, synchronously, inside the calling web process. No broker round-trip, no consumer, no second service. The worker deployment gets deleted outright.

One companion setting matters as much as the flag itself:

```
task_eager_propagates=False
```

Without it, an exception inside an eagerly-executed task raises straight into the HTTP request that triggered it. A user signs up, the row is committed, the account exists, the SMTP server has a bad afternoon, and the API returns a 500 for a request that succeeded. The task failing must not be able to convert a completed write into an error response.

## Why inline is safe here, and where it wouldn't be

This is the part I'd want someone to check before copying the flag into their own project, because "run your background tasks in the foreground" is not universally sound advice. It was sound in this codebase for four specific reasons, and I went and verified each of them rather than assuming.

**Every enqueue already happens after the commit.** This is the one that would have bitten hardest. Every call site in the codebase follows the same order:

```python
db.commit()
db.refresh(story)
moderate_content.delay(story.id)
```

Commit, refresh, then fire. Because the transaction has already closed, the inline task opens its own session and reads a row that exists.

Reverse those two lines and the whole thing falls apart. With `.delay()` before `db.commit()`, an inline task runs inside the caller's still-open transaction, opens a separate session, and queries for a row that has not been committed yet. It finds nothing. It has no work to do, so it does nothing, and it does it without complaint. Under the old setup that same ordering mistake would have been survivable, because the network hop to the broker and back bought enough time for the commit to land. Async execution was papering over a latent bug, and eager mode would have torn the paper off.

**The tasks are idempotent already.** They were built for at-least-once delivery with `task_acks_late`, so running one twice is harmless. Inline execution never re-runs anything, but it costs nothing to have confirmed it.

**The latency is small where it matters.** `moderate_content` is a local `better_profanity` scan. No model call, no network, no IO worth naming. Inline moderation costs milliseconds and the user never perceives it. The only task with real latency is the SMTP round-trip on signup, at roughly one to two seconds, and that one is now sitting inside the user's request.

**The work happens during a request, which is the whole trick.** Because the task executes inline, it runs while the platform has CPU allocated to serve the request that triggered it. That's the condition the entire approach depends on.

## Why not just use BackgroundTasks?

FastAPI ships `BackgroundTasks`, and it looks like the obvious lightweight answer: schedule the work, return the response, let the task run afterwards. On a normal always-on server, that's fine.

On a scale-to-zero platform, it's a trap, and it's a quieter one than the min-instances trap.

`BackgroundTasks` runs your work *after* the response has been sent. At that moment, from the platform's point of view, the request is finished. CPU gets throttled down to near nothing. Your task doesn't crash and doesn't error, it just gets starved of the CPU it needs to make progress, and if the instance is scaled down before it finishes, it disappears mid-execution.

Inline execution works precisely because it happens *before* the response, while the request is still open and the CPU is still yours. The distinction between "during the request" and "after the request" carries no weight on a VM and carries all of the weight here.

## The flag is load-bearing, and it fails silently

Worth stating plainly, because it's the thing most likely to go wrong for someone repeating this: once the worker service is gone, `CELERY_TASK_ALWAYS_EAGER=true` is not an optimisation. It is required.

Deploy without it and four call sites go back to enqueuing tasks to a broker that nobody is consuming. Story publish stays `pending` forever. Signup verification emails are never sent. Webhooks are never delivered. Support escalations never escalate. Every one of them fails silently, and every one of them presents as a completely different bug to whoever notices first.

There's a smaller landmine right next to it. The way you set that variable matters:

```bash
gcloud run services update ... --update-env-vars CELERY_TASK_ALWAYS_EAGER=true   # correct
gcloud run services update ... --set-env-vars    CELERY_TASK_ALWAYS_EAGER=true   # deletes everything else
```

`--set-env-vars` replaces the entire environment list rather than adding to it. Run the second one and you have just silently removed the dozen-odd other variables the service needs, including the database URL. The service then either boots broken or doesn't boot at all, and you get to find out which one on a service you were in the middle of making cheaper.

## So what did it actually cost?

Nothing broke. That's the honest summary, and it's also less interesting than the trade-off, so here is the trade-off.

**Retries went away.** This was the real price and it wasn't a small one. Eager mode does not retry, at all. A transient SMTP failure on signup meant that verification email was lost. At the time I made the call there was no resend path either. Because this is a side project and my availability is strictly capped, I had to make a calculated trade-off between my infrastructure budget and my time budget. I chose to stop the financial bleed immediately and accept a temporarily broken edge case, rather than blocking the cost-fix while I spent weeks building the proper Cloud Tasks push-queue solution. It went straight onto the list of things to fix when time permits.

**Time limits went with them.** `task_time_limit` is ignored in eager mode, so a task is bounded only by whatever timeout the caller imposes on itself.

**Task latency became user-visible.** Signup now carries the SMTP round-trip inside the request. One to two seconds, sitting in a flow where the user is already waiting and already expects a pause. That's the reason this trade is tolerable, and it's why I'd answer differently for a task that took twenty seconds.

**Task-level parallelism disappeared.** Background work competes with request handling for the same process.

What I bought with that: idle infrastructure cost goes to zero, broker polling goes to zero, and there is one fewer service to deploy, secure, and monitor. Redis stays in the stack for rate limiting, caching, and the WebSocket pub/sub backplane, so nothing else in the architecture changed.

For a project with this traffic and this budget, that's a good trade. For a project where a lost email is a lost customer, it's a terrible one, and I'd rather be clear about which of those I'm running than pretend the flag is free.

## What I'd build with more time

The correct answer isn't inline execution. It's push delivery instead of pull.

Replace the polling broker with Cloud Tasks, or Pub/Sub push, targeting an authenticated HTTP endpoint on the backend service that already exists:

```
create story ──► Cloud Tasks enqueue ──► POST /internal/tasks/moderate
                                          └─► Cloud Run cold-starts 0→1,
                                              works, scales back to 0
```

>![Broker pull versus Cloud Tasks push: a polling consumer that must stay alive to receive work, against an HTTP task that wakes a scaled-to-zero service on delivery](https://cdn.hashnode.com/uploads/covers/62db053c38759e6b49828665/eda47f0c-67da-4547-973c-60094fa9604b.svg)

Every problem in this post dissolves. The task arrives as an HTTP request, which is exactly the thing Cloud Run knows how to wake up for. You keep scale-to-zero *and* genuine asynchronous execution *and* managed retries with backoff, which is the thing I gave up. Cloud Tasks' free tier covers a million operations a month, so at this volume it would also be free.

It is strictly better than both the always-on worker and the inline switch. I didn't build it because it's a re-architecture of every enqueue path in the codebase, and the inline flag was a one-line change that solved the bill that week. That's a scheduling decision rather than a technical judgement, and writing it down is how I stop myself pretending otherwise a year from now.

I wrote the revisit conditions down at the time: when task volume grows, when retries start mattering, when a task exceeds about a second of user-visible latency, or when the project is funded by something other than my own wallet.

## Start smaller than this

If you're reading this with a queue somewhere and a vague sense of unease, you don't need to do any of the above today. Do the cheap part first.

Open your billing breakdown and sort by cost. That's the entire discovery step, and no amount of tooling will do it on your behalf. It's worth making it a habit rather than a reaction, because the failure mode here is a cost that never announces itself: nothing pages you, nothing turns red, and the only signal is a line item that's higher than you'd have guessed.

Then, for whatever's at the top, ask one question: does this thing get woken up, or does it wake itself up? Anything that wakes itself up on a platform that bills for allocated CPU is worth ten minutes of attention. That covers pollers, cron loops inside long-lived processes, websocket keepalives, and any consumer that blocks on a broker.

You don't have to fix it. Knowing which category each of your services falls into is most of the value, and it's the part that transfers to the next platform you use.

## What carries over

1. **A pull-based consumer and a request-billed platform are structurally incompatible.** Not a tuning problem. The consumer never receives the thing the platform bills for.
2. **When you make a polling component cheaper, check that anything is still polling.** `min-instances=0` on a worker is broken, not cheap, and it breaks silently.
3. **"During the request" and "after the request" are different execution environments** on a scale-to-zero platform, even though they look identical in your code.
4. **Enqueue after commit, always.** Async transport hides this bug. Inline execution exposes it immediately.
5. **Before you make async work synchronous, check idempotency, latency, and ordering.** Three questions, and if any answer is wrong the flag is unsafe.
6. **Write down the answer you didn't build.** Deferred is not the same as rejected, and future you will not remember which one it was.
7. **Read the flags you deploy with as a sentence.** `--min-instances=1 --no-cpu-throttling` says "bill me continuously" out loud, if you let it.
8. **An agent gives you the conventional architecture, which is a different thing from the correct one here.** It will reason about your constraints well and it will not go looking for them. The failure mode to watch for isn't bad code, it's good code carrying a constraint that lives outside the repository.

The cheapest service in your stack is the one that doesn't exist. The second cheapest is the one that only exists when someone asks it to.

---

*I'm Ashish Kr Jha, a software engineer. I build things in production and then have to live with them, which is the part that actually teaches you something. More at [ashishkrjha.dev](https://ashishkrjha.dev).*