# Persistent AI Agents Without Redis: A MongoDB-Only Architecture
The default stack for background pipelines in a Python web app is Redis + Celery + a queue broker. It's what every guide recommends, it works, and it has a lot of moving parts. For our agent framework — five persistent AI agents that observe, investigate, propose, and execute — we made an unusual call: no Redis, no Celery, no external broker. Just MongoDB. Everything.
Here's why, what broke, and what we learned.
Why not Redis
Every additional service is a failure mode. Redis pods die, brokers get network-partitioned, queues fill up, tasks get retried, and every layer of infrastructure that isn't your product is one more thing that pages you at 2am. For a customer-facing agent framework where correctness matters more than throughput (a stray duplicate agent tick is a real cost — it might double-post a PO), the calculus tilts hard toward simplicity.
We already had Mongo. It's replicated, backed up, and every operator on the team knows how to reason about it. The question wasn't "is Redis better at queuing?" (it is), it was "is Redis better *enough* to justify the operational surface?" For our load profile — a few thousand agent ticks a day across low-hundreds of tenants — the answer was no.
The scheduler
Every agent has a `poll_interval_secs`. The scheduler loop runs every 30 seconds, picks agents whose next-run timestamp is due, and fires their `tick()` method. The trick — and the whole reason we can do this on Mongo alone — is a single atomic operation:
``` find_one_and_update( { "agent_key": key, "next_run_at": { "$lte": now } }, { "$set": { "next_run_at": now + poll_interval } }, returnDocument=AFTER, ) ```
If two workers race for the same tick, exactly one wins. The other's update finds no matching document and moves on. No lock, no coordination, no distributed anything. The scheduler is stateless. The lock is the atomic update.
The compound index `(agent_key, next_run_at)` makes this fast even with thousands of scheduled entries. In our sizing, it consistently runs in single-digit milliseconds.
The unique-index concurrency guarantee
Even with an atomic scheduler, a bug in the tick logic can spawn a duplicate investigation. Two agent instances start ticking at nearly the same moment, both read the same trigger, both open an `agent_investigations` row. The `find_one_and_update` protects the scheduler; it doesn't protect the downstream write.
Fix: a partial unique index on the investigation collection:
``` create_index( [("tenant_id", 1), ("agent_key", 1), ("trigger_signature", 1)], unique=True, partialFilterExpression={"status": {"$in": ["proposed", "investigating"]}}, ) ```
The `partialFilterExpression` is critical. We do want two investigations with the same signature to exist *eventually* — one closed, one active on a new occurrence of the same pattern. What we don't want is two *active* investigations for the same tenant+agent+signature at the same time. The partial index enforces exactly that: two active rows with the same trip attributes trigger a duplicate-key error, which the agent catches and treats as "someone else got there first."
What we gave up
Cross-machine push semantics. Redis lets a worker publish an event that a subscriber consumes immediately. Mongo's polling model has a floor latency equal to the poll interval. For our use case — agents that fire every 4-5 minutes anyway — this doesn't matter. For a fraud-detection agent that needs sub-second reaction, it would.
Broker-managed backpressure. Celery has a whole vocabulary — task expiry, rate limits, worker autoscaling — that Mongo doesn't. We built a much simpler equivalent: `next_run_at` is the pacing mechanism. If a tick takes longer than expected, the next `next_run_at` naturally moves further out. If the worker is overloaded, the queue is the collection itself; new work sits in `next_run_at <= now` until a worker picks it up.
Fan-out ergonomics. Publishing one event to N subscribers is one Redis call and N-1 mostly-free reads. On Mongo it's N inserts. This would matter at scale. At our scale it doesn't — we don't do fan-out in the agent framework; every action has exactly one target agent.
What we gained
One system to reason about. When something breaks, the mental model is "look in the agents collection." No cross-referencing broker state against DB state. The audit trail matches the operational reality by construction.
Free replayability. Every scheduler decision is a document with a timestamp. Ticks, retries, backoffs — all queryable. We can replay any tenant's agent history in a debugger by loading the raw docs.
Deployment simplicity. The standalone worker is one Python process reading from Mongo. We deploy it separately from the API (different scaling profiles) but there's no infrastructure between them. Container up, container down; no external state.
Cost. No Redis pod, no broker cluster, no memory ceiling. On a $50/month MongoDB Atlas tier we run the agent framework for hundreds of tenants.
The failure modes we still have
Long-running ticks block the loop. If one agent's tick hangs, the scheduler can't advance until it times out. Fix: wrap every tick in `asyncio.wait_for` with a hard deadline. Missed tick = logged warning + next iteration.
Poll-interval drift under load. If ticks take longer than the poll interval, `next_run_at` moves faster than wall time. The agents effectively rate-limit themselves. We monitor the drift metric and alert if it exceeds a threshold — usually it's a signal that a downstream API (ERP writeback) is slow.
Cold-start burst. When the worker restarts, every agent with an overdue `next_run_at` is immediately eligible. On a system with 500 tenants and 5 agents each, that's a 2,500-tick backlog. We stagger by inserting a randomized `next_run_at + jitter(0-60s)` on worker startup. Solves it.
When you shouldn't do this
If your agents need to react in sub-second time, use Redis. If you're doing heavy fan-out (100+ subscribers per event), use Kafka. If your job durations are wildly heterogeneous (some 10ms, some 10 minutes), use Celery.
For everything else — polling agents, deterministic schedules, correctness-over-latency workloads — Mongo alone is a perfectly good backbone. It's the boring choice, and boring is exactly what a persistent AI workforce should be built on.