Memory growth in long-running Python workers
Ops pages show a gentle ramp in RSS over days. Restarting workers “fixes” it until the next week. Before you blame CPython, separate leak from cache.
Take two snapshots, hours apart
Enable tracemalloc at worker start. Snapshot after warm-up, snapshot after sustained load. Compare with snapshot.compare_to(old, 'lineno'). Growth tied to a single line number is actionable; diffuse growth across many lines often means fragmentation or allocator behaviour—still worth noting.
Check per-request globals
Module-level dicts that accumulate message IDs are a frequent culprit in webhook workers. The fix is bounded caches or moving state to Redis with TTL—not switching languages.
Celery prefork vs solo pool
Prefork masks some leaks by recycling children; solo pools expose them faster. Match your profiling pool to production if you want comparable numbers.
When memray is worth the install
For extension-heavy code (numpy, lxml), tracemalloc under-reports. A short memray capture on a staging pod can show allocations inside C extensions that Python-level tools miss.
What we deliver after diagnosis
A chart of snapshot diffs, the retaining reference path, and a patch with a regression test that fails if the cache exceeds a agreed bound. No hand-waving about “monitoring it.”