Sidekiq workers consume a lot of memory: how can I fix that?
I have a Rails application that can usually run with about 1 GB of RAM. However there are some Sidekiq workers that constantly grow over time – for days – and consume a lot of memory (e.g. 2.5GB).
Are there any solutions?
The problem is related to Ruby threads and how memory allocation works at low-level. In many cases it’s not a memory leak, but just a memory bloat.
Here’s two effective solution that we have used in production to reduce the memory usage:
- Reduce the number of Sidekiq threads (e.g. use 10 threads instead of 25)
- Set the environment variable MALLOC_ARENA_MAX=2 for tuning glibc memory behavior.
The environment variable is a no-brainer. Here’s the results that we got by adding it (max RAM in 16h):
- web: 1191Mi -> 966Mi (~20% less memory)
- worker type 1: 911Mi -> 348Mi (~60% less memory)
- worker type 2: 549Mi -> 304Mi (~45% less memory)