ruby on rails 3 - Memory leak in background process on Heroku -
i wondering if can advise me on how track down memory leak / issue on background process on heroku.
i have 1 dyno worker running delayed_job queue, processing sorts of different processes. time time, i'm getting sudden jump in memory consumed. subsequent jobs exceed memory limit , fail, , hell breaks loose.
the weird thing can't see jump in memory connected particular job. here's sort of log see:
aug 15 07:13:25 vemmleads heroku/worker.1: source=heroku.10054113.worker.1.4589e3f4-8208-483a-a927-67c4c1cbee46 measure=load_avg_1m val=0.00 aug 15 07:13:25 vemmleads heroku/worker.1: source=heroku.10054113.worker.1.4589e3f4-8208-483a-a927-67c4c1cbee46 measure=load_avg_5m val=0.01 aug 15 07:13:25 vemmleads heroku/worker.1: source=heroku.10054113.worker.1.4589e3f4-8208-483a-a927-67c4c1cbee46 measure=load_avg_15m val=0.01 aug 15 07:13:25 vemmleads heroku/worker.1: source=heroku.10054113.worker.1.4589e3f4-8208-483a-a927-67c4c1cbee46 measure=memory_total val=133.12 units=mb aug 15 07:13:25 vemmleads heroku/worker.1: source=heroku.10054113.worker.1.4589e3f4-8208-483a-a927-67c4c1cbee46 measure=memory_rss val=132.23 units=mb aug 15 07:13:25 vemmleads heroku/worker.1: source=heroku.10054113.worker.1.4589e3f4-8208-483a-a927-67c4c1cbee46 measure=memory_cache val=0.88 units=mb aug 15 07:13:25 vemmleads heroku/worker.1: source=heroku.10054113.worker.1.4589e3f4-8208-483a-a927-67c4c1cbee46 measure=memory_swap val=0.01 units=mb aug 15 07:13:25 vemmleads heroku/worker.1: source=heroku.10054113.worker.1.4589e3f4-8208-483a-a927-67c4c1cbee46 measure=memory_pgpgin val=0 units=pages aug 15 07:13:25 vemmleads heroku/worker.1: source=heroku.10054113.worker.1.4589e3f4-8208-483a-a927-67c4c1cbee46 measure=memory_pgpgout val=45325 units=pages aug 15 07:13:25 vemmleads heroku/worker.1: source=heroku.10054113.worker.1.4589e3f4-8208-483a-a927-67c4c1cbee46 measure=diskmbytes val=0 units=mb aug 15 07:13:31 vemmleads heroku/web.1: source=heroku.10054113.web.1.bf5d3fae-2b1b-4e1d-a974-01d9fa4644db measure=load_avg_1m val=0.15 aug 15 07:13:31 vemmleads heroku/web.1: source=heroku.10054113.web.1.bf5d3fae-2b1b-4e1d-a974-01d9fa4644db measure=load_avg_5m val=0.07 aug 15 07:13:31 vemmleads heroku/web.1: source=heroku.10054113.web.1.bf5d3fae-2b1b-4e1d-a974-01d9fa4644db measure=load_avg_15m val=0.17 aug 15 07:13:31 vemmleads heroku/web.1: source=heroku.10054113.web.1.bf5d3fae-2b1b-4e1d-a974-01d9fa4644db measure=memory_total val=110.88 units=mb aug 15 07:13:31 vemmleads heroku/web.1: source=heroku.10054113.web.1.bf5d3fae-2b1b-4e1d-a974-01d9fa4644db measure=memory_rss val=108.92 units=mb aug 15 07:13:31 vemmleads heroku/web.1: source=heroku.10054113.web.1.bf5d3fae-2b1b-4e1d-a974-01d9fa4644db measure=memory_cache val=1.94 units=mb aug 15 07:13:31 vemmleads heroku/web.1: source=heroku.10054113.web.1.bf5d3fae-2b1b-4e1d-a974-01d9fa4644db measure=memory_swap val=0.01 units=mb aug 15 07:13:31 vemmleads heroku/web.1: source=heroku.10054113.web.1.bf5d3fae-2b1b-4e1d-a974-01d9fa4644db measure=memory_pgpgin val=2908160 units=pages aug 15 07:13:31 vemmleads heroku/web.1: source=heroku.10054113.web.1.bf5d3fae-2b1b-4e1d-a974-01d9fa4644db measure=memory_pgpgout val=42227 units=pages aug 15 07:13:31 vemmleads heroku/web.1: source=heroku.10054113.web.1.bf5d3fae-2b1b-4e1d-a974-01d9fa4644db measure=diskmbytes val=0 units=mb aug 15 07:13:35 vemmleads app/heroku-postgres: source=heroku_postgresql_charcoal measure.current_transaction=1008211 measure.db_size=482260088bytes measure.tables=39 measure.active-connections=6 measure.waiting-connections=0 measure.index-cache-hit-rate=0.99996 measure.table-cache-hit-rate=1 aug 15 07:13:45 vemmleads heroku/run.2472: source=heroku.10054113.run.2472.e811164e-4413-4dcf-8560-1f998f2c2b4e measure=load_avg_1m val=0.00 aug 15 07:13:45 vemmleads heroku/run.2472: source=heroku.10054113.run.2472.e811164e-4413-4dcf-8560-1f998f2c2b4e measure=load_avg_5m val=0.00 aug 15 07:13:45 vemmleads heroku/run.2472: source=heroku.10054113.run.2472.e811164e-4413-4dcf-8560-1f998f2c2b4e measure=load_avg_15m val=0.14 aug 15 07:13:45 vemmleads heroku/run.2472: source=heroku.10054113.run.2472.e811164e-4413-4dcf-8560-1f998f2c2b4e measure=memory_total val=108.00 units=mb aug 15 07:13:45 vemmleads heroku/run.2472: source=heroku.10054113.run.2472.e811164e-4413-4dcf-8560-1f998f2c2b4e measure=memory_rss val=107.85 units=mb aug 15 07:13:45 vemmleads heroku/run.2472: source=heroku.10054113.run.2472.e811164e-4413-4dcf-8560-1f998f2c2b4e measure=memory_cache val=0.15 units=mb aug 15 07:13:45 vemmleads heroku/run.2472: source=heroku.10054113.run.2472.e811164e-4413-4dcf-8560-1f998f2c2b4e measure=memory_swap val=0.01 units=mb aug 15 07:13:45 vemmleads heroku/run.2472: source=heroku.10054113.run.2472.e811164e-4413-4dcf-8560-1f998f2c2b4e measure=memory_pgpgin val=0 units=pages aug 15 07:13:45 vemmleads heroku/run.2472: source=heroku.10054113.run.2472.e811164e-4413-4dcf-8560-1f998f2c2b4e measure=memory_pgpgout val=33609 units=pages aug 15 07:13:45 vemmleads heroku/run.2472: source=heroku.10054113.run.2472.e811164e-4413-4dcf-8560-1f998f2c2b4e measure=diskmbytes val=0 units=mb aug 15 07:13:46 vemmleads heroku/worker.1: source=heroku.10054113.worker.1.4589e3f4-8208-483a-a927-67c4c1cbee46 measure=load_avg_1m val=0.30 aug 15 07:13:46 vemmleads heroku/worker.1: source=heroku.10054113.worker.1.4589e3f4-8208-483a-a927-67c4c1cbee46 measure=load_avg_5m val=0.07 aug 15 07:13:46 vemmleads heroku/worker.1: source=heroku.10054113.worker.1.4589e3f4-8208-483a-a927-67c4c1cbee46 measure=load_avg_15m val=0.04 aug 15 07:13:46 vemmleads heroku/worker.1: source=heroku.10054113.worker.1.4589e3f4-8208-483a-a927-67c4c1cbee46 measure=memory_total val=511.80 units=mb aug 15 07:13:46 vemmleads heroku/worker.1: source=heroku.10054113.worker.1.4589e3f4-8208-483a-a927-67c4c1cbee46 measure=memory_rss val=511.78 units=mb aug 15 07:13:46 vemmleads heroku/worker.1: source=heroku.10054113.worker.1.4589e3f4-8208-483a-a927-67c4c1cbee46 measure=memory_cache val=0.00 units=mb aug 15 07:13:46 vemmleads heroku/worker.1: source=heroku.10054113.worker.1.4589e3f4-8208-483a-a927-67c4c1cbee46 measure=memory_swap val=0.02 units=mb aug 15 07:13:46 vemmleads heroku/worker.1: source=heroku.10054113.worker.1.4589e3f4-8208-483a-a927-67c4c1cbee46 measure=memory_pgpgin val=27303936 units=pages aug 15 07:13:46 vemmleads heroku/worker.1: source=heroku.10054113.worker.1.4589e3f4-8208-483a-a927-67c4c1cbee46 measure=memory_pgpgout val=154826 units=pages aug 15 07:13:46 vemmleads heroku/worker.1: source=heroku.10054113.worker.1.4589e3f4-8208-483a-a927-67c4c1cbee46 measure=diskmbytes val=0 units=mb the memory usage of worker.1 jumps 108.00 551.80 no apparent reason. doesn't jobs processed during time, it's hard understand giant chomp of memory comes from. way later in log, worker1 hits memory limit , fails.
i have newrelic pro running. doesn't @ - in fact doesn't create alerts repeated memory errors. above heroku logs give me no more information.
any ideas or pointers investigate next appreciated.
thanks
simon
there isn't enough information here pinpoint what's going on.
the common cause of memory leaks in rails applications (especially in asynchronous background jobs) failure iterate through large database collections incrementally. example, loading user records statement user.all
for example, if have background job going through every user record in database, should use user.find_each() or user.find_in_batches() process these records in chunks (default 1000 activerecord).
this limits working set of objects loaded memory while still processing of records.
you should un-bounded database lookups loading huge numbers of objects.
Comments
Post a Comment