sql - How to handle Python multiprocessing database concurrency, specifically with django? -

so, i'm trying write application uses django orm, since it'll both need behind scenes processing , easy use front-end. it's core functionality processing data that's in database, in high-cpu process (basically monte carlo simulations) , want implement multiprocessing, using pool (i 4 processes). code runs this, 20 children of parent:

assorted import statements django environment in script multiprocessing import pool random import random time import sleep  def test(child):     x=[]     print child.id     in range(100):         print child.id,         x.append(child.parent.id) #just hit db     return x  if __name__ == '__main__':     parent = parent.objects.get(id=1)     pool = pool()     results = []     results = pool.map(test,parent.children.all())     pool.close()     pool.join()     print results 

with code such, intermittent databaseerrors or picklingerrors. former of form "malformed database" or "lost connection mysql server", latter "cannot pickle model.doesnotexist". random, occur process, , of course there nothing wrong db itself. if set pool = pool(proccesses=1) runs, in single thread fine. throw in various print statements make sure of them running.

i have been changing test to:

def test(child):     x=[]     s= random()     sleep(random())     in range(100):         x.append(child.parent.id)     return x 

which makes each iteration pause less second before running, , makes fine. if random interval down 500ms starts acting up. so, concurrency problem, right? 4 processes hitting. question how solve without making large dumps of data ahead of time? have tested both sqlite , mysql, , both having trouble this.

okay, determined (with of friend) issue django using same database connection processes. normally, when have concurrent db requests, either in same thread (in case gil kicks in) or on separate threads, in case django makes different database connections. multiprocessing, python makes deepcopies of everything, it's passing same database connection subprocesses, , step on each other until breaks.

solution trigger new db connection within each subprocess (which relatively quick).

from django import db ... def sub_process():     db.close_connection()     #the rest of sub_process' routines  #code calls sub_process pool 

went , forth having line, , not having line, , fixes everything.