multithreading - Python Multiprocessing Processes seem to share objects also the should -
in multiprocessing code several workers parallel processing.
the workers should communicating through request_queue , shared value lock.
but seems right because of "forking" when starting e.g. worker #4 , #5 share same dictionary document. found out using id(document) see memory address.
due storing document in mongodb, driver writting _id document, strange errors appear.
it has ensured every worker completly isolated expected queue , shared value, , don't know how right now.
the workers started with:
for in range(workers): worker( request_queue,i,val, lock ).start() class worker(process): def __init__(self, queue,ident,val,lock): super(worker, self).__init__() self.queue= queue self.idstr= str(ident) self.val = val self.lock = lock dbconn = dbconnector.dbconnector() self.mongoconnection = dbconn.getmongoconnection() self.flagcontroller = flagcontroller() print "ident" + self.idstr def run(self): print 'worker started' # initialization here print 'worker loop!' #time.sleep(5) try: data in iter( self.queue.get, none ): mid = data["_id"] print "#" + self.idstr + " : mongoid " + str(mid) #time.sleep(5) try: timestamp = time.time() document = {"rawdata": data, "c": { "quelle": "t", "timestamp": mid.generation_time, "query" : data["query"] } } self.mongoconnection.insert("productive","input",document)
update know tried pass in new document via constructor , use inside worker via self.document, sadly doesn't help.
one way of ensuring each worker thread isolated give each 1 own instance variable document. i.e. referencing document self.document.
Comments
Post a Comment