Final Done Task Repeating #10

Activity

1. graingerkid commented on Apr 25, 2017

   

   graingerkid

   on Apr 25, 2017

   Author

   More actions

   I added a time.sleep(1) before the q.join() and that seems to have fixed the issue.

   I guess it's terminated before it's completed the run.
2. balena commented on Apr 25, 2017

   

   balena

   on Apr 25, 2017

   Last edited by balena

   Owner

   More actions

   Hello @graingerkid, I will try to reproduce your case.

   But a possible explanation is that the last entry is repeated because it was not completely processed (it didn't get a corresponding task_done()), so indeed it restarts from where it stopped.

   Your initial code does not have a join(), so there's a chance that the main thread exits before their children, and in this case the processing may be interrupted in the middle. You can force this case by adding a time.sleep(1) in the worker() function emulating some intensive processing.
3. graingerkid commented on Apr 25, 2017

   

   graingerkid

   on Apr 25, 2017

   Author

   More actions

   Yea this was my full code (sorry)...

   def worker():
       while True:
           try:
               item = q.get()
               print item
           except:
               pass
           finally:
               q.task_done()
   
   for i in range(3):
        t = Thread(target=worker)
        t.daemon = True
        t.start()
   
   for item in range(100):
       q.put(item)
   q.join()
   

   I'd put time .sleep(1) in as follows:

   def worker():
       while True:
           try:
               item = q.get()
               print item
           except:
               pass
           finally:
               q.task_done()
   
   for i in range(3):
        t = Thread(target=worker)
        t.daemon = True
        t.start()
   
   for item in range(100):
       q.put(item)
   
   time.sleep(1)
   q.join()
   

   but you suggest within the worker() function? I'm just concerned about speed, though time.sleep(0.1) might be fine.
4. balena commented on Apr 25, 2017

   

   balena

   on Apr 25, 2017

   Last edited by balena

   Owner

   More actions

   Try the following:

   def worker():
       while True:
           try:
               item = q.get()
               print item
               time.sleep(1)   # emulate an intensive work here
           except:
               pass
           finally:
               q.task_done()

   Then, remove your last time.sleep(1) before q.join(). You would have to wait ~33 seconds if you keep range(100), so you can reduce to just 10 iterations by using range(10) and you will wait 3 seconds each time.

   By repeating the execution, you would not see the item 99 repeating in the second run. If so, we may have a BUG in the code.
5. graingerkid commented on Apr 25, 2017

   

   graingerkid

   on Apr 25, 2017

   Author

   More actions

   It seems to work sometimes and then not others. Any ideas how we could narrow down the issue?
6. graingerkid commented on Apr 25, 2017

   

   graingerkid

   on Apr 25, 2017

   Author

   More actions

   FYI i've got the following set...

   Queue("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/micro_service/queue_data", tempdir="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/micro_service/queue_data")
7. balena commented on Apr 28, 2017

   

   balena

   on Apr 28, 2017

   Last edited by balena

   Owner

   More actions

   @graingerkid, I think you already narrowed down the problem. I'm just without time to work on this issue, but I will certainly do a test case for it.