Python Forum
Why thread and queue only process the last row
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Why thread and queue only process the last row
#1
CSv date,ref_num,desc,debit,credit,payee_id,code,name 2019-01-31,L00001,john,30.00,,150,500,johnkino 2019-01-31,L00001,john,,30.00,150,600,johnkino 2019-01-30,L00002,john,30.00,,150,500,johnkino 2019-01-30,L00002,john,,30.00,150,600,johnkino
import csv from itertools import groupby import threading from queue import Queue def postRequest(): print('thread starting') while True: item_to_process = q.get() print(item_to_process) q.task_done() q = Queue() with open("test.csv", "r") as csv_ledger: r = csv.DictReader(csv_ledger) data = [ dict(d) for d in r ] groups = {} for k, g in groupby(data, lambda r: (r[ 'ref_num' ])): items = [ ] for i in g: chart_of_account = {k: v for k, v in i.items() if k in [ 'name', 'code' ]} item = {k: v for k, v in i.items() if k in [ 'debit', 'credit', 'desc', 'payee_id' ]} item.update({'chart_of_account': chart_of_account}) items.append(item) groups.update({ "date": i[ 'date' ], "desc": i[ 'desc' ], "ref_num": k, "items": items }) print(groups) <--- correct structure q.put(groups) for i in range(1): t = threading.Thread(target=postRequest) t.start() print('Main thread waiting') q.join() print('Done.')
Result
2019-01-30,L00002,john,,30.00,150,600,johnkino 2019-01-30,L00002,john,,30.00,150,600,johnkino 2019-01-30,L00002,john,,30.00,150,600,johnkino 2019-01-30,L00002,john,,30.00,150,600,johnkino
The data of groups is the complete and correct data structure which is loop every row but the result always loops for the last row only after passing to queue for processing.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Process doesn't work but Thread work ! mr_byte31 4 5,744 Oct-18-2021, 06:29 PM
Last Post: mr_byte31
  task queue Valon1981 8 6,330 Jul-07-2020, 07:41 AM
Last Post: freeman
  kill thread or process asap, even during time.sleep nanok66 4 5,527 Apr-29-2020, 10:13 AM
Last Post: nanok66
  Error SQLite objects created in a thread can only be used in that same thread. binhduonggttn 3 21,739 Jan-31-2020, 11:08 AM
Last Post: DeaD_EyE
  Queue in Pygame constantin01 1 5,309 Jan-07-2020, 04:02 PM
Last Post: metulburr
  How to sharing object between multiple process from main process using Pipe Subrata 1 5,046 Sep-03-2019, 09:49 PM
Last Post: woooee
  Queue maxsize mr_byte31 2 6,752 Sep-03-2019, 07:02 PM
Last Post: mr_byte31
  multiprocessing.Queue.join_thread() says "thread" Skaperen 0 3,079 Jul-29-2019, 12:45 AM
Last Post: Skaperen
  Queue.Queue() would not reduce capacity after get() yuan8421 9 16,375 Jan-02-2018, 09:38 PM
Last Post: Windspar
  Good way to have a worker queue accessible by multiple process? cheater 2 3,629 Dec-21-2017, 09:30 PM
Last Post: wavic

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020
This forum uses Lukasz Tkacz MyBB addons.
Forum use Krzysztof "Supryk" Supryczynski addons.