Hacker News Reader

Comment by cess11 | Hacker News Reader

The literature on distributed systems is huge. It depends a lot on your use case what you ought to do. If you're lucky you can avoid shared state, as in no race conditions in either end of your executions.

https://www.youtube.com/watch?v=_9B__0S21y8 is fairly concise and gives some recommendations for literature and techniques, obviously making an effort in promoting PlusCal/TLA+ along the way but showcases how even apparently simple algorithms can be problematic as well as how deep analysis has to go to get you a guarantee that the execution will be bug free.

View on HN

My current concern is a CRUD interface that transcribes audio in the background. The transcription is triggered by user action. I need the "transcription" field disabled until the transcript is complete and stored in the database, then allow the user to edit the transcription in the UI.

Of course, while the transcription is in action the rest of the UI (Qt via Pyside) should remain usable. And multiple transcription requests should be supported - I'm thinking of a pool of transcription threads, but I'm uncertain how many to allocate. Half the quantity of CPUs? All the CPUs under 50% load?

Advise welcome!

Use `concurrent.futures.ThreadPoolExecutor` to submit jobs, and `Future.add_done_callback` to flip the transcription field when the job completes.

Although keep in mind that the callback will be "called in a thread belonging to the process" (say the docs), presumably some thread that is not the UI thread. So the callback needs to post an event to the UI thread's event queue, where it can be picked up by the UI thread's event loop and only then perform the UI updates.

I don't know how that's done in Pyside, though. I couldn't find a clear example. You might have to use a QThread instead to handle it.

Thank you. Perhaps I should trigger the transcription thread from the UI thread, then? It is a UI button that initiates it after all.

The tricky part is coming back onto the UI thread when the background work finishes. Your transcription thread has to somehow trigger the UI work to be done on the UI thread.

It seems the way to do it in Qt is with signals and slots, emitting a signal from your QThread and binding it to a slot in the UI thread, making sure to specify a "queued connection" [1]. There's also a lower-level postEvent method [2] but people disagree [3] on whether that's OK to call from a regular Python thread or has to be called from a QThread.

So I would try doing it with Qt's thread classes, not with concurrent.futures.

[1] https://doc.qt.io/qt-5/threads-synchronizing.html#high-level...

[2] https://doc.qt.io/qt-6/qcoreapplication.html#postEvent

[3] https://www.mail-archive.com/pyqt@riverbankcomputing.com/msg...

Terrific, thank you. You've put me on the right track.

Thank you.

Just use multiprocessing. If each job is independent and you aren’t trying to spread it out over multiple workers, it seems much easier and less risky to spawn a worker for each job.

Use SharedMemory to pass the data back and forth.

Reply on news.ycombinator.com