multithreading in python?

Hey all,

is multithreading python scripts supported in 088? If yes, are there any example showing how to start/stop threads and how to get data from a thread back to the main thread (and/or into a node)

many thanks
Achim

Here is a sample using the ‘threading’ module.
The key thing to remember is you can’t use any object from the td or tdu modules in your other threads. You need to use native Python types or modules (or modules you’ve created yourself that don’t use anything from td or tdu), in the other thread.
Take a look at this and loop back to me with more questions as needed.

To use it, first run the ‘run_to_launch’ script, then press the ‘Queue Work’ button to send some work to the thread. Every frame the ‘checkForResults’ checks to see if the thread has produced any results, and writes any results to the ‘results’ DAT.
python_threading_sample.toe (5.19 KB)

1 Like

I have attached a simple example for input and output.

The script downloads two images and then adds their information to a table, which is being monitored by a Dat Execute.

Does updating that table after download qualify as “affect any objects in the main process”

I am still new to multi-threading but don’t see the need for the queuing.

Could anyone explain if this is problematic?
python_threading_sample.15.toe (6.32 KB)

Yes, this counts as affecting main process objects. The results are undefined. you can put the table in an bad state that can cause it to crash, or the operation itself can cause a crash if you are affecting the table at the same time the main thread is doing something with it.
You must use the queuing or some other type of python thread synchronization object or you can cause crashes.
The thing about threads using TD objects is that it won’t always fail, the main and other thread need to be hitting the same object at the same time for an issue to occur. This is called a race condition and as a general rule if it can happen, it will happen (possibly only at showtime and not before).

Okay, I made significant changes to the code here and implemented queues, I think in the correct way - and would appreciate some advice:

  1. When you say “TD Objects” you are basically referring to any operator, or any part of Touch correct? (I.e. not only tables but also just variables stored in storage as well?)

There are a some parts in my code that seem like they could be problematic:

  1. I am using a counter, accessed through Python storage, in multiple threads. This is supposed to be incremented when the file is written. However, if there are multiple simultaneous threads, it seems like they could be fetching the item out of storage and seeing it as the same value (i.e. 0) instead of its current value like if it were incremented sequentially. However, I do not see a way around this, as I can’t update the counter until the file has been written.

  2. I am appending rows to a Touch Table. This is all done in one for loop, however, as I read your notes, this seems to be problematic if that table was being read at the same time by the main process.

How this is any different than updating a table in a CHOP Execute? In your code, you seem to check for an updated queue value every framestart. Is the correct way to do it, put the results list in another queue, and have another touch process (i.e. Evaluate DAT) check the queue every frame?

Lastly, would doing all of this in another touch process, and updating a table via a TouchIN DAT, prevent these concurrency issues?

Perhaps seeing my code will help:

[code]from urllib import request
import threading, queue, os

url = ‘http://users.dialogfeed.com/en/snippet/dialogfeed-social-wall-twitter-instagram.json?api_key=ac77f8f99310758c70ee9f7a89529023

imgs = [
http://search.it.online.fr/jpgs/placeholder-hollywood.jpg.jpg’,
LPKF: PCB Prototype Technology/Laser Material Processing’,
http://bi1x.caltech.edu/2015/_images/embryogenesis_placeholder.jpg
]

def get_pic(url):
# Fetch image data
data = request.urlopen(url).read()
# This is the part I am concerned about, what if multiple threads fetch the counter before it is updated below
# What happens if the file write fails?
counter = me.fetch(‘count’, 0)

# Download the file
with open(str(counter) + '.jpg', 'wb') as outfile:
	outfile.write(data)
	file_name = 'file_' + str(counter)
	path = os.getcwd() + '\\' + str(counter) + '.jpg'
	me.store('count', counter + 1)
	return file_name, path

def get_url(q, results):
url = q.get_nowait()
file_name, path = get_pic(url)
results.append([file_name, path])
q.task_done()

def fetch():
# Clear the table
op(‘files’).clear()
results = []
url_q = queue.Queue()
# Simulate getting a JSON feed
print(request.urlopen(url).read().decode(‘utf-8’))

for img in imgs:
	# Add url to queue and start a thread
	url_q.put(img)
	t = threading.Thread(target=get_url, args=(url_q, results,))
	t.start()

# Wait for threads to finish before updating table
url_q.join()
for cell in results:
	op('files').appendRow(cell)
return

Start a thread so that the first http get doesn’t block

thread = threading.Thread(target=fetch)
thread.start()[/code]

A good way to know what is allowed and what is not: the code in your thread should be able to run standalone in python33.exe, outside of TouchDesigner. If it is using anything that TouchDesigner provides such as OPs, storage etc, then you are creating race conditions that will potentially cause crashes.

For counters you need to use locks:
eli.thegreenplace.net/2012/01/04 … processing

The difference with a CHOP Execute DAT is that that DAT always executes in the main thread, so there is no chance of memory it is accessing (DAT table for example) changing on it from another OP cooking. There is no threading going on there.

Yes if you do all of this in another process and don’t use threading at all, then you don’t need to worry about threading.

Upcoming build of TouchDesigner will now give a popup warning whenever an operator, or other TouchDesigner object is accessed in another thread.
This should help eliminate potential problems.

Nice! I think that will be a great addition.

Thank you for your ongoing help. I do have one more question:

You say:

but here in the code it looks like both the main thread and the sub thread are accessing the inQ

def workFunc(inQ, outQ): while True: # This will block until a new enty is placed on the queue value = inQ.get() outQ.put(value * 2)

Is this safe because the in.get() blocks until it has data so to the main thread (in framestart) it appears that there is nothing stored (until there is data)? It seems like something could be put in the queue at the same time framestart is getting it from the queue causing a crash.

Lastly, running a script on framestart seems like an inefficient way to get something. If for instance, I just wanted to retrieve data after an http call (using urllib) and I only fetch every so often, it seems a little excessive and I was hoping there was another option.

According to python documentation, the queue class is thread safe, so it should be fine.

docs.python.org/3.4/library/queue.html#

You are correct in that running a script on framestart is probably not the most efficient way to to get data. In your case, you could activate the opexecute when you do your http call, and then turn it off again after you’ve received your data. We are thinking about alternatives to speed up the process.

My sincerest thanks.