Hitting bottleneck controlling close to 4,000 LEDs!

Hi Lucas,
thanks for your support, here is a very simplified and commented .toe, it’s great if you have the time to take a look at it !
I’m currently using WS2812b ledstrips, I know it’s kind of slow and innacurate but it does the job.
This is the controller : [url]http://www.electrondes.com/an6u1903_00/an6u1903_00.pdf[/url]
it looks cheap but works incredibly great !
LEDmapper_v2.10.toe (9.03 KB)

Hi Everyone.
Thanks for the great analysis and working solutions.

Some thoughts on the python script:

Lucasm, can you describe your issue with:
"A side note here, I have to split the led bytes into chunks or packets of 255 or less due to a built in limit that sendBytes has… "

Looking at your sample script, I notice you do things like:

exec(‘op(“%s”).sendBytes(%s)’%(serialConnectorName, str(subArray).strip(‘[]’)))

Instead of:

op(serialConnectorName).sendBytes(subArray)

Though I suspect: .sendBytes(*subArray) may work in your case.

Minimizing the amount of scripting altogether would likely improve performance.
(ie, using a Quantize CHOP to avoid the int() calls, and a shuffle CHOP to get everything in the right format).

It might boil down to:

n = op(‘some_resulting_chop’) #everything interleaved and with shuffle CHOPs
v = n[0].vals #put everything into one float array
op(‘serial_dat’).sendBytes(v)

Also, agreed, FM64, those reorder CHOPs seem unnessarily slow.
Thanks for the examples, we will try to optimize them further.

We’re also researching the idea of changing the formats accepted by the DMX CHOPs to avoid the interleaving altogether.

Cheers,
Rob.

A couple more points:

The latest official, Shuffle CHOP has a relatively recent option (August 11th):
“Sequence All Samples”
Which would take three long channels, (example: r, g, b), and reshuffle it into one
single interleaved channel: (r0, g0, b0, r1, g1, b1, …)
This may be of use in some networks.

Also, we’ve just optimized the Reorder CHOP to be 5 to 10 times faster when dealing with a few hundred channels, which is often the case with these LED networks.
That will be in build 25440 or later.

Cheers,
Rob.

I’ts awesome Rob, thanks a lot for your reactivity, can’t wait to see my network run @60FPS :wink:

Rob, thanks a bunch for sharing your thoughts on that part of the send code. Ill be honest i was looking to optimize everywhere else but there… Everything you suggested did LOADS of good…

Precisely the bits about shuffle CHOP to get the samples ordered correctly.
Building that ordered array via loop in python was also eating up a ton of time. The append function was probably being used WAY too much.

My FPS is around the 60 range and solidly sticking. All of it is split between two instances of Touch Designer with out the help of any other programs . Woo!

Not quite sure why I was approaching it the way I was with strings, but using a direct array with that asterisk in front of it did perfectly… and loads faster.
What’s with the asterisk anyways? I noticed it stripped away the commas when sent through a print statement, I’m curious, is that all it does?


All that being said, here’s the new Serial send code, it’s LOADS faster and WAY simpler.
You’re the man Rob:

[code]
def valueChange(channel, sampleIndex, val, prev):

n = op('topto1') # CHOP to pull rgb data from
serialConnector = op("serialConnector") # serial DAT

vals = [int(i) for i in n[0].vals] # returns a list with values converted to ints.
serialConnector.sendBytes(*vals) # Send data to leds!

return
[/code]

I’ve attached a new TOE file to reflect the changes and updates.
simpleLedMappingAndDrivingWorkflow.6.toe (9.35 KB)

FM64,

I took a look at your file, made a few changes and attached it.
I switched the order of operations up a bit and moved a few things to your ledstrips container.

I was able to cut it down by almost about half, but there might be a way to squeeze a few more MS out of it. It’s getting by at just over 60 though.

I suspect that won’t hold if you have more going on in that file but if you split it off to another touch process I imagine it would hold fine!

You may already be doing this but when you test fps, open your performance monitor and then go into performance mode and then hit analyze. Touch’s UI takes a good chunk it’s self understandably and that can make it seem worse than it is.
LEDmapper_v2.12.toe (8.98 KB)

That’s awesome. Im glad all our bits are making one fast system.
One more thing:

vals = [int(i) for i in n[0].vals] # returns a list with values converted to ints.
serialConnector.sendBytes(*vals) # Send data to leds!

The .sendBytes will cast each entry to int anyways, but to be extra sure, you can use the Limit CHOP to quantize/round a channel to whole numbers, eliminating that line entirely.

As for the python * operator, apparently called the ‘splat’ operator, which expands a list or tuple into separate arguments in a function call.

You can save a bit more by not using a select CHOP but a constant CHOP with a replace CHOP in the dmx_output COMPs.
The expression you are using in the select CHOP can be copied and pasted into the name parameter of the constant CHOP, then use the constant CHOP as the first input to the replace and the select2 CHOP as the second input. This little trick saves you another ~0.18 ms per dmx output…

cheers
Markus

Hey Rob,

That makes sense, I added a limit and set quantitize steps to 1 however it doesn’t seem to to changing the data type? (I had this as last CHOP in the chain so it was sampling directly from it.)

Also, I’ve had that issue before with other node types where I try to reference a channel or something and it gives me the <class variableType … thing. Is there a different way to reference touch arrays to avoid that?

Snaut,

That’s a pretty awesome trick, would you mind explaining a bit more about why that works?
It brought frame time down to mostly under 10 ms ! well within the 60 fps range.

Thanks Lucas for your example, definitely, using one shuffle CHOP (sequence all channels) upstream instead of the reorder CHOP before each output was the best idea.
But look a this .toe, it’s even more quick, I reuse your first ideas to trim the channels and split it up inside of each dmx_outputs COMP

@Snaut, thanks for the tips but i didn’t get better performance with it, the replace CHOP cooking time was around 0,15ms …
LEDmapper_v2.16.toe (8.99 KB)

Nice! You got that frame time WAY down.
Your file is registering 2-5 MS per frame on my end. Glad that’s working out, good to know an artnet workflow could push the same if not better in terms of speed.

In the spirit of optimization I setup some nodes in my setup that captured the time in milliseconds just before serial data was sent to the arduino and just after the arduino finished it’s draw loop (it sends back a single byte to touch signifying it finished a frame update)

Anyways, I’m sitting nicely just at .017 MS between the before and after. That’s pretty much at 60 FPS.

I think the ws2811 leds flicker a little when updating that many led’s with a fast moving pattern so it has an ever so slight jumpy look but I can only see it with a really narrow ramp animating really quickly across the panel.

The replace is fairly simple operation compared to the select as it literally just has to look if the channel is in the second input and replace the first inputs channel value. The select has to do a bit more work and watch for more changes in its input.

Compared to the trim, the replace is still slow but it was about only 1/3 of the cooktime required by the select CHOP.

cheers
Markus

I’ve also just changed the .sendBytes methods to allow passing in an array of floats, so no conversion to ints necessary.
ie: n.sendBytes( *channel.vals ) will work now.

(build 25680+)

Cheers

Thanks Rob! I’m going to snag that update soon then :smiley:

Lucas

25880 is posted now if you wanted to try this out.

Hey everyone, been a few months, I just wanted to update this thread again with the final results of all the tinkering and learning :slight_smile:

The system totals out to 5,280 led’s, portable, modular, and pixel mapped. I’ve VJ’ed with it a few times, and it works pretty well! Still working on setting up / breaking down quickly but that’s a different subject entirely.

compilation:
[url]16 panel compilation - YouTube

This is awesome!

So what hardware would you suggest?

Teensy, Arduino, Pixel Pusher or FadeCandy?
Also, what panels are you using?

Thanks for the contribution!!

I did Touch Designer → Serial Out → Teensy 3.1 → Octows2811 adapter board → Leds

For the led’s, there’s a lot of routes you can go, I built my own custom panels out of led strips and a custom pcb I made that just made soldering everything really fast:
ledPcb.jpg
I 3d printed the diffusers and frames/brackets and other unique pieces and used some fiberboard from home depot.

These guys make some rigid large led pcb’s that look pretty nice if you want something ready to go.
[url]http://rgb-123.com/shop/[/url]

For microcontroller, you could use a lot of different things, I liked the teensy 3.1 and octows2811 adapter board. Really solid performance from that at an excellent price point.

Obviously running serial long lengths is not as ideal as ethernet, but if you go this route, this is a really solid active usb extender cable:
[url]http://www.amazon.com/Monoprice-Female-Extension-Repeater-Compatible/dp/B004PLLA9U/ref=sr_1_10?ie=UTF8&qid=1423812738&sr=8-10&keywords=usb+extension+active+monoprice[/url]

Vj’ed with it a few times and it carries the data reliably and has no problems detecting on the pc side.

The part that I think is most limiting for some is mapping the leds, visually. Especially when you have a custom setup, or custom arrangement.

I uploaded a barebones examples TOE further back in this thread that shows how I do it from A-Z (thanks to everyone here for helping work out the kinks on that!)

Hope that helps!
Lucas

Thanks! I have no problem going DIY as I have an engineering background :wink:

I’ll probably build something similar in the near future, thank for the info it really helps! I’ll post updates when I have something to show!

Update time!

I wanted to drop back in here and put out some updated info on driving led’s with TD, over serial to teensy. I had a few pieces of mis information from before and there’s been a few updates on fastLED octows2811 librarys over the months that make things faster!

Current figures:

  • TD 60 fps
  • Serial over usb (for multiple teensys use a multi-tt hub)
  • Teensy 3.1 running a fastLED/OCTOws2811 mixture (more on this below if you’re curious)
  • 3,680 ws28xx leds per teensy 3.1 (460 per pin) also @ 60 fps

Dependancies: (also attached for conveniance)
Arduino Libraries:
[url]https://github.com/FastLED/FastLED/tree/FastLED3.1[/url] -attached-
[url]https://github.com/PaulStoffregen/OctoWS2811[/url] -attached-
[url]https://www.pjrc.com/teensy/teensyduino.html[/url] (addon for arduino IDE)

Arduino sketch:
-attached-

Touch Designer Barebones example
-attached-

Notes:

You can scale this solution by adding more teensys running the same code to the mix. I have not reached an upper limit but haven’t tested it too much yet, but I imagine at least 3 or 4 teensy’s total could be done with out dipping below 60 fps, perhaps more.

Eventually the USB host card will reach it’s max, and / or TD will choke on sending that much data out via serial, but i’ve found that with the modifications the dev’s helped put forward in this thread you will probably hit a hardware wall before a software wall !

For those integrating this into already complex / heavy networks, it may be wise to offload the communication part to another touch designer process and use a sharedMem chop or spout in/out to offload that work and keep speeds up!

A little info on why / how the led counts have increased with this solution:

The reason the octows2811 library works so damn fast is due to the r/g/b data being formatted in a special way that the teensy’s DMA engine can push out natively, so no bit banging, which means hardware can do it’s thing on 8 strips in parallel. However! The .setPixel() command used previously has been somewhat slow, so Daniel Garcia at fastLED has written some backend code that does that translation much faster using some of fastLED’s code.

more info:
[url]https://plus.google.com/communities/109127054924227823508/s/octows2811Demo[/url]
[url]https://github.com/FastLED/FastLED/wiki/Parallel-Output[/url]

I’ve been researching pixel controllers and various software / hardware configurations, and have not found anything like this before. Paul, Daniel, and everyone at Derivative have made this a painless and extremely elegant (and cost effective!) way of building / controlling custom led art, displays etc.

Thank you for that :slight_smile: Hope this helps some of you looking for a solution and missing bits and pieces of how to make it all work.

Lucas
OctoWS2811.zip (171 KB)
FastLED3_1.zip (250 KB)
ledPixelController_460.zip (832 Bytes)
ws2811-12-12b_DrivingSolution_BareBones.1.toe (5.74 KB)

2 Likes