Very poor GPU performance all of a sudden

A few days ago I noticed that when I opened up TouchDesigner to the default “flower noise” network my frame rate had dropped to 30fps. Initially I thought that maybe my monitor had been accidentally set to 30Hz, but that wasn’t the issue.

So I opened up the Performance Monitor and noticed that my GPU operations are taking an extremely long time (see attached screenshot). It looks like two gfx operations are the culprits:

  • “Uploading a 2D texture” is taking a whopping 13ms and appears to be related to the chopto1 TOP

  • “Flushing GPU Commands and Changing Graphics Content” also takes nearly 13ms

Has anyone experienced this issue? I have a Nvidia GTX 980 Ti and tried updating the graphics driver but it didn’t have any effect. The only thing I can think of that I installed around the time this problem arose is that I installed CUDA 7.5 for Octane Render, but I’m not sure how that would affect Touch.

Thanks!

I would say to triple check you have the newest NVidia driver, but here are some other maybe obvious things to check.

  • Did you accidentally plug your monitor in to your motherboard?
  • Do you have other GPU-intensive programs running simultaneously?
  • Also, I think? that those performance operations are actually an exchange of information from your cpu to your gpu, so if your cpu is oddly bottlenecked that can also cause this slow down?

Hi Emintzer, I’ve been pretty diligent with my tests, so I know that there aren’t any other processes running that could be hogging the GPU. Also double checked that I have the latest drivers today.

Interesting point about those operations actually being exchange between the CPU and GPU, I’ll double check to see if there’s anything weird happening in the background with my CPU.

Thanks for the notes!

Have you tried putting it in perform mode full screen at 60fps and monitoring the fps only from perform mode (without performance monitor window open)?

Hi Elburz, yep, I was thinking the same thing. I tried hooking a Perform CHOP to a Trail CHOP, then going into perform mode for a bit, then going back to check the FPS over time. It never goes above 30fps.

Another method that’s pretty reliable is to use the cook channel from perform chop, it tells you if you lose a frame cook. If you put a slight lag on that, you can connect it to the opacity of a constant top, then just composite it on your output, then you can be in perform mode, not worry about trails cooking, and can visually see if you’re dropping a frame if your constant top is flickering.

unfortunately not quite sure what it could be. did anything change recently?

1 Like

That’s awesome, I’ve never heard of that method of checking frames.

Nothing has changed yet. I’m about to reinstall my OS to see if it’s a hardware issue haha, will report…

Hello
I’m having the same issue that the fps doesn’t go beyond 30fps.
Have checked on all the suggestion give above.
Requesting for help.

Thanks

Could you try Perform Mode and on the Window COMP change the Vsync parameter to ‘disabled’. I am wondering if vsync is doing strange stuff and capping you guys at 30fps. (Or you can change the setting globally in Nvidia Control Panel). We are seeing a lot of vsync problems with Optimus laptops lately over the past few Windows Updates.

I’ve had this bug combined with additional really heavy frame drops whenever I would open a second window.If you haven’t already try to uninstall the Nvidia Geforce Experience software. It seems that it causes quite a lot of problems with TD099.

Hope that helps!

David

I have the same problem , with a Fresh install of win10 and Gforce980m (latest drivers) ,the frame drops to 30 when i open any floating window inside touch ,when Vsync is turn OFF in Nvidia Global Settings the frame go to 60 fps…

Me too :frowning: .

 External Monitors with vsync on drop to 30 fps no matter what  do. Turn V sync off and goes right back up to 60 but screen tears horribly.  

I am running a Desktop with I9’s, Quadro P5000 GPU’s windows 7 pro , no gforce experience. all Monitors are at 60. Only one window coverall all monitors and I get 77-80 fps realtime off and v sync off,
I went back to 088 and same issue.

What are the refresh rates given in the Monitors DAT for your external monitors?
Do you have more than on P5000 in the system?

Hi Malcolm,

  Refresh rates are @ 60 in the monitors Dat, I have 2 P5000's in a SLI Mosaic 2x4 for external display and a P600 for local 1920 x 1080 display. Total external monitor resolution is 15360X4320    :astonished: . I know this is a large canvas....I use a .bat file with gpu affinity to launch TD.

 I have 2 970 Pro's NVME drives in Raid 0 for the media drive and it tests in crystal disk 7gb a sec sequential read.  I also use Geometry for some effects with little to no overhead and same results...30 fps. unless I turn off real time....

Been testing and a few notes…not sure if they are 100% related

  1. when I play hapQ video file at full resolution 15360X4320 and the disk read rate in the moviefilein top exceeds 12,000 mbs the Frame Rate collapses. 2fps is typical if this happens. Performance Monitor says that the moviefile in top is waiting on the hdd. 12000mbs is only about 1.5 GB a sec. My Drives support about 7 gig a sec. Mabe PCI lane saturation? not sure of the issue or if it is related to the 30 fps talked about above. I have tested on many different servers with same results. the same 12,000 mbs wall.

Asus and Gigabyte motherboards with x299 chipsets were tested. NVME drives are in 3.0 4x pci lanes.

Motherboards with the x370 chipset seem t0 only get half thy bandwidth they choke @ 6,000mbs

If I split the video into two files ( each well below the 12,000mbs) no problems and will play a 30 fps video fine at 30 fps. But no matter what i can’t seem to get 60 with v sync on, with it off screen tears.

FYI I am on Windows 7 and latest Nvidia drivers.

You should confirm your setup, I don’t believe you can use both Affinity and Mosaic at the same time. Mosaic essentially hides multiple GPUs from the system and presents a single logical GPU, and Affinity is the opposite where you load each GPU on your own using the bat file and command line extensions to launch a process per GPU.

In Mosaic you’ll want one TouchDesigner window covering all the screens (except the non-mosaic control panel on the P600) and then a second control process only on the P600 screen. When the processes cross screens it incurs a huge performance penalty for transferring data between GPUs.

That’s interesting two files work better than one. How did you encode the Hap videos? Did you use TD or QuickTime? For super high resolution video you’'ll want to encode using TD, or FFmpeg with the chuncked encoding option (check our wiki). This will put the files into a format that can be better decompressed with multiple CPUs.
Another option would be to encode in TD with the ‘Secondary Compression’ option turned off for Hap. This avoids the CPU level compression altogether, which will use more SSDs speed, but you seem to have lots of that. It uses the same amount of PCIe bandwidth, but less CPU power.

1 Like

Thank you both for the responses
@Elburz,
the SLI Mosaic with 2 P5000’s shows up as One GPU in windows/TD “GPU 0”, and I have a P600 for a local 1920x1080 display that is “GPU 1”. I tested without Affinity and it seems to make no difference…with or without Affinity the system responds the same. I guess Affinity is not needed as each instance seems to run on the GPU in which the window is rendered without Affinity. Cool.
I have another similar system I have been running for about 8 months with affinity on…
overwatchleague.com/en-us/
but the res of this LED screen is only 11520X4320 :wink: and no issues at all…

@ Malcom
The Hap Videos were encoded with Adobe and the Vidvox codec. I have asked for CineForm masters that I will encode with TD and FFMpeg and do some testing and share what I find…

The LED products is 3.75 mm pitch and with a resolution @ 15,360 it is a massively large LED screen @ 187 ft wide x 50ft tall...and the main focus point in the Arena we have fast moving horizontal content and my goal is to get 60 fps. I will keep you posted!

Thanks!
Walker

Adobe/Vidvox wouldn’t use chunked encoding, so it’s worth trying another encoder for that.