Blob Track TOP

From TouchDesigner 088 Wiki

Summary

The Blob Track TOP is implemented using source code from OpenCV, much of which was ported to the GPU for faster performance. Blob tracking is done first by determining which pixels in an image are foreground pixels and which are background. Then using the foreground pixels tracking is done to find the blobs. Large groups of connected foreground pixels are considered a blob.

To get the results of the Blob Track TOP, attach an Info DAT or Info CHOP to it. The Info reports the current blob IDs, coordinates and sizes in pixels. The blob ID increases for every new blob that is detected.

Typical usage is connecting a video source (such as Video Device In TOP to the Blob Track TOP. Allow the Blob Track TOP to train/learn the background for a while (1 minute if you can). Then you can move foreground objects into the frame and see the tracking. Use the 'Output FG' parameter to see which pixels are deciding to be foreground (white) and which are background (black). Finally use a Info DAT or Info CHOP to get the results and use them as you need.

NOTE: can track an unlimited number of blobs in Pro and Commercial, and up to 2 blobs in the Non-Commercial version of TouchDesigner.

NOTE: The Blob Track TOP is only supported on Nvidia graphics cards.

PythonIcon.png blobtrackTOP_Class

Parameters - Blob Track

Init Delay Frames /initdelay - This parameter controls how many frames the node will cook for before it does any actual work. This is useful for cameras that takes a little while to adjust their aperture/focus. It avoids using frames that are much brighter/darker than the true image to start tracking.

Reset /reset - Resets all tracking data and learned background data.

BG Detection /bgdetect - Blob tracking works by separating the background pixels from the foreground pixels, and doing blob analysis/tracking on the foreground pixels. When this feature is on the TOP will use an advanced BG detection algorithm that attempts to separate existing background pixels from new elements that have entered the frame. You should train the TOP for as long as possible (a minute at least) so it has a strong idea of what the background is. If you turn this feature off then it is expected that you will give a 2nd input to the TOP that is specifies which pixels are the foreground pixels.

Reset BG /resetbg - Resets all the learn background information.

Train BG /trainbg - When this is on the background fed into the TOP will be used to train the TOP to know what the background looks like. It's the most expensive part of this TOP, so you can turn it off once the background is trained. If you have the GPU/CPU power though you can leave it on even when the TOP is tracking blobs.

BG Learn Rate /bglearn - This controls how quickly new static pixels thought to become background pixels. A higher number means pixels become part of the background faster.

BG Learn Response /learnresponse - Similar to the BG Learn Rate, this also controls how quickly pixels become part of the background.

BG Forget Rate /bgforget - This controls how quickly pixels that were decided to be background pixels are forgotten (so if something comes back into frame the pixels will be thought of as foreground pixels).

FG Source /fgsource - When not using BG Detection this parameter selects which channels from the 2nd input are used to decide which pixels are foreground and which are background.

FG Threshold /fgthreshold - The selected channel (the FG Source parameter) from the 2nd is compared against this value. Values that are higher than the threshold are considered foreground pixels, which values less than are considered background pixels.

Output FG Mask /outputfg - Enabling this parameter will let you see which pixels are being considered foreground pixels and which ones are being considered background pixels (white is foreground, black is background).

Blob Track Method /blobmethod - Once the foreground/background pixels have been determined, the blobs can be tracked. This parameter selects one of the many options implemented in OpenCV. In general there is a trade-off between speed/accuracy that needs to be chosen between on a case by case basis. For more information on the different methods you can refer to the OpenCV documentation.

Draw Blob Bounds /drawblobs - Draws rectangles on the TOP image that shows where the tracked blobs are.

Blob Bound Color /blobcolor - Determines the color of the rectangles that are drawn to show the blobs.

Delete Nearby Blobs /deletenearby - Sometimes (depending on the tracking method) duplicate blobs may be created. This feature allows you to delete blobs that are too close to each other.

Minimum Distance /deletedist - When deleting nearby blobs, blobs will be deleted if they are within this number of pixels of each other. The smaller blob will be deleted.

Delete Area Tolerance /deletenearbytol - Along with the distance, the area of the two blobs can be compared. If this parameter is 1, than the area is ignored. As this parameter gets smaller only blobs that have a significant size difference (between the two blobs being compared) will be deleted.

Delete Overlapping Blobs /deleteoverlap - Deletes blobs that are overlapping.

Delete Overlap Tolerance /deleteoverlaptol - If this parameter is 1 then only blobs that are completely overlapped will be deleted. As this value gets smaller less and less overlap is needed for a blob to get deleted.

Parameters - Common Page

Resolution - quickly change the resolution of the TOP's data.

  • Input - uses the input's resolution.
  • Eighth, Quarter, Half, 2X, 4X, 8X - multiply the input's resolution by that amount.
  • Fit Resolution - Resizes the input to the size specified in Resolution using the best possible match that does not crop any of the input. It will resize the image to be larger than the input resolution if a larger resolution is specified. It's a "fit inside", Aspect Ratio is maintained.
  • Limit Resolution - Limits the input to the size specified in Resolution using the best possible match that does not crop any of the input. It's a "fit inside", Aspect Ratio is maintained.
  • Custom Resolution - enables the Resolution parameter below, giving direct control over width and height.

Resolution - enabled only when the Resolution parameter is set to Custom Resolution. Some Generators like Constant and Ramp do not use inputs and only use this field to determine their size. The drop down menu on the right provides some commonly used resolutions.

Use Global Resolution Multiplier - Uses the Global Resolution Multiplier found in Edit>Preferences>TOPs. This multiplies all the TOPs resolutions by the set amount. This is handy when working on computers with different hardware specifications. If a project is designed on a desktop workstation with lots of graphics memory, a user on a laptop with only 64MB VRAM can set the Global Resolution Multiplier to a value of half or quarter so it runs at an acceptable speed. By checking this checkbox on, this TOP is affected by the global multiplier.

Output Aspect - sets the image aspect ratio allowing any textures to be viewed in any size. Watch for unexpected results when compositing TOPs with different aspect ratios. (You can define images with non-square pixels using xres, yres, aspectx, aspecty where xres/yres != aspectx/aspecty.)

  • Input - uses the input's aspect ratio.
  • Resolution - uses the aspect of the image's defined resolution (ie 512x256 would be 2:1), whereby each pixel is square.
  • Custom Aspect - lets you explicitly define a custom aspect ratio.

Input Smoothness - This controls pixel filtering on the input image of the TOP.

  • Nearest Pixel - uses nearest pixel or accurate image representation. Images will look jaggy when viewing at any zoom level other than Native Resolution.
  • Interpolate Pixels - uses linear filtering between pixels. This is how you get TOP images in viewers to look good at various zoom levels, especially useful when using any Fill Viewer setting other than Native Resolution.
  • Mipmap Pixels - uses mipmap filtering when scaling images. This can be used to reduce artifacts and sparkling in moving/scaling images that have lots of detail.

Fill Viewer - determine how the TOP image is displayed in the viewer.

  • Input - uses the same Fill Viewer settings as it's input.
  • Fill - stretches the image to fit the edges of the viewer.
  • Fit Horizontal - stretches image to fit viewer horizontally.
  • Fit Vertical - stretches image to fit viewer vertically.
  • Fit Best - stretches or squashes image so no part of image is cropped.
  • Fit Outside - stretches or squashes image so image fills viewer while constraining it's proportions. This often leads to part of image getting cropped by viewer.
  • Native Resolution - displays the native resolution of the image in the viewer.

NOTE: To get an understanding of how TOPs works with images, you will want to set this to Native Resolution as you lay down TOPs when starting out. This will let you see what is actually happening without any automatic viewer resizing.

Viewer Smoothness - This controls pixel filtering in the viewers.

  • Nearest Pixel - uses nearest pixel or accurate image representation. Images will look jaggy when viewing at any zoom level other than Native Resolution.
  • Interpolate Pixels - uses linear filtering between pixels. Use this to get TOP images in viewers to look good at various zoom levels, especially useful when using any Fill Viewer setting other than Native Resolution.
  • Mipmap Pixels - uses mipmap filtering when scaling images. This can be used to reduce artifacts and sparkling in moving/scaling images that have lots of detail. When the input is 32-bit float format, only nearest filtering will be used (regardless of what is selected).

Passes - duplicates the operation of the TOP the specified number of times.

Channel Mask - Allows you to choose which channels (R, G, B, or A) the TOP will operate on. All channels are selected by default.

Pixel Format - format used to store data for each channel in the image (ie. R, G, B, and A). Fixed format values are limited to the range [0-1]. Refer to Pixel Formats for more information.

  • Input - uses the input's pixel format.
  • 8-bit fixed (RGBA) - uses 8-bit integer values for each channel.
  • 16-bit float (RGBA) - uses 16-bits per color channel, 64-bits per pixel.
  • 32-bit float (RGBA) - uses 32-bits per color channel, 128-bits per pixels.


  • 10-bit RGB, 2-bit Alpha, fixed (RGBA) - uses 10-bits per color channel and 2-bits for alpha, 32-bits total per pixel.
  • 16-bit fixed (RGBA) - uses 16-bits per color channel, 64-bits total per pixel.
  • 11-bit float (RGB), Positive Values Only - A RGB floating point format that has 11 bits for the Red and Green channels, and 10-bits for the Blue Channel, 32-bits total per pixel (therefore the same memory usage as 8-bit RGBA). The Alpha channel in this format will always be 1. Values can go above one, but can't be negative. ie. the range is [0, infinite).
  • 8-bit fixed (R) - has 8-bits for the red channel, 8-bits total per pixel.
  • 16-bit fixed (R) - has 16-bits for the red channel, 16-bits total per pixel.
  • 16-bit float (R) - has 16-bits for the red channel, 16-bits per pixel.
  • 32-bit float (R) - has 32-bits for the red channel, 32-bits per pixel.
  • 8-bit fixed (RG) - has 8-bits for the red and green channels, 16-bits total per pixel.
  • 16-bit fixed (RG) - has 16-bits for the red and green channels, 32-bits total per pixel.
  • 16-bit float (RG) - has 16-bits for the red and green channels, 32-bits per pixel.
  • 32-bit float (RG) - has 32-bits for the red and green channels, 64-bits per pixel.
  • 8-bit fixed (A) - An Alpha only format that has 8-bits per channel, 8-bits per pixel.
  • 16-bit float (A) - An Alpha only format that has 16-bits per channel, 16-bits per pixel.
  • 32-bit float (A) - An Alpha only format that has 32-bits per channel, 32-bits per pixel.