Here's what I ended up coming up with. It would be cool to figure out how to do the color amplification too.
Basically it does a subtract between the movie at the current time and the movie at (a) frames ago and gives you a monochromatic difference in motion between the two.
Then to calculate the direction in which the image is displaced, the monochromatic difference is sent through 4 different feedback loops in opposing x and y directions. Each chain of feedback is multiplied by the monochromatic difference at (b) frames ago thus giving you a positive brightness only if it travels in the direction of the feedback.
Then each feedback loop gets colored according to its appropriate displacement direction and finally, all of the feedback loops are composited into one signal, added over gray, and sent into a displacement top which displaces the original image (c) frames ago.
Hope this makes sense. Any suggestions of enhancements or modifications would be greatly appreciated.