Moving object projection mapping using HTC VIVE+camschnappr

Hi. I am now implementing Moving object projection mapping using HTC VIVE + camschnappr.

However, it is difficult to change the projection matrix in real time for moving objects.

viewtopic.php?f=27&t=10787&p=47510&hilit=htc+vive#p41394

I refer to this link.
If you know how to change the projection matrix in real time, can you help?
viveCamSchnappr.tox (46.1 KB)

you need to move the SOP so that snapper knows where the points are in vive space.

Then it will calibrate to that world coordinate, then you are good.

Thank you harvey moon.

But i do not understand well.
Do i need to know the rotation matrix and translate matrix of the vive tracker?

best regards.

Thank you harvey moon.
How did you map the z value?
If I use cv2.calibratecamera, the z value disappears.