that there used to be an option in TOP preferences to change the TOP video codec. I am wondering where this option is located now or if it is even necessary to improve video playback.
Those are very old and no applicable anymore. If you convert your videos to HAP and have some SSDs you should be able to play a bunch of layers without issues.