I'm a big basketball fan (mostly college). Where are you sourcing the videos? Also, are you manually tagging each video with player names, teams, and keywords or is there some magic happening on the backend?
Theoretically, it shouldn't be terribly technically difficult to attach a few sensors to each player to get a relatively hi-res 3D representation of each player's position throughout the game, should it?
Annotating the action accurately, not so easy, or might there be some trick?
with that said, bluetooth beacons, along with 3 bluetooth receivers for triangulation, could be used for inexpensive player tracking, but the data would be noisy and not as accurate (probably ~1 ft resolution or so). i've actually been wanting to put such a system together to determine how good it might be.