Hasn't this been done already? IIRC there have been some machine learning AIs that use raw pixels as observations.