Perhaps hard attention could be a thing here. Possibly too tricky to train though.