I think a roving tabindex would be fine in this case. The arrow keys and tab/shift+tab change the tabindex so you always have elements with tabindex="-1" and tabindex="0".

https://www.w3.org/TR/wai-aria-practices-1.1/#kbd_roving_tab...