I use that too: ivy / counsel etc. Then to quickly select among the candidates and to not go "arrow down / arrow down / arrow down / arrow down / {tab,enter,whatever}" I use ivy-avy.

ivy-avy works like vim's EasyMotion or the older Emacs "ace-jump" mode (inspired by EasyMotion): you keep your eyes on the thing you want to select (or the place on the screen where you want your cursor to go to) and then invoke avy (by using a shortcut very conveniently placed as I use it all the time). Then a letter appears and you hit that letter (requires to be a touch typist as you cannot both keep your eyes on the candidate and look at your keyboard) and boom, you just selected the candidate / moved the cursor.

There's no faster way to select / move anywhere on the screen.

I'm using that since years (I used to use ace-jump before ivy/avy was a thing).

Imagine this list of candidates after I asked to open a file containing the regexp "green" (well it's just a word, but it could be a regexp):

    /home/jack/work/green.pdf
    /home/jack/private/greendayschedule.org
    .
    . (omit 10 lines here) 
    .
    /mnt/greenvelvet.txt
    /mnt/backup/green.cfg

Now I invoke "ivy-avy" and the list of candidates becomes this:

    f/home/jack/work/green.pdf
    g/home/jack/private/greendayschedule.org
    n
    b (omit 10 lines here) 
    c
    e/mnt/greenvelvet.txt
    d/mnt/backup/green.cfg

And these individual letters appear in another color, so they stand out.

To select, say, "mnt/greenvelvet.txt", you just hit the letter 'e'. Done.

It's really sweet and it works with nearly everything: selecting a candidate in a list (like a file, a completion, ...), moving the cursor anywhere on the screen, selecting a buffer, etc.