In most cases no. Although the orbit of the bullet is cyclical (below escape velocity), unless you shoot perfectly level with the surface, its orbit will intersect the moon. Obviously if you shoot slightly down, it'll hit the ground. If you shoot slightly up, it'll hit the ground behind you on its return.
> Is this the velocity required for a bullet to leave orbit no matter what angle you shoot it?
Yes. Although if it hits the moon first, it'll slow down and not escape (unless you were to shoot through the moon).