"Bitwise" is really shorthand for "this is an operation that acts on the representation of a number in base-2, rather than on the number itself", so it's no surprise that it doesn't translate well to base 10. The reason why that formula is so gnarly is that it does three things in one go:

- convert x and y from decimal to binary

- calculate x AND y at each digit (by using the property that x AND y is isomorphic to multiplication mod 2)

- convert back to decimal.

Instead of trying to understand bitwise AND in this way, we could try building the equivalent operation in base 10, and see where that leads us. One simple way of achieving this is by simply saying that x AND Y is digit-wise MIN, and x OR y is digit-wise MAX.

Note how, under that interpretation, this works in both binary and decimal (or hex, or octal, or whatever other base you want:

101 AND 110 = 100

101 OR 110 = 111

Or using digits not available in binary:

124 AND 310 = 110

124 OR 310 = 324

Again, this works independently of whether you're using octal, decimal, hex, or something else >= 5.