Some early computers, as early as the 01940s, used binary; others used decimal, typically BCD but sometimes excess-3 BCD or one-hot encoding. Knuth's MIX was originally specified to be either binary or decimal, as you prefer, but by the later volumes of TAOCP he switched to requiring binary for things like tries.

Typically an early (pre-01960) computer was either "scientific" or "business". A "scientific" computer was binary, had word-oriented memory of at least 16 bits width and sometimes as much as 60 or more, was measured by its calculation speed, and had floating point. (Some didn't even have separate integer arithmetic.) A "business" computer was decimal, had character-oriented memory of 6-9 bits, was measured by its I/O speed, and had only integer arithmetic, or not even that (as in the case of the IBM 1620). These categories were blurring a lot by the 01960s, and the IBM 360 largely put an end to the division: it had 8-bit-wide memory, 32-bit-wide CPU registers, floating point and binary integer and binary decimal arithmetic, and different binary-compatible models with varying degrees of speed at I/O and calculation. A few years later there were even models that supported paging and timesharing.