Even though the histories of computers seem to focus upon computational power, it seems as though the most challenging aspect in the development of computers is inexpensive memory. I was going to say fast and inexpensive memory, then I realized that we haven't really achieved that goal.
A lot of early computers depended upon the notoriously difficult to manufacture core memory. Then there are things like drum memories, which sacrifice density in order to use stationary read/write heads. And even though these forms of memory share a lot in common with hard drives, they were actually used as the main memories on some computers.
Earlier memories are even crazier, like mercury delay lines (complex, sequential access only, and a health hazard). Even later semiconductor memories endured a long period of explosive growth before reaching useful costs and densities. (Early minis and workstations often had large boards containing only memory. Early personal computers kept costs down by shipping with single-digit kilobytes of memory.)