A mechanical device would be much easier, and would be well within the capability of a single person with all of the prerequisite knowledge to make. Just look at the Antikythera Mechanism.
Even that's a huge head start over having to mine the raw materials.
It would probably take several lifetimes to build even the most basic computer from complete scratch even with all the manufacturing knowledge available.
I'd go mechanical any day. I recall the Soviets had a computer based on liquid that could compute certain differential equations.
http://hackaday.com/2011/03/25/mechanical-turing-machine-can...
I am not an expert on the subject, but I suspect making a facility that can manufacture thousands or millions of transistors, and then operating that facility would take obscenely more man-hours than one man has. Even if his goal was only a 4004, or even just a bunch of breadboards full of TO-92's.
Vacuum tube based might be possible for a very simple computer, if automating the glass-blowing and delicate assembly isn't too hard. After all, at least it's macro- rather than micro-!
Getting through smelting to machine tools would be impressive enough for me. Say stone-age to mid-industrial revolution.
In "Designing Analog Chips", Hans Camenzind tells the story of designing the 555 timer chip; he did the whole design, without a computer, up to cutting the masks out of Rubylith. It took him several months of nonstop work.
Based on these, I extrapolate that one person could do all the physical steps involved in bootstrapping to a computer from stone-age technology. (I won't say "one man", because that person might have to be Jeri.) And I think they could probably do it in substantially less than a lifetime.
But I don't think it would be possible for one person to do all the intellectual work, or it would have happened long before the advent of agriculture.
And I think it would be a lot more fun to do it with some friends.
http://lists.canonical.org/pipermail/kragen-tol/2010-June/00... are some thoughts about possible low-tech digital logic techniques. https://github.com/kragen/calculusvaporis has some thoughts about how to build a reasonably practical non-bit-serial CPU out of only about 1000 NAND gates (or the equivalent.) I think that with the height-field approach described in the kragen-tol post, you could probably make it out of 1000 moving parts — although most of them would have to be machined to fairly tight tolerances. I think the level of mechanical difficulty is similar to a 1960s V6 engine (fewer parts, but much higher force and demands on precision), or a little higher than that of a Curta calculator (600 parts for Type I).
The Clock of the Long Now uses steel gears on stone and ceramic bearings. It contains some mechanical binary calculation (and the prototypes are the most beautiful machines I've ever seen) but it's not a programmable computer.