IBM never had plans for PCs to be graphically impressive (unlike other gaming-oriented microcomputers which had hardware sprites, buffer scrolling, and various special modes), so conceptually its video cards were pretty simple: you change values in video memory, and letters or pixels on screen change accordingly. The rest was on you. Initial setup could be complex (or hidden behind Video BIOS calls to specific firmware), along with memory layout (for various reasons, including production costs), but the thing was basically just an output buffer.
There were some accelerators providing mind-blowing functions like drawing a line on screen given its start and end coordinates in hardware, or filling a rectangle with a given colour in hardware. If you used professional CAD programs where taking drawing routines away from CPU meant going from 1-2 fps to “tolerable” object movement, there was a point in paying significant sum of money for such niche device. Later, Windows strongly suggested to graphic card makers to implements most often used window drawing primitives in hardware, and offer them to the system via standard driver interface. That also was of little use to most games.
Non-standard modes and hacks to manipulate output settings to make screen scroll without redrawing the whole buffer (which was too slow), and other tricks, like changing mode parameters precisely at the moment some specific part of the image was being sent to the monitor, were sometimes possible, but they don't apply here. Doom engine games render the upper world window on CPU frame by frame, writing resulting pixels to the video memory (status bar can be partially updated, or just ignored for our case). So it's a simple stream of writes, and the difference is in how fast they are processed.
What could be different?
— Memory subsystem and address mapping. Not sure about the details, but some hardware settings could possibly tell the motherboard chipset to ignore the caching/order/coherency, and instantly return with success from any write while it was in flight. That would mean that any program that also needed to read from video memory could get incorrect or corrupted results sometimes. (Though I contradict myself here: spectres and screen wipe effect in Doom needed to read from the screen buffer.)
— System bus type, width, and rate. Smaller delays means more frames per second (if images can be calculated fast enough).
— Video memory type, rate, and timings, video chip speed and architecture, internal bus width, presence of caching buffers, and so on. Most important differences, and there are benchmarks of VGA cards in contemporary magazines and on modern retro forums.
However, the video card itself couldn't make your CPU work with twice the performance, it only could limit the amount of data that went to screen. I suspect that either the card was not very compatible, and traded simplicity and performance in certain graphical modes for bugs and slowdowns in others, or the numbers you saw were actually incorrect. For example, if video card forced 60 Hz refresh rate in canonically 70/75/85/etc Hz mode, program could start calculating nonsensical delays, insert rendering cycles that did nothing, and show crazy stats.