When deciding what color the video scan beams need to be at a particular point on the screen, a Spectrum is just looking up the corresponding pixel in the screen buffer to determine if it's foreground or background, then the corresponding attribute block to find out what the foreground and background colors are.
The NES is keeping track of whether the current pixel is within one of the eight sprites that overlap that scanline (of the 64 total sprites whose positions it knew at the start of the frame), and if so it's grabbing that sprite's corresponding pixel from the tile VRAM; if not, it's working out which tile in the background tilemap it's over, finding the corresponding tile image in VRAM, and grabbing a pixel from that.
That's a really clever trick if what you want to do is display sprite and tilemap based graphics!
But it does you no good if you want to do something like draw a circle.
It also required, on the NES, a dedicated PPU chip specifically to handle all those shenanigans. Sinclair were using a ULA to build their video hardware; Nintendo were making custom chips.
To be concrete, if you have a 256×192 screen divided into 32×24 tiles of 8×8 pels (as the Spectrum ZX did) the biggest full circle you can draw is 192 pixels tall. It only has 74 distinct tiles, counting "all filled" and "all empty" as distinct tiles, and if you can mirror those horizontally and vertically (as the Atari 2600 did for its backgrounds) you can make do with 20.
I think a very reasonable balance would have been a pattern table of 256 distinct 8×8 tiles with 2 bits per pixel, totaling 4096 bytes of RAM; a tilemap of 1024 8-bit tile indices and 1024 16-bit tile attribute words, including three 4-bit colors and two mirroring flags (and two spare bits), totaling another 3072 bytes; and 2–8 sprites of 8×8 or 16×16, also 2 bits per pixel, totaling maybe 256 more bytes. This would have given you spectacular, NES-quality sprite and tilemap graphics, but with enough flexibility to do the 3-D isometric graphics in the Spectrum's most spectacular games. And it would have left you a lot more space for game logic. (Maybe you could have stolen a bit from the tile attribute word to get 512 pattern table entries, nearly as many as the 768 tile positions on the screen.)
Could the gate array in the Spectrum have done it? Maybe. Let's say each of the 192 logical pixel lines gets scanned out three times. On 576 out of 625 PAL lines, 50 times a second, you have to spew out those 256 pixels in PAL's 51.95 μs, a dot clock of 4.928 MHz, 203 ns per pixel. (So far, this is exactly what the Spectrum did do.) Each 2-bit pixel value controls a 4-way mux among four colors, three loaded from the attribute word and one loaded from the background color. (Separately you have a sprite engine that's maybe doing the same thing from a sprite or two, and one pipeline stage later you mux between the background and the sprite foreground under the control of a registered bit from the sprite engine.) Every 8 pixels, 1.62 μs, you need to clock in the colors from the next attribute word—exactly as the Spectrum in fact did, but 12 bits instead of I think 6—out of the 384 bits of colors for the 32 tiles on the current row.
These are the only parts of the display driver that have to be fast: the 512 bits of the current pixel line, the 384 bits of colors, the perhaps 32 bits of pixels and 12 bits of color per sprite, and the shifting and mux logic. This is definitely more than the 256 bits of pixels and I think 192 bits of colors the Spectrum actually did have to handle at this speed, but only moderately more. And the difference in visual expressivity would have been night and day. Maybe you could even have afforded higher vertical resolution (as, again, the NES did).
There are other tasks that need to happen concurrently: every 3 scan lines (192 μs) the pixel line needs to be reloaded from the tilemap and the pattern table, a task which involves indexing into the pattern table 32 times by combining 32 bytes from the tilemap with a (possibly inverted) 3-bit line index, and possibly pixel-reversing the row loaded from the tile; this can be done either during the 64 μs of the third scan line (500 ns per index), if the 512 bits are dual-ported, or during the entire leisurely 192 μs period (1.5 μs per index), if you double-buffer them and switch back and forth between two 512-bit buffers every third scan line. And every 8 pixel lines (1536 μs), the 384 bits of color attributes need to be loaded from the attribute memory. And of course the sprite states need to be updated, which may involve loading a new sprite from RAM or just incrementing a counter or two. Because these tasks are so slow (totaling maybe 2.46 μs/byte) they can all be done by a single state machine timeshared between them, maybe driven by an EPROM.
Since the 512 bits of the current pixel line are accessed in a strictly sequential shift-register fashion, and can be reloaded during the third scan line after scanout, even the old Signetics 2503 dual 512-bit PMOS shift register chip (DIP-8 or TO-99, 10 MHz typical, 4 MHz guaranteed, available 5 years earlier) might have been adequate. But I suspect much better chips were available by 01981. The 384 bits of color attributes are also accessed in a similar sequential fashion but at a much lower rate: they just need to be loaded into the color MUX inputs in 12-bit chunks at 617 kHz.
I count 27 chips on the motherboard in the photo at https://en.wikipedia.org/wiki/ZX_Spectrum#/media/File:ZXspec..., including what I think are 16 memory chips, so adding an extra chip like the 2503 might have increased the cost by 3%.
If you could stick the pattern table off in its own 4096-byte or 8192-byte RAM area instead of sharing the bus with the CPU, it would probably make the machine a bit faster too, but I suspect that wasn't practical because you couldn't get 4-bit-wide or 8-bit-wide RAM chips yet. I guess you could have used a 1-bit-wide pattern-table RAM chip and indexed it bit-serially, especially if your 512-bit pixel buffer was double-buffered (note that the 2503 was in fact two 512-bit buffers.)
But that's all with the benefit of hindsight, and without really knowing how tiny or slow their gate array was, or what alternatives there were for memory chips, or how much they cost. Undoubtedly at the time I would have been as blind as anybody else and, if I didn't make the same mistakes Sinclair did, I would have made others that would likely have been just as bad.
It was definitely not clear at the time which approaches were best. Could Sinclair have made a different computer? Absolutely, yes. Would it have had the same impact? Unknown.
From the perspective of a programmer who grew up in the UK in the 1980s the idea that the Spectrum was a failure riddled with design mistakes is just completely ahistorical. It was the dominant home computer, it was the dominant computer gaming platform, and it sparked the careers of thousands of British coders.
Could the NES run the first isometric games that showed up on the spectrum, for example? Or even some of the first primitive adventure games that used the built in graphics primitives to accompany the text with some graphics.
I'm really interested in the question of how to use an NES-like tile engine to draw a 3-D world without perspective, like Fairlight, Knight Lore, half of Quazatron, Highway Encounter, The Great Escape, or Ant Attack—though remember that the Spectrum also had first-person shooters with perspective, like DeathChase, Tau Ceti, or 3D Starstrike, which maybe would have been harder to incorporate. I think there are at least four reasonable answers and one total cheat.
1. You can take an "oblique" rather than "isometric" approach, like Target Renegade, Golden Axe, and many old technical drawings. Your X-Y plane is the screen, and Z goes off at a diagonal, typically 45°. So your X vector is just [1 0], your Y vector is [0 -1], and your Z vector is, say, [1 -1]. Single tiles of surfaces in the X-Y plane are just single tiles on the screen, and that's adequate for when an X-Y surface occults a surface in any other orientation, as long as it's at a tile boundary. A tile of an X-Z (horizontal) surface becomes two right triangles, one in the lower right of a screen tile and one in the upper left of the tile to its right. Similarly, a tile of a Y-Z (vertical but not screen-parallel) surface becomes two right triangles, one in the upper left of a screen tile and one in the lower right of the tile above it.
So, to make oblique renderings of axis-parallel arbitrary solid shapes out of a single X-Z texture, a single X-Y texture, and a single Y-Z texture (as in Ant Attack, though Ant Attack is isometric), all joined at tile boundaries, you need only 5 tiles: all X-Y, all X-Z, all Y-Z, and two that are half X-Z and half Y-Z. If you can have free-standing X-Z and Y-Z surfaces that occult an X-Y surface, like some of the walls in Highway Encounter, you need 4 more tiles: half X-Y and half either X-Z or Y-Z. An additional texture on the Y-Z plane costs you 3 additional tiles, or 5 with free-standing surfaces and arbitrary occultations. Similarly, non-straight edges, like in Where Time Stood Still, cost you an extra tile or two.
2. You can do "isometric" tiles with 45° angles instead of the correct 30° angles, so your X vector is [-1 -1], your Y vector is [0 -1] as before, and our Z vector is [1 -1]. This requires a larger variety of different tiles, has the same degree of geometric distortion, and might actually look worse: for a single texture on each plane you need not only the X-Y, Y-Z, X-Z, Y-Z/X-Z, and X-Z/Y-Z tiles from before, but also Y-Z\X-Z, X-Y/Y-Z, and 8 other combinations, 15 in all.
3. To do "isometric" with the correct 30° angles and only a little bit of geometric distortion, with your XYZ vectors being [-1 -½], [0 -1], and [1 -½], you can have four positions for a boundary within each tile: \ lower half, / lower half, \ upper half, and / upper half. This results in 27 tiles given one texture per plane, though not all will necessarily occur.
The tradeoff for all this complexity is that now you have four colors per tile instead of two, possibly higher resolution, and faster screen updates, which enable a scrolling full-screen viewport. Note that, of the 3-D Spectrum games I mentioned above, only DeathChase, Knight Lore, and Fairlight have a nearly-full-screen viewport onto the playfield, and the latter two achieve it by having an almost totally static background with a 1–2-second "loading" pause in between rooms. By contrast, a tilemap would allow you to rapidly scroll a large viewport, as long as it was done in multiples of the tile size.
4. Instead of precomposing a pattern table that draws a full-screen "isometric" world, you can draw with total freedom into a small viewport by writing into the pattern table. (On the NES the pattern table was typically in ROM, but of course that would be impossible for a cartridgeless machine like the Spectrum.) This allows you to not only do partial occlusion effects and subpixel scrolling like The Great Escape but even perspective rendering like Driller and 3D Starstrike.
It might enable you to do faster perspective rendering like Driller because you could get better fill rates on the interiors of stippled surfaces by reusing a stippled tile. (Maybe Driller could have managed more than 1fps.) And Driller wouldn't have had to be black and white: four colors in a tile means that a tile containing an edge between two checkerboard stipples can contain four colors, two per checkerboard.
5. If you can change the base address register for the pattern table, maybe you can do it three times per screen during the HBI, thus clawing back the ability to treat the tiled screen as a full-screen raw framebuffer, without having to dedicate RAM to it all the time or losing the ability to scroll it rapidly. This might even enable you to cut the tilemap entries from 8 bits to 7—all you need for ASCII anyway—using the 8th bit for a half-brightness flag or something.
Hindsight.
Don't know about the UK / rest of western world but in eastern europe the spectrum clones were running quite a bit of homemade software for scientific/engineering purposes.
Including graphs. No graphs with a NES tile/sprite system.
Overall I think that would have been better, not worse.
The NES sucked at that kind of thing, in part because the pattern tables were usually stored in cartridge ROMs, but mostly because outside Japan you weren't allowed to program it. People did do this kind of thing on the TI-99/4A, which had the sprite/tile graphics chip TI developed (the same one later used in the ColecoVision), with 256 8×8 redefinable tiles in RAM: http://www.academic.ro/TI994A/
If you really wanted to provide high-resolution data plotting capabilities on a TV you could provide an NES-style 3-bit horizontal scrolling delay register that shifted from 0 to 7 pixels off the start of the line, then reset it during each HBI. That would allow you to displace what would otherwise be a plain vertical line horizontally by 576 different pixel-accurate displacements, allowing you to plot 576 data points, albeit with only 8-bit precision and 90° rotated from the conventional representation.
A sprite engine that allowed you to reload the coordinates of the sprites every scan line would have the same merit, but be capable of plotting potentially several lines in different colors, and without disrupting the background.
If your sprites, unlike those actually present in hardware at the time, additionally had a variable X dimension, and repeated their pixels until reaching that X dimension, you could use each sprite to draw a stippled trapezoid at 2× or 3× the framebuffer vertical resolution by tweaking its X position and X dimension every scan line.
Hindsight.