Mac mini Late 2018 Teardown

Apple theoretically could have a slot for NVMe separate from the system drive, but there is no industry standard for "card with NAND on it that the T2 chip understands".

Yes, but you can achieve these things without a hardware flash controller. Old embedded devices did exactly this: they used the CPU as the flash controller.

It's simplest to understand if you look at a much older technology: floppy disk drives. There was a technology war in the 80s, between systems that had "direct-addressable" floppy disk drives, and systems that had "managed" floppy disk drives. The former (like the Apple II, and the IBM PC) had to use the CPU as a floppy drive controller, encoding individual commands to the drive's motor, managing the timings of these commands using the CPU's own clock. The latter (like the Commodore 64) could just send async commands, because there was another whole CPU inside the floppy disk drive itself, decoding those commands and managing the motor. That CPU was the "floppy disk controller."

Same was true of hard disks, before the ATA standard. The CPU needed to control the motor. This is why tools that could "low-level format" a hard disk existed: the hard disk's motor was directly-addressable to the CPU, so the CPU could literally position the disk head anywhere and write anything. (And a bad POKE to the disk's IO port could literally cause the disk's head to poke a hole in its platters!)

And this is exactly why, for the machines in the Apple and IBM lineages, the invention of a generic Disk Operating System was so foundational to making these systems easy to develop for. (A DOS is not an "Operating System" in the modern sense—though it does often contain an OS—but rather, a DOS, in the strictest sense, is a library kernel that sits beside your program, and provides your program with a software-emulated disk controller to speak to directly-addressable disk motors!)

When NAND came about, and was first used in embedded devices, it was also in directly-addressable form, with the CPU and RTOS using a set of "software-emulated flash controller" routines to speak to it. Actual hardware flash-controller chips came later!

And so, I hope you understand now that what you've said here is completely untrue:

> They only way to get anything close to what you've made up is to not use PCI-E, either.

You could put a NAND flash chip directly behind a PCIe controller, stick it into a PCIe slot, and then have the OS—using a "software-emulated flash controller"—create commands, and have the CPU wrap them into PCIe packets and push them through the PCH over to that board, whereupon the PCIe packet is unwrapped to deliver exactly the same NAND commands—with the same timings†—that you'd get by having the NAND wired directly to the CPU over e.g. the SPI bus.

† You'd think PCIe has delays, but that's only if the PCIe bus is oversubscribed using something like a PCIe splitter daughterboard. When using PCIe normally, despite the packet-oriented nature, the signalling is still synchronous between the board and the CPU. This is achievable because PCIe doesn't use any routing that can impose latency, but instead, the PCIe host controller merely acts as a network switch, where established paths are electrically continuous. If your NAND chip has four input terminals, and the processor has given it four dedicated PCIe lanes, then sending PCIe packets over those lanes (assuming a high-enough PCIe clock frequency) is exactly equivalent to signalling directly to those input terminals. (And, in the more complex case, if your NAND has four input terminals, and you've given it one 4x PCIe lane, then (assuming a high-enough PCIe base clock frequency) the PCIe controller can still internally ser/des and get your packets there in time.) If this didn't work, then you wouldn't have things like Infiniband RDMA working over PCIe. [Also, this guarantee of electrical continuity is the big difference between something being attached by PCIe, and attached by Thunderbolt. The Thunderbolt controller is a router, and so a driver for a Thunderbolt peripheral can't be coded with precise timings in mind.]

Also, I know this works, because devices do this. Ever had a PCIe board with firmware that doesn't get updated ephemerally at boot by the OS, but instead can be reflashed permanently, like a BIOS? (Network cards will often have this architecture, because of the requirement for PXE boot.) Well, if the firmware can be permanently updated, then that firmware is likely held on NAND that's direct-addressable over the PCIe bus (with some board-internal routing given the memory region accessed, that it told the CPU about as part of its ACPI SSDT.) That's why firmware flashers need to be these bulky custom binaries: they've got (simplistic) software-emulated flash controller libraries embedded in them, so that they can generate the packets to feed, over the PCIe bus(!), to the board's NAND. (You think they're using EEPROMs—i.e. how the BIOS calls these "option ROMs"? Well, NAND flash is a type of EEPROM. And the other, older type is way too low-capacity for holding modern firmware images.)

(The motherboard's BIOS firmware is, these days, NAND flash as well; and "BIOS flasher" programs contain software-emulated NAND flash controller libraries, too. But the BIOS isn't usually sitting on the PCIe bus [but rather is just mapped into some memory region, i.e. is directly attached to the PCH] so this example—while more obvious—isn't as powerful.)

> And since they are using an off the shelf CPU, it's using PCI-E not something custom. Which means a flash controller is on the other side of the PCI-E bus.

Nope. Remember that the NAND in the Mac Mini isn't on any bus that the CPU can speak to. The NAND in the Mac Mini is a peripheral of the T2, not a peripheral of the CPU itself. And the T2 has whatever peripheral interfaces Apple wants it to have. There's no reason for the T2 to speak PCIe to the NAND; the NAND isn't even on a shared bus with anything else. It's just right there, soldered beside the T2 chip on the board, output terminals (likely) wired to input terminals.

The flash controller is part of the T2. (If you think of the T2 as a generic CPU, then it's a lot like the Commodore 1541's 6502, in that it would be running an RTOS and that RTOS would have "software-emulated flash controller" routines. It's probably not, though; it probably has a hard flash controller as part of its package. That flash controller has just been customized to be wired to the Secure Enclave part of the T2 package, such that reads and writes going through the T2's flash controller "flow through" the T2's Secure Enclave's encryption, without the T2's generic CPU needing to be involved at all.)

If you want, you can think of the entire T2 and NAND part of the board, as an NVMe device. This actually makes a fair amount of sense, since the T2 is probably on the PCIe bus, and the CPU is probably talking NVMe-over-PCIe to the T2 when it wants to write to the disk. It's just that the CPU is also talking a bunch of other protocols to the T2 as well, when it wants other things.

---

The best way to understand the Mac Mini's CPU's relationship with the T2, is that it's exactly like having a diskless Intel PC, with an iPhone plugged into it over a Thunderbolt cable. (Except, without the aforementioned Thunderbolt latency.) This iPhone has been modified to expose itself to the PCIe bus as, among other things, an NVMe device. When the host writes to this "storage device", it's really just talking to the iPhone's CPU, which can then do whatever it wants with that received command. It probably turns it into a write to its own storage, after doing a bunch of other stuff. Neither PCIe nor [standard] NVMe is involved in that side of the storage process, because the iPhone is not a user-expandable computer that needs to speak standard protocols to its own hardwired chips.

In this analogy-that's-not-really-an-analogy-at-all, the "iPhone's CPU" is the T2, and all the iPhone's chips have been plopped onto a motherboard shared with the Intel PC. However, these chips are still isolated from the Intel PC in exactly the same way they would be if it was just an iPhone-over-Thunderbolt. It's another whole computer in your computer, just like the article says. And the NAND is part of that computer.