Mill CPU Inter-Process Communication
millcomputing.com
millcomputing.com
They don't provide nearly enough ways to transitively grant permissions. Using the mechanisms discussed in the talk, it doesn't seem like you can implement a simple asynchronous queue of units of work to perform, each having their own permissions. The belt architecture encourages these sorts of second-class mechanisms that have to be used in a rigid way, because the details can be hidden in the belt and not be exposed architecturally.
Unless there's something else not mentioned in the talk, it seems like you still need to trust the OS, because when the OS is asked to allocate a page for a spillet there is nothing stopping it from creating a virtual alias of that page elsewhere and allowing another thread to read its data.
The mechanism to support fork() is a total kludge. Why have a single address space if you're just going to add segmentation in such an ad-hoc way for a single use case? Just run the original binary in emulation until exec() or something like that.
transitive permissions are capabilities.
while i'm sympathetic to the lack of market appeal to a capability based system, doesn't it seem like you could implement posix on top of one by compromising it? fd transfer over unix domain is already halfway there.
seems like a better alternative.
It's not, it's "copy on reference".
The guys are Cambridge have running caps systems that store the extra info in outboard data structures. We judge that the overhead is too great for commercial success. Customers buy benchmarks, and there are no security benchmarks.
'Tis true 'tis, 'tis pity. 'Tis pity 'tis, 'tis true.
That being said I'm still not sure how you can get all of fork's semantics out of this mechanism...
(For that matter various sneaky VM aliasing tricks ["magic" circular buffers mapped twice in a row so that any size and position block is contiguous, same file mapped in multiple processes with different (non-contiguous) memory mappings] are going to fail miserably with the virtual-addressed cache. It may well be worth not being able to do those things for the benefit of moving the TLBs, but it also flies in the face of "will run current software.")
> "magic" circular buffers mapped twice in a row so that any size and position block is contiguous
Yes this exact case is one of those that won't work. There are a handful of other things that don't work, like people who roll their own coroutines using assembler. We promise 'don't rewrite, just recompile' but your codebase has to be 64-bit clean and not use any assembly language for any other ISA and not make other unportable assumptions about the hardware.
I can't think of any software that actually uses these circular buffers that wouldn't be worth adapting to the Mill rather than adapting the Mill to them; can you? ;)
For that matter have you guys touched on how stack unwinding is going to be handled yet? How do I implement exception handling, longjmp, CL's non-local GOTO, etc?
> They don't provide nearly enough ways to transitively grant permissions.
Portals are synchronous and transient permissions can only be used for the duration of the call and by the same thread. Asynchronous isn't so 'simple' because its about lifetime. With the synchronous portal the caller knows that the callee cannot retain any access to the buffers that were passed, and can reuse them safely. If those buffers were put on an asynchronous queue, when would the memory be safely reused and when would the owner know that? If you want asynchronous queues, you either have to have a buffered model like Unix pipes etc or you have to have a global hardware-implemented GC that somehow spans turfs and becomes part of the Trusted Computing Base (TCB) (shudder).
> it seems like you still need to trust the OS, because when the OS is asked to allocate a page for a spillet
Well there's plenty not mentioned in the talk and you touch on one aspect :) When a spillet overflows the extension cannot be in the reserved space, so space must be 'carved out' of the part of the address space where programs also have their needs carved out. Someone has to do the carving, whether its for spillets or for programs, and that someone has to be trusted. The problem isn't aliasing (we're Single Address Space), its that they can simply give as many permissions to it to as many turfs as they choose. There is a possibility that the carver is in the BIOS, but any which way there has to be a turf that can do this. This turf is obviously part of the TCB.
> Why have a single address space if you're just going to add segmentation in such an ad-hoc way for a single use case?
Unfortunately there isn't much market for general purpose CPUs if they are fundamentally unable to run Linux ;)
As you will almost certainly be running a Unix, and as you almost certainly will be using libraries that may fork, then all your normal heap and data stack pointers are going to be Local. Shared mmap pointers will be Global, as will your code. We can hope there is a flag this Linux lets you set that says that you forego the ability to fork() and in return all your pointers can be Global, because the Local puts some constraints on your address space use which become clearer if I explain them:
The local bit works like this:
If a pointer has the bit set, then before use it is mangled with a special register called the Local Space register. The hardware does this every time it uses a pointer. Each turf has its own Local Space value, and you can think of it as a simple offset, so if the address is 0x1...10 and the local space is 2 then the effective address in the global space is 0x1...100.
Now 64-bit adds are relatively slow for this use-case because we really want the effective address asap in the FU so a probable implementation of the Local Space is XORing it into the pointer instead of adding.
When a process forks the OS has to find a position in the global address space where the allocated ranges used by the program are free for the child and then set the new Local Space register for the child appropriately.
> Just run the original binary in emulation until exec() or something like that.
All modern OS don't actually COW until there's a page fault, and I'd expect them to use that trick on the Mill too. So the local bit makes it possible to fork(), but the hole-searching is lazy and only happens if you actually use it.
"Since Monte permits mutable state, one author’s code’s behavior could be affected by another author’s code running further up the frame stack. Stale frames make comprehension of code much harder as a result."
I wish that we were having more conversations and classes like [1] to get knowledge of KeyKOS and other capability-safe kernels out to CPU makers. We have ideas to share!
[0] http://monte.readthedocs.io/en/latest/semantics.html#scope-i...
[1] https://pdos.csail.mit.edu/6.828/2009/lec/l-microkernel.html
The Mill isn't a Capability-based Addressing machine, but it is very much a machine that hopes to push normal programs and operating systems in the Capability Object Model direction with first-class support.
Now you are using a bugmenot login so I have no idea who you are, but if you're a caps vet we'd love to know your thoughts :)
My thoughts are mostly that I don't want to wait a decade for a Mill. Find me at SPLASH/OCAP if you want to chat.
The difference between the two models is visible when you pass a graph structure across a protection boundary. With caps is is easy to pass the whole graph, and hard to pass only one node. With grants it is vice versa.
Fork is a kludge. It also happened to be easy to implement on the hardware available at the time and we've been stuck with it ever since. So I happily forgive the mill team that their fork() implementation looks like it is a total kludge, it would be highly surprising if it were not.
The Mill is hardware and architecture, not policy. If you want to use such an OS then you are free to do so. The Mill is designed to efficiently support micro-kernel OSs. Note: micro-kernel, not no-kernel. There always will be a Resource Service that owns the machine. It will be a couple hundred LOC, small enough to be correct by eyeball or proof. Contrast your choice of monolith.
The OS is not involved in allocating spillets. Spillet space is a large statically-allocated matrix in the address space. It is not allocated in memory, only in the address space. As soon as you allocate a turf id and a thread id you have implicitly allocated a spillet. Only on spillet overflow is allocation necessary. Whether allocating turf or thread ids requires OS involvement depends on the policies and models chosen by the OS designer.
When first created the spillet data lives only in backless cache - no memory is allocated. Only if the spillet lives long enough to get evicted from cache is actual memory allocated, using the Backless Memory mechanism described in our Memory talk. The root spillets of apps will live that long; transient spillets from portal calls will likely live only in cache. Consequently truly secire IPC/RPC using Mill portals has overhead, both app and system combined, of the same magnitude of an ordinary function call.
> They don't provide nearly enough ways to transitively grant permissions. Using the mechanisms discussed in the talk, it doesn't seem like you can implement a simple asynchronous queue of units of work to perform, each having their own permissions.
There is a "session" notion that addresses such things. Unfortunately the talks are far enough into details that they must contain background and introduction slides for the viewers who have not already done (and retailed) all the other talks. This limits the amount of new material that can be covered in a single talk, and sessions didn't make the cut this time. We'll get to them.
> The mechanism to support fork() is a total kludge.
Agreed; there seems to be a Law of Conservation of Kludgery. We had as a minimum requirement that the architecture must support Unix shell. The only real problem is fork(). Would that we could issue an edict banning it.
http://chipdesignmag.com/images/idesign/fpga/figure1.jpg
And such don't come cheap.
I was involved in something similar around 10 years ago where we were working on a revolutionary EDA suite for analog and mixed signal circuit design. The owner was quite technically competent but kept upping the ante and theatrics to the point that no customers or suitors took the company seriously, only the desperate employees. They never closed sales nor sold the IP.
I advise extreme caution in dealing with the business side of this.
And yes, Mill Computing, Inc. is not how real companies are run. Is that a bug or a feature?
They haven’t produced an FPGA proof of concept, after claiming they would have one ready last year. They now say they need investors to finish it, yet they previously claimed to not even be looking for funding.
They claim to have angel investors, but they are all secret ones. Of course it’s an investors right to stay private, but the reason you often see investors and companies shouting from roof tops is because the funding event itself can help a company. Publicizing it generates PR, gives the company credibility in dealing with other companies, and is a signal that can generate demand for more investors.
Even putting that aside, the biggest issue is they haven’t made a compelling case for how their ideas will outperform existing CPUs in practical usage scenarios. Yes a running FPGA would be nice, that’s not the only way to show potential.
They could do quantitative analysis, modeling, or start adding a lot more detail to their talks and papers (which tend to sound about as deep as you get in an undergrad architecture classroom), and argue very specifically and comparatively against today’s standards, even for just a few key scenarios.
Maybe they believe even those ways would still have capital/labor/opportunity costs that are prohibitive or for a startup? Another option could be small meetings with a few well respected hardware architects, who will have the best chance of understanding the potential value. Once convinced, they will probably be glad to write about it or just provide a reference, which will make funding, partnerships, hiring, etc all easier.
I dislike being critical of people swinging for the fences, because it’s what many of us here are trying to do, and it’s important that people keep doing it. However in this case it’s not just about long odds. Because of the reasons above and a few other details, things just don’t add up. I don’t believe the FPGA will ever demonstrate anything compelling, and don’t think any investor in their own backyard on sand hill road will bite.
It’s all conjecture of course, I’d be happy to be proven wrong.
Being a software guy I can't say much about the viability. Watching Ivan's lectures and thinking it over however tickles the same part of my brain that enjoys learning a new programming language. It is just fun to see how some problem could be solved differently.
Most of what you'd like to see are things we'd like to see too. At the beginning we decided to bootstrap rather than follow the usual funding model, at least to the point at which we could demonstrate what we had to people who would understand it in detail. We choose bootstrap in large part because most of us were old enough to have had actual experience with other business models. Yes, it has taken far longer to get this far than we wanted, but we have gotten this far.
About evaluation: it has been our experience that the more senior/skilled a hardware (and software) guy is the more they fall in love with the Mill. You don't hear much of that - we want the tech to be judged on its merits, not on some luminariy's say-so. And of course those senior guys tend to work for potential competitors and don't want to say much publicly.
But you are right: the proof will be running code, and we're starting to do that. We'll be doing more talks like the switches talk, with actual code comparisons. Eventually we will put our tool chain and sim on the cloud for you to play with. Patience, waiting is.
edit: i thought they managed to do all of this without segments, but at the end we hear about a special local segment with offset addressing apparently introduced just to handle children of fork()..i lost how cow can be expressed losslessly as local/global
redit: question around 1:00:00 explicitly ask this question, and he said, erroneously i think that while semantically similar this is the first time a direct hardware implementation of a call gate has been proposed
He also doesn't mention the word 'capability' anywhere either - this is all 1980s stuff
the other thing that struck me as really strange was a bit in the question period where he says that 'smart devices' (DMA?) are different from simple devices (pio?) in that they are expected to be first-class multiprocessing citizens. doesn't that imply that high-performance peripherals need to be specially designed for the Mill? (likely with a Mill core attached)
As for peripherals, I presume that would have to be handled in the drivers.
Of course, modern peripherals don't look like that, so there will be adaptors. IBM 360 channels and CDC6600 PPs also haven't been architecturally revisited in a while.
I just have a sinking feeling about hoping a compiler can correctly identify and track all pointers to know how to flag them. The "pointer is a native-word-sized int" assumption may be so ingrained -- from compilers to stdlibs to the wide variety and age of programs -- that it will be nigh impossible to rid existing codebases of it completely.
But I'm not a compiler guy (or hardware, or assembly, or C for that matter) so I could be quite mistaken. Perhaps it's enough to fix the compiler and make it capable of emitting warnings/errors when it detects a violation.
As far as the talk itself goes, I'm a little sad that there was so little new information though I understand that we're quite deep in the technical details and there's a lot of prerequisite background that you can't reasonably expect from a random tech audience. If there are more than a few more talks you might need to reevaluate this method altogether and use a different format.
I'm very glad that you've decided to change the wording to refer to it as an "SSA machine" as opposed to "belt". I think many more people are familiar with SSA or can be convinced that it works ("your current compiler uses it right now" probably helps) by describing it as "SSA where you can only reference the last N results" as opposed to building a whole model based on a "conceptual giant shift register" from before. I've been following the Mill talks since the first few videos and recently I wonder if even the asm programming model should be writing raw SSA instead of belt numbers, especially since genasm assumes an infinite belt anyways.
How does the compiler implement PHIs connecting expressions with different latencies? Let's say I have:
`if (cond) { x = a + b; } else { x = a * b; }`
The MUL is may take a bit longer than the ADD, but the user needs to accept the argument at a given belt position. How do you avoid having the pay the latency cost for the MUL if `cond` is usually true?
Based on my limited understanding of the Mill I'd say (1) (simple_ PHIs are implemented by the pick-phase, (2) it's an exposed pipeline with known latencies, so you can't really have the "I want to avoid the longer latency inside a single basic block", and it's probably not worth doing that for a single mult vs. a single add, (3) for cases where it matters because the latency difference is much better, you'd use two basic blocks, and rearrange belt positions at the end.
If I remember it right, ADD and MUL take the same time, but when instructions differ in time the compiler is expected to reorder them. If it can't reorder, than yes, it has to wait.
The conversion does increase the latency of getting the value of x. If there's nothing else to do then the tool chain will insert explicit nops to wait for the expression. The same stalls will exist on other architectures for the same code, just not visibly in the code. It happens that making the nops explicit is faster than a stall; you can idle through a nop with no added overhead, but you can't restart a stall instantaneously.
Also, is the thread talk close at hand? I feel I learned less from this talk than usual; most of the material was already discussed in the security talk.
Intel MPX has a similar protection model, and that introduces a lot of overhead (of course it is bolted onto an existing arch and it wasn't a high priority feature).
Imagine you load a 7MP image which takes, say, 21MB of RAM. That would be 5184 4K pages in a classic TLB. In the Mill's PLB, that whole part of the address space can be in a single protection entry.
Then, there's a big difference from how things can be organised in software vs hardware. The hardware PLB has some number of entries, and it will check all those entries in parallel.
I understand that Itanium was forced into a low frequency, high power L1 cache by being at a similar in order statically scheduled design point, where you need loads to be fast in order for a compiler to be able to come up with a reasonable static schedule. Unless there's some really nice idea out there that's radically different from TLBs as they are today, I bet PLBs will be the major limiter of performance for general purpose code.
It is just a normal bounds compare to see if each entry covers the access, and the PLB has as long as the top-level cache access takes to do the checks.
So the PLB misses far far less often than a conventional TLB.
Like any cache, the optimal PLB size is determined by the working set. In the typical code we are seeing, the program has a couple of open files, half a dozen mmaps where the heap grew itself, and portal blocks for assorted libraries. The working sets are much smaller than a conventional TLB, and with SAS we have several cycles available in parallel with the caches.
The upshot is that a PLB can be large, cool, and slow. As for the range compares, the PLB permits the same sore of address sub-setting as is done in mixed size TLBs. Think about how many bits in the typical address range differ between lower and upper bounds.
Fine granularity is expensive, which is why the monoliths have one process-granularity. If you have 100,000 graph nodes and want to pass all of them except this one then you will have to pay for the privilege, in any protection model. The Mill lets you pay less.