New Rust runtime turned on. What next?
mail.mozilla.org
mail.mozilla.org
> Despite all these caveats I have a very strong sense that writing the
> runtime in Rust will go a long way to validate Rust in the domains it's
> aiming for: concurrent and systems programming. Even in the task
> scheduler, where there's quite a bit of unsafe code, the shared-nothing
> nature of unique types forces you to consciously break the type system
> to share memory, and that seems to go a long way to making parallel
> programming easier to reason about.
Yes, even just reading it seems much clearer since the mutability,
lifetime and ownership of each value and reference is spelled out, not
just "some Foo* that you have to remember special validity rules about".
It's noticeably easier to reason about. Really interesting!
See: Minecraft, Terraria, Magicka, AI War, and many more.
Go's GC on the other hand is non-generational, which means every GC pause must perform a full scan of all objects in the system.
Quoting from http://talks.golang.org/2012/splash.article (emphasis mine):
"To give the programmer this flexibility, Go must support what we call interior pointers to objects allocated in the heap. The X.buf field in the example above lives within the struct but it is legal to capture the address of this inner field, for instance to pass it to an I/O routine. In Java, as in many garbage-collected languages, it is not possible to construct an interior pointer like this, but in Go it is idiomatic. This design point affects which collection algorithms can be used, and may make them more difficult, but after careful thought we decided that it was necessary to allow interior pointers because of the benefits to the programmer and the ability to reduce pressure on the (perhaps harder to implement) collector."
In practice I think that the biggest problems are compiler related and affect both Go and Rust. Go and Rust both use conservative GC on the stack and as a result they take a lot of shortcuts. For example, LLVM (and GCC as far as I'm aware, though someone like DannyBee may correct me here) loses the distinction between integers and pointers by the time they get to the machine instruction level, which makes a lot of optimizations easier to implement but also making it impossible to generate precise stack maps. Fixing this would be a lot of hard work. It's probably easier in the Plan 9 compilers, of course, though I suspect it's still going to be a lot of hard work.
That said: Aren't the equivalent pointers in .Net _only_ allowed/usable if you pin your object? If you tell the GC explicitly 'please don't move that, I'm pointing to that thing here'?
That's hard to bolt on to Go, imho. For Go it seems to be implicitly allowed, in .Net you have to ask explicitly?
So it's basically as if all objects are pinned in Go. I don't expect that to change. There are probably tons of Go code out there which will break in response to a change.
I guess it will be pushed back to some mystical "2.0" release, along with Generics.
See https://groups.google.com/forum/?fromgroups#!topic/golang-de... for example.
http://www.youtube.com/watch?v=iMMbf6SRb9Q
http://www.youtube.com/watch?v=BMRlY9dFVLg
https://github.com/vova616/GarageEngine
So I say your statement is exagerrated, certainly stop-the-world garbage collectors isn't ideal for games, but it's not 'wholly unsuitable' as long as the gc sweeps aren't costly enough to impact consistent framerate. Had they been 'wholly unsuitable' then the XNA platform would have been dead upon arrival.
One thing that makes me excited about Rust is that GC is optional.
And if you're targeting 60fps, that's 16ms. Stop the world for 20ms, and you've missed 1 and 1/4 of a frame.
These are other requirements mean that most high level AAA games have to be written to be very highly performant, and in particular, to have reliably consistent latencies. Traditional Garbage Collected languages (Java, C#), whilst achieving high throughput, often struggle with predictable latencies as when GC kicks in, particularly in Stop-The-World collectors, you can get a 10-20ms pause which means you will drop a frame or two. This leads to stuttering which is undesirable. Even modern generational collectors still struggle not to have occasional bad pauses. If something like Azul's continuously compacting collector becomes viable for games, this might be solved, but until it does it is hard (but not impossible) to write high performance (soft) real time games in a GCed language.
The result is that most games are written in C++ (sometimes with a small embedded scripting language, mainly Lua, which uses GC) where memory management can be controlled. The downside of this is developer time - it takes large teams of developers lots of time to write games, because the code is largely written at a low level.
Rust is higher level than C++ and allows many useful and safe idioms, including a more functional style and better use of immutability. In general, like other higher level languages, Rust would allow a developer to be more productive than in C++, allowing games to be developed more quickly, with fewer developers. It does this whilst still allowing manual control over memory - some parts can be handed over to GC, whilst others can be carefully allocated to the stack or heap and managed manually. This allows predictable allocation and cleanup overhead, and therefore more deterministic frame times.
As a games developer, I am extremely interested in Rust as a potential game development language.
Disclaimer: I haven't actually written any Rust yet, only looked at code and thought about the potential. I've spent 5 years doing AAA console development in C++.
I have tried Unity (especially since I could leverage my C# experience) but as a developer I like the 'code-first' approach rather than being tied down in a graphical tool with scripting capabilities (especially for 2D games?).
Using Lua inside a development environment/engine could be a good start - there's a couple of Lua-based development frameworks that might be worth checking out. Start with 2D games as they're much simpler to get going with and get used to how a game engine will look.
After that, it depends what you want to do. If you mostly want to create some interesting games and care more about the gameplay design side, using a pre-existing engine/framework is best. If you are interested in the development and high-performance side of things as I am, then learn C (not C++) and OpenGL. This is not as hard as it might seem. The best OpenGL tutorial I have found is http://duriansoftware.com/joe/An-intro-to-modern-OpenGL.-Cha... - this will get you writing graphics code from scratch and starting to understand what is going on. You don't need to become a graphics expert, but actually understanding how this side of things works will be invaluable.
If you have questions, my email is in my profile and I can be reached there. Happy to answer individual queries. I'm not currently working in the Games Industry (day job is Scala in Finance) but my side project of the last 2 years is a cross platform engine + game written in C and Lua.
Note that Lua 5.1 and later use an incremental GC, which is friendlier for latency-sensitive workloads.
On non-custom hardware like x86 and x86_64 Azul's continuous GC as of when I last checked requires a software write barrier, which is pretty expensive, a 20% or greater penalty.
I think you're much more likely to see 51% of new game code written in C# for Unity. I think the median game budget is shrinking. More and more games are being made as small projects by small teams. Steam greenlight, kickstarter, humble bundle, and other avenues are enabling curation and funding of smaller games. Unity is far more suited for this kind of development than C++. If you look at smaller studios' job listings, a lot of them prefer experience with Unity. The trend is in full swing at this point.
I have no sources to site for the above information. Take it with a grain of salt.
C++ makes it way too easy to corrupt random sections of memory.. which leads to random CTDs. Rust was designed from the ground up to prevent these dreadful bugs.
AAA games are overwhelmingly written by relatively inexperienced, overworked cowboy coders on a tight schedule who are expected to write optimized code. Expecting C++ code without memory corruption bugs in that situation is simply unrealistic.
As a customer I truly hope the industry adopts Rust or another language where memory corruption is impossible or only possible if you explicitly ask for it. I am so sick of CTDs.
I've never used static analysis tools for C++, but from what I've heard, the reason they need to be so advanced is because C++ is so difficult to analyse. Rust, on the other hand, with it's strong, static type system, may not even require extra static analysis tools to be used effectively.
If Rust succeeds as a language suitable for games and gains traction there (and that is of course an if), I think the truth will be in the middle: there will be a huge amount of code still written in C++, and that will continue to be maintained and work. But new code might be written in Rust. Rust is designed to integrate well with C and C++ code, so mixed projects are quite feasible: in fact, both rustc (because of LLVM) and Servo (because of SpiderMonkey) are such mixed projects.
> engines might always be done in C/C++ simply because it
> gives them the ability to easily make bindings to the
> majority of languages
As for "further development", what would you like to see? Currently the most prominent capabilities of the runtime are the lightweight task system and garbage collection. It might be theoretically possible to allow uses of the task system to gracefully degrade to using system threads, and likewise it might be possible for uses of the GC to degrade to refcounting (without cycle collection).
As someone else mentioned, C# is a big deal now and comes with a degree of safety over C++. I don't expect to see Rust in the pole position on the whole, but it could easily displace C++ at Id Software since Carmack has been shopping for an alternative, and elsewhere (especially if he runs with it first.)
But I think C++'s share is losing ground, and Unity is an impressive platform. The asm.js buzz seems to whisper promises about ES and browsers becoming a serious AAA gaming platform. If FFOS gets anywhere it may supply the necessary voltage. So the lay of the land is anyone's guess, but all the variables eat into C++'s % share even if only by expanding the market around it.
* Games require portability. Not many platforms will have a rust compiler (Not to mention that current C++ compilers are very mature and highly optimizing).
* Current C++ libraries. They are probably heavily templates-based and will be difficult to port.
Rust uses LLVM as the backend, so any platform that Clang supports, Rust can too. (And also, it has the optimisations built in.)
In fact, there's already support in the compiler for x86, x86-64, arm, and mips. (I'm not sure if mips actually works, but arm definitely does.)
also, even before that there were people using "non-standard" toochains e.g. http://en.wikipedia.org/wiki/Game_Oriented_Assembly_Lisp
And indeed it's possible to have an alternate toolchain, but lots of work that is best avoided most of the time.
Very rare thing. Ok maybe not in language design. But rare otherwise.
For one, it doesn't really make sense to talk about the CLR when you talk about bootstrapping C#. My project (the Roslyn C# compiler) is a 100% C# implementation of the C# compiler. One thing to keep in mind is that we don't target the CLR.
The C# compiler is not a compiler from C# to the CLR, it's a compiler from C# to the CIL (Common Intermediate Language). It's completely reasonable to imagine a machine which runs CIL in hardware instead of in a VM. In this case the answer to the question of whether or not the C# compiler is bootstrapped is, "Yes. Completely."
Moreover, it wouldn't make sense to ask whether or not the CLR is self-hosted -- it's called the "Common Language Runtime" for a reason. It's a language-independent virtual machine. In this sense, implementing it in C++ makes just as much sense as implementing it in machine code.
Now, my argument here was meant to be without loss of generality. To ask whether or not a language is bootstrapped shouldn't really depend on a runtime, since any language which requires a runtime cannot, by definition, have the runtime written in said language. In this sense, we should only ask whether or not the compiler is bootstrapped, not the entire environment.
Rust is in almost the same boat since they are using the LLVM, but the LLVM differs from the JVM and CLR since it can generate stuff to run on hardware without a shim.
> actually those things exist , but they don't
The linear logic of J.-Y. Girard suggests a new type system for functional languages, one which supports operations that ``change the world''. Values belonging to a linear type must be used exactly once: like the world, they cannot be duplicated or destroyed. Such values require no reference counting or garbage collection, and safely admit destructive array update
http://homepages.inf.ed.ac.uk/wadler/topics/linear-logic.htm...
An interesting bit of trivia about linear types is they are in some sense the closest thing to programming a quantum computer this side of qubits (where no cloning dictates qubit variables can only be used once in a function term).
(The original message is in Markdown; I just put it in a Gist where it would be rendered as such.)
Work (task) stealing is very compelling, and a little paradigm shift is no bad thing. If Rust or any new systems language stands a chance, it should aim high and not too close to the past.
It's rudimentary, but it does compile and it does run, and I'm not aware of any particular obstacles to a more featureful kernel.
For an example, see the code at https://github.com/doublec/rust-from-c-example , which is fully runtimeless.
(e.g. https://github.com/huonw/rust-malloc/blob/master/zero.rs)
I thought this sort of thing was usually handled by the OS. Can you even get raw sockets on most systems? Or by "implemented TCP", do they mean they can give you back a Unix TCP/UDP socket?
Go kinda ruined other languages for me with multiple return values, its something I miss when working in every other language now. And reading through the docs I keep hoping I would stumble across that, though that would create a lot of problems when interfacing with C. Type inference is a huge win, standard libraries looks solid, love the potential around marking variables as mutable, and any language that has no null values makes me happy.
Overall, I wish I had more time too.
It will, because the semicolon in Rust actually has a semantics impact: `a` has type T(a) but `a;` has type Unit (~void).
> Go kinda ruined other languages for me with multiple return values
That's sad, because Go has one of the worst MRV implementations out there: it's a special case of the language itself.
In most languages with MRV — and that includes Rust, but also MLs, Haskell, Erlang, Python or Ruby — MRV is simply a natural consequence of being able to unpack or pattern match containers (tuples and/or lists depending on the typing discipline).
Anyway Rust has multiple return values, don't worry about that.
Also, I've never come across a situation with too few/extra semicolons causing any kind of logic errors. The compiler will complain at you if you get it wrong.
Also, the type checker ensures that you'll hear about it if you lack a semicolon and emit a value other than unit from a block, without then using it.
addsub = function(a, b) return a + b, a - b end
a, b = addsub(32, 44)
In contrast I think at least in Python the tuple based solution would be horribly inefficient because it allocates/deallocates a new object every time just to pass values. I don't now if the Rust compiler is smart enough to optimize the tuple creation away.. but I doubt it.
> I don't now if the Rust compiler is smart enough to
> optimize the tuple creation away
fn addsub(a: int, b: int) -> (int, int) {
(a + b, a - b)
}
...
let (a, b) = addsub(32, 44); let a = 1;
let b = 9;
let (b, a) = (a, b);
printf!("a: %i, b: %i", a, b); // a: 9, b: 1
let x = (1, 2);
let (_, y) = x; // the underscore is the pattern for "ignore this"
printf!("y: %i", y); // y: 2 b, a = a, b
Grabbing only one value looks like this
_, y = returnsTwoValues() -- grab only the second
y = returnsTwoValues() -- grab only the first a,_ = two_values()
_,b = two_values()The large amount and rapidity of change makes it difficult to use it seriously. It's nowhere near as stable as other newer languages like Go and Scala are, for instance.
Some experimental programs I wrote a mere 8 months ago are now basically unusable with recent versions of Rust due to language, syntax and standard library changes.
Unless you can constantly track Rust's development on a daily basis, and update your code accordingly, I'd be very hesitant to suggest using it for anything but throw-away code at this point.
It reminds me of Go a couple of years ago, but that had "go fix" which would mostly update your source files to the newest revisions. That made early adoption a lot nicer. I'd love to see a "rust fix"
Is that done by having a single task queue with multiple schedulers, or through work-stealing by schedulers with no ready tasks in their queue?
Would this open the possibility of configuring schedulers (including individually)? E.g. ensuring a given task stays pinned on a specific scheduler, and said scheduler accepts no more task, that kind of things?
Tasks can be 'pinned' to their own scheduler (i.e. thread) with `spawn_sched(SingleThreaded)`, and this is very important for tasks that call foreign code that blocks.
That's about the extent of the configurability at the moment, but I anticipate at least one other 'mode' in the future for coping with blocking tasks that don't want to be pinned to a specific thread.
I've been assured that it's the plan. I have no idea how much of it works right now though. I've only done one build with the new rt and it doesn't involve tasks.
Excellent [twirls mustache]
> I have no idea how much of it works right now though.
Yeah I don't expect it to work at this point, but knowing it's one of the end-goals is good.
Right now it's the former, but Aaron Todd has a pull request to switch it to the latter.