Grappling with Go
blog.ntpsec.org
blog.ntpsec.org
Only for those that never heard of Mesa/Cedar, Algol-68, CPL, Ada, Modula-2+, Modula-3, Oberon, Oberon-2, Active Oberon, Component Pascal, Oberon-07, C+@, Sing#, System C#, D.
Was also the source for many of Plan 9 UI ideas.
What is most successful operating system created in Go or Rust?
For Go, I don't know any (but might be wrong, googling did not turn up anything).
My point is that commercial failures of specific OSes have more to do with politics and economic games than capabilities of programming languages.
As the OP was kind of insinuating just because many are too young to remember the OSes written in those languages, they should be ignored.
Yet, so far none of the OSes being written in Rust or possibly Go, have achieved the same amount of users those old forgotten systems had.
We can only improve our future by learning the lessons from the past.
Different concerns have different emphases in each fad, but the systems built have structural isomorphisms with what came before - they could hardly avoid it, being built to solve the same problems - and the older experience is lost, because people focus on the surface, rather than the larger shape. Older experience is dismissed because it was in an outdated language, despite it containing lessons that will continue to be applicable.
Everyone is myopic. It's like politics; the appearance of things is what matters. It's like fashion; it's the intersection of the ideas associated with the surface structure of your output that define its perception. No wonder older developers tend to leave this industry.
The trade-offs come around and around, the lessons are not as inapplicable as you think. We used to struggle to run programs on small machines; those struggles came back again with mobile, and come back again in a slightly different guise in clusters (where the program is too big to fit on any one machine).
They recur on different scales, too. Just the other day I was looking at dependency injection. Dependency injection solves the problem of coordinating dynamic program composition; the reason the program needs to be composed is because it was decomposed for testing. But this is just another iteration of the same pattern come around again. It's been done before in a large variety of ways; sometimes through ambient configuration (global variables), sometimes with patched jump tables (dynamic linking), sometimes statically (module instantiation). The emphasis is different - testing - but the solution is not structurally novel in the least.
Hah! Reasons for not learning from history:
* history is hidden behind NDAs, bad licences and patents
* history is hard to access due to its environment (printed on out-of-print books, stored on tapes, HDD and other forms of storage in someone's basement)
* history is written in a programming language nobody knows anymore
* history is written in an incompatible programming language, one for which there are no stable bindings to your favorite language (or for which there aren't any bindings, period)... just think how many times the String class or data type has been written and rewritten just in something like C, let alone the copies in Java, PHP, Ruby, Javascript, Go, Rust, etc., etc.
There's a ton of (somewhat) valid reasons history is forgotten and most of them aren't "history was bad, let's just forget it".
For example, I have collected the majority of Xerox and DEC research papers that I could get hold of.
Do you make that statement with the experience to back it up? Or do you make it because "if it were useful someone would have done it"? What if everyone is thinking the same thing as you are?
Beyond what other people have said, we are also a relatively young industry. There aren't that many people with 30, 40 years experience in the field. When I got my CS degree, 13 years ago now, there was no such thing as a "software engineering" degree, and CS was mostly focused on CS rather than the sort of thing you're talking about.
I also wonder how much collective knowledge we throw away with pushing developers into management. Or the outsourcing in the late 90's and early 00's. Or ageism. We are purposefully driving people with knowledge out of the industry.
That is country specific.
I took my degree around 20 ago and is it called "Degree in Software Engineering", certified by our Engineering Society.
http://www.fct.unl.pt/ensino/curso/mestrado-integrado-em-eng...
Created in 1977.
No, I wasn't. It was an honest question.
The only OSes I've used or had any experience with are Dos/Windows, MacOS, iOS, various Unix/Linuxes and the reaally old Commodores/Macs that just ran Basic. I was hoping maybe something I'd heard of wasn't based on C/C++.
Mesa/Cedar was created at Xerox PARC, as an extension of the Xerox Star 8010 system, which was programmed in Mesa, the language that inspired Niklaus Wirth to create Modula-2.
http://www.digibarn.com/collections/systems/xerox-8010/
https://www.youtube.com/watch?v=Cn4vC80Pv6Q
https://www.youtube.com/watch?v=ODZBL80JPqw Mesa evolved to Mesa/Cedar, which allowed for automatic memory management via reference counting and a tracing collector for the cycles.
https://archive.org/details/bitsavers_xeroxparcteCedarProgra...
The environment had a REPL and debugger inspired by the Smalltalk and Lisp systems at Xerox.
Think of Swift Playgrounds, kind of.
This system inspired Niklaus Wirth to create Oberon, which you can read about in his book, about how the OS was built.
http://www.ethoberon.ethz.ch/books.html
In 2013, Niklaus Wirth revised the language and updated the book for a FPGA based computer.
http://people.inf.ethz.ch/wirth/ProjectOberon/index.html
There were several generations of the Oberon OS, the last one before it stop being relevant, the Bluebottle OS coded in Active Oberon variant.
http://www.progtools.org/article.php?name=oberon§ion=com...
It had a strange mix of UI, using a mix of text and mouse input. Basically all applications were dynamic modules and the procedures/functions with a special signature could be used from the UI.
So you could select text or objects from applications, and invoke such operations on the selection.
The Oberon System 3 and Oberon V4 with their Gadgets framework were the best experience, before AOS was designed.
Most relevant OS in Modula-3 was SPIN OS,
http://www-spin.cs.washington.edu/external/overview.html
There were a few others, this is just a small sample.
Go could be used for doing a, lets call it, Goberon.
It just needs someone with enough motivation for doing it.
Clive (https://lsub.org/ls/clive.html)
Sure rewriting "bits" of things, with more features, and adding my python sounds useful. But here we are 500+ days since the project started and there are only minor experiments in moving from C.
I get the feeling this is one of those projects which will never be complete. Perhaps not a bad thing, providing those involved are having fun, but so much verbiage to say so.
I don't know if they build and work, but I'd suspect they do, based on the refactorings being discussed. The commits seem to be clicking along. A diffstat of roughly where I think they branched off of ntp vs. master takes a moment to poke through, because the raw stats are "1032 files changed, 126489 insertions(+), 143789 deletions(-)", but there's a lot of trading C code for test code and more docs in there.
I don't know exactly what metric you're using but this doesn't match my "obvious lack of progress".
The true metric for success for something like ntpsec is the number of meaningful security problems ntpd has been vulnerable to since ntpsec's inception that ntpsec hasn't been.
At the point where they rewrite in a new language, they're embarking on a fundamentally different project with a different value proposition, which sort of moots the progress they've made (or not made) on hardening ntpd.
(Fair warning: I am both an ESR skeptic and an ntpsec skeptic; I genuinely do not like the idea behind the ntpsec project).
Can't 'ifdefs that nobody uses' hide security flaws? Removing them helps the overall security of the codebase, preventing folks from accidentally enabling a long-dormant codepath, no?
> The true metric for success for something like ntpsec is the number of meaningful security problems ntpd has been vulnerable to since ntpsec's inception that ntpsec hasn't been.
True enough. Are there any numbers on that?
Really all I'm saying is that you can't get a reliable metric of progress on something like this from a simple line count.
I don't know the NTP protocol well enough right now, or the aims for the ntpsec project itself. I will say that having a signed response from an NTP server, would raise the trust level of the accuracy of that response. That is, no one can respond with a bogus time and screw up all sorts of stuff.
In terms of privacy/tls etc, that seems less important. But signed messages would be good.
https://roughtime.googlesource.com/roughtime
... but I acknowledge being somewhat mystified by how much complexity is in NTP given the very minimal needs virtually everyone I've ever worked with had for time synchronization. We all need it, but we don't need it to be ntpd-grade featureful. So this is probably a blind spot for me, and maybe roughtime is less promising than NTS.
They talk about 3 areas where bugs have been removed, including:
Much of ntpd’s most convoluted code lives in ntp_proto.c, which
implements the state machine central to the protocol. Of the 29
vulnerabilities that have received CVEs so far in 2016, a couple
of multi-KLOC functions in ntp_proto.c are responsible for 15 of
them — just over half. Of course, "just rip it out" wouldn’t
suffice in this case: this is core business logic, not junk code.
So we rewrote those functions from scratch, cutting line count
considerably and yielding a far more readable result.
The other two areas are guaranteed bug-free because they removed the code.
Further: not all CVEs are equivalent, so in addition to wanting to know how many vulnerabilities ntpsec was also vulnerable to, you also want to know what the distribution both of severity and of exposure in the default configuration those bugs had.
Ultimately: I feel about ntpsec the way I would have felt if, in 1997, someone had proposed SendmailSec. The answer to the Sendmail security problem wasn't a "hardened" Sendmail (though the Sendmail team sure tried); it was Postfix and qmail.
In short this project is very like the forked openssl project. Progress has been made, I can see it, but I think taking two years to turn some existing code into something cleaner is .. surprising.
Especially now that the focus seems to be throwing it away, and starting anew in rust / go (though based on recent updates go seems to be the choice).
The best alternative out there right now seems to be Chrony. But it hasn't had nearly the inspection and usage ntpd has had, so it's hard to know for sure about its security. ntimed was also looking promising but seems stalled. openntpd doesn't keep time very well and systemd-timesyncd isn't really an NTP implementation.
In the past, this approach would blow up because Go would spawn a kernel thread for each blocking system call that was performed. I had to use a rate limiting channel to constrain the number of kernel threads. Is that still the state of affairs?
Think about something like goroutines reading and writing from pipes.
That being said, Go was fun to learn, and taught me that deployment does have to be as painful as it's been in Java for so many years. But I encourage people to also spend some time with Rust, as it has made a very different set of choices. For me the type and generics system are the best I've ever used and I wish I had access to them in Java.
Then you should spend some time with OCaml and Haskell. :)
I don't need to get up to speed in a weekend. It's the speed that I'm getting up to that matters. Of course there is a limit to this principle, I grant you that.
What matters as well is how readable other people's code is, or even my own code after not touching it for a year. And I mean readable for someone as proficient in the language as the original author. This aspect sometimes seems underappreciated by proponents of "powerful/expressive" languages.
I like Go for this reason. It has more noise (err) than I'd like, but when I look at most Go code, I find it relatively easy to follow and modify. There's no other language I've used that achieves this so consistently.
Wait until he realizes that Rust has channels, too. It's not as pure as Go's implementation, but it's a superset.
I'm like 98% sure loc will be faster, but it would be interesting to know by how much.
For loc the best time was: 55.425s and the worst was: 1m30.557s
For loccount the best time was: 42.345s and the worst was: 56.396s
loc supports more languages but I also believe it has more false positives. loc reporting seems nicer as well.
I see ~5 seconds for loc and ~34 seconds for loccount to count a freshly cloned linux repository.
edit: woops, that was user time. These both spin all the cores. loc takes ~0.7 secs and loccount ~6.6 secs real time.
My rustc version is 1.15.0 and go 1.8.
Unless Rust wants to position itself as embedded-only it will compete with Go for higher level tasks.
Put Rust and Go in front of a python coder who wrote dozens of web services using flask - and I am pretty sure I can predict what tool he'll pick. Sure you could write web services in Rust, that's not what I'm arguing, learning Rust for writing stuff like that is simply not worth it for people having to write such services.
Rust's complexity is comparable with C++, and the main reason for this complexity is to address problems that are mainly seen in C/C++ applications. Rust's reason for existing is simple: modern low-level language with built-in (memory) safety with no performance trade-offs (implying concurrency support etc). That makes it squarely aimed at C++, where developers accept the complexity as a trade-off for performance.
Go's reason of existence and goals are completely different, it addresses the gap that existed between compiled and interpreted languages. It wants to be a simple, safe, easy to deploy language that offers more performance over classic scripting languages and thus is able to address more performance-critical issues. It's main target is clear: servers and networking services, where a GC's impact less important due to the I/O bottlenecks in most scenario's.
C++ dev here. Rust's borrow checking system is crazy and even after a week of trying I couldn't understand it in a way that I could reliably use the language.
In C++ everything is fairly straight forward and well documented, in Rust it feels like I'm inserting random symbols in hopes that something will compile. The syntax is verbose and ugly, maybe not as bad as C++ template code can get sometimes, but it's nasty in its own right.
I want to like Rust, I really do, but this just really turned me off it.
I as Java developer had some hurdles, but most of the time, me and borrow checker get along. My greatest improvement came from compiler lifetime elison in one of the pre-1.0 versions of Rust.
As for syntax being ugly, I do somewhat agree, but mostly, when it reminds me of C++ (e.g. T<U<Vec<u8>>>). I hate tire-tracks and Type::static_method() definitions.
This is a rosy-eyed view of C++ if I ever saw one. C++ has a lot of accidental complexity from layers and layers of features and backwards compatibility concerns: maybe some of the features are "straight forward" in isolation, but its interactions with the rest of the language rarely are.
C++ is deceptive, in that the compiler barely keeps track of anything for the programmer, so there are many cases when code should have "random symbols" (or something equivalent) added, but isn't, just being broken instead. The Rust compiler front-loads this: trying to turn runtime brokenness into compile-time errors, which does make it annoying to get code that runs, but it makes it less annoying to get code that actually works correctly.
Wow! C++ is full of undocumented and documented, but non-obvious, behavior. I guess that speaks volumes about your experience with C++: given enough time, anything is straightforward.
Rust has a steep learning curve (I'm still climbing it), but it is nowhere even close to C++. C++ gives you a false sense of security, in the sense that code will often compile, but be obviously wrong (for someone knowledgeable). The Rust compiler will not stop complaining until you correct your program.
It's even worse for beginners. Rust learners will get frustrated because their programs are not compiling. C++ learners will get frustrated because their programs will segfault. Often because of some obscure issue, such as someone who forgot to implement copy constructors or assignment operators. I've been there, these issues are completely non-trivial to debug.
I would rather have a compiler yelling at me, but that's me.
Also, since you mentioned that you've been trying for a week: do you think a novice C++ programmer would be able to actually do anything coherent in C++ after a week? Heck, I don't think that's enough time to even go through the spec!
My opinion is that Rust is at least as complicated as C++. They have different complexities, but one isn't "easier" than the other. I enjoy Rust a lot. More than C++, at this point, but that may be the "newness" factor creating bias. I think Rust is likely a better language for people new to systems programming to pick up. It's complexities are largely up-front: a novice will learn how to program well in Rust by having the compiler break his teeth a few times, while for C++ she'd either have to have high-quality mentoring or else learn the hard way (by long experience).
It depends what part you show. For example Rust has an experimental flask/sinatra/... inspired library (https://rocket.rs/overview/), and although Go has similar things you can't cut boilerplate that much:
So in the end it may depend why someone likes python in the first place: because of the concise high level constructs or because of ease of use.
I agree, but that's the thing... people DO advocate for that!
Of course, there's a lot of double-talk on the subject. "You can TOTALLY do it today!" ... "Eh, but maybe you wouldn't want to use it PRODUCTION today" ... "But we're just a FEW more crates away from being there!"", etc. The Rust community is great, but they seem they hedge their bets too much and are murky with the messaging.
Right or wrong, Rust and Go compete because there are a LOT of web and microservice developers out there (including me) who:
1. Love what Go brings to the table in terms of of type safety, and AOT compilation to single-file native executables...
2. ... but wish that Go had more functional-style constructs (filter, map, immutability, etc), generics, a contemporary package system that doesn't involve putting Git URL's in your actual source code... or any of Rob Pike's other stubborn alien quirks that people don't like.
Rust can't quite figure out whether it wants to market to this crowd or not. Sadly, due its learning curve, I DON'T believe that Rust is the answer for the "Go, but with more popular features" problem. But enough people aren't ready to concede that yet.
People looking for such a language will take a look at them and pick one.
But that is true for any fast-enough high-level language like Java, Scala, C#, OCaml, Haskell, F#, Kotlin, etc.
So it's not Rust vs Go, it's Rust (and Go) vs the world.
Taken from Dave Cheney's https://dave.cheney.net/2015/07/02/why-go-and-rust-are-not-c... ->
"Rust competes for mindshare with C++ and D for programmers who are prepared to accept more complex syntax and semantics (and presumably higher readability costs) in return for the maximum possible performance. For example, micro controllers, AAA game engines, and web rendering engines."
"Go competes for mindshare in the post 2006 Internet 2.0 generation of companies who have outgrown languages like Ruby, Python, and Node.js (v8) and have lost patience with the high deployment costs of JVM based languages."
For example, while you might need Rust to target a PIC or a ESP-32 class processor, Go will run perfectly fine on a ARM Cortex-M3 which also has hardware vendors selling Java compilers, e.g. MicroEJ OS (http://www.microej.com/).
I used to have to explain this in a HN comment every time it came up, but now Mike Hearn has written up [1] better than I ever could, so I can just link there :)
[1]: https://blog.plan99.net/modern-garbage-collection-911ef4f8bd...
Sure it has quite a spartan type system and I would rather use something else for my own work.
On the other hand, as an ex-Oberon user I am aware of many use cases that Go can easily fulfil and like to make people aware of it, more Go less C is good.
Rust and Go might compete in that area, assuming that Go programmers will have interest in such projects.
It's not in the interest of the leaders of each community to admit that though.
For any type of application, including user space drivers, that can be developed with the constraints of a sub-millisecond GC pauses, GC is a good language, in spite of its stance on generics.
As for kernel space, Go could eventually be used as well, in spite of GC hate, as academia including Microsoft have proven a few times.
Me personally, I prefer Rust, but I do see others would think otherwise.
- Both compile to native code.
- Both care a lot about performance.
- Both care a lot about parallelism.
- Both care a lot about safety.
- Both were invented by a major browser vendor, roughly around the same time.
- Both include an unsafe fun zone that lets you play with raw pointers.
Maybe some of that was inevitable just by following the trends in the field (safety and parallelism in particular), and we'll see other new languages making the same choices?
We are already seeing that, .NET and Java are finally paying more attention to the AOT story on their standard toolchains.
And we also have Objective-C and Swift on iDevices.
Rust not only does all that, it provides data race safety, and does so without a runtime or tracing garbage collector or real performance sacrifices. All while maintaining C linkage for zero cost interop. I can use Rust basically anywhere I could use C, Go, not so much.
That seems to put it squarely in a different segment than Go.
Go doesn't reach Java level safety because of a lack of generics.
It seems at this point clear to me, that the previous Rust piece was written out of sheer frustration, and that the entire period while "learning" Rust he really wanted to be writing in Go, which he had already found to enjoy. This is understandable, but creates a clear bias; which is completely fine, but would have been nicer to see this for that reason.
Everyone's experiences are going to lead them in different directions. I learned Go, decided it was missing features that I really wanted, and that's when I decided to learn Rust. That was a decision driven as much from comfort with a language as it was technical. Which brings me back to his original Rust post, it was written as though comparing two languages on their technical merit, when in reality (after reading this post) it was really about comfort. Again this is fine, and would have been clearer if posted in the original order.
For instance it took about 7-8 years before I switched to Java (and then only after establishing a programming style that doesn't rely on frameworks at all). The language wasn't hard to learn, but it took a while for the JVM and the standard library to get to where it needed to be.
C++ never made the cut. Quite possibly because C++ is not really one language. There were at least as many styles of C++ as there had been projects I worked on (except perhaps at Google, which is the only place I've worked where I've consistently seen good quality C++ code).
I'm in the process of evaluating Go as my main language and I have to say that this is the fastest I have gotten comfortable with a new language. I've been noodling around for a few months, doing side-projects, but my experiences so far have been very positive. Unlike most languages I haven't come across any uncomfortable stuff that would be hard to overcome. Just the usual confusion about what the idiomatic ways to do things are.
What is striking is that it is equally comfortable for doing relatively low-level (my first Go program talked to a chipset over SPI on RPi, doing something that is slightly timing sensitive) stuff and stuff that needs you to build abstractions.
I haven't used Rust yet. Mostly because I don't have the time. And I chose Go since some of my friends use it for software that you and I touch every day, thus providing me with access to people who have used it "for real".
Go just might be my next "main language". (Actually, right now it feels like I'll switch to it and never go back, but the tiny rational bit of my brain says "oh yeah, here's a list of other times you had a good feeling about something that turned to crap" :-))
But I dread the feeling of writing repetitive error handling code and copy-pasta to get a simple generic functionality to work across data types. Maybe I should just consider it a better C and move on...
I don't mind a bit of boilerplate as long as it produces understandable code. Writing boilerplate can be automated. Reading and understanding code can't.
I do understand where you are coming from though. Perhaps we're getting a bit spoilt as programmers in feeling that error handling should be ... more elegant for a language designed within the last decade? :-)
I think the same goes for generics, though I have a hunch that I'm not going to miss it. And I'm not sure why, but that's how it feels now. Whenever I do design classes that are genericized in Java I do so sparingly. This is mostly to ensure that the code is easy to reason about. I'm not fond of "clever" uses of generics because while it may feel satisfactory to have things click together in a beautifully complex manner, it can make code hard to read.
Looking at Android as a representative of "embedded" also seems a bit fallacious. The lowest-end android device I can buy has a lot more ram, storage and clocks than an average home router.
EDIT: this comes across pretty snarky - more so than is warranted. I'm surprised by the first part, but have no right to impugn the whole project based on that. The surprise remains, but I apologise for my tone.
Well you can look at ARM Cortex-M3 running MicroEJ instead.
Me: Android doesn't need to meet constraints anywhere near a home router (which needs to run ntpd)
You: there's a proprietary Java-based OS for Cortex-M3
.. still don't get it. Are you trying to sell me something?
"The language does have annoyances. The absence of const is a minor one."
https://gobyexample.com/constants
const s string = "constant"Now when I go back to python or perl (or javascript) I'm super conscious of the choices I make regarding type checking and parameter checking.
So I downloaded the code and compiled it. The binary is 2.5MB; or 1.6MB after running it through `strip -s`.