Mars Curiosity Operating System: VxWorks | Hacker News Reader

Mars Curiosity Operating System: VxWorks

en.wikipedia.org

93 pointsbradgessler44 comments

If anybody else is interested in the topic, the seminal work on software development at NASA is the Fast Company feature from 1996, "They write the right stuff":

http://www.fastcompany.com/node/28121/print?donttaseme

Worth saying that doesn't apply outside manned missions. They notoriously crashed a probe into Mars thanks to erroneous imperial to metric unit conversion.

EDIT: also the VxWorks fan stuff here is a bit odd, given the other notorious error involved setting the task priorities so that the probe spent the whole time checking the weather, but never phoned home. They were just lucky they shipped the thing in debug mode and reprogrammed it once it had landed.

That pathfinder bug was a priority inversion caused by failing to enable priority inheritance when creating mutexes, not quite as simple as setting the priorities wrong. VxWorks wasn't entirely at fault, although the priority inheritance feature was not obvious. Concurrency is hard. See http://research.microsoft.com/en-us/um/people/mbj/mars_pathf... for details.

you really, really should never write things such that task priorities must be of a certain arrangement for the thing to work properly. It's very bad form. Also, from the link "Tasks on the Pathfinder spacecraft were executed as threads with priorities that were assigned in the usual manner reflecting the relative urgency of these tasks." Yeaaaaaugh! "PRIORITIES DO NOT WORK THAT WAY!" - Morbo.

Why the "donttaseme" in the query string?

I had it bookmarked like that, my guess is that it was a re-submission and somebody added that tag in order to make the URL unique

$ vimdiff <(curl http://www.fastcompany.com/node/28121/print?donttaseme) <(curl http://www.fastcompany.com/node/28121/print)

Aesthetics, perhaps?

"Bill Pate...says the group understands the stakes: "If the software isn't perfect, some of the people we go to meetings with might die."

Some days....

I've read that over a dozen times by now and I'm still fascinated by it.

Some features of VxWorks and why it's so deterministic and fast.

http://stackoverflow.com/questions/8800371/what-makes-vxwork...

I worked on VxWorks and Linux on Intel's IXP embedded processors before, and VxWorks' performance and memory footprint were always better than Linux's. For 'Performance' I refer to network throughput because IXP was meant to be for routers and gateways. It was around 2006 so I am not sure what these two OSes are up to now. Don't flame me because I am just speaking from my own experience.

Previosly on Mars & VxWorks: http://www.cs.cmu.edu/afs/cs/user/raj/www/mars.html

I was watching the Planetfest rebroadcast of Curiosity's landing, and mention was made of there needing to be two operating systems: one to deal with entry and landing, and the other to handle roving and research. The speaker went on to say that both OSes don't fit into RAM at once, so they would have to be swapped over post-landing, and that this would take four days to load from Flash.

Raises the question: why does it take four days to swap an operating system in from Flash, and how does the system go about it while having a path to fall back?

Personal theory: The hardware used has to be low powered, and be able to withstand physical and radiological abuse. Such a system probably will not post the fastest clock cycles, what with power consumption being proportional to frequency^2 and all.

I'm also going to guess that most of the available RAM will be used for redundancy.

I'm guessing the copying doesn't take four days unless they are actually sending the program from earth, in which case ~256MB over X-band to Mars could take that long. Rather I would guess that this copy is a many step procedure they do one step at a time and confirm at every step. With a 28 minute round trip to tell the rover to do something and get a response time adds up quick.

I wonder if the time taken is because it must use very tiny amounts of power, and survive at such a low temperature. They may be using a very slow processor, or a special low-power, low-temperature NAND or whatever variant.

It depends on how one defines slow. Curiosity's onboard computer is 100x faster than the previous two Mars rovers, and has twice as much ram. But that's only 200MHz and 256MB respectively.

With the exception of Sojourner, the first rover successfully placed on Mars, all Mars rovers have run VxWorks.

http://en.wikipedia.org/wiki/Comparison_of_embedded_computer...

I remember an HP tape array running VxWorks crashing when a copy of Nessus would scan it. There was no patch for the issue at that stage. This was a tape array backing up some infrastructure that would handle the transactions of a big chunk of Europe's stock exchanges. It kind of left me with a bad impression of VxWorks, but it might just be HP's processes that was a mess at that stage.

Glad to know it has support for IPv6, early adoption in new places is good!

I thought they would just use the martian IPv4 ranges.

Very droll :)

For the non-network engineers in the audience, it's worth documenting why this is funny:

http://en.wikipedia.org/wiki/Martian_packet

"The name is derived from packet from Mars, a place where packets clearly can not originate."

Uh oh. Looks like its time to update some wiki's.

"early adoption"?

Pricey too - according to this from back in 2008, its $16,500 per seat (developer)... http://mihirknows.blogspot.com.au/2008/03/rtos-cost-and-feat...

A drop in the ocean in the scheme of space projects though.

I wonder what the ground control software is written in. I know that for one of the previous ones, they used Tcl and Tk, and the first screenshots they showed with the first image were the X windowing system if I've ever seen it.

I think I peered some SUN workstations, so Solaris would be a fair guess. Pure speculation of course.

How appropriate that a space agency will be using SUN workstations =)

VxWorks is used pretty extensively in space and defense contracting. The list in wikipedia is a pretty small sublist. It's also often used as a wrapper for custom bootroms.

Do people still use VxWorks for new projects? Isn't linux gaining weight in the embedded market?

Try getting some functional safety certification on a Linux distribution and you see why Linux is not really gaining much weight in certain areas of the embedded market.

However, it is getting small inroads in the instrument clusters for Automobiles. (http://www.genivi.org/)

VxWorks is a real-time OS. I didn't know whether there was a real-time variant of Linux and what I found was that there was, but Wind River acquired the company that was working on it and essentially killed it.

http://en.wikipedia.org/wiki/RTLinux

Edit: Wind River is the provider of VxWorks.

From what I last understand, VxWorks, being an actual RTOS, has faster and more consistent latency rates than any Linux embedded system.

For systems which require a guaranteed response time, such as advanced robotics, this is a major deal-breaker.

There was an article in linux-journal that described how linux was being used at JPL : http://www.linuxjournal.com/article/7570

From what I understand, Linux runs the rover simulator environment - known as RSVP. It consists of a DSL that is used to direct the rover's operations and a visualization environment. This way the ground crew can first plan and test the activities they want the rover to perform in this simulated environment and then upload the action plan to the rover - so overall Linux plays a pretty important part in the the rover tech stack.

Most probably the initial rovers ran VxWorks RTOS, because back then it was the best choice (even now it isn't a bad choice at all) - complex systems are built incrementally.

Aside from what others have mentioned keep in mind that something being sent to another planet usually has 5-10 year old technology at best. For reference in 2003 Linux 2.4 was the latest stable version.

In space they increasingly use C++ on Linux (SpaceX, ESA, ...).

Defense and aerospace - even there we are seeing more Linux for non-safety critical software

SpaceX also use it for their Dragon spacecraft: http://www.spacex.com/downloads/dragonlab-datasheet.pdf

Cool, now I know that Apple routers use the same OS as Curiosity.

And Linksys.

What does SpaceX use? Anyone got juicy details?

VxWorks as well, http://www.spacex.com/downloads/dragonlab-datasheet.pdf

From their careers page: "Our Flight Software Group currently has opportunities developing software for embedded flight hardware using Linux and VxWorks as well as ground simulation software using Linux."

anyway to get my hands on it to have a play?

Volunteer to mentor a FIRST Robotics Competition (http://www.usfirst.org/roboticsprograms/frc) team. The robots built for the competition use a National Instrument cRIO (http://en.wikipedia.org/wiki/CRIO) which run VxWorks 6.3.

Reply on news.ycombinator.com