Why does this Java program terminate despite appearing that it shouldn't? | Hacker News Reader

Why does this Java program terminate despite appearing that it shouldn't?

stackoverflow.com

124 pointspatmcguire121 comments

One of my biggest WTF moments as a programmer was dealing with a high performance spectrum analyzer. It was a cost no object bit of experimental military hardware, and was very impressive from that perspective. But the software was buggy. We would see lockups, as well as discontinuities in the readouts, which were screwing up our post-hoc data analysis. When I dug into the code, I realized that the program used several different threads, but there was not a lock or other synchronization mechanism in sight. You had a thread pulling data off a socket from the device and storing it in a (single) memory buffer, a thread writing commands through a socket to the device, a thread pulling data out of the buffer to draw it to a waterfall plot, and a thread writing data to disk. The lockups happened because the device didn't like to change the scan state while reading data, and there was no synchronization between the command thread and the readout thread. The discontinuities happened because there was a single buffer shared by the readout thread and the write out threads, with no synchronization. So as one thread was writing out a sample, the readout thread was replacing part of that data with the next sample.

I remember grepping the source for pthread_mutex or pthread_cond and slowly realizing that whoever wrote the code didn't even see the need for locks. To be fair, I later heard it was some poor hardware guy who knew some C who got roped into writing the software as an afterthought to show off the awesome hardware, a common problem with hardware companies.

> I remember grepping the source for pthread_mutex or pthread_cond and slowly realizing that whoever wrote the code didn't even see the need for locks. To be fair, I later heard it was some poor hardware guy who knew some C who got roped into writing the software as an afterthought to show off the awesome hardware, a common problem with hardware companies.

"Oh, you wrote a quick demo for the sales people? Well, now it's a product sold for some ten thousand!"

You joke, but in military contract world that is totally fine. Everyone already got paid and bugs will certainly not stop the acquisition process in the future. It is all about who knows who and more about jumping around red tape certifications, ATOs, ATCs, rubber stamps and so on.

That is often why soldiers on the ground end up with overpriced yet crappy tools and weapons.

This is also why automatic weapons with fully autonomous control is probably a very bad idea.

The good news is that if Skynet ever happens, it will probably crash before it can take over the world.

To be fair, it's not just red tape. If you are the Army and your one of the kind custom spec piece of hardware has software bugs, what exactly are you going to do? Buy the compatible version produced by the competing vendor?

Ideally you have prototypes and tests before hand that should discover these issues. Then in the future you would pick a different vendor. You can also set up your contract to account for maintenance and for penalties for serious bugs.

A lot of things are possible but they are not happening the incentives are just not there.

How can anyone program sanely in the presence of this:

currentPos = new Point(currentPos.x+1, currentPos.y+1); does a few things, including writing default values to x and y (0) and then writing their initial values in the constructor. Since your object is not safely published those 4 write operations can be freely reordered by the compiler / JVM.

So from the perspective of the reading thread, it is a legal execution to read x with its new value but y with its default value of 0 for example. By the time you reach the println statement (which by the way is synchronized and therefore does influence the read operations), the variables have their initial values and the program prints the expected values.

I'm not anywhere near smart or careful enough for that... I think I'll stick with Haskell.

The One Commandment of multithreaded programming:

1. Thou Shalt Not touch shared data without synchronization.

You must know what data is shared, and you must synchronize all access to shared data (with one exception, when all access is read-only).

People try to find exceptions to this One Commandment. They think, I'm only writing one value, so what's the harm? Well, this is the harm. Follow the One Commandment and you'll be safe(r).

No real argument here, but Haskell does encourage you to use read only data structures unless you truly need writes. That said, there's still a wide variety of clever ways in which you can find your way to multithreading hell.

Read-only data structures are a common idiom in Java as well (the classic example is String), and do indeed help reduce the headaches.

As Josh Bloch said:

Classes should be immutable unless there's a very good reason to make them mutable

Immutability makes life a LOT easier. A great example is the well designed joda-time vs the disaster that is the the mutable Date and Calendar core language classes.

Yes. Now if only the standard Java collections were immutable...

Not sure this is true. There is the concept of final fields, but that doesn't make objects immutable, and thus even when declaring object finals (or arrays) you can pretty easily shoot yourself in the foot by mutating them in-place.

For immutable collections, guava is helpful.

It is -- but you still do have to be aware if the objects in your collection are also immutable... if so, you may end up with an "immutable" list of 5 objects, and down the line it will still have those 5 objects, but the objects themselves may have completely different internal data.

Referring partly to your parent comment... it's instructive to look at the String implementation.

It's immutable, but they have to jump through a few hoops to do it properly... internally, a String has a private char array. An array is not immutable, so the code has to be very careful to never expose that char array externally.

Because of that, calling toCharArray() will copy the internal array and return the copy -- that one's fairly obvious. But less obviously, a String constructed with a char array must also make a copy... otherwise the calling code might then modify the array after creating the String with it.

So sure, immutable objects are possible, and widely encouraged in Java, but they're easy to do wrongly/incompletely.

Heh, my 1st commandment of multithreaded programming is: share as little as possible, isolate data in each thread and hand-off ownership between threads. With this, you will need only very little synchronization (only for queues), and you will not risk deadlock.

I think that's 3 commandments ;-).

.. or 3 ways of saying the same thing, really :)

Corollary: don't put in locks blindly just because a datum is shared. You'll just get deadlocks instead of race conditions.

Synchronization is like commas in English. People don't really know the rules, and there is danger in putting in too many as well as too few. You should never put in a lock without understanding the mechanics of the datum you're touching. Some times you won't need synchronization at all.

I would, on the whole, usually rather have a deadlock than a race since they tend to crash rather than silently corrupt state ... and the ones that cause retries rather than crashes tend to show up in the profiler. But that's an "err on the side of" rule of thumb rather than a defence of blindly locking things.

My exception: If all operations on the shared data are atomic, and always leave it in a valid state, then you can get away without explicit synchronization. Unfortunately, the only way you can really achieve this (without some sort of a synchronization API) is to use assembly code, and take advantage of architecture-specific data access mechanisms.

But you'll need memory barriers. Let's say you have two variables, x and y, satisfying x >= y at all times, and the values of x and y are never allowed to decrease.

    // Thread 1
    atomic_inc(&x);
    atomic_inc(&y);

    // Thread 2
    local_y = atomic_read(&y);
    local_x = atomic_read(&x);

If you go through it logically, it's always in a valid state, right?

No. You need explicit synchronization. If we start at x=1, y=1, then thread 2 can see x=1, y=2, which seems impossible!

Or in short, your exception sucks. (By "sucks" I mean it is not sufficiently detailed enough to prevent errors.)

This is only the case on architectures where atomic operations don't imply memory barriers. On x86, there is a total order on atomic operations, do another CPU will never see the write to "y" before the write to "x". Indeed on x86, there is total order on stores, so if you had two threads like that, where one thread only did reads and the other thread did updates, you can get away with an unlocked increment (x++; y++).

Being atomic is not enough. It's too easy to use two atomics operations and assume they'll be sequentially consistent.

For example, suppose we have:

    A = 0
    B = 0
    Producer:
        while true:
            AtomicIncrement A
            AtomicIncrement B

    Consumer:
        while true:
            postA = AtomicDecrement A
            postB = AtomicDecrement B
            assert postA == postB // wrong

The assertion is wrong, when you're dealing with relaxed atomics. For example, the producer can be optimized into this either by the compiler or by caching layers:

        int savedUpA = 0
        while true:
            savedUpA++
            if (savedUpA == 100)
                AtomicAdd A, 100
            AtomicIncrement B

In which case the consumer will often see that postA != postB.

Thankfully, languages are apparently converging on "every atomic write is a release barrier" and "every atomic read is an acquire barrier", which makes these sorts of optimizations illegal (releasing B's new value requires releasing A's new value).

The Atomic* classes in Java give you essentially direct access to these mechanisms.

Absolutely true. I would say that either atomics count as a form of "synchronization", or they're advanced enough that you can disregard the One Commandment once you get to that stage.

Although atomicity helps, it typically doesn't provide any visibility guarantees, so typically some other form of synchronization or memory barrier will be required.

0. Thou Shalt minimize and reduce shared mutable data as much as possible, in the design stage.

Sometimes just making new copy and passing that around might not cause that much of a performance penalty vs the time spent debugging synchronization problems.

Honestly I don't think Haskell would have helped much here.

That code is terrible from a thread safety perspective. If the author didn't read up on how to write good thread-safe code, it's unlikely he'd read up on Haskell, so he'd probably write bad code there, too.

Haskell has type-safe software transactional memory for this stuff. And it's exponentially harder to get a program that compiles and has error like that.

The author didn't take the time to learn the very basics of thread safety. It's highly unlikely he would take the time to learn Haskell and software transactional memory.

Can you explain what makes it so terrible? At first glance it seems as if it is thread safe, and avoiding synchronization locks can provide a nice speed up.

The notion that currentPos could be set to the new Point before the Point(int x, int y) constructor completed was completely surprising to me.

Is that something that was obvious to you, or do you have other reasons to dislike it?

Yeah, this is very surprising to me. I simply didn't know that java worked that way and I've been programming in java for many years. I assumed that the memory that "currentPos" would eventually be bound to was fully initialized before the assignment operator remapped "currentPos" to the new value. I also assumed that binding a variable name to a new value is atomic across all threads.

currentPos is modified in one thread and read in a different thread, outside of a synchronized block, and without being protected by a mutex.

It's a pretty straightforward recipe for disaster.

I don't program much in java, but I would have thought that the problem was in the assignment of the new Point to currentPost, not the construction of the new Point.

I would have thought that the order of operations would be:

1) Read value from currentPos.x and add 1. Do same for y.

2) Pass values to constructor

3) Construct the new object

   3a) init values with 0

   3b) assign new values to x and y

4) Assign the newly constructed object's pointer to currentPos.

I thought that the problem lay in 4 where you might read a partial update of the pointer, possibly giving weird results. Is he saying that it might get assigned first and then constructed afterwards?

You're assuming that the code is executing in the way you read it, which is false. With the advent of branch prediction and Instruction reordering in CPUS (to gain performance), a CPU have the liberty of reordering operations for efficiency (http://en.wikipedia.org/wiki/Out-of-order_execution) EXCEPT when there are memory barriers (or explicit synchronization instructions). With a multi-core processor things get even more complex, as you have cache locality (a thread reading the point value might be hitting the CPU cache and not main memory). If the thread happens to be executing on a different core than the assigning thread, disaster ensues.

> I thought that the problem lay in 4 where you might read a partial update of the pointer, possibly giving weird results. Is he saying that it might get assigned first and then constructed afterwards?

Yes. In some versions of java the code: x = new X() results in assigning to x a reference to a new uninitialized object X and then a call to the constructor.

Reference assignment in java is always atomic, it is guaranteed by the memory model.

No. x = new X() should not make x assigned such that it is visible inside the constructor of X. The behaviour in the post can only been seen with concurrency.

The issue here is the lack of a write barrier at the point of publishing the reference to the newly constructed object, and the lack of a read barrier at the point of reading the reference to the newly constructed object. You want to stop both writes moving forward in time (the writes in the constructor happening after the write that publishes the object) and reads moving backwards in time (the reads on the other thread reading the old values of the constructed object, rather than the initialized values - may be caused by e.g. satisfying the read from per-CPU cache).

Java's volatile acts both ways; writes are write barriers and reads are read barriers.

In less technical language, "creation of the object is not atomic". You could switch threads in the middle of the creation.

Also, "reading of the data is not atomic" - you can switch threads in the middle of reading the data.

In order to program THREADED CODE sanely with that in mind, you need to minimize the amount of shared data, and be sure to lock / synchronize around the writing AND READING of any shared data.

If you're not writing threaded code then there is no shared data, and you don't care about these rules. Writing threaded code isn't like dusting crops, but it's not impossible.

EDIT: you also said "the println statement [...] is synchronized" but it doesn't look like that's true - there's nothing in the synchronized(this) {} block, and even if there was, all it would do is keep "this" from entering that block twice; it has no effect on other threads in this code.

You misunderstood the part about println. The implementation of println() uses synchronized internally. You can't see it without downloading the Java source code, but it's there.

But the argument to println is evaluated before println is called! Whatever synchronization it's doing internally therefore can't affect what it prints.

Seems to me, either there's something synchronized in Integer.toString (which seems undesirable and therefore unlikely) or, statistically, in the rare cases that the writes don't take effect in time for the 'if' predicate, they do take effect a few nanoseconds later, in time for the conversions involved in the string construction. (Actually p.x is the only one at risk here, statistically speaking; the conversion of its value to a string will take plenty long enough for the write of p.y to take effect. Combine that with the fact that the write of p.x probably still precedes that of p.y, so that the one that's wrong in the 'if' statement is probably p.y, and you see that the odds of the wrong values being printed are vanishingly small.)

In general, you shouldn't assume that tomorrow's implementation behaves like today's, or today's implementation X behaves like today's implementation Y, unless there is documentation that says so. Looking at http://stackoverflow.com/questions/9459657/synchronization-a..., it seems that Java does not guarantee that println will synchronize.

The exception is if you have tight control over your environment, have tests in place to verify that environment upgrades do not break your assumptions, and you desperately need to bend the rules, but even then, you should look really hard for better alternatives.

Ah, thanks. I don't do much Java.

So "println() uses synchronized internally" means two println statements will never overlap each other, but still says nothing about their interaction with other objects, right? Does nothing to prevent switching threads in the middle of the println and getting two different values?

Correct. It synchronizes on the instance of the PrintStream, which has no effect here.

The bottom line is that the Point object should have been immutable, which would have made it safe for publication. That requires its fields to be final (among other things), which was not the case.

The comment about println being synchronized is indeed incorrect. See my reply to Eiwatah4.

You don't have to know the majority of that, though.

These are tricks the compiler/VM can use, because (unless you say otherwise in code) it's assumed that there's no sharing data between threads.

Most programmers (regardless of the language) just learn the rules about programming with threads -- they don't care why exactly following the rules is important.

Sure those programmers learn the rules... then they promptly forget them because they seem too complicated and "it works on my machine with only one client"..

I agree with you, one must know why rules are important.. but in large systems, it's better if the system/language enforces this separation of data (praise Erlang).

insert "dumb code in any language" rant here

Get real

It's amazing what kind of yarns people on SO will spin to get help with their homework.

I often feel that it's necessary to stretch the truth a little bit on SO. Otherwise you ask "How can I do X?" and all your answers will be "Don't do X. Use Y instead."

That happens because of the "old shoe or glass bottle?" question.[1] When newbies (or even oldbies) ask how to do something, often the real question could be solved much better by taking a completely different tack. This has also been called the "XY problem".[2]

The best answers would ideally answer the question as asked (or explain why it can't be done) and also show a still more excellent way.

[1]: http://weblogs.asp.net/alex_papadimoulis/archive/2005/05/25/...

[2]: http://meta.stackoverflow.com/questions/66377/what-is-the-xy...

The best answers would ideally answer the question as asked (or explain why it can't be done) and also show a still more excellent way.

There are times I've asked how to do X, and for various reasons X is really, no kidding, exactly what wanted to do. Still, numerous people would question my desire to do X, turning the discussion into an inquisition on motives or skill comprehension.

I'm sure in some odd way they all meant well, but they were spending a lot of cycles not actually answering the question.

In those cases the better strategy is to, under some crafty pretext, bluntly assert that X cannot be done.

This will evoke numerous rebuttals with full details on just how incredibly wrong and foolish that claim is.

Both cases play to a trait all too common to many people on discussion boards: replying to questions with the primary goal of showing that you are, in fact, smarter than everyone else.

I've seen this argument often enough, especially on IRC, where people smugly assume that there must be a flaw in what the person asking a question is trying to do. Problem with this is, a lot of times, I just want the answer to a direct question, which I can't find easily in the docs (or maybe it's not documented).

Only, more often than not, people think they always know better and everyone asking a question must either not know the proper way of doing things, and why in the world are they trying to do <question_asked> ? This then leads to a debate about me trying to solve problem X which requires me to solve Y which requires me to do Z, which led me to ask the question. At the end, I get the exact answer I was looking for, but that took longer than necessary because other were looking at it the same way you described, that it's the XY problem. I can't help but feel that it mostly just wasted my time a lot.

You don't always have to know the entire story behind the code to give straight answers, is all I'm saying. :)

Despite how much computer people love Eric Raymond's "how to ask questions" thing[1], it really isn't the best way to get your question answered. It's the way that people who get asked questions wish the universe would work.

One thing I've found is that, beyond a pretty low floor, more details often means less answers. If you, forex, post your C code and ask why it doesn't work on IRIX, people will say "it must be something weird about IRIX, I don't care about that" and ignore you. If you post it without mentioning the OS, you can get some good answers.

[1] http://www.catb.org/~esr/faqs/smart-questions.html

Can't spin yarn without Threads! :D

Actually, vice-versa. Spinning is the act of creating threads. You might have said, "You can't make a thread without spinning yarn".

Oh @#it I fell for it.

If you look through the questioner's history it seems like they enjoy making these sorts of threading/synchronization errors, as well as getting angry/trolling. I wouldn't be surprised if the story about losing the 12 mil electron microscope is total BS designed to garner rep (I see it's getting tons of votes).

Most of those upvotes are from one day though, so they only get about 200 reputation due to the cap. (Also, they gave away 50 points on a bounty for the top answer, so I'm not sure if reputation was the motive or not.)

That is a good point, although I still find it suspicious, especially given some of their previous questions. E.g. the one where he/she created two Haskell data types that looked the same but were using different characters in their names and then complained it wasn't working, really?.

Oh come on! This title rewriting is getting silly now. The Java application is not the story - there are thousands of those on SO. The story is the (real or made up) destruction of equipment by a failing program...

At the core, the use of threads for any embedded system required to operate in hard real time is a mistake. Threads and interrupts are the minefields of real-time embedded systems.

Based on study and years of experience I conclude that the ONLY way to build such as system is to use a time-triggered approach as opposed to event-triggered. No threads. Only one interrupt --the timer-- in the entire system if at all possible.

This is where I fall back to the raw simplicity of C. You really have to work hard to do stupid shit like threads in C. By that I mean that you have to go out of your way to bring in the code and a framework that might allow you to do that. I've successfully written and applied at least a couple of RTOS's for use in mission critical applications (definition: you fuck up and someone gets hurt or dies). Again, only one interrupt. No threads. Carefully --and I do mean very carefully-- shared data between tasks only if absolutely necessary.

A corollary to this is that I could not fathom programming an electron microscope (the subject of the SO question) or anything that might lead to millions of dollars in losses with Java. Couldn't pay me enough.

Anyone interested in working with hard real-time embedded systems would be wise to study the following well-known books:

  - Patterns for Time-Triggered Embedded Systems, Michael J. Pont
  - Embedded Systems Building Blocks, Jean J. Labrosse
  - uC/OS-III, Jean J. Labrosse  (uC/OS-II by the same author includes 
    the source to the prior version of the OS)
  - Doing Hard Time, Grady Booch
  - Operating System Concepts, Silbershatz, Galvin, Gagne

There are other good books on the subject. The above should provide a very solid foundation.

Don't just hack away without the necessary background. The consequences of making mistakes out of sheer ignorance could be dire.

I think the real wtf was testing in production. At least their lab now has an environment to do their qa on.

I'm curious, could he (or you) have written a program in any other languages (like Go) and not have worried about these issues?

Honestly, I got some mild goose-bumps after realizing the number of Zeroes that 12 million had.

you could, but it's not very clear why you would want this to be multithreaded to begin with, since it appears the author wants the program to quit as soon as the value fails validation. It looks like it's multithreaded just for the hell of it.

Anyway, you could do it in Go like this: http://play.golang.org/p/3Wy_S42jVj

basically what you do is you create a goroutine that reads a pointer to the Point value, checks it, and if it's bad, exits. In your main loop, send a * Point down the channel. That basically means: I made this thing. It's yours now. Do what you want with it. Then in the reporting channel, the value is read and checked. The same pointer is sent back in a reply, meaning "I'm done with this, you can have it back now". You almost certainly shouldn't be doing this; I'm just doing this to emulate the fact that the original example is using the same reference from both threads. Yes, this is very, very weird and no, you should not do this in your own programs. Anyway, we take that pointer back, dereference it, and assign it to a new Point value, which is completely weird and I still don't understand. (presumably the original author mistakenly thought this was an atomic way to set both fields on the Point.) In this way, the two goroutines (the main and reporting goroutine) are able to use channels to synchronize access to some shared memory.

This is a pretty silly use of concurrency, though, because in this case, it's probably more clear to just use a mutex to lock and unlock your Point. Again, yes, this is sloppy and weird, but it represents the same behavior as the original program, but properly synchronized.

IMHO, there is a subtle difference between your version and the version posted.

You assuming a "real" producer-consumer relationship. But the posted code seems to imply a sort of "supervisor-worker" relationship.

The "supervisor" "snoops" on the worker's state and raises an alarm if the worker is doing something weird.

As you stated at the end, synchronizing around currentPos seems to be most reasonable solution (assuming that's possible). That guarantees that no Point method is running while currentPos's state is being examined.

I'm under the impression that the supervisor wants to stop the worker as soon as its doing something wrong; not do something like check every so often, and stop only when the supervisor notices.

Anyway, what you're describing is pretty easy too. Basically all you do is make a channel of Point, and have the supervisor read on that. The worker just does some work, and tries to send his value down the channel. If someone's listening, send the value. This is pretty straightforward, but probably not safe in the real world, because the worker just keeps on working; it doesn't wait for a confirmation that its work is ok. http://play.golang.org/p/J8Xgh8S18b

If you want to structure it such that the supervisor can stop the worker, ask the worker what it's doing, look at the data, and then reply with "ok, you can continue now", there's a handful of ways to do it. Now we're getting pretty real-world, and things get a little bit more intricate, but you can do it like this: http://play.golang.org/p/hXYNmCm8lg

the trick being that you send a channel over another channel. There's a handful of ways you could structure this; there may be a cleaner way.

That was very informative, thanks! :)

They could have written it in Java and not worried about these issues. I was writing Java code using channels to communicate between threads in 2000 (i.e. the stuff that Go has.) Back then we had to download Doug Lea's concurrency package to get this stuff. For about a decade it has been in the standard library.

Even without Doug Lea's extensions (though they did help making it more efficient and/or easier to understand), it's been possible to do this safely in Java from v1.0 -- just synchronize and notify() or notifyAll() as needed.

I think the chief problem is that the developer hadn't learned the basics of multithreaded programming yet, and didn't realize s/he needed to know them.

That's an extremely harsh way to learn a little lesson, regardless.

There are many languages where you don't have to worry about this particular issue. Clojure for example enforces immutability (which means you cannot unsafely share the reference, since the reference is immutable too) and provides safe sharing mechanisms that would prevent this.

I'd use scala actors and not have to worry about this issue. Anything that avoids shared-memory communication means you're safe from this kind of interaction.

IMO raw threads are too dangerous to use as anything other than a primitive to implement higher-level concurrency on.

You would think that equipment valued at $12m would have safety features to prevent damage from rogue commands. Interesting.

Those who do not read the history of Therac-25 apparently are doomed to repeat it.

If the hardware can be destructive under the control of software then you must have hardware interlocks, not just software safety features.

Researching the Therac-25 was on the list of topics one of my university lecturers assigned.

Also on that list (from memory).

* AT&T's switch cascade failure * Mars Climate Orbiter * Ariane 5

He also took us through some bugs in code he'd written when he was doing massive telecomm's systems.

I came away with a few lessons.

* If it can't fail, test it twice. * If it's designed so it can't possibly fail, test it three times. * All code is buggy.

> If it's designed so it can't possibly fail

When faced with this kind of requirements, I always lift my eyes right above my screen, where it is written:

> The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at or repair.

And for emphasis, next to it is a poster of a nuclear island cut-out.

The stereotypical example of something that "couldn't possibly go wrong" for our electrical division was the interconnecting wiring embedded in the case that simply connected point A to point B.

Sure enough, we once had a motor controller break where the problem was interconnecting wiring...

Huh. I learned a different lesson from Therac-25: "never write mission-critical software." :)

For those who haven't read it, http://sunnyday.mit.edu/papers/therac.pdf

There are three kinds of programmers:

1. Those who read about Therac-25 and decide never to work on any systems where bugs could threaten lives (your chosen course, and mine).

2. Those who read about Therac-25, get frightened, and work very carefully on such systems.

3. Those who read about Therac-25 and think, "Those idiots! Good thing I'd never make a mistake like that."

The prevalence of (3) in all professions and walks of life is kind of frightening.

You should walk through a building construction project sometime. Your "all professions and walks of life" part is ... disturbingly accurate.

It no doubt applies to doctors too, which is even more disturbing.

Not as disturbing as finding this attitude in hospital construction projects.

Your 3 is a bit more complicated, I think:

If you dig into the details of why its programmers and their managers killed people, it's very easy to say "I'd never be that irresponsible", but still screw up with much more subtle bugs.

They may have been (2) and just weren't sufficiently careful. The inclusion of (3) isn't meant as a criticism of the Therac team, but rather humanity in general.

"My comments were not aimed at the Therac team, but at humanity in general."

Don't be offended, I'm insulting all of you equally!

I learned not to write life-critical software. There is no bringing back the dead. Corporate mission-critical software (unless said corporation is power, weapons, transportation, or healthcare related) can usually only lose money.

The upside to writing software for healthcare is that your code might help people recover faster or in some cases survive where they otherwise wouldnt.

But it can be stressful to write software that has a very direct impact on peoples lifes. While I was rolling out an updated version of an inhouse application a serious bug was discovered. The application was used as part of chemotherapy follow-up/planning and on some occasions it gave the wrong answer for a test. I discovered that the bug was old and that wrong answers had been given for a long period. Personally I was relieved that I had not caused the bug, but from a medical perspective it would have been better if the bug was new and had only affected very few patients.

My takeaway from that experience is that care should be taken when writing critical software, but also that we will make mistakes and it is very important to always focus on how we can make fewer mistakes instead of pointing fingers.

In some cases (like avionics) you can't realistically implement hardware interlocks because you don't have many clear-set criteria to go on (eg. a control surface deflection might be OK at one airspeed, but cause serious aircraft damage at another).

Stories like this make me want for all comercial software to have source code published before anyone is allowed to earn even a dollar form it's sale, support or related activities.

If companies are required to publish their source code before selling their software, what keeps a large portion of their market from simply pirating it instead?

What keeps them now? If you publish software it will be pirated. Hiding source code doesn't help with that.

Even if obligation to publish source code were to harm companies (which might even not be the case) the benefit it would bring to the consumers and whole ecosystem is more important.

Companies don't need special protection. The most important advantage of a company is that it can die without killing all people who work there and destroying all its assets. This allows companies to take risks no sane person would take if he were to be responsible with all his money (both savings and potential debt).

Given that he has the source code to the controlling software it's probably a home brew system.

He's no doubt learned some important lessons, but I can't find destroying $12 million in equipment in any way funny. [c0deporn has updated his original to the more accurate and appropriate "interesting".]

It is however very unlikely that he actually lost $12m in equipment. Most likely there was some damage done, but no where near that amount.

If someone broke a $12m machine, surely their career would be over.

If someone broke a $12m machine, surely their career would be over.

Who would you rather have working on your $12m machine, someone with no real experience with that sort of hardware or someone with plenty of experience and some painfully learned lessons about exactly how careful you have to be when working with brittle $12m machines.

Eh, sometimes you fire the guy.

There are two philosophies of justice: retributive and utilitarian. Retributive justice is about punishing the guilty. If you destroy a $12 million machine, you deserve to have $12 million taken away from you. Most people don't have $12 million, so the retributive boss will take away as much as he can: fire you and file suit if he can and badmouth you to other companies so you never work again.

Utilitarian justice is about preventing future harm. It's not about punishing the guy who pushed the button on destroying $12 million of hardware, it's about preventing future hardware losses. You fire the guy if you think he's likely to destroy more hardware in the future, because maybe he was negligent and careless or even malicious. You keep the guy if he was just in the wrong place at the wrong time, and any other competent employee would have made the same call and inflicted the same loss.

So while the "$12 million in training" line is a great ice-breaker for making the poor guy feel better, if someone broke a $1 million machine, then a $2 million machine (after all, he has $1 million of training now), then a $3 million machine, then a $6 million machine, would you say, "Great, he's got $12 million of training! Let's put him in charge of our $12 million machine!"?

This is an important point. We're so accustomed to the low-margin consumer world sometimes that we forget that equipment can be repaired. Broken physical enclosures can be replaced, dead motors can be replaced, PCBs can be replaced and in extreme cases even the SMT components on them can be reworked for much less than the total cost of production (which likely included things like service agreements already).

Why do you say so? Do you suppose that a test pilot gets fired if their plane crashes?

Lockheed test pilot Tom Morganfeld didn't. After crashing a YF-22 prototype, he was selected for the first X-35 test flight.

Another Lockheed test pilot wrecked a F-16E at an airshow practice and ejected with injuries now flies F-35s.

As punishment?

If they pilot recklessly (reckless java code?), then yes they would either not make it out of flight simulator or barely make any flights until they grounded for a long time.

actually, they just received $12M in training on how to not make that type of mistake ever again. Sounds like a great hire to me, especially if they will be undervalued in the market by others

hopefully the code they posted was not theirs.

Good point about a home brew setup. I suppose my "funny" statement was insensitive. I only meant it was an interesting situation.

It's funny because it's not me.

Without looking at it too closely, there's no synchronization on the point. The synchronized this (with an empty block - wtf -- that's meaningless) has nothing to do with the point...so the point is possibly stale. I haven't assessed the logic more than that to see if that's a possible state that would program exit, but it's the first thing I noticed. Also, this program, as others have pointed out, is weirdly complex -- and that's a euphemism.

I think the logic is more to elicit a strange occurrence (bug, dare I say?) that doesn't occur unless under some weird circumstances. In this case, this behavior would be possible unless the value of a variable, currentPos, is set to an object before that object's constructor finished, and all in the same thread!

The only possible explanation is that the JIT decided to reorder the execution so the variable was set to memory allocated for the object, and then ran the objects constructor.

It's all very surprising behavior.

I don't believe the synchronized(this){} block is entirely meaningless. Essentially it acts as a gate where execution will halt until all other synchronized(this) blocks are complete before continuing.

That's not to say another concurrent access can't sneak in between the block and the next statement, of course, but it does change the behaviour somewhat. But I can't really think of a use case.

As a side effect, it may also act as a memory barrier.

That program is shocking. I'd be interested if there was even a need for threading, perhaps you could have just time stepped your actors with a simple Tick interface.

for all actors actor.tick(timeDelta);

Side question: with $12M of equipment, wouldn't it make sense to have an independent limiter of some kind to prevent damage? Is there something about this equipment that makes that greater than $12M in difficulty? Because relying on having no bugs just so your hardware isn't destroyed seems like a bad idea...

Well, $12m breaks my personal best, but the independent limiter is only delaying the inevitable. You'll have just as many bugs in your hardware as in your software:

1) Include software controls to limit motor position 2) Include hardware kill switch to prevent collisions. 3) Drive unit to the outermost SAFE motor position. 4) Motor limit drops into worm gear 5) Send unit back to original position. 6) Unit doesn't move as worm gear eats limit cable. 7) Since the unit didn't actually move, the software limits are now compared with faulty data. 8) Send the unit out to the far edge again 9) Since the software positions are incorrect, the software limits won't help. 10) Since the limit cable was eaten by the gear, the contact switch won't help. 11) Hear super-mirror array drive into concrete wall.

Of course, having a hardware limit switch the trigger on an open circuit would have fixed that problem. Until the time that a faulty solder joint shorts the switch and the mirror drives into the wall.

So you can add in a hardware encoder to check the physical position. Which saves you until the unit is manually moved next to the wall between runs without updating the position in software. Since the encoder only gives relative position, the software limits, encoder, and shorted limit switch still gleefully watch as you again drive the super-mirror into a wall.

I'm willing to bet the question has a fabricated back-story. If you look at the user's question history: there are some interesting ones, but they all could be justified as a collegiate homework problem.

Also, $12 million for an electron microscope seems somewhat on the expensive side, by an order of magnitude.

The poor bastards might have had the entire reticle set (photo mask) set for fabricating a microchip loaded in the microscope for inspection.

It would absolutely make sense. However, my guess if the story is true would be that one (simpler, cheaper) device caused damage to other, more expensive equipment, in a way that was not foreseen. (Another similar scenario: A software controlled pump not stopping like it should, causing an entire lab bench to be flooded with corrosive liquid).

Unfortunately non-software guys often have a lack of appreciation for how subtle some software bugs can be. Re. the mentioned Therac-25 incident in this thread.

Breaking a $12M piece of equipment could be as simple as blowing a $5 fuse. Most expensive equitment I've worked with (in the $100s/$1000s range) seems to be designed so that the expansive components themselves are pretty well protected, and if you break something it is generally relatively cheep to fix.

THIS! is why you should always write tests.

Actually, concurrency is one of the areas where tests aren't particularly reliable. Whether the tests pass might depend on how the threads happened to be scheduled on that particular run.

Tests don't always catch everything, especially in multi-threaded code. Something more was needed in this case.

To those who disagree with me. Testing was needed, and is reliable, not saying there is something in place that would have solved this particular instance but saying "not particularly reliable" means that you haven't properly addressed the problem. If you can't write a test that tells you your code works, then how can you as a human verify it passes all required cases manually reliably.

YES, I have written tests in multi-threaded situations before. Your aversion to trying it doesn't make my statement wrong.

Reply on news.ycombinator.com