firebacon

But how does that result in a vulnerability? At best you can call the function with garbage input (negative number) and still receive a valid buffer from malloc, no?

TFA calls this a "backdoor"; so how do you actually "get in" after you managed to get the backdoor through code review and deployed into production?

Agreed; the explanation does not clarify why the first snippet is considered "totally safe" and the second one is not. How does the second snippet introduce a new security issue? A negative value for num passed into the function will always be forwarded to malloc (after going through the size_t conversion), no?

What does "mofo" mean here. Can you give us a rough estimate on the transaction rate you achieved with this setup? My own experience and all independent benchmarks I can find seem to indicate a limit of 100-1000TPS on reasonable hardware.

Note that you can "batch" up many inserts into a transaction, which gives you a high "nominal" insert rate but still only ~100 actual transactions or so per second. To see why this is not the most useful number when comparing to a database like Postgres, consider that the limiting factor in a SQLite/Posgres design are cache flushes which outweight the costs of actually writing the data, so the number of rows per batch is mostly arbitrary; using this metric you can always claim a huge insert performance by choosing a suitable N. Also, if you do the batching, you of course loose some of SQLite's consistency/durability guarantees for your writes, which is probably fine if you didn't need them in the first place, but begets the question if an embedded ACID database is the best tool for the job at hand.

The less flippant explanation is that SQLite can only handle a single writer at any time and when you try to access it with two concurrent writers (or a concurrent reader and writer in some modes) it will by default return a "BUSY" error instead of blocking.

So, if you're were getting unexpected "BUSY" erorrs than, yes, you would be using it incorrectly. However, AIUI, you are always expected to see some amount of BUSY errors during normal, concurrent operation and have to deal with them explicitly. So the fact that you're seeing BUSY errors alone doesn't mean you're doing anything wrong...

To use SQLite correctly from multiple processes, you have to do one of two things:

  - Add explicit code to retry on BUSY errors everywhere you do SQL queries
  - Serialize all access the database, e.g. by using a mutex

GP appears to have chosen the second option.

It's hard to point those out without coming across as a hater, and because they might seem so obvious. Trying...:

With the way transactions and durability work in SQLite, a normal setup will not do more than on the order of ~100 transactions per second without adding additional layers of complexity. Which is completely fine for it's intended usecase of being an embedded, lightweight database and of course is not due to some bad engineering decisions, but due to inherent tradeoffs in how transactions are handled.

So basically it's all fine until you try to use SQLite for something for which it is not a good fit, like a large volume of inserts. Other solutions exist that are conceptually better equipped for this usecase out of the box (Postgres/MySQL/Elastic/etc). Not a shortcoming in SQLite per-se, but rather a practical shortcoming that comes up when you try to use it for the wrong job.

Case in point is TFA which takes the theoretical SQLite database size (140TB) limit and runs with claiming that is the useful limit on how much data you can store in SQLite. LOL! That value is when SQLite page IDs start overflowing and not an estimate on how much data can be usefully handled in a single SQLite database.

SQLite is, by the nature of what it is, not a good fit for big data volumes. Try loading 100TB of data into SQLite and try to run even a single query. Even assuming that you have a hyper-fast SSD, a single query will take days to complete! But it still get's mentioned as a potential solution for that problem every so often. In this post even...

So you could consider those practical shortcomings or ill-advised usage. It depends on the definition/perspective I guess. At any rate, SQLite, even if generally being excellent, is not the panacea as which it sometimes gets sold here on HN. No database is, there are just too many tradeoffs involved.

My point was less about arguing which is more popular, but more that the same claim can equally be made for at least hundreds if not thousands of other software packages. Still, SQLite stands out to me as the only package I can recall that consistently puts this message all over the documentation front and center. Which is definitely still a useful signal, even if we both agree that it is, in fact, widely used.

They charge money for it...

Note that I'm not saying all of the test suite should be open source. It's clearly a valid/cool business model and SQLite is, for a lot of use cases, an excellent piece of software that I have often used myself. The criticism was honestly more directed towards the cherry-picking of facts that sometimes happens in discussions on hacker news, especially those related to SQLite.

No, I'm referring to the second section.

> MOST WIDELY DEPLOYED SOFTWARE MODULE OF ANY KIND? -- SQLite is probably one of the top five most deployed software modules of any description. Other libraries with similar reach include: zlib... blah blah blah

It stops just short of claiming it's the most deployed software in the world...

That completely depends on the chosen metric/definition though. For example, you could consider the most used database the one that handles the most queries per day. Not so clear who is the winner now. The claim that SQLite is one of the 5 most widely deployed pieces of software worldwide is completely unsubstantiated and most likely untrue once you count in things like ICU, the linux kernel, etc.

About the well-tested bit: Per it's own documentation, SQLite has a massive test suite. [*Not all of] the test suite is actually open source though, so the overlap between commenters selling you on how well tested SQLite is and those that have seen how the sausage is made is probably [close to] zero.

However, pointing this or any of the other practical shortcomings of SQLite out on hacker news is blasphemy and will invariably get you downvoted into oblivion by people who (apparently) never ran into them.

[Edited] Two Generals does not apply since it deals with the problem of guaranteeing that two parties agree on the exact same consistent state after a finite amount of time/messages. MQTT's "exactly once" delivery is basically an eventually consistent mechanism -- it just guarantees that the message is eventually delivered to the broker which then does some simple deduplication. Eventually may be infinite. So, no violation of the space time continuum here...

In other words, MQTT's exactly once delivery conceptually works as if the server would store a big hashmap of all message IDs it has seen once and the rejecting any duplicates using that hashmap. The client the simply keeps retrying with the same message ID forever. This guarantees that the message is received by the server "exactly once". The actual protocol is more involved, but the details are basically just an optimization to prevent the server from having to keep around the "received messages" hashmap forever.

The problem that you're thinking of that MQTT of course doesn't solve is that it is impossible to perform any externally visible action (such as moving an actuator, storing a record into another database system or detonating a bomb) "exactly once" in the general case when you take into account the possibility of failures in between the two steps of performing the action and storing the information that you have performed it. You can either first commit and then perform the action, which means you can not safely retry after having failed between the commit and performing the action, because you might have already performed it ("at most once"). Or you can first perform the action and then commit, potentially redoing the action after a failed commit ("at least once"). There is, conceptually, no way around this unless we get help from the environment, such as an atomic perform+commit primitive. To see how this relates to two generals, think about the exchange of the "performed" bit between the external system and the storage as an exchange of messages that can be lossy. We can not come up with any algorithm that guarantees that the external system and the storage always agree on the state of the performed bit after any finite number of steps.

> The calculation of accurate crash rates of this type depend on reliable counts or estimates of both airbag deployment crashes as well as the mileage travelled exposing vehicles to the risk of a crash. But after obtaining the formerly secret, underlying data through a lawsuit filed under the Freedom of Information Act (FOIA) against the U.S. Department of Transportation, we discovered that the actual mileage at the time the Autosteer software was installed appears to have been reported for fewer than half the vehicles NHTSA studied. For those vehicles that do have apparently exact measurements of exposure mileage both before and after the software’s installation, the change in crash rates associated with Autosteer is the opposite of that claimed by NHTSA – if these data are to be believed.

> Not to forget, C++ template meta-programming was discovered by accident

That's a bit of an uncharitable reading, no? For others that would like to make up their own mind: https://softwareengineering.stackexchange.com/questions/1253...

Technical details seem to matter though. Using the definition of "paying taxes" that TC seems to be using, you could have always paid your taxes using, for example, magic the gathering trading cards. You just need to find somebody that will buy them from you for USD first. I wonder why one of those made the headlines now and the other one didn't...

Seems like fake news. The state of ohio will not accept bitcoins -- they just "allow" you to pay your taxes using bitpay, which is a service that will convert your bitcoin to USD on the fly. How is this any different to what you could always have done, i.e. selling bitcoins for USD first and then paying your taxes with the USD?

Wow, so many negative comments here. I for one applaud the move -- a license like this has been needed for a long time.

Slightly (un)related, but I don't understand why in a forum full of software developers it is the consensus that all infrastructure software must be free (as in beer)? What are you guys planning to live off once that dream has finally been realised?

Very interesting! Is this kind of command capability (e.g. ability to modify memory contents) something that is usually only available on "non-critical" subsystems, or would you generally expect to also find it on critical components, like the communication or navigation modules?

Thanks - that was exactly the kind of info I was looking for!

Amazing that they had the ability to just run ad-hoc LISP on the spacecraft. It appears their method to ensure safety in the face of arbitrary code execution was to divide up the spacecraft into isolation zones and run the parts that have a REPL on a non-essential CPU. From [1]:

> To protect the main DS-1 mission from possible misbehaviors of RA, the design included a “safety net” that allowed the RA experiment to be completely disabled with a single command, issued either from the ground or by on-board fault protection.

[1] https://ti.arc.nasa.gov/m/pub-archive/176h/0176%20(Havelund)...

Thanks for the reply! So what you're saying is that it's just a "normal" over-the-air software update. I.e. you add some new functionality and then do a full system test of all functions of the software before replacing the entire image?

That makes sense, but is almost a bit disappointing. After all, that is exactly how it works for the boring systems here on earth. From various wired & co articles I had the impression that there was possibly something more; a mechanism that would allow users to send elaborate "commands" to the spacecraft to perform "ad-hoc" tasks at runtime. (What I mean by "ad-hoc" tasks are tasks that are unknown at the time of validation/testing of the software.)

Remote updates -- where you replace a full (sub)system -- are one thing, since you can always run the normal software validation procedure on the new version of the software. So an OTA update of a system (even in flight) does not sound like rocket science (yet)...

But: Once you include a REPL or another mechanism to push and execute arbitrary code "ad-hoc", I wonder how that could possibly be tested an validated? Surely as soon as you add the ability to run arbitrary code, there is no way of testing for all possible states of the system as part of the validation process?

In other words, how do you allow the user to push arbitrary code, but prevent them from putting the spacecraft into a condition from which it can not be recovered? The only way I could naively think of would be to only allow the user to push code to a completely isolated CPU that has a remote-reset functionality from the main/comms CPU.

Still, the popsci articles I read made it sound like there might be more to it. It would be excellent to find some first-hand accounts/sources on how this looks like in reality.

The part that I find the most intriguing is "corrections can be made on the fly".

I can see how you would ensure reliability through proper requirements specification, a good software development process, separate independent implementations and extensive verification.

However, every time I read a popsci article about space flight software, they talk about this capability to push new code to the spacecraft while it is in flight.

I'm really curious to learn what this looks like in practice (technical details). Do they really have the ability to do an "ad-hoc" upload and execution of arbitrary code on these systems? If so, how are the ad-hoc programs tested and verified?

Off-topic, but do you actually run ad-hoc SparkSQL queries on the whole dataset sometimes? Are the logs actually stored as text files on disk? How long does such a query take and/or how many racks of machines do you need for that? Should be more than 1 million drive-seconds just to read the data from disk, right?

Why would that help -- genuinely curious? Shouldn't a redis server already not be bound by the secondary storage I/O speed? I thought it was a main memory system with asynchronous commits to disk?

I would not consider setting an exit code to be a fancy feature, but I guess we are living on the bleeding edge here :)

EDIT: I should have said explicitly in my initial comment that I knew about std::process::exit and panic!, but did not consider them to be a clean solution for exiting the program under normal circumstance -- more of an abort() mechanism.

Oh nice! Will start using that as soon as we get to 1.26 :) -- currently stuck on 1.24 (upgrading takes a bunch of work since we're building under openembedded/bitbake, so I wait until the new version hits the upstream layer)

We started using Iron, but later switched to just using hyper because that seemed easier and like a smaller API surface to target. I'm not really working on a web application so I need zero of the routing/web features of these frameworks.

I do need a lot of pretty specific other features though. Unix domain sockets are currently a must (luckily that works with hyperlocal).

The code runs on a pretty resource constrained system so I like the low level of control that hyper gives me of the request/response chunking behaviour (I can not afford to buffer large requests in memory and what exactly I do with the body payload differs between API calls).

One thing that would be lovely is proper TLS/HTTPS support. I need support for x509 client certificates and I need to get access to the full client certificate data, ideally including the full chain that signed it from the request handler. I have to admit that I currently run a hacked together nginx in front of my app and put this stuff into HTTP headers (hence the unix domain sockets) because I could not get it to work with rust ecosystem libraries.

EDIT: The TLS stuff is also true for the HTTP client case. Currently using the libcurl rust binding because it's the only thing that implements all the features I need (--cainfo, --cert, --key, --resolv). Also it needs to run on OpenSSL or something else that supports x509v3 extended attributes (subtree constraints).

> I'm not sure what your point is with `env_logger::init`

That most of my programs have some global (i.e. "for the runtime of the program") stuff that is setup at the beginning of main. And that some of that might want to Drop when the program exits, for example to delete a temporary directory. Now, if I want to return a non-zero exit code I can not do so while still getting all this global stuff destructed correctly (or use a workaround like having a wrapper-main).

> Calling `std::process:exit(1)` at the end of main is identical to `return 1`.

The thing is that it is actually not the same with respect to destructors -- the documentation explicitly calls that out. See also the C++ comparison in my other comment.

Yes, currently doing something similar to that, but I'd much rather have it as a first-citizen feature. Returning result directly from main sounds very interesting! I will have to look into that thx.

EDIT: I assume you mean RFC1937? Looks like it isn't implemented yet, so we will probably have to at least another year before we can get it in stable rust. But yes - without having read the entire RFC - I think that was what I was looking for!

std::process::exit does not call Drop traits (at least the last time I tried -- maybe I am doing it wrong?). So for example, if you're relying on Drop to clean up temporary files in the system, that will not happen. Providing no method to call destructors AND exit non-zero feels like a bug or missing feature.

EDIT: to expand on this: a suggested workaround is to only call std::process::exit at the very end of the main function. But consider stuff like "env_logger::init". What about that? Ok, that doesn't use Drop and if it did you could put it into it's own scope I guess - so there are workarounds - but it in my opinion that gets pretty ugly. Comparing to c++, std::process::exit or panic! is like abort(), but what I want is a "return 1" from main.

Just yesterday our build broke because a cargo update pulled in some dependencies that suddenly required experimental features. I think it was crossbeam via hyper or tokio.

Also, while I recognize it's a third party project, the hyper API keeps changing faster than I can adapt my code. If you are aware of a more stable HTTP server, ideally one that has proper TLS support and support for unix domain sockets, I would be very interested.

EDIT: on the other hand, after working with rust daily for ~1.5 years, I have only run into one compiler bug and had zero actual bugs in my application due to language/compiler updates so far. There have been a few occasions were essential features were missing from the standard library/language though and several times were we had to backport some code to an older, stable rust version that we're one (when a developer had mistakenly tested their code locally with a newer version).

The highest up on my wish list would be that - as I understand it - you can not currently exit the process cleanly (EDIT: early) with a non-zero exit code in stable rust. That makes it pretty hard to implement good command line utilities. Happy to be corrected if I am wrong on this one though.