The undocumented Android change that led to aCropalypse was reported during beta
iliana.fyi
iliana.fyi
Looks like the ball was first dropped when the issue was deferred without acknowledging the reporter's concise suggestions of how to handle it, engineering-wise. Nor a sign that existing code was reviewed for the same problem.
Then looks like the ball was dropped again, in what I'm guessing might've been: "old version, forget all the old open issues, all the issues that still apply, someone will rediscover the hard way".
This "aCropalypse" event is an example of how making the wrong triage of an issue report can turn out very expensive. All the costs to the world of aCropalypse could've been averted. Some of those costs might eventually come back to the company.
It's easy to guess why the problem happened and the report wasn't handled responsibly. By the time the problem exhibited in a way that couldn't be ignored, the pertinent metrics/KPIs/OKRs about clearing issues, resource allocations, and product shipment schedules were already in the past, bonuses had been paid, promotions for shipping new things earned, etc. And security problems are treated as inevitable, even if there's a constant stream of them, and they're produced faster than they're fixed.
We're going to need software engineers to be accountable for things we've done and signed off on.
(Bonus if accountability changes happen near-term: GPT-powered open source laundering gets a pause, because bridges start falling on everyone foolish enough to sign off on that mangled statistical plagiarism.)
Management first. It's not engineers setting incentives.
If a civil engineer signs off on a bridge and it collapses during rush hour, he doesn't get to just say, "Whelp, boss rushed me, blame him."
Imagine if all buildings had to constantly be retrofitted to support a new structures on, below, inside of, around, and next to them, and every retrofit had a different engineer/architect/etc. No civil engineer would ever sign off on anything.
I think even a small amount of accountability would definitely start some much-needed conversations at software companies about things like "do we really need to retrofit this thing onto that working thing?" and "do we fully understand our dependencies and the consequence of merging from them?" and software quality would be better for it. If the consequence of a bug is low, go for it--the accountability should be low, too.
EDIT, just to add another example. I remember when I felt rushed and checked in some client code that did not implement exponential backoff for request failures--it just retried. Of course that code made it to production and our customers started DDOSing our server when a brief outage happened. I felt terrible about it and honestly believe I should have been fired for it. Yes, management rushed me, but they're not the ones that wrote the code. They're not the ones that made an amateur-hour error. It was me.
If you felt so strongly about it, you could have quit. Why wait on management to fire you?
It's fairly easy to blame engineers for everything that goes wrong. Engineering doesn't work in a vacuum, the system and processes have to have guardrails in place that prevent lapses in quality, including security, from reaching the customer. When such guardrails have been systematically dismantled in favour of "improving productivity", the management - which removed the guardrails - becomes the culpable entity.
I've seen a few houses built on top of apartment buildings though.
There’s your problem. You can’t constantly change software contracts without causing bugs. If you’re liable for the code chances are things will slow a bit. Good! We need to develop software slower.
You can't write software without causing bugs, static requirements or not. The software that goes into air/space-crafts is written this way, and it still breaks occasionally. Sometimes bugs are circumvented with hardware because it's cheaper and more reliable than making the software change.
Buildings have fault tolerances. You can use the wrong amount of something, or put something in the wrong place, and have it still work. If buildings had to be designed and built exactly corrects or they fell apart, we wouldn't have buildings.
> We need to develop software slower
To each his own. Most software is fine having bugs. I'd rather have a modern web browser with a few bugs than have an ascii-only command line browser that works flawlessly. The U.S. economy would agree with me there. The software in your pacemaker or car, sure, balance safety and speed to save the most lives/time, but most software operates in domains with far lower stakes.
He doesn't but this is actually a problem. Ideally, he doesn't get to say this because he chose to rush and if he'd said "No" instead then the boss can't do anything about it. In reality, especially in the US, his boss will just fire him and hire somebody else more co-operative and he knows it.
Without a strong, legally supported and well resourced professional organisation behind people with an obligation like this, they will get exploited and then blamed by people who are insulated from the effects.
Do you see such an organisation for software engineers in the United States of America? Or even most of Europe? Because I do not.
The problem is that medicine and law move orders of magnitude more slowly than software, at the moment. Software is evolving unbelieveably faster, and its problems are far more diverse. As it settles down, if it ever does, we may be able to calcify roles more formally, and then regulate those roles. But at the moment it's the wild west.
I'm willing to bet that the civil engineer in that case has much better whistle-blowing protections, or at the very least, some comfort that if they get fired and someone else replaces them that's willing to do it... that person will be the unhireable one in the future.
This is already illegal. Holding the makers of things more accountable would do nothing for you in this example.
Which is weird. Your best people should remain engineers. Have them indeed sign off on these things.
I agree; the problem with an engineer track is that the salary ceiling is too low, and you have to move to management to get higher salary. but then you end up not using your engineering skills, and waste potential.
What needs to happen is a way for people management to be done without a hierarchical structure. Aka, the senior engineer sits not under their manager, but that the manager is subservient.
Joel spolsky have written about this "problem" already in the last couple of decade: https://www.joelonsoftware.com/2009/03/09/how-to-be-a-progra...
> To make sure that the debate happens respectfully and on a rational basis of facts, it’s absolutely critical that the program managers and developers be peers. If developers report to the program manager, at some point during the debate the program manager is going to get sick of the whole thing and just say, “OK, enough talking, now we do it my way.”
The much worse idea is to promote BAD engineers to full time management positions, which seems to be the way most sub par and mediocre companies do it right now.
Non-licensed engineer are not liable in the same sense, their employers and PEs are. Same thing could apply in software, but people whine too much and it probably won't happen. Every time the topic comes up, people chime in with "But software is different. It's hard." and expect that lame ass excuse to be sufficiently persuasive. Software is also critical, not all, but a lot. Critical systems warrant additional scrutiny and liability.
I'm not against some well-thought-through standards for certain safety-critical applications, but it's got to be a lot better than "maybe code would be better if we threatened to throw engineers in jail for contributing to complex and subtle interdependent bugs that caused a privacy bug that didn't have a significant real-world impact".
Not trying to be facetious, but... what costs? It is unlikely that this cost the company any substantial amount. It probably wasn't even a whole lot of extra work for the PR department, as the bug didn't generate mainstream headlines.
By costs to the world I'm thinking all the things that happen because something a user cropped out was revealed because it wasn't really cropped.
Hypotheticals to illustrate costs to the world:
* A company loses a big business deal because IP/plans leaked to competition in reverted-crop.
* A medical provider and personnel get hit by lawsuit and/or HIPPA for patient data de-anonymized due to reverted-crop of an image.
* Someone gets harassed at their workplace when a reverted-crop adult photo revealed their identity and gets circulated among colleagues.
* Semi-automated extortion rackets, going through troves of images, looking for revertible cropping.
* Other businesses caught in the middle of this have to expend resources to mitigate, or even deal with liability for depending on or assuming correctness of third-party tech behavior, etc.
Maybe luck is with us, and not a single instance of harm to the world actually happens. Or maybe there are many such instances of harm to the world.
Regarding costs to the world eventually coming back to a company -- speaking in general, not of this particular situation -- I think that could come in forms including: brand damage, lost business, lawsuits from those harmed, undesirable turns in legislation, regulatory fines, etc.
So ... yesterday it turned out that Microsoft had the control over a certain Bing's search results wide open to anyone with an Azure account, and that enabled anyone to get access to and control over the Azure services of everyone that used Bing for these specific searches. There were probably other ways too, but that hasn't been disclosed or investigated. And it is --of course!-- unknown if this attack ever happened in real life: there were no logs.
That's quite the problem, wouldn't you say? Well, did you see mainstream articles? There was only an article in The Reg, which attracted a mere 11 replies. It simply doesn't get attention, probably because it was framed as "a misconfiguration", yet the costs could be staggering.
Only if management is held equally accountable.
I've worked at places with dedicated security teams, that acted as a 'buffer' between management and engineers. Those places tended to take security seriously.
Places -without- those, I noticed that management has a tendency to sweep security issues 'under the rug' since they impact deadlines.
And see them evaporate from the companies that would implement something like this.
Personally I would neve take a contract where I could be held legally responsible for bugs.
And ultimately it is the owner who let the product out in a wild without spending a year testing billions of permutations first. Try holding them responsible and see how far can you get.
The frivolity with which "Engineer" titles were co-opted by tech has always been distasteful to me. It felt intended to claim all of the prestige with none of the rigor, accountability, or social responsibility.
Broadcom exploits over the years would disagree. So would Samsung's hardware encryption. So would TPM. So would BadUSB. So would everything related to Bluetooth. Et cetera.
No matter what kind of engineering you do, don't try to predict the future. Engineer towards ease of maintenance and ease of update/deployment instead.
None of the engineers I have hired or worked with assume such liability directly.
Usually, their firms limit liability to the price of the contract. I assume insurance does otherwise.
It might be doable, but would also require (at a minimum) a lot more compensation to make up for the increased liability. Which gets to the heart of the problem, it’s a lot more expensive to produce higher quality software.
How quick is a manager at deprioritizing quality to secure a deadline?
So reply to this https://news.ycombinator.com/item?id=35164665
After cloning astrovim as `~/.config/nvim`, do similar with https://github.com/AstroNvim/user_example but with button "Use this template".
Then in `~/.config/nvim/user/plugins/user.lua` add
``` return { { 'Mofiqul/vscode.nvim', priority = 1000, config = function() --vim.cmd.colorscheme 'vscode' vim.o.background = 'light'
local c = require('vscode.colors').get_colors()
require('vscode').setup({
-- Alternatively set style in setup
-- style = 'light'
-- Enable transparent background
transparent = true,
-- Enable italic comment
italic_comments = false,
-- Disable nvim-tree background color
disable_nvimtree_bg = true,
-- Override colors (see ./lua/vscode/colors.lua)
color_overrides = {
vscLineNumber = '#FFFFFF',
},
-- Override highlight groups (see ./lua/vscode/theme.lua)
group_overrides = {
-- this supports the same val table as vim.api.nvim_set_hl
-- use colors from this colorscheme by requiring vscode.colors!
Cursor = { fg=c.vscDarkBlue, bg=c.vscLightGreen, bold=true },
}
})
require('vscode').load()
end,
},and in `~/.config/nvim/lua/user/init.lua`:
``` colorscheme = "vscode", ```
Example here https://github.com/rofrol/AstroNvim--user_example
It's going to keep biting them in the ass. The prick who heads up Golang's security has refused to allow people to control what encryption algorithms are enabled to disabled in Golang's TLS 1.3 implementation (he was one of the designers of the TLS 1.3 spec) because, and I wish I were joking - he feels TLS 1.3's implementation is perfect and nobody would have a reason to, and even if they do, why, they can just wait until google releases the patch to fix it, and then deploy that change.
Because of course that's how production environments work....you can just deploy a new version of the language you're using, instead of, you know, a config file change or a few lines of code.
You start with management and all the way to the top including the CEO and CTO liable for criminal charges.
And, no we're not going to need any of that.
Software is written, like blogs, novels or laws, not engineered. That’s wrong.
Imagine if the toxic waste that is the mental impact of social media, or forever insecure devices littering the Internet, or any number of things was obligated to the companies to do their best to fix it. Nuclear reactors put huge money away so that when they are done they are bulldozed back into a green field (Other than what was supposed to be a solved problem of spent fuel).
Anyways, the likelihood of regulation is nil and a global world means those negative externalities will continue on churning till the world agrees we suck at managing risk. Which is also nil. Mental health crisis and insecure tech just isn't as flashy as nuclear in terms of getting regulated in the interest of public health.
Risk management (especially with the js people) seems to be "lol what's that?"
Also we have the C cowboys pretending they can manage the complexity of their "simple language" in a complex world
How many use cases prefer not truncating files?
A day in my life at my last job. Basically a regular occurrence that it was “urgent” after a customer report.
This is the bigger problem, there will be very little real cost to the company or engineers involved. My guess is that the cost for engineering a fix and blogging it about it falls into the buffer PR spend or security engineering time. If anything the company will learn from this mistakes for a short time and it will actually save them money in the long run.
The vast majority of software used today has very little legal liability. The industry has perfected the art of legal lip service. There is a massive growth at all costs mentality throughout the industry and engineer’s personal pride only goes so far.
If you want to see real change then you need government and the judicial to enforce real legal liability onto software engineers and companies. That would result in real cyber security rather than current wack-a-mole fines or government’s trying to promote or enforce best practices etc.
Not only does my notes app, which has worked flawlessly for years, now crash when I accidentally use one of those characters in a file name / note title. (Unfortunately, the app no longer receives updates.) But suddenly my entire ebook library (which follows the name format "first author & second author: title (year)") no longer synchronizes with my phone (running AOSP / GrapheneOS), either. Fantastic.
[0]: https://android.googlesource.com/platform/packages/providers...
/**
* Sanitizes given name by mutating the file name to make it valid for a FAT filesystem.
*/
Also a case of violating the Principle of Least Astonishment [0]. How many people would expect that when you write a file you would also have specify that you want it truncated?
[0] https://en.wikipedia.org/wiki/Principle_of_least_astonishmen...
Examples of ‘Open’ calls that require setting a bit in the ‘flags’ argument to truncate an existing file:
- https://learn.microsoft.com/en-us/windows/win32/api/winbase/...
- https://man.freebsd.org/cgi/man.cgi?query=open&sektion=2&n=1
System calls are special low-level interfaces. The requirement for reading their docs carefully and general care is expected at that level.
A higher level API shouldn't contain gotchas like this.
- If you don't specify a directory, Windows will look for the file in a bunch of directories, but the current directory is not the first place searched.
- The lpReOpenBuff is annotated as [out], but is an input when uStyle contains the OF_REOPEN flag.
- The function is named OpenFile, but can be used to delete a file. Even better, the documentation is not clear what to do with the return value on success: is it a valid file handle that must be passed to CloseHandle, or just a dummy non-error file handle?
- You can also use it to check the existence of the file -- the documentation says it will open and then close the file. What's the return value in that case? Is it a stale file handle, a dummy, or do you still have to close it?
If you want a prime example of a layer violation, this is it.
Its replacement “CreateFile” (https://learn.microsoft.com/en-us/windows/win32/api/fileapi/...) is a lot saner, but still has this:
“If you rename or delete a file and then restore it shortly afterward, the system searches the cache for file information to restore. Cached information includes its short/long name pair and creation time.”
That’s intended to make it easier for “Save” operations in document-based applications to keep the short name of the file, creation time and file flags around.
“Never attribute to malice that which is adequately explained by stupidity.”
Won't Fix Working As Intended
Declaring Bug Bankruptcy
Google doesn't test. Google relies on the public to test its software and even then most bugs don't get addressed. If the bug is a gap in the spec or open to interpretation it won't get fixed. Spec bugs are not allowed.
The reason that Google has such a high hiring bar is that most of its processes run open-loop and it relies on having smart thorough people to prevent the large swath of issues from being created in the first place.
But there is a large swath of issues. Just look at Google's public bug trackers.
And Google's atrocious practice of auto-closing old bug reports partially covers up how many issues there really are.
Full filesystem/service-engaging integration tests are helpful but far from exhaustive - here, one wouldn’t have just needed to read the cropped file using standard display systems and count the pixels, but hash the underlying file in its entirety.
For critical libraries in an I/O pipeline, encourage team members to read CHANGELOGs - both before choosing a library to make sure they’re maintained, and closely when even doing a minor upgrade.
The original NCSA Mosaic one with the globe could've been a bit much, but it was appropriate, given the impact of the Web, even for pre-Web Internet natives: "OMG, this is a tingly step forward in distributed global hypertext, it's happening right now -- interacting around the world, and also superpowering the world." With implied altruistic and benevolent goodness.
(The earlier Mcom/Netscape throbbing "N", when I guess they were still rebranding Navigator from Mosaic, looked silly even at the time. Then they did a good one. And there were a bunch of creative alternatives, including some easter egg ones, which JWZ cataloged at one time.)
I believe others will start from the beginning of the file and parse each local directory header and 1) build up their own central directory truth, or 2) stop once they've reached the start of the central directory.
I recall that the footer-only format was to enable streaming out a ZIP file esp. when rewinding to the front of the file/stream was not possible, like sending over a network - at least that's one of the use cases I recall.
But that's a lot less than if the central directory was at the beginning and the code had to grow the central directory and shift ALL the compressed data + local directory headers down.
I'm not entirely sure what storage system that would make sense on, but presumably there was one back in the day.
Better than cassette tapes though...
I feel like a lot of people in this thread are missing a crucial part of the historical context:
ZIP files can be split. That is, a single ZIP file can span more than one floppy disk. The entry for each compressed file in the central directory, which is always on the last floppy disk, has a field which says in which disk the compressed file can be found. If you had a multi-disk ZIP file, and wanted to extract a single file from it, you inserted the last disk, pointed the unpacking software at it, and it would tell you which disk(s) to insert (in sequence) to extract that file.
A lot of the strange decisions of the ZIP file format make sense if you think about that use case: you have a directory full of files, which you want to compress into a set of floppy disks; and once you have that set of floppy disks, you want to extract a subset of the compressed files. You also want to be able to extract the undamaged files even when the last floppy disk (with the central directory) is damaged.
atomic move semantics are the single thing that makes writing basic filesystem interactions plausibly deterministic. anything else is just pretending that errors never happen.
This seems to be the original PR for the API in question, ParcelFileDescriptor#parseMode():
https://android.googlesource.com/platform/frameworks/base/+/...
+ public static int parseMode(String mode) {
+ final int modeBits;
+ if ("r".equals(mode)) {
+ modeBits = ParcelFileDescriptor.MODE_READ_ONLY;
+ } else if ("w".equals(mode) || "wt".equals(mode)) {
+ modeBits = ParcelFileDescriptor.MODE_WRITE_ONLY
+ | ParcelFileDescriptor.MODE_CREATE
+ | ParcelFileDescriptor.MODE_TRUNCATE;
+ } else if ("wa".equals(mode)) {
...
But we jump ahead a few years and the code is reworked.
from https://android.googlesource.com/platform/frameworks/base/+/...:
public static int parseMode(String mode) {
- final int modeBits;
- if ("r".equals(mode)) {
- modeBits = ParcelFileDescriptor.MODE_READ_ONLY;
- } else if ("w".equals(mode) || "wt".equals(mode)) {
- modeBits = ParcelFileDescriptor.MODE_WRITE_ONLY
- | ParcelFileDescriptor.MODE_CREATE
- | ParcelFileDescriptor.MODE_TRUNCATE;
- } else if ("wa".equals(mode)) {
...
- return modeBits;
+ return FileUtils.translateModePosixToPfd(FileUtils.translateModeStringToPosix(mode));
}
from https://android.googlesource.com/platform/frameworks/base/+/...:
+ public static int translateModeStringToPosix(String mode) {
+ int res = 0;
+ if (mode.startsWith("rw")) {
+ res |= O_RDWR | O_CREAT;
+ } else if (mode.startsWith("w")) {
+ res |= O_WRONLY | O_CREAT;
+ } else if (mode.startsWith("r")) {
+ res |= O_RDONLY;
+ } else {
+ throw new IllegalArgumentException("Bad mode: " + mode);
+ }
+ if (mode.indexOf('t') != -1) {
+ res |= O_TRUNC;
+ }
+ if (mode.indexOf('a') != -1) {
+ res |= O_APPEND;
+ }
+ return res;
+ }
A dev was careless, plain and simple.
Having code in a monorepo doesn't help if you don't check how your code changes affects consumers/clients of the code.
If they couldn't change translateModeStringToPosix() behaviour, then the dev could've amended parseMode() to map "w' to "wt" to keep the old behaviour (ugly and non-orthogonal, yes, but that's the way the API was designed).
Original code would only accept one of these modes: "r", "w", "wt", "wa", "rw", "rwt".
Anything else, and you'll get an IllegalArgumentException exception.
which matches the docs:
from https://developer.android.com/reference/android/os/ParcelFil...
Parameters
mode String: The string representation of the file mode. Can be "r", "w", "wt", "wa", "rw" or "rwt".
Throws
IllegalArgumentException if the given string does not match a known file mode.
The modified parseMode code will only throw the IllegalArgumentException if the mode string does NOT start with "rw", "w", or "r".
You can append any other character(s) to the mode string ("a", "t", "x", "y", "z", your fav emoji char), and you won't get an IllegalArgumentException exception now.
I wonder what happens when open() gets a mode flag with both "a" and "t" set?
parseMode API contract thrown out the window...
Nothing I expect. O_TRUNC and O_APPEND are not incompatible, O_APPEND is just redundant (because by definition if you truncate the file you're writing at its end).
macOS certainly doesn't seem to care when I do it.
No, O_APPEND is not redundant with O_TRUNC. When you use O_APPEND, you also have the extra behavior that every write goes to the end of the file, instead of the current position; this is important when more than one process is writing to the same file.
It most definitely does help with that because there's fewer places to look. But if you only look at your monorepo then you'll miss a lot of things if the API you're touching is public. In this case there were consumers in the same monorepo were there not?
The issue here is that someone dropped the ball on backwards compatibility. That happens, especially when the people involved are inexperienced.
Years ago I extended a network file system server to include additional information in an Open response (revision tag to enable distributed caching). Turns out a customer had also extended the server response in their builds, to do something else security-related. Our release startled them, resulting in a lawsuit.
It was all silly and fixed in a patch (a boot config file enable/disable of our new extended info) and it went away. But it underlines how with the best of intentions, still things go wrong.
The infamous "Bug Bankruptcy"