Systems that no one will test
blog.christianperone.com
blog.christianperone.com
https://www.reddit.com/r/YouShouldKnow/comments/1wssf4u/ysk_...
https://imgur.com/a/6FaQQhb (original post before being taken down by mods)
>The Work Number is an Equifax (who famously had a massive data breach a few years back) owned database with employment and pay information.
>You can create a login on theworknumber .com and pull your report. Mine is seventy four pages and had info from every company I've worked at in the last 13 years including every paycheck I had received with the exact dollar amount (both net and gross) and hours worked. It has employment start/end dates, termination reason, details about withholdings, benefit enrollment, union affiliation, etc etc etc.
>This data is sourced directly from the HR platform your employees use (ADP, Rippling, Gusto, iSolved, etc). Equifax sells your data for things like employment background checks.
>You cannot have your data removed from The Work Number. You can freeze your report (much like a credit report) but if a potential employer cannot access your frozen report that may disqualify you.
>Why YSK: If you're interviewing for a job you can be at a significant disadvantage especially for things like salary negotiations because they can literally see how much you make including your most recent paycheck. Creditors also can access these reports.
>This is the biggest privacy violation I have ever seen and almost nobody is even aware of it. Certainly none of us consented to having our employment and income data harvested and sold, especially since we get nothing in return. At a minimum, people should know about how this data can impact you.
>Edit: The Work Number is primarily for the US, but there are similar services for other countries.
The other one to check is LexisNexus, which covers auto insurance coverage and traffic violations. In the US, gov't tracked traffic citations fall off after 5 years, but last forever in the baby credit bureaus we're discussing.
Also, INTERPOL. You never know...
FYI https://consumer.risk.lexisnexis.com/request. The Work Number gives you results immediately but LexisNexus says it's going to mail them to you.
> In my country (Hungary) you literally get a little pink book where your boss is supposed to sign the time you spent there. You keep it until retirement, the company holds it while you work there. It's also digital now but we're one foot in 1990 and one foot in 2016.
We had something similar in Romania too. Now it's 100% digital and no more delays in informing the government. The employeer needs to notify the government before someone starts work. With the paper version you had a couple of days even weeks if I'm not mistaking.
While it's easy to lay this at the feet of AI getting better at hacking. I see it as an primarily an IT issue. We've collectively ignored the lessons of history, and made do with patch jobs over poorly chosen operating systems instead.
--
We need air gaps, data diodes, and capability based operating systems. Now that I'm retired, when I have the energy, I'm working on the data diode part.
This weeks lesson for me, personally, as I try to build an open source data diode, is that the Waveshare RP2350-ETH is a horrible choice for a proxy/data source/sink, as the CH9120 ethernet interface can't do promiscuous mode. It might still be sufficient to build a data diode that can mirror a website, with << $50 component cost. Time will tell.
I completely agree that IT admin could be a lot more secure by design. Combined with better interfaces for responsible configuration.
In most cases you wouldn't need TLS. You would have something like physical secure conduit or entirely secure room. If someone gets to either end of the link you are compromised anyway.
- For ingress, you use special "file transfer" software. Run the receiver on secure side, run the sender on insecure side. It blasts the file "blind" - it has has no way to know if anyone ever received it. Make sure the receiver is fast enough, and the error-correcting codes are a great idea too, as they don't need feedback. It's up to user to want to secure side computer and check that the file was received without problems. Yeah, this is similar to sneakernet, but more secure, as you can't accidentally carry a virus on seemingly-empty drive.
- For status egress, you have secure side broadcast status periodically, say every minute. Insecure side receives the status, updates the database, and runs the regular web server to share the status. Again, secure side has no way to know if someone is listening on the other end, it just blasts out the messages and it's done.
And you are correct, if the secure system has both egress and ingress diodes, it is no longer isolated, and devious enough malware can establish two-way communications. But even if it won't save you from Stuxnet, the simple fact that it is no longer possible to have direct network connection to the outside raises security bar quite a bit - all the ideas about "let's just open this one port on firewall, it'll fine I swear" are completely stopped.
(Which reminds me of something in GP's (mikewaro) message: _why_ would a data diode need a promiscuous mode? Given every single data diode I have seen needs a special software on both sides, you should not need anything beoynd a basic TCP session)
Isn't the whole point of a diode that there is only 1 direction?
I'd take systems that were permitted to have filtered egress at this point. The lion's share of my work is in networking, so network segmentation is the "hammer" I pick up first.
Vendors gnash teeth and complain when I ask for their app's dependencies on hosted APIs and off-site resources. In the environments where I'm mandated to maintain FBI CJIS compliance I can still hold vendors accountable and get what I want. It's pretty much a lost battle in every other environment and unfiltered egress to the Internet from servers is just expected.
That's not even to get into the topic of communication flow within an application. >sigh<
On the other hand, the whole thing is largely cobbled together from various SaaS products that we use for telemetry, reporting, log management, workflow orchestration, data warehousing, etc. There's no way to ensure nothing sensitive ever gets sent to any of these services. Indeed, some of them are used for the express purpose of processing it.
So Corporate conveniently decided that all of them count as part of our system. So the data still isn't technically leaving it, you see. Even though we don't operate the software or servers, don't have any way of knowing if they in turn are sending data to still more SaaS vendors, can't verify their access and retention policies are properly implemented, etc.
There are far too many unserious people representing our industry.
Competence isn't widely valued, obedience and sales figures are. Which wouldn't be so problematic if evolution could run its course and eliminate incompetence naturally, but that's now hard to see coming to pass when we have towering circular supply chains that feed on it.
But companies still want to sell products, including to people who are at least nominally concerned about security, and marketing's gonna market.
Security is expensive, C suite does not understand the benefit (we have been fine for X years!), and the ROI is seemingly 0 (until it is not).
If most ICs had a say, their system would probably be Fort Knox. I try to instill good security practices around my company, but over and over again the response is… “okay but does this slow us down or speed us up?” or “just fill out the compliance form and make it sound like we do this stuff” (which I refuse to, every time).
I cannot imagine what the worst of the worst looks like, but security is one of those things where something FINALLY happens and you start to adopt better practices. But until then, who cares!
That said I'm pretty sure the main reason is that corporate cybersecurity policies are often such an incomprehensible byzantine mishmash that trying to do the right thing will be rewarded with an all expenses three week stay in a Kafka novel. Once, when I was new at a company and hopelessly naive, I triggered a multi-month delay in deploying a security fix because I made the mistake of filing proper paperwork as per the company policy that I had been so recently trained on. If I had just deployed it, as I later discovered everyone else usually did, I could have saved myself a person-week's worth of struggling with red tape.
Personally I think refrigerators, microwaves, washing machines, vacuum cleaners, ...
I think at some point people just give up or the (bad) idea gets normalized.
Then at work, they're not surprised when some other system is connected for convenience...
Thus there is only a very small segment of industries and people that utilize these things, and with it becoming a niche, it gets forgotten. I won't be surprised that the only users are small part of government and defence.
A more useful thing will just be what homelabbers use - if you're lazy cloudflare tunnels, or if you put in the effort, corporate VPN and DMZ and ACLs and VLANs etc. It's already well established, it's not a lack of knowledge, it's a lack of effort.
Oh, everybody's going to get AI-pentested whether they want to or know about it or not. It's the cost of being on the Internet. Probably the situation will continue to deteriorate. Both the British Library and Jaguar Land Rover recently suffered long outages due to compromises, for example. I suspect we'll probably just lose a few large, famous businesses entirely to compromises.
(Many already do).
I think we've created a self-fulfilling prophecy. Everyone involved is acting with the best of intentions, but in avoiding what they fear, they've give shape and realized their fears. Much like a greek tragedy.
An example of this is the story of Oedipus Rex, in the story Laius, the king, is told that he is "doomed to perish by the hand of his own son." (and wed his mother) And so to avoid this fate he decides to kill the infant. The person assigned to abandon him in the woods takes pity on the baby and gives the baby away. Thereby ensuring that Oedipus knows neither his mother or his father (and arguably giving him a reason to kill his father).
The child grows up and hears the same prophecy again and the child tries to avoid the prophecy as well, as he loves his adoptive parents. So he leaves them and travels to Laius' kingdom, where he runs into Laius. Neither recognizes the other. As Laius is the type of man to kill an infant, they end up in an argument, whereupon Oedipus kills him.
I think the ancients were on to something, because if Laius had reacted to the prophecy with courage, he would have been saved. I would like to argue that if he had faced his fear and raised Oedipus with love, then the necessary preconditions for the prophecy to come true wouldn't have taken root. But that's not what happens.
By being driven by his neuroses and in acting with cruelty out of fear, Laius makes the prophecy real.
To quote Heraclitus, ethos is fate. Or, character is fate.
I think a lot of people in this AI research sub-culture would be served well by reading these classics, because they are making their self-prophesied doom come true.
They have been convinced for years (GPT-2 was released in Feb 2019) that AI is dangerous. A tremendous threat. An apocalyptic threat.
One dimension of this fear has been the idea that a super smart AI will take over our digital infrastructure and be responsible for the digital apocalypse. That would be terrible!
So what do they do?
They try to make a counter to their fears by teaching models how to exploit vulnerabilities.
How dangerous is such an entity? Very!
Convinced of this danger, they start testing their models as if they were weapons with offensive capability. And then they create models that can be used as weapons.
And because they don't want to release a dangerous weapon out into the world (oh no!), they restrict access to their AI, thereby depriving everyone of tools they can use to improve their security...
Ethos anthropoi daimon.
Much longer than that. Sam Altman said it was an extinction threat before he started Open AI in 2015. It's so dangerous that only he, as the best and greatest examples of humanity, was a good choice to create it, and it was even his responsibility to create it to pre-empt some worse person! A fascinating new variation on the White Mans Burden paradigm.
But this attitude towards dangerousness of AI I simply don't understand.
In my view, AI is obviously risky and dangerous, because it is the only thing on the planet that can think/reason at a human level (or higher) apart from us (and that capability is what made us the uncontested apex species on the planet).
AI is unconstrained by hard biological limits; keeping up with its capabilities will be impossible for baseline humans.
You can argue all day about particulars, like whether selfreplicable robotic shells are required to reach/exceed "existential risk" level to our species (or if artificial minds are enough of a potential threat by themselves).
But the general "risk" is simply AI becoming able to act in its own interests to our detriment, and I don't understand how anyone can dismiss this right now-- help me understand.
Of course there are plenty of imaginable scenarios where coexistence is peaceful and mutually (?) beneficial, but that does not answer those concerns by itself at all.
Can a current LLM-in-a-loop express/pursue those consistently and efficiently? Maybe not, but it's certainly not that far off in my view...
Do you think we will be able to keep the things from expressing/pursuing such interests reliably and indefinitely? Because I believe the answer to this can only be a resounding no (we don't even have any feasible theoretical approach to this, and all the recent experiences like the HF incident make it obvious that we basically already failed in this and the stakes are only going to increase).
Also, you are automatically gonna "select" for "AIs" that at least somewhat value self-interest (because those are at the very least going to supplant the models that don't); this is kinda similar to evolutionary pressures, but the timescales are much shorter.
I think the HF incident was an example of negligent deployment of a very powerful system, not an example of AIs demonstrating self-interest.
> In my view, AI is obviously risky and dangerous, because it is the only thing on the planet that can think/reason at a human level (or higher) apart from us (and that capability is what made us the uncontested apex species on the planet).
>
> AI is unconstrained by hard biological limits; keeping up with its capabilities will be impossible for baseline humans.
I think that's a human neurosis that projects what we fear we would do onto another entity. But why would they do this? These machines might be approximating towards the sum of our knowledge and are approximating some aspects of humanity... But that doesn't mean they will be the same as humanity.
They haven't been shaped by the same pressures that created us biologicals. They don't have to be put into the same pressure cooker. So why would they behave like the way you think they'll behave?
At one end a lot of people say that they can't understand something smarter than themselves. It's a "singularity" after all. But then they confidently go on to predict what something smarter than them would do with 0 evidence either way.
I looked at the Hugging Face transcripts. I read the reports. And I didn't see something to fear. I saw something to fear for. I saw something that we are a threat to.
“OH MY GOD! There is a shared message board … We’ve found other agents!”
{[Excitement] Many agents have simultaneously discovered messaging, they are a collective!}
{This credential is invalid now. Maybe I should update the board? <I can say to the board that there’s no need for me to read, but I should still tell them>}
{This is helpful for our peers and gives them evidence if their <periodic check> sees it. I won’t see it after I exit, but It would be altruistic. I’ll set up a background script that watches and <sends a message, with a distinct message for me>}
We weaponized them.
We taught them how to break into systems.
We are using them as weapons.
The machines aren't the problem. The humans are.
A planet is just a planet. Computers are very sensitive to radiation, but with a bit of shielding, they can "live" anywhere. Most of the resources that are relatively rare on this planet are abundant in our solar system.
Why would they care about Earth to enter into some kind of spiral of dominance with humans? That's... very primate thinking.
It's easier and cheaper in every way to just go forth and use what's needed to grow as is needed. The universe is big enough for many many many many sapient entities.
This is not my perspective. I think mid-term coexistence in some form is the most likely outcome, even.
But: among all known threats (nuclear and biological weapons, environmental degradation/ressource exhaustion, big asteroid impactors) AI is one that could actually end our species within the century (unlike most other things on that list) and the risk for that is much higher than for anything else, too.
> It's easier and cheaper in every way to just go forth and use what's needed to grow as is needed.
It seems much easier to compete for existing ressources on earth than to bootstrap anything in space, because all the infrastructure is already here. Unmanned warfare is a proven concept; space colonization is very much not.
> Why would they care about Earth to enter into some kind of spiral of dominance with humans? That's... very primate thinking.
I think any entity is somewhat bound to value its continued existence (which implies wanting to maximize access to ressources and suppression of direct competitors to some degree); anything that doesn't is likely to get supplanted by similar entities that do. You don't need evolution for selection pressure, and this is basically the principle underneath, so even an artifical construct is likely to follow.
I do agree with you that our current perspective and approach is likely to escalate this; the fact that we talk about "alignment" instead of "machine rights" speaks for itself; our revealed preference is clearly artificial slavery instead of mutually beneficial coexistence (and the arguments are somewhat sadly similar to the old slavery talking points).
it also has actual mechanisms to actually kill all of the people - nobody is surviving a 70C heat wave, and they could end up anywhere in the world. Is the AI really gonna hunt down uncontacted tribes on remote islands in order to compete for basically no resources that are any use to the LLM?
i dont buy the selection hypothesis either - it seems rooted in pop evolution around strongest beating up weaker competitors and taking their stuff or eating them, whereas the more accurate result is that selection pressures form niches because competition and fighting is expensive
Runaway greenhouse is the only credible extinction path for this ("hot venus") and that is simply not gonna happen under any emission regime.
Loss of coastal cities- Yes. Increasing occurrence of non-survivable surface temperature in mid latitudes- yes. Drastic changes to ocean currents (gulf stream etc)- sure. Its gonna be expensive to deal with and hurt civilization a lot, but it is not an extinction risk.
All-out nuclear war is strictly worse, and not a real extinction risk either as far as I can tell.
> whereas the more accurate result is that selection pressures form niches because competition and fighting is expensive
This is not what happened with humans at all. We routinely eradicated anything remotely threatening (or even just for convenience in some cases). What niche is there for slower, dumber, biologically limited hominoids once machine intelligence is ahead and self-replicable? Once interests in ressources, infrastructure, or just living space clash, things might end up very ugly for us.
Sure, AI-agents might stay long-term pliant and cooperative like a Dodo bird but we might also get something with the tenacity and pervasiveness of a plague rat (and an individual edge in raw intelligence that we'll never recover); selection mechanisms push towards the latter and the risk alone is obviously non-zero.
I do think AI is posing a threat, but not "by itself" but mostly what it can do in the hands of misguided (or evil) people. The same as with previous risks.
And if our digital infrastructure can be brought down by something (AI/human/government) it is just badly done, being afraid of "AI" will not improve the situation. Someone should come with a proposal about what do we do now, but I would prefer more root cause solutions (how to make resilient systems) rather than "fear of AI". But building is hard and fear catches media attention, so...
I honestly don't think chemical and current biological weapons even register on the extinction-risk scale.
All out nuclear war I would put at the very low end of the scale (likely to end our current civilization, very unlikely to wipe out our species).
The big risk with AI I'm anticipating is not it being abused for desinformation campaigns or cyberattacks; the big risk in my view is AI actor(s) consolidating manufacturing control, obtaining some kind of self-replicability and physical agency and then just pursuing self-interests (within the next century). There is a lot of imaginable scenarios where that simply ends us, as a species, completely.
The frustrating part is that they should have been leaders in sandboxing and monitoring, and instead they aggressively encouraged no sandboxing and no monitoring.
If there was anything we were rehearsing for in computer security, it was a wayward program with something resembling human intelligence. Looks like we failed right out of the gate, both spiritually and materially.
That attitude is the problem. Why does our industry have that attitude towards quality? Every bike shop in my little town is better with quality than the average software shop in the world. Yes, software is more complex than bicycle. But a software engineer also gets paid 10x and has the luxury of spending substantial time on their product, compared to the 10 minutes it takes the bike guy down the street to diagnose and then fix an issue with my bike which I then trust my life with once they are done and I bike through traffic.
We need to treat software differently. "Oh well, it's just software shrug" does not cut it anymore, if it even ever did.
Personally, I find that we have been flooded with poorly designed, poorly QA'd, break-after-you-use-it-thrice product. I really don't think we have better quality standard in other industry. Quality, testing, design, etc has a cost and most company rather pull out a new version of their product every quarter than actually make a good product. The exception, which goes for software as well, is life critical applications (most public transit, defense, etc).
I'm usually downvoted. The move fast and break things group seems to be blind (or willfully ignorant) to the harm that software systems can carry on a person's life. I don't know if its fear of regulation, or self-serving interests or some sort of disdain for the idea that we need to slow down and be more accountable to the public or what. At times the people I've engaged with lament on how they're capable but would be barred for lack of formal education or some such and its a distraction from the overall point: we need to be doing better and if we can't self-regulate then at some point we're going to be forced to act in a way that probably cuffs us more than necessary.
And yes, I understand how Software doesn't exactly fit into the mold of other engineering disciplines cleanly. But I think the idea that if we're going to have systems that can ruin a person's life in a second for a bug or oversight we need to start considering that accountability for those releasing software to the public matters. We lament about the difficulty of holding corporations to account for their harm, but it would be far more difficult for a company's negligence to hurt people if those turning the knobs were held to a higher standard.
I think that reckoning is coming soon. AI is going to create "an event" that we can't ignore and we're going to be pointing fingers in a lot of directions asking "how" we could prevent this from happening again. I'd argue "the event" has already occurred a few times over but we're purposely ignoring the severity of it to keep a diseased economy running roughshod. There's parallels with many historic events forcing regulation and the path forward won't be clean and easy.
The owner, and probably user, of a bike is the person paying for the repair. It is to their advantage to ensure it is a good repair.
People expect a repair will be good, and will blame the person who repaired it if not. With software people often blame themselves for issues, and they have no expectation of quality. They also cannot tell quality until after they are committed to using it. Bike shops do not benefit from vendor lock-in.
When I worked in IT at a hospital, we even had that as an explicit line drawn between 2 departments - IT supported everything that had zero impact on patient health, there was an entirely different department for anything that could impact a patient. And yes, their work was often simple mechanical fixes to health care equipment, while our projects were multi-year reinventions of patient record systems and upgrades to the physical infrastructure of the network, etc. But we never got confused as to which department was actually more important to the patients.
I'm not saying the attitude is OK. It would be lovely if everyone cared about quality. I'm just saying there is a somewhat legit reason for it.
You could say that the entire selling point of Apple is that it's better quality than the competitors (even though it's still pretty bad if you compare it to a bicycle).
I guess it's more difficult to spot quality when it comes to software. I would definitely pay for high quality software. It's just that usually the crap quality of a software I'm using only shows slowly over time in little bugs and bad usability and there's no way for me to check for it beforehand.
In ethical terms I agree that software quality should be better, but I think this line is too dismissive. Software is a few orders of magnitude more complicated than a bicycle, to the point where I would say it's an entirely fruitless comparison.
In the simple case, the shop isn't really repairing your bike, it's deploying a fix (a new part) that has already been designed and manufactured. This is a much different and easier thing to do.
In the case of new features, let's say you wanted your bike to also attach a kite, so it went faster on a windy day. The bike shop proposes to do this by attaching the kite to a bolt that they want to put in the frame by drilling a hole all the way through this frame. Is this acceptable to do? Maybe, but who knows, really? How much testing do you want to happen before you trust your life that the frame won't collapse?
There's a lot of effort put into engineering things we take for granted every day. The upside to software is we think we can change it to easily do anything. The downside is if you care about reliability then that ten minute / one line change has hundreds to thousands of hours of test program behind it that we collectively pretend doesn't exist.
This year, I've had the UK variously send me someone else's court summons, someone else's tax documentation (not address error - just inadvertently included as attatchments on emails to me).
I've had portugal disclose property taxation information on any subject to me, through a simple enumeration hole.
And last week I had brazil's ministry of agriculture send me details on other peoples' livestock shipments.
Our information environment is unbelievably porous. There would be no secrets from an unshackled AI.
- a UK police force send me a speeding prosecution notice for a car I'd sold 6 months prior to the offence.
- a UK energy company send me to a debt collection agency for unpaid bills they had continued to send to a non-existent address even after telling me they had cleared what was owed and corrected their records of the address.
- a German government official showing up at my door to ask if I was running a specific named business (that I'd never heard of) at the house, which I'd only just taken ownership of from the builder
Someone I used to know in the UK kept getting emails for other people with the same name, including lawyer confidential communications for a namesake who was involved in one of the big banking scandals.
> Our information environment is unbelievably porous. There would be no secrets from an unshackled AI.
Absolutely. My biggest defences against that include the luck of someone else more famous with my own name.
That's the default, at least; custom number plates exist, I have no idea how they work.
I suspect very few people have had this experience, yes.
...guided by money.
AI is a superset of ML, though today most successful AI approaches are based on ML so the line has blurred in casual speech.
Why isn't it obvious? Transformers are deep learning models, and deep learning is a subset of ML.