kevin_nisbet

This was a fairly interesting read, but some of the claims struck me as fairly circumstantial. I ended up almost writing a novel of a comment, but will try and save everyone from that by summarizing.

I started my career in Wireless Telecom, and some of my work was directly on Diameter Routing and working with roaming partners on technical problems.

The SIM card stuff I have no expertise on, so I won't comment on that.

There are a couple of important things to note which I think the report misses.

The first is what a cellular network does for tracking a user. It's not returning a set of GPS coordinates. A cellular network has a problem to solve, which is on which radios do we transmit that the device should wake up from idle about an incoming call (or SMS or packet). We could not track this at all, and then the entire bandwidth is spent notifying the entire country about every call. On the other side we could track this down to every radio, but then you have the problem of your entire network is just signaling traffic about changes in the best radio to reach a device. A tree swaying in the wind causes constant updates type of idea.

So we break the network up into areas, and if the device moves and see's a beacon from a tower that it's in a new area, it tells the network to update it's location records. There is a slightly more precise record which is the Cell ID, but the device doesn't need to keep it up to date. So in the report when you see the references to the Cell ID/E-UTRAN Global Cell Identity, and LAC/TAI this is the concept those identifier tie to.

There are databases that can map the Cell identities to GPS locations, and you can think of that as the assignment to the tower, although there are some deviations that occur (remote locations for a radio, etc). So most often you're not getting the GPS location of the device, you're getting the tower. This is still a privacy implication, and get's more precise over time as the radio networks get more and more dense to support higher speeds. But still might be dozens of KMs away from the actual device location.

I ended up writing a novel, but need to cut it down. So a bunch of the evidence cited on the Diameter protocol and DNS behaviour on GPRs DNS I don't think is as strong as one might conclude when reading the report.

What particular struck me was the DNS NXDomain as an indicator of trade craft to conceal the source, that they refer to several times. To me this is an expected behaviour on roaming DNS if the source network used to make the query does not have a roaming agreement with the other provider. The DNS specifications by the GSMA are a bit awkward, and I've been bitten by them several times, but they don't carry the same DNS related expectations as you would expect on the internet. On the open internet, you expect to see a delegation from com to ycombinator, and to be able to follow that to ycombinator. This isn't the case on the roaming exchange networks, firewalls and answers are only opened up when there is an agreement. So if this provider is a small fry, there might not be many agreements in place, and it's not weird to get back an NXDomain or timeout. This does depend on whether it was the roots or the provider certs that provided the response that they don't go into detail.

Some of the Diameter related statements also struck me as not having a complete understanding of the technology, and suggesting a direct link to trade craft. Things like the Origin-Host and Route-Record 1 being the same, while perhaps a technical violation of the standard, have no impact, and can just as easily be explained by a network operator not wanting to advertise internal details to the rest of the world. Similar, with the IPX provider not detecting the mis-match between realm and host, I'm not even sure that's expected or how they would do it, although from an analysis perspective it is a clear screw up from the adversary that revealed it. But I don't remember sufficient information getting exchanged between providers that would actually allow an enforcement over those fields, but I could be wrong. Also keep in mind, it wasn't just the roaming exchange that didn't enforce it, all the networks also failed to enforce it. And now the adversary might just see this report and fix their bug, so it's not like that enforcement would've completely changed the situation. But they do have a point that if enforced, it might've been detectable earlier that there was a bad actor present.

There are also alternative and resonable explanations for some of the other claims, like was 019Mobile the one actually relaying the messages, or was the second hop tricked into accepting messages, and the adversary was just impersonating 019Mobile. That shifts things around a bit, they talk at length about 019Mobile being the source of these messages, but while likely, there are other plausible explanations for the origins of those messages.

There are also some technical details they got wrong, but probably not in material ways.

So it's an interesting report, it seems like there is a real operation going on, just that I take issue with much of the evidence cited that I think many readers may draw a strong conclusion from then they should.

On this side of the equation I think you start pulling in customer context and risk analysis on the downside. What is the churn risk for operation at 99% vs 99.9% availability.

If your site is for B2B and impacts customers own operations or revenue, you'll likely be wanting to chase the 99.9%, customers won't tolerate the 1.5 hours per week of downtime and will churn.

However, if the value you're site creates is tolerant to those sorts of disruptions, someone is just inconvenienced and can come back later, a large investment to move from 99% to 99.9% wouldn't be justified. There is literally no impact from the investment. The harder part will be the reality, most investments will be somewhere in the middle with ambiguity on the impact. IIRC, SRE principles do talk about this when setting SLOs in different terms.

I've heard some companies refer to the concept as economical thinking, which is I think a great way to think about it. Doesn't mean you'll always get it right, more so that we embed being conscious about the ROI in our work.

I also believe this is an area that I've observed several engineers really struggle with, especially when moving from big tech to startups, where it's really easy to import culture from another company, and in earlier stages of startup life... if you don't have product-market-fit, it doesn't matter how good you're availability is. Attention is a resource, make sure it's allocated to what creates value for the customer.

The tables in the link do a good job of outlining this, 11.9% while an interesting headline number, 11.9% increase over nothing is still basically nothing.

However, with the tables going back to 2022, it's almost 100% increase of businesses and 50% increase for Non-business filings, which I find more interesting.

Would be great to see the totals as population adjusted against historical context though.

My team has been using https://runs-on.com/ for AWS instance runners, had a few glitches but largely been great for using AWS instances for runners.

I've operated at companies using both models, and have observed similar reactions to suggestions of using the cloud.

To me it's like anything else in engineering, are the costs, risks, and benefits fully understood, and worth the tradeoff in the particular context.

I worked for a startup doing internet of things, the consumer would buy a device and get lifetime service baked in. And that company was a step further, just renting space in a colo was incredibly cost efficient, which supported the sales model and competitive landscape of that product. But it was also very costly to attention, one of the most valuable resources. But it can also get costly in non-intuitive ways, an example that comes to mind is we started to get interviews where a generation of candidates no longer had experience with metal, it was a foreign world to them.

With more experience, I find it's really the costs that get severely underestimated, both for and against the suggestion.

I really wanted to like JJ, it was handy for a few months when I used it. But for me in the end I reverted back to regular git.

What triggered me to go back was I never got a really clean mental model for how to keep ontop of Github PRs, bring in changes from origin/main, and ended up really badly mangling a feature branch that multiple contributors were working on when we did want to pull it in. I'll probably try it again at some point, but working in a team through Github PRs that was my main barrier to entry.

Agreed, and this is where I think some more nuanced and conscious use of tech debt can be used when applicable.

It might be OK to place some bets on an initiative or feature, but if we all understand we're placing a bet, this is an area to load up on debt and really minimize the investment. This also requires an org that is mature about cutting the feature if the bet doesn't materialize, and if the market signal is generated will reinvest in paying down the debt. And also has the mega-danger territory of a weak market signal, where it's not clear if there is market signal or not, so the company doubles down into the weak signal.

Also these bets shouldn't be done in isolation in my view, well executed product and market discovery should also provide lots of relevant context on the ROI.

I'm not commenting too much on the details of the article, but the premise does resonate with me. I would argue all the engineering teams I've been on do not spend enough time thinking about how much a piece of work will cost to execute, and whether it will generate a return.

I suspect this is most apparent on things like meeting culture. Something happens and all of a sudden there is another recurring meeting on the calendar, with 15 attendee's, costing x dollars in wages, that produces no value for the customers because the lesson was already learned.

Or when reacting to an incident of some sort, it's so easy to have a long list of action items that may theoretically improve the situation, but in reality are incredibly expensive for the value they produce (or the risks they reduce). It's too easy to say, we'll totally redesign the system to avoid said problem. And what worries me, is often those very expansive actions, then cause you to overlook realistic but small investments that move the needle more than you would think.

And as a hot topic I also think the costs are an input into taking on tech debt. I know we all hate tech debt with a passion, but honestly, I think of it as a tool that can be wielded responsibly or irresponsibly. But if we don't know what our attention costs, we're going to have difficulty making the responsible choices about when and where to take on this debt. And then if we're not conscious about the debt, when it comes do it stings so much harder to pay down.

Thanks for sharing, this was a great read. And it brought up some old memories, of fighting with telecom vendors about how their calculated their memory usage. At that time you only really got what the vendors gave you.

I think the most fun I had with the page cache was when one of the vendors "fixed" a bug where the memory calculation has previously excluded the page cache. These boxes were all network services, they didn't rely on the disk for anything more than holding the binaries, configuration, and logs. Where the fun comes in is the first time you grep through the logs, which fills the page cache, and sets off the alarms that the cellular network is about to die. ;)

Anyways, important concept to know and understand when it comes to how software performs when interacting with a host.

> I chose managed services specifically to avoid ops emergencies. We're a tiny startup paying the premium so someone else handles this. Instead, I spent late night hours debugging VPC routing issues in a networking layer I don't control.

This happens with managed services and I understand the frustration, but vendors are just as fallible as the rest of us and are going to have wonky behaviour and outages, regardless of the stability they advertise. This is always part of build vs buy, buy doesn't always guarentee a friction free result.

It happens with the big cloud providers as well, I've spent hours with AWS chasing why some VMs are missing routing table entries inside the VPC, or on GCP we had to just ban a class of VMs because the packet processing was so bad we couldn't even get a file copy to complete between VMs.

> I see, this sheds some new light on your initial concerns. I'm aware an attacker can keep pretending to be inside an environment once they've seen it. I wasn't accounting for a scenario where an attacker has a huge database for queries like coords -> list of wifi networks

I think this is the issue, is these datasets are out there and at least big tech companies have them since they're used to assist with GPS. I was about to post the same thing as above but saw vessenes beat me to it.

Without thinking about it too hard, the two directions I see are either making observations of the environment in real-time that is only relevant at that time (IE sniffing actual wireless frames, even if they're encrypted and making observations on them, however, most devices won't let you go into promiscuous mode and do this) or encrypting the messages in flight so only participants can decrypt them (IE a model like the signal protocol with E2E message encryption).

Anyways, this is a cool approach, but that risk occurred to me as well about the ability to just brute force the entire dataset to decode every location.

I'm not as sure and would want to consider the angles a bit more.

I ponder how effective this would be against an adversary sufficiently motivated to look like they're not using a VPN. And then does it result in a false sense of trust, since a user thinks the system more reliably detects a VPN then it does. Or an adversary who has bypassed the system to then point to it to build additional trust.

I'm with you, I think most people might think they don't need this reliability, until they do. I'm sure there is some subset of clusters where the claim is correct.

But from the article, turning off fsync and expecting to only lose a few ms of updates. I've tried to recover etcd on volumes that lied about fsync and experienced a power outage, and I don't think we managed to recover it. There might be more options now to recover and ignore corrupted WAL entries, but at that time it was very difficult and I think we ended up just reinstalling from scratch. For clusters where this doesn't matter or the SLOs for recovery account for this, I'm totally onboard, but only if you know what you're doing.

And similar the point from the article that "full control plane data loss isn’t catastrophic in some environments" is correct, in the sense of what the author means by some environments. Because I don't think it's limited to those that are management by gitops as suggested, but where there is enough resiliency and time to redeploy and do all the cleanup.

Anyways, like much advice on the internet, it's not good or bad, just highly situational, and some of the suggestions should only be applied if the implications are fully understood.

Yup, even for smaller business stuff. For a non-profit I'm on the board of, the staff wanted a more useful printer/copy machine than just a store bought thing, it's a small office, so I said sure find something and let us know.

So I get a contract and am told it's been vetted and I should sign it. What I found was outrageous.

- If we cancelled for any reason, including if they just didn't do any of there terms in the contract, we owed the full price of the remaining contract immediately.

- The way they structured it was also as a rental, so we were paying full price for purchase of the equipment embedded into the term of the contract, but it was the vendors equipment, so if we cancelled we still paid them full price for the equipment, and they got to keep it.

- If there were any legal disputes, no matter which party was at fault, my side would pay for all the lawyers.

I said nope, can't do it. And my staff were pissed at me for like a year because everyone just signs those things.

I don’t know about node but a fun abuse of this is background tasks can still sometimes run on a busy lambda as the same process will unsuspend and resuspend the same process. So you can abuse this sometimes for non essential background tasks and to keep things like caches in process. You just cant rely on this since the runtime instead might just cycle out the suspended lambda.

My ISP does this as well, provides Huawei modems with hardcoded backdoor passwords that can easily be found online. So yup, I've got a dedicated firewall between my networks and the modem. With slow updates and backdoors, I'd include any ISP modem and networks as part of my personal threat model.

I kind of like what the folks in British Columbia are doing for their EV prototypes for Semi Trucks (technically a hybrid, EV with onboard generator): https://www.edisonmotors.ca/topsy

I don't know if the point should be that people won't use a startup bank, just that the assets being directed to the startups/disruptors are not presently threatening to the big banks. I would suspect this is currently the case with WealthSimple here in Canada as well. WealthSimple is at something like $50 billion assets under management [1].

Vanguard asset allocation ETFs are at like $1.3T [2]. 4 Of Canada's Big banks appear to add up to just over 2T Assets under management based on what Google just gave me as summary. So while I think this is a great outcome for a startup (even with Power backing them), to me it seems in a similar space as the above article that we're still talking a relatively small market share, and likely still closer to early adopter status.

[1] - https://en.wikipedia.org/wiki/Wealthsimple#:~:text=As%20of%2... [2] - https://www.vanguard.ca/en/product/investment-capabilities/a...

+1 on recommending the Mom Test, it's one of the most important books I've read.

I'd say in addition to entrepreneurs, it's an important book for product teams / product engineers to understand what the Mom Test teaches, and tune the filter on asking the right questions to get the highest signal, and ensure the solution closely matches the value prop for the customer. Then sales and marketing get a whole lot easier when you've asked the right questions and solved the right problems.

Yea, thinking about it for a minute I would expect limited threat models this tool would help with. I think for broad attacks, this would only be somewhat effective if deployed on tens of millions of hosts so it becomes impractical because the adversary is just finding and interacting with the honeypots.

If you are specifically getting targeted, there might be a slight delay by having the adversary try and exploit the honeypot ports, but if you're running a vulnerable service you still get exploited.

Also if you're a vendor, when prospective customers security teams scan you, you'll have some very annoying security questionnaires to answer.

Probably. I'm not sure what you have in mind, but WireGuard would just be an intermediate network layer, so it would be up to however you're configuring multi-wan routing would still be apply/available.

This is a good explanation, I've had to explain this topic a few times as well, it seems like it's one of those topics that is very missunderstood.

To just expand a bit, I believe some of the confusion around printing of money comes from the way some economics reports are built. As a micro example, Assume a 10% required reserve, If Alice deposits $100 and the bank lends $90 to Bob. Alice ($100 deposits) + Bob ($90 cash) think they have $190 in total.

This is mainly useful for economists to understand, study, and report on. However, when the reports get distributed to the public, it looks like the banks printed their own money, as we now see $190 on the report when there is only $100 of cash in our example system.

Whether the system should work on a fractional reserve is it's own debate, but we need to know what it is to debate the merits and risks of the system.

I suspect there is also another angle, which is are the tests maintainable as well? Like you said, if you're not testing intent, this might be one more thing to maintain.

In another view, this might just be a fancy way of doing snapshot testing, use AI to generate all the inputs to produce a robust snapshot, but realize the output isn't unit tests, it's snapshots that report changes in outputs that devs will just rubber stamp.

This is along the lines of how one of the wireless telecom products I really liked worked.

Each running process had a backup on another blade in the chassis. All internal state was replicated. And the process was written in a crash only fashion, anything unexpected happened and the process would just minicore and exit.

One day I think I noticed that we had over a hundred thousand crashes in the previous 24 hours, but no one complained and we just sent over the minicores to the devs and got them fixed. In theory some users would be impacted that were triggering the crashes, their devices might have a glitch and need to re-associate with the network, but the crashes caused no widespread impacts in that case.

To this day I'm a fan of crash only software as a philosophy, even though I haven't had the opportunity to implement it in the software I work on.

I'm with you, I imagine it's mostly people just drawing parallels, they can figure out how to get a web certificate so think SSH is the same thing.

The second order problem I've found is when you dig in there are plenty of people who ask for certs but when push comes to shove really want functionality where when user access is cancelled all active sessions get torn down immediatly as well.

Agreed. Also if they get the ultra wide streaming of a macbook support shipped might change the equation for me as well. If it works as a productivity tool I can stomach the price.

The funny? thing is, this isn't the only YC company I've seen do this.

Well, my concern started to grow when two things happened: - My sister who is not in a tech field asked me how blockchain worked since she had heard about it - A server at a restaurant was describing that they help people buy their first bitcoin as a side hustle.

So while every example may not permeate out of the echo chambers, there are some that are reasonably widespread.

I agree, this is an incredibly valid approach for some companies and startups. If you benefit by being frugal and are doing something that doesn't need incredible availability, a rack of servers in a colo doesn't cost much and you can take it pretty far without a huge amount of effort.

One technique I've introduced but I don't have a long enough track record to know how effective is using a friction log when onboarding.

The core idea is to ask the person joining to keep a log of anytime something wasn't obvious, and they had to chase down an answer. IE they had to pair with someone to figure out a thing.

Then a month or so after they join, meet with them and go over the log. Doesn't mean the team fixes everything found, but it starts to build the picture of where someone joining has to spend most of their time, and can then have targeted fixes, process updates, etc. As long as the outcome isn't something vague like "improve docs" that no one will understand what the means or how to actually address it.

This also works for product UX discovery, for some types of products have every new joiner onboard as a customer would, and write down a friction log.