HNHacker News
TopNewBestAskShowJobs

antoncohen

3,401 karma · joined December 8, 2010

Anton Cohen

Linux sysadmin

http://www.antoncohen.com/

submissionscomments
antoncohen··on Pride Versioning 0.3.0
Oh, but what about Ruby's versioning policy, which they call "Semantic Versioning", but the semantics are:

> MINOR: increased every christmas, may be API incompatible

That's right, the semantic meaning behind minor versions is that they are released on Christmas Day. They may or may not be API compatible, who knows.

https://www.ruby-lang.org/en/news/2013/12/21/ruby-version-po...

antoncohen··on The Fed is behind the Capital One/Discover merger
The debit card behavior is probably bank specific. I had fraudulent transactions on my debit card. The bank caught it after a few transactions, alerted me, and shutdown the card when I told them it wasn't me. I didn't get charged for any of the fraudulent transactions. I also had a restaurant charge my debit card for my bill and another customer's bill (honest mistake, not fraud). The restaurant wouldn't refund the transaction, so I disputed it with my bank, who reversed the transaction. The bank was fully set up to dispute debit card transactions from their website.
antoncohen··on My rude-ass car
> can anyone explain why in the US if I click unlock on the remote, it just unlocks the front door? This keeps happening on rental cars. You have to click twice on unlock to get all the doors to open.

This is likely configurable. Instructions should be in the owner's manual. It is configurable for both my and my wife's vehicles, one via the touchscreen and the other by pressing and holding unlock for 5 seconds.

antoncohen··on My rude-ass car
This is vehicle specific. Not all 2023 vehicles behave this way. I have a current generation vehicle (same generation and tech as 2023).

- Doors stay unlocked. Eventually the engine won't start without pressing unlock on the key fob again, but the doors remain physically unlocked forever.

- Trunk is manually operated.

- It doesn't ding when starting the engine if my seatbelt it on. And I have a programmer that lets me disable the dings when my seatbelt it off. There are no dings when turning the engine off.

- Blind spot warning is configurable: Off, lights, lights + chime. The chime warning doesn't seem annoying.

- No lane keeping assistant.

- Tire pressure monitors work well. They are accurate (same pressure as multiple physical gauges I've tried). Tire pressure increases slightly when driving due to heat. They have never triggered a warning.

- I don't recall ever having to accept terms of service. It certainly hasn't happened multiple times.

I have physical knobs for volume, fan speed, and tuner. Physical buttons for everything else. No controls use resistive touch buttons. No controls are via touch screen (touchscreen has information and setting like blind spot, but not actual controls that don't have physical buttons).

I also have a 1990s vehicle, with an aftermarket touchscreen installed to support Android Auto and Apple CarPlay. The current generation vehicle is no more annoying than the 1990s one.

My wife has a 2023 model year vehicle. Many of the complaints in the post are enabled by default (auto re-lock, blind spot chime that gets confused by multiple lanes). But many of the annoying things are also configurable, including auto re-lock and blind spot.

So it is possible to pick a vehicle that isn't annoying. And I suspect most of the annoying things can be disabled.

antoncohen··on Microsoft Creative Writer (1993)
Microsoft open sourced 3D Movie Maker a few months ago:

https://github.com/microsoft/Microsoft-3D-Movie-Maker

antoncohen··on Google slowing hiring to “technical and critical roles” only
In my experience the recruiters are great about helping you prep. They have a PDF called "Google Interview Prep Guide Site Reliability Engineer" that talks about SRE-SE vs. SRE-SWE vs. SWE, and links to books and topics to study.

Kirk McKusick's "FreeBSD Kernel Internals" course[1] was recommended to me, and it was excellent. But not cheap at $1495.

[1] https://www.mckusick.com/courses/introdescrip.html

antoncohen··on How Google got to rolling Linux releases for Desktops
Container-Optimized OS is based on Chromium OS and is more of a traditional Linux, in the sense that is runs on servers.

https://cloud.google.com/container-optimized-os/docs

antoncohen··on Zero downtime migrations
MySQL has some robust tooling in this space. Some of the tools use triggers to copy to a new table. GitHub's gh-ost[1] is probably the state of the art, and uses the binary log stream to replicate the data.

[1] https://github.com/github/gh-ost

antoncohen··on Google Compute Engine VM takeover via DHCP flood
That is true, I was thinking specifically about the metadata and SSH keys. But DHCP can also set DNS servers, NTP servers, and other things that can either cause disruptions or be used to facilitate a different attack.

There might be a persistence issue, it seems like part of this attack was that the IP was persisted to /etc/hosts even after the real DHCP server took over again. But even just writing to /etc/hosts could open the door redirecting traffic to an attacker controlled server.

antoncohen··on Google Compute Engine VM takeover via DHCP flood
While there are a series of vulnerabilities here, none of them would be exploitable in this way if the metadata server was accessed via an IP instead of the hostname metadata.google.internal.

The metadata server is documented to be at 169.254.169.25, always[1]. But Google software (agents and libraries on VMs) resolves it by looking up metadata.google.internal. If metadata.google.internal isn't in /etc/hosts, as can be the case in containers, this can result in actual DNS lookups over the network to get an address that should be known.

AWS uses the same address for their metadata server, but accesses via the IP address and not some hostname[2].

I've seen Google managed DNS servers (in GKE clusters) fall over under the load of Google libraries querying for the metadata address[3]. I'm guessing Google wants to maintain some flexibility, which is why they are using a hostname, but there are tradeoffs.

[1] https://cloud.google.com/compute/docs/internal-dns

[2] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance...

[3] This is easily solvable with Kubernetes HostAliases that write /etc/hosts in the containers.

antoncohen··on A popular household fern may be the first known eusocial plant
> Eusociality typically requires two other conditions. ... And since the ferns spread asexually on shared roots, they don’t actually exhibit an active system of resource acquisition typical of brood care.

> One key question is what defines an individual fern. If a colony can begin with a small plume of strap fronds sticking up from a few nest fronds and then spread asexually on the same roots, perhaps it is a single plant

There is a lot of talk about the amazing cooperation of these plants. And then they say it is actually one plant. Is it interesting if one plant grows different leaves at different heights?

antoncohen··on PolarDB, yet another open source database system based on PostgreSQL
This is effectively required when open sourcing something that was previously internal. I've done it at a company. There could be company internal things that leak in old revisions of code and commit messages. Even if the latest commit on master is clean, it is really hard to know that every revisions, throughout the whole history is clean.
antoncohen··on Pulumi 3.0
The Terraform provider for Google Cloud uses partial autogeneration, here is the repo that does the autogeneration for multiple automation tools:

https://github.com/GoogleCloudPlatform/magic-modules

antoncohen··on Pulumi 3.0
> Open Source SDK

> The SDK is a CLI and collection of libraries for defining and deploying cloud apps and infrastructure in code.

https://www.pulumi.com/pricing/

With Terraform the open source CLI and libraries for defining and deploying infrastructure are fully usable, even for large systems.

What I would like to know is, how usable Pulumi is without a paid subscription?

I thought when I first looked at Pulumi, the only non-local state backend was the paid Pulumi Service. But looking at it now, they seem to support the normal object store backends that Terraform does (https://www.pulumi.com/docs/intro/concepts/state/). Though the talk at lack of concurrency control seems to imply they don't support locking like Terraform does.

antoncohen··on Google Cloud vs. AWS Onboarding Comparison
Everyone's needs are different. To me GCP is easier to get started with.

I don't want white glove service where they try to architect my architecture for me. Pushing me into cloud specific features (like everything in Lambdas stitched together with API Gateways). I don't need that level of help.

Also, of the three startups where I have heavily used AWS (personally "owned" the AWS infra), I never had that level of service offered. The only time I've had that "we'll help you architect it" service from AWS was at a startup that used GCP, and there were high level (CEO and CTO) discussions with AWS about partnerships. Then AWS wanted to help us architect (with Lambdas and API Gateways).

For me, GCP is way easier. Their products are more complete products. They work together out of the box. They are optimized for the 90% use case to just work. It is far easier to get a really good complete architecture working with GCP. Using their products push you to industry standard ways of writing and packaging applications (12-factor, Docker containers, etc.).

The startups I've seen build on AWS often have very bad setups. Hand rolled and poorly managed EC2 messes. Bad logging and monitoring because AWS doesn't have good logging and monitoring. Having to manage everything themselves because AWS doesn't/didn't have good managed platforms (Elastic Beanstalk has enough problems that people don't use it, and EKS barely counts as a managed service).

I've seen multiple startup with hand configured EC2 instances, running some generic web service. These systems create huge burdens and risks as the company grows. The app then grows and becomes locked into the special snowflake hand configured mess. One company I joined couldn't deploy for 6 months because their last deployment (a manual process) took 7 hours of downtime. When I see those systems I wish they started on Heroku or GCP App Engine (2nd gen). They would be way better off with a fully automated platform and 12-factor apps. Once you grow and hire people with infrastructure expertise you can take that clean 12-factor app and move it somewhere else. Non-experts trying to use AWS results in scary things.

GCP gives you that easy on-ramp. You can start with App Engine. If you need something a bit more custom, you can use Cloud Run (same basic tech as App Engine v2 but with Docker containers). If you need to move to something even more customizable for growing special needs, you can use GKE for a fully managed Kubernetes. With all those services you can write normal apps that use environment variables and log to stdout/stderr. All three give you logs in an easy to use logging platform. All three give you metrics in an easy to use monitoring/metrics/dashboard platform.

At every level GCP is easier. Cloud Pub/Sub is easier than SNS+SQS, Amazon Kinesis, or Amazon MSK (Kafka). It is multi-region and just works and scales. GCP VPCs and networking are easier because GCP VPCs are global (not regional), and subnets are regional (not zonal). No need to stitch VPCs together. GCP even has Shared VPCs so you can have a single address space shared across multiple projects (GCP projects == AW accounts). GCP authentication is simple and sane, even across multiple projects, no federation and role assumption needed. GCP IAM is far easier than AWS IAM. Cloud Datastore provides a really simple yet powerful NoSQL store, totally simple, no tunables, just works and scales, and even has some multi-region options available. Want a multi-region consistent database that spans large geographic regions? Amazon doesn't have an option, GCP has Spanner. Want a multi-region consistent blob/file store? Amazon doesn't have an option (S3 is single region), GCP offers GCS in multi-region, dual-region options that provide consistent multi-region file storage. Do you have some weird old app that writes to disk, and you would like it to be able to survive a zone failure? AWS EBS is single zone, GCP persistent disks can be regional, and because subnets are regional that means an instance in another zone can take over the disk and the IP.

For me, GCP is way easier to get started with. And will result in a better system in the medium term. In the long term, when you have thousands of engineers and experts in infrastructure engineering, it probably doesn't matter.

antoncohen··on Google Cloud vs. AWS Onboarding Comparison
You can get GCP support for as little as $100/month. AWS charges for support too.

https://cloud.google.com/support

antoncohen··on Rocky Linux: A CentOS replacement by the CentOS founder
Do you trust Debian LTS? As much as RHEL? The documentation about Debian LTS always made me think it is not a fully fledged thing. I've always felt like Debian releases reached EOL on their EOL date, not their LTS EOL date.

> Debian LTS is not handled by the Debian security team, but by a separate group of volunteers and companies interested in making it a success.

https://wiki.debian.org/LTS

antoncohen··on Waypoint: Build, deploy, and release applications across any platform
Possibly, thanks, I'll look into it. One key is that I'm not looking for an abstraction, I'm looking to provide an abstraction. I run infrastructure teams that provide the infrastructure as a product to other engineering teams, including build and deployment systems. So I need a system that is flexible enough that I can create the abstraction I want to provide. Most of the tools in this space either abstract away everything, in an opinionated way that I don't agree with. Or they abstract away too little, so I would have to write a heavy abstraction over the abstraction. Waypoint, with its plugin system and if the HCL syntax is expanded to support reusable templates/modules, may provide a good building block for me.
antoncohen··on Waypoint: Build, deploy, and release applications across any platform
This is great. I'm glad a good company is backing an open source project like this. I've built abstractions like this at two companies, and I hope one day I can stop writing my own. Some comments:

I see "promotion" on the roadmap, so this is probably in the works. But the current "Workspaces" each have their own build. I really want to be able to promote the same artifact from one environment (staging) to another (production). I'd also like to be able to choose the artifact build/version to deploy, not just what happens to be in my local repo.

The concept of multiple environments also brings up the need to vary things by environment. App config (env vars) are obvious. But also settings like number of replicas or auto scaling min/max (auto scaling is also required).

The biggest thing most of the tools in the space lack, like all the ones that try to copy the docker-compose.yml syntax, is standardized reusable app settings. Imagine I have two common types of app, "API services" and "background workers". They have common settings, like maybe they all default to using /healthz for health checks and auto-scaling at 70% CPU. Then within each group they vary, "API services" use internal load balancers and "background workers" don't.

I don't want every individual "API service" app to have to say "my health check endpoint is /healthz" and "I run on port 8080 and need a load balancer". Those are the defaults, and if the app uses the defaults it shouldn't need to be configured. But at the same time I want the app to be able to override the defaults. Within a well standardized environment 90% of app infra settings (not env vars) are the same, or can use the same template (like the image is docker.example.com/svc/{app_name}:{build_id}). I want to be able to reuse the settings.

antoncohen··on Launch HN: Doppler (YC W19) – Easily manage your env vars and secrets
This is awesome! The interface looks great, it is the UX I want. It boggles my mind why the major cloud providers who have parameter/secret management don't optimize their UX for the 90% use case of "I have an app, it runs in multiple environments, I want to vary the config by environment, and expose the config as environment variables, all with a simple and easy to audit interface".

On the feature request front, I'd like to be able to vary the config by location (e.g., region, but could be zone, rack, etc.). It is common to have a production app deployed to multiple regions (as Doppler itself does), and it is likely that 80% of the config will be the same between regions, but there may be region specific settings.

Which leads to the next thing I want, a hierarchy of config precedence: app default -> app+env -> app+env+location. So that the common settings don't need to be duplicated. Right now my guess is that to use Doppler with multiple regions I'd create environments like "prod-us-central1" and "prod-us-east1", but then 80% of the config will be the same between them.

Another thing that can be nice is to have a canonical value, and have multiple apps point to that value instead of having their own copy of the value. For example if you have a "production DB host" you can set that once, and multiple apps can point their DB_HOST or DATABASE_HOST at the "production DB host" canonical value. That way when the "production DB host" changes, it only needs to be changed in one place.

antoncohen··on DigitalOcean App Platform
> The App Platform is one of the few PaaS products built on a shared Kubernetes platform.

I wonder how they are doing this from a security standpoint. For customer workload isolation is every container actually a VM? The pricing/sizing kind of makes it look like that.

> App Platform provides predictable, easy-to-understand pricing

How? Outbound network is charged per-GB. I know autoscaling isn't supported yet, but hopefully they support setting a max number of instances. One of the things people like about DigitalOcean over other cloud services is that you can sign up to pay $X per month and know that is what exactly what you will pay, no surprises.

> Upcoming features

The list of things not supported yet includes auto-scaling and VPC. It is hard to imagine using a PaaS without autoscaling. And I wouldn't want to build out a microservices-like architecture without VPC.

Overall I really like the concept of the offering. Simple PaaS, with custom container support, Kubernetes (for what it is worth in a PaaS, I don't know), and predictable pricing. That all sounds really good.

antoncohen··on My friend starts her job today, after learning to program in prison
Have you ever filled out an actual job application in the US? I'm pretty sure every job application I've filled out until San Francisco passed the Fair Chance Ordinance has asked if I have any convictions, and some have asked about arrests (which is crazy). And every job I can remember has done a background check.
antoncohen··on My friend starts her job today, after learning to program in prison
Move to San Francisco. Employers aren't allowed ask about arrests or conviction on a job application. They aren't allowed to do background checks until they have offered you the job. They can only consider convictions directly related to the job. And if they reject you because of a background check, they must notify you, allow you to respond, and reconsider based on the response.

https://sfgov.org/olse/fair-chance-ordinance-fco

https://sfgov.org/olse/sites/default/files/FCO%20poster2020....

antoncohen··on Cloudflare Radar
Cloudflare says they exclude "content servers"[1], I assume that means CDNs. https://www.verizonmedia.com/ says it is an ad network, including video ads, with brands like Yahoo, HuffPost and TechCrunch.

[1] https://radar.cloudflare.com/glossary#trending-domains

antoncohen··on ACLU sues Palo Alto over ‘unconstitutional’ restrictions at residents-only park
For context on why this park restriction comes across as racist, East Palo Alto is a different city from Palo Alto, and these are the demographics:

Palo Alto[1]: 60.6% White, 27.0% Asian, 6.2% Latino, 1.8% Black, 0.2% Pacific Islander

East Palo Alto[2]: 64.5% Latino, 15.8% Black, 7.4% Pacific Islander, 6.2% White, 3.6% Asian

It comes across as a law that prevents Latino, Black, Pacific Islander people living in the area from visiting their local park.

[1] https://en.wikipedia.org/wiki/Palo_Alto,_California#Demograp...

[2] https://en.wikipedia.org/wiki/East_Palo_Alto,_California#Dem...

antoncohen··on How to contact Google SRE by dropping a shell in Cloud SQL
That is interesting. There is some magic networking going on if Google allows every customer to allocate an IP range of their choice, and the customer can use all the IPs in that range, and Google runs multiple customers on the same network (same VPC and subnet).

A project can contain multiple VPCs. And a VPC can contain multiple subnets, but not with overlapping ranges.

https://cloud.google.com/sql/docs/mysql/configure-private-se...

antoncohen··on How to contact Google SRE by dropping a shell in Cloud SQL
> One of our interesting findings was the iptables rules, since when you enable Private IP access (Which cannot be disabled afterwards), access to the MySQL port is not only added for the IP addresses of the specified VPC network, but instead added for the full 10.0.0.0/8 IP range, which includes other Cloud SQL instances.

> Therefore, if a customer ever enabled Private IP access to their instance, they could be targeted by an attacker-controlled Cloud SQL instance. This could go wrong very quickly if the customer solely relied on the instance being isolated from the external world, and didn’t protect it with a proper password.

I'm not convinced by this, I'm not sure it is vulnerable in the way the author is suggesting "they could be targeted by an attacker-controlled Cloud SQL instance".

First of all, GCE has firewall rules outside of iptables. But the main thing is that the way Cloud SQL does Private IP is via VPC peering. Google creates a VPC on their side, runs MySQL in it, and peers that VPC with your VPC. You actually tell Google what CIDR range to use in the their VPC (the Cloud SQL VPC).

I don't think is it fair to assume that all customers are in the same VPC, and same subnets, with routes between them, and no GCE firewall rules blocking them.

antoncohen··on UI for AWS development VM resources
Asgard from Netflix[1] basically did what you are asking for. But it has been deprecated[2] and replaced with Spinnaker[3], which has a bit of a different focus.

But really, with couple dozen VMs, you don't have a cloud management problem. I would take a step back and figure out what the problem is, and where you want to be. Why are people creating VMs? What are the problems with them creating VMs? How do you mitigate those problems? If it is just tracking, then tags + education can help solve the issue. If it is security and standardization, an opinionated deployment system could help.

With a couple dozen VMs, I'd say you probably shouldn't be running any VMs you manage yourself, and you should be using fully manage solutions. Then focus on the tooling that makes those solutions efficient to use for your users, while meeting your security needs.

[1] https://netflixtechblog.com/asgard-web-based-cloud-managemen...

[2] https://github.com/Netflix/asgard

[3] https://spinnaker.io/

antoncohen··on Osquery: Easily ask questions about your Linux, Windows and macOS infrastructure
That is a result of how Homebrew built and packaged it. There are downloads at https://osquery.io/downloads, and for a Homebrew installation that page recommends installing the Cask.

    $ brew cask info osquery
    osquery: 4.4.0
    https://osquery.io/
    Not installed
    From: https://github.com/Homebrew/homebrew-cask/blob/HEAD/Casks/osquery.rb
    ==> Name
    osquery
    ==> Artifacts
    osquery-4.4.0.pkg (Pkg)
    ==> Analytics
    install: 120 (30 days), 198 (90 days), 206 (365 days)
antoncohen··on Dear Google Cloud: Your Deprecation Policy Is Killing You
People are confusing new versions of things, or new APIs, or new products, or not recommenced anymore, with actually removing products. From this thread people are complaining about:

- Datastore -> Firestore in Datastore Mode: This was a seamless transition with no API change, Google changed the underlying tech.

- Firebase -> Firestore: Never happened. They are different products. The Firebase Realtime Database still exists.

- gcloud CLI updates: It is a CLI tool, it gets new versions.

- App Engine with Python 2.7: It still exists, you can still use it. There is a new App Engine v2 that supports Python 3.

- Cloud SQL v1 to v2: They had a tool to automatically upgrade, it set up replication and switched over.

AWS has new versions too. RDS Aurora v1 (MySQL 5.6) can't be upgraded in-place to RDS Aurora v2 (MySQL 5.7)[1]. AWS Lambda runtimes get deprecated[2].

[1] https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...

[2] https://docs.aws.amazon.com/lambda/latest/dg/runtime-support...

Page 1 of 14Next →