antoncohen

We have hit similar issues with GKE. GKE has a soon to be deprecated feature called "metadata concealment"[1], it runs a proxy[2] that intercepts the GCE metadata calls. Some of Google's own libraries made metadata requests at such a high rate that the proxy would lock up and not service any requests. New pods couldn't start on nodes with locked up metadata proxies, because those same libraries that overloaded the proxy would hang if metadata wasn't available.

That was compounded by the metadata requests using DNS and the metadata IP, and until recently Kubernetes didn't have any built-in local DNS cache[3] (GKE still doesn't), which in turn overloaded kube-dns, making other DNS requests fail.

We worked around the issues by disabling metadata concealment, and added metadata to /etc/hosts using pod hostAliases:

    hostAliases:
      - ip: "169.254.169.254"
        hostnames:
          - "metadata.google.internal"
          - "metadata"

[1] https://cloud.google.com/kubernetes-engine/docs/how-to/prote...

[2] https://github.com/GoogleCloudPlatform/k8s-metadata-proxy

[3] https://kubernetes.io/docs/tasks/administer-cluster/nodeloca...

The vendor might not have been compromised.

> we will review our policies for enforcing 2FA on third-party accounts to the extent possible, and continue our transition to single sign-on (SSO) for all of our integrations

I think it is more likely the CircleCI didn't enforce 2FA or SSO[1] with the vendor, and a CircleCI employee reused an already compromised password.

[1] Nearly all SSO identity providers support 2FA, so using SSO is often the easiest way to get 2FA with third-party SaaS providers.

Cacao pulp is delicious. It is interesting that we use the bitter seeds from the fruit, and mostly throw out the tasty pulp. If you are in San Francisco you can get cacao fruit smoothies from Dandelion Chocolate. I hope the pulp starts showing up in more food products instead of going to waste.

Some interesting things for multi-region availability planning. For example Pub/Sub is global, but it seems like if a Publish happens in a region that is having issues, a Subscribe from a region without issues might fail, even if the Publish succeeds.

On the other hand Cloud Storage multi-region buckets seem to really work. The 'us' location experienced 3.5% error rate, higher than the non-affected single regions (~1%), but still much lower than the affected regions (43% - 96%).

It is unclear how Cloud Spanner fared when the instance had only two regions and one of them was affected.

I think more interesting than this RFC is the United States Army Pigeon Service. During WWII the force consisted of 54,000 pigeons. Over 90% of US Army messages sent by pigeons were received.

https://en.wikipedia.org/wiki/United_States_Army_Pigeon_Serv...

Paranoid conspiracy theorists run a paranoid conspiracy website, and are paranoid that a government is conspiring to hack their website.

Also, it is (was) static HTML. It is a bit hard to believe a static HTML site has been hacked, and professional security firm can't prevent it from being hacked.

Figma has a blog post about it, they use Fractional Indexing.

https://www.figma.com/blog/realtime-editing-of-ordered-seque...

Crux Informatics | Infrastructure Engineer | San Francisco (SF) or New York (NYC) | ONSITE | https://jobs.lever.co/cruxinformatics

The Crux Infrastructure team’s mission is to provide infrastructure products that enable Engineering to work efficiently, run reliable services, and remain secure. We build products designed to be used by our fellow engineers. These products can be anything from amazing development environments, to seamless continuous delivery services, to simple out-of-the-box service monitoring, to core compute infrastructure that just works.

You might like this role if you:

- Enjoy writing Python and/or Go (golang), and can write high quality code

- Sweat the details, while focusing on impact

- Aim higher, while keeping it simple

- Hear about a proposed design and immediately start thinking about the end-to-end workflow, edge cases, failure modes, and security risks

- Are excited about building internal products that make your fellow engineers happier and more productive, and our services more reliable and secure

- Think observability is critical to running reliable services

- Love the idea of unleashing a simian army to wreak havoc on systems

- Get satisfaction from helping others

- Can geek-out about Linux to the point that those around you think you are weird

We don’t expect a single candidate to be an expert in all areas, but if some of these ring true to you, you might like this role.

The company:

With Crux, companies can harness data without the high costs. Our data engineers deliver action-ready data to businesses via our cloud platform. By delivering action-ready data through the cloud, we dramatically reduce the need for companies to build infrastructure to manage and process data. We also eliminate the burdensome tasks which eat up 80% of data teams’ time and energy—wrangling data to prepare it for analysis. We help people devote their time and energy at work to finding insights and creating returns—the work that’s truly valuable.

https://www.cruxinformatics.com/

Reading this makes me worry that they will no longer be providing easily installable downloads that are free for commercial use.

I suppose that is a good tradeoff, like Red Hat providing RHEL source but not installation media. But it will make harder to get started with Chef, and I don't know if there is enough community motivation to create a CentOS equivalent for Chef. I hope AWS's Open Distro project takes on more software projects (https://opendistro.github.io/for-elasticsearch/).

The professor that created "Killer T-Cell" was Dr. Elton Stubblefield[1][2]. Both Dr. Elton Stubblefield and Dr. James Allison were at University of Texas MD Anderson Center in Houston.

[1] https://www.upi.com/Archives/1982/10/28/Beating-cancer-the-v...

[2] http://jplaffont.photoshelter.com/image/I0000BBbrdiNVlfI

Log volume can easily exceed reasonable memory sizes. Even a small company can generate TBs of logs each month. Having a single box with TBs of memory wouldn't be desirable.

For logs without full indexing, Loki (https://github.com/grafana/loki) is a recent entry into the space, and it probably a good option to look at. It indexes metadata (labels), so it allows searching by labels but not full text. It is also supposed to be horizontally-scalable, which is probably something you want in a log storage solution.

Be careful. This is licensed under the AGPL, which basically makes it unusable. Or a commercial license can be purchased.

The Vipassana silent courses put on by dhamma.org are entirely free, run by volunteers. The popular locations are fully booked months in advance, but if you have a very flexible schedule you can get on the waitlist and be accepted right before the course, that is how I did my first one.

I've done two 10-day courses from them. After my first one I described it to people as like being in prison, never having gone to prison, but with less freedom than prison. But I still went back for a second time :)

Imagine you have an application like GitHub Enterprise or Phabricator, something that has a web interface and SSH for git access. You might trust SSH, but do you trust the web stack to not have vulnerabilities?

I don't know about you, but I don't trust running third party apps, I won't publicly expose them. Instead I put them behind authenticating proxies, like Cloudflare Access[1], Google Identity-Aware Proxy[2], Bitly oauth2_proxy[3], and Buzzfeed SSO[4]. With an authenticating proxy an attacker can't even hit the web application until authenticating, so they can't exploit a vulnerability in the web application. They will have to pop the auth proxy first, which have less surface area, and then pop the web application.

Applications that combine HTTP and SSH complicate things because the typical HTTP proxy won't proxy your SSH (git) connections, and the web authentication flow doesn't work with SSH.

Cloudflare Access provides the auth proxy, cloudflared does the web authentication flow for a command line app and passes SSH traffic through Cloudflare Access.

[1] https://www.cloudflare.com/products/cloudflare-access/

[2] https://cloud.google.com/iap/

[3] https://github.com/bitly/oauth2_proxy

[4] https://github.com/buzzfeed/sso

10 MB could be an exaggeration on average, but not by a huge amount. Not even trying to find something crazy big, this is the first site I checked because I went to it earlier today and noticed it took a long time to load:

https://help.salesforce.com/articleView?id=000232181&languag...

It is simple help article, but instead of serving the actual help article it spends about 2 seconds serving 4 MB of JS and rendering the article.

After that I typed 'app.' into my URL bar and went to the first site that came up, that was 3.9 MB of JS on the login page.

Numbers are for uncompressed JS, including the larger of the inline scripts.

A single page application (SPA) without server side rendering (SSR) doesn't serve "useful data to the user on the first payload". It serves an empty HTML shell with a script tag (or multiple tags) linking to the Javascript app bundle (or multiple bundles). If the app bundling doesn't break up the app into smaller bundles that are asynchronously loaded, the initial JS payload will probably be 10+ MB. At which point the browser needs to download the 10 MB of JS, execute it, and have the JS fill in the HTML with "useful data".

So no, an SPA without SSR doesn't include useful data in the first payload.

One big difference is that Clustrix (YC'06) was a single datacenter consistent cluster, while Spanner can do multi-region consistency. It will be interesting to see what MariaDB does with Clustrix, now that they have acquired them, fingers crossed that they open source the technology.

Mythbusters tested golf ball like dimples on a car, and apparently is help fuel economy.

https://www.autoblog.com/2009/10/22/mythbusters-golf-ball-li...

As the above poster said, Red Hat open sources everything. They buy proprietary software companies and open source the software. Red Hat officially sponsors the CentOS project.

I'm very familiar[1] with the "obfuscation" you linked to, and I think you are missing context. Red Hat used to ship their Linux kernel source RPMs as vanilla Linux kernels plus a series of patches. They switched to releasing the source RPMs as a fully patched kernel tree. They did this because Oracle started providing commercial support for RHEL (not just Oracle Linux), which is how Red Hat makes money. This had no impact on open source redistributions of RHEL, like CentOS, because they just compile the kernel source without modifying it. AFAIK the kernel is the only package they do that with.

[1] There was a kernel bug between RHEL 6 RC and GA, the RCs had individual patches, the GA kernel source RPM was pre-patched. Lack of patches made it harder to track down, but it was still possible by applying the patches to the RC kernel, then diffing the patched RC source against the GA source.

Facebook runs the largest deployment of Django in the world -- Instagram.

https://instagram-engineering.com/web-service-efficiency-at-...

Man, reading that made me wish Clustrix (YC '06) open sourced their database (https://www.clustrix.com/). They had a MySQL compatible scale our DB nearly 10 years ago, wireline compatible with MySQL without using any MySQL code, could participate in a MySQL replication cluster with normal MySQL servers (made migration easy). It was scale out shared-nothing, writes would scale linearly as you added nodes, unlike POLARDB which is shared-everything with a single master. It used RDMA 10 years ago, and custom PCIe devices because NVMe didn't exist.

But they didn't open source it, so only a small handful of companies get to use it. Sad.

I think it would be pretty cool to use Cloud Functions as a Selenium Grid, sort of like Zalenium (https://github.com/zalando/zalenium) does with Kubernetes. If you could parallelize end-to-end tests enough, you could get massive burstable capacity to run parallel tests.

Why is this Google's fault?

I walk through The East Cut nearly every day, and I admit I was confused by the rebranding. But it wasn't done by Google. There are people cleaning the sidewalks in East Cut shirts, banners that say East Cut, real estate companies that call it East Cut. It would be a little weird if Google ignored the name.

Yes, "The East Cut" is a strange name. But that area as never part of South Beach (which is also a strange name given that there is no beach). It wasn't really Rincon Hill, it is the area below Rincon Hill and is at sea level.

The East Cut is part of the larger South of Market area, and some of it could be considered to be the southern part of the Financial District[1]. But the neighborhood has evolved into its own thing, and deserves its own name.

[1] https://en.wikipedia.org/wiki/List_of_neighborhoods_in_San_F...

It sounds to me like imaging is the solution to attackers modifying systems during shipment.

> Are you someone that needs to worry that a delivered device has been manipulated mid-shipment?

> If you try placing a simple LaunchDaemon on a Secure Boot device’s System volume, it doesn’t stop it from booting in Full Security mode. That’s not one of the things in the signature manifest.

With DEP/MDM the existing OS and files that were there doing shipment would still be there, DEP/MDM would just add additional things or change settings. With imaging you wipe out everything that was there during shipment and start fresh.

Container Builder has been able to trigger based on changes to GitHub repos for a while (tags and merges to branches). What it (and the new branding) can't do, and what the parent comment is referring to, is trigger builds when PRs are opened, and update the PR with the build status.

This Google Cloud Container Builder rebranded, which is a good thing. I've been abusing Container Builder as a general CI system for a while, I hope the rebranding indicates there will be more focus and feature development on general CI functionality.

There's a difference between social classes, which exist in nearly every country, and a caste system. Castes are hereditary, you are born into a caste.

Very few people are born into tech. Anyone working in San Francisco probably realizes that most people in Bay Area tech weren't even born in the Bay Area. Working in Bay Area tech is a choice, one which requires hard work and quite a lot of luck.

People are born with varying degrees of privilege, which makes it easier for some than others to get a tech job, that isn't a caste system.

I've been on a tech career path since I was a teenager, but with one parent who was a teacher and the other who was a construction worker, it wasn't like I was born into it. I've worked with plenty of people who have made the transition to a tech career later in life. Like one person who said he was tired of digging ditches so he went to tech bootcamp.

The classes in SF are extreme. The pay disparity is too large. It is really hard for a lot of people to get by. But it isn't a caste system.

That sounds awesome! I'd really like to see organizational management, i.e., users in orgs. It is something most B2B SaaS apps have to do, and they tend to do it differently and often very poorly. Maybe it is out of the scope of ORY Hive, but I think it would be really useful. I'm thinking it would tie into an RBAC system to give users different permissions within orgs (member, admin, owner, etc.).

There are two main patterns for B2B apps:

- One org per user, like G Suite. A user is a member of a single org, to be a member of another org requires another user account. This creates a challenge when it is a system where someone might be a member of multiple orgs, because it requires logging in as multiple users and having cookies that can handle that.

- One user in many orgs, like GitHub. A users "owns" their user account, and is invited to one or more orgs. This makes working in multiple orgs easier, but can create challenges for companies that want to enforce things within their org, like SSO or 2FA. It also creates issues with routing of notifications (you want work notifications going to work email), and identity of users within a company (an org admin may have trouble identifying users if they aren't tied to company emails).

An ecosystem like ORY obviously can't alleviate all the issues, but maybe it can help with some of them, like org membership and org friendly cookies.

San Francisco went through a similar thing. The best public high schools in SF (namely Lowell) per predominantly Chinese-American. The NAACP sued the school district (NAACP v. San Francisco Unified School District), resulting in race-based admissions in 1983 to reduce the number of Chinese-Americans at the best schools

Then in 1994 Ho v. San Francisco Unified School District happened, and race-based admissions got thrown out.

If you look at the racial demographics of public high schools in San Francisco it is pretty interesting. Of the 10 major schools only two of them have over 10% white students. Those are the only two that only admit students that meet special requirements. One is Lowell (academically the best) with 15.2% white, the other is SOTA (an arts school that requires auditions) with 41.3% white. Lowell is 56.7% Asian, while SOTA is 19.0% Asian.

https://en.wikipedia.org/wiki/Lowell_High_School_(San_Franci...

https://en.wikipedia.org/wiki/Ruth_Asawa_San_Francisco_Schoo...

I once tried out pfSense, about 8 years ago. I was looking for a VPN solution for a small startup. I had used pf on OpenBSD, and really liked it as a firewall, so having "pf" in the name made pfSense sound good to me. My hope was that pfSense would provide an easy out of the box VPN setup, without me having to configure a RADUIS server to talk to LDAP, and have easy user provisioning.

What I found was a mess of PHP and poor quality plugins. The init scripts were written in PHP, and the plugins modified the init scripts. So plugins could, and did, break the init process, preventing the firewall rules from loading.

I don't think people are using pfSense in production environments, I see this class of solution being used by IT at small companies. Think about big corporate networks, where you might have a device from Palo Alto Networks, or a Cisco ASA for a remote access VPN. Then imagine you are at a 20 person company, a decade ago before SaaS really took off, and you think you need a VPN, but can't afford something from a big company, and don't want to glue together a bunch of open source tools. I think that is when people reach for pfSense.

I don't think it makes much sense these days. Small companies can use much dumber (and better) gateway devices. With SaaS and things like Google IAP and Cloudflare Access there is little need for VPNs. As for IPS, I have trouble imagining IPS being needed or even working at a small company. Big companies can have sophisticated detection systems and teams to deal with them. Hopefully big companies were never using pfSense.