1,425 karma · joined November 21, 2012
https://github.com/j3parker
https://www.solidangle.ca
The URL you curl could be a GitHub blob URL (theoretically immutable) but if you check the hash you’re not trusting GH for anything other than availability.
You also would need to do that as an atomic operation (in the mono-repo + especially with a commit queue you're building on the atomicity of git).
Having to unwinding that transaction if you aren't atomic can get you into a big mess at larger scales.
Here's a good related talk: C++ as a "Live at Head" language: https://www.youtube.com/watch?v=tISy7EJQPzI by Titus Winters (from Google).
Was this source previously in the Windows repository? Has all development moved to GitHub? If it was in the Windows repo, are they mirroring it back?
Do they still have the old history? There are multiple ways to deal with this in git (grafts, replacements) I wonder if they're using anything like this to get full history, internally.
Even the "corporate policies" is a security thing, not "making someone's job easier". Presumably Facebook handles their certificates carefully: if any engineer could obtain certificates the odds that they would and that the certificates would be passed around in email attachments is 100%.
I couldn't find your contact information. It might be good to add it to your profile for other people.
A modern way to do this on Linux (specifically) is to use PID-namespaces for isolation.
You can pass CLONE_NEWPID to the flags argument of clone(2) so that the new process is PID 1 in a new PID-namespace. One of the special properties of PID 1 is that when it dies all of other processes in the PID-namespace will receive SIGKILL.
So, create a new "shim" process in a new PID-namespace and have that process fork&exec the rogue program (it's probably better to fork&exec than just exec because of the special properties of PID 1 that the rogue programming you're running may not be built to handle.)
If you want to kill the rogue program, just kill your shim.
You need to be root (have CAP_SYS_ADMIN) to create a new PID-namespace. If you're not (or more likely don't want to be) you can first create a new user-namespace with the CLONE_NEWUSER flag. You'll be root in that user namespace and can create a new PID-namespace. You'll probably also want to launch the rogue program as a non-root user though, so make sure to figure that out.
Docker does this sort of stuff.
man pages:
* http://man7.org/linux/man-pages/man7/pid_namespaces.7.html
* http://man7.org/linux/man-pages/man7/user_namespaces.7.html
Here's a minimal UI for browsing r/politics/rising specifically:
* https://www.f6oclock.com/#ninja if you want something more discrete
It updates live and throttles way back when the page doesn't have focus.) The default links go through https://outline.com which is a reader-mode like service (sadly it requires JS - I've got a prototype of a non-JS/server-side rendered version here: https://www.f6oclock.com/readability-demo/test.out.html )
(viewing the stories from r/politics is a guilty pleasure of mine that I'm a little bit ashamed of.)
---
The reddit feed-reading APIs are very easy to use. Just add .json in front of a URL and make sure to send a User-Agent. You can whip up bespoke UIs to suit your particular needs--e.g. I have some things that I send through IRC.
(It's named f6oclock because there was f5oclock but they made it suck...)
His lecturers were always very well prepared and engaging, the notes he released (iirc there were some supplementary notes) were great and his assignments were interesting. He also had consistently great type setting.
I guess I wanted to say that this book is probably cool.
"The Harmful Consequences of the Robustness Principle" https://tools.ietf.org/html/draft-thomson-postel-was-wrong-0...
https://www.elastic.co/guide/en/elasticsearch/resiliency/cur...
https://aphyr.com/posts/317-call-me-maybe-elasticsearch
https://aphyr.com/posts/323-call-me-maybe-elasticsearch-1-5-...
The statement was blunt but the reputation is not exactly unearned. These problems become worse proportional to scale.
https://github.com/id-Software/Quake-III-Arena/blob/dbe4ddb1...
They implemented adaptive huffman coding but ironically only use it during static initialization, feeding it with static data (see the defn of msg_hData a few lines up.) Note that the only calls to Huff_addRef are in the init, and notice how it gets called - the loop over msg_hData[i]. I found that pretty funny. I'm guessing there was some plans to use it adaptively but it's not easy/obvious how to keep it in sync reliably...
- The client/server network protocol and syncing state
- The VM architecture (e.g. most of the game can be compiled by LCC into a custom byte code and interpreted. This was for safely (although I wouldn't be as sure these days) distributing mods iirc. I think the main game was shipped compiled to x86 though. The interface between the VM and the core stuff is via "syscalls" and its fun to see how that's wired up.)
- The player physics/controls (i.e. walk/jump/collisions/etc.) (it's neat how fake first-person player physics always is. This isn't unique to quake 3 but quake 3 had notably good controls.)
It's a bit fun to trace what happens on 'w' keydown/keyup (when bound to walk forward.) When I first saw how that actually worked I was surprised (hint: it types commands invisibly into the command console ;) lots of string copies to do that.)
Chrome/Google have something like that but more flexible (e.g. you can have nested OWNERS files) called OWNERS https://chromium.googlesource.com/chromium/src/+/master/docs...
There's a more general/kinda crazy system in Gerrit (by Google) that let's you write custom rules in Prolog (which helps for solving for a minimal set of reviewers that could approve a given PR.) I'm not sure if it gets much use but its a neat thought for sure: https://gerrit-review.googlesource.com/Documentation/prolog-... (which links to this email explaining why: https://groups.google.com/d/msg/repo-discuss/wJxTGhlHZMM/Tal... )
1) Anyone can submit a certificate to the CT logs
2) It will become mandatory that certificates are logged for Chrome to trust them in a matter of weeks (other browsers will follow): https://groups.google.com/a/chromium.org/forum/#!msg/ct-poli...
Even still, the difference between this and e2e encryption isn’t something an end user is really equipped to evaluate IMO. The threat model is practically different vs e2e no-encryption.
Cloudflare also supports reencryption over the net which is useful if your hosting provider supports HTTPS but not via a custom domain (e.g. Google clouds GCS (S3))
If your host hasn’t caught up with the times yet it’s very straight-forward to get this for free (along with other benefits) from cloudflare with a few clicks.
This is misinformed FUD.
Cloudflare may have made it more common for the most basic kind of site (with their easy setup and free tier) but at the same time most of those sites probably didn’t use https anyway.
The reasons this has been done are performance (specialized hardware/separation of concerns) load balancers/firewalls needing to decrypt to route/enforce policy (that doesn’t need to imply termination but it often goes hand-in-hand) and protecting keys from your app server (think of it as like an HSM - if your app server gets compromised you probably don’t want the TLS private key to be leaked. Again you could reencrypt with a different key but often this hadn’t been done.)
The threats for last mile network fuckery (e.g. consumer ISP) are quite different then on the backend. Google has to worry about nation states messing with their networks and so they’ve had to reengineer end-to-end encryption within their network. As an end-user you just sort of need to accept that this isn’t within your ability to control or know.
(I'm not sure how much practice there is behind that theory, currently, but that's the plan.)
The fact that S3 offers the form of strong-consistency that it does means that S3 is already suffering most of the "problems" that strong-consistency implies. If S3 suffered a particularly bad network split, it may not be able to serve reads because it can't be sure that a write didn't happen elsewhere. This obviously applies to new keys (due to their documented guarantees.)
To demonstrate why CAP doesn't say enough to be interesting: imagine building an S3 by starting with etcd (which is based on Raft.) etcd is CP but has scalability/availability issues, so it won't scale out well. Now run multiple independent etcd clusters and shard your keyspace onto those clusters. Now you have arbitrary scalability and your availability is better than a single etcd cluster. Now of course if you were building an S3 you wouldn't actually run it like this (the same process could participate in different Paxos/Raft logs at the same time) but that's the high-level idea.
My point: You can get arbitrarily more availability and throughput without sacrificing strong-consistency, except in some really odd scenarios (no machine can talk to another machine, for example, and you must replicate to multiple machines.) CAP is about 100% availability (vs. 99.9999999%) but some scenarios are dumb and 100% isn't usually necessary. The key to this is that keys in S3/GCS are independent; there are no cross-key transactions.
Eric Brewer (coined the term CAP) wrote about this with respect to Spanner here: "Spanner, TrueTime & The CAP Theorem" https://static.googleusercontent.com/media/research.google.c...
The "I want strong consistency but have to use S3 because its the cheapest way to store stuff and its great at serving reads and just otherwise great" option is to maintain an external index that maps between persistent ids (e.g. file paths) and S3 keys. We use this for hosting user content at my company. Our index is in MS SQL of all things ;)
The MVP option is to not to delete stuff. :) This can work well in some scenarios (e.g. you want to keep all versions around anyway) but be careful to not dig a hole that's hard to get out of.
A more esoteric option is lifecycle based cleanup. This can work well for things like a build artifact cache (where an expired key isn't a big deal because you can regenerate it.)
Some of this is discussed in this blog post by Netflix: https://medium.com/netflix-techblog/s3mper-consistency-in-th... (caveat: s3mper is dead(?) and there is an AWS-native solution for HDFS now.)
* S3: https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction...
* GCS: https://cloud.google.com/storage/docs/consistency
* Azure: exercise left to the reader
The best usages of S3 I've seen are designed to only use the one major form of strong-consistency that S3 guarantees: if you write a key that has never been written to or read from, and then do a read, it will always work. Sticking to this guarantee will avoid nasty difficult/impossible-to-diagnose bugs. To take advantage of this you do things like:
A. Use unpredictable keys (e.g. prefix with a GUID) to guarantee that you don't get EC behaviour, communicate those keys to other services/users after the upload has succeeded B. Store immutable/content-addressable data. Use new keys for new versions of the data (don't ever over-write a key.) If necessary, use an external (strongly-consistent) index to map identifiers to S3 keys. C. Avoid "upload if not exists" in favour of just uploading (e.g. in the case of a possible failure just reupload, don't check if the key exists on S3. Uploads are atomic.)
GCS offers a far greater amount of strong-consistency and this allows makes it much more flexible.
The key to scalability/performance/simplicity of services like S3 is lack of transactions between keys, not eventual consistency. It's informative to start from Paxos and figure out how you'd build an S3 and why it might exhibit EC behaviour like S3 (hint: a caching front-end) and question what value that really brings vs. the extreme decrease in usability (consider that a caching front-end doesn't give you scale-out in the way that lack of cross-key transactions does.)