jacobparker

I'm not sure about the claim about CP systems, but I probably have a different opinion on "common" CP systems (etcd, consul, Spanner-likes, which do offer consistency in the sense of ACID.) But it's definitely worth considering things weaker than that - e.g. Azure Blob Storage and Google Cloud Storage (S3-likes) offer strong consistency but no cross-key transactions. These databases are very applicable but weaker than Spanner-likes like the one from TFA.

Agreed that AP vs. CP isn't a good way to describe databases :)

For more information on this perspective you can read _Spanner, TrueTime and the CAP theorem_ by the fella that coined the term CAP: https://static.googleusercontent.com/media/research.google.c...

Yammer concurred with this perspective, stating:

"At Yammer we have experience with AP systems, and we’ve seen loss of availability for both Cassandra and Riak for various reasons. Our AP systems have not been more reliable than our CP systems, yet they have been more difficult to work with and reason about in the presence of inconsistencies. Other companies have also seen outages with AP systems in production. So in practice, AP systems are just as susceptible as CP systems to outages due to issues such as human error and buggy code, both on the client side and the server side."

https://yokota.blog/2017/02/17/dont-settle-for-eventual-cons...

More info: https://github.com/dotnet/csharplang/issues/36

  $ curl -s https://gist.githubusercontent.com/anonymous/8fdec5275b2c3e81e1e1/raw/blannex | wc -l
  23708

Bazel has plans on their roadmap (https://www.bazel.build/roadmap.html) to open-source rules for common packages ("Repository of BUILD files for third party OSS libraries open to the community") as a P2 for v1.0. This would presumably switch to using those when they're available.

Note that apt is not actually installed inside the container. Bazel just has a rule that knows how to unpack a deb.

If you want to trace uses of it in the code, here is a reasonable place to start, I imagine: https://cs.chromium.org/chromium/src/chrome/browser/engageme...

I think you're taking this too personally. I write _a lot_ of bash scripts and like bash (as I've already indicated.) It's not always the right tool for the right job (no tool is...)

> What do you mean by parsing? Copying, reading? The buffers sit in kernel memory, sure… but that's pretty fast if every line in your program is concurrent.

The buffers don't just sit in the kernel, they are copied between processes (via write/read.) This involves many many context switches. By contrast (but this is just an example), libgit2 will hopefully mmap your packfiles, and once the data is mapped in you don't have to leave your process.

Parsing is sometimes obvious, for example multiple passes over the data with grep and sed to shape it the way you want. It's true that components of the pipeline can run in parallel but the problem is they do a lot of unnecessary work - parallelizing unnecessary work doesn't remove it and once you hit the limit of # of cores (which is very relevant when generating HTML - something often done by webservers with many concurrent requests where you don't have cores to spare on parallelizing inefficient algorithms.)

Parsing is sometime not as obvious, for example steps in the pipeline doing redundant Unicode validation. This is an artifact of squeezing things through read/write with reusable components. It's convenient (which makes it great for prototyping, or for things that "don't matter") but this adds up to poor performance.

> Did you have a bad experience with xargs? > Did you have a bad experience with… running out of… PIDs?

I think you're being very uncivil. xargs is useful but it (with the common options like -n and -P) can spawn many processes which has a tremendous overhead vs an alternative like a function call. It's purely a performance thing.

(I probably should be using parallel but I'm used to xargs :( )

They often make great toys in the sense that toys are fun but it's harder to build robust things in bash scripts.

Robustness may not matter (e.g. I have tonnes of scripts I use for doing work) but if you're generating HTML for a website and are worried about escaping and security then I think there are more suitable tools.

> Big shell pipelines are actually blazingly fast, since each command in the pipeline runs in parallel.

It's also possible to do that in programming languages (maybe not as succiently.)

But where bash scripts fall down is the constant re-parsing of text, data has to be passed through the kernel, and extra processes (when you use xargs, which tends to happen for complicated tasks.) This adds up super quick.

I've replaced reasonable (i.e. weren't badly mishandling the data) shell scripts that procesed and formatted the output of git utilities (on large repos) and in one case shrunk a background job from 5 minutes to a few seconds. This is common.

> Casting bash as the insane bad language

I never said that. I said it was great for prototyping. It is bad for the task FTA, though.

> I would struggle to justify the work I'd need to do to write the equivalent in C

I didn't recommend C (I did link to some Go stuff, though.) Unless your company is all bash scripts (heaven forbid) you're probably already using some language. C# at my work. PHP, Ruby, JavaScript, whatever.

The winning strategy is to use a library/framework/whatever for embedding user-provided content into HTML. Sane HTML template libraries will do this. That library has had more time to get it right. Furthermore a well designed API will clearly indicate what is trusted vs. untrusted data and all untrusted data is properly encoded before being embedded. See the "Security Model" section of golangs HTML templates below.

An alternative to using the git tools which is appropriate for serious work (shell pipelines are great for prototyping) is libgit2. It has bindings for many languages. It's very easy to use (sometimes (not always) easier than the CLI) and often much higher performance vs. big shell pipelines (operating on text gets slow pretty fast, and often you end up using xargs...)

An example set of tools: https://golang.org/pkg/html/template/ + https://github.com/libgit2/git2go .

It's not as succinct as a bash script but it's easier to build something that's correct. Use the shell to prototype, build it right in a saner environment.

  {
    std::unique_ptr<T> x = foo();
    if(x->something()) {
      // use x
    }
  }
  // x out of scope, destructed

  if(auto x = foo(); x->something()) {
    // use x
  }
  // x out of scope, destructed

C# is also getting (has got?) this feature.

The previous version was hosted at http://www.cis.upenn.edu/~bcpierce/sf which now redirects to here.

---

This book is very good for self-study. It teaches you Coq, a formal proof assistant/language. Formally-verified programs can be extracted from these proofs into languages like Haskell and OCaml.

Realistically customers are going to fwd those emails anyway so this approach is more robust.

It seems like most information in these emails would be available to support staff typically though. It sounds like a niche case.

Comparing to http://libtins.github.io/benchmark/ at a glance it looks reasonable - libcrafter is a lot slower than both, and libpcap is often much faster. (scapy (Python) is really, really slow.)

Personally, if I was using libtins and really needed more perf I'd strongly consider just using libpcap directly. But the high-level APIs are really handy.

I don't have anything substantive to say about PcapPlusPlus but I'll check it out :)

(Briefly, note that the original article is critiquing that definition of availability in practice which is legitimate but not relevant to this sub-thread.)

What I'm saying is that EC is most definitely NOT a requirement of physics/the speed of light (what your original post claimed.) The speed of light only sets a (theoretical) limits on how fast you can implement a consistent system.

The original Paxos paper ("The part-time parliament") uses an analogy of a quorum of parliamentarians occasionally getting together in the the same building and agreeing on something. Of course it being the same building is arbitrary and doesn't actually matter, but it's easier to intuit that the speed of light isn't an insurmountable road-block at that scale.

It's important to not overstate the importance of that bound, though.

The Racket one is definitely more towards the "practical" side than the C++ one (C++ is more constrained when it comes to how it can embed DSLs) but for me it falls into a similar bucket with the label "interesting and quirky DSLs that probably aren't practically useful but make me very happy". You may disagree regarding practicality and that's okay by me but please don't take offense. :)

Different but similar joke for C++: http://www.eelis.net/C++/analogliterals.xhtml

---

Vulkan is pretty boring to be honest. I'm very doubtful that you want to bother with it if you're just starting. Using these APIs is like filling out tax forms, but Vulkan is a lot more effort than OpenGL (but, OpenGL is weird...) Learning shaders in OpenGL will be pretty transferable knowledge. Vulkan uses a lower-level assembly-like language, SPIR-V that doesn't pretend to be C (like GLSL) but there are GLSL to SPIR-V compilers. I can't comment on DX (I haven't used it.)

Here's a tutorial on drawing you first triangle: https://software.intel.com/en-us/articles/api-without-secret... . Note that there is a lottttt of code to do this (this is part 3...) I really think you could learn more about modern graphics with glBegin(GL_TRIANGLES) and writing GLSL shaders than boring yourself with Vulkan.

If you do want to play with Vulkan there is this book (piggy-backing off the rep from the old "OpenGL Bible") https://www.amazon.com/Vulkan-Programming-Guide-Official-Lea... . I can't give it an honest review because I got bored; I plan to pick it up again later...

For myself and others, are you referring to the video here: https://pron.github.io/posts/correctness-and-complexity ?

I've been reading this book and it has a good mix of history/context, technique and recipes. It is aimed at folks with a regular oven. Its process for Margherita is multi-step and uses the broiler aggressively; more than I would normally be comfortable with. The output from my regular oven (max 550 C) matches the cover image from the book.

The dough is key, though. I like high salt, low yeast, long fermentation styles. I'm still doing trials with different flour (e.g. caputo 00, blue bag (a standard pizza flour) vs. generic all-purpose flour) to suss out the difference flour makes but the book goes into this in depth, including a basic overview of what the different ingredients are for, chemically, to give you a sense of why some recipes are different (long vs. short fermentation influences amount of yeast/salt.)

----

Shameless plug for an abandoned project of mine: (note: these demos appear to work in the latest Chrome on a wide monitor - YMMV; I didn't know what I was doing.)

Back in 2012 I wanted to learn JS and had some ideas about a UI for teaching C programming with good visualizations. I was interested in building tools that would allow you to visualize in-memory data structures and step through the code (so you could watch a sorting algorithm, or tree manipulation, that kinda stuff.)

Anyway the full thing never panned out but I've got some lower-level demos:

Here's a UI for running MIPS-like assembly: http://csclub.uwaterloo.ca/~j3parker/things/evalc/evalc3/mip...

(Hit "run" and click "faster" a few times) It started off as straight-MIPS but I realized emulating a real machine wasn't necessary for the overall goal (teaching C and the C abstract machine) so I started adding convenient op-codes. All words (I think) of memory had tags to specify if the memory was initialized (maintained by the VM) stack/heap (maintained theoretically by the C runtime) or code/data. The goal was to make a C machine that was maximally friendly for education.

Here's a C REPL that compiles to that assembly and runs it on the VM: http://csclub.uwaterloo.ca/~j3parker/things/evalc/evalc/ it compiles on keyup (I had vague goals of very quick feedback) and there is a line-edit below the source code that you can put C expressions into (e.g. type "fib(12)" without the quotes and hit enter - it should print 233 if you haven't modified the code.)

The parser is mostly-complete C99 (typedefs were an issue - the first parser used a YACC-like generator which is known to have troubles here - the next version used a hand-rolled recursive descent parser. That one didn't get finished but also had way better error messages, as expected.) The compiler (semantic analysis + codegen) didn't support much of the language though, e.g. it doesn't know how to code-gen for > but it knows <. So, the compiler is not at a very usable state.

I also had a V2 of the VM with an insane fixed 8-bit instruction set (with 200+ instructions) for a stack-based (not register) machine with a 32 bit address space. It would generate (with a poorly written bash script) the VM from the LaTeX documentation for the CPU which was weird. Here's the PDF: http://csclub.uwaterloo.ca/~j3parker/things/evalc/evalc2/vm/... the .tex file (in the same directory) has the impl of the instructions (not rendered to the PDF - that was the intent though) and it generates this: http://csclub.uwaterloo.ca/~j3parker/things/evalc/evalc2/vm/... It never did get finished but it had some promise.

---

It was a fun short project that got abandoned. I still think there would be value in an educational implementation of C, something that conforms to the spec but is in no way performant--instead, UB sanitizers everywhere and special hooks inside the runtime to aid in visualization and debugging (e.g. I want to look at memory and see all the allocations with links back to what line of code allocated this word of memory, etc.)

Source code (it's all really bad):

- VM: https://github.com/j3parker/mipsjs/blob/master/mips.js

- Jison grammar/parser: https://github.com/j3parker/jcc/blob/master/c.jison

- Better, but incomplete recursive-descent parser: http://csclub.uwaterloo.ca/~j3parker/things/evalc/evalc2/par...

- Compiler: https://github.com/j3parker/jcc/blob/master/jcc.js