duijf

> It's just that function calls can only appear in very limited places in the program (only inside `steps`), and to define a function, you have to create a Git repository.

FYI there is also `on: workflow_call` which you can use to define reusable jobs. You don't have to create a new repository for these

https://docs.github.com/en/actions/writing-workflows/workflo...

All the Nix commands that take an 'installable' can take GitHub URLs. For instance:

    $ nix run github:NixOS/nixpkgs#hello
    Hello world!

That command will download nixpkgs from GitHub, evaluate the `hello` flake attribute, build it (or download it from a cache), and run it.

> But to find out what the flake exposes at all, reading the flake (or its documentation) is pretty much necessary.

If the flake exposes default packages or apps, then you do not need to provide a flake attribute:

    $ nix run github:NixOS/nixpkgs
    error: flake 'github:NixOS/nixpkgs' does not provide attribute 'apps.aarch64-darwin.default', 'defaultApp.aarch64-darwin', 'packages.aarch64-darwin.default' or 'defaultPackage.aarch64-darwin'

So you can run e.g. Alejandra [1], a Nix formatter, like so:

    $ nix run github:kamadorueda/alejandra

[1]: https://github.com/kamadorueda/alejandra

EDIT: For what it's worth, I think this feature can be useful sometimes, but it does also suffer from the same typosquatting problems as we see in other ecosystems.

Wow it's great seeing this here again.

The first uni assignment I made for CS101 was a Mandelbrot set renderer. I got it to work, but that's all the merit it had. I didn't have a clue about what I was actually doing.

When I read this post a couple of months later it answered questions I didn't even know I had. Ever since, I try to keep digging if I have that feeling of "There must be more to this.."

Line of business, I think

> Ports below 1024 can be opened only by root.

Or processes running with the CAP_NET_BIND_SERVICE capability! [1]

Capabilities are a Linux kernel feature. Granting CAP_NET_BIND_SERVICE to nginx means you do not need to start it with full root privileges. This capability gives it the ability to open ports below 1024

Using systemd, you can use this feature like this:

    [Service]
    ExecStart=/usr/bin/nginx -c /etc/my_nginx.conf
    AmbientCapabilities=CAP_NET_BIND_SERVICE
    CapabilityBoundingSet=CAP_NET_BIND_SERVICE
    User=nginx
    Group=nginx

(You probably also want to enable a ton of other sandboxing options, see `systemd-analyze security` for tips)

[1]: https://man7.org/linux/man-pages/man7/capabilities.7.html

Working on that API gateway with you was a lot of fun :)

To be fair, we did run into some issues with the GHC garbage collector performance initially. That took some time to figure out and wasn't the easiest thing ever. Like all tools, there are rough edges sometimes.

I still maintain that the Haskell we wrote at the time was pretty cheap in terms of operational load / bugs to fix (especially compared to the systems that they replaced). When I was back at the office for a reunion, I heard that things were still pretty nice in this respect, but maybe someone still at Channable can chime in with more recent stories! (Or complain to me about the code I wrote back then)

Channable has a tech blog where you can find more details.

The most popular ones discussing technology choices (with HN discussion):

- Haskell (https://news.ycombinator.com/item?id=13782333)

- Nix (https://news.ycombinator.com/item?id=26748696)

You can find more posts on:

- https://www.channable.com/tech

- And the HN discussion with this Algolia query: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

Hope those are useful to you! :-)

Disclaimer: I led DevOps at Channable for a while, but I no longer work there.

Here's a way you can do this with git. This trick relies on `git merge --allow-unrelated-histories`.

Assuming you have repos `foo` and `bar` and want to move them to the new repo `mono`.

    $ ls
    foo
    bar
    
    # Prepare for import: we want to move all files into a new subdir `foo` so
    # we don't get conflicts later. This uses Zsh's extended globs. See
    # https://stackoverflow.com/questions/670460/move-all-files-except-one for
    # bash syntax.
    $ cd foo
    $ setopt extended_glob
    $ mkdir foo
    $ mv ^foo foo
    $ git add .
    $ git commit -m "Prepare foo for import"
    
    # Follow those "move to subdir" steps for `bar` as well.
    
    # Now make the final monorepo
    $ cd ..
    $ mkdir mono
    $ cd mono
    $ git init
    $ touch README.md
    $ git add README.md
    $ git commit -m "Initial commit in mono"
    
    $ git remote add foo ../foo
    $ git fetch foo
    $ git remote add bar ../bar
    $ git fetch bar
    
    # Substitute `main` for `master` or whatever branch you want to import.
    $ git merge --allow-unrelated-histories foo/main
    $ git merge --allow-unrelated-histories bar/main

    # Inspect the final history:
    $ git log --oneline --graph
    *   8aa67e5 (HEAD -> main) Import bar
    |\
    | * eec0abd (bar/main) Prepare bar for import
    | * 9741d6d More stuff in bar
    | * 634ba3d Initial commit bar
    *   43be6e9 Import foo
    |\
    | * d4805a0 (foo/main) Prepare foo for import
    | * 4d2ca10 More stuff in foo
    | * 72072a1 Initial commit foo
    * bfcb339 Initial commit in mono

Form a corporate perspective: 2FA would still force a unique secret per user. That can be useful when your users tend to reuse passwords for different sites or choose poor ones.

I have seen folks use password managers to store their poor non-autogenerated passwords.

For users that do use the PW manager properly, having the PW manager store the TOTP secrets is indeed "putting all of your eggs in one basket".

Thanks for working on this!

I've been using this on and off over the last two months (tend to forget I have it installed and fall back to old habits), but it's really really cool stuff!

More likely May 2021, the date format at the bottom of the post is probably YYYY-MM-DD as common in Germany where the author seems to be located.

EDIT: If Germany has the same conventions as The Netherlands, DD-MM-YYYY is probably even more common

> What is the story in nix for packaging untagged branches of software

Most Nix packages define their sources using `src`. You can pass in whatever you want. E.g. fetch a tarball from GitHub with `fetchUrl`, fetch from a git repository, or point to a local directory.

> or reasoning about "snapshots" where pools of unreleased repos/packages are able to be treated as a single versionable unit?

This is basically what the Nixpkgs collection is minus the "unreleased" part. You can extend Nixpkgs with the packages that you care about using overlays. You have control over the stuff that you put in the overlay, so you can put in unreleased software as well like I described above.

> - I have an existing system for managing packaging metadata which I don't want to migrate from. How much trouble will I get into if I want to generate the metadata on the fly each time (as I currently do for my debs)?

This is difficult to answer without knowing more details.

- You can generate the Nix source files based on the metadata like the sibling comment pointed out.

- Or you could read the metadata with `builtins.fromJSON` and generate Nix derivations programmatically from within Nix.

> - What is the apt/nix interop story?

If you don't want to bother packaging certain applications using Nix (or work towards that gradually), you can. Nothing prevents you from referencing things outside of `/nix/store` in a Nix package. You can launch stuff from `/usr/bin` from a Nix binary if you are so inclined. Libraries are going to be tricky though.

As for the other way around, I'm aware of two options:

- At Channable, we would package the Nix closure of a package in a `.deb`. This works, but you run into trouble when multiple debs need the same store path. You can work around this by packaging the Nix closure under e.g. `/var/lib/<your_package>/nix/store` and bind mounting to `/nix/store` before launching.

- Eventually we realized we didn't want the `/nix/store` stuff in our `.deb` packages. IIRC we created a `postinst` script which ran `nix-env` to realize a store path from our cache + create a GC root so it doesn't get garbage collected.

Some of that stuff may have changed since I left.

Feel free to contact me if you'd like to discuss further. Info is in my profile.

> Also, regarding DevOps, the tooling around Nix makes it a little brittle for anything event based--rapidly changing configurations on the fly due to network conditions (Consul, Ansible, etc). This is where configuration management is heading, and due to the static nature of Nix, delegating dynamic changes is hard/anti pattern.

Channable uses Consul, Vault, etc. for dynamic configuration and it works with Nix just fine.

You don't have to use static configuration files with Nix. Either fetch dynamic stuff using the Consul, Vault, etc. APIs at runtime or use a tool like vaultenv [1] or similar if you don't want this logic in your application code.

Put those tools in your systemd service before launching your app, and you're good to go.

(NB: I was DevOps teamlead at Channable while a part of this work was being done. Sad that I haven't seen the final picture. I imagine it's lovely compared to what we had before)

[1]: https://github.com/channable/vaultenv

There are numerous valid HTTP payloads that are larger than a single TCP/UDP packet.

Ordering is important in these cases.

From https://github.com/ghc-proposals/ghc-proposals/blob/master/p...

.g is a field selector.

f.g is field selection of a record.

f . g is function composition.

f. g is function composition.

My favorite real world example of how you can use these techniques to reduce bugs:

    -- Bad.
    data Session = Session
      { authenticated :: Bool
      , challenge :: Maybe Challenge
      , userId :: Maybe UserId
      }

    -- Good.
    data Session 
      = Unauthenticated
      | Authenticating Challenge
      | Authenticated UserId

(Credits to my friend Arian for the example [1])

The first type permits a value like `Session { authenticated = True, challenge = Challenge "<some challenge>", userId = UserId 1 }`. That's a nonsensical value which doesn't make sense in terms of the business logic.

Generally, when you have types which permit values like this:

- Someone (you or a coworker) will eventually write some code that constructs such nonsensical values.

- This means that you need a lot of tests to ensure that all code using this type works correctly when given such nonsensical values.

By using the second definition of `Session`, you don't have to worry about this at all. Nonsensical values can never exist so you will not accidentally construct them. Therefore you need less tests to ensure your code is correct.

- - -

For some reason, people are really keen to get into code like this when boolean flags are involved. Consider the following code (this time in Python):

    from dataclasses import dataclass

    @dataclass
    class Options:
        connect_tls: bool
        verify_cert: bool
        some_other_setting: int

It does not make sense to have `Options(connect_tls=False, verify_cert=True, ...)`. There is no certificate to validate when you connect without TLS.

(This generally happens when someone is tasked with implementing the `verify_cert` option. They see the existing options type, the existing flag for `connect_tls` and just add a second boolean.)

When I review code like this, I generally advocate for using Enums:

    from dataclasses import dataclass
    from enum import Enum, auto

    class ConnectionOptions(Enum):
        # Could also be: `PLAIN = 'plain'` or `PLAIN = 1`
        PLAIN = auto()
        TLS_UNVERIFIED = auto()
        TLS = auto()

    @dataclass
    class Options:
        connection_options: ConnectionOptions
        some_other_setting: int

It's almost the same safety level as in Haskell (although the Enum has a default serialization, which you need to think about e.g. when you store it in a database).

[1]: https://twitter.com/ProgrammerDude/status/124908893689234637...

Rust looks pretty cool.

Re availability: We had a hard time keeping the system based on Spark available. There were days when the cluster would freak out multiple times in a single day. The 'fix' would be: restart a bunch of spark workers. We spent a lot of time debugging/finding this out (some parts documented in [1]) but couldn't work out what the problem was. (EDIT: Assuming there even was a single problem.)

In this particular case, I'd take the single point of failure over the previous situation.

That being said: we have successfully used PostgreSQL's fail-overs multiple times. In my experience, they work quite alright.

[1]: https://tech.channable.com/posts/2018-04-10-debugging-a-long...

> Reading this article it seems like yet another example of "you don't have big data".

Yes definitely. This thing was a big exercise in "You aren't going to need it".

> so in the end it's mostly...just an app talking to Postgres?

Yes. We solve problems for our customers. It's nice if we can do that without also creating problems for ourselves :)

> seems like they've been jumping on bandwagons before

Just wanted to point out that this system is also written in Haskell. We didn't really switch bandwagons.

If you haven't seen https://haskell-at-work.com/episodes.html yet, I can recommend it

That depends entirely on how/where you implement this.

You can make your application code check the authorization header, get the username from there and do authorization based on that.

There aren't too many standard onrs. Most people seem to either write a custom (embedded) DSL for their tool or go with something more widely known.

If people are curious about examples of DSLs in Haskell projects, they can look at cabal files, persistent's entity syntax, and servant's type level DSL for API definitions. These go on a scale from "fully separate" to "embedded in the language". (Persistent is in between, it uses something called Template Haskell)

The stack Haskell build/project management tool uses YAML files.

- - -

I think Dhall's power is not that it is an alternative syntax to YAML. It's more about the typesystem than anything else. If you're sold on types, then Dhall is definitely worth a look

Dhall heavily borrows both ideas and syntax from the ML family of languages. E.g. Haskell, OCaml, Elm, Purescript

Colons are used for type signatures.

Commas are presumably required because you can have multi line and nested records. (don't quote me on this, not a parser expert)

The comment syntax is from Haskell.

Not saying this syntax is familiar to everyone, but it is familiar to some. The lineage of the syntax might help you understand where the language is coming from

We were able to flip this switch on our services through their API, as we have our Cloudflare config in Terraform.

The API wasn't working perfectly, but with some retries we were able to change the config for our domains.

It's LaTeX, tikz, and pgfplots for the images. Complied with xelatex to pdf and pdf2svg to svg. The rest is custom HTML and CSS. We use the Hakyll static site generator.

> An error doesn't bring down your VM. It gets logged, you can see it, and apply a hot patch and fix it if you need to.

It won't bring down your VM, but it will impact any end-users of your service. Not all errors can be solved by restarting a process or retrying a request (but I recognize the argument that you should aim to catch those types of errors in tests because they are part of the core business logic).

Also just wanted to note that hot-patching has serious costs. Code upgrades/downgrades can be difficult to write and the upgrades themselves also need testing (and this often doesn't get the attention it needs).

There are no silver bullets, basically.

Is anyone else curious why the NYT uses Medium? Their own website is literally about reading stuff

(Sorry if this is off-topic)

Author here, if there is interest, I'd be happy to answer any questions :)

Not exactly at insert mode, but this may be helpful:

    autocmd BufWritePre * :%s/\s\+$//e

Before you save a file, it removes all trailing whitespace (ironically using the same regex as in the article -- hasn't given me trouble yet though)

> All to refute a simple statement that logging might prove useful in such a circumstance.

I don't think that was goal. I think he was questioning whether invading the privacy of a huge amount of individuals is worth it for that cause. How critical is it to have 2 year's worth of car-location data at your fingertips for any investigation? The question is not whether kids ever get kidnapped, but whether such a gigantic dataset aids investigation in any way. A compromise would be to delete the data after a few days, similar to what happens in Minesota, according to the article.

Also: https://en.wikipedia.org/wiki/Children's_interests_(rhetoric...

edit: I mistakenly assumed that hvs and rayiner were the same person. There might be something to be said for your interpretation, but I would assume the best possible one.