ruuda

Chorus One | https://chorus.one/careers | Platforms Engineer | REMOTE (Switzerland ± 6 hours)

Chorus One operates validators on many proof-of-stake blockchains (the ones where security is based on a Byzantine fault-tolerant consensus algorithm rather than wasting energy). We are hiring for several roles, but the one I will highlight is what we call Platforms Engineer. Some companies call this Site Reliability Engineering or Devops.

The main thing we do is take upstream software, build it, run it on our infrastructure, and then monitor it and optimize that setup. Some things that make this interesting are:

    * Building automation that enables us to do this for many networks (70+ currently).
    * Doing this with high uptime, building automation for failover, etc.
    * Working with software that is on the one hand cutting-edge and doing interesting things (consensus algorithms, distributed systems, cryptography), but on the other hand that means it’s immature and often not easy to operate and monitor. Often we have to build custom tools, and dive into the source code of the project. We contribute patches upstream when it makes sense.
    * Some of these projects are exercising the limits of what a machine can do, we have to do some low-level investigation that requires understanding of what the Linux kernel and network hardware are doing to properly identify what’s going on.

We do have a small cloud footprint, but run primarily on bare metal. We are looking for people who can not just configure services offered by the public clouds, but who deeply understand what lies below; people who could build their own cloud. That sounds a bit pretentious and it’s not exactly what we do, but it does involve many of the same aspects.

A very recent example of what I personally find fascinating: last week Ethereum’s Holesky testnet experienced loss of liveness. This is a real-world, globally distributed system that implements Byzantine fault tolerance, with multiple independent implementations of the protocol. Several of these implementations had a bug in an update, which caused a split in the network. The protocol is designed to handle this situation in theory, but in practice it is triggering previously unexplored failure modes in the implementations, that are hard to test for in synthetic small-scale tests. I think there are very few places where you get to be involved in a planet-scale distributed system exhibiting “interesting” behavior, especially one that is not in control of a single entity. Of course, there is also the less fun part that the testnet is now broken, alerts are firing, and it’s hard and chaotic to coordinate a fix when the network is not controlled by a single entity. Fortunately it’s a testnet.

Apply at https://careers.chorus.one/o/platforms-engineer-remote.

Through this article, a few links away, I learnt about tiling logs, explained in https://research.swtch.com/tlog.

I'm working on adding floats to the RCL configuration language (https://rcl-lang.org/) to finally deliver on the json superset promise. Blog post coming soon!

Quod Libet handles album artists properly with the right sorting options? (And also it's one of the few that supports original release date, another seemingly essential feature that few players support.)

In Musium (https://docs.ruuda.nl/musium/) I also handle collaboration albums that have multiple album artists, based on Musicbrainz album artist id.

John Carlos Baez on that article: https://mathstodon.xyz/@johncarlosbaez/113964127171705485

Chorus One | https://chorus.one/careers | Platforms Engineer | REMOTE (Switzerland ± 6 hours)

Chorus One operates validators on many proof-of-stake blockchains (the ones where security is based on a Byzantine fault-tolerant consensus algorithm rather than wasting energy). We are hiring for several roles, but the ones I will highlight is what we call the Platforms Engineer and Infrastructure Software Engineer. Some companies call this Site Reliability Engineering or Devops.

The main thing we do is take upstream software, build it, run it on our infrastructure, and then monitor it and optimize that setup. Some things that make this interesting are:

    * Building automation that enables us to do this for many networks (60+ currently).
    * Doing this with high uptime, building automation for failover, etc.
    * Working with software that is on the one hand cutting-edge and doing interesting things (consensus algorithms, distributed systems, cryptography), but on the other hand that means it’s immature and often not easy to operate and monitor. Often we have to build custom tools, and dive into the source code of the project. We contribute patches upstream when it makes sense.
    * Some of these projects are exercising the limits of what a machine can do, we have to do some low-level investigation that requires understanding of what the Linux kernel and network hardware are doing to properly identify what’s going on.

We do have a small cloud footprint, but run primarily on bare metal. We are looking for people who can not just configure services offered by the public clouds, but who deeply understand what lies below; people who could build their own cloud. (That sounds a bit pretentious and it’s not exactly what we do, but it does involve many of the same aspects.)

If this sounds interesting to you, check out https://careers.chorus.one/o/platforms-engineer-remote and https://careers.chorus.one/o/senior-software-engineer-infras.... We have two job descriptions to appeal to people from different backgrounds; internally we treat it as a spectrum, and the interview process is similar. Whether you’re an infrastructure person who wants to move more into coding, or a software engineer who wants to get into automating operations, feel free to apply to the role that resonates more with you.

We are also hiring for other roles, see https://chorus.one/careers.

Try https://rcl-lang.org/ instead then, it has syntax that you can write without needing ChatGPT.

I think LaTeX is the poster child of this. Nobody writes a LaTeX preamble from scratch, you always copy your previous document and tweak it.

I'm an engineer at a company of about 70 people (about half of that engineers), and I personally review most applications to our engineering roles. About 60-70% of applications we receive are low-effort or automated spam of the kind generated by the author. We have screening questions that ask to describe a personal experience in your own words, specifically without LLM, and yet almost half of the applications we get have LLM answers, or a cover letter that is just an LLM-generated reflection of the job ad. Regularly the same candidate applies to all engineering roles, and then a few weeks later again, and again.

We use Recruitee, and ironically it doesn't have good automatic ways of filtering out the kind of spam generated by the author. On busy weeks, I spend about an hour per day screening and responding to applications. About half that is wasted on low-effort applications and automated spam generated by people like the author, and a significant part of that are repeat offenders. Nowadays I send one warning, and then I ask Recruitee support to ban the person, which due to implementation reasons on Recruitee's end prevents the person from applying at any company using Recruitee. It's harsh and I often feel bad about it, but after having to deal with this nonsense for multiple years now, I'm so sick of it, and I just ran out of patience.

Ctrl+N in Vim, and API docs.

I have a blog where I post a few posts per year. [1] /feed.xml is served with an Expires header of 24 hours. I wrote a tool that allows me to query the webserver logs using SQLite [2]. Over the past 90 days, these are the top 10 requesters grouped by ip address (remote_addr column redacted here):

    requests_per_day  user_agent
    283               Reeder/5050001 CFNetwork/1568.300.101 Darwin/24.2.0
    274               CommaFeed/4.4.0 (https://github.com/Athou/commafeed)
    127               Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
    52                NetNewsWire (RSS Reader; https://netnewswire.com/)
    47                Tiny Tiny RSS/23.04-0578bf80 (https://tt-rss.org/)
    47                Refeed Reader/v1 (+https://www.refeed.dev/)
    46                Selfoss/2.18 (SimplePie/1.5.1; +https://selfoss.aditu.de)
    41                Reeder/5040601 CFNetwork/1568.100.1.1.1 Darwin/24.0.0
    39                Tiny Tiny RSS/23.04 (Unsupported) (https://tt-rss.org/)
    34                FreshRSS/1.24.3 (Linux; https://freshrss.org)

Reeder is loading the feed every 5 minutes, and in the vast majority of cases it’s getting a 301 response because it tries to access the http version that redirects to https. At least it has state and it gets 304 Not Modified in the remaining cases.

If I order by body bytes served rather than number of requests (and group by remote_addr again), these are the worst consumers:

    body_megabytes_per_year  user_agent
    149.75943975             Refeed Reader/v1 (+https://www.refeed.dev/)
    95.90771025              Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
    75.00080025              rss-parser
    73.023702                Tiny Tiny RSS/24.09-0163884ef (Unsupported) (https://tt-rss.org/)
    38.402385                Tiny Tiny RSS/24.11-42ebdb02 (https://tt-rss.org/)
    37.984539                Selfoss/2.20-cf74581 (+https://selfoss.aditu.de)
    30.3982965               NetNewsWire (RSS Reader; https://netnewswire.com/)
    28.18013325              Tiny Tiny RSS/23.04-0578bf80 (https://tt-rss.org/)
    26.330142                Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36
    24.838461                Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36

The top consumer, Refeed, is responsible for about 2.25% of all egress of my webserver. (Counting only body bytes, not http overhead.)

[1]: https://ruudvanasseldonk.com/writing [2]: https://github.com/ruuda/sqlog/blob/d129db35da9bbf95d8c2e97d...

If you're looking for a human-friendly json superset (comments, non-quoted keys) that can also abstract away repetitive configuration with variables and list comprehensions, check out https://rcl-lang.org/.

This has been a good driver for me to add features to https://rcl-lang.org/. I just added List.sort for today’s problem.

Cool, it would be nice if that one aligned the days which were on weekends, as those tend to have harder problems.

My experience from predicting on Metaculus, and following this space for a few years, is that it's hard to operationalize the thing you want to predict precisely. Regularly, things go completely sideways in ways that the author did not foresee, and the market ends up hinging on some technicality, rather than the spirit of the question it tried to predict.

Some examples:

    - A question about asset prices specifies FTX as resolution source, but then FTX stopped existing.
    - There was a question about wether submarine cables in the Red Sea would be destroyed by a hostile act before a certain date. The cables got damaged, but it seemed to be a (suspicious) accident, with very limited independent media coverage.
    - There is a question about wether YouTube would be blocked in a certain country before 2025. It got throttled, to the point where it is unusable in practice, but not technically blocked.
    - There is a question trying to forecast LLM progress. How to quantify that? It chose "What is the state of the art score on the Penn Treebank at a certain date?", which was a standard benchmark at the time. But as LLMs evolved, new benchmarks got developed, and although current LLMs probably score much better on the Penn Treebank than a few years ago, nobody reports Penn Treebank score any more. The question ended up being about the popularity of the benchmark rather than LLM progress.

Some comments here point at impossibility results, but after screening hundreds of job applications at work, it's not hard to pick out the LLM writing, even without watermark. My internal LLM detector is now so sensitive that I can tell when my confirmed-human colleagues used an LLM to rephrase something when it's longer than just one sentence. The writing style is just so different.

Maybe if you prompt it right, it can do a better job of masking itself, but people don't seem to do that.

Pronunciation bears little relation to how words are written. For the longest time I thought I knew how to pronounce Greenwich, because I knew how to pronounce ‘green’ and ‘sandwich’. (Or things like advertising vs. advertisement, etc.) I saw a joke somewhere that western people think Chinese must be difficult because you have to memorize the pronunciation of so many symbols, but English is no different.

Chorus One | https://chorus.one/careers | Platforms Engineer | REMOTE (Switzerland ± 6 hours)

Chorus One operates validators on many proof-of-stake blockchains (the ones where security is based on a Byzantine fault-tolerant consensus algorithm rather than wasting energy). We are hiring for several roles, but the one I will highlight is what we call the Platforms Engineer. Some companies call this Site Reliability Engineering or Devops.

The main thing we do is take upstream software, build it, run it on our infrastructure, and then monitor it and optimize that setup. Some things that make this interesting are:

    * Building automation that enables us to do this for many networks (60+ currently).
    * Doing this with high uptime, building automation for failover, etc.
    * Working with software that is on the one hand cutting-edge and doing interesting things (consensus algorithms, distributed systems, cryptography), but on the other hand that means it’s immature and often not easy to operate and monitor. Often we have to build custom tools, and dive into the source code of the project. We contribute patches upstream when it makes sense.
    * Some of these projects are exercising the limits of what a machine can do, we have to do some low-level investigation that requires understanding of what the Linux kernel and network hardware are doing to properly identify what’s going on.

We do have a small cloud footprint but run primarily on bare metal. We are looking for people who can not just configure services offered by the public clouds, but who deeply understand what lies below; people who could build their own cloud. (That sounds a bit pretentious and it’s not exactly what we do, but it does involve many of the same aspects.)

If this sounds interesting to you, check out https://careers.chorus.one/o/platforms-engineer-remote. Aside from platforms engineers we are also hiring software engineers, see https://chorus.one/careers.

There are some downsides but also huge upsides aside from not wasting energy, for example offering very fast finality.

Bitcoin has not moved to it, because arguably the new system would not be Bitcoin any more. It's a coordination problem where you have to get most users (including centralized exchanges) to stop following the PoW chain and start respecting the PoS chain, but they will only do that if they believe everybody else will. Ethereum was able to pull that off because Ethereum foundation and Vitalik (its creator) announcing and implementing, and practicing the switch many times, has a lot of weight. (And then still, some miners remained, but that chain is now known by a different name and not generally known as "Ethereum".) Bitcoin is a lot less coordinated in that sense, even less invasive changes to the protocol are difficult to pull off.

> That a lot is expected and taken for granted from the security researchers by triagers that behave like you have to “prove to be worth listening to” while in reality they barely care to process and understand what you are saying

The unfortunate reality is that for every well-researched report like this one, you get 57 low-effort spam reports that hope to extract a bug bounty reward, or get a CVE discovery listed on their resume. Especially with the rise of LLMs that kind of spam can easily trick you. It's a sad situation, but I don't entirely blame developers for being skeptic.

There is a growing culture of microdependencies, where one project can depend on hundreds or thousands of libraries, combined with automated "vulnerability" tracking, which means projects are constantly receiving notifications about issues in libraries deep in the dependency tree, most of the time in a part of the library that is not even used by the top-level application. It's no surprise that "security" is eating up more and more time.

To add one to the mix: https://docs.ruuda.nl/squiller/

Yeah those others are in the triple dots menu, but paste is gone. Maybe I’m hallucinating that it was ever there, but I notice that I miss it which would be weird if it was never there in the first place.

I switched to Nemo after I got tired of Nautilus moving all the buttons for no particular reason every Gnome release, but a few versions ago they had a pattern that was genuinely good: a dropdown next to the current directory in the navigation bar, with everything you can do in the current directory (paste, create directory, open terminal, etc.). This was really neat, traditionally you have to access those by right-clicking some whitespace in the list/grid view, but in the list view there is only a narrow band of empty space to right-click, usually you accidentally click a file. So this was a genuine innovation in UI design. Unfortunately they since removed it again.

Yes, we hire worldwide as long as your working hours are compatible with Europe. (So US west coast and east Asia are hard, Europe shouldn't be a problem.)

We also have an open position for infrastructure software engineer, which is very similar to the platforms engineer I described above, but with more focus on software engineering and less on operational knowledge like networking. (In the end we work on the same things, we just have people with different strengths and preferences.)

We received a lot more applications there than we can interview so we de-listed it from the careers page temporarily, but you can still find it and apply through this url: https://careers.chorus.one/o/senior-software-engineer-1

Chorus One | https://chorus.one/careers | Platforms Engineer | REMOTE (Switzerland ± 6 hours)

Chorus One operates validators on many proof-of-stake blockchains (the ones where security is based on a Byzantine fault-tolerant consensus algorithm rather than wasting energy). We are hiring for several roles, but the one I will highlight is what we call the Platforms Engineer. Some companies call this Site Reliability Engineering or Devops.

The main thing we do is take upstream software, build it, run it on our infrastructure, and then monitor it and optimize that setup. Some things that make this interesting are:

    * Building automation that enables us to do this for many networks (60+ currently).
    * Doing this with high uptime, building automation for failover, etc.
    * Working with software that is on the one hand cutting-edge and doing interesting things (consensus algorithms, distributed systems, cryptography), but on the other hand that means it’s immature and often not easy to operate and monitor. Often we have to build custom tools, and dive into the source code of the project. We contribute patches upstream when it makes sense.
    * Some of these projects are exercising the limits of what a machine can do, we have to do some low-level investigation that requires understanding of what the Linux kernel and network hardware are doing to properly identify what’s going on.

We do have a small cloud footprint but run primarily on bare metal. We are looking for people who can not just configure services offered by the public clouds, but who deeply understand what lies below; people who could build their own cloud. (That sounds a bit pretentious and it’s not exactly what we do, but it does involve many of the same aspects.) If this sounds interesting to you, check out https://careers.chorus.one/o/platforms-engineer-remote.

Aside from platforms engineers we are also hiring software engineers, see https://chorus.one/careers.

A toy example: suppose we have some sudoku. You want to show publicly (maybe in a HN comment) that you know the solution, without revealing the solution itself, because then anybody would know it and be able to post that they know it. A zero-knowledge proof enables this. You could also post a hash of the solution, but then you need to know the solution already to verify a submission. (It would also enable others to copy your answers without really knowing the solution, though that can be fixed using a technique that zero-knowledge proofs also use, a blinding factor).

More useful cases include decoupling payment information from users, to preserve their privacy. You can prove that somebody paid for the action you want to perform, without identifying the payer. For example to offer cloud storage without knowing which data belongs to which user, so when there is a data breach or law enforcement order, the answer to "tell me everything you know about user X" is their payment history, but not which data is theirs.

Which is odd, given that it contains zero interactive elements even with js enabled.