mleonhard

The problem is that the system is RAM-constrained but the work scheduler doesn't use a RAM pressure signal.

We could solve this with a crate that rate-limits new tasks based on heap usage.

We could build this into the runtime and make it apply to all task spawns. That could be problematic since any spawn could block.

It would probably be better to make it opt-in and less granular, releasing groups of tasks at a time.

It's hard to tell how much care was put into futures_orchestra crate. The docs are generated. I don't see any benchmark in the repo. Performance claims without measurements are a red flag.

> Inexplicably, major advances in this area 15-20 years ago under the auspices of government programs never bled into the academic literature even though it materially improved the situation.

Would you please share some more info about this? Were the advances implemented in software and never written up and published? What are the names of the government programs?

s/Prolificacy/Verbosity/

Over-exaggeration also works well when learning to play stringed instruments like cello.

Is any datacenter's water use significant compared to other industrial installations? According to that article, all datacenters in North Holland use 550 Ml/yr. North Holland has 2.95M residents [0], who use 129 l/person-day [1], 47 Kl/person-year, 139,000 Ml/year for the whole region. So the data centers use an estimated 0.4% of the region's water. Data centers use about 3% of the Netherlands' electricity.

Why do you think this is a lot of water? What are the alternatives to pulling from the local water utility and are those alternatives preferable?

[0] https://en.wikipedia.org/wiki/North_Holland

[1] https://en.wikipedia.org/wiki/Water_supply_and_sanitation_in...

[2] https://www.dutchdatacenters.nl/en/statistics-2/

> // Start a background task that takes the lock and holds it for a few seconds.

Holding a lock while waiting for IO can destroy a system's performance. With async Rust, we can prevent this by making the MutexGuard !Send, so it cannot be held across an await. Specifically, because it is !Send, it cannot be stored in the Future [2], so it must be dropped immediately, freeing the lock. This also prevents Futurelock deadlock.

This is how I wrote safina::sync::Mutex [0]. I did try to make it Send, like Tokio's MutexGuard, but stopped when I realized that it would become very complicated or require unsafe.

> You could imagine an unfair Mutex that always woke up all waiters and let them race to grab the lock again. That would not suffer from risk of futurelock, but it would have the thundering herd problem plus all the liveness issues associated with unfair synchronization primitives.

Thundering herd is when clients overload servers. This simple Mutex has O(n^2) runtime: every task must acquire and release the mutex, which adds all waiting tasks to the scheduler queue. In practice, scheduling a task is very fast (~600ns). As long as polling the lock-mutex-future is fast and you have <500 waiting tasks, then the O(n^2) runtime is fine.

Performance is hard to predict. I wrote Safina using the simplest possible implementations and assumed they would be slow. Then I wrote some micro-benchmarks and found that some parts (like the async Mutex) actually outperform Tokio's complicated versions [1]. I spent days coding optimizations that did not improve performance (work stealing) or even reduced performance (thread affinity). Now I'm hesitant to believe assumptions and predictions about performance, even if they are based on profiling data.

[0] https://docs.rs/safina/latest/safina/sync/struct.MutexGuard....

[1] https://docs.rs/safina/latest/safina/index.html#benchmark

[2] Multi-threaded async executors require futures to be Send.

I spent some years working for a large NGO (Opportunity International) and living with people who work for NGOs.

NGOs must constantly raise money to fund their operations. The money that an NGO spends on fund-raising & administration is called "overhead". The percentage of annual revenue spent on overhead is the overhead percentage. Most NGOs publish this metric.

When a big donor stops contributing, the NGO must cut pay or lay off people and cut projects. I've never heard of an NGO "succumbing to excessive staff costs" like a startup running out of money. Financial mismanagement does occasionally happen and boards do replace CEOs. Board members are mostly donors, so they tend to donate more to help the NGO recover from mismanagement, instead of walking away.

NGOs pay less than other organizations, so they mostly attract workers who care about the NGO's mission. These are people with intrinsic motivation to make the NGO succeed in its mission. Financial incentives are a small part of their motivations. For example, my supervisor at Opportunity International refused several raises.

> So they go around addressing individual problems, taking sad pictures, and avoid addressing systemic problems.

Work on individual problems is valuable. For example, the Carter Center has prevented many millions of people from going blind from onchocerciasis and trachoma [0].

The Carter Center is not directly addressing the systemic problems of poverty and ineffective government health programs. That would take different expertise and different kinds of donors.

The world is extremely complicated and interconnected. The Carter Center's work preventing blindness directly supports worker productivity in many poor countries. Productivity helps economic growth and reduces poverty. And with more resources, government health programs run better.

Being effective in charity work requires humility and diligence to understand what can be done now, with the available resources. And then it requires tenacity to work in dangerous and backward places. It's an extremely hard job. People burn out. And we are all better off because of the work they do.

When we ignore the value of work on individual problems, because it doesn't address systemic problems, we practice binary thinking [1]. It's good to avoid binary thinking.

[0] https://en.wikipedia.org/wiki/Carter_Center#Implementing_dis...

[1] https://en.wikipedia.org/wiki/Splitting_(psychology)

When I used AWS startup credits in 2019, the AWS console made it very difficult to estimate the bill after the credits ran out. I lost a lot of trust in AWS. Also, there were buried mines in the APIs, like the risk of bad logging running up a $70,000/day bill with CloudWatch Logs.

If I could go back and do it again, I would rent a single machine and deploy with ssh (git pull & docker-compose up) and backup to my laptop.

I think that async in Rust has a significant devex/velocity cost. Unfortunately, nearly all of the effort in Rust libraries has gone into async code, so the async libraries have outpaced the threaded libraries.

There was only one threaded web server, https://lib.rs/crates/rouille . It has 1.1M lines of code (including deps). Its hello-world example reaches only 26Krps on my machine (Apple M4 Pro). It also has a bug that makes it problematic to use in production: https://github.com/tiny-http/tiny-http/issues/221 .

I wrote https://lib.rs/crates/servlin threaded web server. It uses async internally. It has 221K lines of code. Its hello-world example reaches 102Krps on my machine.

https://lib.rs/crates/ehttpd is another one but it has no tests and it seems abandoned. It does an impressive 113Krps without async, using only 8K lines of code.

For comparison, the popular Axum async web server has 4.3M lines of code and its hello-world example reaches 190Krps on my machine.

The popular threaded Postgres client uses Tokio internally and has 1M lines of code: http://lib.rs/postgres .

Recently a threaded Postgres client was released. It has 500K lines of code: https://lib.rs/crates/postgres_sync .

There was no ergonomic way to signal cancellation to threads, so I wrote one: https://crates.io/crates/permit .

Rust's threaded libraries are starting to catch up to the async libraries!

---

I measured lines of code with `rm -rf deps.filtered && cargo vendor-filterer --platform=aarch64-apple-darwin --exclude-crate-path='*#tests' deps.filtered && tokei deps.filtered`.

I ran web servers with `cargo run --release --example hello-world` and measured throughput with `rewrk -c 1000 -d 10s -h http://127.0.0.1:3000/`.

I think Cisco SNMP vulnerabilities have been appearing for 20 years or more. I wish someone would add a fuzzer to their release testing script.

I took an "Architecting on AWS" class and half of the content was how to replicate complicated physical networking architectures on AWS's software-defined network: layers of VPCs, VPC peering, gateways, NATs, and impossible-to-debug firewall rules. AWS knows their customers tho. Without this, a lot of network engineers would block migrations from on-prem to AWS.

How does one handle errors with MESH?

To handle errors in HTMX, I like to use config from [0] to swap responses into error dialogs and `hx-on-htmx-send-error` [1] and `hx-on-htmx-response-error` [2] to show the dialogs. For some components, I also use an `on-htmx-error` attribute handler:

    // https://htmx.org/events/
    document.body.addEventListener('htmx:error', function (event: any) {
        const elt = event.detail.elt as HTMLElement
        const handlerString = elt.getAttribute('on-htmx-error')
        console.log('htmx:error evt.detail.elt.id=' + elt.getAttribute('id') + ' handler=' + handlerString)
        if (handlerString) {
            eval(handlerString)
        }
    });

This gives very good UX on network and server errors.

[0]: https://htmx.org/quirks/#by-default-4xx-5xx-responses-do-not...

[1]: https://htmx.org/events/#htmx:sendError

[2]: https://htmx.org/events/#htmx:responseError

Yes. With HTMX, one can put a page definition and its endpoints in one file. It has high cohesion.

There's no integration with routers, state stores, or rpc handlers. There are no DTOs shared between the frontend and backend. It has low coupling.

High cohesion and low coupling bring benefits in engineering productivity.

VPN providers do not have reputations for making secure or reliable software.

Here's a good privacy proxy (VPN) setup: Set up a second wifi router, enable the "Internet kill switch", and connect it with Wireguard to a reputable VPN service. I recommend GL.iNet routers and Mullvad.

With this setup, one can move individual devices between the privacy wifi and identity-broadcasting wifi.

According to IIHS insurance loss data, https://www.iihs.org/research-areas/auto-insurance/insurance... , here's the chance of being injured while riding in one of these cars (and filing an insurance claim) relative to the average US vehicle:

    - Nissan Leaf -15% (Select "Small" & "4-door Cars" on the page)
    - Chevy Bolt -34%
    - Subaru Crosstrek -28% ("Station wagons" & "Small")
    - Tesla Model 3 +26% ("Luxury cars" & "Midsize")

So the choice of Nissan Leaf was OK from a safety perspective, but the Chevy Bolt is better. The Tesla is much worse.

BART is a government organization and all California government employee pay is public. You can see that BART has about 40 software engineers and they earn about 70% of the market rate:

https://transparentcalifornia.com/salaries/search/?q=compute...

https://transparentcalifornia.com/salaries/search/?q=compute...

It seems to me that they are over-worked & under-paid and are doing a good job given the circumstances.

NIMBYs have blocked BART in Silicon Valley. BART doesn't reach Menlo Park, Palo Alto, Stanford, Mountain View, Sunnyvale, Los Altos, Santa Clara, or Cupertino. A few years ago, it finally reached San Jose.

A separate train (CalTrain) goes from SF through Silicon Valley. Last year they switched to electric trains which are faster and run more frequently. The SF CalTrain station is inconvenient (20-mins walk from downtown, under a highway), but they are working to extend CalTrain to the central SF station: https://en.wikipedia.org/wiki/Salesforce_Transit_Center#Futu... .

So Silicon Valley transit is getting better, slowly.

    Location: San Francisco, CA
    Remote: Yes
    Willing to relocate: Yes
    Technologies: Rust, TypeScript, Golang, Java, Ruby, Python, Postgres, htmx, GitHub Actions, CircleCI, AWS, ECS, Kubernetes, Datadog, Sentry, React, HTML/CSS, Linux, Snowflake, and many others.
    Résumé/CV: https://joblin.app/profile/409373959
    Email: michael206@gmail.com

I'm a generalist software engineer with strong opinions on how to build and maintain software while balancing quality, speed, and cost. I've worked at FAANG and startups. I'm a fan of design docs, lunch&learns, clarity, and deleting code. I care a lot about doing a good job and making things meet user needs. I also love unblocking teammates & other teams.

I've published a lot of code online, mostly Rust libraries, with good tests. I like working in the office, feeling camaraderie with my teammates.

> There is no “simplifying force” acting on the code base other than deliberate choices that you make. Simplifying takes effort, and people are too often in a hurry.

There is a simplifying force: the engineers on the project who care about long-term productivity. Work to simplify the code is rarely tracked or rewarded, which is a problem across our industry. Most codebases I've worked in had some large low-hanging-fruit for increasing team productivity, but it's hard to show the impact of that work so it never gets done.

We need an objective metric of codebase cognitive complexity. Then folks can take credit for making the number go down.

If the leaders of these companies actually cared about worker productivity, they would grab the low-hanging fruit and give their employees workspaces where they can concentrate, not crowded open-plan offices surrounded by meeting rooms.

Does it have tooling to help keep Python [0] and TypeScript [1] DTOs in sync?

[0] https://github.com/litestar-org/litestar-fullstack/blob/0996...

[1] https://github.com/litestar-org/litestar-fullstack/blob/0996...

Please explain.

Your comment expresses what I've been thinking. Doing the Staff Engineer role requires close alignment with one's management chain. Working at level 2 (working for the success of the company) is only possible with a manager who also works at level 2. How to find such people and get hired by them?

I found a useful intro "Mastering Claude Code in 30 minutes - Anthropic" https://www.youtube.com/watch?v=6eBSHbLKuN0

    Location: San Francisco, CA
    Remote: Yes
    Willing to relocate: within the SF Bay Area
    Technologies: TypeScript, Rust, Golang, Java, Ruby, Python, Postgres, htmx, GitHub Actions, CircleCI, AWS, ECS, Kubernetes, Datadog, Sentry, React, Cursor, and many others.
    Résumé/CV: https://joblin.app/profile/409373959
    Email: michael206@gmail.com

I'm a generalist software engineer with 10+ years of experience. I've worked at FAANG and startups. I enjoy writing, fixing, and deleting code and tests. I care a lot about doing a good job and making things meet user needs.

I solved my RSI symptoms by keeping my arms warm all the time, while awake or asleep. Maybe that will work for you, too?

  Location: San Francisco, CA
  Remote: Yes
  Willing to relocate: within the SF Bay Area
  Technologies: TypeScript, Rust, Golang, Java, Ruby, Python, Postgres, htmx, GitHub Actions, CircleCI, Kubernetes, Datadog, Sentry, React, and many others.
  Résumé/CV: https://joblin.app/profile/409373959
  Email: michael206@gmail.com

I'm a generalist software engineer with 10+ years of experience. I've worked at FAANG and startups. I enjoy building and fixing things to meet business & user needs. I also love finding ways to increase velocity.

The website blocks the privacy proxy (VPN) service I use.

Looking at the paper, it seems like they put some silicon-dioxide nanoparticles on a substrate, then add a plastic (poly-ethylene) layer on top and melt it (annealing). The spaces between the nanoparticles gets partially filled with plastic. The ratio of plastic to particles is the poly-ethylene volume fraction (ϕPE). They tested different fractions and found that a certain range caused the wetting behavior.

Their experiments suggest that tiny water droplets appear inside the material at 70% RH (relative humidity). If this is true, then I expect there is a way to extract the droplets using very little energy. Ideas:

- make open collection points on the film

- use ultrasound to bounce the droplets around and consolidate them

- make the film on a material that can be saturated with water so the new droplets can easily join the flow

https://en.wikipedia.org/wiki/Volume_fraction