janjongboom

This false sense of reproducability is why I funded https://docs.stablebuild.com/ some years ago. It lets you pin stuff in dockerfiles that are normally unpinnable like OS package repos, docker hub tags and random files on the internet. So you can go back to a project a year from now and actually get the same container back again.

Still gonna break everyone’s CI until they manually update the tag. (And who guarantees that these tags will stay alive after they pull this)

The removal (or moving) of the Bitnami images from Docker Hub is going to break a ton of systems that depend on them. I helped set up https://www.stablebuild.com/ some years ago to counter these types of issues, it provides (among other things) a transparent cache to Docker Hub which automatically caches image tags and makes them immutable - underlying tag might be deleted or modified, but you’ll get the exact same original image back.

Seconding this - absolutely terrific content.

FYI, I've helped set up StableBuild (https://www.stablebuild.com) to help pin stuff in Docker that's normally virtually impossible to pin (e.g. OS package repos, Docker base images, random files from the internet, etc.)

Not never. E.g. all the capital we as founders put in the business before we raised our seed round was converted into Series Seed Preferred shares at the same rights as angels / seed VC. Small portion of total equity but still.

Packages and versions can be deleted from PyPI, which can be a massive pain in the ass for anyone consuming these packages. Can have your whole Python dependency tree pinned => author pulls a package version => builds broken. As part of StableBuild (https://www.stablebuild.com) we create full daily snapshots of the PyPI registry - so figured it would be nice to make an overview of deleted packages/versions and make the wheels available for download.

E.g. jaxlib 0.4.4 was removed a few days back: https://dashboard.stablebuild.com/pypi-deleted-packages/pkg/... => can download the wheels for free w/o registration

I’m an investor in StableBuild.

Yes, fantastic work. Downside is that snapshot.debian.org is extremely slow, times out / errors out regularly - very annoying. See also e.g. https://github.com/spesmilo/electrum/issues/8496 for complaints (but it's pretty apparent once you integrate this in your builds).

> Any source of that claim?

Any tag like ubuntu:20.04 -> this tag gets overwritten every time there's a new release (which is very often)

https://hub.docker.com/r/nvidia/cuda -> these get removed (see e.g. https://stackoverflow.com/questions/73513439/on-what-conditi...)

Yeah, very similar approach. We did this before, see e.g. https://www.stablebuild.com/blog/create-a-historic-ubuntu-pa... - but then figured everyone needs exactly the same packages cached, so why not set up a generic service for that.

Until someone overwrites or deletes the Docker base image (regularly happens), or when you depend on some packages installed through apt - as you'll get the latest version (impossible to pin those).

Yeah, but it's impossible to properly pin w/o running your own mirrors. Anything you install via apt is unpinnable, as old versions get removed when a new version is released; pinning multi-arch Docker base images is impossible because you can only pin on a tag which is not immutable (pinning on hashes is architecture dependent); Docker base images might get deleted (e.g. nvidia-cuda base images); pinning Python dependencies, even with a tool like Poetry is impossible, because people delete packages / versions from PyPI (e.g. jaxlib 0.4.1 this week); GitHub repos get deleted; the list goes on. So you need to mirror every dependency.

Ah, yes, on mobile it shows the wrong pricing table... Copying here while I get it fixed:

Free => Access to all functionality, 1 user, 15GB traffic/month, 1GB of storage for files/URLs. $0

Pro => Unlimited users, 500GB traffic included (overage fees apply), 1TB of storage included. $199/mo

Enterprise => Unlimited users, 2,000GB traffic included (overage fees apply), 3TB of storage included, SAML/SSO. $499/mo

Apologies for editorializing the title a bit :-)

Reading through the comments in https://news.ycombinator.com/item?id=39720007 I saw a common misconception pop up again: Dockerfiles are not deterministic. But they _look_ like they are, and even are for a while: Build a Dockerfile on your local machine; then build it again => most likely exactly the same container. Stuff starts to break down quickly though; so I did this writeup some time ago that should be informative for the wider community.

And that assumes that `foo` and `bar` are not overwritten or deleted in your package repository, and that the git repository remains available.

That's the interesting bit about Dockerfiles. They look _looks_ deterministic, and they even are for a while while you're looking at it as a developer. I've done a detailed writeup of how it's not deterministic in https://docs.stablebuild.com/why-stablebuild

I've gone down the same path. I love deterministic builds, and I think Docker's biggest fault is that to the average developer a Dockerfile _looks_ deterministic - and it even is for a while (build a container twice in a row on the same machine => same output), but then packages get updated in the package manager, base images get updated w/ the same tag, and when you rebuild a month later you get something completely different. Do that times 40 (the number of containers my team manages) and now fixing containers is a significant part of your job.

So in theory Nix would be perfect. But it's not, because it's so different. Get a tool from a vendor => won't work on Nix. Get an error => impossible to quickly find a solution on the web.

Anyway, out of that frustration I've funded https://www.stablebuild.com. Deterministic builds w/ Docker, but with containers built on Ubuntu, Debian or Alpine. Currently consists of an immutable Docker Hub pull-through cache, full daily copies of the Ubuntu/Debian/Alpine package registries, full daily copies of most popular PPAs, daily copies of the PyPI index (we do a lot of ML), and arbitrary immutable file/URL cache.

So far it's been the best of both worlds in my day job: easy to write, easy to debug, wide software compatibility, and we have seen 0 issues due to non-determinism in containers that we moved over to StableBuild in my day job.

Not sure about Earthly as I’ve never used it, but for Nexus and other artifactory solutions around package repos:

1. Don’t need to set up anything. No need to set up an apt repo and push packages or configure a mirror, because StableBuild already caches the complete registry and thus has everything. 2. Don’t need to think about the complete package list when pushing files to artifactory. Have packages cached from 3 months ago, now want to add another one? Oops, not in Nexus, and the current versions in the Ubuntu package registry are not compatible with your cached versions -> need to update the full dependency tree. 3. Integration is trivial. Three lines to your Dockerfile and done. 4. Can retroactively fix things. Knew that this container built 4 weeks ago? Ok use that as a pin date -> fixed.

Then there’s some stuff, like the immutable Docker pullthrough cache and history pypi mirror that I haven’t seen before (but I’d like to learn if others are doing this :-)).

Things Edge Impulse customers have in production: Sleep stage prediction, fall detection for elderly, fire detection in power lines, voice command recognition on headsets, predicting heath exhaustion for first responders, pet feeders that recognize animals, activity trackers for pets, and many more.

I worked on that for a while, see http://janos.io . Interesting idea, but impractical due to the amount of work that needed to be done per device.

Hey I wrote this :-) But back in 2016 or 2017, so some of the practical stuff might be out of date.

Go to https://archive.ph/ and paste in the original URL.

Reposting an analysis I did 6 months ago, where their numbers did not make any sense to me compared to any other LoRaWAN deployment; and probably impossible - bound by physics - to ever generate enough revenue to pay their gateway operators at their data cost / gateway rewards: https://news.ycombinator.com/item?id=29944979

The math for Helium just does not make any sense. Looking at forum threads every gateway makes about $50 in rewards / month. That's $294,000,000 per year in rewards paid out to gateway owners.

Sending one packet costs $.00001 (100K packets for $1 according to their website). So we need to see 29,400,000,000,000 packets yearly on the Helium network for data fees to cover the gateway rewards.

Looking at data from The Things Network - who operate another LoRaWAN network - from their conference last year they mention routing 600 packets/second using 30,000 gateways => that's 630,720 packets per gateway per year.

Assuming that Helium sees the same ratio of packets, and that every packet is unique, and that every packet is meant for the Helium network (and thus paid) this yields an expected (630,720 * 490,000) 309,052,800,000 paid packets => 309,052,800,000 * 0.00001 = $3,090,528 in data fees.

So ~3 million $ in revenue (in very best case scenario) from data fees, but paying out ~294 million $ to gateway owners.

Naturally the only way this works is because they give out their own invented tokens rather than dollars. This does not make any sense, and will never make any sense. I read that the gateway manufacturers pay the Helium company 50$ per gateway (to provision a private key) which probably pays for some stuff (and to pump up their own coin), but if that's the case then it just looks like a pyramid scheme.

Also scale doesn't help if you pay out 100x more for every gateway than you can potentially earn per gateway. But that should be obvious. You're paying a gateway for 600$ and then you get >600$ in rewards per year. This does not make any sense.

Edge Impulse | Infrastructure, embedded and frontend | Fulltime | REMOTE (US/EU only)

Edge Impulse makes embedded devices understand the real world. We build software that lets engineers use machine learning to analyse sensor data in realtime. Our customers use this to track sleep, monitor power lines, listen for elephants, and much more. We also have an amazing community that has created >50,000 projects on the platform.

We're ~40 people now, all remote, and are looking to grow the team significantly in 2022. Currently looking for an infrastructure engineer, an embedded developer and a frontend engineer. More info: https://docs.edgeimpulse.com/page/jobs

I'd suggest to get rid of the custom mouse pointer on your site, it's pretty annoying to have it lagging behind actual movement.

Edge Impulse | Full-stack Web Engineer | Full-time | Remote (US/Europe only) | https://edgeimpulse.com

Are you excited about bringing true intelligence to the smallest of devices? Edge Impulse lets embedded developers use signal processing and machine learning to make their devices understand the real world. We have a fantastic community of 10,000+ developers, are involved in open source projects like TensorFlow, and have a great team of 25+ people who care about their work-life balance.

We're looking for a full-stack web engineer who can help us build out our product. Your main job will be working on the Edge Impulse Studio, our main product for building machine learning models. It's the place where developers explore data and where customers run thousands of parallel jobs. The Studio is mainly written in TypeScript, but we have some pieces in Python and C++, and would love to talk to anyone who feels they can contribute to (or lead) development of both backend and frontend services. We're fully remote, with meeting spaces in Amsterdam and San Jose, CA.

Interested? See the full job ad here: https://docs.edgeimpulse.com/page/full-stack-web-developer-t... !

The Oculust Quest 2 is my first VR headset, and it can cast to the TV. That helps tremendously with the social aspect, rest of the room can see what you're doing and give feedback. In addition we've been playing Eleven - a table tennis game - in multiplayer with two people in the same room (with two headsets) which is tremendous fun, but requires a second headset.

Edge Impulse | Infrastructure Engineer | Remote (US/Europe only) | Full-time | https://docs.edgeimpulse.com/page/ml-infrastructure-engineer

At Edge Impulse we add real intelligence to tiny devices by letting our users build Machine Learning models that operate on sensor data and run in kilobytes of RAM. Since our launch in January 2020 our community has created >8,000 projects, collected >4 million data samples from real sensors, and we ran almost 200,000 compute jobs. To keep up with that scale we're hiring an infrastructure engineer that can help us run thousands of parallel jobs, keep our infrastructure up and running, and help us design for the thousands of new users we expect in 2021. If you know how to deal with a Kubernetes cluster, know how to manage cloud infra, and want to learn about applying Machine Learning to real usecases, I'd love to talk to you!

About us: We're a remote-equal, well funded startup with significant revenue and founders who care about work-life balance, open source and our community. If that all sounds interesting, let me know at jan@edgeimpulse.com !