Writing a Mini-CDN to Learn Nginx/Prometheus/Grafana/Lua
github.com
github.com
CloudFlare is free at my tier and gives me the ability to have the lowest latency.
The lowest latency solution is to put the content near the user and a CDN is probably the easiest way of doing that if someone needs to server a geographically dispersed audience
Also, geography: ping time from Singapore to the US can never be negligibly short, which is pretty noticeable during TLS handshakes.
But yes, likely these considerations do not matter for 90% of websites (not 99.9% though).
What exactly leads you to believe you can tell what 99.9% of all websites need?
Unless you believe 99.9% of websites are only accessed by your upstairs neighbors, CDNs provide a couple of important business and operational advantages.
Just to illustrate how wrong and misguided your personal assumption is, CDNs are primarily used to cut down latency, which in accesses from other regions can easily go beyond 300ms. Unless you somehow think that it's ok for your users to be subjected to a bad user experience, a basic CDN service is all you need to employ to lower those latencies by an order of magnitude.
I accept that my images are being served a bit slower but with the compromise that I can server a lot more images.
I also think the person your are replying to is correct. The CDN makes the user experience worse but saves me money, that’s an unfortunate sacrifice.
This was a very informative read!
If these are public, put them on /favorites/$USERNAME or something similar. If they are private, don't cache them.
You can cache with specific headers as cache keys, but I would advise against doing this too much / abusing it. It really makes caching complicated. And from a data privacy standpoint it's better to opt-in into caching. I've witnessed incidents where visitors saw the private profile page of another user, because it was cached in the CDN.
Fly.io also has a Grafana dashboard built in for your machines
I periodically consider a grafana & backend setup for when datadog becomes cost prohibitive for metrics with several tags.
Haven't used influxdb yet so can't speak as a comparison but from my usage, I'm sold on Grafana, Loki, Prometheus, and friends over DataDog. It mixed with OTel have been a real pleasure to use.
So a classic build vs buy story I guess. Probably will pay less in the long run but for a worse and rocky road experience so far.
Now we are considering to switch to thanos or mimir.
Thanks!
Per the first line of the link: "The objective of this repo is to build a body of knowledge on how CDNs work by coding one from "scratch". "
That may feel uselessly broad if you're not familiar with the language or ecosystem but I've used lua professionally for years on projects of all different sizes and it's a truth I recognize.
In this situation, usually what you're embedding lua for is defining and exposing a DSL to handle complex configuration or mediate some automation.
So these days I use janet for this because I like lisp, its data structures and core functions are predictable in a way lua's are not, and it has a good standard library that is easy to select subsets of for embedding. Lisps are particularly well suited for making DSLs and it has macros if you need them though I never have.
TCL is another excellent choice. It is small and embeds very simply like lua, but is more suited for making DSLs and even GUI config if you need to go that far. And this is subjective but I think it's less hostile to the probably non-professional programmers that are likely the end users. It has a similar C-centric embedded history so is similarly optimized for that case, but with more focus on users not needing to learn the whole language to effectively use a part of it.
For some cases where what the end user will define is not configuration but processes or procedures over unknowable-at-build-time data, forth is an unusual but strong choice. There are some incredibly lean implementations built for embedding. The paradigm is most likely to be unfamiliar to the users but for some use cases it's such a good fit it's worth it.
The cases where I would use lua are where you expect broad and long-lived community development with high complexity and dedicated system builders involved. Lua's metatables and module system are a good foundation to build a powerful ruby-like OO/FP hybrid environment if it's worth adding all that weight and maintaining it over time. You see something like this in mud client scripting where lua is conventional and I think appropriate, and the clients themselves function more like platforms for lua development than apps that run embedded scripts.
The main thing imo is just to think about what the reason for embedding actually is, who is going to be using it, and for what. Lua isn't the worst default, but its flexibility makes developers think they don't have to make this choice at all if they include it, and that's an error.