If anyone wants to try an early beta, reply with your CircuitLab username and I'll get you set up. We know have a lot more to do, but would love some feedback on this early version!
4,398 karma · joined October 6, 2009
Co-founder of CircuitLab (YC W13): https://www.circuitlab.com/
Formerly I ran Growth at Triplebyte (YC S15): https://triplebyte.com/
If anyone wants to try an early beta, reply with your CircuitLab username and I'll get you set up. We know have a lot more to do, but would love some feedback on this early version!
The default kubelet `syncFrequency` is 1 minute (per Pod). (There can be additional event-driven ones, but this is a floor.) Back of the envelope: 30 Pods * 1440 * 7 = 300k calls to startPodSync in a week.
I’d guess a lot of production clusters don’t run the latest release of Kubernetes, and 1.36 was only released in late April, less than two months before I reported the issue.
Given that the issue shows up as linear memory growth on the order of maybe 1 GiB/month per node, there really is a time-based discovery process here. (Could be artificially accelerated for a CI test, I imagine!) And any restarted nodes or clusters that upgraded point releases would restart that clock.
The only unusual contributing cause that made the issue more visible to me was trying to run everything on a tiny 2 GiB RAM node!
I took a quick look at "containedctx" and it seems like for this case, it would almost be backwards: it would flag the (not-memory-leaking) struct-stored "status.ctx", but wouldn't flag when there is a stored "status.cancelFn" only (which resulted in the memory leak).
Could Go implement a runtime leaked-context detector, like the data race detector? https://go.dev/doc/articles/race_detector
# cat /proc/$(pgrep kubelet)/oom_score_adj
-999
(from OOMScoreAdjust=-999 in /etc/systemd/system/kubelet.service)
With this score, the Linux OOM killer wouldn't touch it, but any of my Pods were fair game. lostcancel: check cancel func returned by context.WithCancel is called
I'm not 100% sure why `go vet` didn't catch this issue, but storing the cancelFn in the struct is probably part of the reason. Any Go experts know if that's the case?If you haven't come across Kvrocks yet, it may be worth a look: https://github.com/apache/kvrocks https://kvrocks.apache.org/ . It's a database with a Redis-compatible wire protocol, but the database is stored on disk. This means your working set is not limited by RAM and can be a few orders of magnitude larger! On modern SSDs this is still very fast. I think it improves the durability story as well. But the big win is the orders of magnitude larger database space.
As I've been improving my side project https://totalrealreturns.com/ recently I've ended up using both Redis and Kvrocks together. Redis is great for small global state that needs to be super fast. Kvrocks is great for larger bulk data storage (large precomputed datasets), but also supports a lot of the Redis data structures as well as Lua scripts.
If my DB health check endpoint is returning 500s for N consecutive checks over M minutes, yeah, please wake me up at 3am!
If one user hit a weird edge case in form validation and got a one-off 500, please don't! We can fix that on Monday.
Not always easy to distinguish those clearly or configure those business hours rules, but for my team at https://heyoncall.com/ that is the goal -- otherwise your team burns out fast. Waking up someone at 3am has a real cost, so you better be sure it's worth it.
raise ArgumentError.new("...") unless ...
which can include type assertions but also a lot more. The agents seem to do well with this.I've also had good results using agents to write Crystal https://crystal-lang.org/ which is Ruby-like but does have the static types and produces blazing fast static binaries. Might be a sweet spot for coding agents if you're building some backend services. But I'd still pick Ruby on Rails for a new full stack project.
The edges drawn are between nodes that have been able to hear each other in the last 24 hours, based on observed traceroute packets.
(Even then, it’s only a subset of the actually-connected nodes: the map only shows nodes that have published their position on the public channel, and have set a flag that their data is okay to uplink to a server over MQTT.)
1. These grid tie inverters are designed and tested to shut off completely if there's no grid power. (This is a big design tradeoff: it means they don't provide any power during a grid power outage, even if it's very sunny out.)
2. Even if I had a beefy generator that was unsafely backfeeding my house while the utility power was still connected, the generator would be trying to power not just my house, but all my neighbors too! And the circuit breaker and/or inverter on the generator would likely trip and shut down almost instantly.
There's still a possible risk from #2, especially if the downed wire being repaired is relatively local (i.e. your house only).
But I think #1 and #2 mitigate this risk very well.
But I think plug-in / balcony solar will be pretty cool. And I think there's a path to inexpensive, larger, safer grid-tie inverters which never backfeed, but prioritize solar input first and make up the difference with grid power.
For example, I'm imagining a box that would plug in to the wall, have a DC input from solar panels, and a power strip for loads supporting up to, ideally, a full 15A normal US 120V circuit.
Currently this box exists in the form of battery power station units (Bluetti, Ecoflow, Anker etc). But I think there could be a much less expensive form that could exist without the battery.
I think this is a key point. One could imagine three layers:
(1) NETLIST: a list of text-only descriptions, like "R1 1 0 1k", or even a sentence description "connect a 1k resistor between node 1 and node 0"
(2) SCHEMATIC: a 2D drawing with canonical symbols, straight lines for wires, and labels/notes
(3) LAYOUT: a 3D (or multi-layer 2D) physical representation for PCB, or in this case breadboarding
All three layers are useful. (Obviously you need layout to make a PCB, and you need a netlist for simulation.)
But for most humans, where we have 2D visual representations baked in, if you're trying to understand or communicate what's going on:
- It's really really hard to keep track of a bunch of text sentences like a netlist and node numbers/names for all but the simplest circuits -- maybe 3-5 elements?
- It's really really hard to follow a 3D layout of PCB tracks that leads to pads, and then having to remember pin orders etc.
- It's easiest to follow a schematic diagram. It's browsable. It contains "idioms", as you say, about signal flow, logical block groupings, etc.: purpose and intent and functionality, in a way that netlists and physical layouts don't.
FYI, for medium-large digital circuits, I don't think this is true: probably just reading VHDL/Verilog, like reading source code, makes more sense. This is closer to the "netlist" level. I think that's because you'd name modules and inputs/outputs in a way similar to how you'd name functions and arguments in software, which doesn't really apply to "Resistor" or "Capacitor" as primitives.
But for a pretty big practical range of mixed-mode and analog things, I'd argue that schematics really are the easiest level for our brains.
(Disclosure: I'm one of the founders of CircuitLab https://www.circuitlab.com/ (YC W13) where we've been building an online circuit simulator & schematic editor for a long time. Although I'm mostly on the simulation engine / netlist side. My cofounder and other teammates have done most of the schematic GUI work.)
IMHO solderless breadboards still have their place for prototyping some slow circuits, ballpark maybe < 1 MHz signals, if you're aware of the extra capacitance and limitations. :)
In the HTTP/1.1 (1997) or HTTP/2 era, the TCP connection is made once and then stays open (Connection: Keep-Alive) for multiple requests. This greatly reduces the number of DNS lookups per HTTP request.
If the web server is configured for a sufficiently long Keep-Alive idle period, then this period is far more relevant than a short DNS TTL.
If the server dies or disconnects in the middle of a Keep-Alive, the client/browser will open a new connection, and at this point, a short DNS TTL can make sense.
(I have not investigated how this works with QUIC HTTP/3 over UDP: how often does the client/browser do a DNS lookup? But my suspicion is that it also does a DNS query only on the initial connection and then sends UDP packets to the same resolved IP address for the life of that connection, and so it behaves exactly like the TCP Keep-Alive case.)
Many larger homes in this area have whole-house generators (powered by utility natural gas) with automatic transfer switches. During the 50-hour outage, we "abandoned ship" and stayed with someone who also had an outage, but had a whole-house generator.
Other areas just 5-10 miles away are like what you describe: maybe one outage in the past 10 years.
I think a pattern like that is reasonable for a 6-day cert:
- renew every 2 days, and have a "4 day debugging window" - renew every 1 day, and have a "5 day debugging window"
Monitoring options: https://letsencrypt.org/docs/monitoring-options/
This makes me wonder if the scripts I published at https://heyoncall.com/blog/barebone-scripts-to-check-ssl-cer... should have the expiry thresholds defined in units of hours, instead of integer days?
This is probably the same kind of "one new row per month" assumption that many data pipelines with any sort of primary date/time column make!
I'd bet that a good chunk of the apparently sudden demand spike could be last month's Microsoft Windows 10 end-of-support finally happening, pushing companies and individuals to replace many years worth of older laptops and desktops all at once.
It's comparing the likelihood of an update introducing a new vulnerability to the likelihood of it fixing a vulnerability.
While the article frames this problem in terms of deliberate, intentional supply chain attacks, I'm sure the majority of bugs and vulnerabilities were never supply chain attacks: they were just ordinary bugs introduced unintentionally in the normal course of software development.
On the unintentional bug/vulnerability side, I think there's a similar argument to be made. Maybe even SemVer can help as a heuristic: a patch version increment is likely safer (less likely to introduce new bugs/regressions/vulnerabilities) than a minor version increment, so a patch version increment could have a shorter cooldown.
If I'm currently running version 2.3.4, and there's a new release 2.4.0, then (unless there's a feature or bugfix I need ASAP), I'm probably better off waiting N days, or until 2.4.1 comes out and fixes the new bugs introduced by 2.4.0!
[1] https://totalrealreturns.com/
[2] https://status.heyoncall.com/svg/uptime/zCFGfCmjJN6XBX0pACYY...
I’ve heard good things about “Practical Electronics for Inventors” but haven’t gone through it myself.
Diode half-wave rectifier https://www.circuitlab.com/editor/4da864/
Diode full-wave (bridge) rectifier https://www.circuitlab.com/editor/f6ex5x/
Diode turn-off time https://www.circuitlab.com/editor/fwr26m/
LED with resistor biasing https://www.circuitlab.com/editor/z79rqm/
Zener diode voltage reference https://www.circuitlab.com/editor/7f3ndq/
Charge Pump Voltage Doubler https://www.circuitlab.com/editor/24t6h3ypc4e5/
Diode Cascade Voltage Multiplier https://www.circuitlab.com/editor/mh9d8k/
(note: I wrote the simulation engine)
For work, https://heyoncall.com/ as the best tool for on-call alerting, website monitoring, cron job monitoring, especially for small teams and solo founders.
I guess they both fall under the category of "how do you build reliable systems out of unreliable distributed components" :)
https://totalrealreturns.com/s/GLD
(Not quite the same due to the compounding 0.40%/year expense ratio of the ETF, but probably close enough for this conversation.)
Why not encode that TXT record value into the CA-signed certificate metadata? And then at runtime, when a browser requests the page, the browser can verify the TXT record as well, and cache that result for an hour or whatever you like?
Or another set of TXT records for revocation, TXT _acme-challenge-revoked.<YOUR_DOMAIN> etc?
It's not perfect, DNS is not at all secure / relatively easy to spoof for a single client on your LAN, I know that. But realistically, if someone has control of your DNS, they can just issue themselves a legit certificate anyway.
Software didn't have that sort of "ticking time bomb" element before, I think?
I think I understand why it's necessary: we have a single, globally shared public namespace of domain names, which we accept will turn over their ownership over the long run, just like real estate changes hands. So we need expiration dates to invalidate "stale" records.
We've already switched over everything to Let's Encrypt. But I don't think anyone should be under the delusion that automation / ACME is failproof:
https://github.com/certbot/certbot/issues?q=is%3Aissue%20ren...
https://github.com/cert-manager/cert-manager/issues?q=is%3Ai...
https://github.com/caddyserver/caddy/issues?q=is%3Aissue%20A...
(These are generally not issues with the software per se, but misconfiguration, third-party DNS API weirdness, IPv6, rate limits, or other weird edge cases.)
Anyway, a gentle reminder that Let's Encrypt suggests monitoring your SSL certificates may be "helpful": https://letsencrypt.org/docs/monitoring-options/ (Full disclosure: I wrote the most recent addition to that list, with the "self-hosted scripts".)
ClickHouse defaults to: "fsync_after_insert = false".
A more fair comparison might at least do "SET synchronous_commit TO OFF" on the PostgreSQL side. (But for this UPDATE benchmark, I think the results would still largely be the same.)