compumike

Oof, you're right, that's rough that it's so soon after they discontinued their email service!

I wrote this blog post a few weeks ago: "Minimal, cron-ready scripts in Bash, Python, Ruby, Node.js (JavaScript), Go, and Powershell to check when your website's SSL certificate expires." https://heiioncall.com/blog/barebone-scripts-to-check-ssl-ce... which may be helpful if you want to roll your own.

(Disclosure: at Heii On-Call we also offer free SSL certificate expiration monitoring, among other things.)

I appreciate the "How to monitor the monitor?" section. Always need a meta-monitor! :)

Hope you might give us a try at https://heiioncall.com/ and let me know if that fits. (Disclosure: our team is building and operating it as a simple monitoring, alerting, on-call rotations solution.) We have the cron job heartbeats, HTTP healthchecks, SSL certificate expiration, etc etc all in one simple package. With mobile app alerts (critical and non-critical), business hours rules, etc. And a free tier for homelabbers / solo projects / etc. :)

Edit: since you mentioned silencing things in your post, we also have very flexible "silence" buttons, which can set silence at various levels of the hierarchy, and can do so with a predefined timer. So if you know you want things to be silenced because you're fixing them, you can click one button and silence that trigger or group of triggers for 24 hours -- and it'll automatically unsilence at that time -- so you don't have to remember to manually manage silence/unsilence status!

Blocks me too. (Infinite loop "Verifying...") Firefox on macOS. No VPN.

Interesting! It could be a confounder as well. For example, maybe people who live in dense (loud) urban dwellings also tend to have more day-to-day life stress generally, leading to reduced sleep quality. (Of course, reduced sleep quality could also be a source of more life stress...)

Would your userbase be up for a little experiment: adding white/pink/rainfall/etc noise at various volumes? I bet you'd see an inverted U shaped curve, with sleep quality increasing at relatively low volume levels, and then hitting some maximum and decreasing when it gets too loud! (Agree with other comments about looking at intra-night variance in noise level.)

> This fact is so terrifying that a lot of people try to remain in denial about it.

I remember talking as college seniors about how: for two decades there’s always been some near-future end-of-school-term date that we’re all marching together toward, and isn’t it so strange that the whole cycle is about to disappear?

    if next_end_date.nil?
      # ?!? FIXME
    end

Some rhythm of starting and efforting and finishing and relaxing, before starting the next cycle.

Of course it’s somewhat possible to join new calendar cycles. A two-week engineering sprint. Even YC’s batch concept recreates this for a few months, to great effect! But not the same.

But for the most part, when the calendar rhythm is no longer the source of medium-term stability and inspiration and motivation, I think this makes a good point that the people you surround yourself with can be. If not, what else?

Thank you, Jessica.

... but a small PR might warn you when your renewal automation breaks, before the cert actually expires: https://heiioncall.com/blog/barebone-scripts-to-check-ssl-ce... (a small blog post I wrote with example scripts for checking SSL cert expiry from Bash, Python, Ruby, more) ;)

I'd add that, in terms of tactical implementation, production tests can be implemented at least two different ways:

(1) You set up an outside service to send an HTTP response (or run a headless browser session) every minute, and your endpoint runs some internal assertions that everything looks good, and returns 200 on success.

(2) You set up a scheduled job to run every minute internal to your service. This job does some internal assertions that everything looks good, and sends a heartbeat to an outside service on success.

For #2: most apps of any complexity will already have some system for background and scheduled jobs, so #2 can make a lot of sense. It can also serve as a production assertion that your background job system (Sidekiq, Celery, Resque, crond, systemd, etc) is healthy and running! But it doesn't test the HTTP side of your stack at all.

For #1: it has the advantage that you also get to assert that all the layers between your user and your application are up and running: DNS, load balancers, SSL certificates, etc. But this means that on failure, it may be less immediately clear whether the failure is internal to your application, or somewhere else in the stack.

My personal take has been to lean toward #2 more heavily (lots of individual check jobs that run once per minute inside Sidekiq, and then check-in on success), but with a little bit of #1 sprinkled in as well (some lightweight health-check endpoints, others that do more intense checks on various parts of the system, a few that monitor various redirects like www->root domain or http->https). And for our team we implement both #1 and #2 with Heii On-Call https://heiioncall.com/ : for #2, sending heartbeats from the cron-style check jobs to the "Inbound Liveness" triggers, and for #1, implementing a bunch of "Outbound Probe" HTTP uptime checks with various assertions on the response headers etc.

And this production monitoring is all in addition to a ton of rspec and capybara tests that run in CI before a build gets deployed. In terms of effort or lines of code, it's probably:

    90% rspec and capybara tests that run on CI (not production tests)
    9% various SystemXyzCheckJob tests that run every minute in production and send a heartbeat
    1% various health check endpoints with different assertions that are hit externally in production

And absolutely agree about requiring multiple consecutive failures before an alarm! Whenever I'm woken up by a false positive, my default timeout (i.e. # of consecutive failures required) gets a little bit higher :)

Using it extensively in production at https://heiioncall.com/ . It’s a joy to work with. Fast to write AND fast to run. :)

We wrote https://github.com/compumike/idle-gc to run GC when the long-running process is otherwise mostly idle. It’s been fine for us.

Is it just the historical legacy of international trade that tariffs apply only to physical goods, but not to something like software-as-a-service subscriptions, digital media, or intellectual property?

As the latter categories are more representative of US exports, it would surely be an interesting escalation if other countries were to start including them in their “retaliatory” tariffs.

I keep having multi-hour power outages a few times a year, so I decided to build a battery system a few months ago. I was happy with it during a recent four hour high-wind winter outage, powering my 120V furnace, fridge, desktop PC, some lights and chargers. Here's what I ended up with:

- Bluetti Elite 200 V2 portable power station: $999, 2073 Wh LiFePO4, 2600 W inverter output (120 V only), 1000 W DC input at 60 V max, Bluetooth & Wi-Fi connectivity to app, 53 lbs [1]

- LiTime 51.2V 100 Ah battery: $999 (includes charger), 5120 Wh LiFePO4, local Bluetooth connectivity to app, 82 lbs [2]

TOTAL: $1,998 for 7.2 kWh = $278 / kWh (before 30% tax credit). Plus tax and a few hundred more for the random bits and bobs.

Here's a video that talks about using the big bulk LiFePO4 battery to extend the power station: [3] This setup can supply 2600 W for about an hour (limited by the Bluetti's inverter and internal battery capacity), or 1000 W average for 7 hours (limited by the Bluetti's DC input). Limited to 120 V loads only, of course.

For me, this made the recent outage smooth sailing, running a few hundred watts average load for furnace / fridge / small conveniences. A friend nearby had pipes freeze and burst ($$$ repair and highly inconvenient) during a 20-hour winter power outage. It was nice to have a longer buffer to get a good night's sleep and know that everything would be fine even if utility power didn't come back quickly!

I think the original poster here is latching on to a basic issue that the consumer-grade UPS companies (APC, CyberPower) are being slow to adopt LiFePO4 batteries. At the same time, the "portable power station" companies (Bluetti, Anker Solix, Ecoflow, etc) have been recklessly marketing that they have UPS functionality, without delivering on that until recently. Be careful if you buy one: many of these portable power stations will not turn back on their AC output unattended if they fully discharge before the utility power comes back on! Just recently, they are starting to roll this out on some models: Bluetti calls this option "System Switch Recovery", and Ecoflow calls it "AC Always On" -- and neither are enabled by default!

And I 100% recognize that I'm in the tiny minority willing to combine these products together to get a lower overall $ / kWh. But maybe you are too :)

[1] https://www.amazon.com/dp/B0DCJV9LTB [2] https://www.amazon.com/dp/B0DG9BDB36 [3] https://www.youtube.com/watch?v=9ODIF2EfsUA

Temp file issue is debunked here: https://news.ycombinator.com/item?id=41684848

Dir module works fine, though maybe the docs could be improved. https://crystal-lang.org/api/1.13.3/Dir.html . The stdlib code is also highly readable: https://github.com/crystal-lang/crystal/blob/d14d04562/src/d... shows that #each_child just calls #read and yields it to the block, so I don't think this is really a wart :)

It depends a lot on compiler flags, especially "--release". See https://crystal-lang.org/reference/1.13/man/crystal/index.ht... : "Compiling without release mode is much faster and the resulting binaries still offer pretty good performance. Building in release mode should be used for production-ready executables and when performing benchmarks. For simple development builds, there is usually no reason to do so."

For our API server https://api.heiioncall.com/ , on my Ryzen 5950X, with a warm CRYSTAL_CACHE_DIR ("where Crystal caches partial compilation results for faster subsequent builds"), making a one-line change:

Without --release:

    6.11user 2.37system 0:08.19elapsed 103%CPU (0avgtext+0avgdata 590844maxresident)k

With --release:

    76.54user 5.49system 1:25.16elapsed 96%CPU (0avgtext+0avgdata 696876maxresident)k

So let's call it about 9 seconds while developing, or 90 seconds for a CI build with full release optimizations.

> It would be a totally different thing if it could run Rails, which is one of the things I do in my job.

At https://heiioncall.com/ we combine the two and use Rails for the human-facing parts, and Crystal for the machine-facing parts (inbound API server calls, and outbound HTTP probes). For us, the number of machine-facing requests per second is so many orders of magnitude higher than human-facing requests that Crystal's performance and lower footprint is valuable there. While on the other hand, the conveniences of Ruby on Rails are still great for a conventional human-facing web and mobile application.

Conceptually, I think the correct time to do garbage collection is when your web server process is idle.

My Crystal implementation of idle-time garbage collection is here: https://github.com/compumike/idle-gc though please note that its idle detection mechanism only works for single-threaded Crystal programs.

An analogy is to imagine a single employee (thread) operating a convenience store. If there are customers waiting in the checkout line (latency-sensitive requests), the employee should priorities serving the customers ASAP! But once the line is empty (thread is idle), that might be a good time to start sweeping.

Right now, with automatic garbage collection, the employee only decides to start sweeping the entire store while in the middle of serving a customer! (Because that's when mallocs are happening, which may trigger automatic GC.) Pretty ridiculous!

With idle-time GC, the sweeping happens entirely or mostly while there are no customers waiting. This may not show latency improvements in an artificial benchmark where the system is running flat-out with a full request queue, but in the real world, it changes GC from something that happens 100% of the time in the middle of a request is being served (because that's when mallocs happen and trigger automatic GC), to something that only rarely or never happens while a request is being served.

Even better would be to combine idle-time GC with incremental GC, so that the employee could put down the broom when a new customer arrives without finishing sweeping the entire store. :)

See also "Idle Time Garbage Collection Scheduling" in Google Chrome (2016): PDF at https://static.googleusercontent.com/media/research.google.c...

We use Crystal extensively at https://heiioncall.com/ for the high-performance stuff, and it’s terrific. :)

As the author notes: the typing system is very pleasant. And the standard library is great!

Generics work fine.

The testing story is very good (almost as good as rspec).

Have some experience with the Threads but won’t comment extensively as that’s about to get some big changes in the next minor release or two.

Overall: highly productive for high-performance backend API servers and background task daemons. Way more fast and pleasant to write than Rust or Go (IMHO).

Does anyone on the inside know, when people talk about login services being down, what’s the actual technical bottleneck?

I’d hope we’re not talking about them simply not having enough CPU to bcrypt_compare() everyone’s passwords as they try to log in…

In-house service-wide rate limits that are too low?

Two-factor auth steps? (Third party providers may be rate limiting?)

Anti-fraud things? GeoIP lookups? Etc

Writing out logs of logins? Some login audit DB?

(Or is it not actually some login-specific service that is down?)

> That's the default health check associated with our load balancer, so it gets hit periodically (built-in cron job).

Be aware that you’re still leaving quite a bit of surface area unmonitored: any sort of issue between your clients and the load balancer could break and not fire an alert. DNS configuration, firewall issue, SSL certificates, network outage, etc etc.

If you’re really trying to assert that some HTTP resource is publicly reachable, it’s still a good idea to have some external testing service periodically hitting your endpoint, running some assertions on the response, and alerting you if it’s failing for more than X minutes. (We do this; see my other reply.)

Hope that helps!

We do this at Heii On-Call: https://heiioncall.com/

Outbound Probes to hit your exposed HTTP services, or Inbound Liveness for your own cron jobs etc to check in.

Hot Tub Wi-Fi remote controller from https://github.com/visualapproach/WiFi-remote-for-Bestway-La...

I'll note that many websites do NOT support this:

https://duckduckgo.com./ - blank page

https://www.google.com./ - 301 redirect to non-trailing-dot

https://www.amazon.com./ - works

But even "works" means that the page makes lots of requests to other subdomains etc which do not have the trailing dot, so are still vulnerable for DNS hijack if you have search domains set on your system. So while using domains with a trailing dot is a strongly good idea for calling APIs, etc. it is not a practical solution for the end user with a web browser.

This is a good reason to use domain names with a trailing dot, right? Such as https://news.ycombinator.com./ to specify a FQDN, instead of https://news.ycombinator.com/ which is not fully qualified. The trailing dot keeps the DNS resolver from trying system domain suffixes.

Quite good, low latency, felt very natural! I did eventually get a secret code. :) Nice demo to get people trying it in a playful way.

Beautiful, thank you. Consider posting this on the Crystal community forum https://forum.crystal-lang.org/ as well!

What about something like https://heiioncall.com/status (disclosure: helped build it) which gives a real-time view into what our monitoring & alerting system sees, both from various HTTP endpoint checks, and cronjob checkins?

But I don’t think most orgs would want a public version of this (too much transparency), which is why we haven’t built that.

The demo video looks great. Any chance of this coming to non-macOS platforms?

Yes. See https://adamswebsearch2.nrc.gov/webSearch2/main.jsp?Accessio... "Rulemaking: Regulatory Framework for Fusion Energy Systems, NRC Public Meeting, July 12, 2023" and especially the graph on page 27, "Relative radiotoxicity" vs "Time after shutdown (years)"

Also see https://www.nrc.gov/docs/ML2325/ML23258A145.pdf "Fusion Systems Rulemaking: United States Nuclear Regulatory Commission Preliminary Proposed Rule Language". The NRC are already proposing to regulate fusion reactors much more like particle accelerators (which can produce medical radioisotopes, for example), and less like fission reactors.

Not the OP, but when we built this functionality for Heii On-Call https://heiioncall.com/outbound_prober we made sure (1) the User-Agent string points you to this page, (2) the User-Agent contains a unique string which we can decode and track back to the unwanted HTTP probe configuration on our end, and (3) also obeys /robots.txt. Trying to be a friendly robot :)

Ah yeah, these are no ordinary phone notifications. :)

Heii On-Call will keep alerting you with these “Critical Alerts” until you’ve manually acknowledged. (Or until it escalates to a teammate and they acknowledge…)

And at least on my phone they sound nothing like normal phone notifications, which I personally always have on vibrate and/or DND anyway.

Give it a try and I think you’ll like it.

If we eliminated the accounting fiction called "depreciation", and instead simply had all expenses deductible in the year they were incurred or paid on a cash basis (with net losses carrying-over to future years), this wouldn't have happened.