gregsadetsky

Related discussion: https://news.ycombinator.com/item?id=45984143

I'm very grateful! Thanks a lot

Yeah, great point.

I was considering working on the ability to dynamically generate eval questions whose solutions would all involve problem solving (and a known, definitive answer). I guess that this would be more valuable than publishing a fixed number of problems with known solutions. (and I get your point that in the end it might not matter because it's still about problem solving, not just rote memorization)

Interesting, yeah! Just tried "summarize this story and list the important figures from it" with Gemini 2.5 Pro and 3 and they both listed 10 names each, but without including Yudkowsky.

Asking the follow up "what are ALL the individuals mentioned in the story" results in both models listing ~40 names and both of those lists include Yudkowsky.

It does! I redacted them, but yes. This was a 3-person call.

You're correct of course - LLMs may get better at any task of course, but I meant that publishing the evals might (optimistically speaking) help LLMs get better at the task. If the eval was actually picked up / used in the training loop, of course.

I just tried "analyze this audio file recording of a meeting and notes along with a transcript labeling all the speakers" (using the language from the parent's comment) and indeed Gemini 3 was significantly better than 2.5 Pro.

3 created a great "Executive Summary", identified the speakers' names, and then gave me a second by second transcript:

    [00:00] Greg: Hello.
    [00:01] X: You great?
    [00:02] Greg: Hi.
    [00:03] X: I'm X.
    [00:04] Y: I'm Y.
    ...

Super impressive!

I asked a semi related question in a different thread [0] -- is the basic idea behind your benchmark that you specifically keep it secret to use it as an "actually real" test that was definitely withheld from training new LLMs?

I've been thinking about making/publishing a new eval - if it's not public, presumably LLMs would never get better at them. But is your fear that generally speaking, LLMs tend to (I don't want to say cheat but) overfit on known problems, but then do (generally speaking) poorly on anything they haven't seen?

Thanks

[0] https://news.ycombinator.com/item?id=45968665

I'm new/uninformed in this world, but I have an idea for an eval that I think has not been tried yet.

Can anyone direct me towards how to ... make one? At the most fundamental level, is it about having test questions with known, golden (verified, valid) answers, and asking different LLM models to find the answer, and comparing scores (how many were found to be correct)?

What are "obvious" things that are important to get right - temperature set to 0? At least ~10 or 20 attempts at the same problem for each llm? What are non-obvious gotchas?

Finally, any known/commonly used frameworks to do this, or any tooling that can call different LLMs would be enough?

Thanks!

Mozzi (for Arduino) is also great!

https://sensorium.github.io/Mozzi/

Just to make sure I'm not misunderstanding:

The relay click sound you're hearing should only happen when a relay turns on, and when it turns off. There is no sound when a button is pressed - you can check this by loading an empty simulator and pressing any of the buttons at the bottom (the simulator won't make any sound):

https://minivac.greg.technology/simulator/

((Once/if I get to see a Minivac in real life, I may learn that buttons do make sounds..))

When a relay turns on and off, a sound is played. In the following circuit, check this by pressing and releasing the button 1:

https://minivac.greg.technology/simulator/#wires=1%2B%2F1C%2...

Amazing! I don’t think I’ve heard of anyone owning many..!

1) Are any of them for sale? :)

2) Would you be open to recording a clean sound of one of its relays turning on and off? And helping me figure out some specifics about the motor wheel (its speed and whether it experiences any speed up)

Thank you! My email is:

hi at greg.technology

Thanks!

The Enigma-cracking Bombe (on display at the extraordinary Bletchley Museum of Computing) also used relays

https://www.tnmoc.org/bh-10-bombe-description

And the Z3 as well!

https://en.wikipedia.org/wiki/Z3_(computer)

More…?! :)

I totally feel you - the one reservation is that I’m using (gasp) not the original Minivac Relay sound. I need to go to a Computer Museum that will let me record one to offer a more authentic simulated experience.

So tldr - let me get a clean real sound and then we can come back to this question :)

Ha, I really get the sentiment! I do recommend perusing the Book 1 manual [0], towards the end there are a number of small problems (on page 30). They're worth tinkering with :-)

[0] https://minivac.greg.technology/manuals/1961-minivac601-book...

The Rotary Switch - which is both input and output in the real Minivac - is absolutely wild yeah.

One of the crucial things I didn't get to yet is making that wheel controllable - some circuits in the manuals do require you to turn it. That's high in the todo list (with mobile support).

I also really want to share all of the videos I've been collecting of similar "wheels" - in (modern-ish) elevators [0] and light shows [1]... And also pinball machines! [2]

Re: RadioShack boards, I did have the "300 in 1" kit growing up, so yes... it's full circle for me. :-)

[0] https://www.instagram.com/reel/C3dk5GkJni6/ and https://www.instagram.com/reel/CzuHXKRpuN-/

[1] https://www.instagram.com/reel/C6qqCdhJbM2/

[2] https://www.youtube.com/watch?v=ue-1JoJQaEgm

Thank you so much everyone, this is something I've worked on for a few years on and off -- I posted about it here in a Show HN a few hours ago [0]

The biggest unlock was finding Willy McAllister's excellent Circuit Sandbox [1], which provides the Minivac Simulator's underlying electrical math. I tried so many approaches to simulate electricity (a doomed DIY approach, Falstad, Spice...) but Circuit Sandbox's DC analysis did the job perfectly.

Ping me for questions, and would love to read your feedback!

[0] https://news.ycombinator.com/item?id=45945762

[1] https://spinningnumbers.org/a/circuit-sandbox.html

Discussion happening over at https://news.ycombinator.com/item?id=45950396 !

Thank you, I really appreciate it!

Tangential question: does anyone know if there's a ~similar device to replace/upgrade a Toshiba T1100's floppy drive?

A friend found a T1100 for me and I'm just trying to think of the best way to boot it. Alternatively... I could get a USB floppy disk drive and a fresh floppy, and write old-school DOS to it, at least to get started?

Thanks!

There seems to be https://notbyai.fyi/ and https://no-ai-icon.com/ ..!

For sure! Just for the record, it's not my project. But your point stands - if I were to run an instance of it anywhere where the Japanese wordplay wasn't obvious (and "black market" wouldn't have this effect in English, obviously!) - then I fully agree.

"Internet Flea Market" sounds pretty great actually! :-)

Cheers

It's worth watching the short video, they specifically address it - link with timestamp:

https://youtu.be/mjWJsE7B1cs?si=vPyBiNeNIvKJ8nKV&t=137

I submitted this [0] story a few weeks ago, which led to some discussion and then being flagged since (I think) people were unsure of how verifiable the stats around COBOL were (the submitted page had more than a tinge of self-promotional language).

I was curious to ask you, as domain experts, if you could talk more to the "70% of the Fortune 500 still run on mainframes" stat you mentioned.

Where do these numbers come from? And also, does it mean that those 70% of Fortune 500s literally run/maintain 1k-1M+ LoC of COBOL? Or do these companies depend on a few downstream specialized providers (financial, aviation/logistics, etc.) which do rely on COBOL?

Like, is it COBOL all the way down, or is everything built in different ways, but basically on top of 3 companies, and those 3 companies are mostly doing COBOL?

Thanks!

[0] https://news.ycombinator.com/item?id=45644205

Also see https://www.youtube.com/watch?v=B6orV_Bldsk for a demonstration

Hey Chris, would love to chat about this - could you email me? Cheers

Redirects going anywhere is super flexible, but is also the unfortunate business model of so many "free qr code generator" sites which end up taking your destination link hostage...! (this just reminds me of that, obv the parent post isn't doing that)

My friend's partner once printed a qr code like that and then had to pay a monthly fee to keep the qr code working. Pure predator behavior.

Similar article on this exact topic, with a bit more detail on the QR encoding side, from Terence Eden:

https://shkspr.mobi/blog/2025/02/why-are-qr-codes-with-capit...

Past discussion: https://news.ycombinator.com/item?id=43149077

(self promo-ish) The above blog article was one of the pieces shown at the QR Show that I helped co-organize earlier this year at the Recurse Center. I'm definitely tempted to have another one in 2026!

The whole lists of pieces is here: https://qrshow.nyc/retrospective.html

"juice" (in terms of game making) will always remind me of this amazing, classic talk - https://www.youtube.com/watch?v=Fy0aCDmgnxg

as the sibling comment notes, the Canadian version also has it:

https://www.jeancoutu.com/en/shop/categories/personal-care/o...

https://www.walmart.ca/en/ip/Sensodyne-Repair-Protect-Sensit...