Xcelerate

Weird thought: someone born in the 1800s was (most likely) alive when the first transformer model ran.

Emma Morano died April 15, 2017, the NIPS submission deadline for "Attention Is All You Need" was May 19, and a Wired article indicates they were testing models for quite a few weeks before then.

I wonder how much of the programming language problem is due to churn of the user base. Looking over many comments in this thread, I see “Oh, back when I did HPC...” I used Titan for my own work back in 2012. But after my PhD, I never touched HPC again. So the people writing the code use what’s there but don’t stay long enough to help incentivize new or better languages. Now on the hardware side (e.g., design of interconnects), that more commonly seems to be a full career.

The other issue is that to really get the value out of these machines, you sort of have to tailor your code to the machine itself to some degree. The DOE likes to fund projects that really show off the unique capabilities of supercomputers, and if your project could in principle be done on the cloud or a university cluster, it’s likely to be rejected at the proposal stage. So it’s sort of “all or nothing” in the sense that many codebases for HPC are one-off or even have machine-specific adaptations (e.g., see LAMMPS). No new general purpose language would really make this easier.

Is there not some concept that utilizes cryptography in a way such that information about people is accessible, but if it's accessed, then the access request is added to a ledger (akin to blockchain) such that who made the access, when, and about whom becomes provably public knowledge?

> they all have an obvious and immediate majesty to them.

"Grandeur" is not the only criteria for nice national parks. I'm from the east coast, and while all of the breathtaking views in California were amazing, after a few years of living there I began to get frustrated that I couldn't find anywhere "cozy" to visit during the weekends. Some locations along the Russian River probably came the closest, but the jagged rocks and coniferous trees still didn't manifest the sort of "warm and snug" feeling one gets while river tubing along a mountain river in the Blue Ridge mountains. Temperature deciduous rainforests are actually quite rare across the planet, and particularly when the leaves change colors, it's a sight to behold.

Would be interesting to think about what works are currently out there, published, yet will not be recognized as great intellectual achievements until much later after the fact for some reason.

I’ve heard that bitterness affects children more intensely. So I wonder how much of it is an acquired taste vs bitterness just becoming “milder” over time.

It depends what I'm working on. If it's a bunch of interdependent systems that involve a large amount of data, a giant monitor is better. If the giant monitor is being used to make visible more application surfaces (Slack, email, VS Code, etc.), it makes focus worse.

The biggest improvement I've found for my focus is to force myself to close any open tabs/windows that are not absolutely necessary roughly every two hours. I used to be one of those people with 800 tabs open in the browser and 20 application windows spread across 8 desktop spaces. Was a concentration mess. Requiring myself to "clean up" periodically has really helped.

22AWG Cat6A is actually what I used (cheap it was not however).

I set up my own home network with a Vertiv Liebert Li-ion UPS a few years ago and was thinking about how inefficient the whole process is regarding power. The current goes from AC to DC back to AC back to DC. Straight from the UPS as DC would work much better, and as I was teaching myself more about networking equipment, I was surprised to learn that most of it isn't DC input by default (i.e., each piece of equipment tends to come with built-in AC-DC conversion).

Then I started routing ethernet with PoE throughout my house and observed that other than a few large appliances, the majority of powered devices in a typical home in 2026 could be supplied via PoE DC current as well! Lighting, laptops, small/medium televisions. The current PoE spec allows up to 100 W, which covers like 80% of the powered devices in most homes. I think it would make more sense to have fewer AC outlets around the modern house and many more terminals for PoE instead (maybe with a more robust connector than RJ45). I wonder what sort of energy efficiency improvements this would yield. No more power bricks all over the place either.

I’ve always wanted kids, ever since I was a kid myself, but I was never really sure what it would be like to be a parent.

Turns out it’s quite strange, because my kids bring me more joy than anything else. I’ll sit there for hours watching them play. You may think “that’s not strange—tons of parents say that”, but for my sort of personality, it’s very strange. I’ve always thought of myself as sort of overly analytical, detached, ambitious, and a bit obsessive. Not the sort of touchy-feely person who chases a two year old around with a smile on my face and likes watching videos of cute babies. Yet here I am. I enjoy it so much I’ve even tried to figure out if there’s a way I can take a sabbatical from work to spend the last two years with my youngest at home before he goes off to school (seems unlikely given how questions about a random two year gap on my resume might affect my long-term career).

It’s funny that as a kid I always wanted to work at a tech company for the interesting tech, but now as an adult my favorite thing about it has been the 4 months of parental leave I was able to have with each newborn.

I've been using ChatGPT to teach myself all sorts of interesting fields of mathematics that I've wanted to learn but never had the time previously. I use the Pro version to pull up as many actual literature references as I can.

I don't use it at all to program despite that being my day job for exactly the reason you mentioned. I know I'll totally forget how to program. During a tight crunch period, I might use it as a quick API reference, but certainly not to generate any code. (Absolutely not saying it's not useful for this purpose—I just know myself well enough to know how this is going to go haha)

I’ve always thought the issue was a bit less “Find the interesting research problem” and more “Find the resources, network, or skills that get you into the position of being able to work on the interesting research problem.”

If you asked a bunch of researchers working on the “boring” stuff to predict what the hot papers of the year will be about, do we really think they’ll be that far off base? I’m not talking about groundbreaking or truly novel ideas that seem to come out of nowhere, but rather the high impact research that’s more typical of a field.

Even in big tech companies, it’s quite obvious what the interesting stuff to work on is. But there are limited spots and many more people who want those spots than are available.

“You’re not [X]—you’re [Y]” is the one that drives me nuts. [X] is typically some negative characterization that, without RLHF, the model would likely just state directly. I get enough politics/subtext from humans. I’d rather the LLM just call it straight.

It’s kind of intriguing that predicting the future state of any quantum system becomes almost trivial—assuming you can diagonalize the Hamiltonian. But good luck with that in general. (In other words, a “simple” reference frame always exists via unitary conjugation, but finding it is very difficult.)

“Live next to your relatives because if they get in a car accident and you live across the country you won’t be there to tell them goodbye.”

^Another one I’ve never understood. Like geez, hopefully my daughter doesn’t give up her life dreams just based on the possibility I might be in a freak accident one day...

Interesting. I agree with most items on this page as overrated, but with an L5-S1 disc herniation, the Aeron is about the only chair I can sit in for an extended period of time without hurting. Then again, I haven't tried dozens of office chairs, but at least for me it was worth the purchase cost.

Yep. I have an Ecobee currently and have had a Nest previously. Am totally perplexed why people like these things. Just opened the Ecobee app and literally at the top is an ad saying "The Holiday sale is here. Shop now." Irritates me to no end.

> If spacetime had a discrete character at scales like the inverse of the universe scale we would see dispersion of light as it traveled cosmological distances and we do not observe this. It is technically possible that the discreteness scale is much, much smaller than the inverse universe scale, of course, but at this point it seems pointless to me to entertain discrete models

A computational universe does not strictly imply discrete spacetime. You can most certainly still have a continuous universe—at least from the perspective of the beings that inhabit it. By way of analogy, consider the fact that ZFC proves the existence of uncomputable real numbers yet itself has a countable model (presuming it is consistent).

As a non-mathematician, I’m confused why so many people think the conjecture (whether true or false) is provable within PA. To me, it seems like something that would be very nicely just right outside the boundary of PA’s capability, sort of like how proving all Goodstein sequences terminate requires transfinite induction up to ε_0. Add that to the fact that the Collatz Conjecture seems to fall in the same “category” of problem as the Turing machines that the Busy Beaver project is having a hard time proving non-halting behavior of, and the heuristic arguments all seem to point to: Collatz is independent of PA.

But I’m interested in hearing the counterarguments that Collatz likely is provable within PA and why this would be the case.

> Eliminate redundant matrix operations (like two transposes next to each other)

In 2016, I was trying to construct orthogonal irreducible matrix representations of various groups (“irreps”). The problem was that most of the papers describing how to construct these matrices used a recursive approach that depended on having already constructed the matrix elements of a lower dimensional irrep. Thus the irrep dimension n became quite an annoying parameter, and function calls were very slow because you had to construct the irrep for each new group element from the ground up on every single call.

I ended up using Julia’s @generated functions to dynamically create new versions of the matrix construction code for each distinct value of n for each type of group. So essentially it would generate “unrolled” code on the fly and then use LLVM to compile that a single time, after which all successive calls for a specific group and irrep dimension were extremely fast. Was really quite cool. The only downside was that you couldn’t generate very high dimensional irreps because LLVM would begin to struggle with the sheer volume of code it needed to compile, but for my project at the time that wasn’t much of a concern.

Hmm... I agree with parts of this and disagree with other parts. In my experience "cross-functional collaboration" splits into two distinct components: leadership and information. Anecdotal, but when leadership is split into too many people at the same level who are each in charge of a domain (that requires heavy interaction with the other domains), nothing gets accomplished—analysis paralysis and politics takes over. You absolutely need one specific person as the final decision maker. They should carefully consider all input from various sources and then make a final decision in a timely fashion. If it turns out to be the wrong path, that's fine, just reverse course quickly as well.

On the other hand, information silos are absolutely horrible. The most effective companies I've worked at have always had tons of information freely available to all employees. Unless there are privacy, cybersecurity, antitrust, or similar risks involved, every employee should have access to all information across all teams. It should be easily searchable as well. There are certainly exceptions—Apple seems to function well despite all the secrecy. But most companies aren't Apple, and I don't think it's generally a good strategy.

Haha, I like to joke that we were on track for the singularity in 2024, but it stalled because the research time gap between "profitable" and "recursive self-improvement" was just a bit too long that we're now stranded on the transformer model for the next two decades until every last cent has been extracted from it.

From the abstract:

> This aligns with number theory conjectures suggesting that at higher orders of magnitude we should see diminishing noise in prime number distributions, with averages (density, AP equidistribution) coming to dominate, while local randomness regularises after scaling by log x. Taken together, these findings point toward an interesting possibility: that machine learning can serve as a new experimental instrument for number theory.

n*log(n) spacing with "local randomness" seems like such a common occurrence that perhaps it should be abstracted into its own term (or maybe it already is?) I believe the description lengths of the minimal programs computing BB(n) (via a Turing machine encoding) follow this pattern as well.

> a computation is a process that maps symbols (or strings of symbols) to other symbols, obeying certain simple rules[1]

There are quite a number of people who believe this is the universe. Namely, that the universe is the manifestation of all rule sets on all inputs at all points in time. How you extract quantum mechanics out of that... not so sure

Would love to read a HN-tailored blog post of your work or an overview of the binary lambda calculus if you ever have the time btw

There are a few concepts at play here. First you have to consider what can be proven given a particular theory of mathematics (presumably a consistent, recursively axiomatizable one). For any such theory, there is some finite N for which that theory cannot prove the exact value of BB(N). So with "infinite time", one could (in principle) enumerate all proofs and confirm successive Busy Beaver values only up to the point where the theory runs out of power. This is the Busy Beaver version of Gödel/Chaitin incompleteness. For BB(5), Peano Arithmetic suffices and RCA₀ likely does as well. Where do more powerful theories come from? That's a bit of a mystery, although there's certainly plenty of research on that (see Feferman's and Friedman's work).

Second, you have to consider what's feasible in finite time. You can enumerate machines and also enumerate proofs, but any concrete strategy has limits. In the case of BB(5), the authors did not use naive brute force. They exhaustively enumerated the 5-state machines (after symmetry reductions), applied a collection of certified deciders to prove halting/non-halting behavior for almost all of them, and then provided manual proofs (also formalized) for some holdout machines.

Homemade cake mixes rarely win blind taste tests against box mix. I baked two cakes with and without glycerol monostearate—it really does make a difference.

Everyone likes to debate the philosophy of whether the reals are “real”, but for me there is a much more practical question at hand: does the existence of something within a mathematical theory (i.e., derivability of a “∃ [...]” sentence) reflect back on our ability to predict the result of symbolic manipulations of arbitrary finite strings according to an arbitrary finite rule set over an arbitrary finite period of time?

For AC and CH, the answer is provably “no” as these axioms have been shown to say nothing about the behavior of halting problems, which any question about the manipulation of symbols can be phrased in terms of (well, any specific question—more general cases move up the arithmetical hierarchy).

If it’s not reflective in this precise sense, then the derivation of, e.g., a set-theoretic ∃ in some instances has no effect on any prediction of known physics (i.e., we are aware of no method of falsification).

Lately I’ve been thinking I might have better odds making a straight shot for ASI on my own over practicing and rehearsing the material that needs to be presented almost perfectly in the AI interviews. I’ve worked at FANG in ML / applied research for almost a decade but still can’t even get a screening interview at the top places without asking someone I know for a referral. And I really hate bugging former coworkers for referrals. Normally end up procrastinating on reaching out until the job postings just disappear haha.

I feel like there’s an interesting follow-up problem which is: what’s the shortest possible proof of FLT in ZFC (or perhaps even a weaker theory like PA or EFA since it’s a Π^0_1 sentence)?

Would love to know whether (in principle obviously) the shortest proof of FLT actually could fit in a notebook margin. Since we have an upper bound, only a finite number of proof candidates to check to find the lower bound :)