fultonn

I know this because I spent a lot of social time around a Math dept in grad school and many of them are moonlighting working for subcontractors on production and evaluation of training data [1].

Anyways, mathematics is a bit of a funny discipline... some buckets:

There are lots of obvious cases where progress has obvious applications for incremental progress, and where you can ask for novel math from the perspective of the application or ask for applications from the perspective of novel math. That seems like the sort of thing that you could throw $10M at get something valuable without much human input. I do this, at much much much smaller dollar amounts, pretty regularly, and with open models, so nothing secret/sota/etc.

But there are also lots of cases where some progress gets made on something very pure and esoteric and the implications aren't really clear. Those are cases where just asking a bunch of nerds to marinate on it while interacting with a messy world in the full social generality of a modern university could make a lot of sense. The connection between the four color theorem and register allocation, for example, feels very... hard to do without humans free-associating in a diverse academic environment.

And, at last, there are cases where there are subfields or problems that are big and important in the way that driving a rare car or wearing a particularly fascinating rolex is important. Some of modern mathematics feels like it has its cultural roots in intellectual gamesmanship amongst wealthy gentleman and/or those under their patronage. A lot of the... less well-argued... objections come from this set.

Like I said, math is kind of a uniquely weird discipline. It's got aspects of practical gritty science, noblesse oblige/high-culture, fashion/taste, craftsmanship, tutelage, and so on.

--

[1] Some enterprising journalist should track down all the subcontractors used by OpenAI for mathematics data, then talk to all of those people and figure out how much breakthrough-adjacent data work was happening... pure conspiracy but I'd be curious for someone to explore the hypothesis that there were maybe people helping find fragments here and there and it wasn't all purely mechanical monkeying.

By verifying that all of its possible behaviors conform with the "it works" spec, regardless of which of those behaviors it chooses.

Monitoring with a known-safe fallback is the easiest case.

USA-specific: At ~$38K/year (including SS) you probably need to own a home without substantial deferred maintenance to make it work. So you can retire on $250K but only if you're over 62 and only if you have substantial wealth outside of those liquid assets (so probably a net worth at least 2x your liquid). If you have a spouse also on SS it much easier.

scala's repl is decent. It has its annoyances, but so does python's (white space sensitivity + repl + terminal emulators stuck in the late mid century don't mix).

If you've ever read through FORTRAN code from a mathematics department or MATLAB/C/C++ from (non-software) engineering disciplines, then you probably understand why productionizing a jupyter notebook is definitely not the worst of all possible worlds.

Also worth mentioning 2026 - 4 = 2022. Not a great year to be stick building from a cost perspective.

I would use a different adjective: exceptional.

There are not not times where lone geniuses produce amazing output. But most of humanity's progress over the last few thousand years (or at least certainly the last few hundred) resulting from a different type of work.

I'm deeply suspicious. I do not yet have a concise statement for why, but a lot of literature on the sociology of knowledge work sort of points at my thoughts.

Section 5 of the Thurston article cited by Tao touches the elephant. Raduchel's article on the economics of software [2] also touches it.

I've tried to put words to this for a few years. I think I'm just going to start writing versions of it as see if that helps me shape the thought into something more concise.

So, in the spirit of this article's style, here are some postulates:

1. There is a sociological process happening in the production function during knowledge work.

2. That production function and the associated sociological process spans years or even decades, and must outlast many of the artifacts that are produced during the early years of the function.

3. You cannot get the right lines of code or the right theorems proved without running that sociological process alongside the artifact production process.

4. It is impossible to completely separate the sociological process from the artifact construction process. If you just iterate on artifacts then too much of the required hidden state is lost to make progress in the right direction. This is true even if you include distilled artifacts capturing pieces of the sociological process (eg meeting notes, documentation, commit logs, prompts).

5. So you need that sociological process, or something like it, to still happen.

6. For a lot of knowledge work that process plays out in extremely high-fidelity social interactions [3] that we have not yet captured in the datasets that would be required to reproduce those dynamics.

7. And even if we do collect that data, our current architectures and training algorithms and hardware would be useless given the size of the datasets.

So: the technology today gives us the ability to iterate on the production of artifacts. But it does not sufficiently simulate the social process which gives rise to the Right artifacts.

This isn't exactly what I actually think, but it's a version of the thing that I intuit when I watch heavy use of AI in both software projects and formalization projects. And simulating that process feels way harder than people are currently assuming.

[1] https://arxiv.org/pdf/math/9404236 Section 5.

[2] https://www.nationalacademies.org/read/11587/chapter/11 pp 166-168.

[3] there is a reason we still gather in-person around white boards, and why doing so is more crucial for some types of work than others.

(And I only reached for the LLM after working through all the flowcharts and tables in the shop manual, reading relevant parts of two engine repair books, and watching a bunch of youtube videos.)

It's especially awesome in domains where:

1. you personally don't have the time to learn,

2. you can trivially evaluate the output ("does engine go brrrrrr or brr-splaugh-crunch"), and

3. there are no bad outcomes; or, there are bad outcomes (destroy engine, hurt myself) but you can augment with ground-truth ("do X as suggested by nlp machine but follow shop manual for doing X").

It was a 3B or 8B model too, so I probably burned more energy on unnecessary oil changes than on running the llm inference.

Say what you will about LLMs, and dear god can the claude-style harness outputs be annoying in the workplace. But they're great tools for accelerating the sorts of ultra-narrow-path learning curves that commonly pop up in tinkering/hacking/similar sorts of activities.

(Also, people without critical thinking skills believing everything they read pre-dates LLMs by thousands of years. This was a widely discussed problem with popular media consumption in the USA during the several years preceding the invention and popularization of LLMs (say 2015 to 2021). There's a different, more fundamental educational diagnosis for that problem, which LLMs may exacerbate, but which they did not cause and cannot cure.)

So, to be fair: most philosophy majors wouldn't have much luck with Rudin.

> Dive into the classics after you have gained the maturity from modern texts.

Diving into old texts is a skill unto itself. That's why a lot of institutions do the great books thing as a core curriculum (so, maybe 2-3 courses taught in this style, as an alternative to more conventional phil 101/history 101 style distribution requirements). Then a more conventional education from there onward. The theory is that this is a mid-point precisely because it provides lots of transferable skills for diving into the classics in your chosen field, while avoiding the "let's learn analysis from descarte" excesses.

Of the lot, Elements feels misplaced.

Lots of people actually do read Elements as part of their course of study. It's niche but there's a whole cottage industry within academia for that sort of thing. There are probably over a dozen institutions that have either a degree program or a core curriculum that is organized around original texts, with Euclid usually serving as the math distribution of that sequence. So running into people who have read (big chunks of) Elements is not that uncommon. That's true even IRL outside of online discussions forums on thread topics that likely select for such people.

My impression is that this is not really true of the other examples. Except maybe Godel's proofs; I do think a sufficiently motivated instructor could pull a decent chunk of college students through the original text in a semester. Probably better ways to spend everyone's time, though.

> IMO that's the same as going on the street door by door and checking if one is left open to steal everything inside the house...

From experience: this does happen regularly in some neighborhoods of some cities in the US, and even that isn't always an enforcement priority. So lack of enforcement on the internet, where most the perpetrators probably aren't even in a jurisdiction with an extradition treaty, isn't exactly surprising.

In a past life I spent a lot of time on that sort of thing for control systems and RL. Spec says what not to do, reward says what to do, implementation can be arbitrarily complex wrt the spec.

There are many opportunities for an analogous move in LLM-assisted software engineering.

The conceptual gap I'm referring to here has nothing to do with formal methods per se. It's just an analogous problem with the quanta of information required to state the spec vs the quanta of information required to state the implementation.

Namely: once your problem has enough of a certain type of essential complexity, there's not a huge delta between "a sufficiently specific description of the problem" and "the source code that solves the problem". The complexity of a sufficiently specific prompt approaches the complexity of the actual solution. At that point, the former does not have the purported benefit and the latter has a lot of huge benefits (determinism, modularity, etc).

When one is operating in that regime of problems, proposing that one can substantially automate the software engineering function has a real "I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question" feel to it.

This is deeply related to the conceptual error a lot of executives are currently making around automation in/of their software engineering orgs.

It has always been true that learning some formal methods probably makes you a better programmer in certain ways, even if you never use them. I think it's increasingly also true that learning some formal methods probably makes you a better manager of people/processes that product software.

edit: edited.

We already arrived at that destination a decade ago.

Probably even further back tbh.

There's just a lot more people playing the game now, without the social indoctrination that made it more tolerable in some circles.

It's not that bad, though. September is annoying but you kind of miss the eternal renewal once you're out.

You are not. A disproportionate amount of value in the computing industry was created by <strike>geniuses</strike> decently smart people who worked together and who decided to just tell people how to do things instead of trying to capture the value of being the first person to figure out how to do those things.

This observation pre-dates the current wave of AI hype by a half century or so.

> driven by greed

I can only speak for myself.

For me it's exactly the opposite. If I want to explain how something works, I can just... do that. If I want to share an artifact demonstrating how to solve a particular type of problem, I can just... do that. If I want to mentor/teach, I can just... do that.

Doing those things within the confines of Academia Approved Institutions is exhausting and distracting.

To wit, and the actual point of this post: the term "Publishing Research" in this article doesn't mean "post it on a .html page and share the source code". It means engaging in a very specific and peculiar and extremely political modality of communication.

And it really only makes sense to do that specific and peculiar and political thing you're at a stage in your professional/personal development where you need to play that particular prestige game. (Which there's nothing wrong with, but it is a deeply cargo culted version of the actual scientific process.)

Wagons (as sold in the US market anyways) are passenger vehicles and therefore have the same problem as the Forester[^1]: your saw leaking some gas or bar&chain is kinda not ok. Or insert any other "thing that makes messes".

The true ideal is a 90's regular cab Tacoma/Ranger or a final gen Transit Connect depending on use-case.

> It's too bad this is america and everyone hates them and buys these dumb trucks instead

Well, at least that hypothesis might get tested.

[^1]: https://news.ycombinator.com/item?id=48968375

Unfortunately I'm 5'7 and even the low roof transit is 6.8. So it's paying >$10K more for a pretty non-ergonomic solution.

> seems too small to even be useful for that.

One advantage of a bed when transporting 2x's is that you get the hypotenuse. So the bed goes further than you'd think in terms of load length. The 60/73 beds of the tacoma are certainly better than the mavericks, and those trucks are still smaller/same size as the transit.

But my actual observation is that, in cities, transits just really are strictly more annoying and dangerous to drive than a mid-sized truck. Maybe it's just me, but maintaining situational awareness on an unprotected bike lane to my right in a transit is way way way more difficult than the same in a mid-sized truck.

The shorty transit connects from the 2010s aren't what the modern line-up looks like. Modern transits are heavy long wheel-base vans with surprisingly large displacement engines and unfortunate blind spots.

I do get the animosity about trucks on city streets (...I bike to work and pass 4 ghost bikes every morning and evening and have had my share of close calls...).

But as a bicyclist, I'd rather ride next to a tacoma or ranger than a current gen transit. And it's not even close.

If you bicycle and prefer vans over trucks on your streets, I encourage you to test drive any 2026 transit. It'll change how you ride around vans, even if you're already a careful cyclist.

Maybe in the past, but the size thing isn't really true of the modern line-up. Relative to eg a base Tacoma or especially a Maverick, The Ford Transit has about the same wheel base, is slightly heavier, has worse visibility (for me at least), has a larger displacement engine, and is on a pragmatic level more difficult to maneuver and park. They also have far fewer creature comforts, which wouldn't be a huge deal if they didn't MSRP at $10K MORE than a base midsized truck!

Vans also don't have 4x4, have much lower ground clearance, no good mechanism for transporting longer stuff, and you can't hose them out nearly as easily as a truck bed. The main thing that those vans are built for is just lots of weight in the van. Great fits for some use-cases, but not mine.

And, again, they're just seriously expensive for what they are because of all the commercial applications.

I did also consider an old minivan or a used van from previous generations when they made the smaller transits. Again, 4x4/awd isn't really super optional in these parts, so pickins were slim.

But it's still funny to me. Because in my mind one of the main perks of an open bed is that you can do so much before having to deal with a trailer (which are super annoying, especially in cities, especially without off-street parking).

Which I guess is just to say: maybe there's a market for a truck that can't tow much, because maybe there's a portion of the market that wants an open bed but doesn't need to tow a massive camper or whatever.

This is why I'm excited for REO.

I live in a pretty dense city and don't have much storage space in my residence. On the weekends I'm clearing and developing a lot a couple of hours away. My vehicle doubles as storage for all the stuff I need for that and don't want to leave unattended (saws, tools, winch, etc).

I used a Forester for the first two years, which was cramped but doable. The bigger issue was messes. Oil has a way of getting everywhere. I also had one gas spill from a faulty saw gas cap which was both dangerous and a huge PITA to clean up. I also got to a phase where the difference between awd and 4x4 matters, and where I was yanking milled planks around which didn't feel safe doing with the forester.

I switched to a truck after a couple of years and went small. The box is completely full between tools, saws, gas, winch, ropes, etc. When I do groceries I can usually fit some of the "hard" bags back there (cans, bottles) but the rest has to go up front.

A REO would be great for my use-case. The price is low enough that for the price of a new Tacoma or F-150 I could (get close enough to) buying both a REO and a used beater 3/4 ton at the property, which is a better fit for the things I need a big truck for anyways. And I wouldn't be driving a dumb big truck around a very cramped city.

For now I'm stuck with a big truck in the city because owning more than one personal vehicle in this town is insanity.

FWIW, transporting (even large) bikes in the bed of a truck is doable and normal. The typical way to do this isn't with a hoist. You just ride or walk it up a ramp and tie it down.

ofc a Forester or RAV4 with a trailer is a far superior solution, especially for interstate driving where the trailer is less of an annoyance.

The concern about poor precedent stemming from poor cases has some rational sense, but we have the benefit of experience. Empirically it just hasn't tended to play out like that in the case of consumer protection statutes in MA. One reason this doesn't happen in practice might be the limited bandwidth of the appellate process. The SJC could (and likely would) prioritize answering questions about the statute in the context of cases brought by the AG.

The longevity pro-consumer laws in MA provides some good empirical data that cuts against the concern about push-back.

It'll have reach because MA has a long-arm statute and there's a rich history of applying that statute in the context of Chapter 93.

It'll have teeth but probably not to the effect that you hope.

This statute was written such that only the Attorney General can bring action; see Section 10(b). This diverges from a long history in the Commonwealth of allowing private individuals to bring civil suits for most types of Chapter 93 violations.

As a result, I anticipate that the most impactful change will be in the quantity and frequency of political donations to Mass AG candidates (and in the case of contested primaries their aligned block of candidates up and down ticket).

Consumer protection laws should always provide for a private cause of action. Otherwise they just function as a mechanism for legalized corruption.

Got your email. Let's connect there.

I was not talking about the output of models.

I'm referring to the model itself. The `.ckpt` file is clearly transformative wrt its training set. Or, at least, substantially more transformative than other things that have long received fair use protection.

> Like, I get that you're invested in the industry

On the contrary, I'm invested quite heavily in the exactly opposite hypothesis -- that the ChatGPT/Claude/Gemini UX you're referring to is not fit-for-purpose.

> How the heck would you train children and adolescents on the responsible use of AI?

By teaching them how it works, how it doesn't work, and to think of it as a unit of computation rather than an anthropomorphic entity.

We are hiring a Senior Scientist to join our team. We're working at the intersection of LLM training and the inference stack.

We're looking for someone who wants to work in the following intersection:

* Both implementation experience and theoretical background in compilers, programming languages, or programming models. We have an absolutely requirement for someone with mathematical maturity in formal systems (programming language theory, type theory, proof assistants, etc.)

* practical experience pretraining and rl tuning LLMs (or skill-adjacent experience / expertise)

* practical experience developing or modifying inference engines for LLMs

From an academic perspective, think "POPL/PLDI/OOPSLA/CAV/etc." + "NeurIPS/ICML/AISTATS/etc.".

The job posting for a Senior Scientist position, with rough salary ranges, will go live soon. I'll edit or post a reply with the link when it's live. For now, if you have expertise at some intersection of the above topics, please feel free to reach out: nathan@ibm.com

Application link for Research Scientist position: https://careers.ibm.com/en_US/careers/JobDetail?jobId=118292...

We also have a Research Engineering role for which we're actively interviewing: https://careers.ibm.com/en_US/careers/JobDetail?jobId=113488...

I've thought a lot about how to safely deploy autonomous systems (even did a whole PhD on the topic, lol).

I think one can ethically deploy a system that has some degree autonomy. It takes a lot of work to do right. And the tooling for LLM-based systems isn't quite as mature as the tooling for e.g. control systems. Part of this is because so many resources in AI safety are misspent on problem statements that are myopic or grandiose. Between "don't say pii" and "prevent ASI extinction" there's a hard but tractable control systems-y view of AI safety.

But I don't think there is any sort of fundamental barrier that prevents us from building appropriately constrained LLM-based systems.

> And even then, there are such copyright issues with it. Is there no practical ethical use for AI? Responsible use doesn't equate with ethical use for me.

When responding to a position, especially on the internet, I try to empathize with the thing I'm responding to. Not just understand it, but sort of put myself in a mental state where I have an emotional attachment to my conversation partner's point of view.

With respect to Copyright as a legal framework in my country (USA): despite my best attempts, I really struggle to develop empathy for the viewpoint that LLMs/diffusion models are not a transformative use. I can certainly sympathize, but trying to actually put myself in the shoes of believing that training an LLM is a purely derivative and non-transformational work just feels far too alien. There are so many things that are "clearly transformative" but required so many orders of magnitude less scientific/technical/engineering genius.

Which isn't to say that the US legal system's definition of copyright is the morally correct one.With respect to copyright beyond the US legal system, or beyond legal denotations generally: I can certainly empathize.