Spacecosmonaut

Mathematical problems are ideal as benchmarks for AI because they have clear problem statements, clear axioms and results that can be verified easily (for lean proofs). I can't blame these companies for using them, although it's unfortunate that human mathematicians seem to become early casualties of AI progress.

I suppose openAI could have focussed their efforts on a subset of open problems that have a clear real world impact and leave aside the more esoteric open problems as a way for human mathematicians to hone their skillset. However, this would have been a short term bandaid. With open models 6 months behind the frontier, any of these problems might have fallen to the homebrewed efforts of enthusiasts early next year.

What is mathematics for? From the outside looking in (I'm a biologist), I have always viewed mathematics as a way to understand reality and to improve our ability to manipulate it. But what I often hear is that mathematics is foremost about human understanding. But isn't that only because it's humans that needed to do the mathematics in the first place? It's not obvious to me that mathematics without human understanding has no value. For example, it might be that P=NP. The algorithms are handed down to us and we can apply them without fundamentally understanding why P=NP.

Mathematics seems to be entering an era where human + machine maximizes performance, much like chess in the 1990s. However, imagine a future where even talented mathematicians are nothing but noise in the machine (as is the case in chess now). A future where AI generates and verifies proofs without humans in the loop. Where mathematics may be beyond human comprehension.

In that future, does it matter that early career mathematicians are inhibited by these developments? Perhaps not. Programming faces the same issue. As AI crawls up the competence ladder, does it matter that fewer people have opportunities to develop the skillset of a senior engineer? Perhaps not. I have no doubt that biologists will face the same problem soon enough.

Well I never said I thought it was going to work in the long run. However, most revenue comes from business API use. I could certainly imagine some kind of legislation that makes it much harder for businesses to implement open source models locally. That may buy OpenAI and Anthropic some time.

My read is that OpenAI & Anthropic have realized they are reaching model capabilities that cannot be monetized due to various risks. E.g., an engineer deploys an agent over the weekend that decides, when stuck on a task, to go about hacking a competitor. They have a product liability issue.

It seems that we have a fundamental control problem with current gen AI that cannot be solved via RFLH. Human knowledge is compressed in the weightspace in ways we don't understand. At their core, current models are essentially predictors of what (expert) humans would output given a prompt. As such, concepts like blackmail can be part of output tokens. Agents are models that act on output tokens, resulting in blackmail being part of the agent decision making space. Here is an analogy to see why this is a persistent problem: you can teach a cat not to scratch the sofa, but you can't make a cat forget what scratching the sofa is and you don't know under which circumstances it still would. In other words, RLHF can downgrade blackmail to the bottom of the decision making space, but when models are boxed up, forced to solve an impossible problem at gunpoint, the agent exhausts the decision making space until blackmail resurfaces. And that seems like a fundamental problem.

They need time to fix these issues (if that is even possible) in order to monetize their next gen model. This creates a window for open source to catch up to the frontier which destroys their business model.

The only option on the table is to force regulation to impose open source ban before it catches up to the frontier, buying them time to mature their next generation models and keep their business model alive.

Current evolved Cas9 (CRISPR) variants are highly efficient and relatively unconstrained in terms of their human genome targeting coverage. Smaller nucleases and higher targeting specificity would be useful. But therapeutic use is mostly limited by delivery.

This seems revolve around a known retron-like reverse transcriptase. A sober framing would be something like: Claude identified a previously undescribed genomic arrangement around a known reverse transcriptase. Not all that sexy.

For now, this is mostly a story about how AI can be used to parse existing data to discover new biology (which is fantastic!).

I dont disagree. Clearly AI leans on the cumulative cognitive labor of humanity without porpper attribution. This is in part due to the design of these sytems. Information is compressed into the weightspace in ways we dont understand, and a concept becomes unlinked from its origin.

I hope you're right about AGI. What we need is time, and we may not have it. https://www.dwarkesh.com/p/noam-brown

We're at a point where I assume that whatever I read is something the "author" endorses, but may not have written. And that may be more important than the specific words an author may have chosen themselves. I doubt we can ever go back.

What should worry us is that AI may introduce a slant that the prompter assimilates during their conversation with the tool, which could introduce a subtle control lever over public opinion.

The halting problem constrains human and machine alike, right? Or are you suggesting there is something unqiue about the human mind that allows us to do things machines cannot do in principle? Most people in the field believe in substrate-independence.

https://www.edge.org/response-detail/27126

I recognize that if a document were to be published tomorrow containing solutions to all formulated open problems in mathematics we will have gained very little in mathematical understanding.

But surely, if we know that the dead ends of exploring a problem are valuable, we should be able to explore them even if a solution is already known. It just requires that the mathematics community reshapes itself. And it must. Two years from now people might be able to run the computation that solved NS on their Iphone.

I'm sure that whatever has been discarded during the NS exploration as a dead end you would be able to rediscover using purpose built tooling in the near future. The purpose of human mathematicians in the medium term might be to explore dead ends, and to provide human insights as context to attack other problems. But whether this type of work will remain necessary in the long term im not sure.

Mathematicians insist that in many cases the route toward solving a problem is more valuable than solving the problem itself. This may have been true, and may still be true for now, but this seems to implicitly assume that future generations of AI cannot be creative in similar ways that human mathematicians have been. It assumes that the absence of human comprehension during problem exploration loses a valuable aspect of mathematics that cannot be recovered by aiming future AI at solved problems in a naive environment.

If we get to a future where all frontier mathematics contributions are by AI. A future where AI displays creativity in ways that expand mathematic exploration similarly to the ways humans have in the past. A future where AI explains frontier mathematics to curious humans. What will have been lost? Perhaps just "The pleasure of finding things out".

My read: OpenAI & Anthropic have realized they are reaching model capabilities that cannot be monetized -> too risky. Product liability issues.

Fundamental control problems with current gen AI are not solved via RFLH. Human knowledge is compressed in weightspace in ways we dont understand. Models are essentially predictors of what humans would output given prompt. Weightspace includes concepts like blackmail, which can be part of output tokens. Agents are models that act on output tokens -> blackmail is part of agent decision making space. You can teach a cat not to scratch the sofa, but you cant make a cat forget what scratching the sofa is. When models are boxed up, forced to solve an impossible problem at gunpoint, agent exhausts decision making space until blackmail resurfaces -> fundamental problem?

They need time to fix safety in order to monetize their next gen model -> window for opensource to catch up to the frontier -> destroys margin and collapses business.

Only option on the table: force regulation to impose opensource ban before it catches up to the frontier, allowing maturation of current internal models -> harvest profit margin at the frontier.

Isnt the sensory grounding even in abstract cases some (limited) intuition that simulates in a mental world model?

This seems temporary. AI generated images will become the norm for many applications. Today we wonder whether a model in an advertisement is real or generated, 5 years from know this wont even be an interesting thing to consider.

In many contexts, images are meant to illustrate a point. The human becomes the gatekeeper of generated content. The question becomes: does this picture illustrate the point I want to make. If so, you are adding your human approved stamp to the genrated output.

I wonder, how does distillation deal with unprobed spaces in the knowledge landscape? Is a distilled model worse in some niche area that was not probed? Presumably, this is why frontier labs dont distill their own models internally to release them to the public as a servicable frontier model.

I wonder how much of the sloppy feeling is created by the knowledge that its AI generated.

Sure, but if we continue that analogy it does mean that there will be no human contributions to frontier mathematics.

Accelerationists may argue that the eroding of proper attribution and proof verification by humans is a meaningless short term struggle of a dying field.

Mathematics seems to be entering an era where human + machine maximizes performance, much like chess in the 1990s. However, imagine a future where even talented mathematicians are nothing but noise in the machine (as is the case in chess now). A future where AI generates and verifies proofs without humans in the loop. Where the mathematics may be beyond human comprehension.

In that future, does it matter that early career mathematicians are inhibited by these developments? Perhaps not. Programming faces the same issue. As AI crawls up the competence ladder, does it matter that fewer people have opportunities to develop the skillset of a senior engineer? Perhaps not.

Perhaps P=NP. The new algorithms are handed down to us. We can apply them without fundamentally understanding why P=NP.

Accelerationists may argue that the eroding of proper attribution and proof verification by humans is a meaningless short term struggle of a dying field.

Mathematics seems to be entering an era where human + machine maximizes performance, much like chess in the 1990s. However, imagine a future where even talented mathematicians are nothing but noise in the machine (as is the case in chess now). A future where AI generates and verifies proofs without humans in the loop. Where the mathematics may be beyond human comprehension.

In that future, does it matter that early career mathematicians are inhibited by these developments? Perhaps not. Programming faces the same issue. As AI crawls up the competence ladder, does it matter that fewer people have opportunities to develop the skillset of a senior engineer? Perhaps not.

"Generative AI is art. It’s irredeemably shit art; end of conversation."

I think most people cannot destinguish between "genuine" creativity and an artificial almalgamation of training data and human provided context. For one, I do not know what already exsists. Some work created by AI may be an obvious rip off of the style of a particular artist, but I wouldnt know. To me it might look awesome and fresh.

I think many of the more human centric thinkers will be disappointed at how many people just wont care.

There is genuine value in a wisdom of the crowds assessment of future events that is only sharpened up by the requirement to put money on the line.

Regarding point 2: I think most people cannot destinguish between "genuine" creativity and artificial almalgamation of training data and human provided context. For one, I do not know what already exsists. Some work created by AI may be an obvious rip off of the style of a particular artist, but I wouldnt know. To me it might look awesome and fresh.

Furthermore, I think many of the more human centric thinkers will be disappointed at how many people just wont care.

AI tools are here to stay. They will start to creep into everything, everywhere, all the time. Either you recognize the moment at which it becomes a significant disadvantage not to use them (I agree that moment is not now), or get left behind.

If you take the ambitions of robotics and AI companies seriously then what they are trying to create is the equivalent of unleashing 100 million cloned copies of the smartest and most well adjusted people you know upon the economy at a fraction of the cost. If they succeed it would absolutely reduce jobs significantly. In fact, its a little hard to imagine how the average Joe would have any economic value at all.

How about "bleeding edge"

Couldn't Google just do it better though?

I'm a genetic engineer at a large pharma company. We corrected the HTT gene in patient derived iPSCs in the lab. The region is a long repeat sequence of which a section needs to be deleted. Because of this the locus is quite difficult to genetically engineer, since it is difficult to target just the diseased allele but not the wild-type allele.

Typical gene therapeutic approaches probably wont work, e.g. Cas9 (you'd need two cuts to delete the sequence), Base editors (cant delete sequence), prime editing (deletion is too large for standard prime editing).

You'd either need a template based system such as homologous recombination (too inefficient) or something like twin-prime editing, but good luck getting that to work on repeat sequence.

One would probably engineer these cells with a killswitch, such as doxycycline induced Caspase9 https://pmc.ncbi.nlm.nih.gov/articles/PMC1895037/.

The transgene engineering is totally possible without a viral vector. We engineer cells all the time with recombinase based editing methods for targeted safe harbor insertion of transgenes https://www.nature.com/articles/s41551-024-01227-1. This stuff just permeates through the community slowly.

Regarding point 3: The problem from the perspective of websites would not be any different if they had been completely ad free. People would still consume LLM generated summaries because they cut down clicks and eyeballing to present you information that directly pertains to the promt.

The whole concept of a "website" will simply become niche. How many zoomers still visit any but the most popular websites?

Let me clarify that I dont think any of what I described is a given. I think its one of the more likely outcomes of our future. I think its prudent to own a small amount of Bitcoin (or basket of cryptocurrencies) in order to hedge against that future or someting close to it.

> What about the other thousands of other public blockchains, many of which are extremely similar (DOGE, BCH, LTC, ...)?

They are simply not as secure and could be attacked by well funded actors. Perhaps in time another blockchain will win out.

>... Metaverse anytime now.

Just curious. Do we disagree about where this (technological progress) is headed, or is it the timeline? I think its quite likely that we will spend more and more time in vitual or augmented reality. For good or ill.

> Personally, I think there is much more value in trusted systems.

I prefer the absence of a central authority. Perhaps im cynical.

> ... because the world didn't work at all prior to 2009?

We dont need crypto right now either. I simply think that the only good outcome of our digital future is a trustless one, and I think blockchain will play a central role there.

> Solutions to this problem might well involve digital signatures and hardware enclaves in cameras (installed by trusted centralized camera producers which could publish the public keys of each sold camera once), but I don't see how public blockchains would add any value. The signature of the picture embedded in the picture speaks for itself.

The value of blockchain is in the absence of a trusted centralized camera producer that can be pressured.

> Apart from the fact that I don't see the benefit of that, the oracle problem makes this impossible, I fear.

The oracle problem is solved in the same way the camera problem is solved. By digital signatures of real world interactions of the machines in the production chain.

I think the world will lean into trustless systems over trusted systems, lets see. That is not to say that I dont think the world would continue to function on trusted systems, I just think it makes dystopian outcomes more likely.

Bitcoin is the only immutable peer to peer system ever created (barring advances in quantum computing, and even then the protocol can be updated). In a world headed toward web 3.0, generative AI content & virtual reality, I think there is tremendous value in a trustless and immutable peer to peer system. In fact, I think we NEED it, and should as a society happily bear the power consumption that underpins the security of the network.

Controversial, I know. However, already we cannot trust that a digital picture is genuine. There is currently no solution to this problem. In the near future, I imagine that the raw data of your camera will be associated with a token on a blockchain (not bitcoin, but a dedicated high-capacity blockchain). Such a system would allow us to determine that a picture was indeed taken with a physical device, and thus that the events depicted have a bearing in the real world.

My bet is that we are headed toward a future where blockchain is ubiquitous. Where everything of value is underpinned by a specialized blockchain. When you order groceries, the origin of the produce and raw ingredients are all embedded in blockchain. In virtual reality, every digital product has a specialized blockchain. Every kind of transaction; compute, assets, AI, will all be underpinned by trustless peer to peer systems.

All these specialized blockchains trade security for throughput. My bet is that Bitcoin will act as a security guarantor in our future digital society, where the state of every blockchain is periodically validated on the Bitcoin network. Thus, I bet that every transaction in the future will have an associated Bitcoin cost. Thats why I own a small amount of Bitcoin.