Hacker News Reader

Longer term computation isn’t really the limiting factor for generative AI, it’s training data. Generative AI is like Google search before the web responded to their search engine existing. There’s a huge quantity of high quality training data which nobody had any reason to pollute ready for the scrapping.

But modern search is hampered by people responding to algorithmic indexes. Algorithms responding to metadata without directly evaluating content enabled a world of SEO and low quality websites suddenly being discoverable as long as they narrow their focus enough.

So longer term it’s going to be an arms race between the output of Generative AI and people trying to keep updating their models. In 20 years people will get much better at using these tools, but the tools themselves may be less useful. I wouldn’t be surprised if eventually someone sneaks advertising into the output of someone else’s model etc.

This has already happened. Search google for a few random terms, and go through the first page of web and image results. A decent chunk will be AI-generated.

I disagree. With more computation you can train a bigger model on the same size training data and it will be better. There is a lot if knowledge on the internet that GPT-4 etc. have not yet learned.

The issue is the training data isn’t some constant. Let’s suppose OpenAI had 10x the computing power but a vastly worse dataset, do you expect a better or worse result?

The question is ambiguous without defining how much worse the dataset is.

But why would the dataset be worse, they can just use the same one as before.

A dataset that’s 2 years old is worse than one that’s 20 years old even if it contains the same data.

Even facts age. In 2004, Pluto was still classified as a planet. Not such a big deal on its own, but stale data gets a little bit worse every day.

> I assume that concrete code will always be faster and the best way to have deterministic results, but I really have to idea how to conceptualize what the future looks like now.

It will likely be a mix of both concrete code and live AI generated experiences, but even the concrete code will likely be partially AI generated and modified. The ratio will depend on how reliable vs creative the software needs to be.

For example, no AI generated code running pacemakers or power plants. But game world experiences could easily be made more dynamic by generative AI.

> ...this has seemingly infinite flexibility based on so little and that's just incredible.

What makes it little? This is the difference between von Neumann architecture and Harvard architecture.

Makes me wonder if there's any company which is trying to train a model to produce three D worlds within Unity (not as a video like oasis).

This at least is a bit more realistic than what’s being presented by Google.

There are already a number of techniques for procedurally-generating a world (including Markov-based systems).

The problem with replacing procedural world generation with LLM generation are a) you need to obtain a data set to train it, which doesn’t commercially exist, or train it yourself, b) there’s a fundamental need to iterate on the design, which LLMs do not cope with well at all, c) you need to somehow debug issues and fix them. That’s quite apart from the quality issues, cost and power usage.

I was hoping that Midjourney would make the leap from 2D to 3D, then start to provide the 3D model. A bit further, you could tell the Midjourney LLM to create a small scene, like: The character runs fast. Then, Midjourney LLM could output whatever script is necessary to make the 3D model "run". If Midjourney doesn't do it first, I am sure someone else will.

> Will traditional programming be used for most "production" workloads with gen AI being used to aid in the prototyping and development of those traditional programs

I mean we're already there with Copilot, Cursor and other tools that use LLMs to assist in coding tasks.