Jack000

None of my relatives who were alive in 1969 have ever heard of the US moon landings. The information was totally suppressed at the time and in the intervening decades they simply had no reason to seek it out.

The younger generations think it’s a hoax but are more uncertain because they know that any information about the event is likely to be propaganda

None of the things you listed mean much when it comes to ethno-nationalism. There are a lot of Chinese that love American brands but still hate the US for nationalist reasons.

Among my relatives I would say all are anti-US. About 5-6 of them vehemently so and want the war to start immediately.

I grew up in China and if you think there’s less propaganda in China compared to the US I don’t know what to tell you.

It's heavy and the extra encumbrance is not great for general purpose computing, though I never felt the need for extra screen real estate while I'm coding, watching youtube, sending emails etc.

That said I still use my AVP regularly, it's a great home theater system for 4k HDR content that's portable for travel. I've owned other headsets like the valve index but the novelty of 3d gaming wore off pretty quickly.

This is kind of the visual equivalent of asking an LLM to count letters. The failure is more related to the tokenization scheme than the underlying quality of the model.

I'm not certain about the specific models tested, but some VLMs just embed the image modality into a single vector, making these tasks literally impossible to solve.

Chess is a great example because it highlights the subtle difference between LLMs and agents. What GPT3.5 does is not quite playing chess but creating realistic chess moves that a human might make.

An LLM could play chess though, all it needs is grounding (by feeding it the current board state) and agency (RF to reward the model for winning games)

I think a lot of the confusion on whether LLMs can think stems from the fact that LLMs are purely models of language and solve intelligence as a kind of accidental side-effect.

The real problem that an LLM is trying to solve is to create a model that can enumerate all meaningful sequences of words. This is just an insane way of approaching the problem of intelligence on the face of it. There's a huge difference between a model of language and an intelligent agent that uses language to communicate.

What LLMs show is that the hardest problem - of how to get emergent capabilities at scale from huge quantities of data - is solved. To get more human-like thinking, all that is needed is to find the right pre-training task that more closely aligns with agentic behavior. This is still a huge problem but it's an engineering problem and not one of linguistic theory or philosophy.

The thing is, it's not gibberish. A sufficiently small language model can be understood by humans: https://twitter.com/karpathy/status/1645115622517542913

The explanation is perfectly sensical, just too complex for humans to understand as the model scales up.

The thing you're looking for - a reductive explanation of the weights of a ANN that's easy to fit in your head, does not exist. If it were simple enough to satisfy your demands, it wouldn't work at all.

I was writing an SVG editor web app but didn't know anything about the fabricjs api. Here's a portion of my prompts with chatgpt (april 2023 version, the date at the top is the share date). The initial code came from a different chatgpt thread.

https://chat.openai.com/share/e630c5d4-d492-43cb-b7e5-214ff8...

I finished the app in 2 days, with a third day for css/visual styling. I previously might have hired someone to do this or tried to figure it out myself, which would have taken about a month.

At the end of this thread chatgpt kind of goes off the rails a bit and fails to center the uploaded image. I think this is because it can't execute the code and see its results, and can only take blind guesses at what the problem might be.

I still had to write about 10% of the code myself, but it's about 10x faster for me to use chatgpt. I think I'd use chatgpt even if it were slower, because I prefer "thinking in natural language" vs "thinking in code"

here is the final app in production: https://tinyurl.com/368w3a9y

In pixel space a convnet uses pixel-wise convolutions and a pixel-kernel. If you represent a vector image as a polygon, the direct equivalent to a convolution would be the Minkowski sum of the vector image and a polygon-kernel.

You could start off with a random polygon and the reverse diffusion process would slowly turn it into a text glyph.

I think this approach isn't ideal because you're representing pixels as 150x150 unique bins. With only 71k fonts it's likely a lot of these bins are never used, especially at the corners. Since you're quantizing anyways, you might as well use a convnet then trace the output, which would better take advantage of the 2d nature of the pixel data.

This kind of reminds me of dalle-1 where the image is represented as 256 image tokens then generated one token at a time. That approach is the most direct way to adapt a causal-LM architecture but it clearly didn't make a lot of sense because images don't have a natural top-down-left-right order.

For vector graphics, the closest analogous concept to pixel-wise convolution would be the Minkowski sum. I wonder if a Minkowski sum-based diffusion model would work for svg images.

> what will LLMs ever do for us?

Hallucinations are an engineering problem and can be solved. Compute per dollar is still growing exponentially. Eventually this technology will be widely proliferated and cheap to operate.

I didn't say they would be better. Most people on earth lack any access to healthcare at all. https://shorturl.at/joA23

A doctor for every person, a teacher for every child, available any time and for free.

As Yann Lecun said, “in high dimension, there is no such thing as interpolation. In high dimension, everything is extrapolation.”

It’s trivial to come up with a prompt that doesn’t exist in the dataset. To generalize, the model cannot memorize.

The foreign buyers tax, the empty home tax, and now the total ban on foreign buyers introduced this year has had zero effect on home prices. It's time to stop blaming foreigners.

The solution is to build more and more quickly.

oxycotton.com and oxyloo.com are taken unfortunately

washluv.com is still available though

There are still some good 6 letter names out there, if you're willing to use a brandable (made-up) word. Some I randomly made just now: pletha.com encryx.com oxiply.com

double descent is a different phenomenon from grokking

Not seeing anything about the dataset, are they still using LAION? There's no mention of LAION in the paper and the results look quite different from 1.5 so I'm guessing no.

> the model may encounter challenges when synthesizing intricate structures, such as human hands

I think there's two main reasons for poor hands/text

- Humans care about certain areas of the image more than others, giving high saliency to faces, hands, body shape etc and lower saliency to backgrounds and textures. Due to the way the unet is trained it cares about all areas of the image equally. This means model capacity per area is uniform, leading to capacity problems for objects with a large number of configurations that humans care more about.

- The sampling procedure implicitly assumes a uniform amount of variance over the entire image. Text glyphs basically never change, which means we should basically have infinite CFG in the parts of the image that contain text.

I'm not sure if there's any point in working on this though, since both can be fixed by simply making a bigger model.

I had a tough time with "Man eating shark"

Why would a man eat a shark? Shouldn't it be the other way around?

This conflict only exists if you believe in a single, inviolable self.

Which cell is the “real” one when it divides? Which branch of a tree is the true tree?

Both versions of you have equal claim to the past, as long as the copy is truly identical.

no it's definitely GPT4.

I'm not sure why the share page says "Model: Default" but it's "Model: GPT-4" in my webui. Seems like a bug in the share feature

I tried to replicate a few of my chats (the displayed date is incorrect, it seems to be the publish date instead of the original chat date):

svg editor:

early april: https://chat.openai.com/share/c235b48e-5a0e-4a89-af1c-0a3e7c...

now: https://chat.openai.com/share/e4362a56-4bc7-45dc-8d1b-5e3842...

originally it correctly inferred that I wanted a framework for svg editors, the latest version assumes I want a js framework (I tried several times) until I clarify. It also insists that the framework cannot do editable text until I nudge it in the right direction.

Overall slightly worse but the code generated is still fine.

word embeddings:

early april: https://chat.openai.com/share/f6bde43a-2fce-47dc-b23c-cc5af3...

now: https://chat.openai.com/share/25c2703e-d89d-465c-9808-4df1b3...

in the latest version it imported "from sklearn.preprocessing import normalize" without using it later. It also erroneously uses pytorch_cos_sim, which expects a pytorch tensor whereas we're putting in a numpy array.

overall I think the quality has degraded slightly, but not by enough that I would stop using it. Still miles ahead of Bard imo.

Here's roughly how my conversation with GPT4 went

Me: I want to make an svg editor, give me some suggestions. I mainly want something with mobile support.

GPT4: gives me some options

I look over the options and choose fabricjs

Me: Start by loading an svg at a predefined url

GPT4: <code>

Me: Ok now implement a save feature, send the json to this url ...

GPT4: <code>

Me: The text is loaded as an image, I want the text to be editable

GPT4: <code>

Me: I'd like to add some google fonts to the text editor

GPT4: <code>

Me: The fonts aren't loading, I think we need to load the fonts first before initializing the canvas.

GPT4: <code>

Me: Ok add a undo/redo feature

GPT4: <code>

Me: Let's add some clickable buttons instead of hotkeys, here is the html..

GPT4: <code>

I probably could have done this myself, but frankly it would have taken me a long time to figure out the fabricjs api. It probably saved me at least a week making this thing.

here's the live app: https://tinyurl.com/2dhh58cn

and the code: https://tinyurl.com/2tu4xrtn

you can tell the GPT generated sections by the (overly) verbose comments

Humans use language for communicating ideas, which is very different from language modeling. Here's some discussion on this topic: https://www.alignmentforum.org/posts/htrZrxduciZ5QaCjw/langu...

I would add that the above comparison is misleading, because humans have a massive advantage in that they have prior knowledge of what words mean. A more apples-to-apples comparison would have the human do next word prediction on a language they don't know.

This would be akin to me giving you a few GBs of Chinese text, with no grounding or translation, then try to communicate with you in Chinese after you've read the whole thing.

There’s too much focus on AGI.

Language models do not emulate human minds - they are models of language. The emergent behavior from these models are only a side effect of their main training task, which is to build a model of all meaningful sequences of words. We then use RFHL to bias the model toward a small area of the language latent space which conforms to our idea of intelligent behavior.

Humans (a GI) have zero ability to do language modeling. Human equivalent AGI would similarly fail at this task.

The technology behind language models is more important than general intelligence - it is a universal induction engine that can model (and truly understand) the latent structure of any signal.

This type of data is actually better than independent human text (specifically for training the LLM that originally produced the output)

GPT4 is trained with PPO+RLHF. The web text that is produced by the LLM then fed back in will be more proximal to the original token distribution.

In other words, by selectively publishing LLM output you’re effectively performing the same action as clicking the thumbs up/thumbs down button on the chatgpt webui.

I agree with openai that this will not be a problem at all, since you would need a process to gauge the quality of the data anyways, even for human text.

LLM failure modes are caused by the lack of external context - they perform poorly on visual tasks because they have no sense of vision for example. Hallucinations are another aspect of this - as embodied agents, humans and animals have a strong bias for counterfactual reasoning because it is needed to survive in a complex information-rich environment (if you believe in something that is false, you tend to get eaten)

The real solution to these problems is to train transformers on a more human-like information context rather than pure text. Hallucinations should naturally decrease as LLMs become more "agentic"

the point is that you have to want to in the first place. This is kind of like looking at “the flame” or some other modern art and saying “yeah that’s easy, I could have done that.” It’s missing the point entirely.

Actually I don't think most programmers are capable of designing something like spacex.com

- every image on the site fits the minimalist theme (and there are hundreds). The photos are always dark in the appropriate areas so the text is readable, and they're dynamically cropped/resized with good positioning for all breakpoints.

- the front page appears to be simple, but if you click through the site there are actually a ton of dynamic illustrations and charts like this: https://www.spacex.com/human-spaceflight/

The key thing is that the aesthetic is executed consistently and with restraint. This video is a bit old now but it's as true now as it's ever been: https://www.youtube.com/watch?v=EUXnJraKM3k

Minimalism is the most difficult aesthetic to pull off, because it's easy for the end result to look unfinished, cheap or lazy.

I think the problem is not that coders can't be designers, but that coders generally don't really care about design, and I say that as a coder. If you gave the spacex project to the average coder, you'll probably end up with something like sap.com or fortinet.com

Web design has converged a lot in the last 20 years because people don't want to learn a new UI for every website they visit. This makes good web design harder, because now if you want to create something memorable and distinctive, you'll need to do it in spite of the navigation and functional parts looking identical to every other site.