alwayshasbeen.png
> The AI effect occurs when onlookers discount the behavior of an artificial intelligence program by arguing that it is not "real" intelligence.[1] > Author Pamela McCorduck writes: "It's part of the history of the field of artificial intelligence that every time somebody figured out how to make a computer do something—play good checkers, solve simple but relatively informal problems—there was a chorus of critics to say, 'that's not thinking'."[2] Researcher Rodney Brooks complains: "Every time we figure out a piece of it, it stops being magical; we say, 'Oh, that's just a computation.'"[3]
> "AI is whatever hasn't been done yet."
> —Larry Tesler
That's very different from code which (perhaps surprisingly) isn't well formalized. The goals are often vague and it's difficult to figure out what is intentional and what incidental behavior (esp. with imperative code).
As someone who was deeply involved in the Go scene since the early 2000s let me emphatically assure you it was not at all clear. Indeed it was a major point of pride among Go enthusiasts that computers could not play it well, for various reasons (some, like the branching factor, one could potentially grant that advances in hardware and software could solve for eventually. Others, like the inherent difficulty in constructing an evaluation function, seemed intractable).
Betting markets at the time of the AlphaGo match still had favorable odds for Sedol, even with the knowledge that Google was super-confident baked in.
It is extreme hindsight-bias of exactly the type the grandparent was talking about to suggest that obviously everybody knew all along that Go was very beatable by "non-real AI".
It's typical for enthusiast of anything to believe their area is special. To be able to judge how feasible besting humans is a question for computer scientists / engineers anyway. You need some knowledge of Go for sure, but don't actually have to be a master Go player.
> It is extreme hindsight-bias of exactly the type the grandparent was talking about to suggest that obviously everybody knew all along that Go was very beatable by "non-real AI".
Maybe it was a surprise for you, but I was familiar with the pre-AlphaGo state of the art, and it was always clear it's just a technical problem. There's no fundamental difference between a game like Go and Chess, it's only the parameters which differ, where Go's space is exponentially larger.
> Others, like the inherent difficulty in constructing an evaluation function, seemed intractable
The branching factor is what makes the evaluation function difficult in Go and is therefore again a technical problem.
> Betting markets at the time of the AlphaGo match still had favorable odds for Sedol, even with the knowledge that Google was super-confident baked in.
This is a non-argument, betting markets are not rational.
Actually it doesn't, as demonstrated by the fact that people have made fairly decent chess implementations in 1K lines of code and things like that. Chess is comparatively easy because it has well-defined rules and well-defined concepts of "good" and "bad". Refactoring something to be multi-threaded is incomparably more complex and any comparison to this is just pointless.
Interesting.
Here's a sample question I just asked the AI:
We are working on a React mapping application. I am going to give you two components from this application. Our goal is to extract functionality to deduplicate code
We can make a single base component, extract code to hooks, or any other strategy which you think is appropriate
Here is travel.tsx:
<snip>
Here is heatmap.tsx:
<snip>
And here is the response:
To deduplicate code and improve maintainability between the `TravelMap` and `Heatmap` components in your React mapping application, we can identify common functionalities and extract them into reusable hooks or components. Here's a strategy considering your code:
1. *Extract Common Hooks*: Both components use similar hooks for managing markers, popups, and map references. We can create custom hooks that encapsulate these functionalities.
2. *Create Shared Utility Functions*: Functions like `initialViewState`, which are identical in both components, can be moved to a separate utility file and imported where needed.
3. *Refactor Map Layers and Sources*: The way map layers and sources are handled is quite similar in both components. A custom hook or component to manage map layers and sources would be beneficial.
4. *Generalize Popup Handling*: The popup handling in both components can be abstracted into a custom hook or a higher-order component.
5. *Combine Common Styling*: Shared styles, such as `MAP_CONTAINER_STYLE`, can be moved to a common CSS module or a styled-component that can be reused.
6. *Refactor Event Handlers*: Functions like `onMapClick`, `onMouseMove`, and `onMouseLeave` have similar structures. These can be part of a custom hook or a utility function with configurable parameters.
7. *Legend and Live Region Components*: These components are used in a similar way in both `TravelMap` and `Heatmap`. Ensure they are designed to be reusable with different props.
8. *Refactor Map Initialization*: The way the map is initialized and configured is quite similar. This can be abstracted into a custom component or hook.
### Example Implementations:
#### 1. Custom Hook for Map Markers:
```typescript // useMapMarkers.ts import { useState, useCallback } from 'react';
export const useMapMarkers = (initialLat, initialLng, world) => { // Similar marker logic from your components // ...
return {
marker,
onMarkerDrag,
onMarkerDragEnd,
updateMarkerPosition,
// other marker related states and functions
};
};
```#### 2. Shared Utility Function:
```typescript // utils/mapUtils.ts export const initialViewState = (map, mapType) => { // Implementation }; ```
#### 3. Custom Hook for Map Layers and Sources:
```typescript // useMapLayers.ts import { useState, useEffect } from 'react';
export const useMapLayers = (props) => { // Logic to handle layers and sources // ...
return {
dataLayer,
patternLayer,
// other layer related states and functions
};
};
```#### 4. Generalized Popup Handling:
```typescript // useMapPopup.ts import { useState } from 'react';
export const useMapPopup = () => { // Popup logic from your components // ...
return {
popupState,
openPopup,
closePopup,
setPopupData,
// other popup related states and functions
};
};
```### Integration:
Once these hooks and utilities are created, integrate them into your `TravelMap` and `Heatmap` components. Replace the duplicated logic with calls to these shared resources. This will significantly reduce code duplication and make future maintenance easier.
### Testing and Refinement:
After refactoring, thoroughly test both components to ensure that all functionalities work as expected. Watch out for any edge cases or specific behaviors that might be affected by the refactoring.
For those suggestions, I might use five out of eight of them, and probably do one or two things differently. But you cannot, with a straight face, say the model did not understand. It clearly did. It suggested reasonable refactors. If being able to refactor means understanding, I guess we have understanding!
I could continue with this conversation, ask it to produce the full code for the hooks (I have in my custom prompt to provide outlines) and once the hooks are complete, ask it to rewrite the components using the shared code.
Have you ever used one of these models?
Cleaning up code also follows some well established patterns, performance work is much less pattern-y.
Codebases like FFMPEG are one of the kind. I bet you need 10 or 100 times more understanding than the react thing you mentioned above.
One day maybe AI can do it, but it probably won't be LLM. It would be something which can understand symbols and math.
> Because refactoring requires understanding, which LLMs completely lack.
<demonstration that an LLM can refactor code>
> Cleaning up code also follows some well established patterns, performance work is much less pattern-y.
Just as writing shitty react apps follow patterns, low-level performance and concurrency work also follow patterns. See [0] for a sample.
> I bet you need 10 or 100 times more understanding
Okay, so a 10 or 100 times larger model? Sounds like something we'll have next year, and certainly within a decade.
> One day maybe AI can do it, but it probably won't be LLM. It would be something which can understand symbols and math.
You do understand that the reason some of the earlier GPTs had trouble with symbols and math was the tokenization scheme, completely separate from how they work in general, right?
[0]: C++ Concurrency in Action: Practical Multithreading 1st Edition https://www.amazon.com/C-Concurrency-Action-Practical-Multit...
It's obvious from context here that the refactoring that was mentioned was specifically around concurrency, not simply cleaning up code.
https://chat.openai.com/share/7c41f59a-c21c-4abd-876c-c95647...
What you've shown is actually a great example of the what folks mean that LLMs lack any sort of understanding. They're fundamentally predict-the-next-token machines; they regurgitate and mix parts of their training data in order to satisfy the token prediction loss function they were trained with.
In the linked example you provided, *you* are the one that needs to provide the understanding. It's a rather lengthly back-and-forth to get that code into a somewhat useable state. Importantly, if you didn't tell it to fix things (sqlite connections over threads, etc.), it would have failed.
And while it's concurrent, it's using threads, so it's not going to be doing any work in parallel. The example you have mixes some IO and compute-bound looking operations.
So, if your need was to refactor your original code to _actually be fast_, ChatGPT demonstrated it doesn't understand nearly enough to actually make this happen. This thread conversation got started around correcting the misnomer that an LLM would actually ever be able to possess enough knowledge to do actually valuable, complex refactoring and programming.
While I believe that LLMs can be good tools for a variety of usecases, they have to be used in short bursts. Since their output is fundamentally unreliable, someone always has to read -- then comprehend -- its output. Giving it too much context and then prompting it in such a way to align its next token prediction with a complex outcome is a highly variable and unstable process. If it outputs millions of tokens, how is someone going to actually review all of this?
In my experience using ChatGPT, GPT4, and a few other LLMs, I've found that it's pretty good at coming up with little bits to jog one's own thinking and problem solving. But doing an actual complex task with lots of nuance and semantics-to-be-understood outright? The technology is not quite there yet.
Let's see how your smooth talking LLM is going to do with things that are not web development or leetcode medium, for which so much stuff has been written. All the best.