GolDDranks

I would be relieved to be mistaken, but I still see quite egregious risks there. For instance, a human bad actor with a powerful AI would have both intelligence and agency.

Secondly, I think that there is a natural pull towards agency even now. Many are trying to make our current, feeble AIs more independent and agentic. Once the capability to effectively behave so is there, it's hard to go back. After all, agents are useful for their owners like minions are for their warlords, but an minion too powerful is still a risk for their lord.

Finally, I'm not convinced that agency and intelligence are orthonogal. It seems more likely to me that to achieve sufficient levels of intelligence, agentic behaviour is a requirement to even get there.

Are you talking about Monte Carlo tree search? I consider it part of the algorithm in AlphaZero's case. But agreed that RL is a lot harder in real-life setting than in a board game setting.

Note that we aren't talking about risks of LLMs specifically here, they embody what I said in the ancestor comment: "current technological paradigm".

> I don’t see anything that would even point into that direction.

I find it a kind of baffling that people claim they can't see the problem. I'm not sure about the risk probabilities, but at least I can see that there clearly exists a potential problem.

In a nutshell: Humans – the most intelligent species on the planet – have absolute power over any other species, specifically because of our intelligence and the accumulated technical prowess.

Introducing another, equally or more intelligent thing into equation is going to risk that we end up with _not_ having the power over our existence.

I think it's very fortunate, because I used to be an AI doomer. I still kinda am, but at least I'm now about 70% convinced that the current technological paradigm is not going to lead us to a short-term AI apocalypse.

The fortunate thing is that we managed to invent an AI that is good at _copying us_ instead of being a truly maveric agent, which kinda limits it to the "average human" output.

However, I still think that all the doomer arguments are valid, in principle. We very well may be doomed in our lifetimes, so we should take the threat very seriously.

It's interesting that whereas old kango are Chinese loanwords, many newer ones are made-up words, and some even got backported into the Chinese language!

Dijkstra _could_ be universally taught in 7th grade if we had the curriculum for that. Maybe I'm biased, but it doesn't seem conceptually significantly more difficult than solving first degree equations, and we teach those in 7th grade, at least in Finland where I'm from.

The problem with computables is that equivalence between them is only semi-decidable. (If the two numbers are different, it is decidable, but if they are not, it isn't. The problem is that you don't know if they are different a priori, so you might get lucky and find difference, but you might as well not.)

We know for sure that algebraic numbers behave nicely in terms of equivalence, and there are other, bigger number systems that are conjectured to behave nicely ( https://en.wikipedia.org/wiki/Period_(algebraic_geometry) ), but the problem with these and computers is that they are hard to represent.

Supposed by who? Maybe it's supposed to be 柔術.

The bank I used to use had a per-verification request code that the app showed. If the party dealing with you knew the code, you could be sure they were the party who initiated the verification request.

Huh, I was thinking that it's accelerated gravitational frames that cause the dilation, and I've encountered a lot of statements that argue the same. This is from wikipedia: "This is because gravitational time dilation is manifested in accelerated frames of reference or, by virtue of the equivalence principle, in the gravitational field of massive objects."

However, according to that logic, an object located in a cavity in the center of earth should experience no more dilation than an object outside the earth's potential well, because the gravitational forces / curvature gradient cancels out, and should be zero. But that isn't the case according to the same sources, for example, Wikipedia says' "Relative to Earth's age in billions of years, Earth's core is in effect 2.5 years younger than its surface."

Something's not right about how we verbalize this story about gravity

To make this even clearer:

Let's imagine that there is a huge amount of time dilation (we live on the surface of a neuron star or something). By climbing a bit, we experience 1.1 seconds instead of 1.0 seconds experienced by someone who left down.

We have a clock that can measure milliseconds as the smallest tick. But climbing up, back down, and comparing the amount of ticks won't let us conclude anything after a single millisecond. If anything, we must spend at least 11 milliseconds up to have a noticeable 11 to 10 millisecond difference.

Now, if the dilation was 1.01 seconds vs 1.00, we would need to spend at least 101 milliseconds up, to get a minimal comparison between 101 and 100 milliseconds.

According to ChatGPT, the speedup factor for getting 10 cm higher is 1 + 1.09e−17. (With ΔT = gh /(c^2) The math seems to check out, but not sure if the formula itself is correct.) Surely, if the clock ticks at rate 1e-19 in a second, i.e. one tick is hundred times smaller than the dilation difference in a second, the clock would still need at least a hundreth of a second to accumulate enough ticks for the count of ticks to differ even by one tick because of the dilation.

I've been to many snack bars, and once in my almost ten years in Japan, I found one that didn't welcome foreigners.

Agreed, I said 95% just to be generous, should've mentioned that.

> TBAA, like C has by default, generally is easier than just no aliasing [note: I assume no aliasing here to refer Rust's mutable references, not C's restrict]

I don't accept that statement. Rust's mutable references are statically checked, so they are impossible to get wrong (modulo compiler bugs) if you aren't using unsafe code. Using raw pointers has no no-aliasing requirements, so again, it's easier than in C.

The hard part is mixing mutable references and raw pointers. In 95% of Rust code, you are not required to do that, and you shouldn't do that. In the remaining 5%, you should understand the aliasing model. In that case, indeed, you need to know more than what TBAA requires you to know. But that's for the case where you can also _DO_ more than TBAA would allow you to do.

The situation is not that bad. The rules of unsafe code were pretty badly defined back then, but they are in process of becoming a lot clearer, and like the grandparent argues, with a well-defined aliasing model like Tree Borrows, they are easier to understand than C's.

If you look into the code you linked, the problem was about accessing undefined bytes though an aliased, differently-typed pointer – something you would have hard time doing in C to begin with. MaybeUninit was a new thing back then. I think that nowadays, a senior Rust developer would clear the hurdles better.

> Unless I'm just misunderstanding the point they're trying to make here.

You misunderstand the word "can". Yes, you can, in unsafe code, do that. And yes, that is undefined behaviour ;)

https://play.rust-lang.org/?version=stable&mode=debug&editio...

However, in this case, we are not talking about a painting, but a woodblock print. (Not disagreeing with your point in general, though.)

And in East Asia, we use a system based on exponents of 10000. I kind of like it, except when I have to think about it and the short/long scales at the same time.

10000^1 = 万 10000^2 = 億 10000^3 = 兆 10000^4 = 京

Thanks for the inline dictionary lookup, I TIL'd that both of those words were popularized by the Simpsons.

Ah, the Saint Hannes cross, or sankthanskors in Sweden, or hannunvaakuna in Finland. It's not so much related to campgrounds, but to mark sightseeing spots in general.

I have written very little Zig and a lot of Rust, but I love both languages. However, Zig having breaking changes has made me wary of not starting anything serious it with – yet. I'm still happy that these changes happen, because I'm willing to wait for a stable version. Meanwhile, I enjoy myself some Rust, and probably continue doing so.

While I agree that good automatic translations are great, the fact that the current unconfigurable mess that doesn't even respect the user's settings (neither the settings of their Google Account nor the settings of their browser), makes it a grating experience to multilinguals, who are believed to be the majority in the world.

Yes, legal trolling is unfortunately a thing.

Which is crazy, because the shapes are just an idea, and you can't copyright ideas. They also couldn't be a trademark infringement (IANAL, though) if you don't use them in your branding.

(To share a recent personal experience about Markov models: I bootstrapped recently a HMM with hand-assigned weights. It was around 15x15 class transitions, 225 weights. That's small. Or rather, microscopic. Then I ran it against real data, and picked up examples of wrong classifications, and made them auxillary training data. Of course, it was not a language model, language model is impossible to fit in such a small space. It was a model of transitions of chapter "types" in novels, where types are something like "Epilogue" , "Prologue", "Chapter 23", "Table of Contents", "Afterword" etc.)

A traditional Markov model trained (rather, just "fitted") on tokens or words is a small language model.

Re: Prompting to ask. Thanks, I'll try that. And I'm gonna try version 4 as soon as I can.

Two problems:

1) Writing a high-performance memory allocator for a game engine in Rust: https://github.com/golddranks/bang/tree/main/libs/arena/src (Still work in progress, so it's in a bit messy state.) Didn't seem to understand the design I had in mind, and/or the requirements and goes on tangents and starts changing the design. In the end, coded the main code myself and used LLM for writing tests with some success. Had to remove tons of inane comments that didn't provide any explanatory value.

2) Trying to fix a Django ORM expression that generates unoptimal and incorrect SQL. Constantly changes opinion whether something is even possible or supported by Django, apologizes when pointing out mistakes / bugs / hallucinations, but then proceeds to not internalize the implications of the said mistakes.

I used the Zed editor with its recently published agentic features. I tried to prompt it with a chat style discussion, but it often did bigger edits I would have liked, and failed to share a high-level plan in advance, something I often requested.

My biggest frustrations were not coding problems per se, but just general inability to follow instructions and see implications, and lacking the awareness to step back and ask for confirmations or better directions if there are "hold on, something's not right" kind of moments. Also, generally following through with "thanks for pointing that out, you are absolutely right!" even if you are NOT right. That yes-man style seriously erodes trust in the output.