montebicyclelo

Yeah... Looks like can get about $1/hr for 10 small VMs, ($0.10 per VM).

So for $3000, that's 3000 hours, or 125 days, (if just wastefully leave them on all the time, instead of turning them on when needed).

Say you wanted to play around for a couple of hours, that's like.. $3.

(That's assuming there's no bonus for joining / free tier, too.)

I stopped using apple's notes app with an ipad pen after it lost 20 minutes of my handwritten notes when trying to sync them. (Which fits the theme of apple losing people's stuff.)

I think OP is suggesting that Apple / AMD / Intel do the work of integrating their NPUs into popular libraries like `llama.cpp`. Which might make sense. My impression is that by the time the vendors support a certain model with their NPUs the model is too old and nobody cares anyway. Whereas llama.cpp keeps up with the latest and greatest.

Although note — people are likely to be infuenced by the recent prevalence of em dash to use it more in their own writing nowadays

> the Python code in the previous example is still readable

Yes, I agree with the author, list comprehensions are readible, and I'd add, practical.

> it gets worse as the complexity of the logic increases

    len(list(filter(lambda line: all([abs(x) >= 1 and abs(x) <= 3 for x in line]) and (all([x > 0 for x in line]) or all([x < 0 for x in line])), diffs)))

Ok, well this is something that someone would be unlikely to write... unless they wanted to make a contrived example to prove a point.

It would be written more like:

    result = sum(my_contrived_condition(x) for line in diffs)

See also the Google Python style guide, which says not to do the kind of thing in the contrived example above: https://google.github.io/styleguide/pyguide.html

(Surely in any language it's possible to write very bad confusing code, using some feature of the language...)

And note:

    x = [line.split() for line in text.splitlines()]

^- list comprehension is just a convenient shorthand for a `for loop`, i.e.:

    x = []
    for line in text.splitlines():
        x.append(line.split())

Just moving the `line.split()` to the front and removing the empty list creation and append.

So, they are included in the normal backpropagation of the model. But there is no one-hot encoding, because, although you are correct that it is equivalent, it would be very inefficient to do it that way. You can make indexing differentiable, i.e. gradient descent flows back to the vectors that were selected, which is more efficient than a one-hot matmul.

(If you're curious about the details, there's an example of making indexing differentiable in my minimal deep learning library here: https://github.com/sradc/SmallPebble/blob/2cd915c4ba72bf2d92...)

Nice tutorial — the contextual vs static embeddings is the important point; many are familiar with word2vec (static), but contextual embeddings are more powerful for many tasks.

(However, there seems to be some serious back-button / browser history hijacking on this page.. Just scolling down the page appends a ton to my browser history, which is lame.)

I always found that badly named things did make learning harder / more jarring; especially if an explanation for the incongruous name wasn't provided.

> if the weights are taken from HF then what's the implementation for

The weights are essentially a bunch of floating point numbers, (grouped into tensors). The code says what operations to do with the weights. E.g. say you load matrix W from the weights, you could do `y = W @ x`, or `y = W.T @ x`, or `y = W @ W @ x` etc.

TLDR, unexpected blank fields

> policy change was inserted into the regional Spanner tables

> This policy data contained unintended blank fields

> Service Control... pulled in blank fields... hit null pointer causing the binaries to go into a crash loop

Forced password rotation and expiry seems the bigger problem; given that it causes people to get locked out so often, (e.g. if pw expires when on holiday), — often then requiring travelling to IT, or at least a few hours trying to get IT on the phone to reset, or chasing up colleagues who aren't locked out to get in touch with IT.

Many (most?) companies still do it, despite it now not being recommended by NIST:

> Verifiers SHOULD NOT require memorized secrets to be changed arbitrarily (e.g., periodically)

https://pages.nist.gov/800-63-3/sp800-63b.html

Or by Microsoft

> Password expiration requirements do more harm than good...

https://learn.microsoft.com/en-us/microsoft-365/admin/misc/p...

But these don't seem to be authoritative enough for IT / security, (and there are still guidelines out there that do recommend the practice IIRC).

I guess where it could potentially bring value is by:

Removing a major concern that prevents individuals / small customers from using GCP in the first place; so more of them do use it

That could then lead to value in two ways:

- They make small projects that go on to be large projects later, (e.g. a small app that grows / becomes successful, becomes a moneymaker)

- Or, they might then be more inclined to get their big corp to use GCP later on, if they've already been using it as an individual

But that's long term, and hard to measure / put a number on

Not providing a cap on spending is a major flaw of GCP for individuals / small projects.

With Cloud Run, AFAIK, spending can effectively be capped by: limiting concurrency, plus limiting the max number of instances it can scale to. (But this is not as good as GCP having a proper cap.)

Reason Cloud Run is so nice compared to other providers is that it has autoscaling, with scaling to 0. Meaning it can cost basically 0 if it's not being used. Also can set a cap on the scaling, e.g. 5 instances max, which caps the max cost of the service too. - Note, I only have experience with the CPU version of Cloud Run, (which is very reliable / easy).

The industry standard has been Poetry for a good few years now, and UV is the newer exciting tool in this space. Both create universal lockfiles from more loosely specified dependencies in pyproject.toml resulting in reproducible environments across systems, (they create isolated Python environments per project).

> The difference between spoilage and fermentation/aging is whether humans like the result at the end

Is not whether it makes humans unwell quite an important factor..

Not sure "original concept" is quite right, given it had been tried earlier, e.g. here's a 2023 attempt to get gpt-4-vision to play pokemon, (it didn't really work, but it's clearly "the concept")

https://x.com/sidradcliffe/status/1722355983643525427

Related to ways of understanding neural networks, I've seen these views expressed a lot, which to me seem like misconceptions:

- LLMs are basically just slightly better `n-gram` models

- The idea of "just" predicting the next token, as if next-token-prediction implies a model must be dumb

(I wonder if this [1] popular response to Karpathy's RNN [2] post is partly to blame for people equating language neural nets with n-gram models. The stochastic parrot paper [3] also somewhat equates LLMs and n-gram models, e.g. "although she primarily had n-gram models in mind, the conclusions remain apt and relevant". I guess there was a time where they were more equivalent, before the nets got really really good)

[1] https://nbviewer.org/gist/yoavg/d76121dfde2618422139

[2] https://karpathy.github.io/2015/05/21/rnn-effectiveness/

[3] https://dl.acm.org/doi/pdf/10.1145/3442188.3445922

> then go on to explain how they aren't useful in any field whatsoever

> Where are the AI-driven breakthroughs

> are we just using AI to remix existing general knowledge, while making no progress of any sort in any field using it?

The obvious example of a highly significant AI-driven breakthrough is Alphafold [1]. It has already had a large impact on biotech, helping with drug discovery, computational biology, protein engineering...

[1] https://blog.google/technology/ai/google-deepmind-isomorphic...

Language vocab seems a good use case. What other things are people here using spaced repition for?

> The results we obtained in Section 7 imply that, at least on simple datasets like TinyStories and Wikipedia, LLM predictions contain much quantifiable structure insofar that they often can be described in terms of our simple statistical rules

> we find that for 79% and 68% of LLM next-token distributions on TinyStories and Wikipedia, respectively, their top-1 predictions agree with those provided by our N-gram rulesets

Two prediction methods may have completely different mechanisms, but agree sometimes, because they are both predicting the same thing.

Seems a fairly large proportion of language can be predicted by a simpler model.. But it's the remaining percent that's the difficult part; which simple `n-gram` models are bad at, and transformers are really good at.

Edit: increased the validation to 10,000 life grids for 100 steps, (taking 16 minutes to check), which is hopefully somewhat more convincing. That's 1,000,000 life steps computed without errors in total. Plus 32,000 steps computed without error during training.

When the attention grid is manually computed (to be equivalent to 3 by 3 conv), the model can be trained to be 100% perfect, verified by checking all 3 by 3 grid states. (And this manually computed attention matrix means that once the tokens reach the classifier layer, each token contains only the information of the relevant 3 by 3 grid, and the whole thing is deterministic as you say.)

However, when the model is computing the attention grid itself, just checking all 3 by 3 sub-grid states crop up is not enough, because the position of the sub-grids can impact the attention matrix, and also the state of other cells can impact the attention matrix. So as shown in the post, it does approximate 3 by 3 conv, but if it doesn't get the approximation quite right, there could be errors. But I would say that it's still computing the Game of Life algorithm in an interpretable way, it's just that maybe it has struggled to create a perfect 3 by 3 convolution via attention in that particular case. (To exhaustively check this, would require checking all 2 * (16x16) grids.)

> This is such an insane statement.

In what way? Maybe you mean something different when you say computing the game of Life algorithm.

> I really didn't want to root the phone, but nothing else did what I needed

Shame that rooting is such a pain, and risks bricking the device. (Apparently Google's introduction of an anti-rollback bootloader this month has caused a few people's devices to get bricked when they tried to root.)

Hmm, just my intuition: training this model was very sensitive to the initial seed and training hyperparameters. It struggles to actually get to the 3x3 conv solution; but once it gets close to that things move much more quickly. This can kind of be seen in the animation of the attention matrix over time, which starts off random / spread out, but then once it starts to get more parts of the attention matrix in place it moves quicker. (Assuming all the experimentation wasn't in some bad part of the hyperparameter space.)

Also, it may just be the nature of the task. Some tasks you might have more to learn, all the time, with each training sample potentially giving information that's different from all the others. But with this, once it gets close to the solution of Life, it's quick.

> the paper does not investigate whether the resulting NN actually performs the game of life algorithm

How could it not be computing the game of life algorithm? Given that it gets 100% accuracy over multiple steps on a bunch random game boards it's never seen before.

And then based on the structure of the net, and by examining the attention layers and finding that it's doing 3 by 3 average pooling, we can see that the attention layer produces a set of tokens, where each token contains the information of the number of neighbours it had, and its previous state. This then goes through a classifier layer, which decides it's next state, given that information.

Further evidence for that: it was possible to use linear probes to confirm that the tokens that had been through the attention layer contained the information about the number of neighbours and the previous state.

From all of this, it's clear that the model is running the Game of Life properly.

it won't AFAIK, without some extra hand coded logic

It wasn't obvious that a transformer could do this, and learn to produce conv via attention

> it's brute forcing a small grid

If by small grid you are referring to the attention matrix plot shown, then that is not a correct interpretation. That diagonal-like pattern it learns, is 3x3 convolution, so it can compare the neighbours of a given cell.

Edit: and note that every grid it is trained on / runs inference on is randomly generated and completely unique, so it cannot just memorise examples

The diagonal-looking attention matrix shown in the post is mathematically equivalent to 3 by 3 convolution. The model learns how to do that via its attention mechanism - it's not obvious that it would be able to do that via attention.

(This can be shown by comparing that attention matrix to a "manually computed Neighbour Attention matrix", which is known to be equivalent to 3 by 3 conv.)