tylerneylon

I think these two things could be improved from that answer:

* I'm suggesting a general approach to measuring track tension, which is the average of the vectors. I didn't see that idea in the answer.

* I think the answer could be communicated a little more simply. For example, we don't need to think in terms of Q[sqrt(3)]; I see that as a distraction.

View each track piece as a 2d vector. Add up the vectors. In a zero-tension setup, the sum is (0, 0).

As a metric for tension, assume any mismatch in position is evenly distributed. Model this as the average of all the vectors. (Thus the same displacement is more meaningful when we have fewer pieces.)

___

That's the full idea. It might seem that it is ignoring rotation, because it doesn't explicitly mention rotations, but they are included because the effective vector that a track piece provides is both a current direction as well as the displacement contributed by that piece. If we wrote some code to model this, a cursor would consist of a direction (an angle) along with an (x, y) position.

___

Some related math concepts:

* The exterior angles of a polygon sum to 360. So we could have another measure which is how far we are away from 360.

* Not useful in this case, but this also reminds me of winding numbers from complex analysis, which is a way to locally walk along a curve to understand which side is the "inside" or how many times a curve goes around a given point.

Machine learning engineer, PhD, 15 years experience

I'd love to work together on a contract basis and give you the AI/LLM-based solution you want, with the understanding that on my side, I'm also creating a related software product. I'm building a new product that integrates custom data with an LLM. The goal is to empower plain English queries of whatever crazy (or sane) data sources you have, such as pdfs, confluence docs, source code, SQL databases, or external web pages.

Looking forward to talking more :)

https://www.linkedin.com/in/tyler-neylon/

tyler@unboxresearch.com

Mathematically, it’s possible to have a differential equation with an initial condition, and still have multiple different solutions. That’s what the author creates as well.

I don’t think this article is about continuous vs discrete physics, as some commenters suggest - nor about reality at all (not directly). Rather, this is pointing out a surprising property of the model of Newtonian mechanics.

I had a hard moment on the gamedev stackexchange where I was stuck trying to learn how to do something in OpenGL. A moderator immediately closed my question as a duplicate because there was a similar question about OpenGL ES, which is a (related but) different API. I tried to plead my case, but was shut down.

Shortly after that, I gave up on the game I'd been working on for a couple years. The mod's decision contributed to that.

I felt stuck by a wall between me and answers to some of my game programming questions. Over-moderation is more than an inconvenience. It can destroy the ability of users to get things done.

Mathematically, we can write v_out = V * w,

where v_out is the vector of output from the attention unit, w is the probability vector from the softmax, and V is the set of input vectors, where each column is an input vector.

For a moment, pretend that the columns of V are orthonormal to each other. This might not be true, but it's an interesting case.

When the model wants the output to be small, it can set w = 1/n, meaning all coordinates of vector w are 1/n. (n = the number of columns in V)

In that case, the length ||v_out|| will be 1/sqrt(n) exactly, which is small compared to the input lengths of 1 (since we're pretending they were orthonormal).

Now if we stop pretending they are orthonormal, the worst case is that they're all the same vector, in which case the weights w can't change anything. But that's a mighty weird case, and in high dimensions, if you have any randomness at all to a set of vectors, they tend to point in wildly different directions with dot products close to zero, in which case the same intuition for the orthonormal case applies, and we'd expect a uniform distribution coming out of the softmax to give us a vector that's much smaller than any of the input vectors.

The author is suggesting that we add 1 to the denominator of the softmax that is used within attention mechanisms (not the final output softmax).

The softmax inside an attention unit allows it to see key/query matches as probabilities; those probabilities support a continuous-valued version of a key-value lookup (instead of 1/0 output of a lookup, we get weights where a high weight = the desired key-value lookup).

Adding 1 to the denominator would change an attention unit by no longer working with a true probability vector of weights, but rather working with weights that add up to less than 1. The motivation is that the network can learn to provide high weights so that the adjusted softmax is very close to a probability vector; and it has a new option to provide all-low weights which give all-low output weights, meaning it can opt out of having high confidence in anything.

(switching to opinion mode)

2. How can we tell if this is good?

2a. We should just try it out: Train an LLM with this, see if it works.

2b. There are two reasons I suspect it won't make a big difference.

First, if an attention node has low confidence, it can already assign similar scores pre-softmax. Then we get what looks like a uniform distribution as output. Then we're basically taking an average of a bunch of vectors (vs a weighted average that is more like choosing one of them). Statistically, we expect that averaged vector to be close to zero. In other words, the node already has a way to effectively opt-out by providing a near-zero output vector.

Second, in a transformer, each attention unit has many other learned weights that can support the ability to opt out. Both the V matrix and the feed-forward layer after the attention unit give that module a way to provide low values to the activation function after the feed-forward layer, which would result in a value as small as you like — again, a way to opt out.

3. I appreciate the non-academic tone of the article and the willingness to play around with fundamental ideas. Although I'm not totally convinced by the note, I'd love to read more stuff like this.

The most popular article is "Learn Lua in 15 Minutes." I write about code, math, and philosophy; eg, neural networks, permutations, or consciousness.

I also run a weekly machine learning / AI newsletter:

https://learnandburn.ai/

Suppose we have an n-dimensional pyramid. It has an (n-1)-dim'l base with volume B, and that base tapers to a point; for simplicity, it has height 1.

Take cross-sections as we travel from the base to the height. How big is a cross section at height y?

Each cross section is a (n-1)-dim'l shape, and its volume is B * y^(n-1) because this is how volumes scale in dimension n-1. [Examples: In 2D if you x2 the sides of a shape, it's "volume" (area) is x4. In 3D if you x2 the sides of a shape, it's volume is x8; in 4D it would be x16, etc.]

Now take the integral to add all of these cross-sections:

integral(from 0 to 1 of B * y^(n-1)) = B * y^n / n evaluated from 0 to 1 = B / n.

If the shape is scaled along the height dimension by h, then we get the more general formula:

volume = h * B / n.

So, if b(n) is the base in b^n=n!, then b(n) is about (n/e). [I see this as the key observation of the article.]

Keeping in mind that (n/e)^n is a bit smaller than n!, a more accurate value for b(n) is a little higher than n/e.

So it makes sense that 25! ~= 10^25 because 25/e is a bit below 10, and the adjusted value for b(n) ought to be a little higher than 25/e; 10 will do in this case.

(And 270! ~= 100^270, etc.)

10000 - x = 1111 - x + 1 = not(x) + 1

In other words, not(x)+1 in binary as the same as subtracting x from the next higher power of two.

Lukas_Skywalker's fun note about doing this in base 10 made me realize that.

Sorry if it was there and I missed it. Thanks!

Just in case you happen to know the answer: How does taking tree samples (as in your third link) not harm the tree? It seems inevitable that it would do so, at least intuitively.

Although I don't have access to the proprietary code used, it's most likely that an LSH algorithm is behind the scenes in every modern search engine (to avoid serving duplicates), many modern ranking systems such as Elasticsearch (because items are typically vectorized and retrieved based on these vectors), and most recommendation systems (for similar reasons as ranking). For example, all of these pages probably have an LSH algorithm at some point (either batch processing before your request, or in some cases real-time lookups):

* Every search result page on Google * Every product page on Amazon (similar products) * All music suggestions on Spotify or similar * Every video recommendation from TikTok, YouTube, or Instagram

etc.

Z is the set of integers {.., -2, -1, 0, 1, 2, ...}.

Z is called "Z" due to influential German mathematicians who chose the letter to stand for "zahlen" ("number" in German).

As a fun fact, the letter "N" stands for "natural numbers," meaning either positive integers, or positive integers and zero. However, that letter (N) is _not_ standardized because mathematicians disagree about whether or not to include zero. When I separately told a couple of my professors this in grad school, they didn't believe me — one of them thinking it was crazy for zero to be excluded, the other thinking it was crazy to include it!

https://en.wikipedia.org/wiki/Natural_number

My original motivation for writing this was to explain (in a follow-up article) an ANN algorithm I published in 2010: https://epubs.siam.org/doi/pdf/10.1137/1.9781611973075.94

I never wrote that up (as a non-academic article), but the fact that people like this article is a nudge for me to do that.

The above paper was the first LSH algorithm which deterministically returns all close-enough points and guarantees no false positives for distant-enough points. I've heard there are also other algorithms with that property published since. If I have time, I would like to one day see how that algorithm compares against the others in Erik Bernhardsson's benchmarks.

It's about one possible future of our relationship with AI. It paints multiple perspectives on this relationship while being implicitly judgmental. I'm being vague because the story reveals itself slowly, and the blossoming of information is itself a work of art -- this unfolding is part of Ishiguro's mastery.

Most people feel that AI currently has zero emotions, and perhaps a near-future version may convince some people that it has _some_ kind of emotions, though these emotions almost certainly will feel wrong, maybe-invalid, and strange to us, merely by them being different and by our awareness that an AI is not human. This and other questions are explored in Klara and the Sun.

I interpret your criticism to mean that ML folks tend to re-use a test set multiple times without worrying that doing so reduces the meaning of the results. If that's what you mean, then I do agree.

Informally, some researchers are aware of this and aim to use a separate validation data set for all parameter tuning, and would like to use a held out test set as few times as possible — ideally just once. But it gets more complicated than that because, for example, different subsets of the data may not really be independent samples from the run-time distribution (example: data points = medical data about patients who lived or died, but only from three hospitals; the model can learn about different success rates per hospital successfully but it would not generalize to other hospitals). In other words, there are a lot of subtle ways in which a held out test set can result in overconfidence, and I always like to learn of better ways to resist that overconfidence.

Ok, I fully agree with their foundational premise: Start simple.

But: They've overstated their case a bit. Saying that deep learning will cost $11,000 and need 14 days on this data set is not reasonable. I believe you can find some code that will cost that much. The readme suggests that this is typical of deep learning, which is not true. DL models have enormous variety. You can train a useful, high-performance model on a laptop CPU in a seconds-to-minutes timeframe; examples include multilayer perceptrons for simple classification, a smaller-scale CNN, or a collaborative filtering model.

While I don't endorse all details of their argument, I do think the culture of applied ML/data science has shifted too far toward default-DL. The truth is that many problems faced by real companies can be solved with simple techniques or pre-trained models.

Another perspective: A DL model is a spacecraft (expensive, sophisticated, powerful). Simple models like logistic regression are bikes and cars (affordable, efficient, less powerful). Using heuristics is like walking. Often your goal is just a few blocks away, in which case it would be inefficient to use a spacecraft.

1. Summarizing papers:

Here are some examples of paper summaries: https://twitter.com/unboxresearch

I wrote the software that uses GPT-3 to produce those. You can look at the papers to judge the accuracy for yourself.

2. Solve math problems:

What is the area of a circle with radius 2?

The area of a circle with radius 2 is 4π.

3. Generate wiki articles:

Write the first paragraph of a wiki article about tigers.

Tigers are the largest members of the cat family. They are apex predators, primarily preying on ungulates such as deer and bovids. They are territorial and generally solitary but social animals, often requiring large contiguous areas of habitat that support their prey requirements. This, coupled with the fact that they are indigenous to some of the more densely populated places on Earth, has caused significant conflicts with humans.

4. Write scientific code:

Write a matlab function to plot the largest three singular values of a matrix.

function plotLargestSingularValues(A)

s = svd(A);

plot(s(1:3));

I'm not trying the others because I think they're more specific to Galactica.

The models, including GPT-3, absolutely do make mistakes. We are in the first few years of these capabilities existing at this level, so much research is needed before the results are consistently good. In my experience, though, GPT-3 is extremely useful and reliable for many use cases. (Building and using tools like this is part of my career.)

I read the MIT Technology Review article, and I was asking myself “what is an example of Galactica making a mistake?” The article could easily have quoted a specific prompt, but doesn’t. It says the model makes mistakes in terms of understanding what’s real/correct or not, but the only concrete example I see in the article is that the model will write about the history of bears in space with the implication that it’s making things up (and I believe the model does make such mistakes). I don’t think it’s a good article because it’s heavy on quoting people who don’t like the work and light on concrete details.

Does the imperfection of a language model really mean the model should not exist? This seems to be what some critics are aiming for.

My own takeaway from the data is a bit different from what the authors say conversationally (vs in the figures). I’d summarize the paper, mostly based on figure 5b, as:

Try to be happy, old, sleep well, and don’t eat too frequently.

In terms of breakfast, the data’s main conclusions are to avoid a lot of sugar (which is hugely important), and to wait a while before eating breakfast (see fig 3).

I see why you mention 1-based indexing, but it’s mostly a different beast. I’ll defend 1-based indexing here because, having used Lua a lot, the indexing issue just goes away.

People like what they’re used to. Most coders are used to 0-based, but we all start life 1-indexed. We begin counts with 1: chapter 1, the 1st floor of a building (in the US), 1 AD, the 1st of Dec, etc.

In C/C++, 0-based makes sense because we often care about pointers and pointer arithmetic. But in languages without pointers, that reason goes away. And you can still use half-open intervals with 1-based indexing.