montebicyclelo

The author has experience of teaching box plots in various organisations.

The author has found that compared to other types of plots, people struggle to learn how to intepret box plots.

The author proposes some alternatives that they believe to be easier for people to interpret:

- Strip plots (for few data points)

- Jittered strip plots (for more data points)

- Distribution heatmap (for even more data points)

----

This aligns with my experience of trying to convey information to non-technical or moderately technical people; box plots are a struggle for them. To me it does seem like the proposed alternatives would be more accessible.

Sure, we could try to better educate people about box plots, (as the author has done professionally); or we could consider using something that requires less effort for people to comprehend.

Related side note: I've always had a suspicion that one of the ideas I was working on using Google Colab was viewed by an employee and leaked, because someone wrote a blog post with the exact same idea (very niche) before I got round to releasing mine (I ended up not bothering due to being gazumped), and a Google Colab employee tweeted that blog post. (Puts on tin foil hat.. I stopped using Colab after that.)

Yep interesting that massively useful and impactful work like this doesn't seem to get as much interest on HN compared to gimicky new architectures that have only been shown work well only on tiny datasets like MNIST and CIFAR, rather than at scale.

IAgreeThatSnakeCaseIsEasierToRead,ThoughIfWeWereToUseCamelCaseMoreWeWouldBeSavingAGoodAmountOfSpace,WhichMightBeUsefulForNarrowBlocksOfText.I'mSuprisedItHasn'tCaughtOnInProse.Actually,LookingAtThis,MaybeIt'sAGoodThingItHasn'tCaughtOnForProse.

This seems likely. I wonder if explaining it through the packaging, e.g. coloured stripe marked "stir space" and an explanation on the back saying don't worry you are only paying per gram of peanut butter...

The introduction / training could be made clearer with some GIFs / animations showing what to press and the effects.

y = x @ W + b

Is how you'd write it in NumPy and most (/all?) deep learning libraries, with x.shape=[batch, ndim]

I'd personally prefer that to be the convention, for consistency.

From the YC guidelines:

> One good trick for describing a project concisely is to explain it as a variant of something the audience already knows. It’s like Wikipedia, but within an organization. It’s like an answering service, but for email. It’s eBay for jobs. This form of description is wonderfully efficient.

Seems the risk of this is a loose simile.

Edit: Though, thinking about it, Airbnb for GPUs is fairly accurate for this model of a marketplace where people let out their GPUs for others to rent.

The success we're seeing with neural networks is tightly coupled with the ability to scale - the algorithm itself works at scale (more layers), but it also scales well with hardware, (neural nets mostly consist of matrix multiplications, and GPUs have specialised matrix multiplication acceleration) - one of the most impactful neural network papers, AlexNet, was impactful because it showed that NNs could be put on the GPU, scaled and accelerated, to great effect.

It's not clear from the paper how well this algorithm will scale, both in terms of the algorithm itself (does it still train well with more layers?), and ability to make use of hardware acceleration, (e.g. it's not clear to me that the structure, with its per-weight activation functions, can make use of fast matmul acceleration).

It's an interesting idea, that seems to work well and have nice properties on a smaller scale; but whether it's a good architecture for imagenet, LLMs, etc. is not clear at this stage.

It does seem to be the norm for interesting things to be presented in a boring way, rather than the exception. I'd guess this is partly because it's difficult to explain concepts clearly in the first place, let alone make the explanation engaging...

Hypothesis: If writing educational material is like throwing a dart, you have a bigger target if you aim for just "clear" — if you aim for "clear ∩ engaging" you have less chance of being "clear".

    size(clear ∩ engaging) < size(clear) 

I think time pressure would come into play with this. If you have the time, you can hone in on the "clear ∩ engaging" zone; but if you are pressed for time you just aim for "clear" which you'll hit more quickly.

And if you're really pressed for time (and/or uncaring), you might aim for "passable" rather than "clear"; something that your colleagues would OK, but that isn't particularly great. This is a larger target than "clear".

    size(passable) > size(clear)

It's fairly easy to use, not that compute intensive (e.g. can run on even a small-ish CPU VM), the embeddings tend to perform well and you can avoid sending your data to a third party. Also, there are models fine tuned for particular domains on HF-hub, that can potentially give better embeddings for content in that domain.

Cosine-similarity is a useful metric. The cases where it is useful are models that have been trained specifically to produce a meaningful cosine distance, (e.g. OpenAI's CLIP [1], Sentence Tranformers [2]) - but these are the types of models that the majority of people are using when they use cosine distances.

> It's my understanding that the delta between two word embeddings, gives a direction, and the magic is from using those directions to get to new words... it's the directions and distances to nearby objects that matters most

Cosine similarity is a kind of "delta" / inverse distance between the represenation of two entities, in the case of these models.

[1] https://arxiv.org/abs/2103.00020

[2] https://www.sbert.net/docs/training/overview.html

Hmm, typically the models where people use cosine similarity on embeddings have been deliberately trained such that the cosine similarity is meaningful. It looks like this paper is looking at examples where the models have not been deliberately trained for the cosine similarities, and hence in these situations it would indeed be unreasonable to assume cosine similarities to be a good idea.. (but that's kind of a given?)

For example, here's the loss from the CLIP paper [1], which ensures cosine similarities are meaningful:

    # joint multimodal embedding [n, d_e]
    I_e = l2_normalize(np.dot(I_f, W_i), axis=1)
    T_e = l2_normalize(np.dot(T_f, W_t), axis=1)
    # scaled pairwise cosine similarities [n, n]
    logits = np.dot(I_e, T_e.T) * np.exp(t)
    # symmetric loss function
    labels = np.arange(n)
    loss_i = cross_entropy_loss(logits, labels, axis=0)
    loss_t = cross_entropy_loss(logits, labels, axis=1)
    loss = (loss_i + loss_t)/2

And Sentence Transformers [2] using CosineSimilarityLoss:

    train_loss = losses.CosineSimilarityLoss(model)
    # Tune the model
    model.fit(train_objectives=[(train_dataloader, train_loss)], epochs=1, warmup_steps=100)

[1] https://arxiv.org/pdf/2103.00020.pdf

[2] https://www.sbert.net/docs/training/overview.html

Nice, here's an attempt to compute the HN-index for all users. (Differs from blog post; not sure if because the data from the two sources is different, or there's a mistake somewhere):

    WITH Ranking AS (
        SELECT 
            by,
            score,
            RANK () OVER (PARTITION BY by ORDER BY score DESC) AS rank
        FROM hackernews_history
    )
    SELECT by, MIN(rank - 1) as h_index
    FROM Ranking
    WHERE rank > score
    GROUP BY by
    ORDER BY h_index DESC
    LIMIT 100

It appears to be working for me:

    SELECT count(), min(time), max(time) FROM hackernews_history

39838828, 1970-01-01 00:00:00, 2024-03-11 13:04:47

The open Google Bigquery table of Hacker News might possibly be a nice way to get this data. (Although last time I checked it was a few years out of date, so maybe not... Hoping that someone reads this and starts updating it again...)

Lineage:

- Peter Norvig (original (?)) - http://norvig.com/21-days.html#answers

- Jeff Dean (slides) - https://www.cs.cornell.edu/projects/ladis2009/talks/dean-key...

- Colin Scott - https://github.com/colin-scott/interactive_latencies?tab=rea...

- this post

Thanks, I do agree with this, and I am rooting for them. Would love to find out at some point that it's turned around.

Guessing there are a few of us in a position where we are frustrated with our past experiences with ROCm software, (e.g. being awful to install, in the past there were long guides with loads of steps to follow, and not the greatest clarity / simplicity in the instructions, and the only option they suggested if you messed it up was to reinstall the operating system, and there were multiple guides / pages and it wasn't clear which was the latest one, and then it only supported older versions of tensorflow / pytorch / jax, and only supported recent / higher end cards, etc. — it may be better now, this is my experience from I guess a few years ago); but who at the same time recognise that it would be great for GPU compute to be more affordable, and for there to be good competitors to Nvidia.

This addresses a real pain point, of Numba not being so easy to profile, (especially compared to Python and line-profiler), so will definitely be trying this out!

There's an example here of using Numba to speed up data science code, (disclaimer, I'm the author): https://sidsite.com/posts/python-corrset-optimization/

> for most "normal" uses, you'd probably be fine with having your code written in numpy or pandas

In my experience this is the case. Occasionally though, there might be an bottleneck, and Numba can be a good way to handle it. E.g. when implementing seam carving, I found normal Python / NumPy too slow, and same with Naive Bayes, so used Numba. (Why was I implementing these, instead of using a library? As an exercise. But I guess the takeaway is that if you want to do something more intensive, and are implementing it yourself rather than using a library, Numba can be a good option).

Consumer laptops, such as apple silicon macbooks with just 32gb ram, can run fairly large models, such as Mixtral-8x7b, an open source model that's comparable to gpt-3.5, well enough for interactive chat, (and seems likely models of this size will keep improving) - ofc, 64gb ram preferable, because it unlocks larger models. And 3090s are the other way to to go, which are faster but not quite as much vram, though you can use multiple.

Yes, I saw that and still consider the methodology questionable. A fairer approach might be: time from the cli, including compilation, for compiled languages. (Or warmup the jit compiled code.)

The reality is that it would be very hard to find Python code that does not use NumPy (or some tensor lib), for matmul.

Including time to JIT compile is questionable, why not also include time to compile the compiled languages?

In addition to the uncertainty of the value, there's also the context of how it will be used. E.g. for some quick business decision that needs to be made, it may be fine to round to 1 sig fig, e.g. 71.244% -> 7/10 -> the majority.

Hmm, seems someone's managed to patent something fairly trivial... the same thing as OpenAI's Clip, but instead of with image-text pairs, with music-text pairs; for vector retrieval. And apparently neural networks aren't software because they can be implemented as hardware...

I've been using Yabai + skhd for years, and found the benefits far outweigh the issues. 99% of the time it's fine, and for the 1% I just disable it temporarily.

Killer feature for me (apart from windows automatically being resized to fit, which is great), is ctrl+i to switch to screen i, and ctrl+shift+i to move the current window to screen i. I tend to have 2 windows max per screen, and use Yabai to flit between screens / flit windows to different screens.

(I used i3 on Linux prior to setting up Yabai, and configured shortcuts to be similar to i3.)

Propelling 1500kg around at speed just to get a 80kg human somewhere, doesn't seem super efficient, (apparently electric bikes are around 30 kg).

Related; 4x improvement in accuracy for experiment 2 of this paper, (with gpt-3.5-turbo), by changing the prompt:

https://sidsite.com/posts/reversal-curse/

Shows what a dramatic effect the prompt can have

Link to the demo: https://huggingface.co/spaces/sradc/visual-content-search-ov...