7,517 karma · joined September 12, 2011
I wonder if a different approach without that issue exists. For instance, using max(0, exp(x)-1) instead of exp(x) in the softmax attention formula. That way when the query is orthogonal to the key (or worse), it does not contribute.
GCP: $0.023/GB/month (standard storage, beyond 5 GB free limit)
Azure: $0.019/GB/month (hot, first 50 TB/month)
Cloudflare: $0.015/GB/month (beyond 10 GB free limit)
Scaleway: ~$0.013/GB/month (single-zone)
Backblaze: $0.006/GB/month
Hetzner: ~$5.49/month + $0.0054/GB/month (beyond 1TB free limit)
Egress costs vary though.
You mention that foldhash does not claim to provide HashDoS resistance against interactive attackers, so perhaps that disqualifies it.
If anything, given this requirement, comparing with wyhash, as they do in the article, is misleading.
Don’t they all indicate being based on the transformer architecture?
> not entirely because of transformers but because of the hardware
Kaplan et al. 2020[0] (figure 7, §3.2.1) shows that LSTMs, the leading language architecture prior to transformers, scaled worse because they plateau’ed quickly with larger context.
Honestly, those are not the missing parts that most matter IMO. The evolution of the concept of attention across many academic papers which fed to the Transformer is the big missing element in this timeline.
1986: Geoffrey Hinton publishes the backpropagation algorithm as applied to neural networks, allowing more efficient training.
2011: Jeff Dean starts Google Brain.
2012: Ilya Sutskever and Geoffrey Hinton publish AlexNet, which demonstrates that using GPUs yields quicker training on deep networks, surpassing non-neural-network participants by a wide margin on an image categorization competition.
2013: Geoffrey Hinton sells his team to the highest bidder. Google Brain wins the bid.
2015: Ilya Sutskever founds OpenAI.
2017: Google Brain publishes the first Transformer, showing impressive performance on language translation.
2018: OpenAI publishes GPT, showing that next-token prediction can solve many language benchmarks at once using Transformers, hinting at foundation models. They later scale it and show increasing performance.
The reality is that the ideas for this could have been combined earlier than they did (and plausibly future ideas could have been found today), but research takes time, and researchers tend to focus on one approach and assume that another has already been explored and doesn’t scale to SOTA (as many did for neural networks). First mover advantage, when finding a workable solution, is strong, and benefited OpenAI.
How are you testing Molmo 72B? If you are interacting with https://molmo.allenai.org/, they are using Molmo-7B-D.
> Our key innovation is a simple but effective data collection strategy that avoids these problems: we ask annotators to describe images in speech
I see this as another example that datasets trump architecture nowadays.
The architecture is not where the innovation is: it is only CLIP embeddings converted to the LLM tokens through MLP with some pooling to reduce the token count.
What other aspects could they harness?
These recommendations are database-specific. It is definitely true of MySQL, since its row storage hinges on the primary key (and other indices yield that primary key). To have a hot page cache, you need to always add to the same page, so an incremented primary index is preferred. A UUIDv4 secondary index would still have cold page caches though, defeating the purpose. (Unless using UUIDv7.)
But in CockroachDB, the data is distributed. Random INSERTs actually have a speedup over sequential ones, as those insertions can happen in parallel (on different machines). Similarly, SELECTs are faster if the primary index is random, because a single machine is not overloaded; it effectively acts as a load balancer.
Either way, the standard blockchain algorithm doesn’t behave like that.
To be fair, I don’t have a stake in this beyond wanting unambiguous definitions.
It is a bit like last-writer-wins, except it uses the block history count instead of timestamps. Clearly they lose a lot of user intents.
I feel like CRDTs are associated with intent preservation. Perhaps rule 2 ought to include that inconsistency resolution ought to not discard operations. That would also exclude LWW and the “empty string” merge as CRDTs.
Similarly, a list of operations with Lamport timestamps fits this definition. However, just like the blockchain, it makes no effort to preserve user intent.
To go to an extreme, a merge operation which replaces the current content with the empty string, is a CRDT per this definition.
I personally find their compact mode the cleanest I have ever seen. This is the entire window: https://imgur.com/hhfyeVz To access the address bar, move the mouse to the top, or type Ctrl+L. For the tabs, move the mouse to the left, or type Ctrl+1, 2… or Ctrl+Tab to cycle through.
I wish Firefox had such a compact mode.
What is the company’s ethical position though? It officially stemmed from Mr Musk’s objection that OpenAI was not open-source, but it too is not open-source. It followed Mr Musk’s letter to stop all AI development on frontier models, but it is a frontier model. It followed complaints that OpenAI trained on tweets, but it also trained on tweets.
Companies like Meta, Mistral, or DeepSeek, address those complaints better, and all now play in the big league.
This 2023 study[0] indicates 48% of people discover apps by browsing through the App Stores, compared to 21% that discover them through Web search engines.
Is this a trend?
This 2015 study[1] indicates 40% of people browse for apps in the App Stores, compared to 25% (“1 in 4”) that discover them through Web search engines.
Does this apply to Patreon?
The CEO of Patreon had this to say:[2]
> iOS is actually now the most used platform for communities on Patreon.
[0]: https://www.semrush.com/blog/app-store-optimization/
[1]: https://www.thinkwithgoogle.com/marketing-strategies/app-and...
> Your app should include features, content, and UI that elevate it beyond a repackaged website.
[0]: https://developer.apple.com/app-store/review/guidelines/
When they are asked to search for a website, they open the app store and search there. If there is no result, they give up.
Additionally, all Llama 3.1 models are available in https://api.together.ai/playground/chat/meta-llama/Meta-Llam... and in https://fireworks.ai/models/fireworks/llama-v3p1-405b-instru... by logging in.
The narrative is independent of the price.
SpaceX has showed that they may refuse service in various situations, such as drone access to Starlink Internet in a battle during Ukraine war. Thus a government being dependent on them for their space fleet, gambles the possibility of being suddenly unable to guarantee commercial services such as Galileo GNSS, or during military operations.
It places Claude 3.5 Sonnet in third position.
That feels like it runs counter to this statement from the Gemini 1.0 technical report[0]:
> Gemini models are trained to accommodate textual input interleaved with a wide variety of audio and visual inputs, such as natural images, charts, screenshots, PDFs, and videos, and they can produce text and image outputs
(Details of the estimation in the source code.)
The way I read the Gemini technical report, it seemed like, unlike GPT-4 vs DALL-E, Gemini was pretrained with multimodal outputs. Is that not the case?
It likely surpasses AlphaGo, and just like Stockfish, it delivers a protocol that can hook into many user interface apps: https://github.com/lightvector/KataGo?tab=readme-ov-file#gui...
From those technologies, also came an interesting visualisation of how human players changed their habits following AlphaGo: https://drive.google.com/file/d/16-ntvk3D1_pgjJ7u64t4jMYMh0z...
In my view, the main value of larger models is distillation (which we particularly witness, for instance, with how Claude Haiku matches release-day GPT-4 despite being less than a tenth of the cost). Hopefully the distilled models will be easier to run.