225 karma · joined March 27, 2018
It is the exact metric you'd choose if you wanted to make the current situation of rsync look like not a big deal.
[0] https://github.com/RsyncProject/rsync/graphs/commit-activity
A big question is what they're using as their draft model; there's ways to do it losslessly, but they could also choose to trade off accuracy for a bigger increase in speed.
It seems they also support only a very short sequence length. (1k tokens)
For example, when they redesigned combat around the 1-Unit-Per-Tile (1UPT) mechanic for CIV 5, this crippled the ability of the AI to wage war. That's because even if a high-difficulty AI could out-produce the player in terms of military, they were logistics-limited in their ability to get those units to the front because of 1UPT. That means that the AI can't threaten a player militarily, and thus loses it's main lever in terms of it's ability to be "difficult."
Contrast this to Civ 4, where high-difficulty AIs were capable of completely overwhelming a player that didn't take them seriously. You couldn't just sit there and tech-up and use a small number of advanced units to fend off an invasion from a much larger and more aggressive neighbor. This was especially the case if you played against advanced fan-created AIs.
I'm hoping they get rid of 1UPT completely for Civ 7, but I have a feeling that it is unlikely because casual players (the majority purchaser for Civ) actually like that 1UPT effectively removes tactical combat from the game.
IMHO, the main reason to use pytorch is actually that the original model used pytorch. What can seem to be identical logic between different model versions may actually cause model drift when infinitesimal floating point errors accumulate due to the huge scale of the data. My experience is that debugging an accuracy mismatches like this in a big model is a torturous ordeal beyond the 10th circle of hell.
Originally discussed here: https://news.ycombinator.com/item?id=41234180
Each of our nodes has a huge amount of DDR attached, in addition to copious amounts of on-chip HBM and SRAM. This allows the system to switch between a variety of different models of different sizes and architectures at lightning speed. A highlight is one based on Llama2 7b, similar to the Groq demo, but executing with bf16/fp32 instead of int8. (And using only 8 chips instead of 568!)
They've got a video here [1] (scroll down slightly) that compares it against a 180B Falcon model that's running on GPUs on HuggingFace. The MoE results are not only just as good quality-wise, but also ridiculously fast. Like, nearly instant. A big benefit is that the experts can be swapped-out and retrained with new data, which is obviously not as easy with the more monolithic 180B model.
This report focuses on the capabilities, limitations, and safety properties of GPT-4. GPT-4 is a Transformer-style model [33 ] pre-trained to predict the next token in a document, using both publicly available data (such as internet data) and data licensed from third-party providers. The model was then fine-tuned using Reinforcement Learning from Human Feedback (RLHF) [34 ]. Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.
This is true at the code level. But at the system-design level, this documentation is the extra coupling.
I feel like it's important to understand this. I agree with the original commenter; in engineering, nothing is truly free. In many cases, this extra coupling helps keep a system strong and stable, like extra nails holding planks of wood together. In other cases, you may find that part of a system's spec actually missed the mark and now needs to be ripped up and redone. That extra coupling might now work against you!
Again, that doesn't mean that it wasn't worth having it. It is just important to understand tradeoffs in engineering.
What I was wondering was: is there a better number? e.g., is there a way to calculate the amount of USD put into a cryptocurrency across a timeframe? What I'm imagining is a metric like (sum of all bitcoins bought by USD purchase price) - (sum of all bitcoins sold by USD sale price) = amount of USD that has been put "into" bitcoin. That first glance, one might expect this number to equal zero, but it should be greater than zero because of the new coins created by mining.
On the other hand, directly because of the above, their hasty self-inflicted take down earlier this year nearly killed the entire hobby. Many series essentially stopped updating for the ~5 months the site was down, and many more are likely never coming back again.
The decision to suddenly take the site down for a full site rewrite feels completely inexplicable from the outside. (A writeup the above or the previous one[1], both of which read like they were written by a Google Product Manager, especially don't help as they conspicuously avoid any comment to the one question on everyone's mind: "leaving aside the supposed security issues with the backend, why on earth also rewrite and redesign the entire front end from scratch at the same time?")