stygiansonic

Looks like Fusion calls a bunch of models and then uses an LLM to synthesize the results, and pass to another model for final output.

Fugu looks like it's doing something different? Using an LLM earlier on in the flow as an orchestrator to decide which other LLMs to call. More coordinator than simply synthesizing results, and more "agentic".

It's interesting because it's all exposed behind a single OpenAI compatible endpoint (Responses API?) and so then presumably someone could use this for one of their single agents. Now you have agent-of-agents, nested in some sense. The token usage increases accordingly!

(Perpetrators also not caught)

  The jury found that Ding stole trade secrets relating to the hardware infrastructure and software platforms that allow Google’s supercomputing data center to train and serve large AI models. The trade secrets contained detailed information about the architecture and functionality of Google’s custom Tensor Processing Unit chips and systems and Google’s Graphics Processing Unit systems, the software that allows the chips to communicate and execute tasks, and the software that orchestrates thousands of chips into a supercomputer capable of training and executing cutting-edge AI workloads. The trade secrets also pertained to Google’s custom-designed SmartNIC, a type of network interface card used to facilitate high speed communication within Google’s AI supercomputers and cloud networking products.

> Gemma 3n leverages a Google DeepMind innovation called Per-Layer Embeddings (PLE) that delivers a significant reduction in RAM usage.

Like you I’m also interested in the architectural details. We can speculate but we’ll probably need to wait for some sort of paper to get the details.

A lot of my teenage years were spent building and playing with PCs and a lot of the knowledge and interest came from reading each and every issue of boot and maximum pc

Jumping into an unknown codebase (which may be a library you depend on) and being able to quickly investigate, debug, and root cause an issue is an extremely invaluable skill in my experience

Acting as if this isn’t useful in the real world won’t help. The real world is messy, documentation is often missing or unreliable, and the person/team who wrote the original code might not be around anymore.

Is this concept (dice room puzzle, doomsday argument) at all related to the st Petersburg paradox? https://en.m.wikipedia.org/wiki/St._Petersburg_paradox

For LLMs/large models the huge cost is memory ops to load each layer weights during the forward pass. This is why doing inference at batch size 1 is extremely wasteful: you pay all the mem ops cost and don’t use enough compute FLOPs to justify.

You want a high enough batch size so that compute:mem ops is close to the ratio for that GPU. This is usually done by batching together multiple user requests.

At times of low usage there is excess capacity because batch size is below this “optimal” ratio. So they can slot in these “relaxed SLA” requests for little marginal increase resource usage on their end.

Basically have a queue of these requests that you use to “top off” your batch size when you can.

Edit: also you may not be able to get optimal batch size depending on when the requests arrive, eg you don’t want to wait forever to fill up a batch. So again having a queue of outstanding/delayed requests to serve allows for smoothing things out and increasing compute utilization

Obviously this doesn’t capture all the nuance.

One of the requirements for third party App Store use by developers is maintaining a standby letter of credit of 1M euros from an A rated financial institution

Edit: was mistaken, this is a requirement for the operator of a third party App Store.

(Not making a judgment as to whether I believe this is true or not)

Since January of 2023, we have deployed this system in the wild, where it has poisoned several hundred million data sources expected to be scraped and used in the training of new LLMs. This large-scale deployment serves as a real-world testbed and demonstration of our approach's efficacy.

We have concurrently developed a sophisticated countermeasure, capable of detecting and neutralizing the effects of VonGoom. Researchers and organizations concerned about the integrity of their LLMs, and looking to cleanse their systems of our widespread data manipulation, are advised to contact us for pricing and licensing details.

https://www.beyondtrust.com/blog/entry/okta-support-unit-bre...