bick_nyers

I disagree. I have to learn by committing to muscle memory. I always say I'm a slow learner, but a fast thinker.

Yes, I can read/hear a concept and understand it in abstract and visualize it pretty easily, but it will leave my brain just as easily as it entered. Just the way my memory works.

Solved problems speeds up that muscle memory learning process significantly as opposed to going line by line and attempting to generate your own problems/solutions. In addition, you can solve a problem correctly, but not have the correct prose, solution manuals can help there as well.

Edit: Honestly the biggest thing about solving problems is that it gives a sense of progression and a dopamine-reward loop that most people just don't get from reading one line at a time. That being said, good problems and good solutions can be time consuming to generate, so it makes total sense to me that the PhD-level textbooks don't follow that format.

I think this is often the reasoning of management, however, what they don't see is that developer skill often follows a power law and the flow state is a multiplier on top of that. The cost of a midday check-in is not the cost of how much you pay that developer per hour because developers work output per time is nonlinear.

I believe the flow state is why it's so easy to believe the idea of the 10x engineer, we've all experienced the feeling of flying through code without barriers, and we have all felt the below average days where you may be tired and just can't quite load all that complexity into your head to get any real work done. So we believe that there are people out there who are so skilled that they exist in this flow state all the time.

Yes, developers should work on themselves to rely less on a state of flow to be high performers, but even so there will always a benefit to being "in the zone".

Objective Function: Optimize on a per-user basis. Constraints: Output generated must be considered legal in user's country.

Both things can co-exist without being in conflict of each other.

My (hot) take is I personally don't believe that any LLM that can fit on a single GPU is capable of significant harm. An LLM that fits on an 8xH100 system perhaps, but I am more concerned about other ways an individual could spend ~$300k with a conviction of harming others. Besides, looking up how to make napalm on Google and then actually doing it and using it to harm others doesn't make Google the one responsible imo.

Alternatively, you include information about the user of the model as part of the context to the inference query, so that the model can uniquely optimize its answer for that user.

Imagine if you could give a model "how you think" and your knowledge, experiences, and values as context, then it's "Explain Like I'm 5" on steroids. Both exciting and terrifying at the same time.

I had an idea similar to this for a model that allows you to parameterize a performance vs. accuracy ratio, essentially an imbalanced MoE-like approach where instead of the "quality score" in your example, you assign a score based on how much computation it used to achieve that answer, then you can dynamically request different code paths be taken at inference time.

I didn't read the paper, but am familiar with: https://mpeg-miv.org/

One challenge here is dedicated hardware support, running any sort of video processing/decoding on the CPU is unfeasible (GPU less so), particularly if the context is a battery-powered device (Quest 3, Apple Vision Pro), with high resolution and framerate (6K-8K, 60fps).

That doesn't even get into the logistics of capturing, serving, and software supporting this video data either

To be fair, you need 2x 4090 to match the VRAM capacity of an RTX 6000 Ada. There is also the rest of the system you need to factor into the cost. When running 10-16x 4090s, you may also need to upgrade your electrical wiring to support that load, you may need to spend more on air conditioning, etc.

I'm not necessarily saying that it's obviously better in terms of total cost, just that there are more factors to consider in a system of this size.

If inference is the only thing that is important to someone building this system, then used 3090s in x8 or even x4 bifurcation is probably the way to go. Things become more complicated if you want to add the ability to train/do other ML stuff, as you will really want to try to hit PCIE 4.0 x16 on every single card.

Agreed, but there are other costs associated with supporting 10-16x GPUs that may not necessarily happen with say 6 GPUs. Having to go from single socket (or Threadripper) to dual socket, PCIE bifurcation, PLX risers, etc.

Not necessarily saying that Quadros are cheaper, just that there's more to the calculation when trying to run 405B size models at home

I'm curious what techniques they used to distill the 405B model down to 70B and 8B. I gave the paper they released a quick skim but couldn't find any details.

Would be interesting to see the performance on a dual-socket EPYC system with DDR5 running at maximum speed.

Assuming NUMA doesn't give you headaches (which it will) you would be looking at nearly 1 TB/s

The best NVLINK you can reasonably purchase is for the 3090, which is capped somewhere around 100 Gbit/s. This is too slow. The 3090 has about 1 TB/s memory bandwidth, and the 4090 is even faster, and the 5090 will be even faster.

PCIE 5.0 x16 is 500 Gbit/s if I'm not mistaken, so using RAM is more viable an alternative in this case.

Edit: 3090 has 1 TB/s, not terabits

Energy costs are an important factor here too. While Quadro cards are much more expensive upfront (higher $/VRAM), they are cheaper over time (lower Watts/Token). Offsetting the energy expense of a 3090/4090/5090 build via solar complicates this calculation but generally speaking can be a "reasonable" way of justifying this much hardware running in a homelab.

I would be curious to see relative failure rates over time of consumer vs Quadro cards as well.

What if the LLM that a specific NPC utilizes was handcrafted/fine-tuned by a human?

Are you asking why pay extra for a CPU and RAM? Not everything can be done on a GPU, for example, .png decompression. If you really analyzed your training code and preprocessed your data substantially you could probably get away with very lightweight CPU/RAM resources but I think the reality is that it's such a minor contribution of cost to the overall system (GPU are expensive) that wasting development cycles on that degree of optimization isn't strictly necessary. When you're a hyperscaler you are likely chasing those fractions of a percent of cost efficiency though. To use my original example, you would likely want to preprocess your .png to either .webp (multi-threaded lossless) or .jpeg (lossy), but likely it wouldn't make sense to turn it into a GPU decompressible format as you would save on CPU cost during training but would pay more in storage (and maybe transfer) cost.

Edit: To be more clear, if the CPU work is bottlenecking training, you want to optimize that as much as possible by preprocessing your data/tweaking training scripts. What I'm discussing here is the gap between "fast enough" and "faster":

CPU is not fast enough for training < CPU is exactly fast enough for training < CPU is faster than needed for training

Great advice.

... you didn't have a UAT environment?

Depends on your definition of simplicity.

Some view simplicity more as minimizing lines of code, e.g., less moving parts.

I view simplicity more as increasing lines of code, the goal being to make the code very verbose. Sometimes this means more moving parts, but smaller "movement" at each step.

There are other views of simplicity as well.

The issue I take with that is that it's another layer of abstraction that can be warped /inflated/etc. Similarly, mobile games often have a "game currency" that you purchase to obfuscate the true cost of a purchase.

It's easier to splurge with a card than it is with cash because it feels more abstract, particularly if it's credit.

M2 consumes about 2.5x the power of MicroSD, which may be fine for most use cases, but anything running on a battery will suffer. 5 watts just for the M2 drive is quite a lot.

I wonder how well dataset distillation would work between NN <-> KAN. Could significantly reduce training time by seeding the initial KAN model, thus making more of these experiments feasible to conduct and thus can experiment on larger models.

Depends on the pay. If more companies start doing this, then you can imagine having to slog through 10-20 of these as an applicant will increase the time to find a job significantly. Most people don't get hired off of their first successful round of interviews. If the company pays as much as the job salary itself then I am fine with this.

Are you using your Shield as an HTPC? In that case you can use the upscaler built into a TV. I prefer my LG C2 upscale (particularly the frame interpolation) compared to most Topaz AI upscales.

I know it's not an amazing safeguard, but providing that alternative would be a deterrent for some teachers just blindly feeding it into an AI and not reviewing the outputs critically. I don't think many teachers would do that, but they are quite overworked as well.

I'm wondering if you could sort the two inputs, add the indicies for the multiplications together, then take the largest of that.

In your 0.1 example, 1000 gets index 2, and 0.1 index 0, combines to 2. This will tie with the 1*8, but I think it would even out with larger vector lengths.

Edit: I could be wrong but I think you can precompute the indices for the weights in advance without a prompt, then you won't need to perform those sorts at runtime.

I've seen it done with a PLX Multiplexer as well, but they add quite a bit of cost:

https://c-payne.com/products/pcie-gen4-switch-backplane-4-x1...

Not sure if there exists an 8-way PCIE Gen 5 Multiplexer that doesn't cost ludicrous amounts of cash. Ludicrous being a highly subjective and relative term of course.

Interesting... 1 Zen 4 EPYC CPU yields a maximum of 128 PCIE lanes so it wouldn't be possible to put 8 full fat GPUs on while maintaining some lanes for storage and networking. Same deal with Threadripper Pro.

A comment I really liked on a previous post about ternary highlighted that what you are actually measuring with {-1, 0, 1} is inverse correlation, no correlation, and correlation.

I think they are moreso saying that GPUs are not optimized for those operations. CPU aren't "designed" for matrix multiplies yet we can still run them, albeit at a slower rate than on a GPU.

Gradient-directed evolutionary algorithm sounds kinda interesting.

It took a lot of human brain flops to get to this point in time though, I wonder how many orders of magnitude more than it took to train ChatGPT...