ozr

RAG and fine-tuning are very different. Few-shot prompting and RAG are both variants of in-context learning.

I’ve taught LLMs imaginary words and their meanings with minute amounts of data (two or three examples) via full fine-tuning, LoRA and QLoRA.

I have no idea where the myth of ‘can’t add new knowledge via fine-tuning’ came from. It’s a sticky meme that makes no sense.

Pretraining obviously adds knowledge to a model. The difference between pretraining and fine-tuning is the number of tokens and learning rate. That’s it.

> the more Shotwell is running things, the more optimistic I am about SpaceX’s future.

Her age is relevant when discussing the future of SpaceX, considering the median retirement age.

She's 60 years old now.

The awesome-X lists seem like a community attempt to do something similar.

He's obviously not closing Jira tickets at Meta, but that doesn't mean he's not an engineer. As an example of the positive impact that the Llama releases have had, this post from him has been doing the rounds lately in response to criticsm like yours:

https://www.facebook.com/notes/775294156352065/

If your product is an AI model (OpenAI, Anthropic, etc) you can't give it away for free.

If your product is a social graph w/ ads (Meta), you can.

It's hardly corporate charity:

* Meta releasing these models creates an improvement and tuning ecosystem around it, giving them access to tons of free developer time.

* It's also a strong recruiting tool, for engineers and researchers frustrated by, e.g., Google and OpenAI becoming increasingly closed. They know they can publish at Meta.

* The cost is insignificant. Meta had 30B in revenue just in Q2 2023.

For a k/v store, a hash table will take you very, very far.

You aren't required to purchase a home or drive a car. If you chose to, as most Americans do, then yes: the demand is inelastic.

But that doesn't imply what you're saying unless the supplier has monopoly power, which they, by law, do not.

Insurance carriers have an incentive to compete on price like any other business. There are plenty of options. The margins are generally pretty small, and there is a lot of regulation around what portion of payments must be used for claims.

There's a few obvious exceptions, but plenty of insurance isn't required.

This is not correct. Fine-tuning can absolutely add new knowledge to a model. It's been repeatedly demonstrated at this point.

LIMA demonstrated that instruction-tuning and output formatting could be trained with a limited number of samples, not that finetuning was incapable of adding new information to the model.

It may be sub-optimal in most cases to RAG, but it does work.

Cartel is maybe overly pejorative, but it's definitely based on relationships.

Postmasters at large ESPs and inbox providers can and will text each other to resolve issues.

This is pretty much how the internet works as well (BGP, etc). It's opaque, but open.

I've been on large and small teams of people that cared and didn't care. It feels noble and reasonable and right to tie it to a greater purpose; like your soup kitchen vs supermarket example. It's not that type of care the OP is getting at.

It's closer to a craftmanship kind of care. Being proud of what you're putting into the world. Caring that it works well, not because the purpose is important, but because it's something you're doing and you want to do it well.

In context, he's not describing a compromise, he's talking about a 'wrecking amendment'. Adding something absurd to a bill. The difference is that the person adding the amendment doesn't actually want or expect their amendment to be passed.

Wikipedia has a fun example:

> The Wyoming House of Representatives Committee of the Whole amended House Bill 0085, a proposal to study the state's emergency preparedness, in order to kill the bill. The amendment would have required the state to consider purchasing an aircraft carrier (even though Wyoming is a landlocked state), purchasing fighter jets, establishing a military, initiating a draft, and creating a currency.

> I don't see how you're expecting things to work differently. Would we expect Democrat-sponsored legislation to fail for reasons apart from Republican opposition?

There is a popular perception that:

- If Democrats are in the minority and want to block a Republican bill, they will attempt to drum up public pressure, advocate on the floor and with individual legislators, etc.

- If Republicans are in the minority and want to block a Democrat bill, they will by any means necessary. They'll use various edge cases of parliamentary procedure, attach/amend/manipulate bills to poison them, etc. 'Playing dirty'.

I have no idea if this is actually true, but it's a common idea I've heard from both sides. The Democrats see it as a sign of their moral superiority, and the Republicans see it as an demonstration of conviction.

The founder ('Beff Jezos') has a large twitter presence.

As someone that has had those titles for many years in the past: I'm only going to object to the one that's paying less at the time.

A good sysadmin was always doing the same thing as a good SRE.

> Traffic within a data center is relatively cheap, is it not?

Complexity is very, very expensive.

I'm less concerned than the OP. It seems like they are using the revenue from search to fund other endeavors. Same model as Google, fundamentally.

> I also found it weird that the author thought being an EM is a good path to being an entrepreneur.

This was the pitch from my manager that led to me moving into an EM role. He knew I was interested in doing a startup in the future, but I didn't have experience with people management.

I don't think it's necessarily a great path to entrepreneurship, but if you are solid on the engineering side it's a good skillset to develop.

I'm an engineer with a passing interest in the law. I've frequently had to explain to otherwise smart and capable people that their one weird trick will just get them a contempt charge.

Salary is one part of comp, which is one part of what makes a company attractive.

There are plenty of levers to pull that are not salary, and not based on geography.

> You can try to pay everyone according to the highest possible local salary you might compete against, but now you're arbitrarily paying more than you need to everywhere else.

This is the difference. I don't see it as arbitrarily paying more than I need to. I see it as paying people the same for doing the same work. My goal is not to pay as little as possible, or even to pay a high rate relative to their local market.

I can't look at someone in Romania doing the same job as someone in Chicago and tell them I'm paying them less because they're in Romania.

I completely understand that this is not economically optimal, and I love capitalism. I just have no interest in saving money this way.

> So you could try to set a median pay that's higher than the LCOL targets but lower than Bay Area salaries. Now you're attractive to some talent, but other people won't consider your company because it pays less than their alternatives.

Salary is only one part of comp. Comp is only one part of what makes a company attractive.

The (good) companies I've been at are location agnostic on pay, or close to it. It's not about getting a discount, it's about expanding the talent pool. We want the best we can get, not the best within 25 miles of a hub.

Definitely lucrative enough. The use case you've described isn't particularly uncommon, but lots of companies just pay for the egress.

The problem is that there are now multiple generations of software engineers that do not know how bandwidth is priced. They've only used managed providers that charge per unit of ingress/egress, at some fractional dollar per GB.

This is why you'd 'charge up' some prepaid store.

For you and me, yeah the training is useless. For someone actually naive enough to pick up gift cards for the 'CEO'? It could help.

It's a sophisticated attack for sure, but the data collection really isn't too difficult now. A minute or two of audio is sufficient for voice, and a single good image.

The bit of code at the center of an operating system that mediates access between userspace code and the hardware.

Whereas in CUDA programming, a kernel is just the code running on the device. When I first heard kernel in relation to CUDA programming, I expected it to be a) the actual OS kernel, and then b) something akin to a GPU driver.

I can see how NVIDIA got there, but it's not immediately obvious if you're coming from a SWE context.

I agree, but the push for 'AI Safety' as a whole is driven by the x-risk crowd.

There are a few people genuinely concerned, however misguidedly, about the outputs of LLMs. The big money coming into the field from EAs and the sort of background fear is about fast takeoff, shoggoths, paperclip maximizers, etc.

The thought is that, if you can't reliably align an LLM to output what some authoritative source wants, then you can't reliably align the inevitable machine god that will destroy us.

In that context, user education doesn't do any good.