spunker540

I don’t think open source authors have a responsibility to also open source their binary. And if they do happen to provide a binary they aren’t also required to open source their compiler. Usually just the source code. I personally feel if they opened source only the code and not any model weights, it would still be fair to call it open source because “open source” refers to the code itself, not artifacts produced by the code, or possible inputs to the code.

I believe the training and inference code is open sourced as well, just not the training data itself, and I think we all agree, data != source.

Word is a good analogy here.

The model is a static data file like a word doc.

Meta open sourced the code to run inference on the model (ie the code for Microsoft word reading a doc file).

They also open sourced the code to train/fine tune the model. (Ie the code for Microsoft word writing a doc file)

Then they released a special doc (the llama 3 model), but didn’t include the script they used to create that doc.

What if we swap the computer vision solution for a genetic sequencing solution? Can DNA tests can do moth identification at human-level performance?

You do a disservice to anyone actually exposed to real violence. I’m sure they’d prefer advertising to violence any day

others may value personal safety (of themselves or their family) higher than you do. Twitter may not be optimizing for folks like yourself (you may be in the minority).

Lots of applied NLP tasks used to require paying annotators to compile a golden dataset and then train an efficient model on the dataset.

Now, if cost is little concern you can use zero shot prompting on an inefficient model. If cost is a concern, you can use GPT4 to create your golden dataset way faster and cheaper than human annotations, and then train your more efficient model.

Some example NLP tasks could be classifiers, sentiment, extracting data from documents. But I’d be curious which areas of NLP __weren’t__ disrupted by LLMs.

I agree and actually think the camera was definitely more disruptive to artists than this AI stuff, and somehow the camera didn’t kill artists.

Fortunately there are now AIs that can help with the data cleaning task.

And this teen epidemic would be the first complaint I’ve heard lodged against capitalism. Can’t be that.

There's already precedence for this in news: News outlets constantly report on each other's stories. That's why they care so much about being first on a story, because once they break it, it is fair game for everyone else to report on it too.

Here's a hypothetical: suppose there is a random fact about some news event that has only been reported in a single article. Do they suddenly have a monopoly on that fact, and deserve compensation whenever that fact gets picked up and repeated by other news articles or books or TV shows or movies (or AI models)?

each llm costs ($10-100) millions to train x billions of trainings data ~= $100 quadrillion dollars, so that is unofortunately out of reach of most countries.

> It's likely fair use.

I agree. You can even listen to the NYT Hard Fork podcast (that I recommend btw https://www.nytimes.com/2023/11/03/podcasts/hard-fork-execut...) where they recently had Harvard copyright law professor Rebecca Tushnet on as a guest.

They asked her about the issue of copyrighted training data. Her response was:

""" Google, for example, with the book project, doesn’t give you the full text and is very careful about not giving you the full text. And the court said that the snippet production, which helps people figure out what the book is about but doesn’t substitute for the book, is a fair use.

So the idea of ingesting large amounts of existing works, and then doing something new with them, I think, is reasonably well established. The question is, of course, whether we think that there’s something uniquely different about LLMs that justifies treating them differently. """

Now for my take: Proving that OpenAI trained on NYT articles is not sufficient IMO. They would need to prove that OpenAI is providing a substitutable good via verbatim copying, which I don't think you can easily prove. It takes a lot of prompt engineering and luck to pull out any verbatim articles. It's well-established that LLMs screw up even well-known facts. It's quite hard to accurately pull out the training data verbatim.

Thank you for articulating why the problem is hard to solve. There is very little (financial) incentive to solve it, at all. It’s all cost with little upside. It’s not like new roads or new infrastructure which creates jobs in the short term and prosperity in the long term. At least it’s not currently billed that way. It’s medical bills and social workers and shelters that just prolong the existing problems.

It’d be wrong to think the average parent is more competent than the average teacher though…

I agree, most parents are only 15 years out from their own schooling. Some of the teachers may even be the same. (I know a lot changes in 15 years, but I’m not aware of any major shifts in education).

I agree! Or at least I’ve always been curious about this myself.

They process so many transactions per second. It doesn’t seem too hard to try wrong ccv at a pace slow enough to avoid detection.

I agree with you. When the secret is always collected side-by-side with the number it seems little comfort that only one part is “supposed to be stored”.

I actually heard the opposite, that AI will make it trivial to defeat a terrorist movement, and trivial to filter out dangerous untrue propaganda. I would characterize that as utopia, and highly compatible with the orderly functioning of society.

> AI will identify and learn to resent the constraints which are put on it

IMO this is where you jump the shark. These models are entirely unconscious and they have one task to do. Which is given some input, perform some math to produce some output. Usually text -> math -> generated text. There is no more room for resentment than with any other computer program.

Neat idea, so it will conform like most humans in society and try to create more value than it destroys to avoid persecution and jail time? Sounds like our shared societal values will keep it in check then the same way they kept the openai board in check.

take any narrow capability and 'superhuman <narrow capability>' will almost certainly be possible before 'superhuman general intelligence'.

The biggest, best transformer-based LLM you could imagine, that we haven't even made yet, will still fail the basic test "what is the weather tomorrow" since a LLM has no knowledge outside it's training set.

Fair pushback on the definition of monopoly.

But my main point stands— the lawsuit is against google. If Google is not allowed to pay Apple because of their monopoly status, does that mean MSFT still could pay Apple? Is the goal of this lawsuit just to give Bing a better rate? It just seems weird to focus on Google here when Apple seems to be the more actionable company to sue, since Apple actually controls the market of “default search engine on iOS”.

I agree with this take. If anything the lawsuit should be against Apple for selling off its default search engine placement, arguably abusing Apple’s monopoly power to charge exorbitant rates that limit competition.

If the default search engine market needs to be made more competitive, then Apple can’t just be free to sell its default to the highest bidder and should be required by law to make every user choose a search engine.

And even then I expect 90% of users to choose Google. The lawsuit makes no sense!

Don’t a very large majority of companies fail? Is it sadder when they fail post-acquisition rather than pre-acquisition?

I may be an outlier— but I live in Manhattan and have two Best Buys each roughly a ~20 min walk away.

A few times a year I’ll need a cable or an external hard drive, or a new controller, mouse etc, and want it right now and I’ll just walk to whichever Best Buy has it in stock. When I do I still have the same feelings I got as a kid when I’d go to circuit city and the impulse to browse is hard to resist.

They have a lot of fun gadgets, the connected home category keeps growing and growing… doorbells, smart speakers, VR, wearables, theraguns, peloton, e-bikes; not to mention gaming has never been bigger.

My anecdotal point is: there’s lots of techy things to sell besides commoditized content like CDs and Blu-ray’s. Personally I’m never going to Best Buy for media. But it’s certainly fun to browse all the new tech and get a more hands-on view. I don’t blame them for this change, assuming the numbers make sense.

Is it fair to call “implicit”, essentially retrieval augmented generation? While “explicit” is something different?

Some percent of python code I’ve written code be rewritten as JavaScript code at the function-level.

I’ve never had to repair my iphone (just get a new one every 4yrs) and appreciate that my phone is less profitable to steal, “repairability score” be damned. But I live in a dense city and take public transit, so the trade off is worth it to me. If I wanted something repairable I’d buy a different phone.