At the end of the day, the value add is also around integration and implementation and that is very difficult to generalize.
Edit: fixed the number - I thought it launched in January. It turns out late March was the launch, while the first hints/discussion about it were January. I got around to it in early July.
I’m not sure what you mean by this; GPT-4 launched 4 months ago.
For example, it's relatively straightforward to generate dialogue or other text for a game, but structurally connecting any of that text to game mechanics is unsolved (though AI Roguelite is trying).
Many users' GPU's are powerful enough to run models locally, but the rub there is that you can't really run the model at the same time you're playing the game normally, unless you want, like, massive frameskips.
If open source models are good enough (within the category of image generators it looks like many Stable Diffusion clone models are), what's the business case for Stability AI or Midjourney Inc.?
Same for OpenAI and LLMs — even though for now they have the hardware edge and a useful RLHF training set from all the ChatGPT users giving thumbs up/down responses, that's not necessarily enough to make an investor happy.
I personally think the only way AI will end up being a benefit to society is if we end up with unencumbered free and open models that run locally and can be refined locally. Every financial incentive is pushing in the other direction though.
Meta is no doubt doing this because it’s in their best interest, but if both the quality and licensing of LLaMA 2 start a trend that’s a pretty effective counter-weight to eyeball scanner world.
And there’s other stuff. George Hotz is pretty unpopular because he does kind of put the crazy in crazy smart (which I personally find a refreshing change to the safe space for relatively neurotypical people in the land of aspy nerds), but tinygrad is a fundamentally more optimizable design than its predecessors with an explicit technical emphasis on accelerator portability and an implicit idealistic agenda around ruining the whole day of The AI Cartel. And it runs the marquee models. Serious megacorp CEOs seem to be glancing nervously in his direction, which is healthy.
It’s not locked-in yet.
If it isn’t a lot, I encourage you to look more closely. Let’s skip all the disputes about his achievements prior to this year (and you really need to take with a grain of salt anything bad you hear about someone who has been actively fucking up the afternoon of powerful people their entire life).
Tinygrad is so obvious in retrospect, but hindsight is 20/20 and I missed it.
Why the hell do we have kernels for any composite tensor operation? The machine economics are that starting a training run is who cares. Doing a training run is machine hours, days, centuries, millennia.
Spend like a drunken sailor customizing the kernel, exploit the aforementioned “who cares” blank check to just dump the friggin CUDA or PTX or OpenCL or Metal or whatever into /tmp/ and hit it with a fork/execve to nvcc or whatever. You can fuse and columnize and whatever to your hears content in the IR.
We spent how much time rigging up a trouś convolutions via IM2Col by hand with template instantiation in ATen? Well, we’re never getting that back. And while dating NVIDIA is fun, being chained in their soundproof basement not so fun. “It lives with the same 24Gb of GDDR6 on it paid for last year on its skin or it gets the p4d.24xlarge pricing again”.
Nobody has jacked anyone so usuriously via API lock-in since Gates and Win32.
Anyone who is doing a YouTube video called “Get in Losers, We’re building a Chatbot” and then live codes for 5 hours to turn 2700 lines of Python that can’t run LLaMA into 3200 lines of Python that can run it on three times the number of platforms that PyTorch?
I’m struggling here.
Where his arrogance, ignorance and complete lack of experience with how teams and companies function effectively was on full display.
You can spout off technical terms all day long but I've worked at two of the FAANG companies and met hundreds of incredible engineers and it wasn't their grasp of the technology that made them great. It was their ability to lead, deliver, communicate, relate etc which are skills that sound far less impressive but are infinitely more important.
"Hackers like to work for people with high standards. But it's not enough just to be exacting. You have to insist on the right things. Which usually means that you have to be a hacker yourself. I've seen occasional articles about how to manage programmers. Really there should be two articles: one about what to do if you are yourself a programmer, and one about what to do if you're not. And the second could probably be condensed into two words: give up."
I've only worked for one FAANG, but it was for like 7-8 years out of a 20+ year career, and it was mostly on stuff where even small mistakes cost a lot of money so I'll counter your anecdote with one of my own: at the beginning of the century the software business was really weak on demographic diversity (a problem we still have and still needs to be addressed) but it was unparalleled in neurodiversity. In particular there's a longitudinal component to my observation. I'm not here to diagnose anyone via the Internet, but it's pretty clear that George (like a lot of us) is pretty friggin "different".
During the period of time (say 1995-2015 +/- 3 years) when most of the current empires of tech were built, staggeringly successful managers in both open-source (e.g. Linus) and industry (e.g. Sheryl) created a lot of space for aspy nerds who "spouted off" about tech stuff in blunt "this is stupid" kinds of ways, and all kinds of other stuff that's now called "being toxic" rather than "being a weird nerd". It was only during that latter half of this time (and really the end of it) that software became a sufficiently high-status occupation to draw in a bunch of people who decided that it was a good idea to outmaneuver those folks on the org chart and re-brand an entire industry of people from "probably on the spectrum at least a bit" to "toxic asshole".
I don't know if packing Linus off to charm school because he was simply too important to write off was a better or worse move than just letting him be Linus on a mailing list he founded, I kinda liked the original Linus. But even if we decided that old-school hacker aspiness couldn't get a blank check anymore (on the Internet we fucking built), it's just a strictly better idea to strike some sort of compromise with the aspy nerds than to redefine the vibe that had prevailed for decades as basically up there with racism, misogyny, and homophobia (you might notice that aspy tech assholes are on average some of the least racist, misogynistic, and homophobic people in any industry: they on average accept anyone who kicks ass at code or is trying hard and are short with anyone who is dismissive of code, irrespective of demo).
I wrote the `im2col` and dilated convolution kernels on NVIDIA and Intel that went into Caffe2 originally, and AFAIK that stuff migrated over to like ATen or whatever (though I imagine someone has rewritten it since), so I don't exactly love the "spout off technical terms all day log" characterization, but mostly I'll admonish you that calling leadership and communication skills "infinitely" more important than technical expertise is dangerously close to "arrogance and ignorance", and that overlooking that someone has been in the public eye on e.g. Twitter since they were 16 doesn't seem like a particularly serious effort to "relate".
In a way rather reminiscent of early Linus, George has organized a few thousand iconoclastic accelerated computing pros and/or people aspiring to become one into a small but surprisingly credible threat to the current round of Bond Villain monopolists. Maybe we cut him some slack on being a bit of an edgelord on his own Twitter account, his own Twitch stream, his own GitHub Issues page, and to set the culture at his own startup?
History has shown time and time again that you need the guardrails that regulation/laws provides in order to channel progress in an effective direction. Otherwise the negatives of human nature destroy progress. We have seen this recently with the rampant criminality and negative behaviours permanently crippling the crypto space.
In the AI space there is no evidence that allowing models with illegal/unethical content is going to magically translate to some boost in performance, capability or efficacy than would otherwise happen.
In fact again history has shown that the opposite is likely to happen. That having models that don't exhibit negative effects e.g. racism, sexism will result in them being used in more places and exposed to wide audiences. Thus translating to a bigger net benefit for society.
Open is its own area, proprietary general models are a race to zero vs OpenAI and Google who are non-economic actors.
Most AI right now is just features tho, very basic without the real thinking needed.
Next year we go enterprise.
> Make models usable is really valuable Sure... but is there really a moat here? Seems like OSS can address "making models usable" and without lockin from a startup.
Business model is called open core hundred of billions in market cap here as most folk don’t want to build/train their own models and will pay for support and help https://en.m.wikipedia.org/wiki/Open-core_model
Can look at databricks and many kfheds
Edit: if AMD plays their cards right, I'd expect that they can get a lot more in on the action, too.
ChatGPT already has a lot of value by itself, the value added by any startup is going to be marginal at best.
LLMs are a lot more like a generalized processor than people are admitting right now. Granted you can talk to it, but it becomes significantly more capable when you learn how to program it -- and thats where the value will be added.
I don't know if you mean, like, LoRAs and similar (actual substantive changes), but the vast majority of "learning how to program" LLMs (accounting for the majority of startup pitches as well) is "prompt engineering" - which, as the meme goes, isn't a moat. There's a skill to it, yes, but if your singular advantage boils down to a few lines of English prose, your product isn't able to control a market - and VCs are (rightly) not interested unless you have the possibility to be a near-monopoly.
But no one would say that now, thats ridiculous. There is a sufficient degree of prompt engineering that is already defensible, I'm already doing it myself IMO. You'll see very sophisticated hybrid programming/prompting systems being developed in the next year that will prove out the case.
For example 30 parallel prompts that then amalgamate into a decision and an audit, with 10 simulation level prompts running chained afterwards to clean the output. These types of atomic configurations will become sufficiently complex to not be just for 'anybody'.
It's pretty effective on complex problems like Spam, Trust and Safety, etc. And the applications of these sort of reasoning atomic configurations I think are unlimited. It's not just 'talking fancy' to an AI, its building processes that systematically improve reasoning to different very hard applied problems.
But overall, hasn't that theme been true for like... all tech ever? You have to set up and build your own innovation path at some point.
They are limited to applications in which the latency slo is O(seconds), knowledge of 2021-present doesn’t matter, and you’re allowed to make things up when you don’t know the answer.
There are, to be fair, many such applications. But it’s not unlimited.
But in general many configurations are possible, but also you need to refine the personas a great deal to get it to work well.
Sure, so you ensemble some results. You're back to the classical "hyperparameter" problem though that's faced ML for a long time-- what those personas are, what those subsequent prompts are, etc. require a fair amount of manual verification and tuning. And the search space is extremely vast.
Not to mention that something like this is likely to be very unperformant.
So... prompt engineering? They're an extremely inefficient processor though and very prone to error (despite what synthetic benchmarks may show).
I think this is very much like the CCD sensor, that Kodak couldn't envision using because it was "so expensive, slow and low resolution."
This assumes that all AI startups are squarely competing with ChatGPT, or that ChatGPT is some kind of AGI that can do most machine learning tasks making AI startups redundant "thin wrappers" around ChatGPT.
How does ChatGPT make say Weights and Biases irrelevant, or a startup detecting bank transaction fraud, or say a product that detects when someone at your door is a stranger.
1980 after the foundations of neural networks, but it was too computationally intensive to be useful
2009 with Watson
https://www.hiig.de/en/a-brief-history-of-ai-ai-in-the-hype-...
That's some extreme cherry picking.
During that time period, the internet and smartphones alone have completely changed society (for better and worse) in the span of only three decades, despite the former going causing a minor economic crash in its infancy.
Almost everything is different except human nature. The scammers are innovating just like everyone else.
We still burn fossil fuels to a large extent, still drive but not fly cars, still live on Earth not in space, still die of the same causes, etc.
I watch a long cargo train that looks like it's form the 80s go by and wonder how much the internet changed cargo hauling. I'm sure with the logistics the internet made things a lot more efficient, but the actual hauling is not much different. It's not like we teleport things around now. You can order online instead of out of a catalog, but brick stores remain. You can read digital books, but still plenty of printed materials, bookstores, libraries.
I honestly think this overstates the case pretty severely. They have certainly caused societal change, but from what I can see, society as a whole is not actually all that different from what it was before all of that.
TBH, I'm not sure how to quantify housing bubbles either. I'd bet most of the country has much higher home prices now than in 2007. I bet they were higher than 2007 in most places and most years between then and now too.
It's the very nature of the hype cycle that it is very hard to distinguish from a real thing.
--
[1] though that has produced more useful output than some of the previous hype cycles, as I think will the current one as it seemingly already is doing
[2] I was barely born for the start of the “AI winter” following the first such hype cycle
Producthunt has basically become that these days, none of it is inspirational nor value adding, just constant "X but with AI"
If you think these aren't all fundamental units of the next web, you're not thinking about it from the right perspective. If you can't pick apart the real mathematical utility and origin behind crypto efforts from a generation of scammers who hijacked a very real thing, then you just lack understanding or nuance.
We are decades away from the most obvious solution but it very likely involves cryptographically-backed digital currency and smart contract systems used by automated neural networks.
AI benefits from the same economies of scale as all the other means of production, and the winners are going to be the ones that can reinvest their profit into growth and outpace competitors.
tl;dr I don't see distributed multi party computation doing a better job than a rack of H100s
[PS despite my dismissive tone I'm curious if there's something I'm missing]
All cryptocurrency is, is cryptographically-backed digital currency. The current wave of cryptocurrencies are rooted in the blockchain scheme laid out in the whitepaper, with various forms of proof. The only thing new is a way to prevent double-spend; cryptographically-backed currencies were not invented with Bitcoin.
This stuff absolutely has a fundamental place in future commerce, especially as generations grow tired of the payment processor mafia acting as global moral arbiters. Smart contracts build upon this, asset classes such as NFTs give distributed ways to work with authentic data. All of the scammers who jumped on to these technologies have nothing to do with the underlying technologies themselves, nor the core group of people who are still interested in progressing this tech.
AI is just automation, which can make use of these tools in a trustless environment, without disruption from untrusted/unwelcome parties. It's not life-changing stuff, but these tools will fundamentally drive the web in ways you won't even notice if you don't look for it.