Hacker News Reader

Comment by appplication | Hacker News Reader

The premise of this is flawed. OpenAI is cheap because of has to be right now. They need to establish market dominance quickly, before competitors slide in. The winner of this horse race is not going to be the company with the best performing AI, it’s going to be the one who does the best job at creating an outstanding UX, ubiquitously presence, entrenching users, and building competitive moats that are not feature differentiated because at best even cutting edge features are only 6-12 months ahead of competition cloning or beating.

This is Uber/AirBnB/Wework/literally every VC subsidized hungry-hungry-hippos market grab all over again. If you’re falling in love because the prices are so low, that is ephemeral at best and is not a moat. Someone try calling an Uber in SF today and tell me how much that costs you and how much worse the experience is vs 2017.

OpenAI is the undisputed future of AI… for timescales 6 months and less. They are still extremely vulnerable to complete disruption and as likely to be the next MySpace as they are Facebook.

View on HN

The winner is going to be the consumers of AI.

It's a race to the bottom on pricing on the provider/infra side. It seems very unlikely that any single LLM provider will achieve a sustained and durable advantage enough to achieve large margins on their product.

Consumers can swap between providers with relative ease, and there is very little stickiness to these LLM APIs, since the interfaces are general and operate on natural language. Versus something like building out a Salesforce integration and then trying to update that to a competitor. Or migrating from Mongo to DynamoDB.

Building the LLMs is where the cool tech lives, but surprised so many are seeing that as a compelling investment opportunity.

But, let's see!

Aye, open models from Meta are tearing at the moat.

Yep. I wonder why there is not more features that would differentiate your LLM API, like for example the Functions in OpenAI LLM APIs. While not perfect it is extremely useful and not aware of a similar offering (I do know that there are Python packages offering similar functionality but it is my understanding that they don't work as well as the OpenAI Functions).

> The winner is going to be the consumers

Cloud infra may be a comparable market, since computation is a big share of AI costs. Did consumers win big from competition between AWS, Azure, and GCP? Not sure. I see an uptick in write ups saying “We switched off cloud and reduced costs by 2/3rds.” Not a scientific sample but may leave the question open.

Much harder to switch cloud providers than to switch LLM models. How much time would it take most companies to move their product from AWS to GCP, for example? What if you use a cloud specific tool like DynamoDB?

Margin is a function of stickiness/cost of switching (among other things).

I suspect eventually we will enter a world where migrating cloud providers is mostly a click of a button, but we're a long ways off from that. Requires vendor agnostic and portable apis/containers/WASM runtimes everywhere.

Swapping an LLM, at least in the current state, is about as close to updating a pointer as you can get.

Well I'd have to change my Terraform provider and the managed Kubernetes resource... Other than that, it'd be the same. So half an hour of coding + half an hour of reconfiguring CI secrets?

Pretty obvious to most that switching cloud providers is not quick or painless for the majority of orgs. There's not really an argument in good faith to suggest otherwise.

Especially given that many orgs use managed or cloud specific solutions that have no 1:1 mapping between vendors

I said I'd have to change that one managed resource. Actually it's two - the managed database. But no more.

> I said I'd have to change that one managed resource. Actually it's two - the managed database. But no more.

As someone who worked on a similar project recently, I'm getting that idea that you obviously don't know what you're talking about.

Didn't you know? It's only one DB ENV variable and you're swapped over from GCP to AWS. /s

I'm running the tech for a global startup with 100M EUR turnover. It's still pretty small, but it's something.

You need to plan ahead, sure. Portability was one of my main concerns (including the possibility to go self-hosted). But it's definitely not impossible, nor too hard to do.

You lost me at Terraform and Kubernetes..

What's a serious cloud provider-portable replacement for Terraform?

And Kubernetes? It goes way beyond containers, you can replace many cloud provider-specific resources with Kubernetes resources. What alternative gives you that?

If you swap an LLM, at the very least, you have to run your entire Eval set again.

This will almost certainly lead to a prompt rebuild, to better accommodate new model idiosyncrasies.

If you are unlucky- your use case may be one where Evals require human review.

Unless you are YOLOing it without evals. In that case this is relevant.

You need to rerun eval after each LLM update. GPTs have a new version every couple of months and their capabilities can change quite drastically pretty much randomly. Maybe they will make it more robust in time, but I think this is the feature of the technology and people will have to adapt to these quirks

Yup. Constant Eval.

There are a HUGE number of startups that would have had a massively harder time if they had to roll their own infra. Who cares if the companies eventually have to move off? (Though even Netflix seems ok with it overall for now). There are a ton of services that just wouldn't exist otherwise

I disagree, at early startup scale you don't need much, you just buy a better VPS when you need to scale up

Learning aws vs learning how to operate a vps are of comparable complexity

It took me less then a day to setup infra for my startup (more than 10 years ago)

If you're just running VPS on the big clouds, you're not going to get much advantage out of it, indeed.

But tell me what third tier cheap provider has managed scale-to-zero-or-infinity functions? Managed storage with S3-like API? Where can I get an API gateway cheaper than Amazon? What about managed databases? These tools allow me to develop insanely scalable software incredibly easily.

Agreed - all the big clouds are very expensive VPS hostings. Don't use it for that.

You get portability. Which the functions do not provide. Open source solutions have a longer career utility than proprietary offerings. I remember when NetWare certs were all the rage. Useless now. I remember msce. But if you learned open tools 35 years ago instead... You get the picture.

You can scale from 1 thread 512MB RAM, to 500 threads and 12TB of RAM (off the shelf). Which is good enough for almost everyone who isn't planet scale.

Auto scaling also comes with auto billing. Oops, your accidental infinite loop spawning functions has bankrupted your company. You don't have that risk starting with a VPS.

I agree completely, but the argument remains the same - there's not much utility in using the big clouds as VPS providers, and it's definitely costly to do so.

That part of true. Using EC2 at AWS now and it's much more expensive than vultr, Hetzner, or akamai cloud.

> Managed storage with S3-like API?

What's third tier here? Pretty much all of them offer S3 compatibility it's basically table stakes

Sorry not sure if I'm missing something obvious but isn't a VPS gonna be hosted by a cloud provider? How would that be an alternative to using the cloud?

It's not that hard. Particularly for simple deployments, which a startup should have or they are doing it wrong.

How would you do it for a simple webapp? Genuine question

Ingress that handles SSL. nginx, or caddy. Then stand up your app server behind that on the same VM. Database can be on the same or different VM.

I try to not use anything else if I can avoid it on a new project.

Ingress gives you the ability to load balance and is threaded and will scale with network transfer and cpu. Database should scale with a bump in VM specs as well, CPU and disk IOPS.

If you keep your app server stateless you can simply multiply it for the number of copies you need for your load.

Systemd can keep your app server running, it you docker it up and use that

> If you keep your app server stateless you can simply multiply it for the number of copies you need for your load.

This right here so many times over.

I'm not going to spend time worrying about scaling, I'm going spend time figuring out how to make stuff stateless.

Shove your state into the database. Done.

No that's all fine, I mean physically, where would I put a server and stuff if we didn't have cloud providers? I'd need to pay an isp for an IP address and maybe port forward and stuff like that right? I don't get why I wouldn't just do what you mentioned on a five dollar digital ocean droplet or an ec2 instance or whatever, the cloud still seems orders of magnitude easier to get off the ground.

If you rent a cloud vps as an ingress you can run an overlay network and your actual hosted services can literally be anywhere. See nebula, netbird, etc. You can also ssh forward, but that doesn't scale well past a handful of services and is a bit fragile.

For new small systems I suggest you start with a cloud VPS. If traffic is low, cost of downtime is low, and system requirements are high then a cheap mini PC ($150) at the home or office can keep your bill microscopic. If your app server and database are small then you can just throw them on the VPS too.

I run light traffic stuff at home in a closet so it doesn't occupy more costly cloud RAM. Production ready saas offerings I'm trying to sell right now are all in the cloud. Hosting all my stuff in the cloud would cost me hundreds per month. My home SLA is fine for the extras. I don't need colocation at this time, but I have spoken with data centers to understand my upgrade path.

You can run a live backup server at a second location and have pretty good redundancy should the primary lose power or connectivity.

When system requirements elevate (SLA, security, etc) you probably want to move into a data center for better physical security, reliable power and network. Bigger VPS is fine if it is big enough. Can also do a colocation if you don't want to rent, and you contract directly with a data center. I wouldn't look at colocation until your actual hosting needs exceed at least $100/mo and you're ready for a year long commitment.

But to the point of the original question that started this thread - the takeaway is still that cloud services made development massively easier right? The answer to the original question seems to still be "Yes cloud providers did lead to big wins for customers" since none of these other suggestions are able to get away from needing a cloud service provider without making starting something intensely difficult. And you wouldn't be able to get a vps for 5 bucks for ingress without all the other cloud competition in the market.

> Cloud infra may be a comparable market, since computation is a big share of AI costs. Did consumers win big from competition between AWS, Azure, and GCP? Not sure. I see an uptick in write ups saying “We switched off cloud and reduced costs by 2/3rds.” Not a scientific sample but may leave the question open.

Renting a server and using standard commodity open source software and standard (outsourced) sysadmins is way cheaper and faster than learning and dealing with all the proprietary AWS and Azure junk.

(Probably also less reliable than "cloud", but who cares if you're a startup.)

It's not the tech it's the data. As far as I know the data is not freely shared or at the very least there will be custom built models from data you can't get anywhere else.

Can confirm: The cloud computing fad is well underway to dying...that is why AI is booming. One need only to follow the wall street dollars to figure that one out.

Several very big profile names have recently begun moving back to self-hosted, hybrid, or dedicated hosting solutions.

Cloud computing never was good in terms of value, however, it was only good in terms of scalability. AI solutions built on top of 'the cloud' will always be even worse.

Note that I pay a certain "third tier" cloud provider less than $100/mo total for hosting large websites that would cost me more than 10 grand a month on AWS/Azure/Google...while having better uptime. (the biggest differences? the complete lack of IO and bandwidth charges, and much lower storage charges)

That should tell you all you need about these types of bubbles, but then again, most of us that watched the entire tech field unfold since the pre-internet phase already knew this.

> Several very big profile names have recently begun moving back to self-hosted, hybrid, or dedicated hosting solutions.

Could you provide some examples?

Dropbox has been the biggest name who did the whole “cloud repatriation” thing, which was all the rage at the beginning of 2023, with claims that the cloud was soon to be dead—but cloud revenues are supposedly expected to be in excess of $1T by 2026, so whatever.

Some random survey from ESG found 60% of respondents repatriated at least some workloads. Who knows what the N was though.

Source: https://www.sdxcentral.com/articles/thinking-about-cloud-rep...

Frankly, it makes sense that companies are deciding what works best and where. But the death knell of cloud providers is simply not happening.

Which cloud provider is it, if you don’t mind sharing?

Hetzner or Oracle, maybe?

Just gonna chip in here and say that anything costing a 100 bucks is not valid as an argument in a conversation about cloud.

The prime selling point for cloud for large enterprises was (and still is):

- a signatory that shares blame on several core security issues (iso stuff) - high amount of flexibility for individual teams used to asking for a vm then waiting four weeks for the itops dept to bring it online

Now the vast majority of cloud moves for large enterprises ends up as a shitshow due to poor implementation sure, but the key points for getting it sold are still there.

And ofc you CAN still cost optimize with cloud, its just harder.

Context: Worked in post and pre sales over some years in MSFT in the enterprise segment.

I got out before the downturn and everyone talking about cost, but my approach in selling azure to the c-suite would be fairly similar today I reckon.

But you also have to account for the counterfactuals.

If the cloud had not existed, those that claim they saved money switching away from cloud might never have been in business in the first place.

Certainly the undisputed winners will be the very few firms with enough engineering resources and GPUs to train their own models (not just fine-tune) where the models in question increase the productivity of workers in their non-ai-related profit centers. After that we have the real question of what the future will be of open source LLMs, on the one hand, and the question most relevant to this article of what sort and whether profitable “AI businesses” can be sustained over time. As Stratechery has analyzed, it is very possible that OpenAI turns out to be a very profitable B2C company with ad revenue in ChatGPT not concerned with their B2B sales or even the objective quality of their AI. Right now is an incredible time for AI: cheap Uber rides never qualitatively changed my life, but the current consumer access to AI models is truly incredible and I hope that only improves. However, even ignoring whatever happens on the regulatory front, I don’t think that is guaranteed at all.

This was definitely a theory that made people burn tons of money on the past couple of years, but I don’t think it holds water. These models are getting obsolete so fast, and there’s so many open ones, I doubt any one’s privately trained model can stay relevant for long

The data is the moat.

(If you can train your internally deployed LLM on data none of your competitors have, that's an advantage).

In that case, X.AI, powered by X/Twitter/Tesla data and possibly Facebook (both closed, and somewhat hard to crawl inside) have the largest moat.

I don’t think they necessarily will be allowed to train on their data unless they get explicit permission. They will try, but the way I see privacy revaluations is that users will have to authorize specific uses of their data and not be surprised by any application.

This could be one of the more interesting privacy fights of the next decade.

I’m sure there are easy cynical takes about how they will just shrink wrap the EULA, and maybe they will. But in a good privacy environment, users should never be surprised and have control over how their data is used. And I think we’ve made some progress there.

> I don’t think they necessarily will be allowed to train on their data unless they get explicit permission. They will try, but the way I see privacy revaluations is that users will have to authorize specific uses of their data and not be surprised by any application.

If there's one company that I don't think cares about user permissions or the law, it'd be Twitter.

The EU officially warned Elon about DSA fines and the response was less than serious.

https://www.cnn.com/2023/10/10/tech/x-europe-israel-misinfor...

China probably has the most comprehensive data on its users from a surveillance perspective

You may not need to train a model to make use of your data though. Maybe a cheap fine tune would work just as well. Maybe just having the data well indexed and/or part of the prompt context is good enough.

Idk. These were trained on pretty public things like Wikipedia.

It's not anymore. If the model is publicly accessible, its skills can be distilled by performing some API calls and recording input-output pairs. This scheme works so well it has become the main mode to prepare data for small models. Model skills leak.

I agree, publicly deployed models seem to be easy to train from. I did say "internally deployed LLM" though. agentcoops said "...where the models in question increase the productivity of workers in their non-ai-related profit centers" above, that's the bit I was thinking about. I think private models, either trained from scratch or fine-tuned, are going to be a big deal though they won't make the PR splash that public models make.

The conclusion for that seems to be that it just yields a model that has the surface look and feel of GPT3 or 4 but without the depth, so the experience quickly becomes unsatisfactory once you go out of the fine tuning dataset.

It seems to be compelling because a genie has been the promise of technological progress forever and let's be honest, it's been marketed as such. Why would people not invest in that unless you were a technological savvy skeptic like yourself.

Because the cost of entry to the market is so absurdly high right now, it is seen as a good investment opportunity. If you throw enough money at it you can make your place no nice and early and then win later by sheer experience in the field. That is the idea.

Possibly unrealistic, but my fear is that they will end up like Netflix more than uber.

Some will scream in horror but I wanted Netflix to be a monopoly. A single place and app and account with all the content I need.

"competition" in streaming space has been nothing but disastrous for me as a consumer. It led to greedy heterogeneous islands of content, with proliferation of crappy apps and pointless restrictions and return to cable package mentality.

Again, Possibly irrationally and ignorantly, my fear is that 5 years from now I'll need a dozen subscriptions to less good services which will hoard their source data and models and be specialized based on which content they got licenses to. I. E. There'll be ai1 with new York times and Wikipedia, and ai2 with Washington post and encyclopedia Britannica, and ai3 with I don't know fox news and RT, and ai4 with mit and Harvard business libraries, and ai5 focused on math with extra subscription to wolfram, and ai6 with rights to stack overflow and JavaScript and so on.

There are many scenarios various writers have posited where we are actually in local maxima lf ll, with data being increasingly closed and or poisoned, and possibly segregated in the near future. :-/

I think this will happen, will be as inconvenient as you envision, but I think it will be for the best.

We're not socially ready for one AI winner. The resulting giant would be too powerful and too influential.

Just like the pirate Bay online cinema is better than every streaming service there could be pirate LLM that uses all the data, maybe it could be even trained by internet users sharing a bit of compute with some program?

Or a cryptocurrency where the proof of work is running a malicious AI...

You mean the basilisk?

Not exactly but it would be a good name for it!

If you run it you get some money and higher balances mean better treatment when it takes over.

Assuming the baseline treatment is very bad, this is exactly what Roko's basilisk is.

RoCoin

Distributed evil AI evaluation in exchange for protection from it. You can trade the protection on the free market.

I don’t think you can really compare that. The streaming service market is not elastic because you cannot easily interchange one series for another.

If other vendors LLMs become good enough it will actually be easily to interchange and then the race for the best UX and integration will be upon us (which the other commenter alluded to).

I think it's a circular argument.

"if other llms are good enough" assumes that in principle they have same opportunities, access to same data, or content. My fear is precisely that this assumption may be taken away - I. E. That news paper publishers or encyclopedia owners or big websites (stack overflow, web Md, etc) will enter into arrangement with specific llm companies - just like Netflix Disney prime etc aren't competing on their app or price or flexibility, but on exclusive underlying content. Nobody WANTS to subscribe to Paramount+ or cbs access... But if they hold enough material hostage some people will 'have to'. I can see a future, not far off, where different llm organizations selling feature is not how good their technology is - to your and overvodys point, THAT moat is likely to even out - but what underlying training data they have legal access to.

> Possibly unrealistic, but my fear is that they will end up like Netflix more than uber.

I am convinced that's how they will end up.

Your Uber/AirBnB/Wework all have physical base units with ascending costs due to inflation and theoretical economies of scale.

AI models have some GPU constraints but could easily reach a state where the cost to opperate falls and becomes relatively trivial with almost no lowerbound, for most use cases.

You are correct there is a race for marketshare. The crux in this case will be keeping it. Easy come, easy go. Models often make the worst business model.

Probably why Altman has been talking so much about how dangerous it is and how regulations are needed. No natural moat, so building a regulatory one.

Only hundreds of billions at stake if successful.

But OpenAI appears to have some sort of data moat. I doubt their model is the best in the world, but more/better data generally beats better model, and GPT-4 definitely beats Claude, Bard, Bernie and the rest probably because they curated the best quality and largest data set. Maybe that moat doesn't last long but perhaps they have exclusive rights to some of that dataset through commercial agreements that could be a more durable moat.

> But OpenAI appears to have some sort of data moat.

I'm willing to bet dollars to doughnuts that Google and Facebook have at least one, possibly 2 or more orders of magnitude more latent training data to work with - not including Googles search index.

My uninformed opinion is that Google and Meta's ML efforts are fragmented - with lots of serious effort going into increasing existing revenue streams with LLMs and the like being treated as a hobby or R&D projects. OpenAI is putting all its effort into a handful of projects that go into a product they sell. The dynamic and headcounts will change if the LLM market grows into billions

> My uninformed opinion is that Google and Meta's ML efforts are fragmented -

It seems more likely that at Google at least they just fell into the classic innovator's dilemma in which they were stuck trying to apply innovation to their current business models in an attempt at incremental innovation instead of seeking an entirely different customer and market.

Yeah, same with Meta.

Both have a large graveyard of failed innovation attempts. Remember Google Buzz? Google Plus?

Continuing the litany: https://killedby.tech/google/

I'm sure some people at google and meta were screaming at the top of their lungs to jump on the ai bandwagon before chatgpt - but you know how things work in large companies.

They're not as good at innovating, that's why they acquire startups all the time. It's a blood transfusion

I got the impression that Google was running Bard on a smaller model with presumably cheaper inference costs. I imagine the unit economics of both Bard and Chat GPT are negative right now and Google is trying to stay in the game without lighting too much money on fire.

Google and Facebook are not interested in pooling all their resources in order to build the next big thing. They are just interested in doing “enough” so that people keep using their platforms. The race is about how often a day every person on the planet spends on either google or Facebook/instagram. It’s about who is “the homepage of the internet”. They just need to be good enough so that traffic doesn’t move off to chat gpt.

I think threads is a cover to get more natural language data from actual people for free, but maybe I’m wrong

Facebook.com already has decades-worth of natural language text and audio/video from uploads and "live" sessions. That is a deep pool, and wide too because Facebook probably has content in all currently-spoken natural languages, with the exception of those exclusively used by uncontacted peoples. That is a data moat.

I'd bet that there are any number of submarine startups out there sitting on top of full downloads of Common Crawl, archive.org, etc. who are only too happy to let OpenAI be the first penguin off the iceberg.

If OpenAI survives all the legal challenges, they'll just click "Go" and be in business in weeks to months.

If OpenAI gets smacked down, they haven't lost much.

There are probably also some submarine operations that are already doing/have done the training. If OpenAI gets bankrupted for copyright violations, we'll just never hear from those.

You might be forgetting the cost to train a 1.8T model.

The cost is peanuts compared to the potential profit. Apple/Google/Facebook could absolutely eat the costs for a skunkworks project to do that training and just sit quietly waiting on the yes/no from legal.

But the question is rather of the cost are peanuts compared to expected cash flow for the next 12 months.

>submarine startups out there sitting on top of full downloads of Common Crawl, archive.org, etc. who are only too happy to let OpenAI be the first penguin off the iceberg.

Not sure any startup would be ok with sitting on a potentially gamechanging model.

Google/Facebook? Maybe?

Amazon's LLM efforts are pretty much in shambles

> I doubt their model is the best in the world

For what reason do you doubt that?

I was using Bard and chatGPT in parallel, but lately I just default to Bard. To me it's a better model with much more accurate answers, while chatGPT just gives you bombastic words.

Just heard Steve today from Builder.io who did an impressive launch of Figma -> code powered by AI.

They trained a custom model for this. Better accuracy, sure, but I was a little surprised to watch how much faster it is than GPT4.

Based on their testing, they’ve become believers in domain specific smaller models, especially for performance.

Codegen is an area that seems to have the best breakout performance compared to the big FMs. In hindsight it should be obvious given that openai created Codex.

However, I am skeptical that smaller and more focused models will do well for more general tasks. I’ve found gpt-4 to just be flatly excellent for tasks that involve emitting a custom DSL and customer-provided business context (the latter being something we cannot train for, not without immense expense).

>hungry-hungry-hippos market grab

That is a beautiful metaphor, seriously.

This point is discussed in the article. Title is not for Google/Meta, they'll invest all the billions that they have to.

It is for the consumers of these models, is there even a point to train your own or experiment with OSS!

Sure, open models often require much less hardware than chatGPT3.5 and offer ballpark (and constantly improving) performance and accuracy. ChatGPT3.5 scores 85 in ARC and the huggingface leaderboard is up to 77.

If you need chatGPT4-quality responses they aren't close yet, but it'll happen.

I was kind of curious because I’ve often felt the same about cheap Uber/Lyft rides. So I checked. I initially signed up with Lyft sometime in late 2015. The first airport ride I got from my house in Austin was in December of that year and they billed me $38 base fare and $1.50 airport surcharge. This was before tipping was available in the app as well.

I just took Lyft again to the airport earlier this month same location and I was billed $49 USD, and a $1.30 “Texas Surcharge”.

An inflation calculator says that $38 usd in 2015 is equivalent to $49 in 2023. Color me surprised. I thought the prices had significantly increased since I signed up but it looks like actually no they didn’t.

Trawling back through those old emails I do see constant “50% off all weekday rides” offers from the time I signed up until about March 2016, at which point they stopped. So there were some subsidized incentives when they were early in Austin but it looks like they stopped sometime in early 2016. So if the money train existed, it happened before that, at least in Austin.

> The premise of this is flawed. OpenAI is cheap because of has to be right now. They need to establish market dominance quickly, before competitors slide in. The winner of this horse race is not going to be the company with the best performing AI, it’s going to be the one who does the best job at creating an outstanding UX, ubiquitously presence, entrenching users, and building competitive moats

I hate this. Not the best will win, but the one with the biggest pockets. Nothing that helps with technofeudalism. Proper competition would be good.

I am about as entrenched to OpenAI as I am to Uber.

I like Uber; it’s convenient and fairly reliable. Five milliseconds after Lyft creates a better experience, I can switch.

I think the main difference is that OpenAI doesn't have network effect. AirBnB/Uber/any social network wins because you want to be where everyone else is. ChatGPT is great, but I can switch tomorrow to something else without any issues.

Right exactly. The founder is basically the king of this business model. Should not be a surprise to anyone!

One mayor difference between Uber and Open AI is that fundamentally openeyes technology will get cheaper for them to run, hardware wise and software wise. They just need to hold their position long enough for variants of Moores law to kick in.

the hardware these LLMs run on isn't going to get 10x faster/cheaper in the span of a couple years, it will get incrementally faster at the cost of having to buy new expensive datacenter GPU hardware. It's not going to magically save them from losing money on serving requests.

This is true. The fields are green and lush with things that don't scale: freemium and low prices to get people hooked. The crack dealers will almost inevitably go full Unity when they are forced by their board of directors to turn a profit.

Silly argument when the Uber cycle has yet to achieve durable profits via monopoly

I don't think Uber and AirBnB are good comparisons.

Both are B2C and have network effects.

> They are ... as likely to be the next MySpace as they are Facebook.

I will happily take a 1:1 bet on them not going the way of MySpace in, say, 5 years!

OpenAI is not cheap, it’s selling at lost.

Not if they pass that sweet sweet regulation. It's sure to start adding additional expenses to start-ups.

Greater uptake -> more data -> better AI

If anything is going to be based on how good the tech is, it's this.

ChatGPT integration is a single API call, I don't think they have much of a lock-in right now.

Completely wrong, the best AI will win. There is insane demand for better models.

There is insane demand for good enough models at extremely good prices.

Better beyond a certain point is unlikely to be competitive with the cheaper models.

Yes, this is correct. And OpenAI are currently the best at this.

Yep, quality over quantity. The difference between 99.9% accurate and 99.999% accurate can be ridiculously valuable in so many real world applications where people would apply LLMs.

Only Big Tech (Microsoft,Google,Facebook) can crawl the web at scale because they own the major content companies and they severly throttle the competition's crawlers, and sometimes outright block them. I'm not saying it's impossible to get around, but it is certainly very difficult, and you could be thrown in prison for violating the CFAA.

I'm not sure if training on a vast amount of content is really necessary in the sense that linguistic competence and knowledge can probably be separated to some extent. That is, the "ChatGPT" paradigm leads to systems that just confabulate and "makes shit up" and making something radically more accurate means going to something retrieval-based or knowledge graph-based.

In that case you might be able to get linguistic competence with a much smaller model that you end up training with a smaller, cleaner, and probably partially synthetic data set.

Common Crawl claims to have 82% of the tokens used to train GPT-3, and it's available to anyone.

Add all the downloadable material at archive.org and you've got a formidable corpus.

https://commoncrawl.org/

The improvements seem to be leveling off already. GPT-4 isn't really worth the extra price to me. It's not that much better.

What I would really want though is an uncensored LLM. OpenAI is basically unusable now, most of its replies are like "I'm only a dumb AI and my lawyers don't want me to answer your question". Yes I work in cyber. But it's pretty insane now.

GPT-4, correctly prompted, is head and shoulders above everything for coding. All the text generation stuff and NLP tasks, it’s a toss-up.

I haven't played with the self-hosted LLMs at all yet, but back when Stable Diffusion was brand new I had a ton of fun creating images that lawyers wouldn't want you to create. ("Abraham Lincoln and Donald Trump riding a battle elephant." It's just so much funnier with living people!) I imagine that Llama-2 and friends offer a similar experience.

It's way better at writing code, or at least I've found it to be so.

Depends how you define quality. This paper reflects my own experience

https://arxiv.org/abs/2305.08377

and shows how LLM technology has a lot more to offer than "ChatGPT". The real takeaway is that by training LLMs with real training data (even with a "less powerful" model) you can get an error rate more than 10x less than you get with the "zero shot" model of asking ChatGPT to answer a question for you the same way that Mickey Mouse asked the broom to clean up for him in Fantasia. The "few-shot" approach of supplying a few examples in the attention window was a little better but not much.

The problem isn't something that will go away with a more powerful model because the problem has a lot to do with the intrinsic fuzziness of language.

People who are waiting for an exponentially more expensive ChatGPT-5 to save them will be pushing a bubble around under a rug endlessly while the grinds who formulate well-defined problems and make training sets will actually cross the finish line.

Remember that Moore's Law is over in the sense that transistors are not getting cheaper generation after generation, that is why the NVIDIA 40xx series is such a disappointment to most people. LLMs have some possibility of getting cheaper from a software perspective as we understand how they work and hardware can be better optimized to make the most of those transistors, but the driving force of the semiconductor revolution is spent unless people find some entirely different way to build chips.

But... people really want to be like Mickey in Fantasia and hope the grinds are going to make magic for them.

> Remember that Moore's Law is over in the sense that transistors are not getting cheaper generation after generation, that is why the NVIDIA 40xx series is such a disappointment to most people.

I am unconvinced by the idea of trying to redefine Moore's Law to be about MSRP. The NVIDIA H100 has twice the FLOPS of the A100 on a smaller die. That's Moore's Law, full stop. When NVIDIA has useful competition in the AI space, they'll be forced to cut prices, as has reliably been the case for every semiconductor vendor for the last 60 years.

Agreed. We need Intel to get the software side of ARC together. Or we need something like the unified ram or apple silicon. Apple have accidental stumbled into being the most competitive way to run llms with their 192gb studio.

Yeah, everyone has their own "Moore's Law" that's over, but original Moore's is kicking along.

If you look back just 2 years we had the grinds build those specialized models for QA, NER, Sentiment, Classification etc. and all their deep investment was rug-pulled by GPT-3 and then GPT-4.

You say that training datasets will win, but this is where OpenAI is currently have a big leg up: Everyone is dumping tons of real data into them, while the LocalLLM crowd is using GPT-4 to try to keep up.

We will see who is faster.

In addition to what the other commenter said about Moores law, innovations like Flash Attention which reduced memory usage by over 10x and FA 2 which made huge leaps in compute efficiency show there is still a lot of room to improve the models and inference algorithms themselves. Even without compute we likely haven’t scratched the surface of efficient transformers.

Moore's law is irrelevant. Large language models are going to leave the digital paradigm behind altogether.

Neural nets don't need fully precise digital computing. Especially with quantization we're seeing that losing a bit of precision in the weights isn't impactful. Now that we're serving huge foundation models with static weights there's an enormous incentive to develop analog hardware to run them.

Mark my words, this will lead to a renaissance in analog computing, and in the future we will be shocked at the enormous waste of having run huge models on digital chips.

Just think, how many multiplications per second is the light refracting through your window right now clocking? More or less than is required to ChatGPT do you think? If only the crystals were configured correctly and the patterns of light coming through could be interpreted...

> Just think, how many multiplications per second is the light refracting through your window right now clocking?

It really depends where you draw the lines, because you could also say that one single transistor in my electrical CPU is doing a kerjillion calculations for all of the atoms and electrons involved.

Exactly. The important quantity is how many calculations are being used.

Yes! Prompts are super finicky.

You have to create a prompt/function that for a wide set of inputs, generates a token sequence that will perpetually expand in a manner that corresponds to an externally observed truth.

Way too often it feels like you have to shove a universal decoding sequence into a prompt.

“Talk your steps, list your clues, etc.”

Just trying to luck into a prompt that keeps decompressing the model/ generating the next token that ensures the next token is true.*

I recall there was a paper with a relevant title recently… https://arxiv.org/abs/2309.10668

Basically - LLMs don’t reason , they regurgitate. If they have the right training data, and the right prompt, they can decompress the training data into something that can be validated as true

——-

* Also this has to be done in a limited context window, there is no long term memory, and there is no real underlying model of thought.

Fresh approaches to AI hardware are emerging, like the Groq Chip which utilizes software-defined memory and networking without caches. To simplify reasoning about the chip, Groq makes it synchronous so the compiler can orchestrate data flows between memory and compute and design network flows between chips. Every run becomes deterministic, removing the need for benchmarking models since execution time can be precisely calculated during compilation. With these innovations, Groq achieved state-of-the-art speed of 240 tokens/s on 70B LLaMA.

Fascinating stuff - a synchronous distributed system allows treating 1000 chips as one, knowing exactly when data will arrive cycle-for-cycle and which network paths are open. The compiler can balance loads. No more indeterminism or complexity in optimizing performance (high compute utilization). A few basic operations suffice, with the compiler handling optimization, instead of 100 kernel variants of CONV for all shapes. Of course, it integrates with Pytorch and other frameworks.

https://youtu.be/hk4QGpQAvSY?t=57

It's too early to say who is winning/will win, of course. But so far the UI and its accessibility have made a huge difference in how different gen AI models are being used.

For example, I struggle to see DALL-E winning over Firefly if Firefly is integrated into a very rich environment, whereas DALL-E is basically prompt UI only (while DALL-E 3 is a better model IMO).

The PI iPhone app has a solid UX and even better UX if Apple (bought it) integrated into Siri.

How would a potential competitor obtain an equivalent body of training data?

OpenAI has been the largest unintentional producer of LLM datasets. Everyone is leaching on GPT-4 even though the license says "no, no!".

I use Uber in SF all the time, and while it's absolutely more expensive than it was during those go-go years, it's actually an even better experience than it was before (specifically on how fast it is to get a driver near you).

You must be experiencing another SF than me...

What's gotten worse in your experience?

> Someone try calling an Uber in SF today and tell me how much that costs you

Yet...what is the biggest U.S. ride share app?

Hungry Hippos is true. But it also works.

It’s a funny point. After only taking Ubers and Lyfts all my adult life (young), I have recently switched to cabs because they are cheaper in my city (and I tell everyone I know to do the same). They have dominance now but if I had to guess whether cabs or Uber will still exist in 100 years… I know which I would bet on.

Cabs are an idea. Uber is a company. I can't think of any company that I'd predict to last 100 years. Even Google and Apple probably won't last 100 years. I honestly wouldn't even bet on cars being a dominant form of transportation in 100 years.

I think Apple will last 100 years. See their latest AR product. They continue to create amazing products that are relevant to the masses.

You're probably right. But FWIW it's not even to the halfway mark yet.

I had the opposite impression on AR. A useless gimmick no sane people will use.

Apple did good with the M processors though

Never underestimate the allure of status symbols. For the longest time I thought the same thing of the iPhone. Why would people spend so much when they can get Android for considerably cheaper ? It's the status stupid. Of course, now they are ubiquitious and no longer really much of a status symbol, but if you go back a decade, you'll know what I'm talking about. The same will happen with VR as the tech improves and prices drop a bit.

They aren't that ubiquitous in the UK/Europe or Asia. I've never used an iPhone. I'm not sure of the market share figures but I'd guess it's about 50/50. Similarly with Mac laptops you get the impression that everyone buys apple, but it's really not the case and it's probably more like 75% windows at a guess.

> Even Google and Apple probably won't last 100 years.

I'm not sure, there are 100+ yo companies like Kodak, Nikon, IBM, Panasonic, GSK, Merck, etc. With that in mind, it's not hard for me to imagine that some of the tech giants will also have a presence in hundred years, maybe not as dominant as today, but they could survive.

> I can't think of any company that I'd predict to last 100 years.

Coca-Cola, Nestle, and Unilever. Easy bet. Coca-Cola is already 137.

Uber's surge pricing is dumb in places where they're competing with traditional taxi ranks. I usually pay $35-$45 for an Uber to/from the airport, but if a couple of flights land at once, suddenly Uber is $85+ and the taxis are only $55.

Sounds to me like it's working perfectly.

The point of surge pricing is to rebalance supply and demand when demand for rides outstrips supply of drivers, whether by attracting more drivers or by discouraging price sensitive riders.

Some riders at the airport will strongly prefer Uber, for whatever reason, so they're less price sensitive than you. Because you're happy to substitute an Uber for a taxi, you decrease the demand as a response to the surge pricing, preserving the limited supply of drivers for the riders who really want them (or are at least are price insensitive enough to pay for that privilege).

> They need to establish market dominance quickly

It's not clear there's any more of a long-term market for this any more than there is for compilers. I'm kind of scratching my head trying to figure out where this assumption there is one comes from.

Reply on news.ycombinator.com