The alignment portion of training requires you to have upvote/downvote data on many LLM responses. Google’s attempt at that (at least according to the news so far) was asking all employees to volunteer time ranking the responses. Combined with no historical feedback from ChatGPT, they are behind.
I’d argue that tokenization is more analogous to having a losslessly encoded image where you don’t have access to the decoder. Humans don’t perform well on that one.
Backtracking is easily solved with a shortest path algorithm. I don’t see any need for masking if you are simply maximizing likelihood of the entire sequence.
You may be shocked to hear this but dijkstra’s short path algorithm is the technical answer to this question. We just don’t use it because it’s expensive.
I’d say that any technical limitation that doesn’t apply to humans is not AGI. Context windows are the most apparent ones; humans don’t have a stroke after reading N characters.
Without accounting for data and model architecture, it’s not a very useful number. For all we know, they may have sparse approximations which would throw this off by a lot. For example, if you measure a fully connected model over images of size N^2 and compare it to a convolutional one, the former would have O(N^4) parameters and the latter would have O(K^2) parameters, for K<N window size. It’s only useful if you know they essentially stacked additional layers on top of GPT3.5, which we know is not the case as they added a vision head.
The common free to play guard is you need to play X number of unranked games before you can play ranked. The account is “paid” for with some proof of work.
Easiest way would be to classify the query to go to either Bing proper or ChatGPT. Example:
“What is today’s date?” -> Bing
“Write a rap song about hippos” -> ChatGPT
My interpretation of his claim is it’s no longer a good career move to focus on big-picture research. It’s better to attack problems with narrow scope or superficial impact. Some of those papers are useful, but they’re not the most important problems in practice.
I mean, why not? I have alerts on cost in my cloud usage corresponding to orders of magnitude increases in expected cost. If they trigger a few hours faster (more accurately) that’s good. But I don’t see what I would do bar shutting down the service, so my argument is that it’s not really a feature that would make a difference for most use cases. And if your service is steady, you can already do a linear extrapolation from yesterday’s costs to see if you’d go over budget.
It’s plausible to do what you’re saying. I still think it’s more trouble than it’s worth. If you somehow managed to engineer everything perfectly and tune it as you say, you’d still have customers who wanted their service to stay online past overcharge. I’d think the predominant business case for such tight guarantees would be small and low budget projects. Additionally, every cloud vendor who doesn’t do this seems to be cheaper, so you’d be priced out (in a commodity market).
I agree you can get something working. However, if 1% of the time, a customer is overcharged $100+ because they assumed the limit was guaranteed, you’d probably reconsider if you want to offer that service at all.
See my other reply. The reason I believe it’s architected this way is to have a good tradeoff between timeliness and overhead cost. Full correctness at global scale would involve serializing all billable events through a series of global transactions, which would bring performance to a crawl (e.g., 10ms just to synchronize the metric).
There actually are some fundamental laws related to this problem. It’s a distributed system, so you can lose availability of the metrics data and be bounded by the CAP theorem. You’d also need to keep the metrics synchronized at some granularity across zones, effectively serializing all billable events at a global scale. For example, if you wrote one byte into storage across the globe, that write would have to be billed, committed, and be globally visible before the next byte could be written and billed. And each communication is bounded by speed of light, etc. You’d effectively be synchronizing all billable events through a database at cloud scale.
The latency to query cost is on the order of hours on user side so the service would be off the limit by the cost rate * cost latency. That’s my guess for why it’s not a feature.
The system approves requests by how trustworthy you are. It does this so:
1. You can’t denial of service the cloud or something else
2. Launder money through mining cryptocurrency
3. Take popular instances away from high profit customers
Usually grad students get projects to investigate or prove a particular idea. It’s especially straightforward to do this when the advisor is iterating on a line of ideas, and you resume where the last grad student left off. There isn’t a point in building anything other than to test the idea, so you don’t want feedback to bias you in what direction to go.