Sorry, I have to ask this: how does this compare to ChatGPT?
What's important is that they're preparing for the future by building all the tooling/UI/UX around coding copilots. This way, when costs and feasibility of building ChatGPT-quality LLM's drop and multiple open-source models are available, Replit has the ability to immediately drop them into their production environment. They'll also have the skills and systems to finetune any new models and wring extra performance out of them.
This is more important to users than it seems at first because current UX of things like GitHub Copilot don't allow me to use their AI against my codebase the way that I want to (the way I use ChatGPT). Right now GitHub Copilot is a glorified auto-complete, but I want it to do widespread scaffolding, refactoring, and analysis across my whole codebase. Microsoft has access to LLM's that can do this through their control of OpenAI -- but Microsoft lacks the tooling/UI/UX to bring the power of ChatGPT to me as a user of VSCode/IntelliJ/PyCharm/Visual Studio.
So if Replit can find more innovative, boundary-pushing ways of integrating LLM's, they won't necessarily need the highest quality LLM's to produce a superior user experience. It's a strong signal that Replit is well-positioned for the future, when ChatGPT-like models are democratized.
Hopefully JetBrains is paying attention. They definitely have time to wait a bit more (1-2 years?), but not a lot of time. JetBrains shouldn't solely rely on Github Copilot plug-in to provide their users with LLM's, because it's not clear that the user experience of that plug-in will stay competitive with the user experience that GitHub Copilot will offer directly in VSCode. The IntelliJ/PyCharm plugin may remain "just a fancy auto-complete" while VSCode gets more interactive workflows.
Future IDE's with LLM integration require novel, smart, clever UX typically invented only by very creative people.
It's also worth noting that Replit is not just trying to be an IDE -- they're also building a marketplace to buy/sell coding work, and establishing a small foothold as a niche cloud computing provider.
Highly recommended.
Generally we have continued finding that the more "other"/general stuff an AI model is trained on, the better it performs on specific tasks. As in, an AI model trained to identify photos of all animals will perform better than an AI model that is only trained to identify breeds of dogs. Even at identifying breeds of dogs.
Taken to the extreme, we've found that training image models with "multi-modal" LLM capabilities improves their ability to identify dogs/etc. A lot of people don't realize that GPT-4 is actually multi-modal...while OpenAI has only allowed API access to use text input, the model itself can also accept image input.
Note that we've moved on from ImageNet-style tests "Choose the most appropriate label for this image from 200 possible labels" to much more advanced "Reasoning" tests[0]. PaLI[1] is potentially the SoTA here but BeIT-3[2] may be better example for my thesis. Notice that BeIT-3 is trained on not just images, but also trained like an LLM. Yet it outperforms purely image-trained models on pure-image tasks like Object Detection and Semantic Segmentation.
More importantly, it can understand human questioning like "What type of flowers are in the blue buckets of this image?" and respond intelligently.
0: https://paperswithcode.com/area/reasoning
1: https://arxiv.org/pdf/2209.06794v2.pdf
2: https://paperswithcode.com/paper/image-as-a-foreign-language...
3: http://www.incompleteideas.net/IncIdeas/BitterLesson.html
Not if that tool is censored, and you need an uncensored version to do your work. Or maybe you have privacy considerations, or your company policies forbid using something hosted remotely or owned by another company, etc...
Currently you don't really use LLMs for designing the structure, just completing the implementation, and I think that will be very doable locally.
I'm very excited about everyone doing work even when they're not beating ChatGPT right now, of course.
But how it compares to ChatGPT right now is extremely relevant to lots of people.
It's also become very common to vaguely reference OpenAI's offerings when announcing new models without saying how they actually compare, or only mentioning some small way in which it compares favorably.
(Though it seems to often be that some comment from the article comparing to OpenAI gets promoted to the title when posted on HN, like here.)
ChatGPT ought to be a non starter for many use cases where data cannot be shared with OpenAI or where the copyright situation of the generated output could become too vague.
Having the option of open source models that potentially could be self hosted could make those use cases viable.
OpenAI probably hasn't gone through all the SOC2/etc/etc/etc/etc audit certification that AWS/GCP/Azure have, but if you're using those, then this decision is just a matter of degree. Plus OpenAI is clearly aware of the concerns and beginning to address them in order to expand their addressable market.
For defense companies, yeah, this is a non-starter. But they often don't even have access to StackOverflow and cell signals are physically, purposefully blocked by the building construction materials. And they only recently even started using cloud computing and use a purpose-built cloud at Azure/GCP/AWS that's specifically walled off for DoD partners.
Saying that they're "working on it" is not useful IMO - at the end of the day, they'll be exactly as unethical as they can get away with. We live in a time where we can comfortably say that that is "very unethical".
How is it different from storing all the sourcecode on a private GitHub.com repo?
Input:
below is a SQL statement:
SELECT
CAST(DATE_TRUNC('week', "t1"."TIMESTAMP") AS DATE) AS "WEEK_START",
COUNT(\*) AS "EVENT_COUNT"
FROM "ANALYTICS"."POSTHOG"."POSTHOG_EVENTS" AS "t1"
GROUP BY
"WEEK_START"
ORDER BY
"WEEK_START"
LIMIT 2000
Explain this SQL. Respond in JSON format with the following keys:
TITLE, DESCRIPTION, TABLES
JSON response:
{
"title": "Weekly Events Count",
"description": "Count of weekly events",
"tables": [
{
"name": "POSTHOG_EVENTS",
"columns": [
"WEEK_START",
"EVENT_COUNT"
]
}
]
}https://platform.openai.com/docs/model-index-for-researchers
https://help.openai.com/en/articles/6195637-getting-started-...