Replit's new Code LLM: Open Source, 77% smaller than Codex, trained in 1 week

I created this as the basis for my origami folding descriptive language. I tried to find something similar, requirements being both well structured and English-like but couldn't find any, so I created it.

The origami folding app will hopefully be out in 2 weeks, so you can see how it's used.

[0] https://github.com/fuzzthink/mation-spec

> We implement near-deduplication in our pre-processing pipeline on top of exact deduplication. We first split the files into words/tokens based on non-alphanumeric characters and remove files with fewer than 10 tokens. Next, we compute the MinHash with 256 permutations of all documents, and use Locality Sensitive Hashing to find clusters of duplicates. We further reduce these clusters by ensuring that each file in the original cluster is similar to at least one other file in the reduced cluster. We consider two files similar when their Jaccard similarity exceeds 0.85.

Near-duplicates are still difficult to measure. So we should expect duplication, and it should be proportional to the number of samples we have (even if the same variance, but I'd wager higher variance with larger duplications).

[0] https://github.com/openai/code-align-evals-data/tree/97446d9...

[1] https://arxiv.org/abs/2211.15533

I wanted to demonstrate what I said above so I came up with some examples of things I think a human would have an easy time implementing but might be hard to implement. BUT a key part is that I expect these to be in the dataset! I just don't expect these to be in hundreds or thousands of githubs because they will be uncommon (but not rare). Also, we'll pretty much ask for few-liners to give the model the biggest advantage we can (errors will compound).

Prompt:

from torch import nn

class LipSwish(nn.Module):

""""

The Swish activation function is defined by a gated linear unit,

where the gate is defined by a sigmoid function and multiplies the input with

a learnable parameter, beta. Beta is initialized as 0.5.

The Lipswish function normalizes the output by the upper bound of 1.1.

""""

    def __init__(self:

        super().__init__()

Discussion: The Swish function should be in the dataset and is a well known activation function (though beta is not in the pytorch version). Despite LipSwish being in the dataset (introduced in 2019 from Residual Flows[0]) it is not common. I could get the code to generate the swish function (initializing beta, and performing the gate) but could not get the code to divide the output by 1.1. I would not expect a human to have difficulties with this.

Okay, so let's try something else that might be a bit more common and older. The same paper uses a concatenated activation function, and those aren't "uncommon". CReLU was introduced in 2016[1] and there's plenty of concatenated activations around since then. The pytorch documentation even uses it as an example[2]. There's far more examples of CReLU (3k python results for "class CReLU" vs 58 for "class LipSwish. Use these numbers as weak hints because search sucks and isn't always accurate).

Prompt:

from torch import nn

from torch.nn import functional as F

class CReLU(nn.Module):

""""

Concatenated version of ReLU. The activation is applied to both the positive and

negative of our input and the result is concatenated.

""""

    def __init__(self):

        super().__init__()

    def forward(self, x):

Discussion: This was a bit surprising, it didn't use functional as we might expect (or hinted). But interestingly it will if we change the class name to "ConcatenatedReLU". I found exact copies on GitHub with the full name (memorization) but the fist page of instances for CReLU I found used functional (I did find one that was exactly the above code, when adding "clamp" to the search, but missing the minus sign. There were plenty of errors in CReLU implementations). Interesting side note: CReLU continues and defines a function CReLU6 with uses the same docstring but clamps with a max of 6 on the positive input whereas Concatenated starts to define a convolutional block (Conv + BatchNorm + ReLU) called Conv2d.

So we have kinda mixed results, and in both cases these are rather odd and probably not what we wanted. We can clearly see that there are issues where a human would not have too much trouble. There's a big issue in these types of problems: we need to memorize a lot of information (otherwise we can't even write code or know library calls) but too much memorization prevents creativity. There is a lot of gray area between the _pure_ "Stochastic Parrot"/"Fancy copy machine" vs a generalized intelligence (with a broad and flexible definition of intelligence). I'd still call them stochastic parrots because to me the evidence suggests that we're closer to the memorization side than the creation side. But that doesn't mean these frameworks aren't useful. We all know a lot of code is boiler plate (otherwise we wouldn't have the joke "copy paste from SO") and these tools can be very useful for that. But I think the utility is highly going to depend on what you are coding for and how you code. If you're doing standard stuff, this probably has high utility to you and can save you a lot of time. The same way writing macros does, but this is FAR more powerful. It can also help novices a lot. Also, if your main errors are reading mistakes (e.g. you're dyslexic) -- this is my largest problem -- then this might make things difficult as you have a tendency to gloss over text and miss minor errors. I also don't think these tools would help if you're a researcher or writing optimized or specialized code. These differences are probably why we see such differences in people's reactions. But it may also be a hint into what people do and how they work when we see who raves and who rants about these.

[0] https://arxiv.org/abs/1906.02735

[1] https://arxiv.org/abs/1603.05201

[2] https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html

Edit: We can also check if code is in the stack[3]. We see that [0] is indeed in the dataset so we know there is information leakage. Interestingly the exact copy I found in the previous comment[4] isn't! (The repo, though the user is)

[3] https://huggingface.co/spaces/bigcode/in-the-stack

[4] https://github.com/bertomartin/stat4701/blob/ec2b64f629cbbf6...

Thanks for the link (which I've now skimmed beyond the abstract). What wasn't obvious to me from the abstract is that different attention heads have different penalty strengths, so if some prediction task requires long range dependencies you might expect one of the less-penalized heads to end up specializing. I wonder what would happen if the penalty for one head is zero? (The paper suggests this might've been tried and just made things worse, but unclear)

I must admit that this is a wonderfully elegant (and interpretable) way to do this... much more intuitive (to me at least, a wannabe practitioner) than all of the trig-based embeddings.

Could you, say, fine-tune the model every week with the latest merges? Every hour?

>You are free to:

>Share — copy and redistribute the material in any medium or format

>Adapt — remix, transform, and build upon the material

>for any purpose, even commercially.

Compare this to the latest release from StabilityAI lab DeepFloyd, "IF", which in addition to various restrictive clauses strictly prohibits commercial use: https://github.com/deep-floyd/IF/blob/develop/LICENSE-MODEL

Repl.it's release is as open as it gets these days, in my book.

I have an expanded list of foundational research that is likely to serve as basis for gpt4 here in my blog: https://kir-gadjello.github.io/posts/gpt4-some-technical-hyp...

Hope it helps!

4000+ upvotes on HN.

https://news.ycombinator.com/item?id=27424195

But the problem is that these models don't exist in a vacuum, and have to go against slightly larger ones that are also compute optimal and use more data, which will definitely perform better.

Then again maybe there is a sweet spot for a model that's small enough to run effortlessly on regular machines while only serving as a control node in an autoGPT style setting, where it fetches the context it can't possibly have from a curated online database to make up for its shortcomings.

Input:

    below is a SQL statement:

    SELECT
      CAST(DATE_TRUNC('week', "t1"."TIMESTAMP") AS DATE) AS "WEEK_START",
      COUNT(\*) AS "EVENT_COUNT"
    FROM "ANALYTICS"."POSTHOG"."POSTHOG_EVENTS" AS "t1"
    GROUP BY
      "WEEK_START"
    ORDER BY
      "WEEK_START"
    LIMIT 2000

    Explain this SQL. Respond in JSON format with the following keys: 
    TITLE, DESCRIPTION, TABLES
    JSON response:

    {
        "title": "Weekly Events Count",
        "description": "Count of weekly events",
        "tables": [
            {
                "name": "POSTHOG_EVENTS",
                "columns": [
                    "WEEK_START",
                    "EVENT_COUNT"
                ]
            }
        ]
    }

https://platform.openai.com/docs/model-index-for-researchers

https://help.openai.com/en/articles/6195637-getting-started-...

See: https://platform.openai.com/docs/guides/code

I'm very excited about everyone doing work even when they're not beating ChatGPT right now, of course.

But how it compares to ChatGPT right now is extremely relevant to lots of people.

It's also become very common to vaguely reference OpenAI's offerings when announcing new models without saying how they actually compare, or only mentioning some small way in which it compares favorably.

(Though it seems to often be that some comment from the article comparing to OpenAI gets promoted to the title when posted on HN, like here.)

ChatGPT ought to be a non starter for many use cases where data cannot be shared with OpenAI or where the copyright situation of the generated output could become too vague.

Having the option of open source models that potentially could be self hosted could make those use cases viable.

Same for questions where you ask “is X good or bad? And why?”. It answers good or bad before it comes up with the reasons. That’s very plausibly ok, but it’s different from how people imagine it works and thinks.

    """python function that returns a random integer between min and max"""
        return random.randint(min, max)


    def gen_random_float(min, max):
        """python function that returns a random float between min and max"""
        return random.uniform(

Pretending I know my Pade from my Maclaurian, I'd follow up with: "use the better Pade approximation".

The software ecosystem is pretty immature, and there are numerous things that need to change before the core technologies are good enough to fine tune competitive LLMs.

I do think fine tuning moderate sized LLMs on your own (pretty expensive) hardware using consumer GPUs maybe possible this year.

Unfortunately all the evidence is that training (as opposed to inference) requires high-precision, and hence high memory. This is something that consumer GPUs for the most part lack. New techniques are likely to be required (eg better sharing of training on low memory GPUs) but it's hard to predict how they will develop.

Sounds a lot like neural networks.

I think libraries are a result of the human brain's limited working memory capacity , while organisms or neural networks aren't so limited in what they can focus on. Perhaps the transformer became successful in syntactic abstraction because it is limited on how many things it can focus on.

Computer libraries are equally a result of limited memory and disk. Javascript is a counterexample of what happens when computing is free (because it runs on someone else's computer)