Learnings:

There's A LOT of data on the internet, good luck getting through all of it. These models show interesting recency bias.

  - The model seems to be aware of its training and how it has changed over time?
  - Due to model size it seems to easily forget information gathered during training so every sample provided is picked in a cycle from the datasets above.
  - The `InterleavedDataset` when `shuffle=True` messes with the above constraint so needs to be fixed.
  - JSONL became necessary due to datasets taking many GBs on disk, currently it's using a primitive method of simply streaming lines from disk during training (offsets are calculated when the datasets are loaded)
  - It displays interesting abilities to reason even at this size.
  - It performs best at summarisation when the information is provided.
  - It hallucinates considerably.
  - Story-telling is very "we went here, then we went there" but there's no "depth" - it's just a sequence of events.

Future work:

I'm going to continue hacking away at this project as I want to see where it leads but I'm very keen to get it out into the wild when time permits. (it's exhausting trying to keep up with this field)

Some examples below:

>>>

Human: Can you help me with a puzzle?

I bought sushi for lunch but I don't remember where I left it.

Here are the steps I took before it went missing:

    I came home, tossed my keys on the table and brought my lunch to the kitchen.

    I took my lunch out of its bag and threw the bag in the bin.

    I put my lunch on a plate then went to my bedroom.

    My phone rang so I went back to the kitchen.

---

Where did I put my sushi? Please think step-by-step.<|endofprompt|>

Assistant: ->Based on the steps I took before it went missing, I came home and placed my keys on the table. I then placed my lunch on a plate then went to my bedroom.

I put my lunch on a plate then went to my bedroom.

Took: 0.7614264488220215s

>>>

Human: Read this sentence: "The Parakeet hopped out of its cage and landed on the table."

Pretend you're the Parakeet. What do you see around the room after you hop out of your cage? Please be descriptive.<|endofprompt|>

Assistant: ->As I look out the window, I see a small room with a large table surrounded by chairs. The walls are lined with various items that make for an immersive experience. The furniture is made of high-quality wood and has a sleek design that adds to the overall ambiance. The table is adorned with colorful paintings and sculptures, giving the space a cozy and inviting feel.

The furniture is made of high-quality wood, which adds to its charm and sophistication. The chairs are made from high-quality materials that add to the overall elegance and sophistication of the space.

The tables are arranged in a way that creates an elegant ambiance. The chairs are arranged in a way that allows for easy movement around the room, making it easy to find what you're looking for.

Overall, I am sure you'll find something that will leave you feeling refreshed and rejuvenated after your workout.

Took: 5.08751106262207s

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

    Human: Please create a website about a new Robot Cafe. Show the entire HTML script.<|endofprompt|>
    
    Assistant: -><!DOCTYPE html>
    <html>
    <head>
        <title> Robot Cafe</title>
        <style>
            body {
                font-family: Arial, sans-serif;
                font-size: 16px;
                line-height: 1.5;
            }
        </style>
    </head>
    <body>
    
        <div class="container">
            <h1> Robot Cafe</h1>
            <p>The Robot Cafe is a new Robot Cafe that offers a variety of products, from electronics to medical devices. The products range from basic medical devices to more advanced medical devices.</p>
    
            <h2>The Robot Cafe is a new Robot Cafe that offers a variety of products, from medical devices to advanced medical devices.</p>
    
            <h2>The Robot Cafe is a new Robot Cafe that offers a variety of products, from medical devices to advanced medical devices.</p>
    
            <h2>The Robot Cafe is a new Robot Cafe that specializes in using natural and sustainable materials. The products range from basic medical devices to more advanced medical devices.</p>
    
            </h2>
    
            <h2>The Robot Cafe is another new robot cafe that specializes in using eco-friendly materials and has a wide range of customizable options. The products range from basic medical devices to more advanced medical devices.</p>
    
            <h3>The Robot Cafe is another new robot cafe that specializes in using eco-friendly materials and has an extensive range of customizable options. The products range from basic medical devices to more advanced medical devices.</p>
    
            </h2>
    
        </div>
    </body>
    
    </html>
    ```
    

Took: 7.056133508682251s