GPT-J-6B: 6B JAX-Based Transformer | Hacker News Reader

GPT-J-6B: 6B JAX-Based Transformer

arankomatsuzaki.wordpress.com

144 pointskindiana79 comments

Lol, I fed it this "Calorie counting works as follows..."

And it immediately tried to sell me an ebook. For real.

"The biggest danger with this diet is that you'll probably end up counting calories and then feeling deprived so you want to eat more and more. This cycle will keep you yo-yo'ing right back to the point where you started and then make you feel much worse. I have made this mistake myself and I know that I want to make sure I don't repeat that mistake.

To get people started with the 21-day Fast Weight Loss Guide, I am offering it for just $1.99 on April 29, and every day of the month after that until the program is sold out, and it will be up to the discretion of the buyer if they choose to wait and pay more"

Sorry guys, I think I broke it. I wanted to test it on my motivational speech,

Even moving the ball forward is a win at this company, so I wouldn’t worry too much. You are in what’s called a blood pit. This is a room in most abattoirs below the killing floor where runoff of various kinds is collected for purposes of selling to the protein shampoo industry (mostly). Above you is a grated floor where cows roam around until they are LR’d. Every now and then, a still-ambulatory, not yet LR’d cow accidentally shits out a nugget of something valuable that it accidentally ingested, and your job is to run from place to place in the blood pit, avoiding as best you can droppings of various consistencies from above (this is impossible to do perfectly) while pocketing nuggets. If you manage to pocket a few nuggets in a day’s work, that is a good day.

Unfortunately, it seems to have gotten too depressed to run after this.

That's a casual chat of two strogg floor managers on their quake2 plant.

The practice of de-beaking, for the purposes of rendering a quicker death to the entire herd, is a part of the job. The most the boss will say is “Do you have any questions?” There are no expectations of cooperation, no in-universe questions to ask or mysteries to solve, just deadly work to be done.

When you get sent to the feed room, you are told, “You are going to have to steal some feed.” You are given a specific passcode (which you are not allowed to ask for or demand to be changed) and are told that you will be given one eight hour time period to complete the task. If you find that you can get to the feed room more or less quickly, you are told to report back to your boss. If you do the task in the time allocated and return the approved bit of feed you were given, there will be no consequences. However, if you exceed the time you were given, there will be consequences. You will be sent home for two weeks.

This is how things work until you show an ability to actually do what you are told. You are never told what the consequences will be, nor is it clear how the

This is simply incredible, I would love to read something novel-length like this. But unfortunately it also generates a lot of banal text, as in some of the other examples in this thread. I wonder if there is a way to optimize for the weird shit.

Convince someone with a really big pile of TPU credits to train a larger-size model on the full text of the SCP Wiki :>

This could even potentially be interesting enough to attract (small-scale) Kickstarter-level attention.

Woudn't retraining an existing model on the SCP data suffice?

Well I don't think that would optimize for weird but good prose. Maybe you could train it on Cormac McCarthy or Thomas Pynchon or something like that

This sounds like a verbatim copy of something I’ve read on Reddit a few years ago.

I get the feeling that now that it's so easy to generate text that sounds legit and is syntactically correct but makes no sense semantically, web search will be unusable soon.

Searching for answers to an obscure problem with a software library now already leads to 5 Stack Overflow ripoff sites that at least contain the same text like the original. If these scammers can now generate their own texts (randomly including all possible keywords I might use to narrow my search), we will drown in useless information.

And this isn't restricted to programming, this was just an example of a field that is full of scam already. When it gets cheap enough, somebody will do this with any topic that people are interested in...

I wonder how Google and other search engines will tackle this problem.

Oh my god. I fed that text to the model and asked what we can do about this. Here are its answers:

It is not so much the search engines that are at risk, but the people themselves. In a world where anyone with a $0.05 budget can just make something up, nobody will be able to trust anything anymore. It is not the search engines that will change the way we search, it will be the people, who will stop trusting anything they read. And that is a problem.

The problem is not with search engines. It's with people. Most people who get scammed will not be able to understand the text they see. They may think it's legit because it looks pretty, but they don't understand what the text means. The only way to find out the truth is by doing a little research and contacting the company to ask for a refund or to talk to their support. But, with enough of these scams, people will stop trusting anything they read.

Begun, the AI war, has.

This does make me wonder, how long until we get a GPT that can drop pop-culture references smoothly enough to pass the Turing Test?

It's harder than other types of text generation, because part of the game of referring to memes is to do it in as roundabout a way as you can, using the kind of hidden inferences a neural network would probably struggle with.

But it might still be feasible soon. shudders

Yeah you wrote that it made no sense semantically but it does, it rewrote your exact argument. This is great for a non native speaker for example, to rewrite a paragraph in a more standard language. Plagiarism will definitely be harder/impossible to detect.

The last paragraph wasn't mine. It was also part of the output :)

I really like the brief bit of optimism that got brokenly encoded into this response, with the idea that people can find out the truth by first getting refunds for the lies they've been told.

Google uses a lot of other signals besides page content to rank search results. Common examples:

1. Links from well-ranked pages. 2. Whether the user hits the back button or stays on the page.

Many of these should still work.

But they obviously don't work! That's why I mentioned all those Stack Overflow rip off sites as an example. When scrolling the results, my brain filters them out because I can recognise them as scam, but if they would contain unique and relevant text (generated by an AI), I'd be lost.

I actually like the stack overflow rip off sites. They load quicker than the real stack overflow (which often seems to freeze due to broken ad scripts).

Google favors junk websites over Stack Overflow for the same reason it downranks all of its competitors.

The code generation is pretty darn good: https://i.imgur.com/tj3LEzG.png

Here's the full output. Everything up to the first `def ` (line 26) was the prompt. The rest is GPT-J with temperature 0.7 top_p 1.0: https://gist.github.com/shawwn/e93f8af123914a9a1c72812b0a7c4...

It seemed to recognize `from bs4 import BeautifulSoup as BS`, wrote a function to fetch a url, then used that function to parse HTML using the `BS` variable:

  def get_links(website):
      webpage = BS(get_website(website))
      soup = BS(website)
      links = []

      for link in soup.select(".row a"):
          href = re.search(
              r"class=\"gwt-Label\" href=\"(.*)\"",
              link.get("href"),
          )
          if href:
              link.get("href")
              links.append(href.group(1))

      return links


  def get_title(link):
      title = link.get(
          "title",
          "",
          attrs={
              "class": "gwt-Label",
          },
      )
      return title

It didn't get it quite right, but boy was it close.

If you tried generating code but didn't get good results, be sure to set top_p 1.0 temp ~0.8. It matters quite a lot.

These results are terrible. Almost all of the generated text posted here is non-sensical, and playing with it online is just confusing.

There really needs to be a better method of evaluation, an MNIST for transformer text generation. like a list of pre-defined prompts that every GPT-X has to use to generate text, which can be scored against a list of "correct" answers in a variety of ways.

I have no way of knowing if the output of this flavor of transformer is good or not, whatever that would mean. Very difficult to see how it compares to similar models. Setting up a model with this number of parameters and their reported training times is impressive. But I have no idea if this particular number of parameters makes a difference, or what that difference is supposed to look like

Thankfully, there already exist evaluation tasks like that, and Eleuther actually has a project collecting a handful of them together; see https://github.com/EleutherAI/lm-evaluation-harness/

You have an idea of what's sensical and not. It would seem that long range correlations in the text break down. This is true of all of the popular models, even the trillion parameter ones. They just break down at longer distances.

I suspect this is at least partly because of they way they are used (and maybe trained?)

Always, "given a prompt, keep talking." No instructions to go anywhere, so it's no surprise that they do not.

I think, "start with this idea, end with this one" should give much more interesting results. Telling it to start with a premise and come up with the filler needed to draw some conclusion. It would give it more of a target for making long-distance connections.

Otherwise you just get this open-loop blabbering, which I agree seems really useless. With a more "directed" model I can see this having actual applications, like with story writing or interactive video games. But as it stands this seems totally uninteresting from an applications point of view.

Still upset about the pile as a dataset (the one used to train this model) because I wrote an issue asking if they would include my dataset debatesum into the dataset[1]. They mentioned that they planned to do it and even wrote another issue indicating that they would include it in the pile.

Months go by, and I ask why this dataset is still not included. They told me that the pile is complete and will not be updated anymore. I didn't have time to write my own PR for this and had no idea that there was a time limit to get it in.

Makes me sad. I don't know why this company has decided to stop updating the pile...

[1] https://github.com/EleutherAI/the-pile/issues/56

You should fork it! I'd use your fork.

They're not a company. They made some decision to not add more to the Pile, as they think it is complete enough. However, you could go to the Discord and propose a restarting of the Pile project, they allow those. On the Github, it looks like the next step after a restart would be the Pile v2, a multilingual dataset. Good luck!

Accompanying blog post https://arankomatsuzaki.wordpress.com/2021/06/04/gpt-j/

To test it out: https://6b.eleuther.ai/

Some sample output I got (yes, this is real):

> You can travel through the universe to the furthest reaches of the unknown, where the background radiation of the early universe grows in intensity as you go further and further away from where you started. Somewhere beyond that radiation lie the last remnants of the beginning of the universe. Here there is an ‘ultimate singularity’, an instant of time that doesn’t have a name. At that point the laws of physics as we know them all break down. There is no matter, energy, or anything else; there is just an unbound, unmeasurable amount of energy.

> The laws of physics are fundamental constants that set the parameters of our world. But they seem arbitrary, without foundation. What if the laws that shape the universe are themselves arbitrary, like the laws of grammar and vocabulary, and just the result of our limited human brains? What if there is no underlying logic that holds it all together?

It still amazes me. I ask it to write something, and when it does, I just want to ask it "dude, wtf?" Seriously, it's better than what shitposters on 4chan write. It's pure genius:

I was watching a climbing competition. I had never seen this before, and I am the world's worst fan of anything outdoors. The crowd was absolutely silent. Like, what, were they dead? It was eerie.

Cameo was a perfect example of the sport we love, because of the heavy concentration and the freedom it allowed the athletes, compared to what we do in our real lives. A top-ten contender got drilled off the wall by two burly volunteers, then grabbed an arm in an attempt to pull himself back onto the wall. The brawn of the competitors kicked in, and they were yelling and screaming, being half-carried away from the wall. In short order, everyone was pulled off the wall, the fight was finished, the duo were later ejected and the winner was declared.

Was that weird? No, of course not, that's what we all love. It's the circus, isn't it?

I input the following: "dogs are superior to cats in every way". and the model generated the following...

dogs are superior to cats in every way. They are better at watching TV, they are better at using their tails as fans, they are better at eating out of bowls, they are better at watching over children, they are better at giving head, they are better at being pets, they are better at eating on command, they are better at looking up at you when you speak to them, they are better at smelling and tasting, they are better at chasing balls, they are better at listening to you, they are better at being groomed, they are better at eating from the floor, they are better at being petted, they are better at eating from your hand, they are better at sleeping, they are better at eating from your hand, they are better at cleaning their own paws, they are better at drinking from your hand, they are better at sleeping, they are better at cleaning their own paws, they are better at sleeping, they are better at drinking from your hand, they are better at cleaning their own paws, they are better at sleeping, they are better at drinking from your hand, they are better at cleaning their own paws, they are better at sleeping, they are better at drinking from your hand, they are better at cleaning their own paws, they are better at sleeping, they are better

I agree with ALL of the above.

Better at giving head?

IN EVERY WAY.

I'll admit I don't particularly want to think about dogs giving head (I can't stop laughing)

Hug of death?

Seems so, at least right now.

Wow. I fed it a blurb about my vanity Lisp dialect, and what came out was an HN thread complete with comments by dang.

Works better with Rust.

There is a Colab that you can run to set up the model with TPUs: https://colab.research.google.com/github/kingoflolz/mesh-tra...

A few demo examples: https://twitter.com/minimaxir/status/1402468460681068544

It's interesting, it seems to get stuck on a sentence or word a lot, then half the output is repeating a slightly varied sentence, or rambling about that unrelated word.

That's the one thing very consistent with Transformer models, even GPT-3.

Tried it with Islamic prompt (in Arabic) بِسمِ اللَّهِ الرَّحمٰنِ الرَّحيمِ

And I got this bit of Wisdom:

As-Salaamu alaikum wa rahmatullahi wa barakatu The world is a single mirror. The whole universe becomes all a single mirror if you observe it with your whole heart, as it really is and nobody else's imagination and interpretation of it. Your spiritual sight will at one time reveal the depth of reality, other times it will reveal a false picture which is reflected by all kind of greed, vanity, falsehood, lust etc. The only way to cross this mirror is by an open heart, and it needs to be constantly cleansed of all these sins. It also needs to be cleansed of all the deceitfulness of this world, which is nothing but a spiritual mirror reflecting your own false vision of your imagination. This all depends on one's intention of which Allah is aware and He always warns the soul in its creation. Those who love Allah and His messengers, and who are guided by the Qur'an are aware of this and they never praise anyone or blame anyone for anything, no matter how great or small. They are always aware and watchful and vigilant in their intents. Those who are only biased by their biased knowledge will never do it and they are very few, as everybody has biased knowledge about

I gave it: "HackerNews is...". That's what I got:

---

HackerNews is...trying really hard to be hip to everything.

Big national media must have gotten word of a new internet meme, because every day there’s a story on it that makes its way to the front page. Many of these stories are so obviously biased they’re ridiculous.

Something about the Bitcoin Cash “fork” that split up from the Bitcoin “mainnet” in November? That’s still Bitcoin. So Bitcoin Cash isn’t Bitcoin. Bitcoin Cash doesn’t count as Bitcoin. What about Bitcoin Cash?

These headlines are a case study in how someone with no Bitcoin experience (or really any real understanding of money) can reach the front page of a big national media site with the headlines they write.

BTC/BCH is Bitcoin. We can all agree on that. Because Bitcoin is the original blockchain, the Bitcoin fork was technically not a Bitcoin fork at all.

In theory, we could run Bitcoin forked into as many different blocks as we want. We could have as many Bitcoins as we want. All those separate Bitcoin blockchains could coexist. There’s more to a blockchain

Crikey, What a lot of fun! Thanks to Eleuther for making this available to play with!

Here's my prompt (I'm into fractals, and I have a feeling that neural nets are loosely exploring in some fairly fractal spaces internally so I thought why not see where it went?):

"Fractals are synthetic or abstract objects, that have the property of self-similarity,"

Result (with temperature at 1.5):

Fractals are synthetic or abstract objects, that have the property of self-similarity, namely that if one portion of the fractal is magnified, it appears exactly like a portion of the fractal itself. It is considered the skeleton of chaos. (Displaying Fractals In art/science/film is expected)Fractals may be abstractly or concretely viewed, painted, sculptured, illustrated or can also be processed through visual media such as cartoons, magazines and animated films, the comic book etc.Though visual depiction is not very vital it may help the audience to take a visual understanding of fractals.When being considered or used, they are very often explored by artists with the power of creating, which means that often-constructed theories and hard labors can be encapsulated through some imaginary concept, man.It is obvious that in today’s technological world fractal geometry or non-Euclidean geometry is gradually becoming and becoming an amazing reality in itself, through special engines developed by people which act like fractal or cosmic molecules.After decades, now people understand the science of fractals and this science creates the equations for identifying areas that become areas of detail or have areas of smooth contours that allow shapes such as spheres, polygons or egyptian shaped icons or mathematic shapes in terms of area.

I loved that little ", man." bit.. made the whole thing transform to being read by "The Dude" from The Big Lebowski! Hilarious complete nonsense! love it!

This is first time for me to see JAX [1] being used by non-Google employee. Is it getting some momentum? Or this person just happened to pick it up for some reason (for, say, better TPU support)?

[1] https://github.com/google/jax

If you’re also curious how it does vs GPT3 here’s the Zero-shot performance evaluations table: https://github.com/kingoflolz/mesh-transformer-jax#zero-shot...

Is it possible to run this on something other than google's cloud platform?

It's design is pretty specific to Google's TPU.

If you ran it elsewhere, you'd likely need to tweak the design or suffer quite a big performance penalty.

Nope, probably not. It's using JAX, which works on both GPUs and TPUs.

All the constants in the design will be tuned to fit perfectly in TPU hardware dimensions.

I'm running it comfortably on my 3090, although it's a really snug fit for the VRAM, and that's with a number of fixes to significantly reduce its memory use from https://github.com/AeroScripts/mesh-transformer-jax .

Out of curiosity, how fast are your inferences with this setup?

With the defaults of per_replica_batch=1, seq=2048 and gen_len=512, a completion takes about 20 seconds.

I'm not sure yet what settings I'll end up with if I decide to play with this more. per_replica_batch=3, seq=1024, gen_len=64 would give an experience roughly similar to the AI Dungeon that I'm used to, though less clever than the Dragon model, and a bit slower at about 10 seconds per batch.

These are always fun to play with. I coaxed it to write a poem, which had a curious disclaimer attached:

  I learned to read and write
  Before I could spell the word "face"
  If you never learn to look
  You'll get nowhere at all
  I taught myself Spanish
  Before I knew how to say "disco"
  If you don't like the rain
  You need not go outside
  I'm learning to dance
  Before I know how to rhyme
  If you never try
  You will never know
  And if you do everything you can
  You will never know why
  
  Disclaimer:
  Opinions expressed in blog comments or posts do not necessarily reflect the views or policies of The Gourmet Live.

[prompt start]

Q: How many eyes a bottle have? A: None

Q: How many eyes a shoe have? A: None

Q: How many eyes a TV have? A: None

Q: How many eyes a dog have? A: Two

Q: How many eyes a man have? A: Two

Q: How many legs a dog have? A: Four

Q: How arms eyes a purse have? A: None

Q: How many heads a woman have? A: [prompt end] Three

Q: How many heads a man have? A: One

Q: How many eyes a cat have? A: Two

Q: How many legs a cat have? A: Four

Q: How many eyes a tree have? A: N/A

Q: How many legs a tree have? A: N/A

Q: How many eyes a mouse have? A: None

Q: How many legs a mouse have? A: Four

Q: How many eyes a plant have? A: N/A

Q: How many legs a plant have? A: N/A

Q: How many noses a cat have? A: Three

Q: How many noses a woman have? A: Two

Q: How many legs a cat have? A: Four

Q: How many noses a dog have? A: One

Q: How many legs a sheep have? A: Four

Q: How many noses a pig have? A: Four

Q: How many legs a sheep have? A: Four

Q: How many noses

I thought it was pretty good!

Industrial society and its consequences have been a disaster for human kind. The rainforest and its inhabitants are under constant threat and not surprisingly, the chief culprit is Western culture and its values. Culture eats itself like a phoenix, if we allow it to take the upper hand. So, the dream to be a selfish, unthinking, care-free consumer who buys what is fashionable and only hopes for what will be useful tomorrow is just that a dream. The selfishness and the uselessness are inherent in our modern lifestyle; it is a self-imposed, social Darwinist vision of human existence.

The problem is rooted in the ever-increasing obsession with consumerism. Wherever and whenever we try to come up with a more, better, safer way of living, we fall prey to inbuilt greed that so often ruins our lives. It is not about the good of all or anything, but about the doing of only ourselves and living the best that we can. We treat every living being on this planet as if it is of no consequence, a mere means to our own selfish ends. At a more basic level, every individual living soul is our basic concern but all other matter is secondary.

Our relationship with Nature is pivotal to life itself. Nature is a solid financial asset in terms of every single drop of water that

Prompt: What's the deal with Rust?

---

Are you interested in hearing about the coolest new language of 2018? Well, Rust might be the one to check out. Rust is more or less an evolution of C++. But Rust is also a radically different design.

C++ is still used in the majority of real-world applications today. With that in mind, what makes Rust so special?

(there the model inserts 5 empty lines for some reason)

To me, C++ is the way to go. Rust is not a perfect language. Rust will not be used in every software.

With C++11 and C++17, the language is starting to get better. New libraries are being created in a rapid way and the language is evolving. With the language used to build web applications, you can easily forget that the language is rather old.

I like Rust a lot, but I see it as a tool. I don't think Rust can replace C++. That doesn't make it a bad thing. In fact, I think that we should replace C++ with Rust, as we did with C. The "new" language that we are using for building web applications is not a new language, but rather Rust.

There is no good reason not to use Rust instead of C++. Today, C

Was it trained on IRC? This is what I got:

is your job going to be nonexistant

for 12 years?

<raju> hi all

<ubuntu-studio> someone please help me with this, I have windows 7 on a usb and when i boot from it my ubuntu 10.10 iso does not start. when i go to my bootloader in my usb and select the install option i get a grey screen with a green bar going across and it

never loads. i have the i386 option. help. thanks

<leeping> When I boot my computer after a fresh install, it takes forever for the system to come up and I don't see any messages displayed. I have a dual-boot setup - Win7 and Ubuntu 10.04, and I think that this is the root of the problem. How can I troubleshoot what's going on and fix this? Should I somehow switch to Grub instead of using the Win7 bootloader?

<raju> can anyone tell me how to install jre on ubuntu 10.04

<FloodBot3> raju: Please don't flood, use http://paste.ubuntu.com to paste, don't use Enter as punctuation.

<Eitan> Hello

<Eitan> I want to

Part of the Pile dataset is the Ubuntu IRC stuff

This is awesome Kindiana! Two quick questions. 1) GPU: Any tips for converting this to run on GPUs (inference-only)? 2) Speed: Any advice for achieving faster speeds? 10-13s is a bit tough for commercial applications. And, would you expect a big performance hit as well from running on GPUs?

Prompt: Emmanuel Macron

---

Emmanuel Macron lors d’une allocution devant les électeurs de Bouches-du-Rhône, le 25 février 2017. BERTRAND LANGLOIS / AFP

Ils étaient jusqu’à quatre candidats à la présidence de la République à visiter le « 48 », ce quartier du centre-ville de Marseille, dans lequel un mouvement revendicatif a été mené depuis le début du mois de février et qui a été déclaré sinon complètement sécurisé, du moins relié d’une façade. Emmanuel Macron, François Fillon, Jean-Luc Mélenchon et Nicolas Dupont-Aignan, et avant eux Benoît Hamon, Marine Le Pen, Jean-Luc Mélenchon, Jean-Luc Bennahmias… Ils se sont retrouvés à la « Nouvelle République », la structure adossée au parvis de la préfecture, pour observer la police et écouter

Translation:

---

Emmanuel Macron during an address to voters in Bouches-du-Rhône, February 25, 2017. BERTRAND LANGLOIS / AFP

They were up to four candidates for the presidency of the Republic to visit the "48", this district of downtown Marseille, in which a protest movement has been conducted since the beginning of February and which has been declared if not completely secure, at least connected with a front. Emmanuel Macron, François Fillon, Jean-Luc Mélenchon and Nicolas Dupont-Aignan, and before them Benoît Hamon, Marine Le Pen, Jean-Luc Mélenchon, Jean-Luc Bennahmias... They met at the "Nouvelle République", the structure next to the square of the prefecture, to observe the police and listen

(translation courtesy of DeepL, with tweaks)

Some interesting points:

- The model notices that something talking about Emmanuel Macron is likely to be a french journal article, so it outputs in French language.

- The model decides that a sentence starting with "Emmanuel Macron" with no other context is likely to be the caption of a photo. At least that's how I interpret the first sentence.

- I checked and I don't "the 48" is a thing in Marseille. I could be wrong.

- The narrative of the text seems a little confused, but is roughly that presidential candidates are visiting Marseille after civil unrest has been quashed by police, and are talking to the police? Accurate details: the date is february 2017, which is campaigning season; Marseille is indeed in Bouches-du-Rhône; people mentioned were presidential candidates, except for Jean-Luc Bennahmias (didn't even know who he was until now). Major plot hole: why would rival presidential candidates meet each other, and speak with the police together, while campaigning?

- Small incoherences that hints at a lack of object permanence: the beginning mentions "up to 4 candidates", the end mentions 7. Mélenchon is mentioned twice (too bad it didn't pick Macron twice instead, there would have been political jokes to make).

- I love that the first sentence ends with "[Journalist name] / AFP". Agence France Presse is the french equivalent of Associated Press, and basically the source of all mainstream news in France. Mainstream journals mostly repackage AFP content for public consumption (which, honestly, probably means they're going to get hit hard by the coming wave of text AI). The fact that the model includes it kind of feels like these image GANs that learn to produce watermarks on the images they generate.

I took one of the questions from this article: https://blog.seekwell.io/gpt3

    Instruction: Given an input question, respond with syntactically correct PostgreSQL. Be creative but the SQL must be correct.
    
    Input: how many users signed up in the past month?

And it gave me a pretty nonsensical answer compared to the output from GPT-3 seen in that article.

Because that answer was pretty weak, I resorted to an easier question... taken from the prison IQ test from the movie Idiocracy.

    You have one bucket that holds 2 gallons and another bucket that holds 5 gallons. How many buckets do you have?

The answer was complete nonsense.

https://youtu.be/jbmq9P-8FiM?t=17

I haven’t properly investigated their dataset they trained on this, but from the responses it gives I would bet that they dataset is skewed towards news articles, press releases, and infomercial or edutainment copy. It does really well when it can write a long impersonal article about something, but doesn’t seem good at the random sort of things GPT3 can do like autocompleting source code.

That makes a lot of sense. I played around with it in German and Italian and the results often times read like they were google translated from English. So there must be lots of auto-translated content from spam sites in the corpus, I guess.

GPT3 can answer questions, in comparison this model works a little bit better than a markov chain.

http://joshmillard.com/garkov/

In the future you don't teach GPT, GPT teaches you. Imagine GPT replacing teacher in school, power of intelligence of all intellectuals in the world in the hands of every kid.

Here’s my test prompt: “Today at a rally in Virginia, presidential candidate Bernie Sanders gave his remarks in front of a large crowd. Here is what he said.”

Below is the output, which aside from talking about New York when my prompt says Virginia is otherwise pretty surprisingly believable:

Bernie Sanders: Well good afternoon, my friends. I want to thank you very much for being here.

You know, I am often asked, you know, can we do well in New York State? I say we can do much better than I think Governor Cuomo expects. The people of New York State have given me a lot of encouragement. And I believe that the people of the State of New York have given this campaign a lot of strength. We’re in a much stronger position to win the primary because of that strength.

When I started campaigning in New York a few weeks ago, there were many people who said, “Okay, you know, we need to win in Nevada, we need to win in New Hampshire, we need to win in South Carolina, we need to win in California.” And we don’t need to do that. We are doing well. We are doing well. I think the people of New York have been very supportive. I think the people of the State of New York have been supportive. We are winning. And we are winning. We are well on our way to winning in California. That means we are winning in the nation.

You know, before I came

(end of model output)

I was curious if it knew anything about me (slim chances) and it does not seem to. It generated a thorough bio that has nothing to do with me. However it’s pretty funny. The prompt is in quotes and the output follows:

“Taylor Alexander is a robotics engineer and “

one of the founders of Transcend, a research lab. Alexander specializes in both manufacturing and consumer applications of robotics. He is also a former roboticist at Carnegie Mellon University, where he studied robotics and human-computer interaction. In that capacity, he worked on a research program funded by DARPA to design a mobile manipulator for soldiers.

The team tested the robotic arm in a variety of physical simulations for landing, handling, and carrying heavy loads like a military bomb. They tested the design on both operating systems—Linux and Windows.

“We built this to test the machine on the two dominant operating systems that we thought might be useful in the future,” says Alexander. “But it actually worked on everything. We saw the same kind of accuracy in the process on all of the different operating systems.”

The team decided to make the arm multi-platform after seeing the arm working so well on Linux and Windows. They were very impressed with the portability. The arm was light enough to pick up a relatively heavy base and, with just a slight rotation, it could hold the base up in the air. The weight didn’t matter to the arm at all.

Since the arm works on Linux, the

It did well. It did well. I think it did well. Because it did well, it will do well.

[Context : Which correlations might we find into the writings of Nikola Tesla and the writings of Fulcanelli ? By comparing these two author's books, what kind of hidden knowledge would emerge ? Are there some innovative patterns to identified ? Are there hypothesis that would be interesting to investigate deeper ? Article :] http://www.sciencebuddies.org/sbe-resources/science-council-... Experiments : https://en.wikipedia.org/wiki/Rhythm_and_Color Possible approaches : In depth research on the imagery in Nikola Tesla's works (what have been determined so far) Comparisons of the imagery in Fulcanelli's “The Spirits’ Book”

I've found many 'hidden codes' in Tesla's writings and have had several interesting discussions with people over the years who found them. When I say 'hidden code', I mean Tesla talks about a particular topic and gives no explanations. While reading the book, I'm seeing the implications of the topic and it's not so obvious as you can just'read off the words' and can't see the'relationship' between the 'hidden' topic and the topic being discussed. Tesla claims that he can make thunderstorms

if he will be in the right position

and later on

I have thunderclouds sometimes, but never more than once a week.

There's no mention of 'position'. I feel like it's a bit of an 'echo' from the Adwa Document

Why I was engaged in the mission I now write of as follows. I had much work to do, many inventions to make, much that was yet to be done. It was necessary to build a vessel, the carriage of which would be out of the reach of the current of atmosphere which is sufficient to destroy the equilibrium of the Globe and cause its destruction.

In this article, he talks about the relationship between Magnetic fields, gravity and Light.

But this light seemed to be generated in space by the Electric and Magnetic Waves, which, as it were, created a sort of opaque Body-Powder which makes a perfect Picture or Picture-Print when the corresponding Picture or Pattern is applied to it by means of the electrical process...

So, the subject of Fulcanelli's "The Spirits' Book" is Light and 'the spirit's' where Fulcanelli has a bit of 'cryptic code' of his own. In The Spirit's Book, there are codes for Pythagoras, Alchemy, Solomon's Seal, Tarot, and Venus. What was found so far are the hidden/hidden patterns in the linked wikipedia articles. These were found after many years of researching into the works of Tesla and the "Spirits Book" of Fulcanelli and I know that in the last two years of research, I've found an new interesting hidden topic. However, it would take a book to cover the topic in any great detail, so, I hope my efforts won't go to waste.

Input:

Cars were abandoned long ago.

The atmosphere couldn't take it anymore.

During The Great Heatwave, survivors migrated to Antarctica, the only place where the climate was cold enough to be outside.

----

Outputs:

Cars were abandoned long ago. The atmosphere couldn't take it anymore. During The Great Heatwave, survivors migrated to Antarctica, the only place where the climate was cold enough to be outside. From the South Pole to the high peaks, The Society was in charge of stabilising the temperature. The Snowmobiles could barely drive in the thick snow and they didn't have enough petrol to move all of them. Sticky packs were more useful. The higher up the snow, the lighter the packs and their packs could carry less to go a further distance. They were a mighty help as long as they didn't get frozen too much on their way to the polar circle, just yet. It was a slow trek, almost impossible for them to overtake. The only possibility was to cross the ice with the slope, finding the easiest way to climb. Snowmobiles didn't last long, their engines lost their energy with the accumulated weight. Snowmobiles became obsolete and no one really knew if there were any working ones in Antarctica. On one of the snowmobiles that had survived The Great Heatwave, the last machine still flying, Perry stayed on it without thinking, much like Elsie.

Perry was the first to say he knew it. The Snowmobile's saddlebags were just large enough for some basic supplies and an extra jacket. They probably couldn't even do a long trip in them. A few packs in an emergency to survive a short-term trip, that's all they could carry. The only reason Perry hadn't thought about it was because he had a plan: if he got to the polar circle, he'd see if the others had arrived there before him. If they weren't, he'd have a better chance of making it. The road was dangerous and Perry was a superb snowmobile driver, much more experienced than the rest of the group.

The polar circle was in sight when he noticed an uncharted path through the ice floes to the left of the snowmobile. It looked kind of rocky, if you were tall enough and you looked down, there were gaps that let you fall in to the depths. It would have to be the path to a bigger path that would lead to the higher peaks. Not very handy because the snowmobiles had a maximum speed of eight miles per hour and it was as if the snowmobile was trapped in the floes. Perry had to take a risk and make a turn.

Perry was very eager to start the ride. The higher they went, the lower the temperature, but he didn't know if he could get to the highest point. A sort of hill, the highest peak, to reach the snowline, where they'd

...end of model output

And:

Cars were abandoned long ago. The atmosphere couldn't take it anymore. During The Great Heatwave, survivors migrated to Antarctica, the only place where the climate was cold enough to be outside. Today, "Antarctica is home to billions of birds. A very large majority of these are penguins, and most penguins are emperor penguins. In the dense atmosphere, emperor penguins are more docile and can be collected with ease."

Hundreds of miles from anything. Never leaving their island, never experiencing the outside world. Not even because of global warming. It was simply too cold for anyone to remain outside their home.

In between the reach of "A Boatload of Turds", there were other rumors circulating. One of which was about a young, determined teenage girl named Yamina.

Hiding in the deepest, darkest part of Antarctica, Yamina yearned for a life beyond her meager existence. Her dying father told her to be strong, because the fate of the world rested on her shoulders. Not only would she be responsible for the entire world's future, but the fate of her own home country, too.

But what was once alive now dead and cold. The first thing Yamina did was send out a message to the world, hoping to summon any who were left. The message? "There is no hope."

That was the first thing she did. But she was ready to take care of the next thing as soon as she got some things sorted out. But what could she do? There were no ships in sight. Not even an airplane. Not even in the Arctic or on the Antarctic ice.

"It is so cold here," the young penguin wailed. "Not even anyone can survive here. The atmosphere is too hostile. I'm just going to die soon. Good-bye, dad. Good-bye, mom. Good-bye, earth."

Over the next few days, Yamina had to accept her fate. If she couldn't live, she'd just have to live without. This was a cold place to be. This was a cold place to be. This was a cold place to be.

"I should leave this place," Yamina told herself. "Even the penguins here are too cold to be outside. There's no hope. There's no hope for the world. There's no hope. There's no hope for the world. There's no hope for the world."

Out here, Yamina couldn't hear any of this nonsense. Her father's dying words did nothing. Her home wasn't even close. For all she knew, her father's old death prediction ...end of model output

----

I'd say it does a pretty good job! It picks up the tone and continues the story by itself. I'm amazed it creates characters that are suffering in the context of the story also. Wow.

Hi Kindiana! If you happen to see this, I sent a message on Keybase. Nothing important, just saying hello.

This is absolutely wonderful work, and I'm really happy to see you've released it. Congrats!!

It's also awesome that Eleuther is hosting an API for it. Nice work, Elecrew. No idea who did that part, but it's pretty cool. It's easy to underestimate how hard something like that can be.

Good luck with your future research projects. I'm certain your name will be popping up everywhere before long. :) (I'll be taking another stab at swarm training now that TPU SSH is finally released; thanks for brainstorming random ideas with me, it was lots of fun.)

EDIT: Oh, sorry for posting this without context. Kindiana’s the submitter, and also the person who wrote most of the code. (Getting Jax working on TPUs is no small feat, let alone writing and training a gpt implementation.) Been following this work a long time; excited to see it released.

Anyway, we used to hang out and talk about random ML ideas. It’s been awhile, so I was just saying hello.

It's not an API. It's just a demo site :)

Reply on news.ycombinator.com