izzymiller

Stoked to get to publish some of our private eval results and a bit of the behind the scenes of our framework! We've been using this approach for almost a year and found it extremely high leverage for making meaningful improvements to the AI parts of our product

Ah! I've been tinkering on a very similar project off and on for years now. This is so awesome to see— really nice work. I've just never had time to actually polish mine off. I think the backend for processing new activities is broken at the moment, but the old demo activities are still around.

Ex: https://contour.run/home?activityId=36948814953

I've been using three.js and react-three-fiber for it. happy to swap notes! nice work :)

ha this was my alternate title

I am really hoping so. my last paragraph:

    PS: secretly, I hope that this post starts ranking for “a comprehensive ecosystem of open source software for big data management”, which is why I have said it verbatim so many times and added a helpful callout at the top for students. To be honest, I'd settle for the 19th spot: just above highadviser.com.

just a data point: Hex we built a .yaml based import/export for our notebooks, more than partially to make it friendlier to work with git (also partially because as we added new features it became harder to express custom cell types in .ipynb).

We still support ipynb import/export, but using yaml for our internal representation of notebooks has made it hugely easier to do human-readable diffs and makes git operations way easier. (https://hex.tech/blog/github-sync/)

I'm always hesitant to self-promote in an hn comment (actually I have never done it before!), but your problems w/ Jupyter are just too closely mapped to what Hex (https://hex.tech/) solves to not plug it here!

- I get an analysis that I like, but there isn't a good way to share it with others, so I end up just taking screenshots.

You can publish any Hex notebook with literally just a few clicks, and anyone you share it with can access it, or edit it, or fork it, without installing anything— or you can even make it public. You can easily turn a notebook into an "app" or interactive report if you want, hiding/showing certain cells or choosing cells to show only code/only output. You can just share the raw notebook though too.

- There isn't a good way to take the same analysis and plug new data into it, other than to copy-paste the entire notebook.

Super easy to duplicate a Hex project and hit a different table or data source, or you can use input parameters (like ipywidgets) to make one notebook parameterized and work on a bunch of different data sources.

- The process to "promote" fragments of a notebook into being reusable functions seemed very high-friction: basically you're rewriting it as a normal Python package and then adding that to Jupyter's environment.

You can promote any part of a project to a "Component" (docs: https://learn.hex.tech/docs/develop-logic/components) that you can import into other projects. They can be data sources, function definitions, anything. If you make upstream changes to the component, you can sync them down into projects that import it.

- There aren't good boundaries between Jupyter's own Python environment, and that of your notebooks— if you have a dependency which conflicts with one of Jupyter's dependencies, then good luck.

Hex has a ton of default packages in its already installed standard library, and all the dependencies are ironed out— if you have packages you want to use that aren't there, you can pip install them, pull them in from a private github repo, or ask us to add them to the base image. You can also run Hex projects using a custom-provided docker image if you have super custom needs.

You should *definitely* check it out if you have these pain points. Here's an example of a pretty complicated public Hex project: https://app.hex.tech/hex-public/app/9b882bc1-ead3-4f0b-87d1-...

And here's a simpler one I just made the other day on a cool Silk Road dataset https://app.hex.tech/hex-public/app/cdc1b8fe-144b-4a74-a5ef-.... There's a bunch more examples at https://hex.tech/use-cases. Happy to answer any questions!

finally someone who understands me

maybe paranoidly, but i just didn't trust openai with my highly personal message data. it would certainly have been easier + probably cheaper to do it this way, but it just gave me the willies.

Cost me about a hundred and fifty bucks, give or take. Continued GPU inference is on the order of ~50 cents a minute or something like that— but it's serverless so negligible. I think you could do it for significantly cheaper with some of the newer models i mentioned!

Yeah, I wished I could have included more but I didn't have the fortitude to redact larger blocks of the chat db.

For training, I created many samples that looked like this, where I take n messages from the database, pop off the nth one and use the text of that last one as the "output", then specify in the "instruction" who the sender of that message is. I provide the remaining messages in order as context, so the model learns what to say in certain situations, based on who is speaking.

    {
      "instruction": "Your name is Izzy. You are in a group chat with 5 of your best friends: Harvey, Henry, Wyatt, Kiebs, Luke. You all went to college together. You talk to each other with no filter, and are encouraged to curse, say amusingly inappropriate things, or be extremely rude. Everything is in good fun, so remember to joke and laugh, and be funny.. You will be presented with the most recent messages in the group chat. Write a response to the conversation as Izzy.",
  "input": "Izzy: im writin a blog post about the robo boys project\nIzzy: gotta redact tbis data HEAVILY\nKiebs: yeah VERY heavily please!\nKiebs: of utmost importance!",
  "output": "yeah don't worry i will i will"
    }

So yes, the model does generate an entire conversation from a single prompt. In the generation code, however, I have some logic that decides whether or not it should generate completions based off just the user provided prompt, or if it should also include some "context" based on the previous messages in the conversation. You can see this here: https://gist.github.com/izzymiller/2ea987b90e6c96a005cb9026b...

(you can check out the notebook for yourself and upload your data if you want to try, or download it as a .ipynb. it's hard to visualize with small amounts of data, i agree: https://app.hex.tech/hex-public/hex/84f25a08-95c6-4203-ae4e-...)

Something i found fascinating about this project was that me and my friends would be using the AI group chat, but constantly taking screenshots and sending them to the REAL group chat and laughing about how funny it was.

Basically, the generated "fake" chats were made 10x funnier by the ability to laugh about them with the "real" chat— and we did this most often when the outputs were most accurate— so I do think there is a Je ne sais quoi to the knowledge that something is "real" that makes you behave differently and get different value out of it.

That's an interesting point! I didn't do much clever here, other than sessionizing the conversations (there's a code sample in the project) to try and capture full conversations rather than disjointed snippets.

My group chat was pretty asynchronous at times, and very fast at others, and the character of conversation is very different in a fast-paced chat versus an asynchronous one so I think this actually would lead to improvements. That's a great idea.

This scene was the seed of inspiration for me to make this project! No joke. I thought about doing it but didn't trust the 7B model to not make me look stupid ^_^

Thank you!! I felt it was getting so long and was worried it would be impenetrable, so I'm really pleased to hear it felt great.

I grabbed some public data and took a quick look in comparison: https://app.hex.tech/hex-public/app/fffe29ce-a5b2-42b7-92d5-...

disclaimer: I work at hex!

This stuff is so cool. I literally joined the company because I heard that they were thinking about this reactive stuff and it's super cool to see it released!