dandiep

I think https://elicit.org/ is also worth a mention in this context.

Costs really aren’t a concern compared to speed of development and quality.

I'm building a webapp backend in Python right now and the ecosystem compared to Java is just horrendous.

* Maven (or at the very least transitive dependency management) is a defacto standard. While some solutions exist such as Poetry, there is no such as thing as a centralized repository with POMs that tell you all the related dependencies easily. And managing virtual environments feels like a huge step backward.

* I ended up using FastAPI + Dependency Injector for the Python project, and while great, it is less robust, less well documented than things like Spring + JAX-RS

* Because Python sucks at multithreading and is slow, everyone has a bunch of C in their libraries, so when there is a bug (which there will always be), it's very hard to debug. In Java land I can easily step through any code, including the JVM.

* Lack of typing metadata - even relatively new libraries like the OpenAI client lack typing information, leaving you guessing about what data is available to work with.

* SQL Alchemy is vastly inferior to Java ORM solutions IMO (no, I don't want to discuss ORMs...)

* IDE support is still relatively primitive for pytest. Until a couple weeks ago, you couldn't see the output of all your tests in the VS Code console log until it was done executing. Or when you get an error, you can't just click on the stack trace to go to the failing line.

* Lack of autogenerated API documentation, e.g. equivalent of Javadoc

Love the Python syntax, but I still feel like I was more productive in Java for writing webapps. YMMV

Using GPT3.5/4 in our language learning app and people seem to enjoy it. [1]

Tried Llama2 and it definitely doesn’t even come close for what we’re doing. Would absolutely need fine tuning.

Maybe customers don’t enjoy chat bots for customer support, but there are a million other uses for these models. I, for example, LOVE github copilot.

1. https://squidgies.app

Who here is using Claude? And can you comment on your experiences with it vs. GPT 3.5/4?

I'm curious, do you have any concerns in unleashing LLMs in front of your kids?

I ask this as someone with no kids who is working on a language learning app that does essentially what you're talking about [1]. We have only done a little bit of red-teaming, but I'm sure there are ways kids could talk about topics that at least some parents may not want.

1. https://squidgies.app

I really don't want another database. I just want to have a solution built in for Postgres, and more specifically, RDS, which we use. I know there will be some extra difficulty that I will have to manage (e.g. reindexing to a new model that is outputting different embeddings), but I really don't want another piece of infrastructure.

If anyone from AWS/Google/Azure is listening, please add pgvector [1] into your managed Postgres offerings!

1. https://github.com/pgvector/pgvector

Thank you for this!

Yes, however, it can generate a training dataset to fine tune a smaller model really cheaply, which can then scale.

There is no such thing as davinci-003 per se. There is text-davinci-003 which you can't fine tune, and davinci which you can.

Does anyone known if OpenAI is allowing fine tuning to GPT35 for foundry customers?

Also, there's a huge difference between "it works for an ML benchmark" and "it works for real life use cases". OpenAI has done a phenomenal job with instruction tuned models, enabling fine tuning for any use case very very easily, and deploying it all at decent scale.

Does OpenAI provide guarantees now around fine tuned models being loaded 24/7? Last time I fine tuned Curie models they took forever to load and that killed it for me as we’re building an interactive application.

A couple points which I don't see elsewhere:

1) They have the best quality model. Better quality means more users. More users means more data. Which means higher quality...

2) operationalizing & scaling these these models is non-trivial. I'm not sure what the state of distillation/pruning is for GPT-3, but I imagine they have figured out some proprietary techniques.

3) It's not just publishing a single model, but making it so people can fine tune and push their own. Because they've gotten good at 2, now anyone can create their own version of GPT customized for their use case.

Will Google or others be able to do the same eventually? Definitely.

The point I'm more making is that it's not just training the model and running it.

Temperature = 0 solves a lot of it, but not all. A list of all options solves a portion of the remaining. Not sure what to do about the rest - maybe ask the LLM?

Yes.

Awesome... but I can't tell from your website what exactly it is.

There's a lot of work going on in this area right now. Check out GPT Index:

https://github.com/jerryjliu/gpt_index https://github.com/jerryjliu/gpt_index/blob/main/examples/pa...

GPT isn't multi-modal yet (so no images), but that's coming.

LangChain is very cool. Templates and composability are the way forward.

I'm guessing that most folks haven't created composable prompts, so a few experiences on why this stuff is necessary. I've been building a language learning app and it makes heavy use of GPT-3 [1], which has made clear a number of things for me:

1) Composability is fundamental to leveraging language models. You can't get language models to just generate things in one go. It's rather like a human. If you're writing a book, you start with an idea, then an outline, then a chapter outline, then writing paragraphs... Or in our case, generate a flashcard deck description, then vocab, then example sentences, etc.

2) Externalized prompt templates are also important. Engineers need to be able interface with experts who can create custom prompts. E.g. in our case, I need experts who speaks to build prompts specific to other languages for a language learning app [2].

3) Unit testing is critical. There is no linter for a prompt. I have made so many typos over the last year that broke things. OpenAI has released several new models over the last 2 years. Anthropic is coming out with a model. You need to have assurances your prompts work. I've actually had to start building a basic unit tester for our prompts because of this... [3] (please someone else do this so I don't have to)

4) external data is the next step forward for large language models. E.g. in my case, someone may want to learn about the history/culture of a country and we may want to reference existing articles on it since LLMs are known to hallucinate. We need to be able to interface with the web and databases easily. I'm not convinced that LangChain is the write layer of abstraction for this. I suspect/hope that the next version of GPT will have some significant advances in this regard.

1. https://squidgies.app

2. https://github.com/squidgyai/squidgy-prompts - open source composable prompts for Squidgies

3. https://github.com/squidgyai/squidgy-prompts/tree/main/tests - unit test in YAML

Perhaps, but language is the common denominator in a multi-model world. E.g., I pass the GPT output into other models which are fine tuned for that sub domain. You can do embedding to embedding conversion, but not sure it's worth the effort.

Also, a pretty sophisticated way to manage transitive dependencies. Python is an absolute mess in this regard (as well as pretty much everything else with dependency management…)

100% agree with this. In the early phases (in my experience) encountering of new word drastically outpaces your ability to do flashcards. And this is fine! Then, as you progress, you don't don't need to review every damn flashcard because it's way more basic than the level you're at.

This is is one of the primary reasons why I think SRS is not more widely adopted for languages, the other being that most people will never go through the effort to learn why and how they should do it.

I have been working on a form of this for a language learning app [1]. The basic idea is that it gets you do free form output (discussion questions, word games, etc), gives you feedback on where you have errors and then sets up SR flashcards automatically (e.g. if you don’t know a word, conjugate it wrong, etc).

It uses embeddings to make inferences across exercises and flashcards. This is handy for:

- detecting that you already have a flashcard for that particular word/phrase/grammar issue

- you know one sense of a word but not another

- you correctly used a word that you have a flashcard for and that review can be put off

It can look at your vocab and grammar skills of a particular area overall using Rasch scoring. Perhaps you answer almost all the questions about “cooking” correctly but none about “parts of the body” - well then the app can focus on the second area.

There is also work to be done to build out better regressions given this data. If a word I am studying is way above my skill level, I will probably forget it faster. Or maybe I forgot a conjugation which is very common. It should be easier to remember because I’ll encounter it more.

Whether or not my app succeeds, I think these types of approaches hold a ton of promise to increase learning efficiency.

1. https://squidgies.app

Apologies - this wasn't clear. I didn't mean to say we generalize to all cases. I should have said it was more helpful for some of the common cases that plague language learners.

I have been a user of the Language Tool API as we're building (experimenting with?) a language learning app [1] and grammar correction is a key component. I really want to love it, and I'm so appreciative of the open source version that has allowed us to get going, but it's just so... limited. I haven't really dug into the code, but I gather that it is very rules based and that seems destined to fail because so much of language is context sensitive.

For example, it really can't determine in German if you're using the correct case or not for a particular verb. Or it can't really tell if you're using the right prepositions.

Ultimately, we ended up building a grammar engine just for language learners using transformers [2] + a version of errant [3] we ported to be multilingual. It's not perfect, but after a couple weeks of development was more helpful than LT.

1. https://squidgies.app

2. e.g. this paper from Grammarly https://arxiv.org/abs/2005.12592, but there are a bazillion others

3. https://github.com/chrisjbryant/errant

There are tons of commercial uses for these models. I've been experimenting with an app targeted toward language learners [1]. We use large language models to:

- Generate vocabulary - e.g. for biking: handlebars, pedals, shifters, etc

- Generate translation exercises for given topic a learner wants to learn about - e.g. I raised the seat on my bike

- Generate questions for the user - e.g. What are the different types of biking?

- Provide more fluent ways to say things - I went on my bike to the store -> I rode my bike to the store

- Provide explanations of the difference in meaning between two words

And we have fine tuned smaller models to do other thing like grammar correction, exercise grading, and embedded search.

These models are going to completely change the field of education in my opinion.

1) https://squidgies.app - be kind it's still a bit alpha

I've spent some time looking into this and found very little. The closest I could get is that if you are a Google Cloud customer, it forbids you from reverse engineering other Google APIs that are not documented as supported, which would probably include this use case. No idea if they're a Google Cloud customer though.

One thing I don't see mentioned here is that the Google Cloud version of Translate is actually different than the user-facing one at translate.google.com. At least when I tried it a year ago, the Google cloud version was vastly inferior. I suspect it has to do with licensing agreements around certain datasets. Very curious if anyone knows more on this...

I skipped the tensorflow-deps and the tensorflow-macos dependencies since it was trying to downgrade a bunch of stuff:

$ pip install -U tensorflow-macos --no-deps

That seemed to work for me at least.

I was able to install this fairly easily (much more so then the crap they dumped out here - https://github.com/apple/tensorflow_macos. Just take a look at the 200 github issues that were ignored for the most part...)

I also noticed that in my project I got a decent speedup immediately when executing my model, but I have not run any benchmarks.

But, where do you go to file bugs? Ask questions? etc. I am not a big Mac developer, so is there something I don't know?