Hey folks -- enjoyed writing this how-to on using SSEs to stream text back. The technologies are fairly specific but the components can easily be used individually.
229 karma · joined February 11, 2021
Now for some speculation from personal experience. Some people leverage language more for internal thought and some people have other intermediate representations. I'm personally more of a language thinker except I have a specific avenue for math/programming that is highly optimized. I imagine many use language to solve math problems (and no way is right or wrong).
I find merging to be extremely tough to work with -- I don't think I actually got good at git until I learned to rebase entirely.
Also, cherry-picking (effectively an atomic rebase-type) is underrated.
Instead, it’s either “welp, pushed this to prod and got promoted and it’s someone else’s problem” or “sorry, this valuable thing is too complex to do right but this cool demo got me promoted...”
This sentiment is echoed in this comment in reddit comment as well: https://www.reddit.com/r/LocalLLaMA/comments/1d4p1t6/comment....
Similarly to this post, I think that the "good" abstractions handle application logic (telemetry, state management, common complexity), and the "bad" abstractions make things abstract away tasks that you really need insight into.
This has been a big part of our philosophy on Burr (https://github.com/dagworks-inc/burr), and basically everything we build -- we never want to tell how people should interact with LLMs, rather solve the common problems. Still learning about what makes a good/bad abstraction in this space -- people really quickly reach for something like langchain then get sick of abstractions right after that and build their own stuff.
The rest is a fun programming problem (which he largely glossed over), and it's clear he put a massive amount of care into the video. Thank you!
Yeah, reacting to so much perceived information being represented in such little code. Versus billions/trillions of weights in LLMs…
Will read!
Makes me wonder if LLMs aren’t the best way to model the world at all…
On the plus side there's a lot of interesting things and it is generally easy to follow/figure out what they did.
On the minus side it's a little exhausting, and there's so much money in it feels like the vast majority of it is grifting. To add to that, the people who are trying to catalog it (the AI guy on LI) are the griftiest of them all.
I've found the best way to keep up is find one topic you want to learn about, deep-dive, and read all the related papers, then explore breadth-first from there until you find another topic...
Hamilton/the UI doesn’t run it, but it does give visibility. So it has the tracking/visibility of airflow and the metrics/artifact tracking of MLFlow bundled together. Can also be used with those systems happily.
Our goal was to provide an all-in-one system that provides a host of data/ML/LLMops needs.
Also I’m sure this is extremely well studied but I’m not super familiar — ML on accounting graphs could identify shapes that indicate illegal transactions, etc… Will dig for reading :)
And good point re: good docs for LLMs! On a meta-level there's something very funny about having to write better human-facing docs for the use of computers. Who would have thought that some of our most powerful computing systems emulate how we think!
It's just a library so it depends on what could be the bottlenecks. Standard ones are: 1. Compute -- just scale horizontally 2. DB -- scale horizontally (and vertically), use intelligent partitioning
(1) is generally easy with autoscaling, (2) is where your queries/data modeling matter, and this is why we made persistence entirely pluggable, figuring we wouldn't know the best usage patterns yet. More reading: https://burr.dagworks.io/concepts/state-persistence/.
One more thing to think about -- Burr isn't opinionated about how you build your app -- E.G. up to you to think about what happens if you have your webapp open in, say, two tabs. You'll need to manage situations like this, but Burr's plugin capabilities should make that easier to iterate on. We're looking to add more examples on this -- also reach out if you have any specific questions/find anything missing/have suggestions.
That said, logging stuff and using regexes (which is the common way to productionize printlines for live/post-hoc debugging) has a lot of challenges, and this is where we've seen a framework become useful.
There's an obvious trade-off between adopting a framework and rolling your own -- we've seen both work in various contexts (we've been in MLOps for quite a while). Our hope is that this will make people more productive by constraining the space they need to think about and allow more mental room for application logic.
If you're getting started with LLMs, the first step is an API call to OpenAI (or some other foundational model) to get a feel for it, then think about how you want to integrate with your application. Burr can help standardize the structure, allowing you to think less about it at any given point (as logic is encapsulated into actions). Furthermore, the UI can help you debug.
With a few POCs at your company (assuming you want to iterate and get into production), Burr can help abstract some complex parts away (state management and telemetry), and make your code cleaner and more extensible, which can help you iterate quickly and explain what you're doing (you can always draw a picture of your app).
We think that it buys companies something they really want -- the ability to swap out implementations/vendors as it decouples the logic from application concerns.
So, if you have logic that's starting to get complicated (and might get more-so), I think Burr is good. If all you're doing is a single wrapper over a GPT-call, it might be overkill (or not! things tend to grow in complexity).
Technical details follow:
You can define an action like this:
@streaming_action(reads=["prompt"], writes=["prompt"])
def streaming_chat_call(state: State, **run_kwargs) -> Generator[dict, None, Tuple[dict, State]]:
client = openai.Client()
response = client.chat.completions.create(
...,
stream=True,
)
buffer = []
for chunk in response:
delta = chunk.choices[0].delta.content
buffer.append(delta)
yield {'response': delta}
full_response = ''.join(buffer)
return {'response': full_response}, state.append(response=full_response)
streaming_result_container = application.stream_result(...)
action_we_just_ran = streaming_result_container.get()
print(f"getting streaming results for action={action_we_just_ran.name}")
for result_component in streaming_result_container:
print(result_component['response']) # this assumes you have a response key in your result
# get the final result
final_state, final_result = streaming_result_container.get()
@action(reads=["prompt"], writes=["response"])
def my_action(state: State) -> Tuple[dict, State]:
prompt = state["prompt"]
# you stick in whatever library you want for the call
llm_response = whatever_library(prompt)
state = state.update(response=llm_response)
return {"response" : llm_response}, state.update(response=llm_response)