claytonjy

I don't think that is obvious. If your use case demands lowest latency at any cost, you might run batch size 1. I believe replit's new code model (announced about a month ago) runs at batch 1 in prod, for example, because code completions have to feel really fast to be useful.

With TensorRT-LLM + in-flight batching you can oversubscribe that one batch slot, by beginning to process request N+1 while finishing request N, which can help a lot at scale.

I think they _are_ saying batch size 1, given that rushingcreek is OP.

Ah, that's fair, and faster than any of the LMDeploy stats for batch size 1; nice work!

Using an H100 for inference, especially without batching, sounds awfully expensive. Is cost much of a concern for you right now?

How have you liked using TensorRT-LLM? Did you come from faster-transformers, vLLM, LMDeploy, TGI, something else?

We started migrating to it the day it came out, very glad to have it, but lots of little annoyances along the way. Biggest one has been loading our model repository; having to hardcode the location of the engine file means we can't use the built-in ways Triton has for downloading from GCS!

is that impressive? I was thinking 100 tok/s on an H100 is really slow considering LMDeploy claims 2000+ on an A100 and a large batch size.

By joining forces with Pyflow, or a different tool, or making yet another one?

The easy part is choosing a better end-state; anyone can do that, and for any of these Rube Goldberg machines at a large-ish company, several people likely have.

What makes someone a staff+ is finding a path to iteratively evolving towards that end-state without breaking anything along the way and while having progress to show off at each step.

I wouldn't make it a high priority. If there's one thing I know about conda users it's that "no conda package available" has never stopped them. In fact they prefer to pip install inside their conda environment, and the only conda packages they use are the ones that touch Nvidia drivers (e.g. pytorch).

This is a big deal. Curious if vLLM or LMDeploy will use this new backend or not; I know LMDeploy currently has a custom Triton backend.

We've been struggling with various limitations of LMDeploy, vLLM, and ctranslate2 for a fine-tuned llama2 model, so I'm hopeful this is a much better solution.

I think it's supposed to? I'm generally running on k8s with base images that don't include tini, so I've settled for exec-style CMD instead

The most important thing I know about signals is that in dockerized applications, your ENTRYPOINT or CMD must use the exec form (`["command", "arg1", "arg2"]`) in order for your application to receive signals.

A lot of applications, like web/API frameworks, will gracefully shutdown if they receive SIGTERM or SIGINT, but if that signal doesn't propagate to the app you'll need to SIGKILL instead. Kubernetes will do this for you, waiting for your process to shutdown and then ungracefully killing it when it doesn't do so in time.

But you can't do env var substitution with the exec form...

Shareholders who got in before that last round, that is. Employees who joined in the last few years will likely have underwater options unless Loom internally repriced already-granted options.

Interesting; for me NiFi is tied with Airflow as my most-hated software I've had to use professionally.

This was a few years ago but NiFi only supported editing the DAG from a GUI; you could store the resulting XML in git, but no human could read or edit it. Python steps were stuck on Python 2.6 or 2.7. Hard to debug pileups. Is it better now?

I agree, thinking in layers is super helpful here. I might prefer the higher-level, simpler API was part of the same project as the low-level API (esp. given the name recognition here), but when it isn't, I can go make my own high-level API.

I'm not a rust user; is there anything preventing someone from making a "base64-simple" crate which uses this base64 crate to make a higher-level API?

This is almost exactly how I did a similar migration, also in Timescale. I used PL/pgSQL and sqitch, did you use a migration tool?

Yup, I've taken the same approach to conferences.

I used to go to a lot of tech-oriented Meetups pre-pandemic, and even helped organize a few.

But then there was COVID, and I moved many states away, and it doesn't feel like that scene bounced back to pre-pandemic levels in either my old or new cities.

I used sqitch in a past job and loved it, but I had to basically implement what pgroll does automatically in order to ensure smooth migrations, and I hit plenty of issues along the way. Learned a lot, but pgroll looks much friendlier.

Cython is similar to python and is fairly easy to write for a python user, while C++ or Rust have much larger learning curves.

Is boring technology even an option in high-velocity fields like ML/AI?

I can pick boring databases and boring servers but when it comes to model architectures, model weights, GPUs, serving frameworks, etc. there's been such massive improvements lately that it doesn't feel like anything has settled enough to be "boring".

If I need a language model, or an image classifier, or an ASR model, there are older options which might be "boring", but they're not nearly as powerful as the latest options. Does choosing boring ML tech mean sitting out on massive recent advancements?

Yes, this absolutely works. Rather than viewing it as a rollback enabler, I think of it as a way to try things while keeping an escape hatch. Instead of deleting a column from a table, you can remove it from the view, and only drop it from the table when you're really sure that's safe.

But dropping a column from the view is an API-breaking change, so rather than updating the existing view I'd make a new view, possibly in a new API schema (e.g. "api_v2"). Then you migrate clients/applications to the new view, and only when nothing uses the old view would I drop both the old view and the column from the underlying table in new migrations.

Yes, I think that was it! Interesting to see they have removed that page in more recent version of the docs.

If your API view is a direct mapping of an underlying table (e.g. "SELECT *"), and you make a breaking change to that table, the view will also change and your application will break.

What I've done instead is change the underlying table as needed, and ensure the view interface stays the same, by changing the view definition. This lets you refactor the low-level schema without breaking clients, and this migration can be done in a single transaction.

> Think of database migrations like you would versioning an API, deprecate and then drop old columns only after declaring a breaking change and giving consumers time.

This ties into my second point as I eventually made an "api_v2" database schema and migrated users as you describe.

I last applied this approach a few years ago at an IIOT startup, where we had billions of rows of sensor data. We needed it all in one place, but with a fairly varied set of users, like an application team showing customers various slices of their data and a data science team applying all kinds of analytics to the data.

When I showed up we had a single Postgres database with a half-dozen different clients. One managed the migrations in addition to doing real work, and the others were a mix of "direct" users and API layers used elsewhere. Tons of overlap, tons of repeated logic, tons of inconsistency.

I used this approach to first stem the plague of direct DB access, and later to consolidate our APIs. It worked pretty well, and the folks I left it to seemed to appreciate my approach.

We did have a lot of debates about what is or isn't "business logic", what belongs in this DB or not. We tried to keep it pretty light; our raw data was many independent tables, and our API schema had functions for insertions and views to make that data usable in ways that didn't feel very application-specific.

I was pushing to replace our handwritten API service(s) with something like Hasura, though that didnt happen before I left. I think that's where this approach gets more powerful: manage your data with your database, get a web API for free. Then your data store is encapsulated behind a singular service with a well-defined API.

You're right that it requires some serious attention to the DB. We wrote all migrations in PL/pgSQL (managed with sqitch), giving us a level of control that is hard to get from a language-specific migration framework. That required a lot of upskilling, but I think it was worth it for us.

I was, but this approach makes it function as a single application, kinda. You're not exposing your underlying data storage directly to consumers.

The weird thing here is your interface is still SQL. Sometimes that's fine, but I think this API-schema approach really takes off when you use something like PostgREST, PostGraphile, or Hasura to automatically turn this API schema into a web API. Lots of nice benefits to those tools, and then you have a single service that is more what users might expect.

I actually discovered this API-schema approach from the docs of one of those tool...I just can't remember which one.

You can, but I'd argue if your API schema is just 1-to-1 views for the underlying tables, you're not getting much value from this approach.

It is difficult, but it's possible, and I've had to take that hard road.

At a high level it looked like

1. create API schema and start filling it out with useful stuff

2. one by one, migrate applications to the new schema. You'll find stuff missing from the API schema, and you'll add it, to support each application. Ideally you'll find commonality across applications, so you end up with a cohesive API schema and not a jumble of application-specific stuff. But this takes discipline!

3. as each application moves over, remove their access from the low-level schema(s)

In postgres, every object (table, index, function, view, etc.) lives in a "schema", which is better thought of as a namespace. I put low-level objects like tables into one or more schemas, then create an "API" schema with views, functions, procedures that operate on tables in other schemas. Then I only grant access to that API schema to the application users.

You're right, but I think GP has a point as well. Anything you expose to the application has a potential for misuse, and a lot of "misuse" of a DB means queries with bad performance.

But, I'd argue the API schema gives you better control over this by pushing what might otherwise be application-level logic into the database. I've solved a lot of bad ORM behavior this way.