claytonjy

Two additional rules I don't see followed often, but made a past life of mine much easier:

1. rollbacks are bullshit, stop pretending they aren't. They work fine for easy changes, but you can't rollback the hard ones (like deleting a field), and you're better off getting comfortable with forward-only migrations

2. never expose real tables to an application. Create an "API" schema which contains only views, functions, procedures, and only allow applications to use this schema. This gives you a layer of indirection on the DB side such that you can nearly eliminate the dance of coordinating application changes and database migrations

You can get away without these rules for a long time, but 2 becomes particularly useful when more than one application uses the database.

Awesome. What about compiled model support? Running most of the listed models without compilation only makes sense for hobby projects.

I see plans for more models via HF partnership, but can I or will I be able to run a custom fine-tuned version of a supported model?

Yeah exactly, 20x is on the high side these days. That's 1.5s for 30s of input, a bit slower than I get for ctranslate2 alone (basis for faster-whisper and whisperX) on a V100 with the large model and beam size 5.

Yes, same prompt and same version. GPT-4 is especially stochastic, even with temperature=0.

Google Speech-to-Text is one I've dealt with recently; costs over 1¢ to transcribe 1 minute of audio. Almost 8¢/minute for the medical version.

I consider Timescale to be more of a hybrid than a "pure" columnar DB. IIRC the latest chunks are row-wise, but it goes columnar for older chunks to compress them. You can do a lot of OLAPish stuff, but it's arguably OLTP-first.

"enshittification" doesn't refer to a change in _investor_ value, but to _user_ value. Facebook does make a ton of money, but at the expense of the user experience. They got huge by providing a great UX and making no money; it was the need to make money that triggered enshittification.

Branding and trust. ML folks love HF, and rightly so. They have great software libraries, host everyone's models, generally great UX all around. They do a great job of making it easy to do cutting edge stuff, while still making it possible to do bleeding edge stuff. They appeal to basically all ML folks in a way few other companies do.

Google could grow GCP by more deeply integrating HF into GCP, I think, while retaining as much of the HF brand and interface as possible.

Big friction point might be ethics, since HF is still seen as the "good guys" and where some folks who left Google over ethical concerns landed.

How does this compare to Nvidia Triton? It's also a dockerized inference server capable of running ONNX, TRT, and many other compiled models. Automatic request queuing, HTTP & GRPC interfaces, configurable delays for dynamic batching, multiple models per GPU, integrations for metrics and tracing, etc. etc.

Is Inference looking to compete directly with Triton, or in a different niche (CV)? Does it make a different set of tradeoffs?

Because they see the lawsuits and shifts in public perception. Now that they've "cheated" their way to the top, they can play by the rules to entrench themselves so others can't catch up.

I think this is a good and important point. I was a DE for a bit and grew tired of it for this reason; thankful to have escaped into MLE, though it was more luck/opportunity than an explicit goal.

At or above any other SWE role at the same company. Often above the ML researchers at the same company since most researchers are earlier in their career than the ML ops/platform folks, though that's much less true at FAANG-tier companies.

This is the way. Provide value on an ML-adjacent team and expand your scope over time.

One thing to watch out for is being too far from ML at an ML-heavy company. Those places often have plenty of devs doing frontend work, integrations, etc. but those roles are not so different from similar roles at non-ML companies. Being awesome at typescript isn't going to help you shift to ML work. You want to be as close to the ML folks as possible: data engineering, data science, ML platform, etc. A common theme there is python, so get very comfortable with the language and ecosystem.

Almost certainly not worth it on anything but an institutional scale, and even then most efforts fail or are huge wastes of time and money.

Stick to good data and simple techniques, and even then be aware you're extraordinarily unlikely to post higher gains than you'd get with DCA and index funds.

No, language models were not a secondary application transformers. From the abstract of the paper you posted:

> The transformer model has been widely leveraged for natural language processing and computer vision tasks, but, to the best of our knowledge, has never been used for stock price prediction task at DSE.

In general, all the fancy DL stuff from the last decade started on either images (e.g. CNNs) or text (e.g. RNNs, transformers). Applying them to tabular data is generally an afterthought, and for time series in particular is rarely worth the time and compute.

The paper you posted is not very good and should not be used to inform your own work. It includes no comparisons to baseline models, and even the abstract describes the results as "acceptable".

Yes, that lawyer was Lawrence Tribe.

I used sqitch to have very explicit tests for procedures, written in PL/pgSQL. Worked great though writing that code was a little weird due to lack of IDE support I'm used to in any other language. Could take it even further with pgTAP.

Yup, between analytics engineers and ML engineers, it seems data scientists are the weird middle we no longer need.

I have worked with "data scientists" that are more rebranded statisticians. Some are overpaid SASS users, but some are true statistical wizards who can also program (R/Python/SQL), and when you need them they're great. I don't need them to write production code. I'd rather call them statisticians but I'm glad to see them getting paid better!

I'm not sure there's much chance of that happening. ONNX seems to be the broadest in coverage, but for basically any model ONNX supports, there's a faster alternative.

For the latest generative/transformer stuff (whisper, llama, etc) it's often specialized C(++) stuff, but torch 2.0 compilation keeps geting better, BetterTransformers, TensorRT, etc.

What is it about the funding environment that makes VCs suddenly willing to fund companies built around open source python libraries? They generally aren't beating the AI drum, but are they benefiting by proximity to AI via python?

I can't remember the last time I _wanted_ to pickle something. Parquet and JSON pretty much cover it for me. I guess pickling made more sense before tools like Pydantic were good and popular?

If I want all the python-specific nuance on disk, I know I'm in a bad place!

My general rule is, if I have a dict with a fixed set of keys, it should be a dataclass. Problem solved!

Then if I need to send or receive it, it's an easy conversion to a Pydantic BaseModel.

Yeah, that's fair. I could have been more precise about what an advantage research talent can be.

As an example, the startup-employed AI researchers I know had already PEFT'd llama2 within a day or two of the weights being out, determined that wasn't good enough for their needs, and began a deeper fine tuning effort. That's not something I can do, nor can most people, and it's a serious competitive advantage for those who can. It's a rather different interpretation of "can adequately fine-tune" than "can follow a tutorial".

When I think "AI startup", I think of the places where these people work. I don't think there's many of those people, and I think their presence is a big competitive advantage for their employers.

It's true we've made this easy, and that's awesome, but this is not what AI startups do; this is what other companies experimenting with AI do.

I think there's rather more to it than that. Two of these guys are on the Llama paper; the hype and momentum from that is surely responsible for a huge chunk of their valuation. If you take the big LLM-relevant papers, most of the folks with this kind of profile are already off doing some kind of startup.

The Mistral folks have impeccable timing, but are leaving FAANG somewhat late compared to their peers.

Is Jasper a counterexample? Good UX, domain-specific, but still a standalone ChatGPT wrapper forced to do layoffs because they have no moat.

Yes, totally. Not something anyone reading this is going to replicate!

1. research talent. There's not actually that many people in the world that can adequately fine-tune a large cutting-edge model, and far fewer that can explore less mainstream paths to produce value from models. Only way to get good researchers is to have name-brand leaders, like a top ML professor.

2. data. Can't do anything custom without good training data! How to get this varies widely across industry. Partnerships with established non-tech companies are a common path, which tend to rely on the network and background of founders.

Even with both those things it's not easy to outcompete a large, motivated company in the same space, like a FAANG. They have the researchers, they have the data and partnerships, so the way to beat them is to move quickly and hope their A- and B-teams are working on something else.

Can someone ELI5 why pypy doesn't or can't work with C-based packages like numpy or psycopg? I know nothing of how pypy does its magic.

If we could use pypy, while still using those packages, I think it'd be the go-to interpreter. Why can't pypy optimize everything else, and leave the C stuff as-is?

How does pypy handle packages written in other languages, like rust? can I use pypy if I depend on Pydantic?