claytonjy

was this before torch 2.0? With the very notable exceptions of a few mispackaged versions, torch now includes all the relevant Nvidia libs, and I haven't seen it grab the CPU version on a GPU box yet, though I'm not sure what it looks for.

A notable open issue in poetry is we can't currently specify one dependency on torch, and have it grab CPU version on some systems and GPU on others. Does PDM solve that?

I have successfully transitioned an ML/AI team of seasoned researchers away from conda and to poetry. Some also use pyenv, I suspect a lot don't bother but may get bitten eventually.

It's definitely a learning curve, but it turns out every conda user has been bit by the irreproducible tendencies of conda quite often. Nobody uses the conda env file, they just start an env and pip install things into it. They don't realize the base env has stuff, too, and conda envs are hierarchical rather than isolated. I know it's possible to use conda in an isolated and reproducible way, but have yet to meet someone that does so.

So it hasn't been hard to pitch poetry to these folks, and while many complain about the learning curve they appreciate the outcomes.

We're a pytorch shop, and torch mostly just works with pip or poetry these days, as long as you skip the versions the torch maintainers mispackaged. We rarely need anything higher-level that only conda could install.

We really like having more than two dependency groups as this allows us to keep research and production in the same repository. main, dev, research. Then researchers contribute to the core library of a project and keep research and production using the same code for running and evaluating models.

pyenv only manages python versions, while poetry manages dependencies and virtual environments. They are complimentary, but do not overlap.

can you say more about the network issues with thousands of k8s nodes? I'm regularly running 2-3000 nodes in a GKE cluster, majority have GPUs, is this something I need to be worrying about?

That is a satirical twitter account.

Even GPT-4 has been hit or miss for me. I think it's less about the model and more about the data. I'm intentionally aiming to write my home-manager stuff as modern as possible (e.g. flakes, recent home-manager syntax) but most of the content out there uses older paradigms, so it's often hard or impossible to get chatGPT to produce quite what I want.

Still helpful, just not as helpful as it is for like, python 3.11.

Doesn't this work pretty well everywhere else now, too?

Torch at least is easy to pip install, as is poetry if you skip all the versions that were mispackaged.

It's unclear to me what the benefit of conda is these days, other than making it much harder to reproduce your environment on other machines.

another comment in here mentioned Zipline Reloaded¹, a fork from a guy who wrote a book using Zipline. Last commit 6 months ago, much better than 3 years from the original.

¹: https://github.com/stefan-jansen/zipline-reloaded

is your strategy more profitable than indexing? or are you aiming for something else, like beta-neutrality?

related, has anyone had success moving from Slurm to Kubernetes for a physical (non-cloud) cluster primarily used for training large models on lots of GPUs?

they said they finish work in 1-2 hours

what kind of role are you in and at what kind of company that there is so little for you to do? Do you never meet with coworkers or bosses?

I think it's awesome some people have found such a place, more power to them/you, but having so little to do is a very alien concept to me.

They offer Obsidian Publish for $8/mo: https://obsidian.md/publish

And even if they do pay, larger companies often insist on net-60 or even net-90...so you have to wait potentially months after delivering work to get paid. For a small or new shop that alone can put you on the ropes, especially since the companies most likely to do this are likely to be large enough and pay well enough to be the bulk of your inflow.

huh, I didn't even know openvino supported anything but CPUs! TIL

Anecdotally I've found ctranslate2 to be even faster than insanely-fast-whisper. On an L4, using ctranslate2 with a batch size as low as 4 beats all their benchmarks except the A100 with flash attention 2.

It's a shame faster-whisper never landed batch mode, as I think that's preventing folks from trying ctranslate2 more easily.

To have a good comparison I think we'd need to run the insanely-fast-whisper code on a 4090. I bet it handily beats both the benchmarks in OP, though you'll need a much smaller batch size than 24.

You can beat these benchmarks on a CPU; 3-4x realtime is very slow for whisper these days!

Can confirm. I worked at a small statistics consultancy where the owner did some expert testimony, and you have to reveal what you charge. The opposing side was often paying 500+/hr for shoddy work!

I think we've seen stories of Ivy League professors charging thousands per hour, and making the bulk of their income this way. Not sure how high it goes in the most high-profile cases.

One of the notable tricks the various LLM serving frameworks provide is a special approaches to batching, e g. continuous, persistent, or in-flight batching depending on the inference framework. At some level they each allow you to start a new generation while in the middle of one or more previous generations.

Is that possible with "just" pytorch? Could it be added to gpt-fast?

I would not consider metrics explorer to be a particularly good product. For queries where the query builder isn't good enough and you write PromQL instead, they don't even let you alias the legend; instead you must see the entire query as the label for that line.

A pretty minor nitpick, but indicative of the level of attention they give the product.

Even in the HBO show Succession, these things take a season, not an episode

I definitely believe he can raise a lot of money quickly, but I'm not sure where he'll get the talent, at least the core modeling talent. That's Ilya's lane, and I get the sense that group are the true believers in the original non-profit mission.

But I suspect a lot of the hires from the last year or so, even in the eng side, are all about the money and would follow sama anywhere given what this signals for OpenAIs economic future. I'm just not sure such a company can work without the core research talent.

Metrics, probably? The article calls out logs, metrics, and traces as the 3 pillars of observability.

Their Ultimate plan includes access to various top-tier LLMs for different uses, including GPT4 https://help.kagi.com/kagi/ai/assistant.html

I used DDG for years and often reached for "!g", yet a year or so into using Kagi and I don't think I've ever needed to fall back to Google. Any particular patterns to the searches you prefer Google for?

Definitely poetry; pipenv is not actively maintained and poetry is it's spiritual successor.

I appreciate the pushback and the links!

The third chart in the apartment list link ("Homeownership has been in decline for decades") is exactly the plot I wanted. That chart makes the gap not look so dramatic as either your numbers or the paragraphs surrounding the chart: millennials are within 10% of the boomers at the same age. For the youngest and oldest millenials it's more like 5%.

I'm curious how this has changed since the article's publishing in 2020. Anecdotally, I know a lot of millennials who bought their first home since, but I don't know how widespread that is.

I think some amount of gap is reasonable, especially for the 20s cohort: less likely to have kids, more likely to want to live in a big dense city. Looks like millenials are closing the gap once they hit mid 30s or so, right around when they have their first or second kid and move to the burbs or further. I just can't see being worried or upset about that.

Aren't age-adjusted homeownership rates roughly the same for millennials as it was for baby boomers?

I won't argue that it's harder and more expensive, it sure is, but millennials seem to prioritize it enough to do whatever it takes to own a home, so home prices keep going up.

I built an ML pipeline in pachyderm years ago, maybe 2016, and loved using it then. But my data was small-to-medium, and I stuck with CSVs because back then you couldn't get many of the benefits with a binary format.

I think it's a shame it took Airflow and similar several more years to realize "each step is a docker container" is the right way to build a dag. It's not clear to me why Pachyderm was left behind while Prefect and Dagster became serious contenders, and Airflow/Astronomer started recommending everyone use it just like Pachyderm (container per step).

which things do you do with HF, and which with WandB?

HF seems like "GitHub for models and datasets", it has a cool brand and everyone in ML uses it in some capacity. But when it comes to _company_ needs like private datasets/models, experiment tracking, CI integrations, etc. it seems WandB is a superset of HF.

HF has an enterprise offering, but it seems to be de-prioritized, and I think you'd still need WandB or MLFlow for experiment tracking?