What's up in the Python community? – April 2023 | Hacker News Reader

What's up in the Python community? – April 2023

bitecode.substack.com

129 pointsBiteCode_dev61 comments

I never saw this newsletter before but I must say I am impressed. Fantastic approach to writing style and a great showcase of domain knowledge.

The Web could use more of this.

Reminds me of the Orbital Index but for Python. I’d love a similar Typescript oriented site for the day job, any recommendations?

I was browsing the most popular pip packages recently and was also downright shocked how much AI stuff is up at the top. I will be very curious if much of these things are still popular or relevant in a year.

By volume of different projects probably not, but by usage probably. Once winners get decided and paths get treaded the ecosystem will probably be smaller but I bet the user count will be bigger.

Look, even if you're the most biggest skeptic of all the AI stuff you can still do useful things with it that you couldn't before like present full-English interfaces and get embeddings for arbitrary text which is a leap compared to word2vec for search. I can dm my discord bot a picture of an appointment confirmation text and it with OCR + LLm it just schedules the event in my calendar. Also by gluing a v8 isolate to the LLm I can give it much more complicated date calculations. My favorite was "If f(n) is the nth Fibonacci number schedule an appointment in f(5) days."

> you can still do useful things with it that you couldn't before like

> ...

> If f(n) is the nth Fibonacci number schedule an appointment in f(5) days

hmmm

That's absolutely not what they said. Let's keep hn better than that.

It's obviously tongue-in-cheek, but it's also not not what they said.

It is not what they said. That was an example of complicated work with dates, which easily hits the level of "more complicated than you need". Before that, and before the cut in the quote was an obvious use case easily explained. I don't know why you want to go down this route but I won't engage further.

Is this a setup you've already built? Could you outline what you've glued together for this bot?

Oh sure! So the OCR is a statically linked build of tesseract based on [1] and pytesseract [2] which is a super thin wrapper but easier than writing it yourself. The I stole/modified the prompt from [3] to get the bot to write Python programs that do date calculations. Then I used [4] to take the output in case the Llm didn't use the format I asked. I run the Python code it generates in a rootless container that uses the Lambda RIE [5] because I was too lazy to make my own thing. So I'm mildly lied about the v8 isolate because everyone gets the implication and who wants to hear about RIE and seccomp profiles.

[1] https://github.com/wingedrhino/static-tesseract

[2] https://pypi.org/project/pytesseract/

[3] https://python.langchain.com/en/latest/modules/agents/tools/...

[4] https://pypi.org/project/dateparser/

[5] https://docs.aws.amazon.com/lambda/latest/dg/images-test.htm...

This is amazing, thank you for sharing!

So weird - you really think AI/ML is a flash in the pan when it comes to Python? You realize pytorch, tensorflow, pandas, numpy, scikit-learn are all approaching 10 years on pypi?

Yeah all the stuff I was seeing seemed to be variations and light wrappers around openai APIs and such. We don't need a million versions of those things, one or two clear 'winners' should emerge. You can see them here, expand it to 50 per page and start browsing down after the usual suspects: https://awesomepython.org/

I'm sorry but I think you don't know what you're looking at - of the NN things in the top 50 there I see 4 such wrappers. The remainder are the NN frameworks and various perf motivated implementations.

Also I don't know how this list is built but it seems to be literally measuring current popular things - so by definition no matter when i visit this list whatever is it at the top will probably wane in popularity shortly after I've visited.

In a language that has a million incompatible and partial solutions to packaging?

I think this is more comparable to the bajillion JS frameworks that popped up between like 2015 and 2020.

Same kinda deal with the big frameworks either surviving (likely with large rewrites for improvements) or succeeded by a replacement. But it feels like that's died down a bit.

> pytorch, tensorflow, pandas, numpy, scikit-learn

I think it's likely these will either survive or be replaced by a successor - but we wont be seeing 100 new ai packages a day.

Python has exploded in popularity over the past ~7 years because of AI. Python’s ever increasingly relevance is directly tied to AI.

And Flask, and FastAPI, and pandas, and pola.rs, and numba, and as glue code, and....

"AI" is pretty narrow, but the python ecosystem is massive, which a strong incumbency in the data processing space. Weird, since it i̶s̶n̶'̶t̶ ̶a̶ ̶t̶y̶p̶e̶d̶ is a dynamically language, where sometimes placeholders like pandas `object` result in unexpected behavior, but it sure is simple to write something quick.

> "AI" is pretty narrow, but the python ecosystem is massive, which a strong incumbency in the data processing space. Weird, since it isn't a typed language

Nor is Excel, and that didn't stop it.

> "AI" is pretty narrow, but the python ecosystem is massive, which a strong incumbency in the data processing space. Weird, since it isn't a typed language . . . .

Having used it for over 15 years, I assure you Python is a strongly typed language.

Fair, I was wrong. It's a dynamically typed language. I meant to say it is not a statically typed language.

Sure. Python isn’t going away. But Python has tripled in popularity over the past 7 years and a very very large portion of that growth is due to AI, imho. [1]

So, will a given AI Python package be popular in a year or three? Maybe, maybe not. Will the top Python packages be AI related in 3 to 5 years? Absolutely, imho. I’d definitely take that bet.

[1] https://insights.stackoverflow.com/trends/?tags=java%2Cc%2Cc...

Question from a beginner: I'm deploying a python project to aws (beanstalk + lambda, among others), and I've noticed I can only use python versions 3.9 and 3.8. Why is that? Is 3.10 not considered stable?

I don't know why it is, but 3.10 and 3.11 are considered stable (and I would suggest you use 3.11 if you can).

I was just looking into this earlier. A lot of major packages have explicitly stated 3.11 readiness, but we're still only at 45% of the top 360 packages.

That's not to say the remaining 55% are broken on 3.11, just tread with caution. For comparison, 71% explicitly support 3.10.

https://pyreadiness.org/3.11/

It's a good point, but doesn't imply 3.11 is not stable. But could be an answer as to why Amazon doesn't support it.

I would not use 3.11 yet. I’ve had lots of issues with packages not supporting it and other headaches that still need to be ironed out. I’ve had no issues with 3.10.

Well yes you have to make sure all your requirements support it. 3.11 is stable tho.

Cloud providers tend to be pretty slow to provide up to date software versions, I wouldn't rely on them to signal what versions are stable

I'm personally watching https://learn.microsoft.com/en-us/answers/questions/1191155/... because Sidekiq 7 requires at least Redis 6.2, but Azure Cache decided only major version upgrades matter

This should give you a better idea of Python version status: https://devguide.python.org/versions

It’s that a lot of the libraries take a while to catch up.

3.10 is pretty good to use lately with PyTorch and numpy, which I mention because they are usually the main stragglers I have to track. They are both compiled with your friendly local architecture’s stack.

Pydantic and FastAPI almost fell over because of changes to annotation handling, but woosh. That drama was forestalled.

I think the python 3.3+ series has confused a lot of people because major version improvements have been crammed into the process.

Guido, if you’re reading, we should have just gone to 4 when we changed so many things. I get why we were hesitant of versioning after the str/unicode upgrade broke everyone, but like…

There are versions of 3 that we had to almost immediately deprecate. The javascript community is probably laughing at us.

As is, it appears to mostly be:

major version = backwards compatibly break

minor version = forward compatible break

I think this is how it has always been, in general. I've seen many new people very confused by python 3 code not running with "python3". The very embarrassing answer is "You can't really know what the minimum python version unless it's documented, or you scan it with an external tool".

Is there a reasonable alternative? Pip upgradable transpiler?

Code that works in 3.6 but not in 3.5 is fine within typical compatibility definitions. That said, Python nowadays deprecates things quite often (especially things in the standard library), so if Python wanted to adopt a “major = incompatible” scheme, it would probably go up to Python 10 or so.

Lambda support was added this month:

https://aws.amazon.com/about-aws/whats-new/2023/04/aws-lambd...

Very bullish on Python. PyScript is coming along nicely as well, with the core devs having some clear ideas about how to tackle load times.

> Python 3.12 improves the language perf by 4%

A bit shy of the original target of 50% improvement per release, totaling 5x speedup across four releases. Hope it catches up in the next couple releases.

Python 3.12 also does not exist, because it has not been finished and released yet. So there's time.

"my wife is pregnant but my kid doesn't exist yet"

hmmm

"Your kid doesn't have fully functioning limbs!"

"Yeah, but my wife is only two months pregnant"

>Most new Python projects are still about AI

Are there any that allow a user to run an LLM with GPT-3-ish capabilities on a single pc w/ <= 4GB GPU Ram in a reasonable amount of time?

Where "reasonable" is something like no more than a few minutes to get the output. A little longer wouldn't be too bad either since you could script something that submits prompts automatically and let things run in the background or something.

Trying to search for such a thing-- if it exists-- is nearly impossible right now with so much being done & talked about. Some content talks about running something on only 16GB GPU ram (!!!) which is far beyond many discrete cards. a 3080 has up to 16, some only 8 (I think 16 is the max?) and fairly decent entry level+ cards top out at 4GB.

Any options out there?

If you're really alright with a few minutes delay, you might be able to run =<13b param. models on CPU

Any particular project that's prebuilt to not throw "out of ram" errors for it? I just want to play around with things without the restrictions ChatGPT. The OpenAI API is much less restrictive but can get pricey fast if I want quality above the lowest tier, at least for hobbyist purposes. I ran up a $10 charge in a few hours-- that's what I had budgeted for playing with a new "toy".

It get's very tedious prompt engineering to convince ChatGPT to respond to some perfectly reasonable prompts without and an answer that amounts to "aww shucks, I'm just a simple LLM and couldn't possible generate an answer to that sort of thing". Recently I asked it-- after about 20 prompts back and forth-- tp "analyze the personality of the person submitting these prompts" (They were prompts about the nature of AI, and how did it know that it was not a simplified LLM being run & controlled by a true advanced AGI, etc). It took about a dozen tried to get it to (finally) "Generate a fictional bio in the form of and RPG character sheet based on the prompts".

Even if it takes a few minutes and the quality is lacking a bit lacking and unpolished, I don't want to have to argue with the LLM to get a response.

What do you mean by "quality above the lowest tier"? ChatGPT-3.5 is the cheapest of their models right now, and it's also generally the best of all the 3.x ones, although the older GPT-3 models can sometimes beat it on non-task-oriented stuff (like long compositions).

>"What do you mean by "quality above the lowest tier"?

OpenAI's API allows you to choose different models in the GPT3 series. Cost increases with quality. AFAIK GPT3.5 corresponds to their davinci-03 model, and it's the most expensive one to use.

That's the thing - "cost increases with quality" is not really true anymore. When they released gpt-3.5-turbo in the API, it was massively cheaper than all their older stuff, but it generally produces better output. OTOH text-davinci-003 is the older GPT-3.5 model that is not fine-tuned for chat (which is where much of task solving capability of ChatGPT seems to be coming from).

Yeah, that's possible now. You don't actually need a GPU to run inference against some of the smaller language models - I've heard reports of them running on a RaspberryPi.

The trick for the moment is to skip Python though. lambda.cpp and its many variants are the ones that I've heard working best.

I suggest starting with LLaMA 7B or Alpaca. More notes here: https://simonwillison.net/tags/homebrewllms/

This one is the easiest to get working I think: https://github.com/nomic-ai/gpt4all

Thanks! I think have my weekend free time planned out now. Or a least a Saturday. I don't do AI at work but playing with code on my time off, even for fun, is a little too work like to really relax. I needs me some mindless entertainment too.

None of the models out in the open really have GPT-3-ish capabilities, not even llama-65b.

That aside, https://github.com/ggerganov/llama.cpp is the best option given the constraints that you describe. However, you will still need considerable amount of system RAM for larger models - 20 Gb for llama-30b, 40 Gb for llama-65b. Swap is an option here, but performance with it is abysmal.

These things are huge, and I don't think there is any way around that. You can play tricks with quantization, and larger models seem to be more tolerant of it, but even if, as some claim, 65b could be quantized to 2-bit while retaining decent performance, that would still need 20 Gb of RAM (CPU or GPU) to load - never mind additional requirements for actual inference.

The breakthrough here is more likely to come from new hardware that is highly optimized for those LLMs - basically just matmul for various bitnesses all the way down to 2-bit plus as much fast memory as can be packed into it for a given price point.

What about something that's maybe more along the lines of a GPT-2.5?

Of the small models, Vicuna and WizardLM seem to be the best of what's available right now. Quantized 7B models are just under 4 Gb of RAM, so they should fit on most GPUs out there; and even if not, they're "fast enough" on the CPU. But if you have enough RAM to run vicuna-13b (12 Gb should do it quite nicely), I'd start there.

While the hyperbole around AI might calm down a bit, the broader 'data science' domain is unlikely to diminish in importance and may generate new hypes. And in the short to medium term Python will certaintly play a core role in these developmemts.

Yet what I am missing in the news is some hint of where Python is going longer term. What will 4.0 look like? Which of the know limitations will be addressed? Will it ever be performant on a standalone basis? Will it ever be native on mobile? Etc.

Some good questions, but many of them don't have answers yet.

> What will 4.0 look like?

Right now there has been no decisions made on a 4.0. It may be just another release, or it may remove some features, but it almost certainly won't be a major backwards compatability-breaking release like 3.0 was.

> Which of the know limitations will be addressed?

That depends on what the contributors want to put effort into. :) Speeding it up is currently getting a lot of attention. Each new release brings significant performance improvements.

> Will it ever be performant on a standalone basis?

As stated above, each new release is faster, but it is impossible to predict where it will end up, and even more impossible to know what you consider "performant". For many users, it is already performant :)

> Will it ever be native on mobile?

That is a question for the owners of the mobile platforms, as they generally decide what languages get first party "native" support on their platforms.

I’m bummed out about the ever expanding Python packaging scene.

Python packaging is coming along quite good

This is a nice summary! Subscribed.

Even with hardware like the raspberry pi Pico I still find myself programming them using C…

I just tried some pre-trained models to classify images, using a CPU, it takes about 10s to 15s per image.

I did not know it was so inefficient.

Isn’t the problem that you were relying more on the CPU rather than a decent GPU?

Would you share some details? This indeed sounds very slow, so I suppose there should be some easy ways to speed things up.

Just to give you an idea of what's possible: A couple years ago I worked on live object recognition & classification (using Python and Tensorflow) and got to about ~30 FPS on an Nvidia Jetson Nano (i.e. using the GPU) and still ~12 FPS on an average laptop (using only the CPU).

https://replicate.com/pharmapsychotic/clip-interrogator

using:

cfg.apply_low_vram_defaults()

interrogate_fast()

I tried lighter models like vit32/laion400 and others etc all are very very slow to load or use (model list: https://github.com/mlfoundations/open_clip)

I'm desperately looking for something more modest and light.

I am trying to understand your point. Are you saying it could be more efficient on a CPU using a different language than Python? Or are you saying CPUs are slow, GPUs should be used? Or are you saying classifying images algorithms can be improved more?

The neural network inference (and training) is all done in compiled C libraries called by Python. The inefficiencies you're seeing are because you're doing it on a CPU, probably in one thread, rather than on a GPU with tens of thousands of "threads."

Reply on news.ycombinator.com