pypeline --> pypeln
multiprocessing pipeline --> pr
threads pipeline --> th
asyncio pipeline --> io
this is totally unnecessary
If I want to use short abbreviated names in my code I can always `from pypeline import multiprocess_pipeline as pr`
Your library shouldn't export them like this as the default.
`io` is especially bad since this overshadows the `io` module in the Python stdlib
I am thinking about expanding the module names to their worker names: * pr --> process * th --> thread * io --> task
And then have the conventions * from pypeln import process as pr * from pypeln import thread as th * from pypeln import task as io # as ta?
This conversation is very valuable, thank you all for the feedback.
I see your reasoning here (`import pypeln as pl`) but I still think where you have submodules you should use unabbreviated words for their names.
For me I'd be happy with `pl.process.map` in my code, but `pl.pr.map` feels a bit too obscure to have as the default.
These things are quite subjective of course, but part of that subjective judgement comes from the experience of what is commonly done in other Python libraries (the stdlib is a bit of a mixed bag in this regard unfortunately, riddled with CamelCase and other abominations).
There are a lot of "pipe" projects in PyPi, but your project is also about process management. Maybe you should avoid "pipe" in your name perhaps? FlowProcessor? nFlow? xFlow?
I do agree that you should avoid io for asyncio. You should probably at least use aio, but there's no reason you can't have asyncio_task, thread_task, multiprocessing_task.
Lastly, in my mind the killer app for this would be to allow something that works on top of Celery in production, but then be able to fall back to say multiprocessing or threading when running locally. That would allow me to prototype something, and then when I want to scale, I can just change a config setting.
The goal I have for Pypeline is much simpler: let you easily setup data pipelines where you leverage processes, threads and asyncio where they are good at. So in my mind a killer app would be a pipeline that maybe starts with an asyncio stage for e.g. downloading images, maybe then a multiprocess stage for e.g. doing image processing, and finally a threading stage for e.g. interacting with the OS.
Right now I see Pypeline more as an easy to use single machine tool instead of a higher level distributed abstraction like Celery. Maybe other framework could leverage Pypeline to ease their work.
"Assumptions are the root of all evil."
With autocomplete a coder has no reason to use shortnames anyway.
You want to be explicit but still concise while writing python.
I do not have any experience with Java, but I guess when writing Java you can feel that you use too many words than needed (i.e. definition of verbose [0]), wikipedia has a hello world example[1] and it feels just heavy.
IMHO if you write pythonic code, very often it feels like writing/reading prose, seems like talking to computer. You are not too implicit, neither too verbose.
[0] https://www.merriam-webster.com/dictionary/verbose
[1] https://en.wikipedia.org/wiki/Java_(programming_language)#"H...
A library author should provide names which are descriptive and clear. Users can then abbreviate them however much or little as we choose (by import aliasing). For example it is very common to see `import numpy as np`... but you wouldn't want them to publish the library as `np`. It should have its proper name.
One reason for this is if I'm exploring code in a REPL or IDE with tab-completion. You want to have some idea what a module is for, without having to play 'guess the abbreviation'.