Django 1.8 and Python 3: Complex app tutorial, end-to-end | Hacker News Reader

Django 1.8 and Python 3: Complex app tutorial, end-to-end

marinamele.com

185 pointsdlcmh29 comments

Great job. For me personally, it's very helpful to have an end-to-end tutorial, where all aspects of an (example) app are discussed. I don't have a CS-background nor many years of coding experience, so a tutorial written by somebody who does have the proper experience and knowledge and puts it all together is extremely valuable to me. That's why I'd happily pay a fee for tutorials like this.

At the moment I'm working on building an app in Node.js with Express and sadly, there are not many tutorials of this quality.

It will surely work in Django 1.8. But whether the Django 1.8 recommendations like turning `urlpattern` into a list [1] or using `TEMPLATES` setting [2] have been applied is a different matter altogether.

[1]: https://github.com/mineta/taskbuster-boilerplate/blob/master...

[2]: https://github.com/mineta/taskbuster-boilerplate/blob/master...

Can you point me to a list of such recommendations for Django 1.8? Are these official ones?

It is in the official release notes:

https://docs.djangoproject.com/en/1.8/releases/1.8/#template...

https://docs.djangoproject.com/en/1.8/releases/1.8/#django-c...

Thanks arocks! I'll update that!

Done! :-)

Just looked at the summary, but it doesn't seem to include a chapter on asynchronous jobs ( such as report creations, emailing, or any kind of long running tasks). Along with websockets / push architecture, this is often when website designs start to be really complex.

The issues in sample project are regular challenges. I also think it's not a complex project without real-time parts, searching, async and concurrency tasks. Especially in Python 3.

Django makes asynchronous jobs very easy through "management commands" - simple but perfect framework for running any kind of job. The sample runs on Heroku - use a job scheduler and run your commands.

Alternative is to use something like Celery: http://docs.celeryproject.org/en/latest/index.html

Yeap, celery is really when i started to feel the pain of using python rather than a more enterprise-ready language&framework. Although in my case it wasn't using it with django but with flask ( but i don't think it makes a big difference).

The main issues when you start to have a stack built of big independant systems ( such as a db, a message queue, worker processes, and web server) are more related to deployment, monitoring and failover management. But they are really the big issues that justify the choice of a technology instead of another.

I understand the complexity in python of tying together Flask + Celery + DB + app server etc, but what would you use instead (your comment seems to imply there's a language/framework that has the aforementioned things bundles/more integrated)?

Implementing a task queue is not that hard in Django's management commands. See https://docs.djangoproject.com/en/1.8/howto/custom-managemen... - it covers 90% of the what you'll need. When you start to scale out, you might need a different solution, but this will work and be easy to deploy for the mostly everything.

As an example - I use this to create contracts with LaTeX asynchronously. The contract is created and then sent to the user by email. A management command runs every 5 minutes to check if there are any contracts that need to be generated.

In terms of languages that facilitate doing work concurrently or in parallel with the HTTP request/response cycle:

  - Java / Scala / Clojure
  - Go
  - Erlang (add Elixir if you are comfortable with its maturity)

Not meant to be an exhaustive list of languages. Celery is great but there are plenty of situations where chucking units of work into a job queue or running a cron job feels like an incomplete solution.

These for the most part don't help you persist/offload/schedule asynchronous jobs off the main process. If your process dies or gets restarted, the async job dies with it. Celery persists, queues, schedules and distributes jobs for you, it's different.

For something more similar to Go/Java/Scala/Clojure/whatever, you can use Python's gevent or asyncio.

the .net ecosystem is the kind of thing i was thinking about, but i'm pretty sure java has the same kind of solution.

as for the other persons answering me: i'm not only talking about long request. think about something like scheduling emailings or reports or ios notification , periodically or based on some db state change.

not something you'd want to handle with just an async framework.

the downvote made me look for a more precise example , so look at quartz for a robust enterprise grade lib for jobs and task management ( and its .net alternative).

i suppose celery is, on the paper, comparable in terms of fearures, yet the one time i had to use it made me feel like playing with a brittle toy, and in the end, i had to use a "recycle on every jobs" setting to prevent memory leaks from crashing everything. not that i believe celery is leaking memory, but that tying all the pieces together is a real pita.

I suspect people jump to Celery when something much simpler would probably be enough. I've seen people use Celery with all the associated deployment and monitoring complexity when all they wanted to do was run a slow task without blocking a request.

You could simply use this: if all you need is to run a slow task and not block a request there's this: https://github.com/defrex/django-after-response

I've usually used the system crontab (sometimes via https://github.com/andybak/django-cron ).

If you really really need a proper task queue then Huey is much lighter than Celery: https://huey.readthedocs.org/en/latest/

If you do need something more than

django-after-response looks fantastic, it'll make 50% of my apps much easier. Does anyone know what the lifecycle of the spawned threads is like? I'm guessing it depends on the server, but it looks like request + tasks need to take less than 30 seconds total in order to not time out.

Ya, it depends on the upstream timeout for the process. I'm pretty sure to default for gunicorn is 30 seconds. After that the process will be killed, which kills the thread.

> celery is really when i started to feel the pain of using python rather than a more enterprise-ready

The hendrix project - and its crosstown_traffic API specifically, is designed to alleviate this pain and remind that python (well, Twisted) is absolutely an enterprise-ready asynchronous network solution.

http://hendrix.readthedocs.org/

"Yeap, celery is really when i started to feel the pain of using python rather than a more enterprise-ready language&framework"

Say what?

The suckiness of celery doesn't come from its use of python. I'd be interested to know how you consider python not to be "enterprise ready".

I'm not sure what the suckiness of celery is.

We've been running it 4 years in production with no problems. It's really a nicely performant option with nice fail over options, and great debugging.

Nice tutorial :) I really appreciate that it covers a lot of the bases and doesn't assume the reader will only ever run on localhost (like many others do). I'm sure given that its a good start with Django, the gaps identified in the comments here can be filled in later via google or further tutorials in your series.

(shameless self promotion) we've just released something similar, but for Ruby on Rails - https://prograils.com/courses/saas/ch/01-the-welcome-word - our plan is to make step-by-step tutorial on creating quite complex todo-like saas app.

the boilerplate seems to ignore django's "multiapp" project layout:

This might make it not-that-suitable for actually complex projects.

EDIT: ah, looks like the formatting here did't support what I tried to do...

Indent the tree with 4 spaces.

Something like this probably?

    |-manage.py 
    |-MyApp/ 
    |---- settings.py, urls.py, wsgi.py,... etc. 
    |-AnotherApp1 
    |---- urls.py, models.py, views.py, migrations/... etc. 
    |-AnotherApp2 
    |---- urls.py, models.py, views.py, migrations/..

Err, are `AnotherApp{1,2}` intentionally missing `settings.py`, and all the other incongruencies?

I'd think that `AnotherApp{1,2}` would be subdirectories of `MyApp/`, right?

There's only one settings file, which is defined in the first/main app.

Reply on news.ycombinator.com