Dynamic Django Models with Model Models
protoapi.net
protoapi.net
I have often wondered about pushing the customizability down a level though, like what the author does here, where each form creates a separate database table. It's too crazy for me, but it's tempting. And I have actually done it before where users' forms automatically generate individual tables in a separate flattened reporting database. It works really well for giving enterprise customers SQL access to an OLAP view of their data, even if I wouldn't use it for my source-of-truth OLTP structure. (Some day I'm going to write a blog post about that.)
Even just for derived reporting tables, you quickly discover all kinds of limits in your database: reserved words, max columns per table, max columns per query, number of foreign keys pointing at a central table, etc. I haven't hit anything fatal though (with Postgres).
Anyway, I'm excited to see that someone actually pulled this off, complete with Django models and migrations, even if it's just for fun.
But I never used it beyond that proof-of-concept; I still reach for EAV and/or JSON columns when I need to do this sort of thing for production software.
Good blog post though.
What's wrong with using a JSONField with a different Cerberus validator for each client, and then using a Django form (and possibly some custom admin template code) to make it possible for the client to perform CRUD operations in the admin?
I personally would still just build a React front end and do the CRUD operations via REST endpoints, but if you really want to use the Django Admin then that's a much better way to do it than monkey patching the migrations system.
The opening line should have stopped here. Dynamic data structures in untyped languages are risky, hard to debug, and prone to breaking One can argue that document stores (e.g. CouchDB and MangoDB) have been around for a while but their strength (and that of JSONField), is the ability to store hierarchy, not arbitrary structure (which they also do). A dynamic model like the one in the article serves as a database-stored deserializer, which is more of the domain of the business logic, rather than data.
Having said all this, the article is dope :o
What exactly do you mean by "untyped languages"? Python and every other high level language like it definitely has types. Maybe you meant "dynamically typed"?
It's closer to getting a bag of bytes in assembly than it is to the behaviour you'd get in any actually-typed language.
Nobody should be using such crap in 2018.
Then, when the form is saved, take the resulting dictionary and save it to a JSONField in a 'result' model. If you want to re-display the form with the results, just create the form from the configuration model and add the data from the result model.
I've had great results with this technique, and you can model any field except a foreign key. Django forms provides the validation, and the backing storage is dynamic so you can change fields easily. It seems less fragile than creating models like this article proposes.
However, this is only more flexible in the front end. If you want to do anything with your data in the back end, a JSONField will help you less than fully structured data. While you can do some queries on JSON fields, most of the richer queries will have to be done in python, slowing processing down.
If you're looking for runtime-changeable, structured data that you can run complex queries on, ModelModels sound like a good alternative. Most enterprise applications are in this area.
Typing that out now, I'm both enjoying the cleverness of that solution and wondering if it was at all justifiable. It probably would have made a good blog post.
* Django has facilities for mapping Python classes to database tables
* Author uses metaprogramming to construct classes to match table definitions decided at runtime
* Author uses built-in schema migration tools to synchronize backend
There's nothing about this that wows me, although it does demonstrate knowledge of Django internals and Python metaprogramming. It doesn't impress me because this is the bread-and-butter of what Django does.
This is generally what causes headaches later on, when you accidentally delete a field in production or forget to add a new one to mimic your dev environment.
But we had something like that where I last worked and it was okay. There was also no concept of dev db which was a different topic altogether...
Its used a lot in defining DLS's. Or Programming languages in general. Also editors often need to work one level higher in the abstraction chain. Abstractions around sourcecode.
Lisp is the master is this.
If you wanted to _fundamentally_ build your product around this concept that's fine, but Django probably isn't the right framework, however no feature is worth the pain of this.
For what it's worth, Django models and doing good relational design solve a huge number of problems. We've got ~380 Django apps and ~350 models and haven't yet needed dynamic models.
It's fairly common for example to see an app introduce some models, Python APIs to work with them, and business logic, and then to have a sub-app for the web API and bits relating to that, and a sub-app for the admin interfaces (we don't use the Django admin).
Example:
- orders
- refunds
- shipping
- royal_mail
- click_and_collect
- fraud
- fraud_blacklists
- returns
I find it a pretty nice codebase to work on. We're hiring too https://www.thread.com/jobs/software-engineer
...
I agree, but the main demand for something like this is going to be a client who wants to create their own forms on demand without touching Model code.
It is very much a hack-- the technical debt of it just gets kicked down the road for someone else to correct, the first of which is the unfortunate analyst who used to churn out weekly reports with a handful of SQL statements but now has to run an entire ETL pipeline/data science marathon to correct all the inconsistencies.
I can see the use-cases, but I think that level of dynamism should be built into the data model, not built by building a dynamic data model.
To take an extreme example, if you had a single table with the columns "key" and "value", you'd be able to encode any structure you wanted at the application level.
This would likely be a bad idea (at least for general purposes), but there are levels in between that would work. Models like "Form" and "Form Item", and maybe even "FormItemType" and "FormItemField", things like that, could represent fairly complex dynamic structures.
If I googled "Dynamic Django models", _this_ is what I would like to see.
- the most obvious one I don't see mentionned is that you should use a separate database than your main one for all the generated tables. Pretty sure it would save on operational headaches.
- it would probably be useful to generate all the Python code that could be used for making migrations, if only for debugging purposes
- understand early on which Fields you want to use with which options. I wonder if ForeignKeyField would be possible. That would be a great feature, but probably bring headaches.
- obviously, try to submit a PR to Django for the migrations loader hack, maintaining Monkey-patchs is troublesome
All in all, it's a pretty cool hack, but even with those modifications, I'd still be very wary to use that in production instead of battle-tested alternatives, unless you have a use-case that really warrants having full-blown SQL schema for your dynamic data.
Edit: on second thought, I can think of a use case where using that in a CMS would have saved me troubles (at least different tables for different Models). When you have a production site receiving Model A data (for example, your clients orders) AND you want to edit Model B data (for example, your site dynamic pages), you can't do that with EAV all in a single table, but with using dynamic models, you could just copy the Model B data, or even go with more complicated scripts.
Re-building the app around django cms would be the way to go - but being able to do this from scratch in an existing application is pretty nice.
What the article proposes is much better than this: A way to create models (ie tables in the database) dynamically. To make it more clear if you are not familiar with django, it'd be the same as if the CREATE TABLE sql was generated dynamically depending on the user selections.
So in this case each model will be in its own table and the fields will have proper types. You may even be able to have referential consistency using a ForeignKey field!
Since you say "you may" ... , do you happen to see any problem that may arise with this implementation ?
So the MakeField model will have an optional field to denote the class to which you want to create the FK; probably an FK to ContentType would suffice for this field (https://docs.djangoproject.com/en/2.1/ref/contrib/contenttyp...).
I know I've written field and FK too many times, I hope it makes sense; if it doesn't tell me and I'll provide a more thorough example.
For instance, a religious institution may not want to invest in a company that derives revenue from tobacco. So, we might have:
1. Is the company in an industry or sector we have flagged as tobacco?
2. Upon review, did one of our analyst create an exception list for companies partially involved in selling tobacco i.e. CVS, LVMH with duty-free, etc.?
3. Did the religious foundation also provide its own list of companies associated with tobacco usually through a consultant or several.
Therefore, we used a schema like:
table: exclusion_lists columns: company_id, exclusion_list_id, source_id
An account would register a restriction, which would group the baseline restricted companies from say sector to along with any customized lists that the account subscribed to, and a view would output all the restricted stocks and reasons why.
It didn't seem like an anti-pattern at the time. The benefit seemed to be if a restriction code was changed, it would all happen in one place versus updated in a bunch of JSON objects. Adding a column for a client for some special restriction with three companies would create a very sparse table so that didn't seem like an elegant solution either.