Goodbye MongoDB, Hello PostgreSQL
developer.olery.com
developer.olery.com
The only useful piece of advice I can give a younger developer is... be careful when drinking the newtech koolaid.
And one more thing:
star = Sequel.lit('*')
User.select(:locale)
.select_append { count(star).as(:amount) }
.select_append { ((count(star) / sum(count(star)).over) * 100.0).as(:percentage) }
.group(:locale)
.order(Sequel.desc(:percentage))
For the edge cases where you need complex joins and/or extra performance, don't try and bother with the ORM, just drop to SQL.
Others should not be allowed in an interactive app anyway: developer should be warned/prevented using a "batch" query by accident.
It gives you total choice over what level of abstraction you want, so this is a particularly egregious use of it, being as nothing about that query is dynamic :)
You can use it purely to execute handwritten SQL queries loaded from files, or stored procedures, or any level of abstraction between raw SQL and the monstrosity you posted. The model layer is totally optional and is built cleanly on top of the query/relational layer, not inside of it.
The thing is a fucking work of art to be honest. The design shows impeccable taste, ruby's more exotic features are deployed only when really needed, everything is so clear, especially if youre used to activerecord.
BTW also the developer is super helpful and pops up everywhere to answer questions.
if you are learning sequel, stop by the irc channel and he'll probably answer your question.
EDIT: I just went to look at the github: 1821 stars, zero issues. And yes, github is the projects official bug tracker. There are 662 closed issues.
I must see if I can arrange for work to send him some money...
1) All my code lives in Visual Studio with compile-time type-checking. No maintaining stored procedures outside of my main codebase, no mucking about with strings. And because Entity Framework puts the Select after the From clause, I even get good intellisense on column-names.
2) I can quickly shift code into being in-memory queries from database queries.
3) While it's a little muddy mixing SQL and C#, there are places where C#/EF's syntax is so much more brief and clean since it's hyper-aware of relationships. For example, instead of writing 2 joins to get the country of a user, I can say User.Centre.Country.
Of course, the Entity Framework is adding a layer of complexity to my work - you still have to know what's going on in SQL, and that mental overhead is a downside. But it pays off.
SQL is a brilliant language, and that's why it's stood the test of time in spite of its advanced age.
But seriously, it has a lot of bad flaws - it's often difficult to make reusable components in SQL like you can in other platforms.
The underlying relational algebra is brilliant. That's why I like ORMs - they admit that SQL is ideologically beautiful, but the SQL language itself could use some loving.
I will say EF requires extra training/knowledge. So it isn't a "free" framework. However being able to migrate your entire code base from one database to another only changing a single annotation in the Code First classes is pretty nice. As is being able to generate your databases via the initialisers.
Is it for every project? No. However for "classical" database projects where you're going to be writing a bunch of CRUD and shoving it into C# classes, it is very helpful.
I'm also guessing John's problem with that query is that it's a simple aggregate query in SQL, there's no good reason to use the ORM, mainly because you have no idea what convoluted SQL it's going to spit out. There's only downsides with no upsides.
I've been involved in several migrations. All of which were planned over years in some cases. There was no "serious shit" at any point in the process.
One popular reason for migrations is licensing costs (e.g. escaping Oracle or IBM), or due to mergers where the other company had a different database system and they wanted to consolidate both technology but also expertise.
> I'll also guess you've never done it because the EF + MySQL = a world of pain.
I've never done what? I've never done EF with MySQL. I never claimed I did however.
I've used EF with other database systems including Oracle 11g, MS SQL (several), and other integrated it with some smaller database systems that had EF providers available.
> MySQL does not like nested queries, the EF uses them like they're crack, one of the reasons the SQL it produces is so hard to read.
You, the developer, formulate the queries. If you're seeing excess nesting then examine how you're doing things.
> I'd guess that the EF + [any db that's not MS SQL] probably suffers from similar "holy shit why did the DB just die, oh it's the EF" problem.
That's not been my experience. While I have found some other EF providers "limited" compared to Microsoft's provider, most of the problems encountered were due to underlying bad configuration or bad entity layout rather than EF itself. I will say Code First is significantly better than EDMX-style modelling. In particular in larger projects (the Visual Studio designer kind of sucks).
> mainly because you have no idea what convoluted SQL it's going to spit out.
It is very easy to see exactly what EF spits out. You can even pipe that output all over the place (e.g. error system, performance logs, etc). There's no excuse for being ignorant of what EF is doing under the hood anymore.
Because in my office the alternative is PL/SQL functions that dynamically build WHERE clauses, and I'll choose EF over that any day.
I think that you're confusing LINQ with the ORM.
LINQ is a language feature, Language INtegrated Query. It operates on IEnumerable<T>. They introduced a new interface, IQueryable<T> which extends IEnumerable<T> with Expression. Basically it exposes your "query" as an Expression Tree - this can then be used by QueryProviders to do cool stuff like generate SQL.
A number of ORMs implement LINQ QueryProviders; Entity Framework, NHibernate, BLToolkit are all good examples.
EntityFramework, like a lot of fully-featured ORMs suffers from all of the traditional ORM problems: runtime performance is sub-optimal, it's query heavy and if you use the visual designer it encourages bad development practices.
On the other hand it allows people with very little knowledge to be productive; for small-scale projects the bang-for-buck is hard to compete against.
- Fits well with a programming language; {a: data.x} is better than 'WHERE A="' + data.x + '"' (sanitize?) or similar which are harder to read. SQL queries are good for direct input, mongodb queries styles are better to be used with a programming language;
The mongodb syntax looks like it supports composing statements much more readily than SQL. It makes me wonder if there's been any serious work done making a query language that's fully as general as SQL, but is designed to be safely and easily composable so that you can build queries up from parts.
Anyone know of such a thing? The mongodb approach of query-as-data-structure is as close as I've seen.
Haven't you noticed that it's exactly what the Sequel snippet does? It's also what SQLAlchemy's Expression Language does. It's a reification of SQL queries in the host language, so it can/should give pretty much all the tools of SQL.
I haven't used SQLAlchemy's expression language, but glancing over it now it looks like it might be just what I was looking for (at least for python), so thanks for the pointer.
In a sane language that query might look something like
cursor.execute("select field from tbl where long_obtuse_column_name=:foo and status != :status", foo=query, status='open')
All parameters are properly escaped by the api of course, so even if foo is "'; drop table audit; " you don't have any problems.
Of course not, but your developers yes. Use a complex syntax and more bugs will appear. Do you care about bugs? No, but your users will.
> If you chose Mongo over e.g. a SQL DB because you like your code to look pretty then I have to admit that I wouldn't trust your software at all.
This is not my only reason.
So for a while it didn't support use cases as well as various NoSQL/NewSQL databases. However, PostgreSQL has also been built by great bunch of developers who are happy to adapt and implement the new, instead of harping on about the past.
So personally I adapt that advice to instead "use evidence to guide decisions".
I find that people that don't understand the problem like MongoDB for the ability to be "flexible" in designing and modifying down the road. Though I like to argue you have flexibility in SQL also.
SQL does many things very well that noSQL stuff won't. Also vice-versa.
those kinds of applications are not usually kinds of problems we have to solve.
You mean in every one of your application's code. You are using a schema no matter what, but it's a missing feature in mongo.
Another use case: if you have an ad-hoc group of developers working on a project in a decentralized fashion, they can each work on local codebases of the project without needing an up-to-date setup script or migration scripts.
I'd maybe even argue that a lot of these "goodbye MongoDB" developer blog posts are sort of unnecessary. MongoDB allows for fast deployment and the convenience of a flexible database at the cost of speed, so eventual migration to a more solid relational database could/should be part of a long-term plan when devs choose to use MongoDB. It's all about using the right tool for the job.
MongoDB also has some performance benefits over many traditional databases, and has "native" support for binary data.
I personally prefer CouchDB over MongoDB for most application that are well suited to use a "NoSQL" database, but MongoDB is sometimes decent too.
Unfortunately this seems to be a regular pattern in tech, a mistake (mostly) repeated by the young fresh developers coming in. Usually after you get burned once or twice you stop making that mistake.
I'm not a Comp Sci grad so can't speak from experience, but most of the syllabuses I've seen seem to focus on a very short term, single semester (or at most year) approach to projects. It would be good if they could start with a project and be required to re-write, adapt or improve some kind of project from beginning of the college time to the end. It would be a good introduction to technical debt, amongst other problems. (Heck if you really want to go for broke, maybe even mix things up a little, next semester you'll be working with someone elses code! That would be painful but might teach people the value of good comments, tests etc.)
Exactly. People report "X makes Y so easy!" and because of the breathy excited "look what I did in just an hour" blog posts the naive believe that using X for Y or similar things negates the need to properly design and think ahead.
If you take that approach then even if you are accidentally using the right tool doe the job, you may well be using it very wrongly and creating an unmaintainable and/or inefficient mess that you'll have to fix later.
A few days coding can save you a good hour or three of design and thinking beforehand.
(1) SQL server software itself was clunky, hard to scale, complex to deploy in a clustered fashion, and generally "old" in a bad way. This made NoSQL seem like a breath of fresh air.
(2) Startups! Ship now! MVP! Fail fast! The whole industry has been operating on this ship-fast-fail-fast way of doing things that prioritizes development speed and agility over correctness. That's because half the time what you are developing will be thrown away anyway -- it will fail, or you'll "pivot" and rip it all apart and almost start over. No point in static languages, SQL schemas, or other explicit design elements in that world.
#2 is I think the more significant of the two for things like schema-less MongoDB or dynamic languages. Once you get product/market fit, you typically end up taking the crap you wrote with no types and schemas and re-doing it in a "real language" with a "real database." But as they say in Startupistan, that's a "good problem to have."
I am yet to do any project that doesn't use SQL servers for data storage.
In fact, a traditional RDBMS is designed to allow for any kind of ad-hoc querying you desire with reasonable performance, and the ability to normalize/denormalize, index and materialize views in order to optimize unforeseen use cases. The excuse of poor scalability is just a rationalization that some kids who didn't understand SQL used to justify using something that has a shiny marketing page that they ignorantly found more viscerally appealing. The tradeoff was all wrong, because 99.9% of projects will never need to scale beyond a single DB server, and for those that do, the flexibility of a well-defined schema and ad-hoc queryability will give them an early advantage in what kind of pivots are reasonable—if they are lucky enough to have to scale then they can replace the RDBMS with the context of knowing what exactly their use case that they must scale is. And at that point you'll presumably have the resources to do it, which is much more effective than prematurely attempting to design a scalable infrastructure and discovering that you didn't have the first clue what the real bottlenecks or even the full use case would be.
We have started using jOOQ and it strikes a great balance of raw sql/typed sql and result -> object mapping. I don't ever seen going back to a heavyweight ORM again. There is just too much magic and lost time spend figuring out performance issues for a minimal gain in ramp up for those who do not know SQL.
The builder syntax is convenient because it allows queries to be composable without ugly string-smashing. For instance, you could take the object built out of that "User.select..." and pass it around into other functions that append extra 'where' clauses, etc.
Not this argument again. I'm all for learning SQL (just like one should always learn foundational concepts), but for some reason there's this notion popping up that Real Programmers use Raw SQL, just like 20 years ago Real Programmers used Assembly. For the love of God, stop hating on basic software engineering principles like code-reuse and allow us to move up the abstraction hierarchy like we do in literally every other facet of software development. Here's a comment I made on the issue of ORMs vs Raw SQL a while back that goes into more detail: https://news.ycombinator.com/item?id=8134205
In the past year I've also grown very fond of type safety within my ORM coding as well, something that's hard to achieve at compile time with just raw SQL.
My experience with ORMs has been that I eventually end up regretting using one if I try to model my data as objects because with ORMs it's easy to code yourself into a corner where you end up wishing that your design supported the relational model instead.
I do understand the desire for a good ORM, though. SQL is extremely powerful and a well-designed database is a joy to work with, but a sequence of tuples is often not the most convenient datastructure to process in most programming languages.
And that's fine, but it does mean that you should pick an ORM that plays nicely with that workflow (or otherwise design your ORM to be compatible with it). Back when I was working with Perl I found that you could do some pretty good stuff with RoseDB -- it could load objects from your custom queries and they'd still work like normal ORM objects (i.e. you can load related objects from them). I've missed that capability on several occasions since.
Your argument would've made sense a few years ago but not now.
Seems like this thread has gone down a very old, well worn rabbit hole.
Dang.
While I agree that many ORMs go too far or even worse, not implement certain powerful features, Sequel (and similar tools I imagine) strikes a nice balance. The particular examples I gave on their own are not super useful, but we already have quite a few queries that are composed/re-used without having to concat strings.
So tl;dr: it's about composability, not being "lazy" or "ignorant" to SQL.
I can never, ever remember the intricacies of the Sequel API or any one of these query builder APIs. I am always looking up something that is rather trivial because I am thinking in SQL, the language, and always have to convert back to Ruby or whatever language I am working in.
CTEs and SQL functions in PostgreSQL strike a good balance in the composibility department for me, while still being able to write SQL.
How exactly are you going to just write SQL if the actual query statement needs to change based on the user input? Are you going to hardcode every possible permutation, or are you going to use a query builder? Or are there some advanced Postgres features that I'm not aware of that would allow all these combinations in some kind of prepared statement?
edit: The question I'm asking is pointed towards the people who are implying that you can "just use SQL" as static statements that are not dynamically assembled. Like a static function or prepared statement that takes some parameters, and at runtime you only pass in those parameters - not rejigger the actual SQL statement fragments. Your ad-hoc query builder implementations with SQL injection vulnerabilities are not relevant to what I'm asking.
sql = 'SELECT * FROM products'
args = []
if title:
sql += ' WHERE title LIKE %?%'
args.append(title)
if upc:
if args:
sql += ' AND'
else:
sql += ' WHERE'
sql += ' upc LIKE %?%'
args.append(upc)
if price_min:
if args:
sql += ' AND'
else:
sql += ' WHERE'
sql += ' price >= ?'
args.append(price_min)
if price_max:
if args:
sql += ' AND'
else:
sql += ' WHERE'
sql += ' price <= ?'
args.append(price_max)
if order_by_name:
sql += ' ORDER BY name'
elif order_by_price_low:
sql += ' ORDER BY price'
elif order_by_price_high:
sql += ' ORDER BY price DESC'
conn.query(sql, args)The question I'm asking is pointed towards the people who are implying that you can "just use SQL" as static statements that are not dynamically assembled. Like a static function or prepared statement that takes some parameters, and at runtime you only pass in those parameters - not rejigger the actual SQL statement fragments.
Seriously, how is this different from "just use SQL". SQL was never really intended to be used by itself (except for manually typing queries into a console). Yes, what I wrote is code that writes code, but it's far different from a larger ORM or querying framework.
It's easy to see how libraries like Sequel get invented to help people do this, and I don't really see what the problem is when the DSL stays close to SQL semantics.
Trying to differentiate between a DSL that interfaces with SQL and SQL is nonsensical. SQL is the original DSL.
Hmm I thought that is how it was intended to be used? If it didn't, it would look like datalog, lisp or some binary protocol with prefixed lengths and whatnot.
It was intended to be typed in by analysts at a console. Who would then print the report on the dot matrix printer and mail to the headquarters or something of that sort.
I guess a better wording may have been to say that SQL was never intended to be used in isolation. It can be, but you've always had interfaces to other languages / platforms. SQL is a data manipulation and retrieval language - the rest of the business logic has to be performed somewhere else (excepting complex stored procedures).
I understand the impulse to "Just write SQL." But in practice, with all the string concatenation needed to generate actual queries, you can't really see what the SQL will be without running all the code in your head anyway.
Wait a minute...
sql = "SELECT * FROM products WHERE TRUE"
....
if upc:
sql += ' AND upc LIKE %?%'
....
SELECT
users.id,
users.email,
1
FROM
users
I think the irony is that much of the problems of sql are related to a lack of tooling.
The lack of tooling is result of large numbers of devs being distracted by ORM which moves that particular problem into the language ide/editor.
Essentially we lack great *.sql editing, ide, macro, refactoring because of the lack of focus on sql itself.
edit: as an aside, string concatenation in the language of choice is its own road to hell.
Except that my other tools are just an editor like emacs and makefiles. So now to be productive I have to have and ide for sql, learn sql, learn macros.
Interesting that SQL came first. If it was so easy and obvious to use it we would not have seen any ORMs by now. But every other project that uses SQL databases ends up with some kind of ORM.
One of the reason is probably because there is an impedance mismatch. Code is already written dealing with objects, functions, pointers, data. But when it talks to the database it has to talk via a command line string interface so to speak.
There is another type of impedance mismatch and that is learning a new language. One can argue SQL is there to keep & manipulate the most precious things you have -- data, so one should spend time learning it well. But the other point is, SQL is there to keep & manipulate the most precious things you -- data, so don't mess with it if you don't know SQL very well. Let someone else learn it (the author of the ORM).
I am not arguing for one point or the other, just kind saying how it is interesting how we have been going in a circle the last 15 years or so.
sql-mode for some databases(mysql and oracle iirc) reads the DDL of the tables you are typing about, why is this not better and more general?
Sql is so well established and general as a solution that not learning it is a poor choice.
I find LINQ + Entity Framework to be a blessing in C. Though I use the monad/function syntax not the LINQ query sugar when I use it.
Notice that its only a yearish old I think?
The SQL code completion is quite good (and it caches your entire database schema so it is very quick). That's not particularly special on it's own, but what makes this especially useful is IDEA's "language injection" [2] feature. This allows you to, for example, get completion for SQL when it is contained in some other language, which could be anything (e.g. a Java String, Ruby String, XML, etc). It will also analyze and report errors in this SQL on the fly.
And SQL statements contained in concatenated strings are no longer a problem because you can edit those in a separate editor window where you are only editing the SQL, and it automatically gets placed into the concatenated string.
Not sure if it addresses all of your specific concerns, though.
[1] https://www.jetbrains.com/idea/features/database_tools.html
[2] https://www.jetbrains.com/idea/help/using-language-injection...
Pgtap is on my list of things to poke around with.
edit: thanks btw
How about something like this:
s = Select.new
s.add "WHERE title LIKE #{title}" if title
s.add "WHERE price <= #{price_range}" if price_range
s.add "LIMIT #{limit}" if limit
s.add "OFFSET #{offset}" if offset
s.add "id, blah, boo FROM products"
s.execute
Also note how anyone who knows SQL could immediately work with this, learning curve: 5 seconds.
Why is there no ORM that works like this?
Edit [responding to moe, below]: that's a matter of taste. I prefer to have my SQL _outside_ my code. If I have to write a little XML to make it happen, so be it. Additionally, unlike your example, a strict separation of SQL and data ensures SQL injection is not possible. That's also a worthy goal.
Does MyBatis have an API similar to what I outlined above?
Because the page that you link to shows a horror that I can't even begin to describe:
<select id="findActiveBlogWithTitleLike"
resultType="Blog">
SELECT * FROM BLOG
WHERE state = ‘ACTIVE’
<if test="title != null">
AND title like #{title}
</if>
</select>
query = session.query(Products)
if limit:
query = query.limit(limit_val)
if offset:
query = query.offset(offset_val)
if sort_type == SORT_REVERSE_CONST:
sort_obj = sort_obj.desc()
else:
sort_obj = sort_obj.asc()
query = query.order_by(sort_obj)
query = session.query(retrieve_db_object(table_name))
return query.all()All ORMs work "a bit like this". I don't want "a bit like this". I want exactly like this.
Because with my proposed interface I could be productive immediately and permanently. I would never have to refer to any documentation. Not once.
With every other ORM this is a pipe dream. Their "fancy" chainable wrapper-APIs are nothing but a ball on a chain.
The strings don't even have to be sent to the DB at all. You've utterly missed his point. He wants the language to be intelligent about what the SQL means and do the right thing.
It's not raw SQL, it's an abstraction.
The original argument was that if one could "just write SQL" one wouldn't need to refer to documentation to become productive, "not once". I don't personally think it's practical to use any database library without reading the documentation, but certainly the moment you get away from "the SQL I type in goes straight to the database" you're going to need documentation to tell you what's going to change.
What's being described here sounds like a library that parses SQL fragments, combines them into whole queries, and sends them off to the database. To implement this this without resorting to simple string concatenation needs an intermediate representation of SQL language components. That intermediate representation is going to look a lot like SQLAlchemy, or Sequel, or Arel. That means that instead of "just writing SQL" you're actually adding an extra layer of abstraction and another chunk of documentation to read.
All that said, the "just use SQL" approach is already fairly well served. Most languages have mature bindings to specific databases as well as at least one ODBC-style API to simplify access to those bindings. Nearly all of these have support for client-prepared statements, which handles the interpolation problem. If you want to build more complex queries on the fly, you can use string concatenation or CTEs or decide to use a SQLAlchemy/Sequel/Arel-type library for just that one case.
And, honestly, you don't need to give up SQL to make use of Sequel. I personally really dislike the query composition API demonstrated at the top of this thread, but you can do useful things with Sequel without it:
db = Sequel.sqlite
db.execute "create table foos (id int not null, name text)"
db.execute "insert into foos (id, name) values (1, 'bar')"
# or
db[:foos].insert(id: 2, name: "baz")
db["select * from foos where id = ?", 1].first # ==> {:id=>1, :name=>"bar"}
# or
db[:foos]["id = ?", 2] # ==> {:id=>2, :name=>"baz"}
So is everything else anyone types, code-like or not. He even said "Note how I deliberately shuffled the order and didn't bother with escaping.".
The response was flippant, intelligence-insulting, and obviously the result of failing to read thoroughly.
And speaking of intelligence-insulting, we all know you can run raw SQL through Sequel.
You're not having a useful dialogue, you're being combative, like the person I initially replied to.
It was just a mockup. But you are right, in reality it would end up looking more like this (and use custom interpolation for escaping):
s.add "WHERE foo > $(bar)"
I believe both of these problems should be solvable without compromising the simplicity of the approach.
s.add "WHERE foo > ?", bar
Or even this:
s.Where "foo > ?", bar
At that point, you've reinvented ActiveRecord or hundreds of other query builders, it also avoids you having to remember the ordering rules of sql as to which part of the query must be built first. There's a reason these query builders have converged on similar syntax.
I see where you're coming from - you don't want to learn two sets of syntax, but many query builders nowadays are very well thought out, and have a simple syntax which just echoes SQL while avoiding sqli and statement order issues, they also usually have an sql() option which lets you just send SQL if you wish for a complex query.
There are very good reasons people keep inventing layers over SQL:
They help avoid sqli
They centralise db access so you can optimise in one place by say adding caching layers to queries
They mean you don't have to deal with the vagaries of SQL for simple queries unless you want or need to
That can be done automatically in many languages. At least in the sense you can do:
"where value=$(bar)"
and not:
"where value = ?", bar
OpenACS was an old web framework for the Tcl language, but it even had that 20 years ago!. The company backing it failed in the .com era, and the lack of types was a turnoff for many - but it made SQL much much more readable. And of course the queries were turned into prepared statements. A query looked like this:
select foo from table where age>:min_age and name != :user_name
Things like this can also be done in node.js (disclaimer: I wrote the blog post below):
I'd say stick with keeping it a string and use:
s.add "WHERE foo > $(bar)"
And that's the problem. Everyone "echos" SQL, just slightly differently, each with their own idiosyncraticies.
If all you do is echo anyway, why don't you just let me write plain SQL, for christ's sake?
It's much easier for humans to memorize a slight SQL dialect than a different god-awful, nested ORM API boilerplate for every language they're working with.
I can write this without thinking, for queries of almost any complexity:
s.add "SELECT id from users WHERE"
s.add "name == $(name)" unless name.nil?
s.add "AND age > $(age)" unless age.nil?
s.add "GROUP BY age"
Ask me to formulate the same query in any ORM and I'll have to refer to the documentation. And almost every time I'll waste a stupendous amount of time on getting some irrelevant, syntactic detail just right. Only because the ORM insists on an autistic API.
They mean you don't have to deal with the vagaries of SQL for simple queries unless you want or need to
After having worked with Hibernate, SQLAlchemy (the one-eyed amongst the blind), ActiveRecord, DataMapper, Sequel and a few others that I don't even remember, I conclude with passion: They have it backwards.
It's not the vagaries of SQL that I don't want to deal with. It's the vagaries of the ORM du jour that I don't want to deal with.
They usually do more. SQL is not very pretty or friendly IMO, ORMs clean it up a bit (for example allowing arbitrary ordering, sqli, construction of multiple queries from one base). The example I gave was an arel query - very similar to yours if you move the WHERE out into the function name.
Your ORM proposal above looks fine (and very similar to existing ones but with more strings), but personally I find it uglier than the many which are available, and you haven't actually tackled the protection against sqli or explained how that would work. Normally you'd pass that through to the db driver as a $n param, rather than trying to escape a string at the ORM level. If your ORM is going to infer details from the sql, it basically has to parse that sql first and handle all the same things that current ORMs do by using named functions to add SQL - you'd find the same issues with consistency and edge cases I imagine, and of course have to be very careful about parameters and sqli.
So if you don't like ORMs, don't use them, but you do have to be sure that your own ad-hoc home-grown ORM that you're writing instead covers all the bases current ones do. I prefer the syntax of the ones you're denigrating for simple queries, even if for complex queries it might be better to write sql directly (via the functions they provide), but would be interested to see it in action - when are you going to start :)
They give you the ability to manipulate the query in interesting ways at any point before you execute it. They let you join different queries together, built by different parts of the system, in a safe way. They let you work with the native language you're working in instead of having to construct clauses in a foreign language, using strings. They let you post optimise your loading strategies.
You might want "exactly like that", but it's not going to be as flexible as a chainable wrapper system.
Hell, I could probably give you pretty close to that in python with a little work, but it's not something I'd want to use myself.
I get it. I like working in SQL too. I know it really really well, and I cringe when I see developers writing totally sub-optimal code because they don't understand the relational data model.
But there are other ways to do things, and what you're describing doesn't give you much more than raw SQL, so why not just use raw SQL? You've added some syntactic sugar that, in my language (Python), would be a bit of a horror show (strings access local variables implicitly, no thanks). What else do you gain?
This is more verbose, granted:
q = session.query(Products)
if title:
q = q.filter(Products.title.like(title))
if price_range:
q = q.filter(Products.price.lte(price_range))
if offset:
q = q.offset(offset)
if limit:
q = q.limit(limit)
products = q.all()
p = products[0]
p.supplier.contracts[0]
q = q.options(
joinedload('supplier').
subqueryload('contracts')
)
Use whatever makes sense for your use-case. Don't limit yourself because you'd have to learn something new. Honestly, before using SQLAlchemy I mostly felt the same as you do. Many ORMs get in the way, but that's not really a problem intrinsic to ORMs.
If you are fine with the new debugging this entails. Take the joined load example you gave. Almost every time I encounter SQLAlchemy code, the author didn't understand the lazy joining semantics and every attribute access on each record is emitting a new query and you now have O(n) queries. It's not obvious that is happening until it bites you once you have thousands of records.
For another example, I can count on one hand the number of developers I have encountered that can understand the expiration of the objects in the session or what the engine builder stuff is doing in the boilerplate they paste in to get it to work. It always requires experimentation, a trip back to the SQLAlchemy docs, and then finally logging all SQL queries to see if it's finally emitting the optimal queries with the transaction guarantees you are looking for.
My gripe about SQLAlchemy is that it separates you too much from what is happening underneath.
But, taking the lazy joining example. Someone didn't understand how it worked and now you have to fix it, which is probably as simple as changing that loading strategy to something more suitable for the way you're using the data.
Let's rewind to the late 90s. If I wanted two tables of data to write into the page in an interlaced fashion I would look at how wide they were. I'd make a decision about using two queries or using one larger join. The two query approach was sometimes required due to the size of the data, but it meant cryptic output logic to track positions of cursors relative to each other. The single joined query was simpler to deal with, but still required tracking a last_parent_id so you could swap during the interlacing.
Other developers (those same ones that didn't understand lazy joining) would loop over the first query, running extra queries in the loop (I saw this a lot). Same bad performance as the lazy join.
When you discovered this issue in the code it was a total pain to fix. You're talking about rewriting whole load of code to the point of being unrecognisable from the code you started with.
Contrast that with how easy it is to fix in the SQLAlchemy case. I mostly don't worry about loading strategies now until I'm deep in development. Something's running a bit slow, take a look at some logs, tweak a couple of things and you're golden again. That's such a powerful abstraction.
Regarding the config of the engines etc, again, it's something you need to learn. But really, someone's starting an application from scratch and they just want to dump some code in to handle all their db interactions, but they don't want to know how it works? That's on them, either learn it or use the sensible defaults (in, for example, flask-sqlalchemy).
SQLAlchemy ORM separates you from what's underneath, but it's an ORM, that's kind of the point. If you need to be closer to the metal, use SQLAlchemy Core.
I'll try to write an example like that when I get home.
Of course, they're not "just SQL", and I do understand where the above comments come from.
And yes, sometimes I have to go look up how to do things I'd do trivially on SQL.
Knex is a very convenient, Promise-oriented query abstraction, and it has "raw" methods when only SQL snippets will do. http://knexjs.org/
select id, blah, boo
from products
where 1
and (title like :title} or if(:title = '', false, true))
and (price <= :price_range or if(:price_range = -1, false, true)
limit :limit
offset :offset
I really despise orm's its essentially taking a black box (sql/database) and throwing it inside a much less tested, less optimized, less documented black box.
For what reason, I have no idea, nobody has ever been able to convince me there is a problem that needs solving. Almost every argument i've heard amounts to problems sql already solved a decade ago. I can only speculate but I'll hazard a guess it comes down to lack of enthusiasm to really learn sql properly (even though the person uses it / debugs it daily)
Because you're only showing a query builder, the "relational" not the "object mapper".
From an OOD point of view, if the end result of that query will be Product instances, why am I using a Select object to create them and why is it having to do some sort of string parsing to determine the objects I'm loading?
Because we can just infer the type to be returned via the FROM-clause of the query.
and why is it having to do some sort of string parsing to determine the objects I'm loading?
Because, to cite the immortal Larry Wall:
The computer should be doing the hard work.
That's what it's paid to do, after all.
-- Larry Wall
s.add "count(*) / sum(count(*)) * 100 as percentage"
.select_append { ((count(star) / sum(count(star)).over) * 100.0).as(:percentage) }
The latter version may not look much more complicated at a glance. But as we all know these seemingly trivial fragments, more often than not, take ungodly amounts of very frustrating trial & error before they play along.
# db is a Sequel::Database
s = db[:foobars]
s = s.select Sequel.lit("max(id) as best_id")
s = s.select_append Sequel.lit("count(*) / sum(count(*)) * 100 as percentage")
s.sql
# ==> "SELECT max(id) as best_id, count(*) / sum(count(*)) * 100 as percentage FROM `foobars`"
It's entirely feasible to use Sequel this way. I've used it like that in production. I loathe the query building DSL, myself, but it's strictly optional.
Well, no.
I repeat: The real difference is that most people can't write even this utterly trivial snippet without studying the Sequel documentation first.
Now what if I want a WHERE-clause? Do I have to use s.where? Or s.select_append("where ...")? What if I need to combine them with AND/OR?
It's not ok that we have to think about all this boilerplate that has nothing to do with our actual query.
We shouldn't have to translate our perfectly unambiguous request from english (SQL) to klingon (Sequel API) in order to have it processed.
There might be a sweet spot yet unreached in terms of allowing developers to work with SQL on top of as minimal a native binding as possible. I'm certainly not claiming that Sequel/SQLAlchemy/etc. is at or near that sweet spot. It might be necessary to use a language with language-level support (or macros) to really reach it.
So if we remove this non-requirement of "anybody can write it", we're back to my original question: Ok, we the coders know we want a Product instance, but to build it we create a Select instance and then ask a library do so some sort of parsing?
Possibly no ORM exists that meet your criteria because there are issues with the design goals you're trying to achieve?
Because no one wants to write a SQL fragment parser. You would have write a parser that:
a) Could be started at an arbitrary place in a query and figure out what it needed to do.
b) Would be SQL-dialect aware. (Maybe -- it might be possible to skip this.)
Were it not for (a), I'd imagine you could quickly build this by using some kind of debug feature of the database -- sending it a query to parse and getting back a parse tree.
For example, here is :
{ type: 'select', columns: [{ expression: 'orgJobs.org_job_code', alias: 'job_code' }], joins: [{ table: 'org_jobs', alias: 'orgJobs' }], limit: 100, offset: 0 }
which yields:
select orgJobs.org_job_code as job_code from org_jobs as orgJobs limit 100 offset 0
and so its easy to do things like:
query.limit = 100;
and of course:
query.columns.push(additionalColumn);
and use something like lodash/underscore (or equivalent for your language) to add collection/array/object helper methods so you never have to write for-loops for things like mapping and property plucking.
http://www.ivc.com/blog/better-sql-strings-in-io-js-nodejs-p...
You don’t need another DSL to access relational databases SQL is already the best DSL for accessing relational databases. We don’t need to invent something new. Moreover the SQL syntax and features can differ from one database vendor to another.
query = sprintf("select count(*) from data where %s = ?", column)
results = sql.execute(query, filter_value);
WHERE (_title IS NULL OR title LIKE _title)
AND (_minPrice IS NULL OR price > _minPrice)
AND (_maxPrice IS NULL OR price < _maxPrice)And I don't know what it would look like if the requirements changed to allow ordering by multiple columns with different possible sort directions... that might get back into dynamic SQL using a RETURN QUERY EXECUTE type of thing, which is basically using a query builder in your query language.
select c1, c2, c3, ... order by 2;
would order by "c2"
However, the point is that I'd rather use SQL, it's simple as that.
I can't speak specifically to Postgres, but, in my experience, there is nearly always a way to do what you want with bound parameters. No advanced features are required. It often results in poor performance and redundant code that is hard on the eyes. You will get what you deserve, but sometimes you don't have a choice.
Here is an example of the horror:
WHERE (:1 IS NULL OR :1 = item_id)
AND item_num BETWEEN NVL(:2,0) AND NVL(:2,9999999999)
AND item_date BETWEEN NVL(:3,'01-jan-1900') AND NVL(:3,'31-dec-4712')
AND item_name LIKE NVL(:4,'%')
ORDER BY
CASE :5 WHEN 'NUM' THEN item_num WHEN 'DATE' THEN item_date ELSE item_name END,
CASE :6 WHEN 'NUM' THEN item_num WHEN 'DATE' THEN item_date ELSE item_name END
I am not advocating any of this but am pointing out that bind parameters can be abused in surprising ways if you are backed into a corner.
http://www.postgresql.org/docs/9.4/static/functions-string.h...
select(from('tab'), where(and(like('name','Mo%'),not(is('id',NULL))) EXECUTE format('SELECT * FROM tab %s', format('WHERE %I IS NOT NULL', 'id'));I have written TSQL that changes the entire query (which was on the order of 80 joins) dynamically based on about ~20 different parameters, only a few actually passed into the stored procedure, the rest based on settings and configuration. It took a few hours to grok how everything came together at first, but a few tricks that the optimizer will remove means you can add arbitrary conditionals in whatever order you please.
The example you give is VERY trivial for dynamic SQL. postgres - http://www.postgresql.org/docs/9.1/static/ecpg-dynamic.html mssql - https://msdn.microsoft.com/en-us/library/ms188001.aspx
The only time that I've found ORMs useful for composing queries when I have to dynamically create a query at runtime based upon user input. And even in that case, today, I'd probably still just concatenate strings for a proper SQL statement.
I guess the underlying question is how much your development style involves meta-programming (and how much someone enjoy reinventing the wheel with their own query builder engine instead of trusting a mature ORM).
I spent the other day writing a big query, joining about 10 tables. So usually I start by joining the two main ones. Check that gives me the results I want. Add in another table or conditions, check again. Repeat until all tables are joined into the query and conditions are added.
Then I noticed the results of my GROUP_CONCAT were not as I expected. I had a couple of suspect joins that I tried removing. Same problem. In these situations, it is often easier to go through the same process from scratch of adding in one table at a time and ensure that it is working.
We need some kind of unit test equivalent for SQL.
However, sometimes I want one part of my program to be responsible for one bit of a query, and a separate piece to be responsible for something else. To take a trivial example, say I want one object/function to be responsible for doing the right sorting, another to be responsible for any pagination, and yet another to be able to group results when needed. In that case, having a programmatic abstraction over a query (whether it be an object or a datatype, doesn't matter) can be very useful.
I always wonder why developers (usually young and enthusiastic ones...) pick a set of the most challenging requirements they can think of and then use them to justify the usage of some library or pattern that makes life a nightmare in the simpler cases, which are obviously the majority of the use cases.
It's like using a tank for going to work, because the roads might possibly have been destroyed and there might be enemies shooting at you. Then ending up destroying and having to rebuild your own garage because the tank is too big to fit in. Complete nonsense.
It's a pain, because what we really want in such cases is a datatype server, not a relational database. For all of SQL's merits, the NoSQL backlash was not without cause. For too long the SQL hammer has been wielded to pound in too many screws.
Of course, programmers being programmers, NoSQL turned into just another hammer used in just as many inappropriate situations. The example I outlined is not a theoretical one. I've dealt with such situations...but I've done so with Datomic, which is designed with datatype-composability in mind. Similarly, I've found a healthy mix of SQL, Redis, memcache, and Datomic all in moderation can go a long way.
...but I've still not found a solution where Mongo is the obvious answer.
Yes, you still have to understand what you're querying against and how you should build your queries to make best use, but I'd much rather write this kind of code than try to concatenate SQL.
IQueryable<int> Source()
{
return Enumerable.Range(0, int.MaxValue).AsQueryable();
}
class UserFilter
{
public bool? EvensOnly { get; set; }
public int? Minimum { get; set; }
public int? Maximum { get; set; }
}
IEnumerable<int> Search(IQueryable<int> source, int currentPage, int pageSize, UserFilter filter)
{
var result = source;
if (filter.EvensOnly.HasValue && filter.EvensOnly.Value)
{
result = result.Where(i => i % 2 == 0);
}
if (filter.Minimum.HasValue)
{
result = result.Where(i => i >= filter.Minimum.Value);
}
if (filter.Maximum.HasValue)
{
result = result.Where(i => i <= filter.Maximum.Value);
}
return result.Skip(currentPage * pageSize).Take(pageSize);
}
void Main()
{
var currentPage = 4;
var pageSize = 5;
var filter = new UserFilter {
EvensOnly = true,
Minimum = 1000
};
var searchResults = Search(Source(), currentPage, pageSize, filter);
} IEnumerable<int> Search(...) {
return result
.Where(i => !f.EvensOnly || i % 2 == 0)
.Where(i => !f.Minimum.HasValue || i >= f.Minimum)
.Where(i => !f.Maximum.HasValue || i <= f.Maximum)
.Skip(currentPage * pageSize).Take(pageSize);
}
.FilterByEvensOnly()
.FilterByMinimum()
.FilterByMaximum()
Unfortunately EF produces an unreadable query when there are a few dozen filters, but the LINQ code is readable, so there hasn't been any difficulty debugging.
I'm absolutely in favour of using a query builder when it avoids the need to fiddle with strings, but at the same time any decent programmer can write in half a day a query builder that satisfies the vast majority of the use cases.
So the question is: once you have covered those cases, do you need to use the query builder to write static queries? No. Do you need an extremely complex query builder to build extremely complex queries? Again, no. SQL is great for writing very readable and optimized static queries, which are almost all the SQL queries you need when you're not dealing with search forms.
Bottom line: use your favourite query builder or write a tiny one, but use it only when the resulting code is simpler than the plain SQL + string joining.
I myself come from the R/Python background, and my hypothesis is this difficulty around composability (especially for dplyr/pandas crowd) is what deters data scientists from writing more SQL.
If you want composability you should factor the common sql fragments out to parameterized views (table valued functions), or just regular views. Doing this across a large project can really improve performance as well.
Yes? So? When is this really a problem? Even if you do generate queries dynamically. When does this become such a big problem that it is worthwhile killing readability and bogging down the system with lots of extra complexity?
As for ORMs: I've done software for 25 years. I still have not seen a single example of ORMs resulting in better software. Not a single time.
As another greying developer, one thing that was seen with ORMs were implicit schemas, often motivated as bureaucracy shortcuts.
So, you need a dictionary of configuration data for your user, but the bureaucratic overhead of adding that to your schema is too much? Why, then take your fancy OO serialization technology and save that little dictionary as a binary blob and stick it into just one new column in the database.
Problem solved...until years later when information in one of those blobs becomes tied (by a maintenance programmer who didn't entirely understand the system architecture) to what is effectively an implicit schema.
Tools like MongoDB can do things that are extremely difficult or impossible with MySQL/PostgreSQL and they are a great choice for those situations. Using it simply out of laziness or misunderstanding, though, is probably going to create problems later.
The problem with being a young person who wants to make his or her mark on the world is that if the people before you did an excellent job, you can only make things worse. In fact, I think this is one of our larger problems not only in technology but in society (see the recent move to extremism in many aspects of our political and religious life the people who grew up under more moderate times are pushing for).
Some things should be boring and stable. Build on top of those things. Redoing the foundation every six months is just sub-optimal. Accept that you may not make that big mark in the world, or if you do, it'll be in ways you completely did not expect, just like everyone else who made their mark.
Technology changes. It improves. It gets faster, easier and more responsive to business requirements. If you don't embrace change in the IT industry then get out. Because you simply won't survive.
Yes.
It improves.
Sometimes.
It gets faster, easier and more responsive to business requirements.
Occasionally.
There are a lot of ideas in technology that are widely accepted, particularly by younger, less experienced generations, as being The Best Way to do things. However, if challenged, most of those people don't really know why they believe that. They have never personally seen any hard evidence that The Best Way is actually better than known alternatives. They have never spent significant time trying those alternatives for themselves to gain first-hand experience of the pros and cons. In fact, their belief is usually nothing more than a matter of faith, a blind trust in the dogma of the day and the rhetoric they find on-line or hear from their peers or managers.
See also: OOP, design patterns, UML and round-trip development processes, frameworks, ORMs, XML, dynamically typed languages, more frameworks, TDD, NoSQL databases, and so on, many of which were either invented or popularised as a way to work around the artificial problems created by one or more of the preceding items.
Of course I'm being a little facetious here. But if you think I'm entirely kidding, just spend a few minutes with Google and see how much advocacy there has been for each of those ideas in their day and how much actual data there has ever been to justify that advocacy. And really I'm being kind here, because you talked about things that are supposedly "faster, easier and more responsive to business requirements" and I didn't even mention words like "agile" or "craftsmanship" in my list.
If you don't embrace change in the IT industry then get out. Because you simply won't survive.
I embrace using the right tool for the job. Whether it's new and shiny or old but tried-and-tested doesn't really matter, though if I had to pick one or the other for some bizarre reason, I'd pick tried-and-tested eight days a week. There have been only a few truly significant advances in the past decade or two, but very many technologies that were initially much-hyped but then failed to stand the test of time for exactly the reasons that critics outside the hype bubble had always predicted.
Will our profession ever stop being fad-driven?
class User < ActiveRecord::Base
def self.negate(scope)
where_values = scope.where_values
scope.except(:where).where where_values.reduce(:and).not
end
def self.suspended
where.not(confirmed: true).where(arel_table[:created_at].lt 2.weeks.ago)
end
end
unsuspended_users = User.negate User.suspended
You are less likely to suffer from SQL injection if you use at least some kind of wrapper, although a minimal wrapper that takes a SQL with placeholders and arguments would do for that.
My preferred way to wrap is to use EF with a dash of Linq, which is excellent and yields great readability and flexibility.
There is no way I am going to be willingly writing SQL statements in production code (unless I am forced to do so by some legacy problem). Worse case if the SQL is so complex that it is too much for Linq, then write a stored procedure.
Back in 2002 I had to write hand crafted SQL before mainstream ORMs and at the time I thought it sucked bad! I was looking for ways to generate the code from schema but didn't have time to go and back-fix all the legacy code ad-hoc SQL statments!
Pretty much all of our coding guidelines on my team are just guidelines. The one absolute law is that all data going to the DB must be paramaterized, nothing goes in as string substitutions.
But virtually all of our DB access (like, at least 99%) is by stored proc anyway. We're just barely able to keep up with performance requirements by tuning things just right, and in my experience, EF isn't able to generate queries that are as efficient as we can by hand with some fiddling. I know that sounds like "I write in assembler, 'cause no compiler is as good as me", but that is what the reality seems like.
Surely that is something that could be disabled at the driver level for security? Make sure it only send one statement at a time.
Correct me if I have missed something.
Passing data properly through parameters isn't really very hard, there's no point in looking for incomplete work-arounds.
In the end, whether you speak SQL or use NoSQL construct matters as much as whether you write a program in C or Pascal. You can have endless debates about form and function and you might want to see and identify with a winner, but in the end whether the tools and the community surrounding a technology fits people's needs is the only thing that really matters.
If you're using Java (JDBC), there's an embedded kit so you don't have to run a standalone server. If you're not using Java, you can run the standalone server which also emulates the Postgres ODBC protocol so you can (for example) use `psql` to connect and run queries.
Wikipedia runs on MySQL. They have replicated read-only copies of the database and ngnix caches which handle most read-type operations. Is your site busier than Wikipedia?
In my my mind SQL as a language is a huge PITA. First, parsing of complex statements is expensive (there's workloads where SQL parsing takes more time then processing the results).
Second, as SQL exists today (SQL2011) it's a large, complex language that's not implemented uniformly. So I understand why people want to build programmable interfaces for generating queries versus writing giant string statements that expand to SQL.
I personally would wish that there was an alternative language for interacting with relational databases that isn't SQL. Just expose the whole relational expression tree to users say ala S expressions. It's not like the relational engine is going to optimize / re-order it anyways.
I mean something along the lines of:
[ GROUP_AGGREGATE,
[ "name" ],
[ [ SUM, "COUNT(*)" ],
[ SUM, "cost" ]],
[ JOIN, [ ==, "user_id", "customer_id"],
[ FILTER, [
[ &&,
[ >=, "age", "30" ],
[ IN, "state", "NY", "CT", "NJ"]],
[ TABLE, "customer" ]
[ TABLE, "orders" ]]]]]
(ssql->sql #f '(select (columns (col actors name)
(count (col roles id)))
(from roles actors)
(where (= (col roles actor_id) (col actors id)))
(group (col actors name))))
In the above sexpr, the SQL query is expressed as a Scheme/Lisp hierarchical list, which I think is a quite elegant notation. I'm sure other Scheme implementations provide variations on this theme of SQL generation. Using some macro magic, there would certainly be a lot of ways to organize it and probably it's not even all that hard to do.
[0] documentation at: http://wiki.call-cc.org/eggref/4/ssql
The downside (IMO) is that it's still reflecting SQL the language ... not the relational model.
It's like writing an app from scratch in assembly, versus C.. The compiler will do quite a bit of the dirty work for you, and if you look at it in a disassembler no doubt you could find tons of improvements to be made (errmmm depending on the compiler I suppose). Doesn't mean either one is the wrong way to go about it. Now if all you do is stuff ASM code in your C app, yeah.. Why bother?
The crazy thing is most of them are working on a single node, so no horizontal scaling. And they seem to think non-relational is the new way to do things.
Importantly, I don't think that's an indictment of MongoDB. Instead, it highlights the key advantages of NoSQL: ease of use and rapid iteration.
When you're first working on a project, MongoDB is very easy to slap in. You don't even have to create tables/collections. As you iterate, you don't have to constantly be updating schemas as your product direction shifts. Thus, MongoDB is perfect for early startups.
As a company scales, of course it's warts start to show. The very attributes (like schemaless storage) which were an asset when just a few developers were hacking on the project become a drawback as the company grows in size. Reliability becomes more of a concern.
Naturally, people shift to a more traditional database. Personally, I'm almost ready to switch some of our core functionality from MongoDB to Postgres. This isn't an indictment of Mongo though.
It's silly to expect that every single software project requires the same level of correctness or maintainability. A rapidly-developed speculative SaaS for example may be able to trade long-term maintainability or scalability for faster development time (often more important if you have no idea whether or not your idea will even work).
This lets you do things like writing shitty hyperlinear naive solutions to get all the pieces of a system in place, and then going back and optimizing each in turn. You don't spend a lot of time fussing over little details, and instead make rapid progress.
If you don't spend time on the interfaces between the pieces, though, you're absolutely screwed.
"Plan to throw one away; you will anyway."
The quote is Fred Brooks from his book "Mythical Man Month." It seems to me the author's post reflects that quote. Granted they didn't throw the entirety of every application out, but arguably it's an entirely different application now. I would even wager there's months of refactorings ahead of them as they take advantage of Postgresql's features.
The author of the comment I'm replying to seems to advocate a different approach where you choose tech and techniques that are good for short-term but maybe not long-term. This way you have planned technical debt plus unplanned technical debt. Also the comment author seems to be missing one of the post author's main points, which is that MongoDB's schemalessness is awkward even in the early stages of the app, and this is worked around by a less-than-ideal technique of moving the schema from the database layer to the application layer.
This post reflects on developers being lazy, instead of doing it right the first time around. Oh no, you have to log in to the db and run a CREATE TABLE statement every few months when you need to scale. Cry some more.
And even then, 'lazy' is subjective. I don't find SQL that hard to implement. There's a library for practically every language you can think of. What's your excuse?
Put another way, it's like "what? you couldn't spend 10 minutes declaring types everywhere?" - yeah, it's less robust, yet dynamically typed languages remain popular.
My excuse: When I'm just past the mock stage, and still playing with what UI functionality should be, sometimes I just want to get some JSON persisted. I'm changing the shape of the data a lot, discovering the schema as I build. The persistence is frankly a technical nuisance I wish I didn't have to think about.
Sometimes, the result is "good enough" and I don't need to go through the ceremony of glueing in an ORM.
Right now though, I have to write a script to pull some crap out of an Excel file and into my database. I use Scala with the Play framework, but what a nightmare getting typed information out of an Excel sheet is. No thanks, I'll just pull it out magically with Ruby and construct the JSON requests to my real server to create the data. It's a smart business decision and I would defend it all day. I'm sure there's corollaries for "dynamic databases" such as MongoDB.
To be clear though, I'm with the grandparent. I really think decisions for MongoDB are made out of a combination of laziness and eagerness most of the time. I mean honestly, if your data is going to be relatable, why would you employ a scheme that forces you to make those relationships manually? So you go to a relational databases which ironically lacks relationships, but hey there's plenty of support and libraries that will do it for you. But really if you had to ask me my personal opinion? Graph databases where relationships are FIRST CLASS CITIZENS are typically the best for serious projects containing lots of relationships.
Mongo won't. It will take you at your word. Even if that word is wrong.
Choosing mySQL over mongoDB to start with really can't add that much time to a project, especially if you're doing it in a language like Ruby. A couple hours extra here seems very worth it to avoid a multi-week migration process a couple years down the road.
Bah. It's like they didn't know that schema-free data stores mean "there is no schema; different objects may have different fields". This is the whole point of MongoDB: you assume the responsibility of managing the schema. That's a "feature": you get greater flexibility by assuming more responsibility. Whether it's a useful feature, I won't say.
Their second reason is much more valid: MongoDB is not consistent/durable, all that. It's good for a cache, but not for long term data.
Here's a third reason I'd give against it as your primary data store: it's expensive. You have to keep your entire dataset in RAM, but that's not always necessary. My favorite example is from Foursquare. They had every single check-in ever in MongoDB in RAM. That's absolutely unnecessary, and quite silly to do so. Old check-ins are archived data. You don't need them. No user ever wants to know when/where they checked in three years ago. This is why at the time they were paying for 68 GB RAM Amazon boxes instead of 4-8 GB boxes. (I have no idea what they do now. I remember chatting with them on HN after a catastrophic out of memory failure when they filled up the entire 68 GB's).
Regarding consistent/durable: during the past four years, we've not had any problems on this front that weren't caused by us in. We've had an issue that was a misconfiguration on our part where we allowed writes to a server because we pulled it out of a replica set. We also ran out of space on the logging volume once and that caused downtime - but, we didn't have log rotation or anything setup.
In general, we've found the failover very reliable and new primaries have come online without any problems.
That said, schemaless is both a blessing and a curse. Now that Postgres and MySQL have online alter built-in I'd possibly choose one of them if we were starting everything again.
I think we are in agreement about the working data vs all data. I am saying that in most applications your working dataset is much smaller than your total data set. So why pay for hardware capable of holding your entire dataset in RAM when you don't need it?
I am surprised you are able to do this with Mongo. Last I checked, it simply did not handle this case, and started failing miserably if it was not able to fit all data into RAM.
Re: durability: I am not talking about the server going down, coming back up. I am talking about whether there is an fsync() when writing data. Set up a test case where you are writing data very rapidly to MongoDB, then pull the plug on the box it's running on. It'll come back up, but the data it told you it just wrote won't be there because it didn't checkpoint. Did you check that all your writes succeeded when you had node failures in your cluster? Most applications don't have the machinery to do this because generally the state necessary to check this is stored in the database, yet it's the database you are testing. The only way to test this is to also write logs (also atomically), and then verify DB data against logs. Or, just use a database that guarantees durability.
Re: consistency: MongoDB doesn't support transactions [1]. That's enough to exclude it from a large number of applications. Anything to do with money, for example, is out since you really don't want double spending to be a thing.
[1] http://docs.mongodb.org/manual/core/write-operations-atomici...
This may not suit everyone and absolutely does not suit financial transactions. You can bend Mongo to do it using additional collections and money movement logs... but, why bother when it's simpler to use MySQL/Postgres?
Who/what is the 'you' there? Don't 'you' have the responsibility of managing the schema either way? It's a question of whether you want to manage the schema through an rdbms, or... just in your application logic, I guess?
The flexibility you gain seems to mostly be... the flexibility to choose not to ensure adherence to the schema.
Which may be a valid choice in some cases, but I suspect a lot of people don't quite realize they are making it let alone analyze whether it is the right choice for them, and come to regret it.
MongoDB is durable. While it doesn't quite support SQL transactions, it is durable. The data is journaled before being confirmed, and once confirmed will be written to disk. It can be consistent, but this sort of breaks the whole idea of scaling and distributing the load in Mongo, or causes massive performance problems, so that is something to consider when using it. Understanding and adopting eventual consistency is tough, but it's an issue with every distributed database, not just MongoDB.
And you don't "keep your entire data set in memory". You should make sure your indexes fit in memory. Your data can be as large as you like. Most people with terabytes of data in MongoDB don't keep terabytes of RAM in their servers.
Foursquare did NOT keep every single check-in in RAM. They kept an index of them in RAM, sure. But the problem was they had a sharded MongoDB deployment and one of the shards became unbalanced and exceeded the available RAM. If that happens (harder to do these days, but not impossible), it can be very difficult to recover.
MongoDB tries to bridge that gap between NoSQL and SQL. I think the MongoDB folks originally ignored decades of database research when developing MongoDB, but they've been forced to adopt it as the years have passed. Is it an Oracle killer? No. But it can be a useful and productive tool if you understand and apply it appropriately.
It comes down to this: there is no way to tell MongoDB to commit a write and not return until it is durable. You can tell it to fsync() it within at most 33 ms or so from when it returns, but not immediately. There is no control in it to say "fsync() this value because it is important, then return".
That is not as durable as other database engines, which do this based on number of commits that have not been fsynced to the journal (this value can be set to 1 causing every commit to be fsynced).
Consistency is a large topic, and I am sorry I mentioned it vs something more specific, such as:
MongoDB doesn't have transactions, so you can't make multiple updates truly atomic. That's annoying at least, and unusable at most. Two phase commits let you get about 50% there, but with no rollback your application code has to implement rollbacks. In lots of scenarios your application is not smart enough to rollback a "transaction", so your data will be screwed.
Eventual consistency is starting to come under fire precisely because it avoids the hard problem of database design and is offloading it on the application developer, who is typically not as versed in how to create a system where temporary inconsistencies are OK. At best, it means that the developer is now wasting productivity on addressing consistency issues.
You may be right about Foursquare keeping only an index of all the check-ins, not the entire dataset. The main point remains: why keep essentially archived data in RAM? MongoDB does not let you keep a partial index.
MongoDB tries to give up basic datastore guarantees to gain speed and flexibility. Once again, I am not arguing that it's a useless tool. It is useful. As a cache. Or in cases where data loss is acceptable. It is much less useful in cases where dataloss is catastrophic, such as financial transactions. It will also be much more expensive at scale, whereas other datastores optimize for the conditions where RAM is expensive and disk is cheap. You can still run into this case when you are talking about a terabytes to petabytes of data.
> when defining a field as int(11) you can just happily insert textual data and MySQL will try to convert it.
this is dependent on the SQL Mode, which is quite flexible. for example, the STRICT_ALL_TABLES will prevent strings to be inserted in INT fields:
mysql> create table example ( `number` int(11) not null );
mysql> insert into example (number) values ('wat'); Query OK, 1 row affected, 1 warning (0,00 sec)
mysql> SET sql_mode = 'STRICT_ALL_TABLES';
mysql> insert into example (number) values ('wat'); ERROR 1366 (HY000): Incorrect integer value: 'wat' for column 'number' at row 1
There are certainly advantages in choosing PostgreSQL over MySQL... this is just not one of them :-)
Example: The TIMESTAMP type defaults to the current time instead of null.
There are more, but those are the two that have bitten me in the past (and no I do not use MySQL at all if I can help it)
It is possible to change timestamps to default to Null. For backwards compatibility, the previous behavior defaults:
http://dev.mysql.com/doc/refman/5.7/en/server-system-variabl...
MySQL has a large legacy of being used as a very-immediate-although-somewhat-toy database at its roots.
For example, in absolute terms, I would find much more troubling the usage of non-transactional tables, justified by meaningless microbenchmarks, which has been somewhat common for some time.
Nowadays MySQL is definitely reliable, and it has a much more expert surrounding culture than the past, so if a person/company is willing to put time and knowledge, it's a reasonable choice.
I don't find automatic conversion so damaging that people should stay away just because it's default.
When you reach some level, you definitely need to have a relatively intimate knowledge of your tools, and at such point, one is far from the "defaults".
- STRICT_TRANS_TABLES is on by default for "new installations" starting from MySQL 5.6 (2013). What this means is that the bundled config files all turn it on.
- Starting from MySQL 5.7, it is a compiled default, along with several other more "strict" options. Effectively making it default to ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,ERROR_FOR_DIVISION_BY_ZERO, NO_ZERO_DATE,NO_ZERO_IN_DATE,NO_AUTO_CREATE_USER.
I have some sample configuration files to make 5.6 behave like 5.7's strictness. For example: https://github.com/morgo/mysql-compatibility-config/blob/mas...
It does make upgrades harder, so some applications may also need to use a whitelist/blacklist approach to transition: http://www.tocker.ca/2014/09/01/suggestions-for-transitionin...
Actually, with modern versions of Rails, using Sequel in place of ActiveRecord isn't bad at all. Nothing in Rails is really tied to ActiveRecord anymore. There are dependencies on ActiveModel, but you can easily make Sequel::Model objects conform to this interface. Sequel-rails helps with most of that: https://github.com/TalentBox/sequel-rails
It's really just the migration that's difficult, as that's tough to do piecemeal and requires good test coverage.
Here's a recent side project of mine that I switched from ActiveRecord to Sequel pretty quickly once I remembered how extremely limiting the querying capabilities of ActiveRecord can be: https://github.com/bgentry/portfolio-api
And if you really want to keep the option for schemaless data storage, Postgres can now do that with better performance than MongoDB, while keeping full indexing capabilities: http://blogs.enterprisedb.com/2014/09/24/postgres-outperform...
The main reason we started using it the first place was composite primary keys (not supported in AR, at all), and we've never looked back!
Developers, who reads the article beware of reality, you can look back 5 years now, not looking forward 5 years ahead. Don't waste time in dealing with database columns and schemas, instead build the business faster, I found Mongo DB or any Document DB is good fit for agility.
1000's of business move to Document DB because it is schema-less.
While working on Document DB, you should be master in writing stand-alone scripts in Python/Perl/Ruby to run every-time you break the structure, or fix inconsistency.
edit: fixed typo errors
As in a few terabytes of data, >10k ops/second territory.
I've been really disappointed with its reliability and performance in situations where I've been around that.
It doesn't scale without tremendous effort and implicit schemas are a very dangerous thing to introduce into your application, they're insidious, and require enormous diligence in the application to codify the schemas (I would only feel comfortable using a loose document store with something like Haskell in which I can model the schema with strong types).
I've encountered MongoDB in three different companies / products and expended much effort to immediately move away from it in every case. In each case, the solution (which has been different each time) was far more appropriate to what was needed.
There's a sad inclination by developers to pick "one ring to rule them all" tools and MongoDB I believe even sells itself that way. It is not.
You can always run db.currentOp() in the mongo shell to see what process is taking forever as well.
Let me clarify why though, there methods of optimizing a query by adding another field, but since I have to traverse my records with the sort() cursor my queries take that long.
I remember querying databases with 100 tables and millions of records in foxpro a century ago and it took less than a second.
What has happened to the world while I was in cryogenic state? Take me back to the nitrogen pool!
I think scaling any system takes careful planning.
IMHO, this is an area where MongoDB has improved a lot in the past two years, especially with 3.0, but there is still a lot of work to do.
http://dev.mysql.com/doc/refman/5.7/en/innodb-create-index-o...
If you specify ALGORITHM=INPLACE,LOCK=NONE you can alter table without blocking reads and writes. We have used this method successfully in Amazon RDS when updating schemas.
> If you specify ALGORITHM=INPLACE,LOCK=NONE you can alter table without blocking reads and writes. We have used this method successfully in Amazon RDS when updating schemas.
The use-case of ALGORITHM=INPLACE and LOCK=NONE is to produce an error if the modification you are attempting is not supported in this mode. i.e. even if you don't specify LOCK=NONE, that doesn't mean it will lock.
This is useful in preventing guessing games (i.e. you think its LOCK=NONE, but for some reason it's not compatible...)
Plus your problem with checking if a field exists is because Ruby doesn't support property attributes. This is easily solved in C# using attributes. Which you can fake in Ruby. http://stackoverflow.com/questions/1085070/how-do-i-fake-c-s...
Data consistency is what your models are for. No data should be inserted before being assigned to a model.
Again it sounds like you needed a shared library or just a service wrapped around your database.
I'm glad that the OP is growing as a developer and starting to understand the merits of SQL and why having defined schemas is important. SQL is awesome and its not going to be replaced by NoSQL (at least not fully). However please make sure that you are building the foundation of your understanding by fully grasping the technologies that you are already using.
With all due respect, you should have had this list before selecting MongoDB. Consistency, as in "eventual consistency" and also in the way that you describe it is better supported outside the NoSQL group.
Also, I don't fully agree with the "schemaless" discussion. The moment you think about your data, you build a schema in your head, which, then, is translated into code etc. It's about how far you take this rigid model. Maybe you should see it as "flexible schema".
FoundationDB looks great, I haven't used it yet but they appear to have their heads on right.
PostgreSQL, the newer versions, have indexable BJSON data types so you can get the same exact behavior from Postgres as you do from Mongo but with a true RDBMS along with it, a dependable storage engine, etc...
Postgres is harder to scale horizontally though - if you have really high-volume data writes, you should be using something else for that.
I typically use PostgreSQL for all of my highly "structured" data and Riak for high-volume and "flatter" data (Postgres also often serves as an index into those objects).
Don't prematurely scale, just pay attention to your metrics, scale vertically first, then tackle the very specific pain points.
If you need schema-less data storage in a "real" database, use PostgreSQL's JSON type.
http://clarkdave.net/2013/06/what-can-you-do-with-postgresql...
Replication is for high-availability not for consistency. As long as you can live with that, the reliability is ok.
A bit of both really.
NoSQL databases allow for rapid prototyping, as do weakly and dynamically typed languages. It's amazing if you want to just get a product out of the door. NoSQL is the short term answer. And MongoDB is the answer if writing your data to /dev/null feels like a good idea to you..
However, strong typing and regular databases offer consistency. You have no way of going wrong because it would have refused to compile twenty times before you even think about pushing your (wrong) code in production. Sure, you can force yourself to get such consistency in MongoDB. But first, it's pretty taxing mentally, and secondly if you're doing that, why not go the way of a relational DB which offer you tools to enforce that and is faster ?
KV columns in databases feel wrong to me too, I feel dirty using PGSQL's json storage because I feel like I'm throwing normal forms away. But at the end of the day, what matters is that your product works.
>But at the end of the day, what matters is that your product works.
With my requirements, I could just write to a flat file and be fine... I seem to like complicating things just enough that I no longer understand how what I'm building works. LOL.
That said, there are applications for other less-structured datastores like Redis. When you need to store data with an expiration quickly, and use common datastructures like sets, Redis can be fantastic.
Okay, you lost me there. Why does it need to be schema-less?
1) Implicit Schemas. We avoid this completely by doing production migrations and re-indexes on every new field we add. Its expensive, and we have to write/test/run scripts in production. At times I wish I could just write a Rails migration on an RDS instance and call it a day. N-1 compatibility isn't hard to accomplish with good code reviews.
2) Search. Want to search and join like in the old days? Good luck. Using a NoSQL DB as the primary authoritative store of record is great, but you'll need a secondary indexes for any searches you want to do. If you need to look up an object on a new field (or even one that already exists in all of your data) if you haven't built an index for it you will have to.
3) Serving Clients. Because of the schemaless-blobby nature of writing clients, things get real messy the minute you have multiple services or applications reading or writing to a DB. To get around this, you have to put a service in front of it and serve the data from some RPC technology, which is an extra step and requires more development and maintenance.
4) Administration. There aren't a long history yet of robust toolsets and "science" behind different schemaless NoSQL databases. Meaning which one you choose has a huge impact on your ability to fine tune it, debug consistency or other expectation issues, and do things like proper failover, backups, restores, etc. Knowledge between similar NoSQL DBs doesn't transfer as well so your mileage will vary more so than on SQL databases (MySQL vs PostgresSQL, for instance).
NoSQL has its place and purpose, but it is rarely as the "one database that rules them all" that many businesses end up with. I'd be interested in counter stories.
Guess what. Same thing applies to SQL.
The issue I have with SQL (MS sql in the case of work) is the amount of cleverness involved in some queries I have seen. Among the old timers, it seems almost a badge of honer to develop the longest, most clever SQL query that does everything in one step. Inevitably, there are problems, and people have trouble figuring out why because they can't debug parts of the statement in isolation. In this case, it's as if they have written an entire program in one line and can't test parts in isolation. No doubt this is abuse of the language... and admittedly my SQL skills are not world class, but I keep scratching my head and wondering why they do this to themselves. If a query is so complex you can't tell what it is doing, and it misbehaves, perhaps you would have been better off with some smaller queries you join in the program... nothing against joins... one or two or three of them... but really....
I keep asking myself if I'm missing something, but I sort of doubt it.
When you need really complex queries, you should think about a new db structure.
Thanks for the insight.
The version of MongoDb used is not mentioned in the article. The changes between versions is a problem with Mongo but it improved much recently.
One problem clearly identified is the lack of model. There are solutions to this. For instance, with Opa (http://opalang.org) we use strong static typing to generate a database model and guarantee that the whole applications sticks to it. That leaves out model changes, but there are solutions for that.
Also, there is no need to have the whole collections in RAM, but clearly enough RAM for the "working set" helps.
In the end, Mongo is no magic so do SQL databases which have their share of problems too.
Here's the thing; anyone that claims one is superior over the other without adding a specific use case or context is doing something akin to religious preaching. Databases are tools. Pick the tool you need for the job. If you don't understand exactly what a hammer is for and how it does what it does put effort into gaining that understanding. If after that research you think the tool has flaws that are important to you then don't use it.
It is BigTable.
Generally, if your data is not very large, you should use a SQL database. NoSQL is mainly used for easy scaling, not for its schema-less feature.
I think it would very helpful to use here ToroDB (https://github.com/torodb/torodb). While being MongoDB-compatible, it stores data structured into PostgreSQL tables, automatically identifying the schema of the data. Then, just by looking at the created tables, you very easily have the schema (and the data migrated to that schema). It would make migration easier.
Disclaimer: I am a ToroDB developer
The trend to go back to SQL databases now is fueled by the "same" poor reasons why people were going for NoSQL databases. You can't have everything...
I've been on two straight projects where Mongo has been fine, no better or worse than an SQL database, but certainly nowhere as bad as one would assume reading all the negativity around here. I have experience with it being stable in two different contexts in production (social games, a very common use case, being one). A search for "social games mongo" should show you some success stories.
The thing that makes it a good choice for me when starting a new project is that you don't really think about it, especially if you're following lean startup methodology where the goal is to get a product into the hands of the users as quickly as possible & you anticipate changes to your models. Mongo is pretty fantastic for this. And when things stabilize, moving data really isn't that bad or expensive if you modeled based on this assumption (which you should). And now that PostgreSQL has a pretty good JSON datatype, it presents a pretty painless path if and when you outgrow Mongo.
I would agree that having too many nullable fields could indicate a problem with the schema, i.e. you should break things up into more tables.
If so many people migrate to other technologies, why is MongoDB still so popular? Have a look at http://db-engines.com/en/ranking_trend, MongoDB has just passed PostgreSQL
My eyes practically fell out of my head when reading this.
If the code has to handle both page.title and page_title, this is a feature of using a schemaless technology.
Also, lots of the issues the author had with MongoDB are also to be found in MySQL, e.g. taking hours to recover from a corrupt database/datastore.
With a DB that enforces a schema, the overhead of modifying the schema tends to moderate that nature.
I once worked in a lab (Ph.D. students are atrocious programmers BTW) where I introduced a schema-less store (cheesy K/V store) to handle some mundane metadata caching on some medical imaging. It was intended to store 4-5 attributes per PK, I never touched it after setting it up for what I needed but showed it to colleagues.
Fast forward 10 years and there were something like 3000 attributes defined. Several hundreds of which were serialized blobs. Huge amount of overlap between the different attributes.
Almost all that because people didn't know what was already in there so they just did their own thing.
Supposedly WiredTiger for M 3.0 will be more everything, including faster and better.
We'll see.
Aside from redesigning the schema/model and changing all the code, what about new backup/restore procedures? Scaling and performance best practices? Did you need to hire a postgres expert?
Backup/restoring is handled by Amazon RDS in our case, in the past we had a custom backup system for Mongo that backed data up to Amazon S3.
Most of our applications require write access in some shape or form, so at least the default replication of Amazon RDS doesn't cut it. Besides that we don't really need it so far, don't see the need for it in the coming months either.
Not even vanilla replication helps, right now. I think I'd have to use something like slony to replicate between different versions of postgresql, but I never tried it.
How does upgrading postgresql versions work in RDS?
Am I the only one here who's thinking that this is correct behavior on the part of the DBMS? Three result codes from an operation: 1) everything is okay, 2) I'm sorry Dave, I can't do that (error) and 3) Okay, if you insist, but I'm going to change your data to make it work.
Am I the only one who thinks: the programmer should be aware of and respond appropriately to ALL THREE, not just 1 and 2.? That anything else is just laziness?
Or is that just me? Am I missing some subtle consideration here that results in my thought process being naive? If I'm being naive I do want to understand what I'm missing, because getting schemas right and having my code react when improper data types are being used is sometimes a pain, ORM or not. But I've always thought it was the right thing to do.
Schema says int? Give it an int or fail. Not "it's sort of okay if you give it a thing which could be coerced into an int according to the database's ideas of coercability".
If anybody is going to talk about scalability, there are many good alternatives to scale PostgreSQL too.
That changes those few hours into few seconds.
https://gist.github.com/neunhoef/9b6749089775e472d44cFor everything else, there are relational DBMS.
There are so many things that I've heard you "can't do in SQL" that are false, at least pertaining to Postgres. Semi-structured data, full-text search, "web scale" programming, geographical indexes... all of these things, people say "you can't do in a relational database" and that's not true. Postgres is fucking powerful and can do a lot, very well.
Arguably, PostgreSQL isn't always a "relational database". You can use it as a key-value store. It just happens that you often want relational logic in a multi-purpose, long-lived data store. The relational database seems to be an attractor; the requirements that accrue to a typical in-house "we can do it better"/NIH non-relational database often converge on it.
NoSQLs have their place at very large scale (100+ TB) and there are plenty of specialized reasons to use alternative databases-- I doubt that Postgres's full-text search is competitive with Elasticsearch-- but I feel like most of the anti-SQL sentiment is against the language. And sure, it's an ugly and outmoded language, but the database is one place where I'd rather have an ugly language and rock-solid tech than the other way around.
Finally, fuck ORMs.
Usually the same people that think running an application on a cloud platform magically makes it fast, when in reality a VPS would be cheaper and faster.
Amen to that. Many times I wondered, geez I could've written this with a simple SQL query, here I am reading the docs.
Not to say people aren't running these types of Postgres clusters, but its something not easily done.
I think the anti-SQL sentiment is a very MongoDB-esque thing, in that MongoDB tried to replace everything. Most of the other popular NoSQL solutions have very clear and cut use cases, and are pretty upfront about selecting the right database.
#1: No schema. If you're prototyping, schemas just slow you down. Maybe that changes as your application matures.
#2: Great drivers. The amount of pain to get a basic CRUD app running with a SQL backend is just too high: push the data from the browser to the server in JSON or XML, validate, convert into database schema, sanitize using prepared statements or stored procedures, and send it off. Compare that with MongoDB: it's just JSON™.
The one thing that killed MongoDB was lack of reliability. Even when you're a startup, you may not lose data. I think MongoDB could have easily dropped 50% of performance & scalability for data security and it would be well off now. It was never intended to be the safe, sane choice for big enterprises.