“Fashion Is Hard. PostgreSQL Is Easy” [video]
tech.zalando.com
tech.zalando.com
I think it's better to default to the tried-and-true piece of software than the new and shiny, rather than the other way around and have to argue why Postgres is a good fit for your data.
If it's system wide and you need PostgreSQL itself able to handle it, using PG's async features is one potential option but setting up and managing a Postgres-XC cluster would be the next best. XC allows scale-out for writes. If you're at a company with a budget for that type of thing I think I remember reading that EnterpriseDB (the PG company) is offering first class support for PG XC.
In most cases though, I find that the write heavy parts of a system are so isolated that diverting them to a simple NoSQL solution tends to be easiest (Mongo, Couchbase, DynamoDB from AWS, etc).
That's the one I'm familiar with, anyway.
(+) Write capacity is O(n) as you add machines but an individual write's time is pretty constant and cluster-wide maintenance operations do start taking longer as you add machines and they gossip to each other. It's not magic, obviously :)
Ok, that not completely serious, but almost completely.
That is actually exactly what you should expect. SIGINT cancels most things you run in the CLI.
Interrupt a ping because you forgot a count parameter? Ctrl+C
Honestly, the reason I don't use Postgres has nothing to do with the CLI and everything to do with the ecosystem for Multi-Master being inferior to Galera. But yeah, I'm happy with mysql-cli as well and don't really see the Postgres equivalent as a noteworthy improvement.
Deleted comment
The following things (off the top of my head, and quickly verified) all handle Ctrl-C so that it kills only the current command and not the containing process:
bash, zsh, csh, ksh, mail, gnuplot, gdb, psql, vsql, python (interactive), ghci
While I object to captive user interfaces in general, if you're going to have one then handling ctrl-c inside it is the right thing to do.
I've got my biases, but whenever something like this comes up, I just make some popcorn and enjoy the show.
In MySQL the backslash sequences are commands to the mysql client for things that are implemented in the client and have to do solely with the command line client software. These things are not available using other clients. Vs things like listing columns and databases and functions have their own DDL language constructs entries and are queries in their own right that can be sent by any client software to the server and return iteratable results. MySQL is more like Oracle in this regard.
That distinction isn't obvious in psql and that contributes to a larger hurdle when using the command line client. This makes psql attractive to power users, as there are a lot of shortcuts for things that might require longer DDL statements to be typed or queries against information_schema.
Cool! How?
I often invent mnemonics that only mean something to me until muscle memory kicks in.
(I even had to check the fact that C- is the ctrl key when looking up this command. I noticed commands listed as M-w - I have no idea what keys that represents).
Thanks a ton, I didn't know that yet. I also find MySQL cli very easy to work. But that's going to change now, it seems.
Anyway, you might want to check http://pgcli.com/ for a free alternative cli with completion. I started using it over the standard psql and I recommend it to everybody.
That... scares me, more than a little. I'm okay with it crashing, or losing queries I wrote. I really want failure cases to be restricted to NOT affecting the database, though. I'll probably stay away or try it out on my personal machine on a toy database for now.
Prefer it to MySQL Workbench even which I liked.
For instance, the "explain query" visualization is great, and very helpful, but there's no way I can find to zoom out, at all. And that window doesn't support mouse scrolling. And there's no way to export the visualization somewhere else. So when I'm trying to figure out why a very, very large query is running slowly, it's agony to use pgadmin3.
Do you know any good postgres query analysis tools?
I hate the backslash commands of postgres, about as much as I hate the show commands of mysql (what's not nearly as much as I hate the oracle introspection tools).
Why couldn't SQL standardize database introspection?
Thanks.
90% of the time, I don't need to remember any special commands for introspecting and since it's on Windows - I can use the keyboard to navigate the entire UI with ease. When I write SQL, I get glorious Intellisense (that's autocomplete to you) for every single object (edit: and command) in the db.
But for a personal project I went with MongoDB, because my data set was a perfect match for mongo's design.
Now I love mongo too, I'm amazed with how easy it has been to maintain 100% uptime on comodity hardware (one server is literally in a room in my apartment) through all the random server downtimes, upgrades, migrations, etc.
And now I have more ideas for some personal projects, and they would go very well with postgres, but I'm so missing replica sets from mongo.
If postgres would have something similar to replica sets in mongodb, that would be amazing.
https://www.compose.io/articles/is-postgresql-your-next-json...
But I would love it even more if it were easier to distribute (see replica sets in mongodb, with auto-failover and other goodies).
http://www.postgresql.org/docs/devel/static/functions-json.h...
That said, if you're doing a lot of mutation of large JSON documents stored in a single Postgres row value, the storage/concurrency control behavior still won't be ideal.
I meant that it was a valid/recommended use case for MongoDB since I didn't really have any relational data (every document inserted was pretty much standalone).
I've mentioned it because I've seen plenty of posts around HN where "mongo sucks" because they tried to fit a round shape through a square.
The extra goodies from MongoDB helped too.
Like automatic failover, I can literally go and unplug a node and everything will still be fine.
Having tail -f functionality in the db was also pretty handy (for my project).
The sysadmin in me was happy too, it's not everyday you see software that allows you to upgrade between major versions / storage engines without downtime (when using replica sets, not standalone, of course).
It is a desktop (as-in, it has normal desktop components, although used headless like a server), but it's got plenty of RAM, good CPU, RAID1 on WD Re hard drives, 100mbit connection and hooked up to a UPS.
In the last year it has been more stable than some of the cheaper hosting I was using.
Besides, it's not running the whole replica set, just one member (out of 3).
1) It doesn't matter, which means the time, energy and money spent setting it up was squandered when it could have been spent on marketing or product dev.
or
2) It does matter, which means now you have to blow even more time, energy and money recovering it and standing it back up. Hope you've rehearsed your DR plan!
I'm not trying to preach, I apologize if it's coming off that way. But this highly resembles tinkering, and tinkering doesn't generally pay the bills. Usually the opposite.
Can't that happen anywhere? Regardless of the type of hardware.
> But this highly resembles tinkering
Guilty pleasure.
> [...], and tinkering doesn't generally pay the bills.
Thankfully, I was aware that it most likely won't be paying the bills, and considering I've made 35€ from it in the past year and a half, I guess I was right :-)
I've made it for myself (and opened it for the rest of the world if they need it), but I'm my most demanding customer, that's probably why I expected nothing less than 100% uptime since I launched it.
And I've managed to do that, without breaking the bank.
I don't know how my tone sounds (I'm not native), I'm just trying to emphasize that with the right tools, you don't need a shiny cloud for really good uptime.
Which is probably a more unorthodox use of databases these days, at least for Postgres (I've heard it was more common for SQL Server and I once had the questionable joy of debugging a petri net solver in Oracle).
Though, with modern PostgreSQL, views-for-everything would probably work at least as well as stored-procs-for-everything.
It was a pretty great interview, and I almost took them up on their offer. So don't get me wrong, I neither want to praise their stack nor put it down, I just thought that their ubiquitous use of stored procedures was an interesting fact (and Postgres with its pluggable languages supports this pretty well, these days you can even put your js/v8 code in it, if you really want to JavaScript all the thigns).
They've got some code about it on github: https://github.com/zalando/java-sproc-wrapper
He also mentioned that Zelando does 100 data model (schema?) changes per week, and that senior dev make code changes without code review.
Once you get past building todo lists, you'll end up with complicated transactions that do many different data mutations, usually along with some simple logic. You can either maintain your virtue and pay the latency price for each of those, or just code a proc that implements the whole thing. It might offend your software dogma, but it'll get shit done so you can get home to the wife in time for dinner. These days, that is the only deadline I care about.
Obviously if your just writing a tool that will be used 2 or 3 times that - you can make trade offs - but you should never use $sql = "SELECT * FROM tblBobbins"
We have a setup where our WAL log is rotated every minute and backed up to S3. Replica databases can either load from S3 or connect to the master for streaming - so they will catch up using S3 and then connect, reducing load on the master.
MySQL has some features that I'm really missing in PostgreSQL, like more flexible compression, and batched index writeout.
It looks great on paper, but unless you're doing development I would discourage its use. That said your original point about the replication is true, but there are shortcomings that come together with it.
Some things that you will learn if you use RDS:
- if you decide to increase volume size, change type to SSD or use provisioned IOPS, you might have database down for an hour or more. Regardless whether you use single instance or HA.
- want to upgrade 9.3.x to 9.4.x? Tough luck you have stop dump the data and provision a new instance. You can't use postgres' inplace upgrade method
- you have limited control regarding tuning, many settings require rebooting the whole thing, when normally you would just restart the process (with HA there's still several seconds of interruption)
- you can only use extensions that they provide, there's almost no extension to monitor performance
- you can't login to use shell to monitor the process (obviously, but it's a still shortcoming)
- can't replicate data across regions or outside of RDS (it could resolve some of the issues above)
- it can fail, we had two times a failure caused by AWS, it happened during backup period. Normally AWS does backup on the secondary database, but we learned that in those instances it got confused and was attempting to do it on primary, rebooted it and for about 15 min the database was unavailable.
I think I am fine with most of those, since I am saving on labor costs. Periodically taking an app down for maintenance is par for most courses. And even on the last bullet, I've seen really talented database guys make mistakes and have small amounts of downtime. Perhaps what I should have said was "high enough availability". If I needed 5 9's, I agree that RDS is probably not the tree I want to be barking up.
Cheers!
I was more concerned about relying on it for 24/7 operation. Such as for transactions on a webpage. The zabbix scenario in our case also requires 24/7 uptime, but if it goes down our site is still up we just won't be alerted when something else breaks at the same time.
I agree that pgAdmin leaves a lot to be desired. It needs something on par with SSMS.
BTW I would like to send out a BIG THANK YOU to the guys that did the videos - this is such an important and great service for everybody who could not attend the conference, so: THANK YOU VERY MUCH!
There are some great content management tools out there, my favorite being apache jackrabbit, and on top of that, apache sling for displaying/accessing the content. It requires a different way of thinking than the standard mvc process - it is a more content focused display process.
I don't know... if you're using Postgres to store non-relational data you may want to reassess your strategy. You may find you're doing the right thing still, but I think most people would be better served with a system like jackrabbit/sling, which is is used by publishers, fashion houses, etc (via Adobe's product that is built on sling) to store and display their content.
(that said, you can use postgres on the back-end with sling)
What kind of data lacks any relations whatsoever? Yes, there are cases where fitting data to the relational model is more awkward than others. But I would be really, really cautious about thinking that I could design on my own a data model that's better than the relational model. Relational databases are incredibly battle-tested across every type of data storage problem out there.
> What kind of data lacks any relations whatsoever?
Think of a traditional RDBMS application such as an ordering system where there are strong relational constraints that need to be enforced before committing your data: ie, your order items had damn well better correlate with an order, which damn well better correlate with a user, or else nothing makes sense -- you had better raise an exception and should definitely not store those things in your database if the relational integrity is not there.
Now constrast that with other types of data collection where relational integrity is much more relaxed.
Generally, these examples would be (near) real time data collection - times when you're collecting data with a "collect first, analyze/correlate later" mentality. Specific examples would be a service that aggregates logs from a server farm in real time, or things like tracking user behavior statistics on the web. Or even an onboard computer that collects data from a car's engine - you want to read that data every 50ms no matter what, and maaaaaybe you will correlate it with events (accidents? combustion issues? whatever) later on oooooorrrr maybe not.
But if you're running a website and you have fewer than 100 servers and you think you need to start rethinking the entire relational data model, you may very well be getting ahead of yourself.
When you look at it more closely, the Samwers have effectively managed to reallocate billions of dollars into startups all over the world in places where raising money, or working for a startup, was practically impossible. Remember the second largest location for venture capital funding, the UK, only receives about 1/20th as much as the Bay Area alone.
The more mature Rocket companies (btw Zalando isn't, strictly, a Rocket company - it had a distinct set of actual founders) are now pretty desirable employers in their respective countries, and deliver relatively good customer service, certainly better than many much bigger and more resourced companies, and usually crushing the non-existent, overwhelmed service you can expect in most emerging markets. Is the backend a little messy and "move fast and break things", sure, but I was surprised by how BAD most backends actually were in the business world period, and the codebases I had a look into were, by those standards, alright (for example: enforcing foreign keys! a luxury in the Era of Mongo and "logic is in the ORM"). What about the warehouses? Perhaps not quite Amazon, but definitely a huge step up from the average e-commerce startup. At least you know where your inventory is, you're not somehow missing 3%.
Yes, Berlin can be a bit abrasive on egos. I left in part because of that, or at least it tipped me over the edge of setting up my own thing after dealing with one too many fake-angry German (hi H!). But with a bit of time and distance I really appreciate what the Samwers tried to do, and there's definitely a lot of founders who got their "change in life" thanks to a few months/years in a Rocket company, making their mistakes at someone else's expense.
It's also quite impressive that Oli is still as hands on as he is, very numbers driven despite flying continuously around the world, spending less than a day in each business yet knowing their numbers better than their managers. I think the tone of the email, and his motivational speeches, is taken out of context; also, the "vision" is, when you strip the hyperbole away (in Australia: "I do not want to be before the wave, or after the wave, I want to surf the wave"), relatively accurate. By modern and especially American standards, he's quite a polite fellow if demanding.
They have a huge selection, expensive advertising campaigns and lenient shipping & return policies, fueled by huge sums of money, so they have a huge market share. Its nice, but execution is merely decent imho.
Addidas jogging pants -- 59,95 €
Volcom T-shirt print -- 17,45 €
Under Armour T-shirt -- 39.95 €
Like really now -- who needs this stuff? Especially when you can just go to Aldi or Tchibo (or any of a number of other discount retailers readily visible at nearly every shopping mall in Germany) and buy essentially same stuff (minus the logos, of course) for 1/10th of the price. So perhaps a better title might be:
“Fashion generates revenue, and handsome pay-outs. But is basically pointless. And a soul-sucking waste of time.”
Big brands like Adidas or Under Armour don't have a clean track record either, but at least they are somewhat accountable to consumers because their #1 asset is their reputation. To fix global trade, it's better to do a bit of research into what kind of company you want to support, rather than blindly buying the cheapest imported thing "because they all suck anyway".
That is, at the end of the day, 80% of the 10x overhead you pay at the premium retailers is just for the brand, not for safer products or better labor conditions (which are at best a secondary matter of concern to these outlets).
The "poor people" won't benefit from most of the branded product's premium, of course. But it's more likely that it's been produced by a subcontractor that at least respects local laws like minimum wage, construction standards and regulations concerning toxic materials.
As it ever been studied if the work conditions of the workers making branded stuff are generally better than their counterparts working on cheaper clothes?
Or that the 10x factor in price overhead is in any substantive way directly invested in safer product standards. No matter how you slice it, to a large extent all we're really getting when we shop at specifically fashion-outlets (as their marketing gurus know all to well) is... the brand.