Yandex.Mail's successful migration from Oracle to Postgres [pdf]
pgcon.org
pgcon.org
* Reasons to migrate were: unresponsive support, inconvenient deployment, closed source, huge cost.
* It took just 10 man-years to rewrite whole project from Oracle to Postgres (e.g. 10 developers, 1 year), which is amazing.
* They benefited from Postgres data versioning, arrays and composite types.
* They liked writing logic on PL/pgSQL more than on Oracle PL/SQL: reduced code size, increased test coverage.
* Easier deployment of changes due no library cache locks.
Great thing for community is that Yandex now commited to Postgres, meaning it will get more testing and bug-fixes faster.
That remains to be seen. There's a fair amount of huge postgres users that don't even bother to report bugs they find...
If Uber DBAs came to IRC or any public place (see http://Postgres.chat) where Postgres experts are present and didn't hesitate asking questions, they would still be running Postgres.
Thinking about switching untrusted one over to Fastmail. Still surveillance state but less surveillance and supposedly good uptime.
Rob and the Fastmail team are most likely the best in the world at what they do, and as an IT guy with decades of experience, I don't say that lightly. They have more than earned my shekels every year. Give them a go... you will be very pleasantly surprised.
My endorsement of Switzerland is about legal jurisdiction, culture that respects privacy, and stability. Well-intentioned owners won't be forced to do the kinds of things we see in America or Russia. At worst, it will be something selective with a warrant. A nice foundation to build on.
Certain people in various countries could make a similar argument about US hosted emails. This is how the world works.
In Russia, literally ANY road policeman not only takes bribes, but has a bag of tricks how to get more from you.
Unfortunately it's well-known to me, from my own experience. (10+ years of driving on Russian roads, 4+ in California)
But between bribing and things you mentioned there is one essential difference, that changes everything. It's this question:
- Are rules known in advance, are they written and available in public?
I believe this question helps to distinguish black and white.
Even though everyone I know here calls the process "bribing", it's greasing the wheels - in some ways, it's bad that it happens. At the same time though, everyone is allowed to participate in the bribing and whoever takes the bribes can and will settle for what they can get.
I wouldn't say dishonesty is more pervasive - quite the opposite, most people are brutally honest, sometimes to a fault. When it comes to official things, it's true, you can grease the wheels and get preferential treatment. Certain problems can go away with a few well placed notes, and so on. Schools are a bit different too since the collapse of the Soviet Union. From what I'm told, there's a lot of pressure from the state to ensure that students pass, so students will repeat through classes as long as necessary to pass them, and often professors will just give out the lowest possible passing grade to get rid of bad students. A little bit of cash will net you a better grade, which will get you access to better universities.
Just as an outsider looking in and watching, I see it more as the same bad system that exists elsewhere but at least the system is open to everyone.
"Greasing the wheels" is a popular euphemism for bribery, not a distinct different act which contrasts with bribery.
See there's another slight disconnect with service in Russia versus service in the US. You ever have someone go out of their way to make something easier or better for you as part of their job? Go the extra mile? That's not really the status quo in Russia. You goof up your registration form at the post office (one mistake on the 4th page of a 10 page document)? Guess who's filling it out again, even if the office worker has white out handy. Only have 30 minutes to take care of something at the nearby government office but it happens to be 1:45 pm and time for the official's tea break? Guess who's not getting their document done.
When people are talking about greasing the wheel, it's getting access to systems they otherwise don't have. It's people going the extra mile on demand, getting the inside scoop, making sure a document gets processed today, taking care of problems for you internally. Do the big bribes happen? Yeah, I'm not denying that. My point more is that the "bribes" that are talked about so much aren't really what we normally think of as bribes.
That's pretty much my experience in most government offices in the US, too.
> When people are talking about greasing the wheel, it's getting access to systems they otherwise don't have.
Paying privately to the official for access for service from public officials that is not generally made available is "bribery". Yes, "greasing the wheel" is a euphemism for bribery that is often used particularly for small bribes for small favors (though "small" is relative to who is speaking.)
> My point more is that the "bribes" that are talked about so much aren't really what we normally think of as bribes.
They are exactly what most people with experience in domains where there is the kind of lack of accountability which makes pervasive bribery a thing expect as the most common kind of bribery. (When there is some accountability, the perceived risk/reward of smaller transactions becomes less favorable more quickly than is the case for larger transactions -- if as a public employee you are going to get fired and be unemployable in the public sector, and maybe prosecuted, if you are caught taking any bribe, and there is even a modest risk of detection in even small bribes, its no longer worth it to take small bribes in any case.)
To use a restaurant analogy, tipping the bartender to get quicker service is "greasing the wheels" but tipping the bartender to get drinks "on the house" or to get an exclusive event without paying the bar owner - that would be bribery.
That's not the usual definition of "bribery" (though generally it is true of all bribery, including the type you are trying to distinguish, since taking extra personal pay for service is usually formally prohibited, even in places where it isn't effectively enforced, so even what you try to distinguish as "greasing the wheels" as distinct from "bribery" is asking the official to do something that they must not do, and acting against the employer.)
The real problem is the economic situation and people that focus on personal enrichment by any means instead of building something sustainable. Basically, in an environment where you cannot be sure of what next year is going to look like, it's brutally rational to try and extract as much value now, as opposed to invest for the future.
As the result, you get everything from poor infrastructure investment and lack of true small business growth, all the way to bribing because the official taking the bribe isn't sure he or she will have this access tomorrow, so better to enrich now and as fast as possible.
This is not something that is endemic to Russian culture, but rather to the survival situation. Unfortunately, Russia seems to teeter in this state for centuries at a time.
Nah, perish the though!
Now Libya is a failed state thanks to US and HRW lies.
You completely invalidate any point you had when you resort to ridiculous hyperbole.
http://abc7.com/news/video-shows-unarmed-mentally-ill-man-sh...
https://www.youtube.com/watch?v=W-WjXLZbJnU
https://www.youtube.com/watch?v=KOwRMq_Fg5w
http://www.dailymail.co.uk/news/article-2515428/Police-shoot...
> seen by their willingness to bet and lose large amounts of money to take down sports bookies
https://www.washingtonpost.com/news/local/wp/2015/01/27/fair...
http://www.washingtonpost.com/wp-dyn/content/article/2006/01...
http://www.cardplayer.com/poker-news/18352-poker-players-tak...
https://www.pokertube.com/poker-news/poker-gossip-opinion/sw...
http://www.salon.com/2013/07/07/%E2%80%9Cwhy_did_you_shoot_m...
I can't find the highly graphic one about the one who got shot, like, 18 times, while was unarmed in front of a shop. Anyway poker shooting is a reality: It's illegal gambling, therefore it's considered high-level criminality.
Not to judge who's right or wrong, who's "supposed to know" and who's at risk - I'm only saying the risk of being shot dead by the USA police is immensely higher than in any other country in the world, including Russia and China.
Nervousness with regard to both police and local citizenry.
Vladimir Putin is not our friend.
If you live in US but use yandex, Russian gov spies on you but you live in USA and therefore cannot be arrested as easily.
http://www.stratcomcoe.org/internet-trolling-hybrid-warfare-...
For PostgreSQL we use commodity servers and right now they are not really efficiently balanced (i.e. on some hosts we don't have enough disk space but other resources are utilized ~30%). So we are working on improving that. But I can't tell you exact numbers of hosts for Oracle or PostgreSQL (which is actually ~x3 from Oracle hosts number :)).
So for the moment I'm using zoho.com.
I tried Zoho last year after hearing multiple recommendations and it was nothing but problems with spam coming in and my outbound emails being incorrectly marked as spam. I even had SPF + DKIM tested and working from the start! Not sure I would try them again after that experience.
Doesn't seem to support addresses from other domains though.
So as much as I want to talk about Oracle vs. Postgres, these slides aren't giving me anything technical to debate.
I'm a postgres fan and booster myself but I'd need a good reason to migrate a data backend on any project I'm managing.
Judging by the salaries from job sites and my yandex interview, the cost would most certainly top up at $0.5 million (that's a very generous guess, probably half or two thirds that, realistically). Not sure how much Oracle costs at Yandex scale (a LOT more?).
Still though: your point is valid in that, Oracle RAC cluster licensing and support is serious bucks.
Couple of years ago ruble's exchange rates dropped significantly (it was ~33RUR for USD, now it's 64-65 rubles).
Monthly salaries for a good engineer in Moscow are usually 100-200k rubles, or 1.2-2.4M rubles/year.
It's only $18-36k per year. Looks insane from typical californian HR's point of view, isn't it?
Taxes take additional ~50% (in Russia, employer pays all the taxes and payments to the pension fund), so $500k was a pretty good estimate.
http://www.oracle.com/us/corporate/contracts/processor-core-...
So if you look at just enterprise licensing, it's $47k per "processor" - which in their world, a single intel core counts as 0.5 processors. So if you've got a 36 core machine (which I think they mentioned in the video) that's 18 "processors". Or $846,000 for one box. You want clustering (RAC) - that's another $315,000. Partitioning? $207,000. So now we're up to ~$1.4 million for a single server. YAY!
Obviously these are list prices, so you should be paying less than that, but it gets ugly FAST.
It took much more time to rewrite backends logic because of lots of legacy code in many applications without using abstraction libraries.
First type is still in the database. Some part of type 2 was moved to our abstraction library (that is used by all applications). Logic of thrird type was moved to our applications and it greatly reduced stored code size.
2. The other concern was Oracle's slow bugfixing and reaction to feedback.
So, this is a good case showing how a company can stop wasting millions of $ for proprietary software, joining Open Source community, contributing to it and taking benefits from tech collaboration with core developers of the product.
- PL/SQL was harder to deploy than PL/pgSQL (as stated later)
- Only synchronous interface in Oracle's OCCI
- Problems with Oracle's development environments
- Not very responsive support from Oracle.
But the dev environment and support for Postgres are not talked about.
Also I can't quite figure out if "3x more hardware" is supposed to be a good thing on their last slide. It's possible that it's a good thing (if it's, say, we can replicate to more servers), but to me it doesn't read like one.
Initial architecture with 2 servers was set up that way to save money on Oracle per-core licensing. They wanted to add more, but that was too costly.
I'm guessing it was a few million dollars worth of wow-factor.
Not having to deal with stuff like this while having the same features/performance/stability/etc may qualify as a 'wow':
That's actually huge for companies that have invested in Oracle and feel trapped.
And the companies that buy Oracle, have a slight tendency to write all their core business logic inside the database.
Their mobile app (Android) is quite pleasant and lightweight.
What might be a problem for non-Russian speaking people is the lack of documentation in English. Quite often a random link redirects to a page in Russian.
We're looking to make the switch from SQL Server when 9.6 comes out (and proves to be more scalable).
On the other hand, PostgreSQL can take advantage of ZFS compression which is great for archiving cold data.
For analytical workloads, SQL is often a poor choice.
Full disclosure, I work for an analytics and BI Microsoft partner. There are plenty of great alternative semantic layers beyond SSAS, but I am not as comfortable with each of those.
Tells you something about how insanely costly Oracle's licenses are. Whatever you say about Oracle though - Oracle DB is an incredibly well performing and reliable piece of software. (Attested by experience - we had several Oracle DBs running on HP-UX Itanium that only needed to be handled when OS and Oracle patches were needed. Massively used too - think 36 PA-RISC cores at peak, 24 IA64 ones. It also helps that there are lot of DBAs with great deal of Oracle experience.)
While I never tried yandex's email, I do share the impression that gmail is too magical and shouldn't be labelling my emails outside of the explicitly rules that i have set.
> To be fair Russia along with the US, France & Japan is one of my favorite countries, so I might be very biased.
Haha same here! It makes me so sad the propaganda war that russia and the us are having. Those two countries are similar in so many ways. Russia was unfortunate that communism had a successful revolution, so they are currently some 30 years behind culturally :-/
Like invading and annexing their neighbours?
https://en.wikipedia.org/wiki/List_of_invasions
CTRL-F united states, 38 FUCKING results.
That "unfortunate" revolution took Russia from the feudal age into the nuclear age in a single generation, so without it they'd presumably be about 100 years behind the USA.
This may explain why your negative opinion of the USSR/communism is not necessarily shared by Russians.
The most popular leaders of the 20th century, as voted by Russians, are Brezhnev, Lenin and Stalin, while the most unpopular are Gorbachev and Yeltsin, responsible for the collapse and break up of the USSR.
Only a rigged and unfair election prevented the Communist Party leader being elected president by the people of Russia in 1996, after they had experienced the joys of several years of capitalism.
In 2016, a quarter of a century after the breakup of the USSR (which happened against the express wishes of the Soviet people), the Communist Party is still the second most popular party in Russia, and most Russians (as demonstrated in repeated polls) would like to see the restoration of the USSR.
As for Yandex, the founder started in business in the USSR, and he is not Russian.
Brezhnev is more responsible for the collapse of the USSR than either Gorbachev or Yeltsin (or anyone else, except maybe Yuri Andropov.) Oh, sure, he was dead and the other two were around when it happened, but it was his intervention in Afghanistan that essentially sealed the fate of the USSR.
https://pdd.yandex.ru/domains_add/
This is in Russian. Maybe there is English version
Just replace .ru with .com and everything else is the same, except the language.
I just registered and yandex mail really seems like a fine service.
Not related with the migration itself but I tried to use yandex and its mailing service a couple of years ago when attempting to become google independent but it didn´t work well with spanish queries so I eventually went back to google. But I recall that yandex mail was really nice.
That Yandex chiefly targets the Russian, other Cyrillic, and Turkish markets limits its exposure to people who aren't in those markets. And some of it has to do with the kitchen-sink nature of what Yandex does (a lot of tech and some media), in that it's easy to mistake them to be similar to other such companies (Yahoo, Verizon, AOL -- incidentally they're now all the same) that have a similar spread of services, and yet they're not generally complimented in their tech achievements.
[1] https://en.wiktionary.org/wiki/everything_but_the_kitchen_si...
[2] http://dictionary.cambridge.org/us/dictionary/english/everyt...
Yeah ok but you could say the same thing about google and they're doing well. You have yandex mail, web analytics, ads etc. How is that different from what google is going? In fact, I would say that yandex is trying to google's roadmap on how to conquer the world and see if they can pull it off in russia with the advantage that they currently have there.
- There are a lot of Googlers on HN, not so many Yandex-ers
- Google, in the general public's minds, is buoyed by positive mindshare that was generated a very, very long time ago (early-to-mid-2000s). Today's Google has appreciably changed from those days, and yet their likeability has mostly been unaffected.
- Google tried to make their lack of focus official by spinning off a corporate umbrella parent by the name 'Alphabet', like Philip Morris Companies the "tobacco" company renamed itself to 'Altria' in 2003 because at the time it owned 84% of Kraft Foods Inc., a large food and beverage company [1].
I maintain that to an American, Yandex is most analogous to Yahoo or AOL rather than Google, because Google is, as of writing, a company that makes money primarily from ads and is under transition to pursue higher revenue streams from business-to-business sources (like Google Cloud Services) more so than by brokering advertising. Whereas Yahoo or AOL are tech-media hybrids. I know Yahoo and AOL don't evoke such positive brand power anymore than, say, Google, but their structure I believe is more reflective of Yandex.
[1] https://en.wikipedia.org/wiki/Kraft_Foods_Inc#Philip_Morris_...
Like for Google, the main revenue stream for Yandex is also ads – they have >50% in market share in Russia, context ads network and RTB. With good Partner Network accepting websites. From recent products, Yandex.Taxi is a huge success, also taking leadership on Russian market, beating Uber and Gett. So with strong leadership in search, ads and young taxi markets in Russia it doesn't even close to Yahoo in comparison, it's definitely "Russian Google".
A useful series of tweets by @SwiftOnSecurity about why storing data in the US isn't a bad idea:
https://twitter.com/SwiftOnSecurity/status/77549120598337126...
I think you've been so thoroughly brain-washed by russian-hating american media that you can't even think straight any more. I realise that this isn't really an argument, but I already presented my argument (that with the usa gov you have facts, whereas the russian gov you have suspicions) and it didn't seem to make a difference, so i don't really care to continue this conversation.
It ought to be obvious to anyone that applying black and white judgements in these situations is idiotic. Do I think the US government is beyond reproach? No. Do I think the US government is more open and accountable than the Russian government? Yes.
> (that with the usa gov you have facts, whereas the russian gov you have suspicions)
Yes, because the US government is open and accountable enough that this information is public. The same cannot be said for the Russian government.
At the end of the day, the people who have the guns and the authority to use them against you take a more prominent place in one's threat model than a foreign entity with no jurisdiction.
Yes. In fact, I'd say that's the case even if your only concern is with the US government.
Virtually all of the controversy over US government actions with regard to local data is about violation of expectations (some of which are codified into law) about domestic surveillance by doing things which are uncontroversial in the context of foreign surveillance, and its not like the US lacks capacity or will for foreign surveillance.
That said, Russia doesn't even have the constraints (legal or mere expectations) on domestic surveillance that the US domestic actions raise concern for violating, so if your concern extends beyond US government surveillance, then that compounds the problem when hosting in Russia.
(Everything just works and there are no insane bribes to pay to various mafias.)
http://www.nytimes.com/2009/09/12/business/global/12ikea.htm...
Edit: there are many ways to hide bribes, of course. Typically it involves overpaying for something. It would be hard for even a non-corrupt tax authority to tell the difference between having bad negotiation skills and bribes.
Let's say someone claims that their programmer genius Ivan saved a company ~1m USD/year with latest commit; and awards him $100k. And you see this unusual bonus, how are you going to know 99% of that doesn't go to someone other than Ivan?
The direct bribes is really for low-level clerks like police officers or fire hazard inspector s.
I was possibly confused by this article just before posting that comment:
http://calvertjournal.com/news/show/3035/founder-of-yandex-r...
(It's hit #4 in Google for his name.)
It's Pavel Durov. https://twitter.com/durov
[1] https://www.bloomberg.com/view/articles/2014-04-25/how-putin...
There are a couple of downsides, but really - there's a lot more upsides to Postgres than downsides (IMO).
So the biggest in my experience is not enterprise support. Folks generally expect it to be. The biggest is actually flexibility. Oracle DB is good at handling some edge-case scenarios, such as a RAC cluster with a failover cluster running an Advanced Dataguard Reporting instance in a whole different state.
Basically you are running a (super expensive) near real-time cluster that's doing compressed log-shipping to another cluster, and you have a standby on that secondary cluster that's open in a read-only mode. And none of it requires cron jobs or excessively complicated setup.
However... even though I've done stuff like this for a number of clients, there are SO MANY better ways to address these sorts of scenarios with creativity and elbow-grease that Postgres tends to be a better tool.
The caveat, of course, is when the person who sets it up leaves, and some other person has to take over. With Oracle's scenario, it's mostly just a slightly unusual config of a standard set of tools. The more creative Postgres solution might be quite a bit more of a learning curve.
This is just an example off the top of my head, but it is one that came up recently. A client had a shipping/receiving system that could not go down, so the 20k+ per node was trivial. To reduce the load for reporting, they just simply had us open a standby DB for read-only reporting on the failover cluster - and they liked that so much, we set up another one for them local to the offshore reporting team.
It's absolutely possible to do something like that with Postgres clustering, but there's a whole lot fewer folks doing it.
So #1 benefit of Postgres for Oracle shops is transition cost. It's cheaper than Oracle (by sometimes substantial amounts), and relatively painless to switch to. A lot of code can move unmodified, all the same platforms, and often even the same teams of folks in the same jobs. Postgres scales very well, at the very least it can handle 90% of a given company's Oracle load with no real issue. Things like clustering, backups, failovers, and reporting systems can be re-engineered to work in similar ways on Postgres, though some will require substantial hackery.
The #2 benefit is stability. Oracle has a very bad track record (especially lately) when it comes to patching and supporting older databases. If you just want a DB to sit in the corner and work for a year, with push-button easy security patches at most... Oracle isn't really the product anymore. Oracle CPU (critical patches) are cumulative now, and the bloat is getting out of hand. I recently just installed a set of patches that were almost an extra 5GB for the footprint... minor for a 20TB database, but ridiculous if you're thinking of thousands of small distributed hosts.
For example - on a QA system we were running a couple of custom one-off patches delivered by Oracle Support. These blew up the entire patch process. I had to stop, back them out, re-patch, then re-apply the one-off patches.
Oh, and 12c is a seriously whiny bitch about XDB. Just for added fun.