Evernote is moving all its data, machine learning to Google Cloud
techcrunch.com
techcrunch.com
Good riddance! Deploying Openstack Swift was a nightmare in my personal experience. A few outages and I decided I'll never work a job supporting it again.
If a recruiter mentions "Openstack" to me what I hear is a shamble mess of a services with components you'll be on-call for which will break in unique ways. Oh and of course you're expected to maintain the entire infrastructure while you do your normal job.
Just move to the cloud already.
It's better to look at each situation, and decide whether the cloud is a good idea or not. Let's stop pretending this is the only good solution.
And you can run a dedicated, private client if you want for most services.
A listenable connection to your client is entirely different than handing your entire infrastructure and data to a third-party. I fail to see the merit of putting your whole business in the hands of a potential competitor.
Anyway, infrastructure in cases like these is really like a public utility. No one says companies are putting their money maker in other's hands because they use public grid electricity (or sewage, or water supply, etc.).
Evernote's money-maker is the sum of those parts.
> No one says companies are putting their money maker in other's hands because they use public grid electricity (or sewage, or water supply, etc.).
Yeah, because that's generally not the product. Those are typically by-products. They happen after the product is produced.
I narrowly avoided a long term (career-derailing) OpenStack assignment...four days in...oh, this is all based on OpenStack?...transfer approved
I myself am a PTL (project technical lead) in one of the OpenStack projects and although I do understand where you are coming from I wonder what the gist of the experience you had was...
Care to share what experiences you have that made you think 'total dumpster fire'?
I know the software isn't perfect, but then what software is? Also I've seen that the OpenStack 'having a bad time experience' is as much of a cultural change as it is a software change; ie, OSS in general isn't a take and not contribute kind of thing, that's not how it really works/gets better.
There was no CVE mailing list like other projects. We had to scrape their launchpad bug tracker. On top of that none of the package repos we checked (Canonical's, Debian's .. couple of others) would even have package updates for up to three weeks on some of the CVEs. We started building out own CI so we could build and patch ourselves. Then I got pulled from the open stack team onto something that was not a bottomless pit of haemorrhaging money.
There are a lot of on-premise workloads which cannot be moved to someone else's remote cloud (for varying reasons). This is where OpenStack fills in the gap and provides something better than an ad-hoc brew-it-yourself cloud platform. That being said it still requires a good understanding of the requirements as well as a well chosen distribution to match the foreseen use-cases.
I'm curious as to what really made you hate it. From my vantage point of running openstack in production from Essex to Juno, Openstack Swift after Folsom release was the most reliable part of openstack outside of keystone and glance.
(Nevermind that it was also the easiest to upgrade as well).
Or you can use the ceph stuff, which is also pretty nice.
I am Genuinely curious, I've not used Evernote for several years, it just got out of hand and bloated I found.
Machine learning is really starting to feel just like another trigger phrase.
https://blog.evernote.com/blog/2015/01/23/search-handwriting...
https://gist.github.com/dannguyen/50dcb5a8f4230e7a8a40bfe2d0...
Yes, and its close cousin "data science" fits the same description, to me. They're quickly becoming what "the cloud" is: marketing jargon that doesn't provide useful information.
The point of treating data science as a field is to distinguish the people doing it from "regular programmers." This creates a distinction of caste: the "regular programmers" are relegated to maintenance work, never trusted to touch any of the mathematical and CS-degree-requiring parts of their own discipline. The "data scientists" are then hired from outside, to do precisely the parts of programming that every programmer at the company would have already been proved competent at, through degree-qualification and technical interviews that seek out exactly these talents. And the "data scientists" are paid better for it, and given better working conditions, and less micromanagement.
Same with machine learning: it's just programming—certainly, a kind of programming that is heavy on CS, but certainly one you can pick up quickly enough to get some good practical results. But you hire ML people separately from "programmers", and you never hand your "programmers" ML tasks.
From yesterday's subthread on the wind-down of Starfighter, I gather that a large part of its goal was to reveal that the emperor has no clothes here: that nearly any programmer is capable of doing "data science" or "machine learning" (and, as well, "reverse-engineering", "low-level programming", and a host of other things) if actually given a chance to solve the problems.
And the flaw in the logic is revealed here, I think: > nearly any programmer is capable of doing "data science" or "machine learning"
Nearly any programmer might very well be capable of learning these disciplines, but they're different disciplines from business logic or database theory or networking; they're non-discrete and statistics-interpretation heavy, and it's entirely plausible that you can have an entire company full of programmers who are neither excited about nor prepared to take on these categories of challenge. Some programmers will (lots of people are polymaths), but it's not nonsense to consider them different categories of discipline.
I think smart companies provide an internal channel to get into these sub-disciplines, but it's not necessarily a problem space you can throw at any developer and say "Here you go; good luck tuning those training and test sets!"
I recently worked at a company who employed "data scientists", they rarely worked in isolation as far as I could tell, and usually required heavy assistance form system/database/storage administrators to be successful and aided by (but not always) with software engineers.
In the case of ML, I still see it as just software engineering / computer science. Sure there are pioneers in the field of deep learning, but aren't most of us just leveraging APIs or existing algorithms?
Honestly, I'm pretty confused myself lately and I work with some of this tech. Open to enlightenment on this topic by the way.
The fact that data science is basically statistics supports the argument of the guy who is saying data science and software engineering are different jobs.
If you are a software engineer and want to be a data scientist, you are free to switch careers. Or maybe work as an engineer for a startup that could use someone to do statistics on the side. But that might be a bad move with no experts around. With data science, your work product is often conclusions and advice. Those are highly susceptible to bullshit. You don't get a compiler or unit tests that tell you that you are coming to the wrong conclusion and giving bad advice. It takes experience and judgement to get that right, not just hacking on an IPython notebook and accepting what the model spits out.
What you describe with respect to data scientists needing help is what I'd call "data engineering." A data science person who is supposed to be doing production-facing engineering should be a competent enough data engineer that he or she can extract information from the data stores without having to have outside help most of the time, but strictly speaking, in my view, these are two different kinds of development.
There are a few Bigcorps with real, novel Big Data datasets that need original statistical work slung at them. These companies need data scientists, for real: people with Ph.Ds in statistics. These people are indeed not interchangeable with programmers. They're partially interchangeable with academic computer scientists, and mathematicians, and physicists; but not with any trades-workers, including your average programmer. Their job is Science, capital-S: to do sound, replicable, high-powered experiments to learn things about the data. That is a skill in-and-of-itself, and one you need a lot of practice with—under the scrutiny of people like Journal editors—to get right (because it's hard to tell from your own observation if you've got it wrong.)
Everyone else, though—that is, all the companies but the SV Big Five and maybe some mobile-game and advertising firms—when they say they're looking for "data scientists", all they're really looking for is data engineers. The actual "data science" they're trying to do is a solved problem: you can find out the same thing a data scientist would tell you from five minutes of Googling the problem, because dozens of companies have solved the same problem for their own data-sets already and written case-studies on how they did it. You only need to implement said solution. Which, like I said, any competent programmer could do, after a few hours spent reading the API reference of a statistics package.
The majority of "data science" (read: data engineering) jobs aren't looking for people to Do Science; they're looking for regular programmers clever enough to have self-selected into the higher caste by doing nothing more than having learned some statistics-package APIs and then marketing that skill. (Much like the "officer class" of the American military in the 1800s allowed people to self-select into it just by being voracious readers.) It's basically an implicit open question at the beginning of the hiring process, "what do you want to be doing all day?", that is answered by choosing a different (pretend) job title.
Perhaps it's because people with formal training and degrees often have better skills than some random kid who taught himself javascript and thinks that that makes him a software engineer?
And then, having found all this great talent, they let it rot, under-employed in maintenance work. Because they were hired to "do programming", not to "do data science" or "do machine-learning" or any of the other 'specialized disciplines' that they could totally pick up given about two hours, given how freakishly intelligent they already have been proven to be.
(If you're wondering, this isn't bitterness; I've never worked for these companies myself. I'm just summarizing a number of rants I've heard from others.)
I suppressed this because I found it a distraction, but if their searching and matching gets better I might un-disable it and give it another try.
Evernote’s new CTO Anirban Kundu told me the first two areas that will be replaced by Google’s machine learning APIs are its voice recognition for speech-to-text translations; and natural language processing, used to help search for contextual content.
> Evernote’s new CTO Anirban Kundu told me the first two areas that will be replaced by Google’s machine learning APIs are its voice recognition for speech-to-text translations; and natural language processing, used to help search for contextual content.
There's some kind of cluster of OCR expertise in the Russian computer science community given ABBYY Finereader as best-in-class printed OCR and Evernote as best handwriting OCR.
{Search for (notes tagged household or notes tagged personal) and (tagged 2016)}
Or even something like: not notebook:"Personal Finance"
I love evernote, but it often feels like the team writing the search engine quit before the job was quite done.
but not -notebook:receipt
I hardly ever use it anymore, using mostly notebooks for actual note taking now. I used to use the web clipper to save articles, but I found that when I would put something in Evernote, I hardly ever went back to access it later. It was like putting things in a vault. That probably all speaks more to me than the product. I am curious what new features they can provide with access to all of Google's machine learning APIs.
The awful search with the web clipper plugin. It seems to match any single word in a saved article. It just ends up being faster to ignore the evernote typically unrelated results and look through Google results.
Inability to embed YouTube videos. Once again with recipes I'm better off using a browser since I'll have to open one up to view the video anyway. Most of the good cooking sites include a video embedded with the recipe, making for a more pleasant experience in the browser.
I really don't understand what happened to Evernote. It was very good and getting better all the time; then a few years back they decided to pivot to...something else, I'm still not sure what...and essentially abandoned all the features I used to start pushing lots of features I never would.
Are there any downsides moving from a free Evernote account to a free OneNote account in your opinion?
- OneNote doesn't have tags
- search function is pretty primitive compared to Evernote
- The Webclipper is frustrating: Websites are often clipped as an image. Very frustrating if you actually need to work with the content afterwards.
- ON seems to generate much more traffic on my phone.
- publishing notes with Evernote is much better: You get a public link, etc. Everything looks more polished. Also, when publishing clipped websites you automatically get the link to the source. In ON it just looks bad. (I use Evernote for press articles.)
- Spellchecking is a mess in ON. For some bizarre reason they've decided to set the notes language to your keyboard locale. It is ugly as hell. But there's a macro for OneTastic to reset the locale.
- no Context.
Good things in ON:
- free form canvas
- no more messing your notes up when you work on them cross platform.
- Excel-tables natively supported
- as somebody noted, Evernote is practically a vault where you never touch stuff again. you basically hide everything behind tags. Then you use the search to get only relevant stuff. That's partly due to the UI. In OneNote you organise your notes more "optically" and actually reread stuff.
I introduced ON at work and we now couldn't do without it. I've been using Evernote in privat for several years. Now I'm using both - EN for press and my literature library, ON for everything else.
Also, Onenote seems to handle sync conflicts better than Evernote. I once emptied my trash in Evernote and it decided to delete the live versions of all the old sync-conflict duplicates I'd trashed. I lost about a third of my notes. That's when I realized I needed to change.
One other downside is that the OS X version of Onenote is not quite full-featured. It's not missing anything essential, but e.g. you can only have one window at a time, for some reason.
Basically, my use case is personal notes and data, mainly formatted text rather than images or websites, for my own use, organized hierarchically for easy browsing. Onenote is ideal for that. If you're big on group sharing, or clipping web pages, or tag search, or whatever the hell else Evernote thinks they're optimizing for, then I can't tell you much.
There's no mention in TFA of the data belonging in any way to the users. And maybe that's accurate.
Yes, we have web/cloud apps now. But that is—or should be—orthogonal to where data is stored and hosted. Of course app vendors have every incentive to create lock-in. And most users are just happy to hear that "you do not need to take any action" and your data will just be magically faster and more secure. But at least around here, wouldn't people rather own their storage? Does remoteStorage have any legs?[0]
This kind of happened with authentication, right? It's common now to use OAuth to provide login using a third party. Of course, they're mostly silos, too, but at least in principle an app could let you choose a provider.
[0] https://tools.ietf.org/html/draft-dejong-remotestorage-07
[1] https://blog.evernote.com/blog/2016/09/13/evernotes-future-c...
https://evernote.com/legal/tos.php
But choosing where to store the data is sort of obviously inside the umbrella of providing the service.
This is just a round about way of saying "I sure am surprised people think they 'own' their data if it's stored and managed by a commercial entity as a business and not spelled out as a primary feature of the service".
That person is neglecting the fact that it's not in the business' interest to "acquire all rights". The main reason being that _some_ layer would read it and _everyone_ would read it the next in the New York Times.
A secondary reason is that it'd be illegal and/or unenforceable in many jurisdictions.
After many years in this industry I've personally come to the conclusion that nobody actually cares about security, privacy, or data ownership. People say they do but they are lying, as evidenced by the fact that they do not change their behavior. People gripe about security and privacy but continue to use "free" surveillanceware/spyware and continue to migrate everything to cloud hosts they don't control.
The only thing people care about in reality is convenience and user experience, and they're obviously willing to give up an almost unlimited amount of personal privacy and control to get those things.
Will I be hurt today if I put my data on someone else's cloud? probably not. Will I 10 years from now? possibly, but by then, my brain will have disassociated those two.
Like that E2E vs spam filtering issue explaination from GMail engineers.
The real question is, how do we fix that?
At the moment, I use GMail, IRC, a Matrix server I set up for my hackspace, a couple of niche forums, and a niche social network. I'm in the process of organising some hackspace servers to provide email hosting, so hopefully I'll be able to switch off of GMail at some point. The next step after that would be some form of cloud file hosting.
I'm not overly privacy-conscious though - I've used Tor once to see what the fuss is about, and I don't have AdBlock or NoScript installed. But at least I avoid explicitly putting my data in the hands of companies I have no control over as much as possible.
--
What we need is separation of virtual machines from the code they run. Exchangable backends, if you like. We should be able to use cloud tools like e.g. Google Docs while also being able to choose our own backend - whether this or that cloud, or even our own servers. Right now, instead of running someone else's code on our data (as it is with files and local work), we're giving third parties our data for them to run the code for us.
I wish too, but this is a moot point - everyone has a shit ton of things to care about in their adult lives. By the time we're done putting out fires at work, dealing with mortgage, bills, grocery shopping, sick parents, broken sinks, searching for a good school for our kids, etc. there's precious little care left to be given. Greedy and socially destructive cloud/SaaS business practices are a distant position on a long list of things to care about, so you can't say that "if people don't care about something then it means they're ok with that". It only means the issue isn't immediately pressing/dangerous enough to warrant prioritizing it.
It's not about individual happiness.
To give an extreme example, there are people who don't care about police violence. While they are might fine, and probably wont ever have to suffer it, the resulting society is worse than if they cared.
Ok, but people obviously don't think the issue is that big since the majority doesn't care.
Whether the majority (or a minority) merely likes something is not a justification for it.
As to whether its not harmful, just, etc, that requires argumentation and further inquiry.
I know I'd rather not own any storage at home. I have a 9TB storage array (well, I'm only using about half) that I use for pictures, music and movies, and if I had a faster and more reliable internet connection, I'd happily move that all to the cloud.
I'd want my own encryption key though, so the provider can't see any of the data.
All of that data already lives at a cloud provider as backup -- took a couple months to upload it all.
The thing is... I don't want to basically trap myself into AWS or Dropbox. I run a 6Tb array like you, and kinda set up my own cloud, drag and drop, etc.
I don't think I'd ever give it up because:
1) It's cheaper to run myself (long term) - I'd have to consistently pay someone else ever increasing amounts to store data (which seems dumb since storage gets cheaper every year, their service doesn't).
2) Security - I don't want anyone to have access to those files but me. I'm 100% sure Amazon or Google would be able to access those files if ordered too by a government. Hell, they might do it without being ordered, who knows what's happened since Snowden.
3) Ease of use - my setup is based on a local network (as well as an ssh component). Basically, my setup works like Dropbox. The only drawback is no mobile component. If I want to change that I have access to all the parts and can do what I want. With AWS (which I use all day for work) I cant swap out a drive or stream a movie to my TV, phone, or desktop with a click.
Now I will say, if I could trust a cloud service that wouldn't charge a ton of money for 60+tb then I would jump on it.
Its actually been that price for about 2.5 years now. With competition from Google, Microsoft, and Backblaze, its possible we could see a push toward even cheaper prices before too long.
As for security, S3 does support SSE at rest with a user-provided encryption key. So you're good to go there.
On cost, I'm using Nearline myself ($.01/GB/month so about $10/TB/month) as an offsite backup of my home NAS setup. Paying for actual bytes stored instead of bytes of capacity gets you at least 20% (raid overhead) but honestly closer to 50% or more depending on how full you are. The poster above had about 3 TiB apparently which works out to $30/month. That's pretty good for an unthrottled replicated storage system offsite.
(And like you, I'm considering making my media directly read-only available via App Engine or something, because once you're talking about 1+ TiB of photos all photo hosting sites fall over).
Disclosure: I work on Google Cloud.
Its my hunch that the statement you quote is just simply wrong. If you're keeping a computer at home that is doing something valuable enough to keep it around, it should valuable enough to you that there has to be _some_ price cheap enough, and _some_ capabilities good enough, that the cloud makes sense. The exception to this is, of course, the hobbyists and the tinfoil hatters.
I don't care about my internet connection but it was looking like hundreds of dollars a month or more to store that sort of data.
(I only post in the hope that I'm missing a magic, affordable bullet.)
- Companies do not (or should not) make cloud purchasing decisions be based entirely on some kind of preferential pricing. Cloud is far from a commodity, especially in ML/Big Data space. In both Evernote and Spotify cases, they highlight strategic technological advantages of Google Cloud. In fact, customers tend to be fairly specific in outlining their findings[0], cost rarely being the main factor.
- I urge you to share evidence that Spotify received "huge credits" in exchange for PR.
- It's not unusual for companies to make such announcements. It's very much standard practice for AWS customers. It's good PR for both parties, imho.
- There are many other "big" names disclosed in 2016. And it's no secret that what Snapchat has achieved on Google's PaaS is nothing short of extraordinary.
[0] https://medium.com/@thetinot/i-think-google-cloud-is-the-bes...
(disc: Work on Google Cloud and put together [0] )
I am surprised to see Evernote moving to GCP to be honest. The cost increase in using GCP vs self-hosted hardly ever make sense for storage providers (which is kind of what Evernote boils down to). And it's not like Evernote is doing well financially.
It's why Dropbox went the opposite direction.
On your second point, per [0], Evernote is interested in Google's ML capabilities.
[0] fortune.com/2016/09/13/evernote-google-cloud/
> Some critics, surprised by such a high-profile company going with what is seen as the distant Number 3 in the hyper-scale public cloud market, have suggested that Google extended a sweetheart deal as part of the move. Details of the agreement haven't been disclosed, but Nicholas Harteau, Spotify's vice president of engineering and infrastructure, told The Wall Street Journal that the company "negotiated hard on price."[1]
> Of course, for such a high profile customer and for the priviledge of a case study, we can assume Spotify has some heavy discounts. [2]
Not "proof," but it's pretty well-mentioned across the industry that Google benefited from this deal probably more than Spotify did.
[1] http://searchcloudcomputing.techtarget.com/news/4500276342/S...
[2] https://medium.com/@davidmytton/how-much-is-spotify-paying-g...
Unfortunately [2] is 100% opinion.
As far as [1], every single "large" deal is negotiated. That is standard practice for deals of this size, including your employer and AWS. It's also very different from "Google is the one benefitted" and "Spotify chose Google because of sweetheart pricing" and "Spotify got paid to do PR for Google". As I've said before, these narratives diminish both parties. It's Trump's "many people say that..."
Step 2: Migrate to Google platform.
Step 3: Exit via Google acquisition.
Maybe Evernote will become a integrated Google+ service.
I don't begrudge Evernote their business model, but I'm much more comfortable with upfront software licensing than subscriptions. That's especially true when it means I no longer have to trust anyone else's terms of service when it comes to keeping my private stuff private.
(Evernote employee here)
Level 2: Machine Learning, however, has been well-defined in the past (starting in the 50s), is now widely used, contains a whole set of (old but also) new innovative techniques and actually contributes massive value to many domains.
Level 3: VR is far from dead.
EDIT: a word EDIT_2: I would have agreed if you had said 'trend'. Yes, it is a trend.
And there goes my data, my privacy, ...
That said I really liked evernote. Decent alternatives, cross-platform, mobile app, good usability, ...?
They have been improving lately though.
Are companies that host on EC2 or Azure giving the customer data or giving up on privacy?
But if I read about that deep-learning magic they plan to do, I'm seriously concerned. Especially since I find the already existing note context feature quite invasive. Companies should only be allowed to do with my data what I give consent to.
I'm not sure why this would be the case. They're just using google cloud services (computing, storage, and apparently tensor flow). You use tons of other services right now which are built on the same back end. It would be no different for you and me (end users I mean) if they had chosen Amazon.
I'm careful what services I use, sometimes I prefer local p2p such as bittorrent sync over uncontrollable clouds for personal data.
I'm mostly concerned about what they will use the deep-learning thing for though. Not sure I'll want it to work with my notes.
OneNote (http://www.onenote.com/), Notebook from Zoho (http://zoho.com/notebook), Apple Notes (https://www.icloud.com/#notes), Keep from Google (https://www.google.com/keep/), Simplenote (https://simplenote.com/)
TFA:
> Will Google be able to index my notes or use it for targeted advertising?
> No. As a cloud provider, Google is subject to strict security and legal obligations which limit Google’s access to Evernote data. The data you put into Evernote belongs to you. Google will not process data for any purpose other than to fulfill our contractual obligations.
Brilliant. Truly.
The closest thing to collaboration is the ability to access google drive documents from within evernote.
https://cloud.google.com/security/encryption-at-rest/
(work at GCP)