obulpathi

Parquet is the Open Source implementation of Big Query's first generation columnar storage format. It is based on record shredding and assembly techniques described in Dremel paper (Google's internal name for Big Query). What they are talking here is the next generation (second generation) storage format.

rajeevk, I can not reply to your comment, so posting here. Just take a look at my comments on hn. I compared Aws and Google cloud in terms of ease of use, pricing, open source friendliness, scalability, security, performance etc .. You can find a detail answer as to why I came to this conclusion.

I did PhD in Cloud and Big Data (Created a Cloud and Big Data platform for crunching large scale images on satellite clusters). I spent about 10 years of my life working on nothing but Cloud and Big Data. And I rarely do comment on anything outside my area of expertise. If you find any of my comments incorrect or interesting, I will be more than happy to engage in a conversation with you. This is also third or fourth time an account has been created to reply to my comments. I find it very interesting!

Google Cloud is by far the best Cloud. Beats Aws and other cloud providers by miles margin. Also keeping in mind that a service that just entered beta is the fastest by itself is impressive. I would say, let's wait a while before commenting on features.

> I don't see the harm in offering the ability to reserve an instance for a discount.

There is no harm with the discount. The harm is only with lock in. Once you reserve instances, you can not change the type (different CPU / memory ratio), region or the number of instances. Once you reserve, you have to pay for your instances 24x7, whether you use it or not. If you require more capacity than reserved, you need to pay the full rate. If you require less capacity, then you need to pay for the unused capacity. You don't have all these headaches with Google Cloud pay per go and automatic discounts. You simply use as many resources as you want and you get the discounts automatically.

Yep, that's true. Capacity planning and reserving capacity should be taken care by the Cloud provider and users should be free to spin up and down the resources as and when they need them.

Google Cloud if feature equivalent of AWS.

If you build inside AWS, you have to pay for cross-region bandwidth like transferring data between the east region and the west region.

If you choose Google Cloud, cross-region bandwidth is also free!

Give GCE a try and I am sure you will be blown away!

I find Google Cloud miles ahead of AWS. AWS does not play well with open source services. Google on other hand embraces OpenSources (Google OpenSourced Kubernetes, Dataflow, TensorFlow, MagLev the load balancer, integration with OpenStack ... ). This reduces your risk of vendor lock-in. I have not seen AWS OpenSoruce a single Cloud service.

Yes, Google Cloud offers IAM. I find it easier to use than AWS IAM. Google Cloud IAM gives you curated roles, hierarchy of resources and thus is easier to use.

Yes, Google Cloud can manage secrets for you. For example, you can spin up an instance and let Google Cloud handle SSH key creation, copying it to instance, auto-rotating it periodically, deleting it when a user is removed from project, connecting to services from Cloud instances is taken care for you.

Google Network and SSDs offer an order of magnitude more performance / price. Disks and Storage are about 2 times fast. Booting is 5 times fast.

Not just the performance, all Google Cloud services are fully managed (Think of S3 and Lambda on AWS). No servers! On AWS, only a couple of services are fully managed. Rest of them are a thin layer on top of EC2. If AWS is DevOps, Google Cloud is NoOps.

GCE is less expensive than AWS even with AWS's long-term contracts. If you pay money upfront (losing the interest) and reserve instances for 3 years, then you can beat GCE prices by a small margin. But, if you take into consideration of interest on upfront capital expense, lost flexibility due to reserved instances, cloud cost cuts, GCE beats AWS even with the 3-year commitment. We go to Cloud because Cloud gives us agility and enables us to create infrastructure in a matter of minutes without any constraints (Imagine singing up a contract with your electricity company agreeing that you will consume the electricity for TV for 3 years on a 24x7 basis. That what reserving capacity looks like). Reserving capacity does not belong to the age of Cloud, it belongs to the age of data centers.

Google Cloud wins hands down!

Places where Google Cloud shines over AWS: Excellent UI, Ease of use, Scalability, Security, Data services, Big Data, Machine Learning, Network, Disks. The only place where Google loses to AWS is Relational Databases.

Google Cloud Network and SSDs are about an order of magnitude better (in terms of performance / price) compared to AWS Network and SSDs. Googles Big Data tools (BigQuery, PubSub, Bigtable, Dataflow) can scale to petabyte scale without any problem. AWS can be OK for terabyte scale data workloads, but beyond that, it's hard to scale and manage. Other pieces like Kubernetes, TensorFlow and Dataflow (Opensource, if you want to run in-house) and available as native Cloud services, all services are fully managed makes Google Cloud the best Cloud.

Nop. Probably did too much copy-pasting :( Mearly wanted to highlight the point.

> Finally, to underscore how seriously we are taking this event, we are offering GCE and VPN service credits to all impacted GCP applications equal to (respectively) 10% and 25% of their monthly charges for GCE and VPN.

These credits exceed what is promised by Google Cloud in their SLA's for Compute Engine and VPN service!

Very informative video on SRE at Google: https://www.youtube.com/watch?v=H4vMcD7zKM0&list=PLIivdWyY5s...

I can clearly Google Cloud winning the Cloud industry. It's only a matter of time and not a matter of if. Cases like this and Spotify, will make the shift happen sooner than rather.

This move will be a "GAME CHANGER" for the Cloud industry.

I see a constant flux of features, but the whole Google Cloud ecosystem is well thought of and coherent as if the ease of use is primary thing. App Engine, after 8 years, is still as easy to use as the day it was released.

Google Cloud wins big on simplicity and ease of use. That can save lot of developer time. Here is the article that I am writing on "Ease of use comparing Google vs AWS": https://docs.google.com/document/d/1B5XUiQUClxdoFpl2Md5MZOv6...

Any feedback is welcome.

Citing from OP on ease of use with Google Cloud:

> Over the past couple of years, I’ve never struggled to grasp or understand any of the GCE offered services. The predefined machine types are also very clear, shared core, standard, high memory and high CPU, I know them all by heart now with their memory configurations and pricing to some extent. I clearly understand the difference between the IOPS for a standard disk and an SSD disk and how the chosen size impacts their performance. There is no information overload, disk pricing and other information is kept separate, it’s simple and easy to understand.

Now compare this with EC2 VMs, it’s overwhelming, current/previous, etc… generation VMs. Disk information and network configurations all lumped together with VM configurations, paragraphs and paragraphs of different price configurations. For me, it was painful just trying to understand which VM type is suitable for my needs. My first encounter with SSD configurations and maximum provisioned IOPS for AWS RDS was one of pain. Instead of spending time working on my project, I found myself spending valuable time trying to select which IaaS offerings best fit my needs. Things like trying to figure out if low to moderate or high network connectively is best for my needs!. No wonder I still hear many say they find Cloud offerings confusing!, I think this is no more with GCP.

Google’s network is so fast, however, that this kind of multi-cloud might just be possible. To illustrate the difference in speeds, we ran a bandwidth benchmark in which we copied a single, 500 Mb file between two regions. It took 242 seconds on AWS at an average speed of 15 Mbit/s, and 15 seconds on GCE with an average speed of 300Mbit/s. GCE came out 20x faster.

Reference: https://gigaom.com/2013/03/15/by-the-numbers-how-google-comp...

1. For micro-instance and 32 core instance, the difference AWS and Google is not a big deal. For rest of the instances, Google Cloud is 4 - 7 times faster. That's not a "synthetic" benchmark.

2. Yes, everyone should automate Ops, but if the Cloud provider takes away some of the pain, its a win.

We don't use GUI to manage our resources. We use CloudFormation for AWS and Deployment Manager for Google. Let me tell you a couple of things about those services. In AWS some resources are zonal, some regional and some are global. It's a mess to work with. For example, same AMI image has different ids in different regions. You need to create maps and stuff to make your code work across regions. Come to Google Cloud, no more zonal/regional/global fuss. An image is a global resource. It's available by the same id in all regions. Your infrastructure template looks much cleaner. Combine the power of Jinja, you can create far powerful templates and evaluate them on the fly. AWS has "three" queuing systems, "two" storage solutions with different API's and different quirks. Google just has one and its nails the use cases for queuing and storage. AWS micro-instance go poof, without any notice. Their NATS are known for being unreliable. Load balancers can't scale. Every service that I looked into, Google is way better than AWS.

We did a performance analysis of Google Cloud vs AWS. The results are in line with what is published in the post. The biggest thing that we can not quantify is "ease of use". Google Cloud is a pleasure to work with. AWS feel so clunky compared to Google Cloud. Don't take my word for it. Create a VM, login into it on AWS and Google Cloud, you will change your opinion about what a good cloud is.

Yep. That's right.Google has built their custom networking. It's like when a machine A sends packet to machine B, the packet is handed over to Google SDN (Software-defined Network), which carries to over their private network and then injects the packet into machine B. Googles network is an order of magnitude faster compared to AWS.

Support is much better on GCP than AWS. Reference: https://news.spotify.com/us/2016/02/23/announcing-spotify-in...

One of the reasons why Spotify went with Google Cloud is because of their superior support.

Highly recommended! I was able to get a pretty good understanding of Deep Learning using this book. The book contains really good analogies (like explaining how a basic neuron works using a real life decision model) to help you understand the complex topic of Deep Learning.

> That being said, Google Cloud support is overly expensive. $150/month Vs. $49/month for AWS Vs. $29/month

Your basis for ordering is their support prices which is just a tiny tiny fraction of Cloud infrastructure pricing. Google Cloud is 50% cheaper compared to AWS. Which is huge. Also, thanks to ease of use (Cloud Shell, SSH from browser ..), Google Cloud will save you lot of man hours.

Also talking about commitment, you need to commit for 1-year / 3-year usage of resources and pay partial or full bill in upfront for discounts. Google beats those prices without any commitment.

Let's compare the Cloud providers on these factors.

I am sure Google Cloud has VPC stuff for all the components, including Cloud SQL. It lacks the variety (No PostgreSQL, Oracle, MSSQL, or MariaDB). All traffic is encrypted by default on all posts (Google Cloud does this for free) too.

App Engine expects your web apps to return almost immediately (in a matter of few ms). For anything that takes longer, you need to use task queues in App Engine. Or maybe try AppEngine Managed VM's?