maslam

Joe's a really nice guy. Have you ever worked with him or been in a meeting with him? Sorry, this type of ad-hominem attack really does not belong here.

"Third party vendor". Is this Qubole? If so, would be nice to give him an Internet high-five by naming them :)

Someone's trying to be GDPR compliant :D

Spelling error in video is a big red flag for me. "Expidited" should be "expedited".

It's a thing in Seattle. One of my friends lives in a wonderful neighborhood in South Bellevue. Multi million dollar homes. At least 40% sitting empty.

Be oh so careful. We ran into issues with alpine node images that would only show up in minimal images like that. Switched to Debian and never looked back.

We use urlbox.io and are really happy with it. Don't recommend building this yourself.

(product person at Appuri in Seattle)

We work with customer data, so lots of time-series. We use SQL for exploration and feature engineering. Model-building happens inside a Docker container that runs on ECS. Scores and predictions are inserted back into the data warehouse (Redshift). It's a simple, practical system that works well without needing a ton of infrastructure.

Yes

This, SOOO much this! Please don't enter the morass that is Hadoop / Hive / Presto / Spark [esp. Spark] unless you really, really need to. Redshift sounds really good for your needs.

George - fan of Fivetran. What, in your opinion, are common cases where customers choose BQ over Redshift? Or is it really a function of which cloud you're on?

We use Amazon Redshift a lot. The biggest pain point is its lack of elasticity and strong coupling between compute and storage. You can see this pattern emerging in warehouses from Azure and Snowflake. I fear that Redshift will never get there because it's based on old Paraccel tech that was never really designed for the cloud to begin with.

Is Redshift worth it? Yes, it's a powerful EDW system. That said, it's going to cost you more than BQ because of always-on nodes.

Is it better than BQ? It really depends on your analytics workload.

This was exactly my reaction. The tips around Amazon Redshift were spot-on including a few obscure-but-critical ones e.g. the one about many small tables taking up a ton of disk space!

@rjurney - we started with ECS right around when DCOS was coming out of alpha (?). Anyway, it looks slick!

Yes. This is pretty much what we saw.

@SteveWatson - thanks for reporting, should be fixed now.

@yeukhon - for the performance you get, Redshift is quite affordable. Keep in mind it's an always-on data warehouse - there's really no way to make it much cheaper than AWS does because they're incurring compute costs. What are you comparing it to? Hadoop?

That's a great blog post. Thanks for sharing!

>Another annoying thing not mentioned in the article is that the default AMI used for ECS is not at all production ready, you really have to bake your own images if you want something usable. I suppose this is maybe because there's subjectively no "good" defaults, I'm not sure, but it's a bit of a pain.

We ran into this as well - I forgot to add this to the post. The Amazon Linux AMI for ECS has _very specific defaults_ that need tweaking.

Thank you!

@madeofpalk, I haven't seen that, actually. I'll look into it.

There are lots of cases where AWS builds services that are not revolutionary but purely evolutionary. Consider Amazon Redshift, their data warehouse. You would _think_ Redshift, with its poor elasticity and tight coupling of compute and storage would suck compared to BigQuery, which is a truly new way of doing things. However, the combination of "it's just Postgres(-ish)" and rapid feature iteration make it a pretty good system to use. Most importantly, the cost model is very good. TL;DR it really depends on the service.

Thanks @advisedwang. We're looking into it.

HN, I'm a co-founder at Appuri. Happy to answer questions! PS: We LOVE most AWS services like Amazon Redshift. Just not ECS ;)

I have vivid memories of working on Longhorn as an intern. It was an incredibly ambitious project - rewrite of the "filesystem" (really, a relational object store!), driver framework and display stack to name a few. The project failed the same exact way many projects fail - they are too ambitious and not well-scoped. The demo Hillel showed, IIRC, was complete vaporware.

We absolutely love CircleCI. They should charge more.

Same here, we just started using it and are happy with it!

Rahul, congrats on the release!

Plus the "Did you see that creepy van / solicitor / someone who's not the same race as me?" post? It gets tiring.

Same here. We use Quay and are not happy. Their service seems more down than up. The UX is awful.