zten

In my experience, this is because the other drivers are just rolling the dice and passing on blind curves or passing in the same lane with barely any clearance. It's not an example of good driving ... but it is predictable.

Edit: on slightly closer inspection, this isn't true for the 27" models!

The compression actually works as advertised, but that document outlines why you don't want to have the sortkey skew from other columns. The disk I/O balloons as it goes and fetches a larger number of blocks from the other columns that match what you're filtering in the sortkey.

Consider Spark and PySpark. The Python bits of Spark are in a sidecar process to the JVM running Spark. If you ask PySpark to create a DataFrame from Parquet data, it'll instruct the Java process to load the data. Its in-memory form will be Arrow. Now, if you want to manipulate that data in PySpark using Python-only libraries, prior to the adoption of Arrow it used to serialize and deserialize the data between processes on the same host. With Arrow, this process is simplified -- however, I'm not sure if it's simplified by exchanging bytes that don't require serialization/deserialization between the processes or by literally sharing memory between the processes. The docs do mention zero-copied shared memory.

I am unsure where the preference lies. I don't always think there's a preference, but you can get forced into it by other economic factors. Young people move into a city, getting adapted a mass transit lifestyle, but they move to a more car-centric area when they inevitably get priced out of car-free living downtown or on a popular rail/bus line because they want to start a family. Personally I don't really want to give up being able to walk outside and down the street to go to restaurants and shop or connect to transit easily, but it would be short-sighted to say that everyone has the same preferences as me.

There's benefits to living in a detached house with a yard. The price is usually being tethered to a car in the US. If you pick a low density area you're probably not going to have an oppressive commute and running errands will be easy (if predictably lengthy, but knowing it'll always be a 20 minute drive and not 20-60 minutes based on traffic is valuable) because there's large roads and every business was mandated to build a large parking lot.

The problem comes when inevitably other people make the same decision and development happens and more people move in. A Californian pattern of development (and in the many areas of the country that have since copied this pattern) is incredibly hostile to good public transit, but due to the geometry of cars, you inevitably have to do _something_... it's just probably going to suck and get underused. So, an alternative strategy emerges: fight tooth and nail against growth.

- Performance from tables not cached on the warehouse instance is awful. That the price you pay for shutting a warehouse down.

- I wish it were cheaper. If you run queries against a warehouse 24/7, preventing from auto-shutdown, you better hope it's tiny. And even then, the cost might incentivize you to employ a different strategy entirely.

The other stuff you mention is them building a moat around your data lake. They're pretty good at that. I'd happily get locked into Snowflake for the moment. Redshift really does look like an amateur hour product in comparison to the tools you get with Snowflake.

Even then, as good as Snowflake is, our internal users went from complaining about the performance of Looker and Redshift to complaining about the performance of Tableau and Snowflake. I don't know if you can ever please anyone in this space...

What I wanted to emphasize, mostly, was that there exists a tier of companies that offer stratospheric compensation primarily through equity, but there are many more people working at other publicly-traded companies that offer RSUs that are worth closer to what you might consider to be a yearly bonus.

There's outrageous equity offered by like 5 companies. Everybody else is getting stock option grants worth 5-10% of what Facebook, Amazon, Apple, Google, and Microsoft are offering (the situation is a little different for Netflix).

> Am I a giant dumbass for staying in Colorado?

Probably not, Colorado is great.

Yes, it's just a building block. The easiest way to use Parquet on Java is to use Spark's integration with it, because it provides the query engine for you. But, if I'm not mistaken, it's much bigger than Pandas.

I also strongly dislike the Hadoop coupling (hey, it's called parquet-mr for a reason...), but it's more or less an invisible annoyance if you use Spark in an environment like Amazon EMR.

I think you'll get a lot of confused responses from people if you want to try to build an application that reads Parquet directly. It's the disk format for a bunch of distributed database engines. They'll wonder how you plan on querying it.

But down in reality, most companies are probably running their own made-to-measure deployment and operation schemes with lower quality and consistency than Old Navy.

Zipp and ENVE seem to be in some kind of internal width arms race with the release of the Zipp 303-S, and the ENVE SES AR series rims. I was exaggerating a little bit :) You'll still find many, many road bikes with disc brakes maxing out with clearance for 32mm width, but that's an improvement over their rim brake cousins. But look around in the growing "all-road" segment and you'll find wider tire clearances on road geometry frames showing up.

I am unsure if this hookless is the same hookless that is from way back in the day. With the way things go in the bike industry (cyclically, no pun intended), I wouldn't be surprised if it were.

You pay a premium to have this stuff loaded up on faster hardware. If you look at how Redshift works on their compute-optimized nodes, everything's sitting in local NVMe SSDs (a slightly different strategy is used on ra3 nodes). They handle the fact that this is ephemeral with automatic backups. Cluster restores aren't exactly fast. I actually agree with you; if you've got some monster SSDs attached, which are comparatively cheap, why focus on the RAM... I believe there are reasons to do that sometimes, but not everything demands quite that level of performance.

For this point:

>> You _could_ write something on your own that just forks out a bunch of threads in a single process to rip through the data, but why?

> Because it's simple and easy. I wrote one in a few days. Not much code.

I think it depends on what formats you're using and how it's laid out on disk. A lot of people reshape their data into a table-like structured or semi-structured format, and that makes it a candidate for putting it into a database like Postgres or Redshift, and other times it makes more sense to bring the database to the data (the Hadoop ecosystem of stuff.)

For example my company still has stuff that writes out row-like objects into S3, and it's not too hard to write a single process job that spawns threads, reads the input files, and does some computation on those. They're on S3, so throughput kinda sucks, and copying it to local SSD only makes sense if you want to make multiple passes. But the Parquet format alternative of this data is tremendously faster to work with and genuinely easier to use, and the only barrier to entry is that you run Spark on your local machine and commit to using Spark with Elastic MapReduce to do processing for this data. Sure, you might only end up filtering through tens or hundreds of gigabytes; terabytes is usually rare. But part of that is because you only read 10-20% of every input file to do the work. It also integrates extremely well with other stuff we're using - it's even easier to just use Snowflake against the same data set, for example.

Sorry, I think I'm rambling at this point and not presenting a really coherent argument.

If you start from the assumption that you've been ingesting data and storing it in a compressed columnar format like Parquet or ORC, then you're already locked into a solution that exists in the Hadoop ecosystem. This turns out to be an effective way to deal with terabytes of data because depending on the query you want to execute, the file format (and a smart partitioning structure) helps turn your problem of reading terabytes into one of reading gigabytes. And, everything in Hadoop land is generally a query, so you're going to need a query engine like Hive, Impala, Spark, etc. AND, you probably need something like Hive metadata so that you're not crawling some directory structure for the schema and input files every time you start up a new process to run a query.

You _could_ write something on your own that just forks out a bunch of threads in a single process to rip through the data, but why? Think about what you're effectively implementing - a bespoke query engine that runs one query plan. Spark has already written an API and a query engine (multi-process distributed, unlike whatever you're likely to hand-roll) and lots of input/output code. You can spark-submit --master local[*] and use up all of the cores and ram everything into one monster JVM if you really wanted to.

Finally, you could stuff all of this in memory, but at what cost, and what are you going to use to do it? (Worse, what happens if the VM goes down - those terabytes are going to take their sweet time reloading over a network link.)