kwillets

My experience comes from around the same time frame -- I spent about a year on an aborted spectral dimension reduction project, and I only recently realized how similar the problem still is today.

I'm not sure if that makes me more or less qualified to do vector DB's -- I tend to block out things that I learned a lot about in the past without much result.

Thanks for the writeup; in my opinion WCOJ is the most interesting development in core technology.

I'll have to disagree about database customers becoming more savvy; cloud has brought in a whole new crackpipe for managers to smoke.

I miss the days when they just chanted about Developers.

I think they're trying to keep cryptographically secure data out of scope.

Message queueing triggers a host of psychological needs. Synchronous jobs rarely need monitoring and management features, but move the same work to a queue, and everybody loses their minds.

"For example, you've run 234,745 INSERTs into [table name deleted]_HOURLY and ALL of them were under 1000 rows."

That's an actual quote from our org's snowflake support slack, about pretty much the same problem: lighting up the cluster with a bunch of tiny transactions spread out in time. The product is not made for common use cases like trickle loading.

I'm an actual experienced data warehouse admin, so I'm watching them rediscover my job description one cost overrun at a time. They unironically say things like "It's self-managing, but you need to monitor costs and rewrite loads and queries to run more efficiently". I'm afraid to ask them what they think I do all day.

He said a few thousand queries per week, which is maybe 3000, or 156k/year. That's 1.56 million cluster minutes, or $500k/1.56M = $.32/minute. A single snowflake node is $.05/minute, so maybe it's an 8-node cluster. They could shrink the node count down to 4 or 2 to save more money.

The final methods seem equivalent to reading a non-float binary fractional number .xyz... from a random bitstream and stopping at the point where further digits will be truncated in the float representation.

eg if x = 1 read 53 more mantissa bits and stop, if x = 0 decrement the exponent and repeat. There's no bias in such a process.

Ironically one of the major uses of analytics has been to highlight the impact of slow response time on user retention for a wide class of applications.

I also feel that speed builds trust, although I don't know specifically why. Perhaps people envision more errors or error-prone processes when a system is slow. It certainly shows more understanding of the data to present it quickly.

It's really bad for in-product analytics. Slow and expensive to keep running on a 24/7 SLA.

Even with Vertica doing that, we're seeing 10x costs just doing back-office DWH. My job now is keeping Vertica running so we can pay our Snowflake bill.

No, most revenue was just people buffing their FarmVille plots. The company aimed away from online gambling or even hardcore gaming.

It was hard to understand what whales were doing because the games were so far from gambling; there was no legal way to cash out, and even goods sales were policed. Money spent was clearly for amusement purposes only.

These were people who spent thousands of dollars leveling up in Mafia Wars. The interpretation at the time was that these were affluent individuals who simply enjoyed competing with money.

The article makes it sound like they just discovered whales in 2014 -- I was at Zynga in the early days, and we were definitely aware of them even on early games, but there wasn't a formal VIP program.

The push to give them special treatment didn't take off until the Facebook tax appeared; at that point it became important to move big spenders to a separate payment channel without the 30% skim.

Good news: There's probably a linear time solution to all the versions of the problem presented.

Bad news: You have to walk this stable genius through 50 years of Suffix Tree and constant time LCA algorithms in 20 minutes.

If they wanted to increase risk/return, why did they buy T bonds in the first place?

The irony of these riskmeister VC types failing to manage a Treasury Bond portfolio is pretty thick.

It also seems like they should have made some credible loans instead of buying bonds - that whole "reinvest in the community" thing. Although they might be required to keep some low-risk (when managed by adults) assets.

I'm not Korean, but I can confirm that every cab driver in Seoul searches GPS this way.

> innovative storage features like zero-copy cloning and time travel.

These are not original to SF. COW snapshot cloning has been around for decades, and eg Vertica has had a copy_table function since IDK how long.

Time travel was in V1 of Vertica, and a quick search shows the same feature in DB2 and other platforms.

left-nimby's, as they're called, obstruct housing with price controls and other uneconomical demands on housing producers and buyers. Often they reject projects out-of-hand if they show any possibility of making a profit.

MinHash is a CPU hog due to precisely this step -- cheap rehashes are the order of the day.

I don't think xor works, but simple arithmetic, IIRC h_i(key) = i*h(key) + C mod 2^32, is viable and SIMD-friendly. Look up minwise-independent hash functions.

The net result of the hashing etc. is to shuffle the unique elements of A union B. In that shuffled union, the first element is in at least one of A or B; if it's in both it's in the intersection. The chance of that is J.

He could have become a tech company middle manager and avoided all this trouble.

random was what came to my mind after your first sentence. It could really use a deterministic test harness for the different layers.

The buffer in this case is the input to a max() function; his point is simply that sorting the dictionary indices before looking them up is a good idea for common data structures like arrays, disk blocks, or anything with caching or readahead.

Is there any benefit to a multipivot sort? The partitioning would be more complex but possibly save memory accesses.

Autonomy took advantage of their UK base to avoid Sarbanes-Oxley and simulated growth by buying defunct US companies and rolling up their revenue. I don't see why this guy hasn't been extradited yet due to the fact that virtually all of the activity, especially the acquisition, took place in the US.

I once saw a TV clip where they challenged viewers to tell the difference between per-province election results and the response to Sejong's tax system at the beginnings of the Joseon dynasty -- they both had the same GSD/JLD split.

Unfortunately I can't find it now though; it was very insightful.

In Korea it's more the pay for conscripted soldiers and general living conditions, including bullying, incompetent leadership, suicide, etc.

And of course there's a K-drama about that -- D.P.

Samsung is considered the epitome of state capitalism, so there may be some confusion about that last sentence.

+1 for comparing wind tunnel capacity. Also TX has no dirigible hangars.