dswalter

It feels like Cantrill being a creator of DTrace earns at least the credibility to be called a 'DTrace expert' without quotation marks.

We've managed to accomplish something similar for cryptocurrencies, too, with overpowered exchanges!

Trino is the community-open-source fork of Presto (they were recently required to change the name from PrestoSQL, which they had been using to make the Presto lineage clear).

(not affiliated in any way). https://trino.io/

According to the comments in the history of that user on Github, it is someone who claims to be an engineer from Facebook in an earlier post: https://github.com/WICG/trust-token-api/issues/28#issue-6447...

Imperva | data scientist-bot detection | Raleigh/REMOTE (us-east-1 timezone-compatible)

This position is for the team I'm on, developing and implementing new ways to protect customers from bots.

I'm happy to answer questions about the position, the team, or Imperva, which acquired Distil Networks in the summer of 2019.

Apply here: http://app.jobvite.com/m?3xY9tlwX

An update has now been posted to that account: https://twitter.com/SlackStatus/status/1346112838176149505.

Text below:

Customers may experience issues connecting to Slack to loading channels at this time. Our team is on the case and we will keep you posted. Apologies for any disruption.

Why must Amazon pass off its savings to your org, in this example, rather than taking most of those realized R&D savings to their own investors?

Sure, AWS may be constantly innovating on doing things more cheaply and scalably. Sometimes, those savings get passed off to the customer; but not most of it, and not necessarily all that frequently.

Anecdotally, some messages seem to be failing to send.

I am on a 2019 macbook pro with decent internet speed, and it took >20 seconds to load.

According to their pricing page, a 50TB SSD from Nimbus costs $12,500. That would have been worth the expense in DBA time alone, right?

Edit: added the link. https://nimbusdata.com/products/exadrive/pricing/

If I could swap over to the parallel universe where the big data landscape was implemented in Rust instead of Java, I would make that trade.

Hopefully there would be other good things about that universe, too.

There are many jobs where a lack of attention to detail (i.e. spelling) on your resume will get you tossed aside from consideration. It's not unreasonable to request a Director of Engineering be held to a similar standard on a public-facing document like this.

I think you may find R an excellent tool/ecosystem for that kind of work.

That's a bit of a misconception. From what I understand, Amazon's non-AWS branches don't get deeply-discounted services from AWS. There is a discount, but it's not enough to turn dark skies into sunshine and rainbows.

Amazon tends to want every part of itself to be in ship-shape, and giving itself a massive discount would discourage efficiency in non-AWS parts of the business.

Disclosure: neither a current nor former Amazon employee.

If you're using AWS athena for querying, you're also using the aws glue catalog (managed hive metastore-ish service) to know where partitions are, but yeah, you'll need to partition and sort your data to make sure you're not doing full table scans.

Presumably one reason to use this is latency: materialize is built on differential and timely dataflow, some innovative frameworks (by Frank McSherry, who was at Microsoft Silicon Valley Research back in the day and is elsewhere on this HN discussion) that are intended to reduce the amount of computation in certain kinds of calculations. Materialized views are particularly ripe for those advances.

It's also written in rust instead of Java, so there's no JVM RAM penalty or GC to contend with.

I may be in the minority here, but I don't think the changes we needed were to make it easier for higher-ups to cash out of a company before rank-and-file get a chance to.

Is there a reason not to consider apache airflow? https://airflow.apache.org/

You could run it on a digital ocean droplet so you don't have to worry about your laptop turning off. It covers retry and cron-style job kickoffs.

Yeah, that's scary. The maps integration makes that insanely informative.

And presto's rich functionality for complex data types (maps and arrays) as well as window functions makes some pretty challenging things possible.

When you add in the cost reductions from employees who quit to go somewhere they can get any work done, the savings just keep stacking!

I just finished the book; it's uneven, but incredible. Some of her conclusions and how she frames the problem are arguable, but her presentation of the facts and her underlying hypotheses about how the situation has arisen are fascinating.

This passage, and really her whole summary of the history of Google, (which goes beyond just this excerpt) are particularly compelling.

Your version only works as long as there is only one browser.

There are a bunch of questions that attempt to proxy how mature the data pipeline/data engineering is of the company you're looking at.

* Do you do batch and/or streaming computation?

* What kind of access do you have for querying/accessing the data this role will predominately be working using?

* Do you have dedicated data engineers and data infrastructure people

* What's your workflow orchestration engine?

* What is the data access pattern for historical data (there had better be at least SQL access here).

* Do you have built-in feedback loops for your machine learning products?

* What is your serialization format of record for production and for OLAP?

* How often do schemas change in your databases?

What your casual elitism disregards is that people who actually do the work can have different ideas, especially related to the work they are doing. Put another way, since they have a different perspective than the higher-ups, they are sometimes able to find potential efficiencies and product improvements that wouldn't occur to others.

Additionally, front-line workers can see real problems with how customers experience the product that may not be reflected in the metrics used to evaluate the product, since metrics are inevitably gamed.

The ratio of good to bad ideas doesn't matter if you miss the one good idea you really needed. A company that invests in ways to effectively use the ideas of 'untrained people', as you so delicately put it, can derive a lot of value from those ideas.

6 months after an IPO is often the conclusion of a typical employee lockout period on shares, which can also be a time of more frequent team members leaving.

I wonder what the retention incentive period was for Github employees; it's often a year, and the Github acquisition was mid-2019...

I'm actually excited about the possibilities. I've watched DataFusion from afar, and I have spent a decent amount of time wishing the Big Data ecosystem had arrived during a time when something like Rust was a viable option, both for memory and for parallel computing.

I use Presto all the time, I love how fully-featured it is, but garbage collection is a non-trivial component of time-to-execute for my queries.

> it can become more powerful than governments.

We've probably crossed that bridge already, but this would certainly speed up that outcome.

Depending on who you follow, it's one of the best link aggregation sites around. Care deeply about refugee rights in Myanmar? Follow a few folks and you'll gain insight into the kinds of things like-minded folks are thinking about. There is the on-Twitter conversation, too, but the best thing about the site is what people are linking to out of it.

Let's say you have a company that tries to make online checkout more efficient by predicting which payment the user will prefer, and research shows that if the user is first presented with a payment method they use, they are more likely to buy.

If I subscribe to your API and get the outputs of your predictions, since I can see the kinds of inputs that you tell me are associated with preferring PayPal, I can approximate your PayPal-preference model.

This works for basically any machine learning model-as-service. So if I invest heavily in approximating and then serving models more at lower cost than anyone else, that might be a viable businese.