massaman_yams

Yep. Some similarity to TFDV too, but the UI here looks to be more or less lifted directly from Trifacta/Cloud Dataprep.

pro: - Trifacta can be slow, and part of that might be the way it stores the data (I'm assuming js data structures); if so Pandas/Bamboolib could improve that.

con: - Trifacta/Cloud Dataprep is directly integrated with Cloud Dataflow and can handle jobs that would crash Pandas.

Depends heavily on your use case, but there are a lot of advanced features.

From a review last year (so some of this may be outdated):

- Broader datasource support - 20 sources in Superset, vs 12 in Metabase

Superset supports:

- view creation by users ('create table as'), as well as materialization

- geo/GIS based visualizations

- day over day, week over week, etc. charting; moving averages, etc.

Superset is missing data catalog-like functions, which metabase has, though I didn't evaluate metabase's data catalog in depth. It also requires you to write SQL, where Metabase includes a query builder UI - though there was a rather strange difference in capabilities within Metabase between the SQL-based charting and the query builder UI.

Superset's very powerful, but other options can be easier for common use cases.

Superset's not the best in terms of looks, but it beats Metabase hands down for functionality.

For single addresses that specifically request it, which is both fine and hugely different from bulk notifications to any/all addresses observed in a breach, which is what I was referring to.

But I realize the wording in the original post is a little ambiguous; I had read "provide ... directly" as implying "push", but that may not be the case, and if so my comment above is not relevant.

Bulk emailing notifications to all affected addresses would be a deliverability nightmare, and would require manual intervention at most ISPs to prevent these messages from being blocked, which said ISPs may or may not be willing to do.

Just think of the number of clueless users who would mark such a notification as spam, and the number of old, dead addresses, some of which are now spamtraps.

edit: clarify bulk vs. individual notifications

Right. Split up by IP/subdomain if possible, preferably within one provider.

That said, you may be surprised by the number of large organizations that have a dozen or more services that send mail on their behalf, for various different purposes... it's not optimal, from a deliverability perspective, but it's also not usually going to cause deliverability issues by itself as long as everything else is in order.

The other comments here don't quite tell the whole story; "I never had any problems" is anecdotal - there are plenty of legitimate senders who do run into deliverability issues.

Using an ESP will generally simplify the amount of work necessary to achieve good delivery, but due to the complexities of domain reputation and fingerprinting, very occasionally they can also be the cause of delivery problems.

General guidelines:

- It's a bit harder to predict how deliverability will turn out at smaller ESPs, vs. well-established ones like Sendgrid, SES, Mailgun, and perhaps Mailchimp.

- DKIM, SPF, DMARC (p=none is fine to start; gmail has a guide for deploying dmarc https://support.google.com/a/answer/2466563)

- Use the same root domain everywhere within your email: From, Return-Path, DKIM signature headers; URLs for links and images in the message body. (Separate subdomains are fine, e.g., img.example.com, bounce.example.com)

- Having your sending domain show up in third party email can cause problems. Try to avoid it if possible; use a separate domain as a last resort.

- Send consistently. Some variation in sending patterns will be tolerated, but you'll probably have trouble if you send 500 messages a day most of the time, then once a month you send 100,000 messages.

- Consider a dedicated IP if you're sending over 50,000-ish messages a month.

- You may want to split up transactional email from marketing, sending these from separate IPs and/or subdomains.

- Finally, make sure you're sending email your recipients find valuable and are interested in receiving. Too much disinterest/disengagement can cause delivery issues.

There's less of a clear line between bulk and "business" email than you might think, particularly prospecting email. Looks 1:1 in terms of content, but is sent in bulk and has tracking pixels.

But other than that, I agree. From a privacy perspective, there's some advantage to just being one row of millions in the marketing database; it's the aggregate behavior that they care about, not individual.

That message going to spam is almost certainly an issue with the domain reputation of one or more of the URLs you included.

Easy to test, really. Send her a new message with the same text as before, but without any URLs.

In short, stability. If the performance hit is low double digits, that may not even be noticeable for many use cases.

From the study: "Useful Mg compounds are those that have a sufficiently low first stability constant to be bioavailable and include magnesium acetate, chloride, citrate, gluconate, glycinate, lactate, malate, oxalate, succinate, sulphate and tartrate (Furia, 1968)."

Yeah, I can't imagine everything working perfectly when using various combinations of timescale/citus/pipelinedb functionality.

cstore_fdw is... yeah, there might be other edge cases there, but frankly, it's not a particularly good solution as it severely underperforms vs. any mature column store DB. Though - there's an argument for using it in order to avoid more complicated ETL from the production relational DB to an analytics DB.

That's part of the effect, but I don't think you can reasonably claim it's most of it - demand for rental housing is still high.

Simple. Demand for housing. Supply issues aside, a huge number of people want to live in the SFBA, and the rents reflect that.

This is the reason for measurements of intermodulation distortion and THD+N by frequency. A proper evaluation of, say, a power amplifier will look at a dozen, maybe a couple dozen different measurements; taken all together, these provide a reasonably complete picture of how the amp performs.

But - from an engineering perspective, music can in fact be decomposed into a series of sine waves - and effectively all digitally recorded music goes through this process as part of a/d conversion (sort of; it's complicated) or FFT-based digital equalization. See: https://en.wikipedia.org/wiki/Fourier_analysis

Doing a task queue in an RDBMS is not as simple as some might think, though. https://www.2ndquadrant.com/en/blog/what-is-select-skip-lock...

Ah, I suppose I come from a world where most models are automatically rebuilt at least daily, and sometimes as often as a few times an hour.

Good examples, but re: #1, do a lot of places really deploy models with a static distribution? It should be relatively trivial to calculate this directly from the data within most ML libraries/systems - using a static distribution seems like such an obvious novice mistake.

It's not just data format changes; model accuracy can be impacted by changes in the distributions of values, even if their types remain the same.

That's why production ML systems should monitor for data drift, model accuracy, and a host of other factors that may not be obvious at first. That's part of what TensorFlow Extended does.

See also: "What’s your ML Test Score? A rubric for ML production systems" https://ai.google/research/pubs/pub45742

It's a certification, yes, but in practice it is arguably more effective from a marketing perspective than it is from a farming practices/audit perspective.

You may be surprised by how many people think Organic == pesticide free, for example.

It's not the best solution in cases of clear false positives, but they do provide a help document outlining what kind of practices result in reduced risk of these kinds of problems. (It's oriented toward high volume senders, but many of the recommendations apply at any scale.)

https://support.google.com/mail/answer/81126?hl=en

Definitely. I'd also look at pg version support (GCP is still on 9.6) and extension support.

DMARC already allows a domain owner to specify a recommended action (no action, quarantine, or reject) to take in the event an ISP receives a message that fails DMARC. This DMARC policy can be published at the subdomain level, and best practice is for organizations to use separate IPs/subdomains for sending different types of mail traffic (security notifications vs. marketing, for example), so security notifications can have a significantly stricter policy than marketing mail.

In the paypal example, the 'via' in the from address line is an indicator the message is not DMARC aligned (but doesn't tell us whether or not the actual originating domain publishes a formal DMARC record - I can't tell that since it's blacked out in the GH post).

So for the paypal example, you're right, it looks like they're most likely not following best practices. It may be possible that the 'via' domain is simply a different legitimate paypal domain, which is an edge case that DMARC isn't designed to cover - I think we'll see a technical standard emerge in the next 5 years to handle that kind of thing; maybe built on some of what BIMI is doing - https://www.ietf.org/id/draft-bkl-bimi-overview-00.txt

You're right; complaint rate and similar metrics tracking were implemented by major consumer email providers in the early 00s, if I recall correctly. The technique has been around long enough for those providers to have systems in place to control for edge cases like the one you mentioned - there are enough subtle behavioral differences between an average user reporting something as spam, vs. a deliberate 'complaint brigade', in order to be able to discern reasonably well between the two - or at least, to reduce the impact of the latter. Users marking 'this is not spam' for messages that mistakenly land in spam is a key metric as well, and is part of how a lot of senders recover from short-term spam folder delivery.

Whether or not additional complexity is justified is something that can be measured in a case like this - and that's exactly what Gmail does, using a variety of different metrics, which allows them to make an informed decision about whether to take on the additional complexity cost.

Should they weight false positives for this message type more heavily in their accuracy metrics? Definitely, these are particularly critical to classify correctly.

It's less about the presence of DMARC in general, so much as the fact that the presence of full DMARC alignment (even in the absence of a formal DMARC record) allows ISPs to make more confident assessments of that sender's reputation.

Gmail and Yahoo will do retroactive classification to some extent, but not broadly. It's a little more common to defer delivery by varying amounts based on historical sender reputation, but again that impacts a minority of legitimate email - maybe 10-20%-ish, off the top of my head - and mostly only by a few minutes.

And yes, every major consumer email provider tracks complaint and response rate metrics (as well as many other metrics and indicators) and uses those as part of their filtering. A spam ratio of > 2% is often enough to cause filtering - that's actually toward the very top end of the complaint rate spectrum for messages that are delivered to the inbox.

And allow the occasional obvious phish to deliver to the inbox.

BIMI is... sort of what you're looking for. https://authindicators.github.io/rfc-brand-indicators-for-me...

Adding DMARC would put a stop to that.