ERCOT operates most of Texas’ grid: https://en.wikipedia.org/wiki/Electric_Reliability_Council_o...
586 karma · joined August 9, 2012
VP Product, Materialize (https://materialize.io/)
Co-founder of Perssist (https://www.perssist.com). Formerly at Datadog
[ my public key: https://keybase.io/awang; my proof: https://keybase.io/awang/sigs/A9k8bWf7Dlozsz1jeKR5Fy6G3VVYcNV4TgdmPJuU0yc ]
Just as there are many MBAs who were or are veteran software developers, the HN community is large enough that there are many members who are professional investors.
https://youtube.com/watch?v=3CxZ2PrXQ1M
https://www.metmuseum.org/exhibitions/listings/2021/medici-p...
Re: custom code -- our codebase is fully source-available and open to contributions, but the source+sink code going through some refactoring to make it more beginner-friendly. Depending on your consistency requirements, we also support Debezium and our own CDC format (https://materialize.com/docs/connect/materialize-cdc/) for folks who want to bring in their own data sources. (For quick prototypes, we also support csv/json/plaintext source types, as well as SQL INSERTs!)
Re: the second point — you’re right, Materialize has historically leveraged existing upstream systems (like Kafka) for things like persistence. But we also hear you loud and clear that not everyone wants to stand up Kafka :)
Link to press release: https://www.prnewswire.com/news-releases/lillys-donanemab-sl...
But to answer your question, yes, our intention is to support separate cloud-native storage layers.
In addition, while Materialize does support connecting to other databases, to power "real-time materialized views", a common architecture we are used for is to present a SQL view on top of streaming systems (such as Kafka or Kinesis).
Here's an overview of Materialize that explains the relationship between Timely Dataflow, Differential Dataflow, and Materialize, starting at 23:20 - https://materialize.io/blog-cmudb/
Building realtime, low-latency (< 10 sec) dashboards. The type of things where you previously would have had to wait for several hours or a day for ETL pipelines to crunch through a lot of numbers.
We're also fielding interest for streaming ML applications. Ie, moving from batch models to streaming models.
Also worth pointing out that Differential Dataflow has been around for awhile, while Materialize is fairly young. We're still constantly learning about new applications!
The closest benchmark I could find was the "FASTER State Management for Timely Dataflow" paper from ETH Zurich, but that wasn't run on a laptop.
#1 -- Postgres is built in C, while Timely/Differential (which underpin Materialize) are built in Rust. Materialize could be a Postgres extension at some point, but for now we want to control the stack (to more easily do cross-dataflow optimization, etc)
#2 -- We are absolutely interested in exposing an API to enable new data sources (and data sinks!). It currently takes a bit of time to understand some basic Timely Dataflow concepts, but we intend on documenting this and opening things up. We're also trying to understand user requirements around things like durability guarantees and UPDATE/DELETEs (Feel free to email me or hit us up in our public Slack as well).
If anyone is interested in sending data via HTTP POST, we'd love to hear more: https://github.com/MaterializeInc/materialize/issues/1701.
Some more info linked here: https://news.ycombinator.com/item?id=16570548
It links to the original lawsuit (https://arstechnica.com/wp-content/uploads/2016/03/IBM.Group...) as well as the 4 patents: 5,796,967, 7,072,849, 5,961,601, and 7,631,346
From the abstract: Using algorithms to analyze the blockchain data, we find that purchases with Tether are timed following market downturns and result in sizable increases in Bitcoin prices. Less than 1% of hours with such heavy Tether transactions are associated with 50% of the meteoric rise in Bitcoin and 64% of other top cryptocurrencies. The flow clusters below round prices, induces asymmetric autocorrelations in Bitcoin, and suggests incomplete Tether backing before month-ends. These patterns cannot be explained by investor demand proxies but are most consistent with the supply-based hypothesis where Tether is used to provide price support and manipulate cryptocurrency prices.
Tracing requests across services and hosts to identify bottlenecks is one reason why Datadog decided to build a tracing component in APM! A flamegraph is one such method to visually infer critical path analysis that Datadog APM supports. Some obvious things to consider are ease of setup and whether the libraries and languages you use are supported. Some not-so-obvious things to consider are:
(1) whether you wish to infer this from a single request, or across multiple requests,
(2) how to define "interesting" traces -- as you could be collecting millions or billions of traces in production
(3) how to create a bigger, "aggregate" picture across individual traces.
Source: https://support.robinhood.com/hc/en-us/articles/115000902306...
In other words, Graphite can be used to aggregate any metric you're interested in (such as CPU usage), which you would then correlate with other relevant metrics (such as number of web requests/second). If you're specifically interested in tracking application-level performance (what is my website throughput or where are my endpoint bottlenecks), you would need to find an integration for your specific environment, or custom instrument your application.
AppDynamics also has graphing features, but aims to bundle any other relevant use cases an application developer or operations person might need. For instance, it includes service topology maps (see which services are running on your infrastructure, and are talking to which other machines) and application logs (to parse unstructured text). They have also built many integrations (to collect relevant metrics), usually for large enterprises.