roskilli

Well one thing is, there are package updates that could masquerade a backdoor much like XZ Utils[1].

The post in question points to dependency package managers however not system packages, such as NPM, which has pre and post build scripts, install scripts, etc.

[1] https://en.wikipedia.org/wiki/XZ_Utils_backdoor

If you don’t mind me asking: which popular LLM(s) have you been using for this and how are you providing the code base into the context window?

Not sure but I believe it’s possible you may have read the parent comment unintentionally in the inverse? I might be wrong but I believe you posited their desired focus is to manage people from parent comment, but I actually think it’s opposite. They don’t want to manage people (especially career wise), instead they want to manage and guide the work of teams and people across one or many teams.

My interpretation of it was to pursue the type of work and things you focus on as an Engineering Manager in terms of getting the most of out teams for the goals of the organization but doing so without the need to manage people directly. Which I would agree is nice since it’s really hard to wear the technical hat and also directly manage people, so separation of concerns makes for far less context switching and folks to naturally align doing the part of the job that they do best. I also agree with this definition since it’s how I think of it too.

Thanks for the detailed response.

I am surprised there is no gauge update API yet (instead of callback only), this is a common use case and I don't think folks should be expected to implement their own. Especially since it will lead to potentially allocation heavy bespoke implementations, depending on use case given mutex+callback+other structures that likely need to be heap allocated (vs a simple int64 wrapper with atomic update/load APIs).

Also I would just say that the fact the APIs differ a lot to more common popular Prometheus client libraries does beg the question of do we need more complicated APIs that folks have a harder time using. Now is the time to modernize these before everyone is instrumented with some generation of a client library that would need to change/evolve. The whole idea of an OTel SDK is instrument once and then avoid needing to re-instrument again when making changes to your observability pipeline and where it's pointed. This becomes a hard sell if OTel SDK needs to shift fairly significantly to support more popular & common use cases with more typical APIs and by doing so leaves a whole bunch of OTel instrumented code that needs to be modernized to a different looking API.

> Moreover, we encountered some rough edges in the metrics-related functionality of the Go SDK referenced above. Ultimately, we had to write a conversion layer on top of the OTel metrics API that allowed for simple, Prometheus-like counters, gauges, and histograms.

Have encountered this a lot from teams attempting to use the metrics SDK.

Are you open to comment on specifics here and also what kind of shim you had to put in front of the SDK? It would be great to continue to retrieve feedback so that we can as a community have a good idea of what remains before it's possible to use the SDK for real world production use cases in anger. Just wiring up the setup in your app used to be fairly painful but that has gotten somewhat better over the last 12-24 months, I'd love to also hear what is currently causing compatibility issues w/ the metric types themselves using the SDK which requires a shim and what the shim is doing to achieve compatibility.

One feature I’d love to see is a transformer that instead of providing a random value provides a cryptographic one way hash of the data (ie sha2) - that way key uniqueness stays the same (to avoid unique constraints on columns) and also the same value used in one place will match another value in another table after transformation which more accurately reflects the “shape” of the data.

Right exactly. As a point of reference, within M3DB each unique time series has a list of “in-order” compressed timestamp/float64 tuple streams. When a datapoint is written the series finds an encoder that it can append while keeping the stream in order (timestamp ascending), and if no such stream exists a new stream is created and becomes writeable for any datapoints that arrive with time stamps greater than the last written point.

At query time these streams are read by evaluating the next timestamp of all written streams for a block of time and then taking the datapoint with the lowest timestamp of the streams.

M3DB also runs a background tick that targets to complete within a few minutes each run to amortize CPU. During this process each series merges any streams that have sibling streams created due to out of order writes, producing one single in order stream. This is done by the same process used at query time to read the datapoints in order and they are consequently written to a new single compressed stream. This way extra computation due to our of order writes is amortized and only if a large percentage of series are written in time descending order do you end up with a significant overhead at write and read time. It also reduces the cost of persisting the current mutable data to a volume on disk (whether for snapshot or for persisting data for a completed time time window).

Hey jaren hope things are well at Robinhood. Good question, there's a diagram on what a default deployment looks like alongside the M3 v1.0 announcement https://medium.com/chronosphere/m3-v1-0-released-a-productio... and in depth documentation on the website https://m3db.io/docs/overview/.

Storage, aggregation and compute are all separate and scale up/down independently. The coordinator and query services are both stateless and you just add more instances, DB nodes do not do compression/decompression for instance all this happens as part of computation on the query service.

M3DB for storage has a k8s operator that can manage clusters (expansion, etc), and the M3 aggregator can be deployed as a stateful set in k8s and also can be independently expanded.

Modules that rely on global state for anything other than memory pooling or what have you should be avoided. It’s a lot more testable and clean to return a high level data structure that contains any of that state you would have held before globally in the module and have that be the “context” or just the parent data structure to any others that are spawned.

Global state makes thing impossible like parallel unit tests that all use another module, or changing things like “MaxConcurrency” in a way that is synchronized across goroutines that might already be calling into the third party module.

My 2c.

M3 is meant to be an open source, central, horizontally scalable metrics store - but your mileage may vary. Either way, check it out: https://m3db.io

And M3DB too if you want to cluster and scale out, vs cloud store

Curious: What is your strategy on replication? Is it some form of synchronous replication or asynchronous (i.e. active/passive with potential for data loss in event of hard loss of primary)? Also curious why you might look at UDP replication given unless using a protocol like QUIC on top of it, UDP replication would be inherently lossy (i.e. not even eventually consistent).

Thank you for mentioning that, Andrew's post is really fantastic covering many things altogether: fundamentals, data structure, real world impact and examples.

FYI M3 is now backed by M3DB, a distributed quorum read/write replicated time-series based columnar store specialized for realtime metrics. You can associate multiple values/timeseries with a single set of dimensions if you use Protobuf's to write data, for more see the storage engine documentation[0]. The current recommendation is not to limit your queries but limit the global data queried per second[1] by a single DB node by using a limit on the number of datapoints (inferred by blocks of datapoints per series). M3DB also uses an inverted index using FST segments that are mmap'd[2] similar to Apache Lucene and Elastic Search to make multi-dimensional searches on very large data sets fast (hundreds of trillions of datapoints, petabytes of data) which is a bit different to traditional columnar databases which focus on column stores and rarely is accompanied by a full text search inverted index.

[0]: https://docs.m3db.io/m3db/architecture/engine/

[1]: https://docs.m3db.io/operational_guide/resource_limits/

[2]: https://fosdem.org/2020/schedule/event/m3db/, https://fosdem.org/2020/schedule/event/m3db/attachments/audi... (PDF)

If you're looking at scaling monitoring timeseries data you may also wanter to consider more Availability leaning architecture (in the CAP theory sense) with respect to replication (i.e. quorum write/read replication, strictly not leader/follower - active/passive architecture) then you might also want to check out the Apache 2 project M3 and M3DB at m3db.io.

I am biased obviously as a contributor. Having said that I think it's always worth understanding active/passive type replication and the implications and see how other solutions handle this scaling and reliability problem to better understand the underlying challenges that will be faced with instance upgrades, failover and failures in a cluster.

In the companies I've worked at, H1-B workers get compensated in software engineering roles just as well as their peers, I've also seen outliers where it's significantly higher (depending on when they joined, how much in demand they were with competing offers, etc).

There are plenty of >200k H1B workers in SV, and I do believe these engineers and specialists will wonder how they can do their work elsewhere if they can't reasonably plan a few years into the future:

https://h1bdata.info/index.php?em=netflix&year=2020

https://h1bdata.info/index.php?em=apple&year=2020

https://h1bdata.info/index.php?em=linkedin&year=2020

https://h1bdata.info/index.php?em=uber&year=2020

There's a lot of interest in this space with respect to analytics on top of monitoring and observability data.

Anyone interested in this topic might want to check out an issue thread on the Thanos GitHub project. I would love to see M3, Thanos, Cortex and other Prometheus long term storage solutions all be able to benefit from a project in this space that could dynamically pull back data from any form of Prometheus long term storage using the Prometheus Remote Read protocol: https://github.com/thanos-io/thanos/issues/2682

Spark and Presto both support predicate push down to a data layer, which can be a Prometheus long term metrics store, and are able to perform queries on arbitrary sets of data.

Spark is also super useful for ETLing data into a warehouse (such as HDFS or other backends, i.e. see the BigQuery connector for Spark[1] that could write a query from say a Prometheus long term store metrics and export it into BigQuery for further querying).

[1]: https://cloud.google.com/dataproc/docs/tutorials/bigquery-co...

M3 separates query and ingestion if you're interested in clustered storage for metrics, slide in question here: https://www.slideshare.net/RobSkillington/fosdem-2019-m3-pro...

You may also want to checkout https://eng.uber.com/m3 which is a highly available RF=3 multi-node TSDB metrics backend and is used with heavy Prometheus workloads and is used to ingest tens of millions of timeseries per second.

Interesting, didn't realize it imposed a lot of client side changes, I'm curious to hear your experience on that? Was it the change in MySQL compatibility that makes it an upfront cost?

With Vitess I believe most get started with just deploying VTGate and just proxying their existing queries to their existing MySQL server, then once that is working nicely, start to range shard and expand the cluster sitting behind VTGate.

Since it's range based sharding, was Vitess ever considered (a popular CNCF MySQL range sharding cluster approach that came out of YouTube)?

Chronosphere | Senior Frontend Engineer, Senior UX Designer | NYC, Seattle | Full-time Onsite (Remote for now) | VISA

Chronosphere is a Series A startup delivering the most highly scalable and reliable open source metric data platform to the world. That platform is M3, which was created by the founders of Chronosphere while at Uber. It is the only open source metric data platform proven at petabyte scale in production, stores over tens of billions of metric time series and ingests and serves billions of data points per second. Chronosphere is a modern and highly scalable monitoring monitoring and observability tool built on M3 for companies faced with scaling challenges. We deliver products and APIs that run in the cloud as a single hosted platform.

Senior Frontend Engineer:

Are you driven by how users digest, explore and build proficiency using a product? As a front-end product engineer you will build interfaces that engineers, data scientists, business analysts and operations staff will use every day to build, analyze and operate products at scale for the companies they work at. Your natural design instincts will be paired with great Javascript fundamentals and how to best design and consume APIs, either using GraphQL or other APIs. For our customers, our tools are leveraged by thousands of engineers on a daily basis to measure and monitor their products and systems. Your work will continue to drive fundamental change to how software is created, delivered and operates all over the world. You will work alongside the engineering team, the head of product and product team, the CEO and the CTO.

Technologies: Javascript (ES6), React, GraphQL on the front end. Go, GRPC, Kubernetes, MySQL and M3DB (distributed time series database) on the back end.

Senior UX Designer:

We're looking for a UX designer to help build Chronosphere. You will drive and complete the design process for our cloud web application from beginning to end. You will work with multiple teams to design a clean, usable console focusing on metrics, data visualization, altering, dashboards, infrastructure, and more. We are open to this role being in either New York or our Bellevue, Washington office. You will work alongside the head of product, the engineering team, the CEO and the CTO.

What we offer: Excellent benefits, competitive pay and equity at Series A, flexible working environment, challenging problems at scale and pragmatically harnessing millions of instrumentation being collected per second.

Senior Frontend Engineer: https://boards.greenhouse.io/chronosphere/jobs/4012681003

Senior UX Designer: https://boards.greenhouse.io/chronosphere/jobs/4012682003

Email me with questions at rob at chronosphere.io

Regarding remote for now: If you are applying to work in the NYC or Seattle office you are not expected to interview in person (it will be remote), or work in person until the CV-19 crisis is resolved (work will be remote). The roles are not remote, but no one is in the office currently.

New York is now testing more daily per-capita than South Korea and China, so naturally due to the higher test rate (which gives ability to find the clusters) New York is going to pull ahead of the rest of the country because it will have the ability to actually show the cases that are active but weren't able to be tested.

The Melbourne Symphony Orchestra is doing just this, and it’s been pretty successful they believe (!)

https://www.mso.com.au/contact/mso-coronavirus-statement/

CDC published numbers on Wednesday night that 40% of hospitalizations for COVID-19 in the US are adults younger than the age of 54.

Sure they're not fatalities, but do you really want to go to an overcrowded hospital for COVID-19? (Will be worse when ventilators run out and hospitalization can't help you nearly as much)

Very interested as someone looking to work with other HN folks, ready to hire people who could onboard and start during the work from home mandate (we had an employee onboard Monday from their home).

That's interesting, I had not heard of ClickHouse as a backend for Graphite with an inverted index. Let me know if you have any links to that.

I'm assuming this is an out of process inverted index used alongside ClickHouse? Or is it more of a secondary table contained by ClickHouse which can be searched to find the metrics, then the data is looked up?

The latter scales not as well with billions of unique metrics since it's always a scan across the unique metrics stored in the time window your query searches for (since any arbitrary dimensions can be specified, all must be evaluated). This is the drawback of PromHouse which is an implementation of Prometheus remote storage on top of ClickHouse - and the major reason why PromHouse was only ever a proof of concept rather than a production offering.

M3DB ingested 30 million datapoints per second (so 1.8 billion per minute) with each node writing hundreds of thousands of writes per second. The dataset was in the petabytes.

For us the cost savings vs OpenTSDB (millions of dollars of hardware), the faster query time and the reduction in oncall overhead was worthwhile.

Whether it saves space or not, looking at metrics over period of months or years when the data is raw is far more slow/expensive than looking at downsampled data.

If you still want to be able to quickly graph and view old data, downsampling is the only way to keep your queries interactive.

Take for instance 30s data vs 10min data. 20x more computation, network exchange and everything else of that nature needs to happen.

Also if you want to keep only a subset of your data for a very long time, you need to have retention policies - otherwise you end up storing all that extra data forever.

At large numbers (terabytes to petabytes) this stuff is impactful, at smaller numbers (gigabytes) my points here are far less relevant.