We haven’t decided that—Substrait[0] is still the IR for query plans across different execution engines—but projects like QueryBrew are a great side effect of what a SOTA planner can accomplish.
[0]: https://substrait.io
2,328 karma · joined September 29, 2009
[0]: https://substrait.io
Their positioning is: You already have all your teams spending routing through us so that through greater transparency to finance you get better spending controls. We now apply the same to inference, which is spread out across all parts of your company in a similar way.
All summed up really well with their tagline “Tokens are money. Save both.”
Don’t think I’ll be using model routing / shared gateway from Ramp any time soon, but the value of this kind of capability is definitely there. And there’s no shortage of vendors aiming to offer this kind of gateway/router functionality. But most of those come from the technology side of the organization. This gets to come from the office of the CFO.
Super excited to try AudioMuse
[0]: https://developer.spotify.com/documentation/web-api/referenc...
Also I’ve heard wonderful things about The Great Miscalculation[0], a recently released book about the Citicorp Tower incident
Under the hood it's powered by Mosaic[0], a dataviz library built on top of DuckDB that's designed to handle coordinated interactive plots over huge datasets, the kind of thing where you interact with one plot and the rest all respond, which requires going back to the database to recalculate all the aggregations.
I've been fanboying Mosaic for the past year but finally have this to point to as an illustration of what's possible with it.
With v4 of h3 they (finally) have a clean syntax for this with polygonToCellsExperimental[0].
Now there’s options for
- Cell center is contained in the shape (default) - Cell is fully contained in the shape - Overlapping (covering): Cell overlaps the shape at any point - BBOX: Cell bounding box overlaps shape
Makes life a fair bit easier if you’ve gotta deal with H3 polys. And if you’re working locally, DuckDB Spatial’s r-tree indexing[1] can make for a nice stand-in for PostGIS as a quick point-in-polygon solution without the need to spin up a service.
[0]: https://h3geo.org/docs/api/regions/#polygontocellsexperiment... [1]: https://duckdb.org/docs/stable/extensions/spatial/r-tree_ind...
Australia’s national library offers Trove[2], which has a huge collection of Australian public domain newspapers.
Most of their repository is likely funded by your tax dollars and is there for the public to use.
[1]: https://www.neh.gov/divisions/preservation/national-digital-... [2]:https://trove.nla.gov.au/newspaper/
From my [fairly-out-of-the-loop-for-the-past-few-years] vantage point, Mozilla’s been a lot less invested in the PWA ecosystem since they abandoned their Firefox Phone / Boot2Gecko initiative, which was intended to create a middle tier between the expensive smartphones of the early 2010’s and ubiquitous and cheap feature phones (flip phones, classic Nokia candy bars), and expand access to the web across the world with it.
All the apps were PWAs, which made it simple to build out. Eventually Mozilla stopped the project, but KaiOS became a commercial implementation and it still runs on a fair number of feature phones to this day.
But without that pressure for PWA support in Firefox as a critical mobile feature, it was largely serving as an expensive bookmark launcher in the Firefox code base so that folks could alt-tab to the small number of sites that supported it on their desktops. Not a noble end for Mozilla’s support for what should have been / could have been an incredible leverage point for the web ecosystem and open development.
You know the drill: SQLite and DuckDB mean you’ve got transactional database and data warehouse that live in the single machine and unless it’s terabytes runs analytics performance comparable to bigquery and snowflake.
No need to write an etl pipeline that runs on a Hadoop cluster, provision that Hadoop cluster, spin up a Hive/Pig instance for analysis. Nah, it fits in *your* computer and is scripted using the same language in the pipelines as in the analysis without a performance cost.
If you need to scale it, it’s not technical scaling, it’s team knowledge scaling (still hard, but not a fundamental stumbling block). So bring in DBT/Dagster (or airflow) and now it’s got supported frameworks that others already know and use.
Gmail absolutely hits that sweet spot of API capabilities and where the users are, so I can’t fault the project creators (or most every email client business these days) for building first (or exclusively) for it.
That said, seeing Outlook as coming soon on their login page is reassuring that they’re building in a way that won’t tie them to Gmail forever. And while few email providers outside of Fastmail are offering JMAP support, as an API it’s much closer to the degree of functionality expected by anyone building on top of Gmail’s API today. A great new client that gives a big section of the public a better way to “do email” might be what it takes for more services to start offering JMAP.
So hats off to y’all and fingers crossed on incorporating open standards.
But that ship sailed long ago. Very few were still able to offer Sunday service before this:
- NYPL (Manhattan/Bronx/Staten Island): 8/92 sites
- Brooklyn Public Library: 8/66 sites
- Queens Public Library: 2/66
(Yes, there are three separate public library systems for New York City. They pre-date the consolidation of the city and no matter how hard folks have tried every study on consolidating the three systems into a single organization, winds up costing significantly more than the current status quo.)
That said, I’m glad to see Webmention adoption. It’s got a much clearer purpose than FOAF (it might’ve been a little too expressive) and fits nicely into the current web ecosystem.
Totally agreed, though where real-time data is being put through an analytics lens is where CDW's start to creak and get costly. In my experience, these real-time uses shift the burden from being about human-decision-makers to automated decision-making and it becomes more a part of the product. And that's cool, but it gets costly, fast.
It also makes perfect sense to fake-it-til-you-make-it for real-time use cases on an existing Cloud Data Warehouse/dbt style _modern data stack_ if your data team's already using it for the rest of their data platform; after all they already know it and it's allowed that team to scale.
But a huge part of the challenge is that once you've made it, the alternative for a data-intensive use case is a bespoke microservice or a streaming pipeline, often in a language or on a platform that's foreign to the existing data team who's built the thing. If most of your code is dbt sql and airflow jobs, working with Kafka and streaming spark is pretty foreign (not to mention entirely outside of the observability infrastructure your team already has in place). Now we've got rewrites across languages/platforms, and leave teams with the cognitive overhead of multiple architectures & toolchains (and split focus). The alternative would be having a separate team to hand off real-time systems to and only that's if the company can afford to have that many engineers. Might as well just allocate that spend to your cloud budget and let the existing data team run up a crazy bill on Snowflake or BigQuery as long as it's less than the cost of a new engineering team.
------
There's something incredible about the ruthless efficiency of sql data platforms that allows data teams to scale the number of components/engineer. Once you have a Modern-Data-Stack system in place, the marginal cost of new pipelines or transformations is negligible (and they build atop one another). That platform-enabled compounding effect doesn't really occur with data-intensive microservices/streaming pipelines and means only the biggest business-critical applications (or skunk works shadow projects) will get the data-intensive applications[1] treatment, and business stakeholders will be hesitant to greenlight it.
I think Materialize is trying to build that Modern-Data-Stack type platform for real-time use cases: one that doesn't come with the cognitive cost of a completely separate architecture or the divide of completely separate teams and tools. If I already had a go-to system in place for streaming data that could be prototyped with the data warehouse, then shifted over to a streaming platform, the same teams could manage it and we'd actually get that cumulative compounding effect. Not to mention it becomes a lot easier to then justify using a real-time application the next time.
[1]: https://martin.kleppmann.com/2014/10/16/real-time-data-produ...
I also fondly remember how wonderful the web development deployment workflow was in Coda. This was back before source control was ubiquitous and CI/CD was required for any real production environment. It was the cleanest way I'd seen to go from saving in your test editor and having a clear path to SFTP it to the server. I half recoil in horror and am still in awe of how considered their experience was.
The real-time use-case has a nice wow factor to it; I've used it as a way to pair program for data science problems. The input cells sync in real-time (a la Google Docs), and so too do the output cells when one person runs a cell. And it's nice to be able to leave comment threads on a cell that can be resolved as a form of peer review.
But what made Colab a game-changer for me is how it let me seamlessly put my notebooks and a VM into Google Drive, making anything I put in a notebook accessible to anyone within my organization without needing to set up an environment, be it shared or local.
My last organization was a small rare disease research foundation, and I primarily worked on the fundraising side of the house; it was not a technical organization. When thinking about the longevity of my work, I realized that even the one person managing IT for them probably couldn't set up, let alone justify maintaining a networked Jupyter environment. So rather than ask for that and store all my analyses and small utilities on GitHub, I built everything on top of Google Drive and Collab. Folks were used to using Drive for everything else, so it meant my work was adjacent and discoverable to the team it was pertinent to and they could get access to both the outcomes of prior runs or change a few variables and run it again without me being needed. I left recently and I've still heard from a few former colleagues that they're still using many of the these notebooks and discovering others I'd built on their own time.
For a small data analysis operation in a Google Apps organization, Colaboratory is a godsend.
It'd be great to see this fed into the OSM Tasking Manager [1] or Maproulette[2] for folks to validate (and then build even stronger training data).
[1]: http://tasks.openstreetmap.us/ [2]: http://maproulette.org/
a. how the internet archive is building its political archive
b. proving to other cultural heritage institutions that they can get their act together and preserve culture today and make it relevant today
Ok, so Ted Cruz ads are a particular part of culture, but bravo.
I used to work at The New York Public Library's Labs and we had to fight to do anything with anything that wasn't blatantly in the public domain. Meanwhile, one of my colleagues there, Leonard Richardson, would go off and create massive archives like the Minecraft Geological Survey (http://mcgs.crummy.com/#mcgs) in the evenings.
Seriously, we need more folks working in these big organizations that are very good at surviving for more than a hundred years, acting as arks of human culture, that can collect and make useful today's materials. It's just our ability to remember today at stake.
I'm pretty excited that this is the first time someone has been able to release into the open an address analysis engine like underpins Google's geocoder. You'd be surprised just how hand-tuned most of the other address parsing engines are (regex and case statements all the way down). This feels like a huge leap forward.