5,396 karma · joined January 21, 2012
meet.hn/city/us-New-York
Socials: - x.com/pradnelluru
---
It raises a funny "innovator's dilemma" that might happen. Where an incumbent has to serve chatty consumers, and therefore gets little technical/professional training data. And a more sober workplace chatbot provider is able to advance past the incumbent because they have better training data. Or maybe in a more subtle way, chatbot personas give you access to varying market segments, and varying data flywheels.
So why is not all information organized in structured, open formats? Because there's not enough of an incentive to label/structure your documents/data that way. That's if you even want to open your data to the public - paywalls fund business models.
There have been some smaller successes with semantic web, however. While a recipe site might not want to make it easy for everyone to scrape their recipes, people do want Twitter to generate a useful link preview from their sites' metadata. They do that with special tags Twitter recognizes, and other sites can use as well.
The good news is that LLMs can generate structured data from unstructured documents. It's not perfect, but has two advantages: it's cheaper than humans doing it manually, and you don't have to ask the author to do anything. The structuring can happen on the read side, not the write side - that's powerful. This means we could generate large corpuses of open data from previously-inaccessible opaque documents.
This massive conversion of unstructured to structured data has already been happening in private, with efforts like Google's internal Knowledge Graph. That project has probably seen billions in cumultative investment over the years.
What we need is open data orgs like Wikipedia pick up this mantle. They already have Wikidata, whose facts you can query with a graph querying language. The flag example in the article could be decomposed into motifs by an LLM and added to the flag's entry. And then you could use SPARQL to do the structured query. (And that structured query can be generated from LLMs, too!)
LLMs and structured data are friends.
Cyberpunk is certainly an achievement for technical graphics - fog, light, reflections etc. These are all enhanced with cutting-edge hardware.
What I'm trying to say is that we're quite far along in one visual axis, but for the whole experience to be amazing, we have a lot of work to do in other axes. Clunky character animations and body shapes take away from the amazing fidelity we get with light/fog, etc.
I've been playing Cyberpunk 2077, and while the graphics are great, it's clear they could do more in the visual realm. It doesn't use current gen hardware to the maximum, in every way, because they also targeted last-gen consoles. I'm thinking in particular of the PS5s incredibly fast IO engine with specialized decompression hardware. In a game like Rachet and Clank: A Rift Apart, that hardware is used to jump you through multiple worlds incredibly quickly, loading a miraculous amount of assets. In Cyberpunk, you still have to wait around in elevators, which seem like diegetic loading screens.
And also the general clunkiness of the animations, the way there's only like two or three body shapes that everyone conforms to - these things would go farther in creating a living/breathing world, in the visual realm.
In other realms, the way you can't talk to everyone or go into every building is a bit of a bummer.
GPT-5 finally on the horizon!
I'm not sure what the perfect solution is, but it is hard to sync a ton of shadow files to cloud storage, versus one big catalog file.
Is the metadata in an open format, so I can take the edits to other programs?
I am glad there's alternatives to having to shell out for Light Room every month. I only need to edit RAW files after holidays!
One thing though - if you do take the time to learn the original "perfect" versions of these things, it helps you become a much better system designer. I'm constantly worried about API design because it has such large and hard-to-change consequences.
On the other hand, we as an industry have also succeeded quite a bit! So many of our abstractions work really well.
But for a client, UI or otherwise, to make use of a dynamic set of URIs/verbs would require it to either look for a specific keyword (hard coding the intents it can satisfy) or be able to semantically understand the API (which is hard, requires a human).
Oddly, all this stuff is full circle with the AI stuff. The MCP protocol is designed to give AIs text-based descriptions of APIs, so they can reason about how to use them.
I think I finally understand what Fielding is getting at. His REST principles boil down to allowing dynamic discovery of verbs for entities that are typed only by their media types. There's a level of indirection to allow for dynamic discovery. And there's a level of abstraction in saying entities are generic media objects. These two conceptual leaps allow the REST API to be used in a more dynamic, generic way - with benefits at the API level that the other levels of the web stack has ("client decoupling, evolvability, dynamic interaction").
Being in the EU without using the Euro has been pretty good for Poland.
So fully implementing a perfect version of REST is usually not necessary for most types of problems users actually encounter.
What REST has given us is an industry-wide lingua franca. At the basic level, it's a basic understanding of how to map nouns/verbs to HTTP verbs and URLs. Users get to use the basic HTTP response codes. There's still a ton of design and subtlety to all this. Do you really get to do things that are technically allowed, but might break at a typical load balancer (returning bodies with certain error codes)? Is your returning 500 retriable in all cases, with what preferred backoff behavior?
"The authors observed a speedup of 1000 between [the commercial MILP solvers of] 2001 and 2020 (50 due to algorithms, 20 due to faster computers)."
I wonder if we can collect these speedup factors across computing subfields, decomposed by the contribution of algorithmic improvements, and faster computers.
In compilers, there's "Proebsting's Law": compiler advances double computing power every 18 years.
One avenue these phone OS, and any consumer OS, can pursue is making it easy to string together app sub-steps. An app can be "cracked open" into sub-step a) either by developers themselves b) or by a backend process at app submission time. The backend process could look at the screens, and see what the user journeys are - this is within reach for current LLMs. An "sub-step" here is like taking a flights app and turning it into different types of search functions - search by date, location, points etc. So an app becomes a bunch of interfaces.
Once you have these "sub-steps" a local LLM can string them together, because it can understand their inputs, outputs, and behaviors.
Actually executing the sub-steps would require the OS to execute the app in the background and run the sub-steps for the user. This would be akin to what browser agents do right now
So this is a way of semantically extracting the "verbs" in existing apps.
With a library of these "sub-steps" in apps, combined with similar ones extracted from the internet - you could chain together the web and native worlds, in the service of the user.
It's easier on phone OSs because phone apps are usually already logged in. It's realistic for iOS to just do a bunch of stuff for you in the background. You really can probably get finance info from all your finance apps, for example.
I'm not saying this is necessarily the answer - but re-thinking the OS in this sort of what is what would be an actually ambitious thing for Apple or Google to do. All these small tweaks are opiates.
1) Combining OLTP and OLAP databases into one system
2) Using an open data format to be able to read/write from many system (OLTP/PostGres, analytics engine/Spark)
> I'd argue the bigger value is keeping the data in one storage place and bringing the compute to it.
Yes, I agree with you. This observation is the idea behind #2, and why Iceberg has so much momentum now.
Fully using PostGres without awareness of Iceberg would require full decoupling, and a translation layer in between (Debezium, etc). That comes with its own problems.
So perhaps some intimacy between the PostGres and Iceberg schemas is a good thing - especially to support transparent schema evolution.
DuckLake and CrunchyBridge both support SQL queries on the backing Iceberg tables. That's a good option. But a big part of the value of Iceberg comes in being able to read using Spark, Flink, etc.
The current Iceberg architecture requires table reads to do so many small reads, of the files in the metadata tree.
The brand new DuckLake post makes all this clear.
https://duckdb.org/2025/05/27/ducklake.html
Still Iceberg will probably do just fine because every data warehousing vendor is adding support for it. Worse is better.