Post author here — just wanted to say I emphatically agree with this and have found your work on rr and Pernosco very inspiring! Anyone who hasn’t seen this work should check it out.
1,474 karma · joined February 2, 2012
One simple solution might be to only run the recording process on test data and never run in prod.
https://omar.website/posts/notes-from-dynamicland-geokit/
I also wrote a short explanation of some little experiments I did there when I visited a couple years ago:
While I agree that commercial incentives are critical to consider, two caveats I'd add:
1) I think the nature and depth of the incentives problem varies a lot by industry. Social media is a really complex case, for example. But I'm most interested in collaborative productivity tools, where I think many companies are okay being incentivized to build the best product rather than build data moats.
2) Cutthroat commercial incentives aren't the only thing that got us here. There are also tech barriers.
If I was starting a Google Docs competitor today, even if I wanted to make it open, I think it would be hard to pull off. What kind of API would I expose to enable realtime editing with good offline mode and conflict resolution? How would I deal with clients that have slightly different rich text representations than my own?
In an alternate universe where we already had a user-owned "web filesystem" with good answers to these questions, I could just hook up my client to that existing system and not even worry about persistence at all. It needs to be _easier_ for devs to build the right thing for users, not harder. And it needs to be a _better_ experience for end users, not a sacrifice. The convenient thing will win.
Sounds like I should add a button to disable the gradient.
The best place I've found is Ruben Verborgh's blog (he's a researcher who's affiliated with Solid). For example here's a nice post which goes into more detail on the ideas behind Solid:
https://ruben.verborgh.org/blog/2017/12/20/paradigm-shifts-f...
2) In terms of applications, I'd personally like to see more of a focus on productivity applications, rather than so much focus on social media. Better interop between cloud SaaS apps would be valuable to businesses and professionals, and would sidestep many thorny challenges like decentralized moderation.
As a concrete starting point: what if I could store my Google Docs text files on my own storage layer, and edit them in realtime using a variety of editor apps? This would resemble my ability to edit a .txt file on my computer in vim or TextEdit, but would port that metaphor to the world of modern online collaboration.
Here's a short Twitter thread I wrote on this topic:
https://twitter.com/geoffreylitt/status/1355255162626068482
3) I'm very curious to see more incremental paths from the current web to a decentralized approach. For example, what if we could start annotating existing websites with private data and sharing those annotations P2P, rather than starting over from scratch?
I've explored this a bit with my Wildcard project, where users can store annotations in a "spreadsheet" that is linked to a web page:
Twitter is one of my favorite places for developing ideas. Writing tweet threads [1] and having conversations with people on Twitter has helped me develop many of my opinions about computing.
The problem is that Twitter was designed for quick status updates, not developing ideas, so the product has some gaps. I started out building this browser extension to fix my own frustrations, and eventually enough people asked me about it that I decided to share it as a product.
One tip/reflection on launch strategy: I initially shared this on Twitter and had people DM me to get access. This was phenomenally successful for getting lots of early feedback—once there's a casual private messaging channel established, a user is way more likely to send feedback. I got meaningful feedback from the majority of people who tried the app. I finally made a website this week because the DM distribution strategy couldn't scale anymore.
Happy to answer questions here about the product, development, or distribution.
[1] more about threads: https://www.geoffreylitt.com/2020/09/21/twitter-and-media-fo...
I wrote a blog post about this idea of "Software as Curation", and how we might imagine more software being built in this way:
https://www.geoffreylitt.com/2020/07/19/tools-over-apps-for-...
We hesitated to mention database migrations too much in the essay for the reason you mention: in some sense, the whole point of Cambria is that you never do a one-time "migration", but rather you just continuously translate the data.
Still, using the system does feel pretty similar to doing database migrations, and we intentionally modeled some of the dev workflow after ActiveRecord migrations:
https://www.inkandswitch.com/cambria.html#developer-workflow
This is the best overview I've found of the basic history:
I'm a PhD student so I have a limited budget, but I don't mind the $100/year price point. It's become a key tool in my research workflow so it's totally worth it. I find it a bit odd that people think Netflix is worth more money than a tool that might transform your creative output.
Unfortunately, after reading this blog post I think I'm even more confused than I was before. Are Reason and Rescript separate projects? It seems like there are now _three_ different syntax options: OCaml, Reason, and ReScript? Even if it's a good change in the long run, the lack of clarity is concerning...
Seems like it may be more stable to just stick with OCaml syntax and js_of_ocaml [1] for now...
Also interesting that OCaml is not mentioned a single time on the new website [2]. I guess they're trying to appeal to JS devs and sell it as a new thing, but seems useful to at least be aware that there's a connection to an established language?
[1]: https://ocsigen.org/js_of_ocaml/3.7.0/manual/overview [2]: https://rescript-lang.org/docs/manual/latest/introduction
Joel Spolsky explaining how it works: https://youtu.be/0nbkaYsR94c?t=567
* professionalism vs individual creativity * the value of credentials * terrible interview practices * gender roles / biases * theory/academia vs pragmatism/industry
Recommend the book for getting some historical perspective on these topics.
But I have a major frustration with user scripts: writing them requires experience with Javascript and reverse engineering websites. This is fine for the HN crowd, but locks out most web users, who can't program at all.
I bet that if it were slightly easier to develop user scripts, there'd be 10x as many of them. I'm sure I'm not the only one who's helped a coworker write a bookmarklet / user script essential for their workplace sanity.
Would be curious what people's experiences have been helping nontechnical people extend websites, or if you know of tools in this area.
My current attempt at this is a project called Wildcard, which requires a programmer to write some site-specific scraping code, but then shows the scraped data to the end user in a spreadsheet and lets them decide what to do with it:
Here's a demo of a prototype live programming environment I made for jq, which similarly shows step-by-step views of the data but also gives live feedback as you construct your pipeline:
https://twitter.com/geoffreylitt/status/1161033775872118789
By the end of that Twitter thread I ultimately morphed it into a tool for building interactive GUIs (eg, get API data in JSON, use jq to morph it into the right shape for your UI, output to a HTML template).
(But even the draft version is a fantastic read!)
I think another useful analogy for thinking about abstract data representations is text streams in UNIX. It turns out many types of data can be represented as newline-delimited text, which enables you to use a suite of generic tools with that data. Inevitably, some data doesn't fit into that format, but it's perhaps surprising how much does.
Brittle site-specific logic is indeed a challenge. So far we've started with the simplest thing possible of programmers manually writing scraping code, so we can focus on how the system works once you have the data available. That has been enough to test the system out and build lots of useful modifications ourselves.
I think eventually some degree of automation will be an important way to help end users use this tool with any website. The "wrapper induction" problem has been well studied and there are lots of working solutions for end-user web scraping, so I expect to be able to integrate some of that work.
We're also interested in a community of shared scrapers, but as you point out there are security considerations. I'm considering trying central code review from the project to approve new site adapters and make sure they aren't doing anything obviously malicious. Another solution could be carefully restricting the expressivity of our scraping system (eg only specify CSS selectors, no arbitrary code) but I doubt that would be sufficient for all cases. Would appreciate any suggestions here.
So far, we've decided to defer thinking about that limitation, and first focus on other questions like getting the spreadsheet interactions right. We're making new site adapters every week and finding that we can build lots of useful modifications for ourselves which work even with only one page of a paginated list. For one example, see my demo of modifying HN front page [1], which I find useful even though it only loads the current front page articles.
At some point, we're considering adding more features around fetching subsequent pages of a table (as explored in Sifter [2], which sorts an entire list of search results across pagination boundaries) or scraping each detail page from a table (as explored in Helena [3]).
[1]: https://twitter.com/geoffreylitt/status/1229251217118892032
re: scraping, it's true that semantic HTML makes things easier, but we've also been building site adapters for a variety of modern sites that use frontend frameworks, "utility CSS", etc. Most promising solution so far is something I'm calling "AJAX scraping" -- observe the JSON request made by the client and just directly extract structured data from there.
To bring it full circle, here are a couple recent demo videos I made of using Wildcard to customize the HN front page:
1: https://twitter.com/geoffreylitt/status/1177607448682582016