The whole point of taking notes for me is to read a source critically, fit it in my mental model, and then document that. Then sometimes I look it up for the details. But for me the shaping of the mental model is what counts
Highly debatable whether it’s possible to create anything truly valuable (valuable for the owner of the product that is) with this approach, though. I’m not convinced that it will ever be possible to create valuable products from just a prompt and an agent harness. At that point, the product itself can be (re)created by anyone, product development has been commodified, and the only thing of value is tokens.
My hypothesis is that “do things that don’t scale”[0] will still apply well into the future, but the “things that don’t scale” will change.
All that said, I’ve finally started using Obsidian after setting up some skills for note taking, researching, linking, splitting, and restructuring the knowledge base. I’ve never been able to spend time on keeping it structured, but I now have a digital secretary that can do all of the work I’m too lazy to do. I can just jot down random thoughts and ideas, and the agent helps me structure it, ask follow-up questions, relate it to other ongoing work, and so on. I’m still putting in the work of reading sources and building a mental model, but I’m also getting high-quality notes almost for free.
I think your take is right. This isn't going to help with the internalization of knowledge that note taking will get you. I do think that there is some value in the way we've set up blueprints of agents if you haven't set up a business before to either teach about role functions in a business or get a head start on business that doesn't create something new. At the very least it's a quick setup to getting to experiment.
To the part about note taking (and disclosure) - we are working on a context graph product that lessens the work of reading sources, especially over time and breath to help with a lot of the structure you've mentioned.
I actually think that the harnesses which do end up building products, the harness will be the product.
As an example, I have a harness which I have my entire team use consistently. The harness is designed for one thing: to get the results I get with less nuanced understanding of why I get it.
Mind you, most of my team members are non-technical, or at least would be considered non-technical, two years ago.
These days, I spend most of my time fine-tuning the harness. What that gives me is a team which is producing at 5x their capacity from three months ago, and I get easier to review, more robust pull requests that I have more confidence in merging.
It's still a far cry from automating the entire process. I still think humans need to give the outcomes to even the harnesses to produce the results.
Say more?
Sure writing code was not the majority of my workday since I moved up in responsibility chain - but that's because I don't get enough uninterrupted time to do the actual coding from all the meetings, syncs, planning, production investigations, mentoring, team activities.
Now that I can delegate code writing to LLM instead of mid/junior devs the dynamics of those tasks change dramatically. The overhead of managing more junior devs is completely gone and no need to have soft skills with an LLM. And communication/iteration speed with LLM is not comparable.
Not to mention I don't need soo much time to get back "into the zone" - LLM can keep working through my meeting - when I'm back it's already working on something and I can quickly get back into the gist of it - much faster than before when I had to take a break and lost all context of what I was doing an hour later. LLM has all that context + more progress right there.
After soo many years of dev I have a pretty good idea of what I want the code to look like - no need to debate the overzealous mid dev about his over abstracted system that's going to haunt me in a production outage next month when we both forgot what he put in - I simply get to say "this is shit rewrite it how I requested". No need to "talk about latest lib that's all the rage on the YouTube blogs etc."
Sure sometimes I realize that doing stuff without LLM would have taken less time - but when I consider how many interruptions I have between my coding sessions - LLMs are empowering precisely because I get to dedicate so little time to my code.
Also less teammates means less communication overhead.
I think the sweet spot is human curation of these documents, but unsupervised management is never the answer, especially if you don’t consciously think about debt / drift in these.
I'd love to see other sources that seek to academically understand how LLM's use context, specifically ones using modern frontier models.
My takeaway from these CLAUDE.md/AGENTS.md efforts isn't that agents can't maintain any form of context at all, rather, that bloated CLAUDE.md files filled with data that agents can gather on the spot very quickly are counter-productive.
For information which cannot be gathered on the spot quickly, clearly (to me) context helps improve quality, and in my experience, having AI summarize some key information in a thread and write to a file, and organize that, has been helpful and useful.
Most things do not need to end up in your notes, and LLMs add too much noise, one that you likely never personally verify/filter out at all.
JA Westenberg made a good video essay about it a few days ago:
It circles back to the question, is this unimportant enough for me to delegate it to a LLM that might get it wrong? If the answer is yes, why even do it to begin with. If the answer is no, you have to do it manually.
I personally though, see value in this type of automation. Stuff like tag categorization, indexing, that otherwise would've been lost seems like a good fit for LLMs. Whether or not they're an ideal solution and something else like a search engine would've been a better fit, is a different question.
Storing my Health Insurance's Member ID, RxBin and other data. Recording the serial number of a product I will be calling technical support for. Organizing files to be more logical and deduplicating or consolidating as needed.
Whenever I want this info, I'll just ask my LLM to pull it up.
(that's one of my biggest hurdles for really adopting any useful assistant type of agent)
The other thing is I will ask an agent via Telegram to code stuff, so I want an agent that is smart enough to do it all. I prefer brute forcing with money right now. I hate when LLM make bizarre mistakes, I end up spending way too much time figuring out the issue.
I use Openrouter, so hopefully no one has built a perfect replica of me in their storage. I flip between models too.
But to be clear, I am living dangerously with agentic workflows in general. Haven't been burnt yet (other than accidentally running up a huge Gemini bill which made me switch to Codex Oauth and Openrouter for cheap Minimax 2.7)
I am moving to a commander/orchestrator model to use both frontier and cheap models and eventually a better local LLM once I buy a 5070 Ti, 3090, 64GB Mac M1 Max, 128GB Strix Halo (probably missed that train) or the AMD R9700.
it's neat, i can create a new sprite/whatever, point claude at the root, and tell it to setup zswap and it will know exactly how to do so in that environment. if something changes, and there's some fiddling to make it work, i can ask it to write a report and send it in to fold into the existing docs.
I've been coding like this lately: if I'm too lazy to review a new non-critical section/unit tests, I'll mark it as `// SLOP`; later, if I have to, I'll go through the entire thing, and unmark
shitty tests are better than no tests, as long as you your expectations are low enough