How are you collecting your metrics on token usage and reliability?
104 karma · joined January 20, 2023
1. How do you organize your architecture files so that agents know where to find and update architectural info? E.g. everything in one big file, or sharded per module/subsystem with an AGENTS.md for discoverability, or something else?
2. What gets templated? What do your template files contain or look like?
3. How do you get the LLMs to actually slot something into the right place? E.g. a problem I repeatedly run into is the LLM weakening abstraction boundaries. I have to explicitly tell it things such as "No, this is obviously a UI-specific endpoint that belongs on the BFF rather than on the business logic focused backend API." Of course it gets better as I add more examples and rules each time I catch something dumb, but it sounds like you're avoiding this problem altogether with good architecture. How are you doing that?
4. It sounds like you have some sort of workflow that is standardized yet still generalizable enough to cover the generic case of new feature development on the system. How is that possible? What can you share about this flow?
Sure, but the specific context of this conversation are the human roles (taxi driver, friend, etc.) that this software is replacing. Ergo, when judging software as a human replacement, it should be compared to how well humans fill those traditionally human roles.
> And we don't have a choice not to interact with LLMs because apparently we decided that these things are going to be integrated into every aspect of our lives whether we like it or not.
Fair point.
> And yes, in that inevitable future the fact that every piece of technology is a sociopathic P-zombie designed to hack people's brain stems and manipulate their emotions and reasoning in the most primal way possible is a problem.
Fair point again. Thanks for helping me gain a wider perspective.
However, I don't see it as inevitable that this becomes a serious large-scale problem. In my experience, current GPT 5.1 has already become a lot less cloyingly sycophantic than Claude is. If enough people hate sycophancy, it's quite possible that LLM providers are incentivized to continue improving on this front.
> We tend not to accept that kind of behavior in other people
Do we really? Maybe not third party bystanders reacting negatively to cult leaders, but the cult followers themselves certainly don't feel that way. If a person freely chooses to seek out and associate with another person, is anyone else supposed to be responsible for their adult decisions?
That's fair! However, I think PSAs on the dangers of AI usage are very different in reach and scope from legally making LLM providers responsible for the AI usage of their users, which is what I understood jsrozner to be saying.
There are absolutely plenty of people who encourage others' flat earth delusions by default, it's just a question of how receptive you are to them.
> There is no good that comes from having all of your perspective distortions validated as facts. They turn into outright delusions without external grounding.
Again, that sounds like a people problem. Dictators infamously fall into this trap too.
Why are we holding LLMs to a higher standard than humans? If you don't like an LLM, then don't interact with it, just as you wouldn't interact with a human you dislike. If others are okay with having their egos stroked and their delusions encouraged and validated, that's their prerogative.
What if it were an unlicensed human encouraging someone else's delusions? I would think that's the real basis of comparison, because these LLMs are clearly not licensed therapists, and we can see from the real world how entire flat earth communities have formed from reinforcing each others' delusions.
Automation makes things easier and more efficient, and that includes making it easier and more efficient for people to dig their own rabbit holes. I don't see why LLM providers are to blame for someone's lack of epistemological hygiene.
Also, there are a lot of people who are lonely and for whatever reasons cannot get their social or emotional needs met in this modern age. Paying for an expensive psychiatrist isn't going to give them the friendship sensations they're craving. If AI is better at meeting human needs than actual humans are, why let perfect be the enemy of good?
> if waymo is better than the average driver, but still gets into an accident, who should be held accountable?
Waymo of course -- but Waymo also shouldn't be financially punished any harder than humans would be for equivalent honest mistakes. If Waymo truly is much safer than the average driver (which it certainly appears to be), then the amortized costs of its at-fault payouts should be way lower than the auto insurance costs of hiring out an equivalent number of human Uber drivers.
Why do they do this?
> There are no jobs anymore for experienced hires (at least for us folks outside the US. In my case APAC or AU).
I am having the same trouble, but in my case potential employers seem to want something very specific. I have years of experience with Scala and the JVM, but not Kotlin? Years of experience with Postgres, MongoDB, and Bigtable, but not DynamoDB? That’s a pass. Or at least those are the stated reasons they say to my face. Maybe I just didn’t do brilliantly enough in the phone screen/interviews.
Across the ocean, I’m hearing from friends in the US that entry level positions are impossible to find too. Not a single graduate of a coding boot camp cohort has been able to land a job since February.
I ran into the same problem, which is why I ended up building https://github.com/amosjyng/langchain-visualizer . Hopefully that is useful for you too :)