Marceltan

Nice, this was helpful for us internally. Good call on allowing importing of existing .claude/settings.json, makes my life easier on personal projects.

Am curious what the personalization looks like implementation-wise. Anything that makes it stand out compared to other wealth management platforms?

Initial setup takes <10 mins (including time spent testing that traces get recorded), we have a `tusk init` setup wizard to walk you through creating a config.

Thanks! Good question. Tusk Drift isn't quite designed for these use cases.

Currently, Drift is language specific. You'd need the SDK installed in your backend while recording tests. This is because Drift captures not just the HTTP request/response pairs, but also all underlying dependency calls (DB queries, Redis operations, etc.) to properly mock them during replay.

A use case we do support is refactors within the same language. You'd record traces in your current implementation, refactor your code, then replay those traces to catch regressions.

For cross-language rewrites or browser-exported requests, you might want to look at tools that focus purely on HTTP-level recording/replay like Postman Collections. Hope this helps!

We instrument JWT libraries directly (jsonwebtoken, jwks-rsa). Both `jwt.sign()` and `jwt.verify()` are captured during recording and replayed with the original results. During replay, you get back the recorded verification result. So if the token was valid during recording, it stays valid during replay, even if it would be expired "now". The test runs in the temporal context of when it was recorded.

Thanks for sharing this. :)

Also yes, appreciate you calling this out. The deviation classification after replay + automated RCA for unintended deviations is another differentiator. Let me know if you have feedback when you get time to explore.

Fair shout. Our instrumentations (https://github.com/Use-Tusk/drift-node-sdk?tab=readme-ov-fil...) hook directly into pg, mysql2, ioredis, firestore, etc., at the library level.

We capture the actual DB queries, Redis cache hits, JWT generation, and not just the HTTP calls (like you would see with mitmproxy), which lets us replay the full request chain without needing a live database or cache. This way each test runs idempotently.

Sounds good Chris, would love to hear your thoughts once you've played around with it.

Good questions. I'll respond one by one:

1. With our Cloud offering, Tusk Drift detects schema changes, then automatically re-records traces from new live traffic to replace the stale traces in the test suite. If using Drift purely locally though, you'd need to manually re-record traces for affected endpoints by hitting them in record mode to capture the updated behavior.

2. Our CLI tool includes built-in dynamic field rules that handle common non-deterministic values with standard UUID, timestamp, and date formats during response comparison. You can also configure custom matching rules in your `.tusk/config.yaml` to handle application-specific non-deterministic data.

3. Our classification workflow correlates deviations with your actual code changes in the PR/MR (including context from your PR/MR title and body). Classification is "fine-tuned" over time for each service based on past feedback on test results.

Agree. My biggest pain point with LLM code review tools is that they sometimes add 40 comments for a PR changing 100 lines of code. Gets noisy and hard to decipher what really matters.

Along the lines of verifiability, my take is that running a comprehensive suite of tests in CI/CD is going to be table stakes soon given that LLMs are only going to be contributing more and more code.

Re: "Can we reliably gauge appropriate moments for interaction?"

One design pattern for LLM tools that I'm very interested in exploring is having an LLM only engage when the user (human) is not in a flow state.

Take writing for example. There are times when text auto-complete is actually a welcome feature because you might be experiencing writer's block. An LLM tool providing proactive guidance then becomes a way to get unstuck. But if you're in flow and flying away at the keyboard, auto-complete becomes a distraction and can take you away from your train of thought.

There are obviously some signals that we could programmatically make use of in this scenario (words per minute, pauses between keystrokes, etc.) but I do think defining "flow" in those terms is nebulous. I can imagine a world where stuff like skin conductance tracking a la Oura Ring becomes part of the mix. Would be pretty cool.