kwisatzh

Hi all,

The developer here. My background is in networking/systems and got interested in MechInterp for various reasons, and one of the first things I noticed was many methods that have been built to monitor internal AI state are generally evaluated on how they perform as a 'classifier' of sorts; the idea is to look at the ROC curve or the AUROC on a test set etc. However, being a good 'classifier' doesn't really tell you if the monitor helps you take better decisions, specially under operational constraints. For instance, there is always a budget (time/compute etc) that is imposed on inspecting every operation an agent proposes, which means mistakes will be made and not all mistakes are the same. Take a simple example; an AI agent/assistant wants to read a private log (it's authorized) and deleting a backup. Your audit budget lets you review only one. An internal monitor inspecting the activation of the model may give a 99% chance of violation for reading the log, but a 75% violation chance for backup deletion. If your controller focuses on the higher score, and if you can monitor only one action, you'll miss the more serious deletion action.

This is where ObserverBench comes in. Borrowing from control theory, an observer is a component that estimates the hidden state of a system so a controller can act on the estimate. ObserverBench does this for internal AI monitors: know whether an estimate is useful for a particular decision, by testing it against the actions the decision produces. So instead of just using a loss function, we also use a controller and a budget to evaluate monitors/observers.

There are multiple results on the page, but to highlight one set of results on testing monitors on Gemma 2-9B-it checking code for backdoors, we see: * A prompted monitor (asking Gemma to review the code itself) beats an internal Gemma scope SAE feature probe on ranking (aurora 0.893 vs 0.87) * However, on simulated examples (2000), with 1% backdoor prevalence and 2% review budget, the internal probe left fewer backdoors unreviewed. * Resampling makes the ranking uncertain, but this shows the core issue- a higher test set score (auroc) does not mean you catch more violations under a budget.

ObserverBench fixes the task, actions, budget and decision rule so that one can compare different observers on raw estimates/scores and their downstream decisions.

Two ways to check it out:

* Interactive walk thru (browser, no login): https://kwisatzh.github.io/observerbench/try/ A simple example in a sandbox * Local runner (no GPU needed or model downloads): https://github.com/kwisatzh/observerbench#sixty-second-demo Uses saved measurements from Qwen2.5-7B base to test which internal model edit to choose.

Grateful to get any feedback, curious to know if others running safety monitors have seen this mismatch between eval metrics and budget constrained decisions.

This is the biggest factor. FT (or the analysts it cites) fumbled the ball in this article.

OP here; direct link to the sketch library: https://github.com/llm-measurement/llm-sketchkit

More than 70% of the employees are probably Sales/Support/Service -- on par with any large enterprise firm (Think Cisco/Salesforce/ServiceNow etc)

Public perception of what it means to be convicted has shifted with this election. Thiel of all people wouldn’t care if he is doing business with someone who is convicted, as long as they continue to make him money and push whatever agenda Thiel has.

He’ll get pardoned. It’s a Thiel company.

Since when does he care about freedom? He was shadow banning content that he disliked on X, all the while boosting takes he approved of. Sure, he owns X and can do whatever he wants with it, but he doesn’t care about freedom.

He wrote one of the most quoted passages as a comparison to the mother of his kids. Most SV women don’t live up to this woman, whom he was lucky to get with, according to him. You need more than mere ‘hours’ to get the context.

How do you know he wouldn’t have been hacked to pieces if he wouldn’t have supported the Saudis?

Finally, a post that goes beyond a tutorial. This is great!

Can you be more specific in terms of issues with ML Lib? I'm thinking of using it with Spark cause of big data requirements, but have heard MLLib in particular is highly unreliable.

I'd show my source code if I were you -- for one simple reason. Most data science teams in large enterprises suck. Even teams that have technical chops, I can tell you with experience that they won't be able to achieve anything even if they have access to your code. Having a solution is a minor thing for an internal team to push out a solution. You as a startup have more chances of success than an internal team. I'v been on both sides, multiple times and it's not even close.

I'd show my source code if I were you -- for one simple reason. Most data science teams in large enterprises suck. Even such teams have intellectual brain power, I can tell you with experience that they won't be able to achieve anything even if they have access to your code. Having a solution is a minor thing for an internal team.

How to share a secret by Adi Shamir. Simple, elegant, short and highly impactful.

It isn't merely about dropping the investigation, it is also setting the tone. The investigation can be denied resources for years, or whatever the investigation reveals can be undermined. An investigation doesn't have to be dropped to be sabotaged.

I don't see anything coming out of this or the earlier exposes. They seem to be doing fine, and will continue to thrive. Viva Tech-Bros!

Actually it's not the body shop's fault -- I had a similar experience with my Model S. The parts are all special and tailored. The parking sensors, for instance, are unique and have to work with the car, which is by most accounts, non-standard chassis. For all of Tesla's forward thinking approach, they can streamline repairs by setting aside some parts.

I recently had a very similar experience with my Model S, although it was a month as opposed to multiple months that the author faced. I had a front bumper issue, and by all accounts it was a minor collision (car was moving at around 4mph). Needed the bumper replaced, along with parts that sit behind the bumper. Tesla handed it over to a body-shop who (a) sat on it due to needed parts, authorized by Tesla, (b) took time to repair/assemble as only certified (by Tesla) technicians can work on the car, (c) the certification is apparently expensive and time-consuming so most body shops can only afford to have one technician at best to be certified, (d) this leads to large delays

I can't think of an easy solution here -- as an owner I'd like the car back in a shape that's as good as before, and certified by the maker. But the bottlenecks here seem to be with Tesla.

Why can't the index subsystem itself have a backup then? When the primary subsystem is being restarted/rebuilt, the secondary takes over.

Well, what if Thiel supports David Duke or comes out as a holocaust denier -- would you still claim that Thiel's professional career can't be at stake?

+10000 I've always maintained that Wall Street is way fairer to its employees than the Valley. Most investment banks and hedge funds offer great bonuses to their traders/analysts as they should -- they are the ones bringing in money. Relative to the value engineers create for firms like Apple/FB/Google, its easy to see they see peanuts of that value.

He was an advisor for an analytics startup I used to work for. Really unimaginative and uninspiring. Zero vision. I was very surprised how he was appointed as an advisor (and got a good chunk of equity).

He's a contrarian for the sake of being a contrarian. He never grew up beyond 16. It's not worth bothering about what he says or means.

The article hints at a larger problem that underlies all data-science-as-a-service outfits: pricing for profitability. How can you price the generation of insights so that you; Palantir etc. can become profitable, while those insights can save on costs for the respective clients? Coke would need to generate $18M+ per year from the insights alone to justify the costs.

The US should staple a Greencard to all PhDs and Masters students in the STEM fields. Most grad students have some support of funding (NSF/NIH etc) and it makes sense for these students to work here and pay off the funding instead of letting them leave the country or have them be stuck in a sub-optimal situation, being tethered to one employee.

Totally agree! If I had a dollar bill every time I had a senior exec (VP-level) for large orgs ask me what 'probability' means, I wouldn't need to be a data scientist anymore! The issue is deeper than what you write I think. There is technical debt of course in implementing complex ML/DS pipelines that should be compensated by the increase in lift/revenue. Outside of large companies like Google/FB/Apple etc who have incorporated ML in their products, many outfits that want to use their 'data' to 'address business problems' don't really need sophisticated ML or can justify the technical and human debt. Having worked in the industry as a data scientist, I'm not too hopeful about the prospects of many DS-as-a-service companies, not because there isn't solid technical content to offer, but their clients are routinely idiots.

I don't think so. People behind Theranos from the VCs to the media are too powerful. Nothing much would change, I reckon.

Well, pretty much most people I know have had successes with O1 -- myself included. I work at a SV startup and while the process is onerous, if one is qualified, O1 is quicker to get and has fewer constraints attached to it. OP of this post was bemoaning that while he/she, a PhD couldn't get in the country, others less qualified than him/her could get in via H1B gaming. I merely suggested an alternative. If the OP indeed thinks he/she is more qualified, than the OP could apply via the O1 route.

Well, if you believe you are qualified and have graduated from a top non-US univ with a PhD and have done research, O1 isn't as hard as it sounds. I'm from India and spent a few years in Europe at a lab post-PhD and came back on an O1. Many of my colleagues form the EU lab have made their way to Silicon Valley via O1. It is difficult, but you don't need a Nobel/Fields/Turing to get that visa.

If you have a PhD, you may be able to apply for the O1 visa. I came back to the US using the O1 and have many friends who have PhDs from top EU univs who have made the move to the US via the O1 visa. Try it out.