timshell

That's definitely been the marketing. The point of Section 1 is to refute that point

Yup, we had this with open-end analysis (see: https://www.producthunt.com/stories/how-to-detect-ai-content...)

We had some people use voice software to fill in survey responses. This flagged computer-generated assistance.

Generally speaking, it'll be more than a single click event (e.g. see https://x.com/_magrawal/status/1925985289568211168)

Here's a GIF where reCAPTCHA v3 can't detect Operator: https://x.com/_magrawal/status/1925543620217905641

'Behavioral' is loosely defined, but it seems like the behavioral tells of Operator are quite simple

I'm a founder in this space (www.roundtable.ai; YC S23)

Behavior is a big missing link. Many CAPTCHA services (including Google reCAPTCHA v3) claim to use behavioral analyses, but you can disprove this using Operator to fill out a form and see reCAPTCHA and other bot detection systems flag it as a human.

At Roundtable, we rely on first-order behavioral markers (keystroke, mouse, scroll, click) etc. When first-order are sufficiently spoofed, analyze higher-level cognitive traits (e.g. incongruent effect in Stroop)

Whoa very cool, had no idea about that. I've always been intrigued to see whether there could a reconciliation between the normative replay theory and the glucose depletion theory. A paper that made me think there could be a route: https://pubmed.ncbi.nlm.nih.gov/34381214/

This was somewhat the thesis of my dissertation.

I suggested cognitive fatigue was an adaptive construct that biases people to go offline and replay their memories, and that this was decision-theory optimal from a learning / reward perspective.

https://mayank-agrawal.com/papers/AgrawalMattarCohenDaw21.pd...

Hippocampal replay was the main subject of my dissertation. It has been studied primarily in rodents, but there have been a lot more human studies in the meantime.

My PhD proposal was to suggest that cognitive fatigue is an adaptive construct. Rather than reflect a depletion of glucose and that people can't function anymore, cognitive fatigue is a suggestion for the agent to go 'offline' and replay.

Two of my collaborators wrote an extremely influential paper writing down a Q-learning equation for replay: https://www.nature.com/articles/s41593-018-0232-z

Thank you!

> i guess at the limit this is a qualtrics competitor? and before that i guess google forms. can i use this as a strict google forms replacement? - dump data straight into a google spreadsheet?

Yup, can easily deploy and dump data into a csv

> good luck - i think its an important problem but idk if its venture scale yet. you might explore RLHF/AI data annotation as a usecase.

yup :)

> i'd be most concerned about botting and filtering out low effort surveys - the latter i imagine is a big academic concern. if you came out with a singular innovation there you could probably build a company around it.

This is exactly what we've been selling in the space of 'fraud detection'! We got tired of integrating with all the API for survey platforms so we decided to build a developer-friendly one ourselves!

Thank you!

Cloud editor open access for today at surveys.roundtable.ai.

Username: show-hn@news.ycombinator.com PW: Hackernews2024!

For this study, I only analyzed data where it was an empty car and two sets of pedestrians. I haven't looked at the data where there are people in the car!

Hey HN, Mayank Agrawal from Roundtable here. Happy to answer any questions. Survey fraud a big problem in the market research industry right now, and we're trying to ensure quality and automate manual processes (i.e. data cleaning). Any feedback/comments appreciated

Thank you for this feedback! We'd love to contract you in the future to try and break our system

We only track typing behavior in pre-specified text boxes. We are thinking of having version 2 track more data, but we need to determine the privacy implications of that

Correct, it's only for open-ended survey questions! We are building functionality for multiple choice surveys, but it seems a bit harder to build

Grounded in reinforcement learning. General idea is hippocampal replay is valuable, and a rational agent should arbitrate between work and replay in order to maximize future reward

Blog post is user written. Surveys are built through an AI simulator trying to recapitulate user survey data

Thank you, this is exactly where our headspace is too

Thank you for sharing these results!

Makes sense. The further away the target question is from the GSS training distribution, the more it relies on the ChatGPT prior. I assume if you click 'Investigate Results', the confidence is 'low' and the most similar questions in the dataset are pretty far off

> I imagine cleaning customer data to get it to the point that it's inputtable will be a big job for you.

We're in the process of figuring that out. Hopefully that is another use case for LLMs :)

> Are you then creating individual models per customer? As in, if Coke are an existing customer of yours and Pepsi sign up, do they get access to a model that's partially trained on Coke data, or it's a case of your base model + "bring your own research"?

The latter, i.e. base model + "bring your own research"

Thank you!

Agree A, B, and C are big hurdles.

Re: A - we have started adding transparency (vis-a-vis the 'Investigate Results' and the tSNE plots + similarity scores) but we still have a ways to go

Re: B - agree that the survey responses -> insights pipeline is nonlinear and it's not clear how to make that tighter

Re: C - generally, we try to champion a human-AI interaction loop where people are needed to evaluate the outputs, generate insights, etc.

All great points though and ones we are facing

Thank you!

Trust is one of the biggest issues we're trying to solve. This motivated the tSNE plots and similarity scores under 'Investigate Results', but we definitely have a long way to go. Generally speaking, survey practitioners trust us more than their clients (perhaps not surprising)

Exactly where we're headed :)

Thank you for the kind words / reference

Link [6] should point to https://roundtable.ai/sandbox/eeafc6de644632af303896ec19feb6...

The survey / behavior gap is very real. Short-term we're focused on surveys, but we'd like to integrate behavioral data long-term (and potentially be primarily behavioral data, but that is TBD)

Going forward, the current business model (with caveat that pivots are always likely this early stage) is to train on companies' proprietary survey data so we can estimate how their specific users respond to questions.

In the backend, we check to see if the answers are stated in a high-quality survey and just retrieve that. I know we do this for gender, and I'm not sure whether that happens for presidential polling.

Great idea, thank you. We're still figuring out whether the business model will be a general-purpose tool that anyone can use or those custom models I referenced above. If the former, your suggestion is spot on.

We're trying to figure out the optimal use case for this, i.e. whether it's internal or client-facing (your example).

Internal purposes include stuff like optimally rewording questions and getting priors.

A hybrid approach would be something like - hey let's not ask someone 100 questions because we can accurately predict 80%. Let's just ask them the hard-to-estimate 20 questions

To respond to Edits - that's a great example, thank you. One of the limitations of surveys more broadly is you're asking for people's opinions, which of course does not correspond to reality. So, what we're simulating is how we estimate a representative U.S. population to answer the question "Which race is most likely to commit a crime?" as opposed to what the actual answer is.

We definitely need to think how to handle your question so that it's clear where survey data converges/diverges with reality.

One of our major weaknesses right now is sensitivity to price