Hacker News Reader

Comment by rushingcreek | Hacker News Reader

I think this is very interesting, but it is reminiscent of what we built with Phind 2 where the answer could include dynamic, pre-built widgets.

The problem with this approach is precisely that these apps/widgets have hard-coded input and output schema. They can work quite well when the user asks something within the widget's capabilities, but the brittleness of this approach starts showing quickly in real-world use. What if you want to use more advanced filters with Zillow? Or perhaps cross-reference with StreetEasy? If those features aren't supported by the widget's hard-coded schema, you're out of luck as a user.

What I think it much more exciting is the ability to completely create generative UI answers on the fly. We'll have more to say on this soon from Phind (I'm the founder).

View on HN

Phind is awesome. I often forget to use it until legacy search engines fail to surface what I’m looking for after a dozen searches. Phind usually finds it.

That said, I used it a lot more a year ago. Lately I’ve been using regular LLMs since they’ve gotten better at searching.

Thanks for the feedback. I think that our main differentiator going forward will be this generative UI on the fly for answering questions as opposed to search alone.

In a similar boat, but have been increasingly returning to for its quick notebook/charting capabilities. Would be awesome to somehow be able to select between different UI modes offering search, ranking, graphing or else depending on user needs.

Given there is already a MCP-UI project, I’m not surprised it can be done. But even that I’m not very convinced that it’s the right approach. After all, it’s still far too slow for real usage…

Totally agree that it's too slow with conventional approaches, which is why we're training custom models for this that we can run fast

Ah, that’s interesting. I’m considering building something similar for our product, and my solution to the schema constraints you mentioned thus far is breaking my widgets into blocks as universal as possible, as to still be useful. All of this is just ideas yet mind you, but my thinking was—maybe I can get the model to pick from a range of composable widgets depending on the task that are interoperable?

For a concrete example, think a search result listing that can be broken down into a single result or a matrix to compare results, as well as a filter section. So you could ask for different facets of your current context, to iterate over a search session and interact with the results. Dunno, I’m still researching.

Have you written somewhere about your experience with Phind in this area?

Yes! We have a blog post here on how we designed these models and widgets: https://www.phind.com/blog/phind-2-model-creation.

Now that models have gotten much more capable, I'd suggest to give the executing model as much freedom with setting (and even determining) the schema as possible.

> If those features aren't supported by the widget's hard-coded schema, you're out of luck as a user.

Chat paired to the pre-built and on-demand widgets address this limitation.

For example, in the keynote demo, they showed how the chat interface lets you perform advanced filtering that pulls together information from multiple sources, like filtering only Zillow housers near a dog park.

> Chat paired to the pre-built and on-demand widgets address this limitation

The only place I can see this working is if the LLM is generating a rich UI on the fly. Otherwise, you're arguing that a text-based UX is going to beat flashy, colourful things.

Yes, because it seems that Zillow exposes those specific filters as a part of the input schema. As long as it's a part of the schema, then ChatGPT can generate a useful input to the widget. But my point is that is very brittle.

Isn't that as brittle as any system being constrained to providing only some type of outputs? Please elaborate.

A fully generative UI with on-the-fly schema would be less brittle because you can guarantee that the schema and the intelligent widget can fully satisfy the user’s request. The bottleneck here is the intelligence of the model computing this, but we are already at the point where this is not much of a problem and it will disappear as the models continue to improve.

I think most software will follow this trend and become generated on-demand over the next decade.

The problem is not the limitations of the capabilities per se but their discoverability (https://en.wikipedia.org/wiki/Discoverability). The user doesn't know what the capabilities are, as they are added and -- infuriatingly -- removed. Google Assistant is a perfect example of this.

Conservational user interfaces are opaque; they lack affordances. https://en.wikipedia.org/wiki/Affordance

Yep, this is a big problem as well. If the user doesn't know what features will or won't work, they lose confidence overall.

Thank you for this word. I have felt it my whole life and never learned the exact word.

I immediately knew the last generation of voice assistants was dead garbage when there was no way to know what it could do, they just expected you to try 100 things, until it worked randomly

They don't lack affordances, you can do stuff. They lack signifiers, ie it's not easy to discover the stuff you can do.

Affordance is not what it can do, it is what it signals that it can do. It needs to be perceptible, by the definition I use (Norman's). I see others go by different definitions that even admit hidden affordances. I do not.

From The Design of Everyday Things:

> Affordances represent the possibilities in the world for how an agent (a person, animal, or machine) can interact with something. Some affordances are perceivable, others are invisible. Signifiers are signals. Some signifiers are signs, labels, and drawings placed in the world, such as the signs labeled “push,” “pull,” or “exit” on doors, or arrows and diagrams indicating what is to be acted upon or in which direction to gesture, or other instructions. Some signifiers are simply the perceived affordances, such as the handle of a door or the physical structure of a switch. Note that some perceived affordances may not be real: they may look like doors or places to push, or an impediment to entry, when in fact they are not.

With Norman's definition, if a conversational interface can perform an action, it affords that action. The fact that you don't know that it affords that action means there's a lack of a signifier.

As you say, this is a matter of definition, I'm just commenting on Norman's specific definition from the book.

Voice interfaces actually remind me a lot of command-line interfaces: If you know the a working "rune" on the tip of your tongue (e.g., "Set a timer for 10 mintues", "Play <exact title rune that gets the song you want>") it's great. But as you say, it's not always that easy to figure out new "runes". LLMs should be somewhat better for that, though.

The LLM is phenomenal at figuring out what you want, but it still has to map it to the schema of the tool. So while the job of figuring out the working “rune” is offloaded from you to the LLM, it doesn’t solve the fundamental problem of the available “runes” likely being brittle and insufficient for any given task even when the LLM knows exactly what you want to do.

That’s solved by MCP though. You can update your MCP’s servers schema dynamically without ever having to touch the app itself but the app will be aware of the new schema.

I'm not saying that the schema can't change from time to time, I'm saying that having any fixed schema at request time is not an ideal user experience because it may not be clear what is supported and what is not supported. From first principles, it's much better if the app schema can be created dynamically at request time so we can guarantee that we can fully serve the user's request exactly as they asked it.

I know that AG-UI from copilot kit is in this space. But it hasn't worked well with the MCP model AFAIK

Reply on news.ycombinator.com