px1999

It's not that weird - Google isn't a search engine company, they're an advertising company.

I'm not just flogging a dead horse here - they don't care how good the search results for these types of queries are, they need to be just good enough to stop you from moving to search via Perplexity, Chatgpt, Claude etc. They also need to be as cheap as possible to generate.

Google makes money when you search for things to buy which is why they'll give you an ad-free experience on things that they never made much money off anyways (eg "why is the sky blue?") - they want you to be around when you search for something they can (eg "jewellery"). Note that they never give you AI slop when you're doing a product search.

Unfortunately people both expect answers phrased in response to their questions, and people have generally poor skills at picking up on authoritative-sounding bullshit. Also if they give you "an answer" and stop you from moving to another site with more detail (and possibly ads from someone else!) you're more likely to ask follow up questions on google.

Initially I thought this was a useless/dumb idea.

On reflection, I like it - it's weird. Old internet weird. Like someone with a couple hundred bucks and some time on their hands wants to do something they think is fun or funny... it feels human. I think we need more of this. Extra points for using a sketchy o365 form that looks like a scam.

My org now sends some portion of our requests to non-anthropic models because refusal has become common from Claude. The requests themselves aren't dangerous, we find that benign requests in biological science wind up being blocked semi-frequently.

If it gets worse in future releases, we'd likely step fully away towards more useful (for us) models even if they're less capable.

Very well said.

I think that "deciding what types of code can be reliably handed off to AI" might be missing from the list. It's orders of magnitude easier to nail 80% all the time than 100% all the time. I could see standalone products even developing in this space.

Tools exist to be an energy/effort multiplier, so it's pretty intuitive that increasing that multiplier will make it easier to get more done.

In practice it's pretty difficult to find the balance between yak shaving and piling in unnecessary manual labour by just trying to do the work with existing (possibly poorly fitting) tools.

If you're planning to stick with your current tools for a long time, each 1% improvement compounds massively over time, so that balance is probably much closer to yak shaving than most people might realise.

Imo there's a huge blind spot forming between 6 and 8 when talking to people and in reading posts by various agent evangelists - few people seem to be focussing on building "high quality" changes vs maximising throughput of low quality work items.

My (boring b2b/b2e) org has scripts that wrap a small handful of agent calls to handle/automate our workflow. These have been incredibly valuable.

We still 'yolo' into PRs, use agents to improve code quality, do initial checks via gating. We're trying to get docs working through the same approach. We see huge value in automating and lightweight orchestration of agents, but other parts of the whole system are the bottleneck, so theres no real point in running more than a couple of agents concurrently - claude could already build a low quality version our entire backlog in a week.

Is anyone exploring the (imo more practically useful today) space of using agents to put together better changes vs "more commits"?

My org has built internal tooling that approximates this. It's incredibly valuable from a manual test perspective though we haven't managed to get the agent part working well, app startup times (10+ min) make iterating hard.

Do you have customers who have faced/solved this problem? If so, how did they do it -- it seems like a killer on the approach?

This is really nice and a very original take. It feels good on mobile / other touch devices.

I'd love to see it feel a bit more polished on desktop (maybe I'll give that a shot if I find a bit of spare time!) - I could see a few simple things like adding up/down arrows to the picked item and wiring into up and down arrow presses going a long way to making it work really well there too.

Genuinely, thank you for sharing this, it's something different and interesting.

Following this logic, why write anything at all? Shakespeare's sonnets are arrangements of existing words that were possible before he wrote them. Every mathematical proof, novel, piece of journalism is simply a configuration of symbols that existed in the space of all possible configurations. The fact that something could be generated doesn't negate its value when it is generated for a specific purpose, context, and audience.

Consider this (possibly very bad) take:

RAG could largely be replaced with tool use to a search engine. You could keep some of the approach around indexing/embeddings/semantic search, but it just becomes another tool call to a separate system.

How would you feel about becoming an expert in something that is so in flux and might disappear? That might help give you your answer.

That said, there's a lot of comparatively low hanging fruit in LLM adjacent areas atm.

The Umbraco CMS was amazing during the time that it used and supported XSLT.

While it evaluated the xslt serverside it was a really neat and simple approach.

I expect it will wind up like search engines where you either submit urls for indexing/inclusion or wait for a crawl to pick your information up.

Until the tech catches up it will have a stifling effect on progress toward and adoption of new things (which imo is pretty common of new/immature tech, eg how culture has more generally kind of stagnated since the early 2000s)

Except value isnt polarised like that.

In a research context, it provides pointers, and keywords for further investigation. In a report-writing context it provides textual content.

Neither of these or the thousand other uses are worthless. Its when you expect working and complete work product that it's (subjectively, maybe) worthless but frankly aiming for that with current gen technology is a fool's errand.

devoir de désobéissance is _duty_ of disobedience.

If they choose to follow orders they know are illegal they can be personally liable.

AMD's offer was more than fair. Hotz was throwing a trantrum.

The business model doesn't matter.

I can write something with Microsoft tech and expect it with reasonable likelihood to work in 10 years (even their service-based stuff), but can't say the same about anything from Google.

That alone stops me/my org buying stuff from Google.

Imo the con is picking the metric that makes others look artificially bad when it doesn't seem to be all that different (at least on the surface)

> we use a stricter evaluation setting: a model is only considered to solve a question if it gets the answer right in four out of four attempts ("4/4 reliability"), not just one

This surely makes the other models post smaller numbers. I'd be curious how it stacks up if doing eg 1/1 attempt or 1/4 attempts.

Specifically within the last week, I have used Claude and Claude via cursor to:

- write some moderately complex powershell to perform a one-off process

- add typescript annotations to a random file in my org's codebase

- land a minor feature quickly in another codebase

- suggest libraries and write sample(ish) code to see what their rough use would look like to help choose between them for a future feature design

- provide text to fill out an extensive sales RFT spreadsheet based on notes and some RAG

- generat some very domain-specific realistic sounding test data (just naming)

- scaffold out some PowerPoint slides for a training session

There are likely others (LLMs have helped with research and in my personal life too)

All of these are things that I could do (and probably do better) but I have a young baby at the moment and the situation means that my focus windows are small and I'm time poor. With this workflow I'm achieving more than I was when I had fully uninterrupted time.

This is great, but I wish there was a shorter and more to the point version for me to link folks to.

Each of the ideas in here is solid, but there's too much writing around the core idea -- a sentence or two for each point and then a tldr like "put in some basic level of effort if you're going to ask for others' valuable time." would do it for me personally.

The aftermarket for these things means that the cost winds up being split between multiple parties in a lot of cases.

Anecdotally, most parents within my circle bought their Snoo used and sold it after use. I bought an unopened snoo from facebook marketplace for $X and sold it after 6 months for $X-200.

I was a little annoyed that Happiest Baby is meddling with the resale value (because I was expecting to be able to sell it on after a few months of use)

IMO even though the product is overpriced, I'd have happily paid 5k for the extra sleep I believe it gave me.

My org uses codegen as a starting point for one of our test layers.

It works for us probably because we sidestep the pain points you list - the environments we run in are pristine complete copies of known datasets, we remove as many sources of randomness as possible, and our environment flakiness level is very low.

They still break but usually because the locators in use have been chosen poorly (or we've made planned changes to a page/component)

We're a web based b2b saas that runs an instance of the entire environment for each of our customers. Our non prod setup consists of a bajillion static test environments but more importantly we use testcontainers to spin up the transient test environments from database snapshots. Using the recorder on the static environments (before the transient ones existed) _was_ a pain

A level of fear allows the introduction of regulatory moats that protect the organisations who are currently building and deploying these models at scale.

"It's dangerous" is a beneficial lie for eg openai to push because they can afford any compliance/certification process that's introduced (hell, they'd probably be heavily involved in designing the process)

> Price discrimination is not illegal

The Play store operates in many jurisdictions, including some where this is borderline or could be deemed to be illegal

I hope that some of those jurisdictions start showing some teeth on this sort of anticompetitive behavior

But it's free and you didn't really answer the question.

Were you looking for a soapbox to stand on?

More interesting than the original statement is how many people seem to have a chip of their shoulder/take the statement as a personal insult.

The professor is correct in that the majority of Web developers could get by without much theoretical/academic background (and anecdotally, do get by without using those skills/knowledge much)

Maybe there's an industry-wide Dunning-Kruger effect?

Thank you for writing this, it was really informative and interesting (as someone with little ML background).

Manually, probably not (exceptional circumstances aside).

Recording generally means transcription and the ability to do automatic summarisation and to query action items, which _are_ useful

Also it means folks can be away or if someone didn't quite catch/understand something relevant to them they might want to rewatch some of the meeting.

Microsoft almost look to be building out a single copilot product across a bunch of their products (ie the eventual goal could be to share context between office365, Windows, sales, security, dynamics, Bing and github). If this were the case, it'd make sense to keep it separate

It also means they won't need to be the best for any one use, just "good enough" (and available enough) and they'll become the de facto ai for business (in the same way teams beat slack etc)

Many people underestimate how much easier it is as a business to throw money at things than to actually solve possibly hard problems. "Database failover and scale to z1d.12xlarge" as a strategy is insanely cheap in comparison to spending days/weeks/months trying to prevent spikes from impacting performance.

As much as folks here might not want to hear it, throwing an army of mediocre (or, ideally, decent) developers at a problem and paying through the nose for managed infrastructure often winds up being a much (much!) cheaper way to arrive at a good outcome than a smaller number of more skilled engineers with a shoestring infrastructure budget.

I love this, and wish other projects would follow suit and not be afraid to say "no" to people asking for changes/fixes.

The obligations that the industry places on open source project maintained by unpaid volunteers are unfair and unrealistic.

If someone actually cares about specific changes/PRs they should fork the library and implement or apply those appropriate patches/changes

A "no" is IMO infinitely more useful than an indefinite "maybe" (what most projects do, because they are afraid to say no to anyone) when it comes to these things.