Hacker News Reader

Comment by skohan | Hacker News Reader

> They thought they needed to build a machine learning model when they really needed to build an entirely new organization, one that possessed the technical and cultural mindset necessary to succeed in this space.

I totally agree. It's not impossible to imagine their model working: why couldn't you serve as a market-maker for homes at a large scale, especially with the unique insights Zillow could have based on their datasets.

However I think where the hubris lay is in how they thought they could leapfrog all the way to an automated solution before building a competency as a house-flipping company.

From what I understand, where they failed was partly in building a rich enough model to properly account for the less easily quantifiable elements which ultimately account for a property's value. I.e. the price per square foot might make a property look like a steal, while something like a sewer main nearby, or problematic neighbor could radically change the value proposition to anyone standing at the site. That's a non-trivial problem to solve for even the best ML and it's not clear how you would automate this.

If you ask me, instead of focusing on building an automated price discovery system, they should have started by trying to build a quality home-flipping organization, and figuring out how to super-charge manual work using their datasets. Over time you might find ways to optimize the process and increase the level of automation to scale output relative to head-count.

View on HN

I don’t think it’s just that they had a poor model, but the combination of that and adverse selection.

If you pledge to purchase at the Zestimate then people who reasonably think they can get more than the Zestimate on the open market don’t have an incentive to sell their house to Zillow (besides convenience). But people who think the Zestimate is an over estimate will of course sell to Zillow. So instead of a normal distribution of actual value:estimated value you end up with a skew towards the end where the estimate is over the actual value.

Trading housing is very different from normal market making because houses are not fungible commodities like most securities are. For most entities trading securities at low frequency it does not really matter whether a market maker skims off a few pennies on their trade; it’s worth it for the liquidity. Houses are less liquid (because they are non fungible) so the liquidity is more valuable, but the price improvement routing around a MM can also be many percentage points of a trade because there are not only so many factors affecting their valuation, but also just chance and random noise (bidding war, a particular buyer falling in love with the property, not-price-conscious buyers).

>adverse selection

According to Matt Levine's recent column, while you might think that, it wasn't what sunk them in practice. Bidding low in fact worked; it just was inherently limited in scale, which is why they switched to bidding higher. Unfortunately, being wrong in the other direction is very bad.

"I know, I know, the traders are saying: “No, this is stupid, your algorithms will not be 100% precise, some of your ‘lowball’ bids will in fact be too high, and those will be the ones that sellers accept. You’ll get adverse selection and end up losing money.” But that was not Zillow’s actual experience in the first quarter! The actual experience is presumably that some people accidentally got too-high bids, realized they were good and accepted them, but mostly Zillow sent too-low bids to everyone, and some people, for whatever irrational reason — market ignorance or financial necessity or laziness or whatever — accepted the too-low bids. The general point is that there is no reason at all to think that the people on the other side of these trades from Zillow are generally better informed than Zillow is. Sure they know more about their houses than Zillow does, but Zillow knows more about the market, and has more money"

"If you systematically bid too low, you will not do many trades, but you will make a lot of money on each trade. If you systematically bid too high, you will lose money on each trade, and also you will do a whole ton of trades. This is much worse!"

I wonder -- does it really matter if the previous homeowner is more informed than Zillow? For things like "annoying neighbor" or other hard to quantify/quickly detect annoyances, the buyer doesn't know about those things either, so I guess the information asymmetry is almost 100% in Zillow's favor, right?

no - because machine data of the deal is not complete, therefore cannot be represented in the models. As any computer-vision researcher knows, the code sometimes does not see what is "obvious" to almost any person.

The buyer might not realise about the annoying neighbour (unless the most annoying thing about the neighbour is the mess they leave everywhere) but will definitely pick up on things that Zillow's algorithm doesn't.

This is one of the things that's still useful about agents. They want to keep doing deals with each other, so the selling agent is motivated to pass on information like this.

Maybe that's been your experience. Mine has been that they're eager to cover up issues and explain away those they can't. I mean no hate, that's their job.

It has been my experience. My agent took a particular likening to me for whatever reason, probably because I'm a very low drama client, and took me around on the open houses that are mostly intended for agents to connect. In my town these happen on Wednesday morning. Selling agent usually has some basic amenities out like a coffee station, pastries, occasionally pizza. A large number of agents just cruise through continuously. It's something of a festive atmosphere. I was usually the only client vs agent in these situations, and while all agents have a degree of salesmanship to what they do, I saw first hand very frank discussions of potential problems that would cause a deal to feel sour after it was done.

As an agent you make your money on the overall volume, not squeezing the last bit out of any particular deal. Near as I can tell getting a reputation for deception is nearly career suicide unless you're at the very low end of the market. I think you're making a fundamental mistake in thinking about incentives in the context of a single sale vs having a career in a business that is very strongly driven by personal networking. This is a common fallacy in microeconomic thinking.

My observation where I live is that there are somewhat exclusive networks and not everybody is in the same one -- they might be clustered around catering to a clientele speaking a particular foreign language, for instance, and be somewhat apart from more "mainstream" agents. Anyway, I personally watched the selling agent try and convince me and my agent that the water damage in a property I was looking at was hardly damage at all and anyway, it doesn't rain in California, so don't worry about it. Sure, that's just an anecdote. So is your story, though.

I don't doubt that two agents who had frequently worked with each other amicably might be more frank, but I do doubt that agents all, or even mostly, presumptively do this with every counterparty they meet.

If you bid low, almost no one will take the opposite side of the deal, so your overall deal flow is low and total profit is low (even if margins are high).

If you ‘open up’ the flood gates on the other end, then yes you’ll do a lot of deal flow - Buyers sense a sucker - and open up a lot of opportunities for matches. It just so happens you’re also losing your shirt.

It’s easy to ‘make money’ (close deals) by giving money to people, and losing money in the actual business.

I think the question of fungibility comes into play here, too. If I’m a HFT and I accidentally post a too-high bid for Anacott Steel then there are well-capitalized players in a position to sell me a whole lot of Anacott until I lower the bid. (They may even be other HFTs who can naked short it to me.) But if I’m an iBuyer and post a too-high bid for 742 Evergreen Terrace, only the Simpson family can hit that bid, and only the one time. If I’m systematically overbidding, then that’s bad, but not every counterparty is informed enough to take advantage (or willing to stomach the considerable transaction costs), and there’s not a well-capitalized player to step in and arbitrage away the difference.

But what if iBuyer calls the Wallstreet Chronicle and tells them that Blue Horseshoe is buying 742 Evergreen Terrace?

I think a useful analogy here is trade-ins. Everybody knows that a private sale gets you a better price for a car than selling to a dealer. But a lot of people don't want the headaches that come with one. Why wouldn't it be the same for housing, especially when people may have life circumstances that mean they need to sell ASAP?

Because of shear dollar amount. Let's say on trade, you can sell your car to a dealership for $8000. Privately, maybe you can get $10000 or 10,500. Now, of course, $2,500 is not nothing but for some people, that is a trade they're willing to make to simply get the car off of their hands rather than go thru with a full-scale private sale process.

On the other hand, for a house that you could sell "instantly" for, say, $450,000, but that you could potentially get by selling privately for $480,000 or $500,000, that is now leaving 10 times more dollars on the table. Proportionally, the difference between the car and the house might be similar but in absolute dollars, it's a huge difference.

I disagree for a few reasons:

1. Your numbers are somewhat arbitrarily chosen... a car could easily be worth $40k and in some markets a home could be worth $200k.

2. Precisely because the house is worth so much, most people cannot afford to pay two mortgages, or their mortgage and rent on a similar home, at the same time, so if they must move by a certain date, they're under pressure to sell.

3. If you decide to sell to Opendoor you can pretty sure that the sale is going to go through; no worries about financing, for instance.

4. It's true that the numbers are larger in absolute terms, but people are often not rational in this way. Plenty of sellers leave tens of thousands on the table for somewhat frivolous reasons.

5. There is a lot of fat as far as people skimming off the top of transactions -- brokers, title companies, etc. IBuyers can get much better rates on these services by being bulk buyers, meaning they don't actually have to come that far off of the next-best offer to make a profit.

I don't think Levine successfully disputes the importance of adverse selection here. Sure, if you make a million low-ball offers, a handful will be accepted for God knows what reason. That doesn't mean that adverse selection isn't a serious danger when you attempt to scale up, as Zillow did.

I mean, why did Zillow go straight from making profitable purchases of few houses to losing money on tons? If there's some sweet spot in the middle that would have allowed them to scale up profitably, would it really be that hard to land on that spot?

I think it's obvious that homeowners know much more about the houses and neighborhoods they live in than Zillow, and this creates a big risk for Zillow.

I think he was saying that while no doubt it occurs on a case by case basis, it appears to be relatively insignificant either way.

When they were paying less than market price, they still made a lot per deal, and when they were paying more than market price, adverse selection hardly pushed them over the edge.

If you make "a million high ball offers" then the fact that some of them are particularly bad deals isn't the core problem.

I'm not 100% confident of this but it seems like the picture that is being painted.

And also, that it is particularly hard to hit the spot in between - I don't know if he's correct about that.

I won't say he's definitely wrong but it seems odd to argue that Zillow has so much market knowledge and money as to render adverse selection irrelevant, and yet they just couldn't resist making tons of unprofitable offers out of sheer impatience to scale up. What happened to all that market knowledge? There's got to be more to it than this.

This is where we get into internal incentives and theory of a firm.

My first project at a FANG made something like $42M for the company its first year, about $100M in total. There were 3 engineers working on it, and it cost the company maybe $500K in total. Great return on investment, right?

Except my Director wasn't in charge of making money, his job was to increase user happiness, typically measured as the proportion of interactions that were "successful". And this project didn't do this - since it encouraged browsing behavior (poking around without a goal), it actually decreased "successful" interactions. So the project was canceled and threatened with unlaunching about 18 months in.

I was like "Well can I buy it off you? $42M might be rounding error for you, but I'd love to have a business with a $42M ARR." But ultimately this was a no-go as well, because it used company infrastructure, user data, sale relationships, etc and negotiating that contract (along with all the legal and reputational risks to the parent company) would've cost a bunch more than $42M. Ultimately he was like "Well, if it's making that much money, maybe it should be transferred over to department that's actually in charge of making money", and that's where it landed for the next few years, until $100M became rounding error in the company's annual revenue and it wasn't worth that executive's time to sponsor it.

Projects have to move the needle for the executive that sponsors them, otherwise it's not worth their attention. Zillow made about $2.7B in 2019, with 47% gross margins. If Zillow Offers was profitable but could only flip say 1000 houses/year at a profit of $50K/house, that's only $50M, basically 2% of their existing revenue. It just doesn't move the needle for the shareholders, which means it won't move the needle for the CEO, which means it's not worth his attention. They needed it to be a substantial fraction of sales in the U.S. - if it had made a profit of $100K on $3M homes/year, that's a $3B business, more than double Zillow's existing revenue, at potentially higher margins.

This doesn't sound right. 2% of the firm's revenue in its first year, combined with a plausible growth trajectory to 5x or 50x that, is great!

I don't doubt that your €42m project got shut down, and that your employer might well have been right to do it. But this particular Zillow project wasn't shut down because it didn't move the needle - if anything, it was shut down because it did.

It sounds like the problem with Zillow Offers is that it didn't have the plausible growth trajectory to 5x or 50x that. It could be a profitable $50M business or an unprofitable $1B business, but not a profitable $1B business. The earnings call shutting it down mentioned that the company wasn't getting the economies of scale that they thought they would be getting.

> I think it's obvious that homeowners know much more about the houses and neighborhoods they live in than Zillow, and this creates a big risk for Zillow.

I really question the idea that most people have any idea what their home is worth without relying on... Services like Zillow. Especially if they've lived there a while.

Most people don't know what their home is worth, and if you get two appraisers or realtors to give you an estimate they'll likely be quite different.

Similarly, if you think a house is worth $500k but you list it for $600k, you don't know if someone will decide they love that house and they're willing to pay that amount or not.

That's one reason Opendoor tends to list houses at a fair premium to what they paid, as sometimes someone decides it is worth it and they make 20%+ on homes like that.

Yeah, and by the same token, you don't know if the buyer offering the top amount is really going to pull through, giving an offer from OpenDoor a certain appeal.

Zillow just has Zillow. The homeowner has Zillow, local realtors, inspections, personal experience of the house...

The local Realtors are ultimately just going off sales data, and inspections rarely happen before an agreement is made in principle to buy.

> inspections rarely happen before an agreement is made in principle to buy

That depends on the market.

Where is that done? I've bought in two states and sold in one and typically the way it's done is you have a contingency in the home sale that lets you walk if you don't like what your inspector finds. A seller could theoretically do their own but there's little upside, since this just creates a record of any issue that turns up that you now can't claim ignorance of.

In Oakland at the moment the sellers typically get the inspection done, provide the report in the disclosure packet, and then expect an offer with inspection contingency waived.

On the downside, there are the obvious risks in the structure of the incentives - in principle these may be sufficiently mitigated by the need to maintain a good reputation among buyers' agents; in practice they may not be.

On the upside, having had the opportunity (and motivation) to read many tens of such reports meant that I was (I believe) much better at reading them - and had a sense of what was typical amongst the homes I was looking at - by the time I read through the report for the house I wound up buying, and there haven't been any surprises so far...

Came here to say the same thing, Matt Levine's article is at https://www.bloomberg.com/opinion/articles/2021-11-18/zillow...

Well, that makes sense, but wouldn't it sink Opendoor too then? They are doing fine, as far as I know.

Rabois talks a lot of noise about how well they're doing, but Opendoor is a miniscule fraction of the size of Zillow, and it's hard to know exactly how good Opendoor is doing anyway. We'll see as the years roll on with a lot of economic uncertainty coming up.

Isn't is also true that the original pricing algorithm was built for a very different purpose? It was useful for getting a ball park estimate of value, but it was hardly accurate in the underwriting sense (for the reasons you point out). The hubris of assuming that those prices were so accurate that Zillow was willing to buy at them sight unseen is mind blowing, particularly when one takes into account the adverse selection (if Zillow's estimate is above what I can get from in-person bidders, I am more likely to take it than if the error is in the other direction).

Yes, you might discover that your average price is accurate, which is just fine for a reporting site. But beneath that there could be some structure, for instance there might be blobs of houses that your model makes too cheap vs reality, and blobs that are too expensive. If those are identifiable, eg via some sort of local knowledge, you might find that people will sell you houses that you've marked too high, but you can't buy the ones that you've marked too low.

Totally. There’s a surly creep who lives on our street and the houses next to his are worth less because of that. But Zillow would never know that.

This “annoying neighbor” problem keeps getting cited as an example of something that affects housing prices that the automatic algorithms don’t know about, but does it really exist? What is the mechanism by which the potential buyers find out about the annoying neighbor? I’m highly skeptical. I’ve had annoying neighbors before and there is zero chance that any potential buyers knew about them when I sold.

Are people interviewing all the neighbors before making a house purchase? Are those neighbors not incentivized to withhold any negative information about the neighborhood because it affects their own property values?

Two covered cars in the driveway, bushes growing out halfway into the sidewalk, house in disrepair, taller than allowed fencing with Beware of Dog signs, it’s sometimes not hard to spot someone you don’t want to live next to.

Huh. Some variation of this house exists on any given block in my city except maybe the neighborhoods where the prices are well over $2 million. No HOAs. I guess my threshold for “annoying neighbor” is a bit different from yours.

Sure, these houses exist everywhere, but most people avoid buying houses next to them. If Zillow bought such a house they’d take a loss to unload it unless they baked it into their model.

I think I’m just used to a housing market (seattle) where that is such a tiny blip it couldn’t possibly affect the price. Maybe untrue elsewhere.

Edit: it also seems like a small thing to worry about at a point in time since neighbors change. Annoying neighbors move away, or a new annoying neighbor can buy the house next door at any time. I’m not paying extra for something that is completely out of my control and could change tomorrow. The potential for annoying neighbors is not a feature of a given house so much as a feature of all houses, in my mind. But this attitude is probably more prevalent in people who don’t live in neighborhoods governed by HOAs where certain “annoying” behaviors are regulated away.

I can't speak for everyone. But I definitely think that a lot of people are put off by buying a house next to a Beware of Dog guy even though they know that one could move in tomorrow. Also, in California, a lot of people are in neighborhoods who could never afford to move in but own the home and don't have to worry about increasing taxes.

Where I am all the houses sell for above asking, often before an open house. It seems hard to imagine a buyer would even have a chance to get scared off by the neighbors.

This neighbor has external signals of the kind of person he is.

Even if their ML model provided excellent predictions, another potential problem they may not have accounted for is adverse selection: the only takers may have been on houses whose bids were too high.

It seems very difficult for me to imagine that they didn't anticipate adverse selection and try to account for it. I mean, clearly something went wrong somewhere, but I feel like people are acting like they had never heard of adverse selection until now.

Makes me wonder if Zillow wasn't planning to "tune" the models, surely they wouldn't want to publicly publish what they think things are worth and then offer 5% less. I wonder whether accurate estimates or making money from flipping would have dictated their decision-making...

[W]hy couldn't you serve as a market-maker for homes at a large scale, especially with the unique insights Zillow could have based on their datasets.

Indeed, I believe this is what OpenDoor does. From The Economist article [1],

"They [OpenDoor] charge a fee for the services they provide: buying and selling homes immediately, with zero fuss. The quick in-and-out makes them more like marketmakers than property investors, who buy to hold.

...

"A former Zillow employee told Business Insider that management had been hellbent on catching up with Opendoor, the front-runner. In order to compete, the employee alleged, the company pushed to offer generous deals to potential clients. It called this “Project Ketchup”. Now it has its own fake blood on its hands."

[1] https://www.economist.com/finance-and-economics/2021/11/13/a...

> a market-maker for homes at a large scale…a house-flipping company

These are different things.

Archetypal market making involves simultaneously buying and selling an asset. Flipping involves buying, improving and later selling. One might be able to deal with the heterogeneity of houses by operating at scale. (Zillow attempted this.) One might also deal with the delay between buying and selling by hedging. (Zillow never seems to have thought about this.) But the improvement function makes what Zillow attempted fundamentally separate from market making.

They weren’t paid to provide liquidity. If anything, they paid a premium for scale and immediacy. They were a real estate operation masquerading as a tech outfit. WeWork in different stripes.

> Archetypal market making involves simultaneously buying and selling an asset

Does it? I worked for a few years for a market maker, and that's not what we did. Simultaneous buying and selling is what the arb guys did. We'd buy and sell with generally short hold times. Which makes sense to me given that the exchange has market makers to provide liquidity. If something can be simultaneously bought and sold, then the market-maker is unnecessary.

> that's not what we did

Archetypal, not predominant.

> Simultaneous buying and selling is what the arb guys did. We'd buy and sell with generally short hold times

The ideal market maker is arbitraging (and eliminating the arbitrage-able inefficiency). That’s why humans were replaced by faster-trading machines everywhere they could be. In most cases, the arbitrage is synthetic or approximate, e.g. hedging an options or swaps book. But a fundamental separation between speculating and marketing making is the latter does not take a view on the assets per se, and should not be betting on their future price movement.

No market maker always achieves the ideal. But they tend towards it. Zillow didn’t have that tendency. In fact, they erected fundamental obstacles between themselves and that ideal.

Sorry, what's your source for this archetype? I thought maybe the place I worked for was just weird, but I've just looked at a half-dozen sources and as far as I can tell, we were pretty typical.

> what's your source for this archetype?

I’d have to dig up the textbook sources, but the key bit is in the definition: market makers quote a two-sided market and make money from the spread [1], i.e. buying at the bid and selling at the offer. If it happens simultaneously, that’s ideal. Every second one is long or short, risk and cost are incurred. Market makers seek to minimise and manage these.

In practice, arbitrage is tough. So most market makers simulate simultaneity by hedging. For example, if longs are accumulating (e.g. due to specialist obligations) one might open shorts or buy positional puts or wing it by shorting SPYs.

An unhedged market maker is just day trading.

[1] https://www.investopedia.com/terms/m/marketmaker.asp#what-is...

> i.e. buying at the bid and selling at the offer.

Really they _quote_ simultaneously the bid and offer (although there will be times when they do only one or neither).

Saying they simultaneously buy/sell is wrong/confusing.

> Saying they simultaneously buy/sell is wrong/confusing

That wasn’t claimed. What was said is the archetype is simultaneity. That is 100% accurate for how the term “market maker” has been used, globally, since at least 1999. (Pre-GLB/LTCM and post-ECN, the term was used more broadly.)

Drift from simultaneity incurs cost and risk. Those costs and risks must be managed. If you aren’t thinking in those terms, you aren’t market making.

Zillow’s downfall mirrors that of the money-centre banks in securities dealing post-GLB leading up to the crisis. What does and does not constitute market making, which is risky but less so than leveraged day trading, was a huge area of policy concern. When non MMs think of themselves as market makers, there is a predictable set of risks they get downed by. Zillow, like so many others, fell prey to that misconception. (There is loose analogy in the ABS markets, where banks holding inventory of esoteric products, either badly hedged or hedged with a busted counterparty, got hosed.)

You can't garauntee your (bid/ask) resting orders are executed against in the same epsilonic time window, nor would you want to. No market making practioners would think in these terms.

> You can't garauntee your (bid/ask) resting orders are executed against in the same epsilonic time window, nor would you want to

Of course you can. This is the entire thesis with which HFT beat out old-school market makers in securities.

I think you maybe have some misunderstandings around the practicalities limit orders and market microstructure (not withstand some theoretical model of risk free market-making, which has broadly been superseded, if you care about the theory at all).

> you maybe have some misunderstandings around the practicalities limit orders and market microstructure

Perhaps. I haven’t been on a market-making desk for close to a decade now.

But to correct one misconception in your comment, market makers don’t commonly use limit orders. (You’d submit a quote and try to hit rebates.)

How are quotes different from limit orders? [Edit: apart from the originator, I mean.]

> But to correct one misconception in your comment, market makers don’t commonly use limit orders. (You’d submit a quote and try to hit rebates.)

They do in the asset classes I work with, YMMV

> Drift from simultaneity incurs cost and risk. Those costs and risks must be managed.

That’s the point of being a market maker. Managing those costs and risks well enough to make money from the spread.

If we look at archetypal on Wikipedia, we get:

> 1) a statement, pattern of behavior, prototype, "first" form, or a main model that other statements, patterns of behavior, and objects copy, emulate, or "merge" into. Informal synonyms frequently used for this definition include "standard example," "basic example," and the longer-form "archetypal example;" mathematical archetypes often appear as "canonical examples."

> 2) the Platonic concept of pure form, believed to embody the fundamental characteristics of a thing.

The confusion between you two seems (to me at least) to fit almost entirely within the difference between those two definition. If you are describing the ideal market maker as essentially performing arbitrage, that seems to fit the second definition pretty well, right?

Meanwhile if wpietri says that most of the work at his believed-to-be-typical example of a market maker was doing non-arbitrage stuff, that'd make sense, right? I guess in most places the main work would be managing the divergence from idealness.

That could be it. Except that if market-makers were ideal in that sense, they wouldn't need to exist. If a buyer and a seller simultaneously exist at a given price, they can just trade with one another. Market-makers are valuable to markets only when they provide liquidity through non-simultaneous buy/sell pairs.

I think it also leaves out that not every market maker wants to be flat instantly. The one I worked for, and at least some of our peers were sometimes happy to hold inventory for a bit when they thought the market would even out.

> one I worked for, and at least some of our peers were sometimes happy to hold inventory for a bit

May I ask if this was before or after GLB (1999, for the kids following at home)? (My experience as a derivative market maker was post GLB and post crisis.)

Managing a flat book is less exciting than placing directional bets. It’s also less profitable, at least when the bets go well. Add to that the complexity of market making, particularly when derivatives enter the equation, and you get the two decades–following the advent of ECNs, turbocharged by GLB and ending in the financial crisis—in which market making desks were proprietary traders first and liquidity providers second.

Also, if you buy a swap and I simultaneously sell stock, there is both a simultaneous trade and liquidity being provisioned in a value-adding way.

Theres a good video here from a british hedge fund manager on why zillows real estate market making doesn't make sense: https://youtu.be/eDc4saE5m9k

The main insights are that market makers hold assets for a short period of time making money on the spread between buyers and sellers offers. Zillow had to hold on to houses for a long time and was speculating that the houses would be worth more in the future which is not market making.

I never got WeWork because it looked like they onboarded all of a 10-year lease's duration risk and then hoped to make up the difference somehow?

> somehow?

By marking it up and/or appreciation. They buy it for $500/sqft and then rent it for $100/sqft. In that hypothetical the breakeven is 5 years, plus overhead.

If the occupancy doesn’t work out in their favor, they may still make it up in appreciation.

What’s a duration risk?

People forget that tech is able to automate workflows. You don’t often yield success when you attempt to automate and invent the workflows in parallel.

I would say the opposite is true. Dying companies are stuck in their own routines because they're trying to automate their poorly designed processes that require humans at multiple steps. Smart companies are designing newer, better processes that are enabled by tech.

Starting from scratch can be a huge advantage.

This is a statement I would have agreed with wholeheartedly 20 years ago, and that I disagree with wholeheartedly now.

I'd be curious to learn why. I've seen the pain of companies tricked into thinking robotic process automation to do their horrendous excel workflows is a good idea. I've seen the benefit of a decent python data engineer with a small AWS budget.

The techier folks definitely have a different set of problems but the speed at which hings get done is night and day. Companies with old school work patterns (which, in my personal experience, means dusty old banks) are terminally entrenched in their ways.

I think you’re both right.

Taking some hopelessly byzantine, spreadsheet-driven process and “automating” it by building a Rube Goldberg scripting framework around it is the kind of totally stupid automation that doesn’t work.

Actually getting down to surface level and understanding fundamentally what each of those humans is accomplishing via those spreadsheets, extracting that all the way back out to a domain model and process flow diagram, and then selectively replacing process steps, whole cloth, with tech designed to be an actual subservice with SLA targets, is the right way to do it.

Throwing the spreadsheets and/or humans out altogether and starting “from scratch” is so exceedingly and needlessly risky from an information loss and hubris point that, well, good luck, but you’re nearly certain to fail.

Personally, I would treat the GP's mindset of "inventing workflows" differently than your mindset of redesigning at "poorly designed processes".

Yes, a poorly designed process sucks but it works at some level. That means the rough flow of it is figured out. Yes, there are exceptions and complications and all kinds of odd things but it's fundamentally different. It's not "from scratch" as you're taking an existing working-but-broken process where you know the input, know the output, and rethinking everything in between.

In an "inventing" scenario, you have what you think should be the input, a notion of what the output should be, and you're trying to build towards that notion.. without the validation that you're thinking of it correctly.

The first is a harder social problem (aka getting people to change) while the second is a harder technical problem.

Ultimately you have to build within your sphere of competence. If you have a well-established but inefficient manual process, it may sometimes be the case that burning it down and replacing it with a tech-driven approach might be the best way forward.

But if you are trying to solve a novel problem, and the proposed solution involves "ML will magically predict the future", you'd better have a very good idea of exactly how the problems will be solved, or else you're probably better off starting with good old-fashioned human intelligence.

It’s more complex than this in practice because in a large organization you have a significant change management component to every process change, whereas automation of an existing process immediately frees up bandwidth even if the process isn’t great. I interview global executives for a living; I hear this every day and I fully believe it.

Ideally, you want to understand the whole process from beginning to end, including all the complex edge cases, before trying to automate it, then automate the whole shebang in one giant undertaking. You need a tremendous amount of high-level buy-in to pull this off, as people will have to wait and suffer with the old process until you're completely done building the new one.

What often ends up happening is a large manual processes is automated bit by bit, and you end up with the situation you describe: a poorly designed manual process painstakingly replicated in code. Full automation is often never actually achieved here.

The absolute worst thing to do, though, is to begin automating the thing without fully understanding it. It's putting rocket boosters on your self-driving car without first understanding the rules of the road.

This is a critical concept that appears to be poorly understood in at least the web development circles I run in.

It reminds me of one of my favorite Bill Gates quotes:

"The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency."

> a machine learning model

Not to mention that generally ML models are not useful for assessing risk. ML nearly always focuses almost exclusively on some point estimate rather than a distribution of what you believe about a value. The former case is all about expectation and the latter about variance. Correctly modeling variance is far more essential to risk modeling than expectation alone.

I recall talking to a startup that was attempting to model credit risk by building a binary classier for defaulting, and trying to figure out a way to use this to score people for credit (obviously they chose to ignore the fact that there is a huge industry with decades of experience in assessing consumer credit risk).

They focused exclusively on finding more advanced models to get better AUC without even realizing that that's not important. I mentioned that the most simplistic credit score model should at least model P(default|info) and then set the interest rate to - P(default|X)/(P(default|X)-1) to break even and they couldn't comprehend this basic reasoning. It was doubly hilarious since their population's base default rate was such that the solution to this equation was higher than the legal limit they could charge for interest.

In the early part of the current startup/tech boom there was a focus on "disruption", the idea that new ideas could easily dominate old ways of doing things. But for many industries, such as credit/lending and real estate, you should at least understand the basic principles of how these "old ways" work before trying to disrupt them.

> Not to mention that generally ML models are not useful for assessing risk. ML nearly always focuses almost exclusively on some point estimate rather than a distribution of what you believe about a value.

It is actually quite a common practice to design neural networks that output probability distributions.

That distribution is still a point estimate for a multinomial, not truly the distribution of your certainty in that estimate itself. This is essentially a generalization of logistic regression, which will of course give the probability of a binary outcome, but in order to understand the variance of your prediction itself you need to take into account the uncertainty around your parameters themselves.

This can be done for neural networks, through either bootsrap resampling of the training data or more formal bayesian neural networks, both of these are fairly computationally intensive and not typically done in practice.

I was going to say, that seems like an "easy" second step once you get your ML to output hard numbers -- tack on ranges and confidence intervals.

> I.e. the price per square foot might make a property look like a steal, while something like a sewer main nearby, or problematic neighbor

This is the real problem.

Even if they have the historical data for that exact house/unit, it won't help them in cases such as:

* That nice view of the woods out the window is now blocked by a massive radio antenna that was just built there

* The river running through the back yard is now heavily polluted by something up-stream

* The new neighbor across the street is a huge nuisance and says they will never move

* The house just had a mass-murder event in it

Just because something is now cheaper than "comps" at price/sq ft and other metrics doesn't mean it's comparable.

anectodal evidence, but yes, an apartment I lived in was 30k cheaper than everything in the house, and the estimate. So I was really happy to have caught a bargain. Then after moving in it turned out the neighbors were unbearable. The father had developed a DIY habit during covid and he would regularly put together furniture at 11PM. Then they had two monsters for kids. Would literally jump around uncontrollably for hours, and for some reason, through some defect or lack in the sound proofing, the sounds were even amplified. Every time they went down to the playground the kids were literally screaming at the top of their lungs the whole way down. I nearly lost my sanity and fortunately could sell it at at 2k loss in a year.

You could sue your seller for failure to disclose.

"i never really noticed"

Yes, you'd have to be prepare to counter that with e.g. testimony from neighbors, or friends who heard the previous owners complain. It's not a slam-dunk, but it is an actionable tort.

Also good luck getting positive ROI on that multi-year lawsuit over things that buyer can reasonably say ‘never bothered me really, I don’t know what they’re talking about - sounds like they’re just irritated at life’. Especially when you factor in all the legal fees and several years of hassle going through the courts.

None of these things are necessarily unusual in most neighborhoods. All at once is irritating to most people - but hard to objectively prove are a true nuisance in a legal sense.

Apart from that of course you can sue anybody for anything (with more or less success); why would that be the case here? I mean isnt if neighbours are annoying a subjective thing? Do you know of any court ruling which implies one has to disclose the state of the neighbours?

Yes, and even if something is super annoying, it might be legal. For example they listened to music until 2-3 in the morning a lot of cases. Thumping reggaeton. And while it for sure wasn't over legal limit by decibel, the bass made my bed shake.

It all depends. But in general you have a common law right to peaceful enjoyment of your property.

http://bryancrews.com/private-nuisance-right-peace-quiet/

The one noise ordinance I've read in full also said that, even if the decibel limit is not exceeded, an audible beat or bass can also be cited.

There is a famous (among first year law students) case[1] that seems relevant given the nature of the issue is one a buyer would not reasonably be able to ascertain on their own. One possible point of differentiation: ghosts are a permanent defect on the value of the property while loud children living next door would probably only torment the homeowner for a decade or so at most.

The opinion is famous not just for its unusual fact pattern but also because the Judge clearly had quite a lot of fun working in other-worldly puns and references while writing it.

[1]https://en.m.wikipedia.org/wiki/Stambovsky_v._Ackley

Despite quoting the ruling at length multiple times in the article, there's not a single link to the actual ruling on that page, so far as I can find. Never change, Wikipedia.

This seems to be the opinion: https://scholar.google.com/scholar_case?case=329068483649083...

You indeed can if there are nuisance neighbors and I believe that is considered a material fact in most states. However, most documents, I believe contain a Real Estate Transfer Disclosure Statement which would have had a line indicating a "yes" if there were nuisance neighbors and it would have been up to you to ask for more details.

This is the key insight. Systems like Zillow's model a dozen or so big factors that are easy to collect (square footage, exact location, nearby comps, sales history, etc.) -- and then treat the rest as minor random noise.

Minor? Usually.

Random? Not at all. A minor annoyance like a cracked driveway ($1,500 to fix) is also likely to be associated with older kitchen appliances, faulty water pressure, deteriorating deck; poorly seated windows, etc. And then, buying that house for what the algo tells you -- or even algo minus 3% -- isn't likely to be a happy choice. Its fair market price may be algo minus 10% or worse.

Also worth bearing in mind, the Realtor community is not going to make life easy for Zillow. Once it's known that Zillow is loading up on clunkers, buyers' agents are likely to tell their customers: There's a Zillow house on the market, too. It's probably got problems. I'd demand a full inspection and some indemnities if I were you.

Common flaw of market disruptors. They assume that the existing players will remain neutral and indifferent to their arrival. The real world tends to be much tougher.

Or a house full of cats. I had a 'cat lady' friend who struggled to sell her home because she had 13 cats. 13 'indoor' cats. Even at a great price the house would not sell. Enter the wonderful folks at Zillow that bought her house based purely on the numbers. Last I heard they still hadn't been able to move that house at any price.

Toxoplasmosis is a scary thing.

Not exactly. From what I read it’s almost completely benign in most humans.

Certainly it isn't a issue after they cats no longer live there and it's been cleaned to a reasonable standard. If it knocks 20 grand off the price of the house it's worth spending 2k to have everything deep cleaned.

Cat pee permanently stains flooring and is also extremely hard to get the smell out. 2k will not be nearly enough if there’s extensive cat damage. Wood floors turn black with it and must be replaced.

Yes, even after extensive renovations cat pee smell can persist and some people are bothered by that smell. Im one of those people but I do like cats and wouldn’t mind having cats if they wondered around the neighborhood rather than be inside only.

Killing small animals and birds.

There are billions of old and ill small animals and birds each year who would normally be taken care by various predators. Around humans pretty much only cats can do that important and necessary job. The popular Smithsonian study that people like to cite which states billions of killed by cats somehow fails to mention what share of killed are ill/old vs healthy. There is no surprise though in such a glaring deficiency of that study once one learns that that Smithsonian team has a researcher convicted for animal, specifically toward cats, torture.

Another aspect - rats, a human civilization companion, raid nests for eggs thus decimating birds population around humans. By controlling rats cats help to maintain birds population.

This is not true, see my reply to your sibling comment for a more detailed answer. Toxoplasmosis is highly correlated with severe mental disorders like schizophrenia.

Indoor cats don't magically get diseases. Just like an indoor pet bat isn't going to magically contract rabies. They might if you are letting them out to go roam the terrain. I'm really tired of these silly perpetuated mythologies.

How can an indoor cat ever clean themselves though? It doesn’t seem like most indoor cat owners bathe the cats, do they?

Cats lick themselves to get clean. They don't bathe even when they're outside.

Cats are not "clean" after they lick themselves, if anything this behavior is primarily responsible for the cultivation of a parasite lifecycle that has co-evolved to the cat's gut.

Cats, especially ones who share litter boxes, routinely get covered in potentially parasite-bearing feces via litter boxes and ingest it via grooming. This allows for easy additional spread/cross infection.

"Cats lick themselves, therefore they are clean" is a long-discredited but persistent myth.

The question was not "are cats clean," but "how do indoor cats clean themselves?"

Indoor cats clean themselves in the same way as outdoor cats, by licking themselves.

Bathing cats can give them infections if the water gets in their ears. Generally, they clean themselves in a freshly cleaned litter box, what's called a "sand bath" by rolling around. They also clean themselves by licking themselves. They do a very good job, I've been around cats my whole life and none of them smell except perchance the fat ones that cannot lick their butts and sometimes have a bit of a dirty butt.

I'm not sure if you asked the question in regard to disease -- but even indoor cat poop will not carry diseases. Perhaps E. coli or other bacteria already in the cat's gut (and our guts too), but toxoplasmosis is not going to pop up unless it's in the cat food (uncooked pork, etc.) or the cat already has it. It is another common mythology that somehow poop has "disease" in it. It has no more disease than we already possess...

>what's called a "sand bath" by rolling around.

I'm sorry, but is this a joke and I'm just missing it? I find it hard to believe that you are seriously suggesting a cat rolling around in a feces-covered litter box is "cleaning itself".

>freshly cleaned litter box

There is essentially no such thing as this. You would need to completely remove the litter and sanitize the box with bleach _each time your cat poops_. No one does this, they scoop out poop and put the contaminated scoop in a bucket or something. The next time the cat enters the "clean" box it is covered in it's own feces, and your scoop acts as a cross-media reinfection device. Litter boxes are, in practice, perpetually dirty things.

All it takes is one exposure, and your "indoor" cat can infect your remaining pets (and you).

This entire topic is is marred with silly nonsense like "your only risk is mishandling cat feces", which totally ignores things like:

1) The cat litter box is covered in microscopic parasite-containing cat feces

2) The cat tracks feces-covered litter out of the box into the living space, which further spreads the parasite. If you walk barefoot through your house, you will step on cat liter pieces eventually which are again covered in microscopic feces.

3) The cat steps in litter covered in cat feces from other cats, gets it on its paws, which it then licks, starting the infection cycle over again. Multiple cats can easily keep a perpetual infection cycle going.

4) Cats, even well trained ones, routinely climb on top of furniture/eating/cooking surface and coat them in their feces. My parents were shocked after installing an interior camera when they saw their "well-behaved cats that never do that" walk all over their dinner table, kitchen counters, open and go inside cabinets etc etc. They are smart enough to know when you are around or not, and to not act like this when you can see them.

5) Cats have sharp claws, and even well behaved ones can accidentally scratch you or scratch a toddler that gets too close etc. This will then directly introduce cat feces to your blood stream.

Once exposed, the initial symptoms are mild flu-like symptoms and not a huge deal (unless you are pregnant, in which you will likely experience a miscarriage. Pregnant women should not be exposed to cats, or be in a household that has indoor cats). The real issue is the long term cysts they leave behind in your brain, causing a latent inflammation and immune response that seemingly interacts with your dopamine system in poorly understood ways. For example, latent cyst-stage toxoplasmosis significantly increases your risk of developing schizophrenia.

Okay even accepting that Zillow made big unforced errors, that doesn't sound believable. Like, they don't make the offer conditional on someone looking at it in person for red flags?

As I understand, they were buying sight unseen.

This happens in hot real estate markets. If you don’t want to miss out or start a bidding war, you have to be the most frictionless buyer.

In my area, OpenDoor was buying sight unseen. All they required was a 10 minute zoom call where you walk them through the house. Zillow actually sent out someone to do a home inspection (although a bit more cursory than what a typical buyer would pay for). This may have varied by market though.

Walking through a house with a high quality N95 mask in a hurry you might not notice the cat pee smell - or chalk it up to there being 13 cats and once they're gone the smell will go away.

It's a pretty interesting discussion as I sold my house just this year, and was frequently watching Zillow trends and information.

Originally the estimate on Zillow said my house was 20% over the value I actually sold my house for just last month. I listed with a traditional realtor for a 5% commission, because when I looked up the service and other fees for Zillow sales, I found they included around 20% of cost for buying homes and closing within generally 10 days.

As I listed my house, and as I reduced price on it for it to gain attention, I noticed the zillow estimate also went down to always stay below my listed price. I believe the estimate that both Zillow and Redfin display prominently were purely based on what my list price was changed to last, not on any meaningful algorithm, which can be very harmful to sellers and buyers, because it makes the process a bit deceptive by nature. Luckily Zillow also displays the price history on homes, which apparently cannot be "gamed" as much as the "zestimate" can be. Another thing I noticed was that the view stats on my listing that zillow regularly provided changed, even after days passed, that was very concerning because stats of that kind aren't supposed to change... They indicate real interest in a property, that guide decisions for sellers to reduce price, and they also indicate what is truly a "hot home".

No matter what, there is always the "human factor" that can corrupt or even destroy any company, where realtors can game the process to maximize their own sales profit or positions, or where appraisers can inflate an estimate as a favor for a personal friend, even despite laws against doing so. In a bad economy, the lengths people will go to to suit their advantage are wild. This type of issue can never be properly addressed by any algorithm, and that's why trusting technology too much can so easily lead to failure in any setting.

Ultimately I am glad I did not sell to Zillow, because of all of the potential for hidden costs and because they manipulate the process even when you don't use their service, but I am not feeling sorry for them as a company... I felt the impact of their presence in the market whether I involved them or not, and that's a big problem when it comes to preserving the value of traditional investment and stable investment in a house that should be properly addressed by regulation.

>as I reduced price on it for it to gain attention, I noticed the zillow estimate also went down to always stay below my listed price. I believe the estimate that both Zillow and Redfin display prominently were purely based on what my list price was changed to last, not on any meaningful algorithm

The fact that a house is for sale at a given price, but has not sold after some time, is a strong signal that it's overpriced. The longer it's been sitting, the stronger that signal is. They'd be crazy not to include that data in the Zestimate.

Now, if it's extremely fast, eg they adjust the price down within a day or so, then it seems a little ridiculous. OTOH the Zestimate has always been a rough indicator at best.

When you reduce price on a house listed on Zillow and Redfin right now, it bumps it to the top of taxonomy-based cues on those sites because of the information update, which increases overall recommendations/listing promotion to potential buyers. It's a new step in getting a property sold introduced by technology dynamics. Price reductions in traditional real estate listings worked differently (You were not refreshed on MRIS). with everyone performing price reductions though, that can have mis-leading effect on economic indicators, and it can also create harmful price reduction "panic" in certain markets though, so this is going to be a burgeoning issue moving forward.

I am luckily both a web developer and knowledgeable about real estate, most people don't properly understand the dynamics that are impacted/introduced into the market by technology and algorithms... People assume the traditional real estate market rules are still in play primarily still, but technology has complicated everything... That's also why Zillow overbought homes, because people making critical decisions too often put "traditional pre-tech" real estate market concerns over considering modern impacts of IT to their decisions.

My house sold within 2.5 months overall, it was not on the market for a long time.

Yes!! You nailed it! I spent a long career in software and now work in real estate, and you are spot on that they are not a company who understands real estate well enough to be buying and selling it. There are plenty of bad real estate agents in the world, but the amount of know-how and connections that good ones have is exactly the encapsulated in the examples you gave - local, specific knowledge that a national/international player isn’t going to have and isn’t going to be able to scale without a whole lot of human investment… gee wiz, kinda like real estate firms.

I wish I had the data that I assume they have internally, because watching their actions I’m not convinced they understand what questions would actually be interesting to explore with ml.

> why couldn't you serve as a market-maker for homes at a large scale, especially with the unique insights Zillow could have based on their datasets.

Why would Zillow have unique insights? With the exception of Texas, I thought real estate sales information is public information in the US.

Can you not imagine how useful it is to know user data e.g. what neighborhoods receive the most clicks, what type of homes generate the most favorites, how long people view one listing vs. another, … that is unrelated to public MLS data?

Maybe, but I was under the impression that Redfin/Trulia/Realtor.com would have the same information.

Also, unless Zillow started imposing confidentiality agreements on their bids, then competing buyers would just have to bid $1 more without their dataset, right?

Zillow is the largest aggregator. They own Trulia. I can squint and see the thought process here by Zillow, though execution, as evident, did not go as planned.

I somehow missed they had bought Trulia way back when.

> Can you not imagine how useful it is to know user data e.g. what neighborhoods receive the most clicks, what type of homes generate the most favorites, how long people view one listing vs. another, … that is unrelated to public MLS data?

Click data is much less valuable that the recent sale price data available in MLS. Using 90s style dwell time and click counts would likely yeild a lot of very noisy data. False positives from people's browser reopening with 15 tabs looking at different houses. False positives from social and paid advertising boosting a particular home or neighborhood's numbers. False positives from enterprising real estate entrepreneurs doing everything they can to get the clicks up in areas they own property to drive up prices. Meanwhile, the recent sale prices tell you much more, with certainty and are very expensive to manipulate.

Surely Zillow was also using that data.

> Surely Zillow was also using that data.

One would hope.

Evidently not as useful as Zillow expected!

One example, User search behavior should contain several leading indicators.

My understanding is that they believed they had an advantage in terms of buyer intent information. Everyone can see who buys what, but Zillow has access to more information about how people shop for homes, and the events leading up to the actual sale.

I have given Zillow a lot of non-public information over years of searching.

How many people search on bedrooms but not bathrooms? When people search on both, what’s the pattern they use? If we highlight prices and BRs on the map does that give more clicks than just prices? How important are photos (times 50 different questions there)? How strong a signal is repeat views spaced over time? Saving a house to favorites? Sending a link to a friend? Clicking on comps in the neighborhood? Which comps do people zero in on (as evidenced by spending more time on the page)? How strong a signal is sending a message to the real estate agent on the listing? What areas of the country are seeing an uptick in search traffic? How long between claiming a house as an owner on the site, updating the information, and listing it for sale?

They are sitting on a (well-earned) treasure trove of data and it’s not unreasonable to think they could use that to be better informed than another buyer without that information.

Where they seem to have failed is in not augmenting that advantageous data with regular old boots-on-the-ground observations.

I think the opposite: there is not much valuable data, it is just noise.

It is very difficult to go from what users browse to what they actually buy. People very often say one thing, then do something completely different.

And sometimes they browse stuff just to make sure that their current decision is correct, so they will look at a lot of items they're not going to buy.

(oh, and everybody and their mother knows photos are important. No need for ML to find that out)

Do you think that A/B or multi-variate testing works in general?

In some cases A/B testing works very well and in other cases not at all.

So it is easy to test UI changes, but difficult to find out why people do what the do.

> However I think where the hubris lay is in how they thought they could leapfrog all the way to an automated solution before building a competency as a house-flipping company.

In my mind this is the problem with consultants who try to automate processes. It’s really difficult (maybe even impossible?) to successfully write a program to make a computer do $thing if you don’t understand the intricacies of how to do $thing manually.

Would investors pour their money if they were more conservative and said something along the lines of "look, this is a very complex subject driven by and for humans, we should hire a bunch of non-technical people with relevant industry experience and try to make some bucks of profit before going full scale engineering and AI"?

Theoretically investors should reward a realistic and well-reasoned business plan, and punish hand-wavy science fiction. The fact that this is largely not the case (cough metaverse cough) is probably an indicator about how frothy the market currently is.

Very much this! I can count on one hand the number of successful businesses I've seen that use machine learning to do some magic inferences humans could never do. I've seen hundreds where humans are pretty good at doing their job, but machines can do it faster, more consistently and cheaper.

One huge difficult-to-quantify risk is the public opinion risk of being a very high profile company that flips houses. If it had succeeded to actually push up housing prices considerably, the whole company could be destroyed in the court of public opinion and therefore probably would be destroyed legally, regulatorally, and legislatively as well.

Even if Zillow had an algorithm that was 100% accurate at predicting current house prices, the housing market is just incompatible with market making. A market maker isn’t exposed to changes in the price, they clip the ticket on providing liquidity regardless of price direction. Zillow may have been able to successfully speculate on house prices with an accurate model, but they would not be a market maker.

Houses trade slowly, so would sit on Zillows books for a long time (days/months). Market makers on the stock market can have assets sit on the books for under a second. Houses are not fungible, which extenuates the slow trade problem.

When we were looking for a house we rejected many because of the “wrong sort” of neighbor, ascertained entirely (and possibly erroneously) without meeting them. I doubt Zillow can model that with public data.

Orrr maybe someone in the org could've practiced some basic morality and compassion and refrained from further contributing to the housing shortage. Just because you can make money being a sociopath doesn't mean you should...

Reply on news.ycombinator.com