How Bad Are Zillow “Zestimates”?
blog.duvora.com
blog.duvora.com
(1) Author discovers that Zillow has a specific claim about accuracy: "around 90% of the listing Zestimate is “Within 20% of Sale Price” of the home". Author then gripes that this is "a really large margin for error". Really? Because I was prepared to praise Zillow for such honest and clear reporting. It only costs a few hundred dollars per home (Zillow covers nearly every home in the US) to get a professional to produce an assessment, and these assessments are usually only a little more accurate than what Zillow is claiming.
(2) Author pulls up tables from Zillow showing a median error. Which is an excellent way to analyze this and is (I think) extraordinarily small. Author then proceeds to do other stuff.
(3) Author decides to verify the values by checking a total of NINE houses. Is the accuracy of that test expected to impress us? Furthermore, one of these 9 is rejected because "(obviously a data glitch or input error. Dismiss this result)" -- but that's the whole POINT of using median error; it isn't affected by outliers.
(4) Author finds that all of the values (all 3 of them) in Staten Island were pretty far off. Author asks professionals who say that the local market o Staten Island is behaving quite unusually at the moment. Reporting on this is the only thing I think the author did right.
Zillow's aggregate economic data is solid, often a better predictor than the Case Shiller Index. I expect that Zestimates will improve over time as the data trickles down and lessons are learned.
The funny thing is the estimate doesn't change when a sale closes. I would think the surest indication of what you can get for a property is what it sold for last week.
Could Redfin not explain why?
I've never bought a home in an area Redfin is in, so I havent used the site before. If Redfin knows the formulas/parameters used for the estimates couldn't they break it down and do side by side comp of $ amounts?
1) Pick three homes that you feel are comparable.
2) Pick out the material differences between those homes and your client's
3) Make up a value for each of those differences, and adjust accordingly
(Unofficially speaking) If you don't like the number you come up with, just make different decisions in steps 1-3 until you get the number you want.
For the most part, the banks and brokers re-sell those loans immediately, so they don't stay on their books. All they need to protect them from lawsuits later is proof that they followed the process to the letter -- which involves getting a written appraisal.
No mortgage broker or Realtor ever got rich by picking a fight with the appraiser to lower the appraised value ... there is literally nobody present at the table who has an incentive to push the value down.
If the bank won't write a loan for the agreed price, the original contract is void, since it's contingent on loan approval. If the seller's heart is set on moving, he might well lower the price.
But it's been a decade now since the housing bubble revealed that appraisals are meaningless.
Why has no one eliminated them?
Whether those efforts were successful or not is another discussion.
We ended up just guessing high and lowering our price a couple of times until someone bit.
The appraiser just needs to make sure that the number isn't fraudulent or crazy.
It's different now, not necessarily less subjective but the incentive for creative appraisals has been drastically reduced. Prices have to be justified and mortgage brokers can't interact with the appraisers now. Apprasing has to be entirely separate from the underwriting process and is now entirely blind. In some states, appraisers are now accountable for significant deviations. The turnaround time is also significantly longer. I've heard reports from some people of appraisals in bigger markets taking as long 9 weeks.
It's actually a pretty interesting process, ripe for disruption if anyone wants to take a crack at it.
Yes, this sounds pretty good to me. I spent some time trying to do something sophisticated for the estimates on my site https://houseprices.io/ using repeat sales of similar properties weighted by distance, but I wasn't getting anywhere near that. I gave up and just used the Land Registry index for the county in the end. There are just so many potential local factors which you're never going to capture algorithmically, not least of which is how much the current owner might have spent doing it up. Although I do get the occasional irate message in my Contact Us form telling me exactly how much!
"Research" of this type should probably never be trusted.
At least give me 10-20 homes in each of 10-20 cities if you are trying to convince me of something systematic.
The ostensibly data-driven part was by far the least convincing section - as soon as I saw the sample size I decided it was nothing more than a starting spot-check.
It seems to me that the real news here is actually that the Zillow median error is really good, given the data they have to work with and the scale on which they are operating. Of course, this does not mean that the Zestimate will be valuable in all cases. Individual buyers, sellers, and their agents and appraisers will often have fer better information about any given home. But everyone should have realized this already.
So Yeah Zillow has an effect. It's worth it to create a login & enter in your upgrades, so your ZESTIMATE goes up.
I did it before Zillow, and they start with the property value as estimated by the city for tax purposed and look at recent sales that are similar in the area, and adjust.
Very nice, thank you, my Zillow estimate is now $5965 higher. We have no intention to sell but if we do, at least I can start the negotiations at a higher price point by pointing to Zillow.
Of course, if I compare my house to actual sales in the neighborhood over the past few months, that's probably 10% high, so correcting Zillow's data about my house made the zestimate less accurate, not more.
But yeah, your conspiracy theory nonsense is probably just as legitimate.
Zestimates are a tool that homeowners like to use. Zillow knows that. Good Realtors won't feel threatened by it but instead leverage it to provide prospects with a much more complete and accurate estimate.
Realtors need to stop blaming Zillow & Zestimates for their own weaknesses. It is getting old.
Some basic UI changes could avoid all of that: e.g. instead of showing $123,456, show a range like $115,000-135,000, and keep the last few digits at zeros based on their estimated error rates. Similarly, their graphs could use shading to make it clear that they're showing the mid-point of a range rather than a discrete value.
So true. When you know that you are not displaying an exact dollar amount you should not use exact dollar values. Exact dollar values afford exactness even though the branding is an estimate.
I'm sure the data is inaccurate, but it's somewhere to start the conversation and let people feel like they have some power in the process of buying and selling homes.
And as people like to say about Uber, realtors are welcome to make a competing site that's as free and easy to use.
Financing still lives and dies by the professional appriasal; as long as banks prefer that, Zillow has zero pull.
https://www.biggerpockets.com/rei/real-estate-comps-house-va...
Yet again the tech community believing a problem exists to be solved that has already been solved.
> In most markets comparables are acquired through an MLS database, and access to MLS is usually not free.
This is true; you need to pay for access to the data source (no different than having to pay for access to equities market data). Redfin succeeds at this because they're a brokerage that happens to be a tech company, unlike Zillow who tries to just pay for feeds (that usually have stale data).
You can hire or become friends with a real estate professional that pays for access to MLS (or become one yourself).
But before sites like Zillow, there wasn't anything like it easily available to the average person.
Zillow unfortunately doesn't help though, as no one takes the value seriously (except perhaps uneducated consumers entering the real estate market). Bad data is worse than no data.
Some properties are only on the market for hours/days; you'd never have a chance as a market participant if you only relied on stale public records.
I don't think appraisers can use MLS listings exclusively when valuing a property. The primary source is typically public records data.
So people who take percentage of the deal disparaged a tool that would net them less money...quelle surprise.
The real estate industry is one of the most protectionist industries out there, and any threat of disruption, no matter how minor, is met by an army of vitriol.
I'm the Bay Area I feel like a seller's agent is grossly overpaid for what is often a weekend of open house showings and then fending off multiple above list price zero contingency offers.
I countered that I don't care. In this day and age buyers are savvy enough to jump on Zillow and Redfin and find my house, and they will demand they see it. If I hired a buyer agent and they tried to talk me out of showing a house over a commission I'd fire them right there.
A few years back I called Wells Fargo to get some info on refinancing, and the first thing the loan agent did was get on Zillow to calc the existing LTV.
In a case like this, we'd probably want to use locations based on number of home sales; and even then, we'd need to account for the amount of time between homes changing hands i/e houses that are flipped. There's likely something to be said about the difference between the nation's average sale price and each area's average sale price; assuming that may affect Zillow's estimations.
Anyways, I like that you're doing studies like this, regardless.
Equally important, is that the author of this article doesn't appear to know much about how the real estate data industry works (especially in regards to market trends and how they are used to generate the Automated Valuation Models (AVMs for short). A bit surprising given the nature of the business he is representing.
The Zestimate is one of a handful of Automated Valuation Models (AVMs in industry speak) - it's the only one with a consumer-facing brand, and as a result sees the most scrutiny despite the fact that the other AVMs on the market are the ones actually used by the banks during the appraisal process.
If you think the Zestimate is bad, you should see some of the other commercial AVMs.
The reality is that creating an automated valuation for the ~100M properties in the US is an incredibly difficult task given the availability of data (or lack thereof!). Zillow does a pretty darn good job and has a bit of a data advantage given its incredibly high coverage of for sale listings (though they lack the experience - some firms have been producting AVMs for decades!).
Another thing to note, is that AVMs (like any statistical model), usually aim for a sweet spot that covers as many homes as possible. As a result, ultra high-end properties are often incorrectly valued. E.g. that $25M mansion down the street will be very hard for the model to price. Additionally, certain luxury features, like say, a tennis court, are almost impossible to incorporate into these models on a national scale.
Actually computing the market value as a "cost-to-build-today" is rarely done, at least for home sales the typical consumer is going to be involved in.
And those appraisers can be very biased in either direction.
But lenders generally don't care so much about the cost to build, so an appraisal ordered by a lender generally won't.
That being said, my recent appraisal (I was financing the purchase) said "All 3 approaches to value considered with most weight placed on sales comparison approach."
I could mathematically prove this statement false, as the cost to build had zero consideration on the value. I think it is mostly a canned statement. Maybe they have a different definition of the words "considered", and "most". :)
[I]n 2007the New York State attorney general sued First American: relying on internal company documents, the complaint alleged the corporation improperly let Washington Mutual’s loan production staff ‘’hand-pick appraisers who bring in appraisal values high enough to permit WaMu’s loans to close, and improperly permit WaMu to pressure …. appraisers to change appraisal values that are too low to permit loans to close” [FCIC 2011: 92].
Looking at other houses in the area, my house appears to have a much higher perceived vs actual value, based on Zillow estimates alone. For example, my neighbor's house on Zillow is supposedly worth $90k, but I'd put it closer to $100k based on her lot size, house size, overall condition, and age of the house (I'm friendly with my neighbor so I know a little about her property's history and makeup). In other words, her Zestimate is a bit lower than what a reasonable person would put it at, whereas mine seems greatly inflated. Perhaps this is due to Zillow "grading on a curve"; it would seem odd if one house was rated at half the value of its neighbors, so mine gets brought up to the lower end of the average range even though it doesn't deserve it. That's purely speculation on my part though.
Its a worthless number, and they should stop trying to manipulate the market. Having a secret measurement formula that works for 70% of the houses but not the other 30% needs to either have a disclaimer that its all fantasy, or just outright be removed. I'm guesstimating those 70/30 numbers, but their site gives some locations 2 stars on their own scale: http://www.zillow.com/zestimate/#acc. If its not 4 stars, then don't publish it.
Also, it is important to remember that no appraiser from Zillow has actually looked at your house; the data is drawn entirely from sales data, public records, and volunteered information. So there's no way it can really account for every factor, nor does it even try. And there's a lag; the market can change faster than Zillow can. It is, at best, a rough ballpark estimate of what's going on in your neighborhood.
1) its ridiculous to also "fix" the zestimate history. Put the old one in the history, and explain near the chart what rejiggered.
2) The market in my immediate area, as evidence by the comps from zillow and the sales prices in the newspaper, has only increased. The conservative comps are near the old zestimate - this is a suburban area. The wild ones are higher than my private estimate.
What happens is that the model uses trends to extrapolate from each "real" data point (in this case, a house sale in your neighborhood). The problem is, and what Kalman Filters help manage, is the uncertainty propagation between each house sale. When it has been a long time from when a sale has occurred, it is unclear what the real/actual price is of a home. This means that on the estimated price the error bounds are large, and zestimate still just reports the mean value of this huge uncertainty.
What then happens is that a house is sold in your area, a new data point is recorded, and the filter re-adjusts itself and collapses its uncertainty/error bounds in the time of that measurement around the measurement. And you get correction. This is why the "old zestimate" is updated.
A couple of weeks ago, the Zestimate dropped to $435,000. This would definitely be closer to an accurate value, but there's also no record of the recalculation at all in the graph.
At this point, it's basically noise.
I just bought a home myself and looked this up, but the takeaway is that the Zestimate is within +- 5% 50% of the time. So, not that accurate. Are people really using it in negotiations though? I'm sure people do, but you'd have to be pretty uninformed to do so, and I'm also sure you're real estate agent would tell you exactly that up front.
As with anything, the 'value' is what a willing and able seller and a willing a able buyer agrees to. An appraiser attempts to determine the most likely price for something (in this case a home) within reasonable amount of marketing time. The appraisal report will usually specify the expected marketing time. I think in most areas this will be 3-6 months, some areas longer and others shorter, depending on the market.
Of course, appraisals effect prices, because most buyers need to finance, and most financiers won't do it without an appraisal that is at least as much as the purchase amount. (That is the "able" part of the willing and able requirement).
Two appraisals performed on the same property at the same time can come back with very different numbers. I have bought two houses in the last 4 years- both had 2 appraisals done, and one the values were almost almost 17% different- a difference greater than most of the "errors" that the articles notes for Zestimate vs sale price. On the other house, the appraisals were came back with nearly identical values, but the first appraisal was done wrong. Still this value (the appraised value from both appraisals) was 16.2% higher than what I paid for the property.
My experiences are just as anecdotal as the articles, though. But I have spoken with real estate professionals all over the country that have the same experiences repeatedly with appraisals.
I paid a couple hundred bucks for an ex-general contractor with a thick binder to inspect my house for hours, looking inside the furnace heat exchanger and testing every wall outlet and inspecting the plumbing and roof inside and out and insulation and foundation and everything. That is a detailed analysis of what you're likely to spend on maintenance and upgrades over the next ten years. Its probably more useful than an appraisal but it takes a lot more time and money.
http://www.freddiemac.com/loanadvisorsuite/loancollateraladv... http://www.freddiemac.com/loanadvisorsuite/faq/loancollatera...
This doesn't even take into account the process for appraisal management companies (third parties).
What Zillow does with its Zestimate product really isn't that novel. There is a whole class of products available called AVMs (Automated Valuation Models) which are used in the industry which pre-date Zillow's Zestimates. What Zillow did which was innovative was brought this type of product to the general public, for free.
In some cases an AVM is all that is needed to secure certain types of loans such as HELOCs.
Interestingly, Zillow recently bumped up our Zestimate from $190k to $230k. What's puzzling is that they also bumped up the historical Zestimates—there isn't a big jump shown on the graph on their site (though there is a big jump shown in my Personal Capital account, which uses Zestimates as part of their net worth scores). Either they modified their algorithm in a way that affected values retroactively, or they're trying to disguise the fact that their previous Zestimates were way off.
Probably the latter. Although, it could be that the data points are calculated on the fly or starting from some arbitrary value.
Then again, since it's shady to begin with, disguising their mistakes wouldn't surprise me.
There are two things a "historical estimate" could mean:
1) What did Zillow say the value was on day X?
2) What does Zillow now think the value was on day X?
You and others seem to be thinking of (1), but I think Zillow actually is doing (2). Since the goal is to figure out what the house is worth and how its value has changed, I think (2) is actually the right metric here.
The useful part of these charts isn't really the current Zestimate, but rather the trend line of the home values, especially at the neighborhood level.
If you don't recalculate the historical values, then you break the ability to see the trend line.
If you really want to see the results of previous models, then the way to do that is to plot a separate line for each model, but you can see why they wouldn't want to do that from a product perspective.
Algorithms have outliers, and quantitative analysis requires adequate sample size.
That said, they do a pretty decent job overall. Property valuation is art and science, and is probably a lot easier in a place like LA with lots of tract homes built in similar timeframes to similar spec.
I live in a city founded in the 1600s. Housing within a 5 block radius is built between 1890 and 2016. It's not a market with a lot of flipping, so it's hard to value as a house that looks like mine (reasonably modern internals) may have 1960s kitchens and bathrooms.
Edit since I wasn't clear: I know they show a range in tiny text, but they main number they show should be the range if they want to avoid all of the "is Zillow accurate" articles.
"Zestimate $472,603", and immediately to the right: "Zestimate range $425k - $506k". The little question mark reveals:
"The Value Range is the high and low estimate market value for which Zillow values a home. The more information, the smaller the range, and the more accurate the Zestimate. See data coverage and accuracy table."
Using two identical homes in my neighborhood about 3 doors apart from each other, mine / neighbors.
Zillow - $387,546 / $384,326
Redfin - $417,808 / $420,737
TCAD - $368,876 / 372,802
(Recent Appraisal) $427,000 / (Recent Sale) $429,000
What's worse, is that when I contested my taxes Zillow used the TCAD numbers to lower the value of my home on their site. Uh... Zillow, you know it's in our best interest to contest our taxes, right? Has nothing really to do with the value of our homes... it's a mix of us wanting to pay less and the City / County wanting us to pay more.
I'm sure there are more scientific studies, but for my house I'd say Zillow is ~10% under-market. When I went in to contest my taxes this last year, I was told by a friendly TCAD employee that Redfin numbers were better and that I should use Redfin when trying to cite examples of comparable properties with lower taxable values.
Keep in mind most of the value depends on your realtor's ability to show the house, how well you stage your house, and the number of of interested buyers.
Many authors in this vein have read about, for example, cognitive bias, the many invalid results in scientific journals and the hoax/fake news/infowars trend. They see every human action as a result of bias, every result as invalid, and every news article as part of a misinformation campaign.
Having the most accurate possible understanding of the world we live in is only possibly by trusting information sources which deserve it, understanding the limits of what "quick and dirty" statistical analysis is capable of, and recognizing when people are influenced by bias, and also when people are acting with full awareness of bias.
In my experience, Zillow has played big part in house pricing. Seller in hot market will almost always set the max of Zestimate and price in their mind. That prices decreases only slowly over time if buyers don't show up. In hot market usually some sucker will show up anyway and Zestimate becomes self-fulfilling prophecy.
I know Zillow doesn't make a claim about being super accurate, but not everyone is an intellectual buyer doing math in their head. For a lot of people, they treat zillow as a roughly true indicator and are surprised when that doesn't match reality. Or they completely dismiss it in the very beginning and rely on their agents or intuition.
You'd be surprised how seriously people who aren't quant inclined can take a prediction or estimate. Well, given that all the quant inclined among us believed Hillary would win, it's obvious that humans tend to put trust in other people's estimates if they essentially trust those people would do a good job unless proven otherwise. It reduces the cognitive burden of calculating it ourselves.
No, it should not. My personal example: When I bought my house, I bought it for 20% less than what Zillow said. But Zillow would have been right on the money if my house were in average condition. But it wasn't, it was unliveable. The first thing we did, before we moved in, was spend 9 weeks renovating it. Even then we didn't even bring it up to the local average. It took us five years to finally finish all the work.
Zillow estimates assume the house is in average condition, because they have no way of knowing otherwise.
Incidentally we bought it from the original owners who had owned for 55 years. I wonder if there is a correlation between condition and how long the current owner had it.
If we go by the "market" rate of what it would/could actually sell for then Zillow is the most accurate out of all those estimates.
Now I've done quite a bit of interior work to the house and Zillow's is most likely off by quite a bit now that my house has a new kitchen and bathrooms but I'm not sure how it could take that into account since there's no way for that data to reach Zillow.
Status Quo in my experience goes:
1. List your house (usually at slightly above whatever the realtor thinks it might sell at)
2. Someone's willing to buy at listing price +/- 5%
3. Bank sends an appraiser whose job it is to figure out what a house is worth (guess what: see #2 for relevant demand evidence), and magically the number comes remarkably close to the price sorted out in #2 massively most of the time
I've never heard a real estate agent complain that zillow's numbers are too low, mind you... only ever that they're too high, which makes sense to be frustrated by if your biggest challenge is controlling your time-costs.
Only seemingly unrelated, just in case, the classic from Frakonomics:
Zillow counts it as two data points at half price!
It's the only sale on the property so they have two half price "historical" inputs to the Zestimate, and that makes the real listing price seem nuts to an Internet home buyer.
They refuse to correct the data, insisting it sold twice at full price on the same day instead of two halves, and saying it's on me to get the town hall to reflect it as a single transaction, which it was not.
"You gotta eat your own dog food," goes the old zaying.
http://www.inman.com/2016/05/18/zillow-ceo-spencer-rascoff-s...
I have a friend who just bought a 3,000sf house on 1/2 of an acre in Lexington, VA for $31k. Zillow has it at $400,000. He's either incredibly lucky and should buy me a beer, or Zillow is the worst. Leaning towards the latter.
Drop a 0 there? It seems likely that $31k would go beyond lucky into probable theft....
As to the disperity, this could be easily explained by seller factors. Maybe the house is actually worth $400k, but they needed to sell it, even if less than it was worth (had a contract on another home, home was a foreclosure, etc etc).
I suspect their data quality is pretty terrible if they're unable to do any curation, so it's no wonder the zestimates are all over the place.
According to Zillow, 90% of the houses in one region were overpriced and selling like hotcakes. And 90% of houses in another region were underpriced but staying on the market.
I think most of the value is not an absolute measure, but as a relative one. I can compare Zestimates to each other in the same city with fair accuracy.
For example, I'm giving serious thoughts to moving to SoCal, and I see a place that looks pretty nice... it's listed as being $2000/month, but then when I go to the listing it says Zestimate $2650/month. What's up with that?
That seems... not useful in any way.
Second, my house is lakefront. Lake frontage is probably the single biggest factor in valuing the property. Yet Zillow ignores lake frontage as a feature. In fact a recent check of so-called "comps" included no lakefront homes at all. A complaint to Zillow was met with a yawn and a "well its just a number people can use as a starting point, and real estate professionals in the area will know the particulars." So basically they told me the "Zestimate" has no value and they know it.
We recently moved, and rented our condo, and we were able to rent it for 2k more than the rental zestimate. So, I agree that it's to be taken with a grain of salt, and not that important.