Zillow.com is a popular website for many homeowners and those who intend to access it to search for appraisals on specific properties. But how accurate can an expert system be that is entirely computer generated without the benefit of actually seeing the property in question, much less comparables, or even the location?
It was my brother-in-law who first told me about Ziilow’s Zestimate utility when I was new in the summer of 2006. As an appraiser and real estate agent in a major metropolitan area for over 50 years, he was impressed by how close the Zestimate was for a real evaluation, this from a guy who has little use for computers in general. I took this with a grain of salt as our company had put a lot of effort into developing such a utility that uses statistical analysis to determine value.
To understand the problem, it is important to understand the basics of property valuation. Since the savings and loan crisis of the 1980s and 1990s, bank appraisals have been conducted according to a rigid method called the Uniform Standards of Professional Appraisal Practice (USPAP). Generally, the specification calls for three to five similar properties to be compared within a half mile and sold within six months. These schemes can be extended when necessary. Differences are then adjusted to arrive at a valuation. The problem was that given the strict restrictions on what properties the appraiser could use and the small sampling (3 to 5 comparables), the results tended to be like comparing apples to oranges. And just like the arbitrator’s decision, it may not always be correct, but it’s always the bottom line for lenders.
Our utility company takes a different approach by comparing thousands of sales over decades, going back to 1991. The concept was to remove variables by total volume (large sample). Zillow must use something like this because they all have a large sample of properties (72 million nationally as of 2006) even though their algorithms are unpublished.
For our algorithms, differences in the date of a sale are adjusted for area appreciation (and recently depreciation) location is based on the school district, which is relevant in our market, rather than a restricted area around the property in question that has an unfortunate tendency to cross borders between areas of very different values. An appraiser who is not completely familiar with the area, as is common when online lenders order it, might compare properties in different school districts or not adjust for proximity to an undesirable feature, such as a major highway or industrial area, although comparables fall within the prohibited distance of the subject property. A good appraiser will take these adjustments into account, although there is no universal standard other than how a particular geographic feature affects an average of other sales prices for similar properties. USPAP valuations tend to vary by 10% to 15% from appraiser to appraiser, especially if each uses different comparables. This is one of the weaknesses of small samples.
In general, our utility worked well. Interestingly, considering how much effort went into its creation, we rarely use it. This is because any statistical analysis ignores the really important aesthetic considerations, such as warmth and charm, elegance, style, etc. In addition, the utility does not take into account variables such as mortgage interest rates, availability of funds (which hampers today’s market), and general consumer confidence. When the market is perceived to be appreciating, buyers are willing to pay more, but when it is perceived to be depreciating, buyers tend to insist on a larger trade to cover any further market erosion. Neither are competing properties currently on the market considered. These tend to have a greater effect on a given sale price than historically comparable ones. All of these factors have a strong influence on how much a given property will sell for. And yet, the Zestimate still manages to get a good estimate, at least sometimes.
As of June 13, 2011, Zestimate is in its third algorithm update. But how accurate is it really? After all, a lot can depend on it. Homebuyers often base their offer on a Zestimate and sellers often base their sales price on the same Zestimate. Fortunately, lenders don’t.
As a real estate agent, I am often confronted by both buyers and sellers who have placed too much faith in Zestimate over what a municipal appraisal, a real estate agent’s competitive market analysis (CMA), or even a property appraisal suggests. USPAP, especially if that’s what they want to hear.
To find out how accurate the Zestimate really is, at least in our marketplace, we simply compare the actual closed sale price taken directly from our Multiple Listing Service (MLS) to the Zillow Zestimate using last month’s sales data for counties. . of Albany, Schenectady, Rensselaer and Saratoga which includes the metropolitan areas of Albany, Schenectady, Troy and Saratoga, New York. Of course, every market is different and may not produce the same results. What we found was that Zestimate was surprisingly accurate in predicting the actual selling price with a few notable exceptions. Zillow uses a similar method to judge its own accuracy. However, Zillow’s accuracies are presented as a mean error of actual sales prices for Zestimate. As of their latest algorithm update, they claim an error of only 8.5%. (http://www.zillow.com/blog/research/2011/06/14/upgrading-the-zestimate) What is not stated is how far the Zestimate is when it is off. Let’s just say at this point that no appraiser, municipal appraiser, or real estate agent is likely to miss the mark as much as Zillow fails when it fails.
Moving on to the numbers:
In terms of average error, what we found was a staggering 2.8% error (overvalue) across all price ranges. In general, the lower price ranges saw a Zestimate higher than the actual sales prices, while in the higher price range, the Zestimate tended to undervalue properties.
Price Range – Zestimate – Percent Error
0 to $100K 113.85% 13.85% ON
$100 to $200K 104.13% 4.13% ON
$200 to $300K 98.28% 1.72% LESS
$300 to $400K 94.42% 5.58% LESS
$400 to $500K 87.22% 12.78% LESS
$500K and up 85.30% 14.70% LESS
As you can see in the table above, the undervalue in the higher price ranges eliminates the overvalue in the lower price ranges. Gold Zestimate was overvalued 47% of the time; undervalued 53% about half the time.
However, when the Zestimate failed, it missed by a mile. The largest error was an understatement of $221,329, or an understatement of 61%. The largest overvalue was $137,850 or an overvalue of 44%. This did not happen often, as noted. The graph below shows a large central tendency towards precision and a larger error towards either extreme.
Bottom line: If you have to be right, use an experienced and qualified USPAP appraiser who knows the area. If the intended use of the valuation is not critical, Zillow is ideal for an excellent rough valuation taking into account the possibility of material error.