Data Daze

John Nelson, MD, FACP

As I write this column, our nation’s economy is looking pretty shaky. In early January, the labor market report showed a significant uptick in the unemployment rate. This led worried investors to sell stocks, and the Federal Reserve lowered the interest rate dramatically in response to the sudden fall in the stock market. Or at least that’s the version of events we’re being fed by most of the press.

Do investors overreact to job numbers? There is a lot of debate about the accuracy of job and unemployment statistics. Clearly they are valuable, but there are all kinds of problems with the way the labor surveys are conducted and the resulting data analyzed.

There may not be a better way to collect the data, so despite their flaws surveys may provide the best information on the labor market that we can get.

The real problem arises when investors—sort of like you and me but a lot richer—look at these data and fail to keep in mind all its strengths and weaknesses. There is a risk people will focus on a single number and overestimate its precision. This has been called the “salience bias.” Writing in The New Yorker, James Surowiecki says this salience bias can lead to “a hard-to-break feedback loop: The fact that traders act as if the jobs report were definitive makes it so. A little information can be a dangerous thing.”1

It can be valuable to trend some data over successive surveys. For example, you may want to know the trend in the average hospitalist’s wRVU productivity over the past two surveys.

I discuss hospitalist survey data with people all the time. I’m struck by how often they seem misled by salience bias, among other things. With SHM’s release this month of its latest biannual survey of hospitalist productivity and compensation, now seems like a good time to discuss the strengths and weaknesses in the data—and cautions when interpreting it.

Understand the strengths and limitations of the survey. SHM’s “Bi-Annual Survey on the State of the Hospital Medicine Movement” is a self-reporting survey in which each practice leader (or his/her designee) completes the survey. The responses aren’t verified or audited, so some respondents might submit shoddy data. Perhaps a busy group leader might complete the survey from memory and estimate things like each doctor’s production of work-only relative value units (wRVUs). When I’ve looked at the raw data, I’ve wondered if some respondents are trying to “spin” their numbers higher or lower for a variety of reasons (e.g., to look unusually good or show how hard their doctors can work). And there may be a response bias: Those who think their practice is atypical might not respond to the survey.

Leave a Reply

Your email address will not be published. Required fields are marked *