Most people who’ve been through an economics PhD course sequence have probably learned about the perils of controlling for potentially endogenous variables, the so-called “bad controls” problem. And yet I still encounter projects that (1) claim to have a quasi-experimental source of variation and (2) control for potentially endogenous time-varying characteristics or outcomes (by “potentially endogenous”, I mean measured post-treatment).

I think there’s less misunderstanding of the pitfalls when it comes to non-quasi-experimental studies, where you need controls by the definition and where the primary concern is often that you don’t have enough of them. So I’d like to explain, in an intuitive way, why including such time-varying controls in quasi-experimental studies is a bad idea.

The first conceptual problem with including time-varying controls is that you shouldn’t need them if your source of variation is as good as random. If you feel that you need these time-varying controls to estimate the treatment effect correctly, then you should probably rethink your study design or the claim that you’re exploiting variation that’s as good as random.

If you believe in your variation in general but are concerned about specific differences between treated and control units, then better alternative include controls like region-by-year fixed effects, unit-specific time trends, or year fixed effects interacted with baseline unit characteristics. In an ideal world, you shouldn’t need these either, but they’re reasonable control variables that don’t suffer from the same problem as time-varying characteristics.

At this point, what I typically hear is, “OK, maybe I’ll show my estimates with and without these time-varying controls.” This may be a reasonable publication strategy given that some reviewers and journal editors also don’t understand the perils of controlling for potentially endogenous variables. It’s still wrong, however, because there’s no theoretical reason why your estimated “treatment effect” should be the same regardless of what you put on the right-hand side of the equation. At best—if the estimated coefficient doesn’t change much—you may conclude that those variables aren’t endogenous or at least not “that” endogenous. But then you’ll be perpetuating the myth that treatment effect estimates should be robust to time-varying controls.

However, it’s also possible that including time-varying controls does change your treatment effect estimate. This brings us to the second reason not to include time-varying controls: you could reach the wrong conclusion about whatever it is that you’re trying to study.

Let’s take a simple hypothetical example.  Suppose you are studying the rollout of a program that created more green spaces in neighborhoods of a huge city. The variation is fantastic: the city didn’t have the capacity to create all the green spaces at once, so they randomized which neighborhoods got the parks first. You want to see what these green spaces are worth, so you obtain data on housing values in each neighborhood. You estimate a simple diff-in-diff, and the treatment effect looks large: parks are worth a lot.

But then you decide to also control for neighborhood population and median income. Maybe you even realize that they could have changed as a result of the park program, but you think you’re getting a “cleaner” value of parks by controlling for them. And then your treatment effects shrinks by, let’s say, 75%.

Obviously, the story here is that the park changed the neighborhood composition. Let’s pretend none of the moves were within-city to set aside issues of spillovers and SUTVA violations (which would make the story more complicated but would not change the takeaway as far as potentially endogenous controls go). Why wouldn’t we want our valuation estimates to reflect the neighborhood composition channel? It’s absolutely part of the average treatment effect and trying to shut down that channel is going to produce misleading conclusions about what parks are actually worth. And if you’re thinking, “Well, at least this is going to give me the treatment effect holding income and population constant,” that’s only true under specific functional form assumptions. So you’re not getting anything that’s easily interpretable when you’re controlling for these time-varying variables.

Think about the situation from policymakers’ perspective. If they are considering implementing the policy you’re studying, they’re going to want to know the total impact. They can’t just shut down the neighborhood composition channel when building parks, so those estimates—even if they meant something—aren’t providing them with an estimate of the total effect they should expect if they put the policy in place.

This doesn’t mean those potentially endogenous variables are useless. Rather than include them as controls and risk messing up your conclusions, you can study them as outcomes. That will produce a lot more useful insights than burying them in the X’s.