|Main » Articles » Musings on Economics|
|Entries in category: 41
Shown entries: 1-10
|Pages: 1 2 3 4 5 »|
Sort by: Date · Name · Rating · Comments · Views
Now that I’ve written about why randomized controlled experiments are so great, it’s time to talk about some of the common ways in which they can go wrong. But first I’d like to make an important caveat: finding potential flaws with any research, even randomized controlled experiments, is actually pretty easy. I haven’t come across any study that couldn’t be criticized on one or more grounds. So with the power to criticize also comes great responsibility: don’t use it to dismiss results you don’t like. Don’t selectively apply these criticisms to some studies and accept the findings of others that could be subject to similar criticisms. Use the knowledge wisely.
The main concern with randomized controlled experiments is the question of “external validity”. Sure, you’ve shown that something works in the laboratory or in a carefully controlled setting, but does it work in the real world? If people in the laboratory are different from those who will be subject to the treatment in the real world or if people (including those administering the treatment) behave differently in the experiment.
For example, maybe you run a clinical trial for a drug and only recruit men to participate in the trial. Will the drug work as well on women? Will there be different side effects for them? For a long time, clinical trials frequently omitted or under-enrolled women, although that is now changing. Or maybe you enroll obese individuals in a weight-loss trial but only includes ones without other health problems like diabetes. But once the drug goes to market, it may be prescribed to all types of obese individuals, and potentially have different effects than what you observed in the laboratory. Or maybe the nurses working on your trial are really good at getting patients to take the drug on time, but in the real world people forget to take it and you observe much lower effectiveness.
External validity is a potential problem with all experiments, not just clinical trials and not just stylized laboratory experiments. As long as people know they are part of an experiment, they may change how they act (maybe to make the experimenter happy, maybe to hide socially unacceptable views or behaviors, or maybe because they don’t take the experimental treatment as seriously as they do things in the real world). This is known as the Hawthorne effect, and it’s essentially impossible to rule out unless your subjects do not know that they are being studied.
Finally, external validity can also be a concern if you’re trying to say something about high-stakes decisions by running a low-stakes experiment. For example, you’re open to this criticism if you want to say something about how people save for retirement and you either run a hypothetical choice experiment or an experiment with low stakes (because who can afford to run an experiment where tens of thousands of dollars are at stake?). In some cases, the low-stakes findings survive in a high-stakes environment, but in others they don’t.
The bottom lines is that the most convincing experimental conclusions are those that are based on a representative population that faces stakes similar to what they would be in the real world, and where the experiment closely resembles real-world conditions (including individuals being unaware that they are part of an experiment).
(click here for part 1)
I was going to write more about quasi-experimental methods, but then I realized why these are usually discussed last in econometrics/empirical methods books. In order to see why quasi-experimental methods are useful, it’s first helpful to understand why experiments are good and where non-experimental methods can falter. Of course, experiments have drawbacks too and non-experimental non-quasi-experimental methods can produce valid results under some conditions. But we’ll talk about all that later.
When properly designed and executed, an experiment will easily allow you to estimate the causal effect of a randomly assigned condition (“treatment”), X, on any outcome Y: effect of a job training program on employment, effect of teacher training on student outcomes, effect of a drug on mortality, effect of dog ownership on health, etc. At a very basic level, a valid experiment only requires two things: (1) a control group (let’s say one composed of people) that is not exposed to the treatment X and (2) random assignment to treatment. This kind of setup is called a “randomized controlled experiment”. In this case, you can just compare the differences in Y’s in the two groups to arrive at the causal effect of X (divide by differences in X between the two groups if X is continuous).
Why do you need a control group? Because things change over time. Over longer time scales, people age, get sick, get better, gain/lose weight, get/lose jobs, learn/forget things, move, and generally act in ways that could affect Y even without X. Over shorter time scales, people might be affected by the time of day, by the temperature, by changes in their mood, by the building into which you bring them, or even by the fact that they are taking part in an experiment. If you don’t have a control group, it’s essentially impossible to tease out the effect of X on Y from the influence of other forces on Y. Most researchers know this and use a control group to ensure that the estimated effect of X on Y is not confounded by anything else happening to the treated group.
One exception I found (there surely are others) is this study, which recruited 4-10 month old infants and mothers for a sleep lab study of “crying it out” (a method by which some parents teach babies to fall asleep on their own by letting them cry and learn to self-soothe). All mothers were instructed to let the babies “cry it out” when falling asleep, so no control group was used. Even after the babies stopped crying on the third day, their cortisol levels were elevated, suggesting that they were stressed out. As this Slate article points out, it is impossible to know whether the babies were stressed out by exposure to “cry it out” (as the research article claims) or by the fact that they were in a foreign environment – the sleep lab. The absence of a control group that faced the same conditions without being exposed to “crying it out” thus fundamentally limits this study’s ability to say anything definitive about how crying it out affects stress levels.
Now you might say, “Sure, for some things, a control group that’s part of the experiment is important. But for outcomes like mortality or income, why can’t we just compare outcomes of people who enrolled in the experiment to outcomes similar people who are not part of the experiment? That seems easier and cheaper.” The problem with this approach is that it’s hard to be sure you’re comparing treated “oranges” to untreated “oranges” as opposed to treated “oranges” to untreated “apples”. Even if you collect information on hundreds of individual characteristics, it’s hard to be sure that there aren’t other characteristics that differ between your experimental treatment group and your real-world control group. And those unobserved differences might themselves influence outcomes. For example, maybe the group that signed up for your job training experiment is more (less) motivated and would have gotten jobs at higher (lower) rates than the real-world control group even if they didn’t take part in your experiment. Or maybe the experimental group is healthier (sicker) in ways that you aren’t capturing and they would have lived longer (died sooner) than the real-world control group. For these reasons, you should always be suspicious of “experiments” where the control group is non-existent or isn’t drawn from the group that signed up for the study.
Finally, why can’t you let people decide themselves whether to be in the control group or not? For the same reason that your control group needs to consist of people who signed up for your experiment – if you don’t assign people to the treatment group randomly you can’t be sure that the two groups – treatment and control – are alike in every single way that affects Y except for X. It could be that people who sign up for the treatment are more desperate for whatever reason, and desperate people may behave differently in all sorts of ways that then affect all sort of outcome. Or it could be that they are more adventurous, which again could affect them in all sorts of ways. Or they eat more broccoli/cheese/ice cream and you didn’t think to ask about that. If there are any such differences that you don’t observe and control for adequately, you can never be sure that differences in Y between the two groups are solely due to the treatment X.
But what if you’re ABSOLUTELY SURE that there’s nothing different between your treatment and non-randomly selected control group that could affect Y other than X and other things you’ve controlled for? The thing is, you can never be sure, otherwise you probably wouldn’t be running an experiment. To be absolutely sure would imply that you know everything about how Y is determined except for the effect of X on Y. And there’s just no way that we know that much about anything that we’d want to study (at least as far as social science and medicine are concerned). But if you have a good counter-example, email me!
That was a long one! Next time, we’ll talk about how even randomized controlled experiments can go wrong.
You just read a fascinating article suggesting that drinking a glass of red wine is equivalent to spending an hour at the gym, that morning people are better positioned for success, or that gun control reduces policy shootings. Let’s pretend that instead of immediately posting the article on your favorite social media website (which I’ll admit I’m sometimes guilty of myself), you instead wonder if the scientific methods behind the study are sound and if you can draw conclusions about cause and effect. How do you figure that out?
Unsurprisingly, it can be really hard. Alex Edmans, a Professor of Finance, has a recent excellent blog post about separating causation and correlation. After seeing lots of (often subtly) flawed research shared on social media, I’ve also been planning to write a guide to separating solid findings from not-so-convincing ones. It was going to be a cool flowchart that you can make your way through, with explanations along the way about why each step matters. But after having it on my “fun” to do list for months, I realized that the only way this flowchart will ever see the light of day is if I write it as a series of blog posts and then summarize things in a flowchart. This is part one.
The first question to ask when evaluating a study is whether it is based on an experiment (where researchers manipulated something, either in a laboratory or in the “field”) or is observational (where researchers collected some data). Experiments may be more reliable if done correctly, but they are not panaceas: there are many ways experiments can go wrong and a big issue is whether experimental findings translate to the real world. But we do evaluate experiments slightly differently from observational studies, so this is the first fork in our imaginary flowchart.
Let’s start with observational studies (this will repeat Alex’s post a bit, but I think it’s useful repetition). The first question to ask yourself is whether the researchers used any “quasi-experimental” variation to come to their conclusion. In general, studies that do are more credible than studies that do not. For example, sometimes researchers get lucky and stumble on a seemingly arbitrary rule that separates subjects (firms, individuals, regions) into two or more different groups. Certain scholarships are given to individuals who meet a specific cutoff on a standardized test score. Because it’s very difficult to control your score down to the point, people right below and right above the cutoff should be very similar in ability, except that the ones right below the cutoff did not get a scholarship and those above the cutoff did. Voila – you can study the effect of getting a scholarship on, for example, college completion, without worrying whether people without scholarships are fundamentally different from people with scholarships!
In order for this approach – called a “regression discontinuity” – to work well, (a) it must be impossible, or at least very difficult, for entities to manipulate whether they’re right below or above the cutoff and (b) researchers must not stray so far from the cutoff that the similarity of subjects below and above the cutoff starts becoming questionable. Ultimately, whether these two conditions hold depends on the context and how narrow of a range around the cutoff researchers select. For example, it’s hard to control whether your SAT score is 1480 or 1490, but scoring 1300 versus 1400 is unlikely to be mostly due to chance. In other contexts, small manipulations are easy to do – for example, many firms have enough flexibility in accounting to turn slightly negative earnings into slightly positive earnings, making a regression discontinuity approach not-so-credible in this setting.
In the next post in this series (which may or may not be the next post chronologically), we’ll talk about other kinds of quasi-experimental variation. Bonus points to people who email me an article about a study they want scrutinized!
In my talks with graduate students, I realized that many of them look for research ideas in the conclusion of a paper, where author(s) will frequently say that something is "a fruitful avenue for future research." I always tell the students that this is a terrible idea, and I thought I'd share why that is.
When I write that something is "a fruitful avenue for future research", it generally means one of three things. The first is that it actually IS a great idea, and I'm already working on it. So you'd probably be behind. The second is that the direction for future research is a great idea, but I have no clue how to do it correctly. Of course, a clever graduate student or other researcher could come up with a novel research strategy, but unless you're reading terrible papers, chances are that the paper's author already thought hard about how to do it and gave up. The third possibility is that the "area for future research" is very doable and straightforward but not very interesting (e.g., replicating the findings in a different sample). And unless you're just looking for an additional paper to pad your resume, you shouldn't do this either.
In short, don't look for great research ideas in statements like these. There's no such thing as a free lunch.
Many charter schools appear to work quite well. Here are two quotes from two articles summarizing the research:
“sound research has shown that, when properly managed and overseen, well-run charter schools give families a desperately needed alternative to inadequate traditional schools in poor urban neighborhoods.” (NY Times, October 13, 2016)
“The briefest summary is this: Many charter schools fail to live up to their promise, but one type has repeatedly shown impressive results.” (NY Times, November 4, 2016)
Because in many cases admissions to charter schools is done through a lottery, assignment to charter schools is literally random, for students that apply. So the level of confidence in these results should be as high as it gets. There’s also no reason to think that the “one type” of charters that has shown significant results cannot be replicated elsewhere (in fact, it has). Then why do so many liberals appear to be against charter schools?
I don’t have a good answer to that question. Liberals’ resistance to charter schools in any way, shape or form reminds me of conservatives’ resistance to any gun control regulation. No matter what type of gun control legislation is proposed, their answer is always “this is a terrible idea”. They also frequently invoke a slippery slope argument – “first, the Democrats will impose more thorough background checks, next, they will take away all our guns”. My sense is that liberal voters see charter schools as a similar existential threat to public school funding. But just like in the case of gun control, to me that logic is very dubious.
We need more evidence-based education reform. Charter schools that have been shown to work seem worthy of our support. I agree with Sue Dynarski, a prominent economics of education scholar, who was quoted in the second article as saying “To me, it is immoral to deny children a better education because charters don’t meet some voters’ ideal of what a public school should be. Children don’t live in the long term. They need us to deliver now.”
I teach masters students the basics of micro- and macro-economics. When we talk about government intervention, one of the first topics is the effect of taxes in an otherwise competitive market. By this point, it’s pretty easy for them to see that taxes hurt both consumers and producers in that market because, generally, (1) buyers have to pay more for the good than before and sellers receive less in revenue than before and (2) taxes reduce the activity that is being taxed, lowering surplus for everyone. For example, if it costs a seller $1 to make a cup of coffee and every day she was selling one to a buyer who was willing to pay only $1.05 (presumably for some amount between $1 and $1.05), placing a 10-cent tax on that market will probably eliminate that transaction. This second effect is called the “deadweight loss” of taxation because losing these transactions creates only costs (to the affected buyers and sellers) and no benefits (because the government doesn’t get tax revenue and consumers/producers do not benefit from transactions that don’t happen). That doesn’t mean we should never have taxes in competitive markets: if the government puts the tax revenue to good use, then social gains can overcome the deadweight loss. It just means there’s no free lunch!
It’s important to note that the assumption here is that we don’t want to limit the economic activity itself (e.g., because it generates pollution). When we talk about “externalities” such as pollution and how taxes can be used to resolve them, I usually ask “Do taxes to correct an environmental externality create deadweight loss?” By this point, a lot of my students have learned to equate taxes with “deadweight loss”, so many will generally say “yes”. However, that is not the case (but I’ll save that for another post).
After we cover taxes, I ask my students: “Do subsidies (in the form of a payment per unit of something produced/sold) in otherwise competitive markets create deadweight loss?” I always think this is an easy question because a subsidy is just a negative tax. The answer then should clearly be “yes”, but the students are usually stumped. So I thought I would write a post about the economics of subsidies.
Unsurprisingly, subsidies work in the opposite way that taxes do: they generally benefit both buyers and sellers by raining the amount a seller receives for selling a good and lowering the amount a buyer pays. No one participating in a subsidized market has an incentive to want to get rid of the subsidy because both sides benefit! Subsidies also increase the amount of the subsidized activity – add a 10-cent-per-cup subsidy for coffee and people will drink more coffee. Someone who wasn’t willing to pay more than $0.95 for that cup of coffee may now buy it for $1 because they also get a ten-cent subsidy that offsets some of that cost. Alternatively, if the subsidy goes to the seller, the seller may lower the price to $0.93, also inducing the buyer to buy.
But this increase in economic activity is not a good thing because the additional “units” being produced and exchanged are costing more to make than the buyers value them at. The net benefit (value to consumer minus cost to producer) to society of this additional economic activity is negative because the buyer values the good less than what it costs to produce. On top of that, subsidies need to be paid for by taxes, which means possibly creating deadweight loss in another market!
One justification people give for supporting subsidies is distributional concerns. Maybe we’re losing some efficiency, but we’re making sure that (presumably poor) people can afford to buy the good in question. However, subsidies are a crude and expensive way for achieving distributional goals because they help everyone who buys in the market, rich or poor. For example, subsidizing college education will certainly help poor students, but if the subsidy is given to everyone, it becomes much more expensive in terms of the amount of revenue (and deadweight loss of taxation) that needs to be generated.
An obvious way to improve on subsidizing something for everyone is more targeted subsidies (like financial aid for poor students). However, even that is not ideal because it distorts individuals’ choices. If we start subsidizing coffee for low-income individuals, coffee will be more affordable, but people will also drink more of it relative to other goods, and it’s not clear that we (or the individuals) want that. Rather, economists advocate giving poor individuals money and letting them decide what to spend it on. That comes with its own set of issues because it creates a larger incentive to pretend to be low-income, but it also respects individuals’ choices and does not lead to unnecessary distortions.
My representative, Rodney Davis, recently introduced a health care bill "to protect people with pre-existing conditions from discrimination against insurance companies." (yes, if you think about it, that sentence is poorly written).
I just wrote to him to ask a few details about his plan. I'm sharing the letter below because it demonstrates the difficulty of ensuring that individuals with pre-existing conditions can buy affordable insurance.
"I read about your new health care bill to make sure people with pre-existing conditions can buy health insurance. I'm just curious as to what happens if insurers offer someone who has cancer insurance for, say, $50,000 per year. Would you consider that acceptable? If not, what provisions does your plan have in place to ensure that does not happen?
If your plan has limits on whether insurers can charge different prices based on pre-existing conditions, how will the plan ensure that younger and healthier people do not have a disincentive to sign up because they are being offered insurance at a price that is much higher than their expected healthcare costs?"
There are really only two ways (that I can think of) to ensure that (1) people with pre-existing conditions are not being offered health insurance only at exorbitant prices and (2) you don't create a "death spiral" where people buying insurance on the individual market are increasingly sick because the healthier people drop out due to rising prices. The first is having an individual mandate (a stick) and the second is a generous tax credit that makes buying health insurance very cheap on the margin even if the pre-credit price is very high (a carrot). I look forward to seeing what Davis's actual plan is (the "Better Way" Republican agenda does mention a tax credit).
(This is based on a true story, but I may have changed some details like field of study and gender to protect the student’s anonymity)
Shortly after Trump got elected president, a student made an appointment to talk to me. She was in the last year of her finance degree and had a good job lined up, but was doubting whether she should continue with her life plan in light of the election. She realized that she wanted to make a difference in the world and a career path in finance didn’t seem like a good way to do so. Instead, she was considering going to work for a women’s reproductive rights organization (I definitely changed this detail, but it roughly captures the spirit of this student’s desires).
I told her to consider sticking to finance and donating a large part of her salary to her favorite organization. Why? Because individuals who hold high-paying jobs can often make a lot more of a difference this way. Her starting finance salary would have been probably at least $120,000 a year. If she left finance and went to work for the non-profit, she would make at best $40,000 a year. But what if she donated $80,000 of her finance salary to the non-profit instead? Well, the non-profit could hire TWO people like her and she would still earn $40,000 per year, as much as she would have at the non-profit.
Of course, there are some caveats to this. She would probably have to work longer hours in finance and maybe she would enjoy it less than the non-profit job. So to stay indifferent between the two, maybe she would donate “only” $50,000. Still, the organization might prefer having that money to having her work there, especially if she didn’t have any special training.
That brings me to the second piece of advice I gave her. If, after considering the high-paying-job-plus-donations option, she still thought going into the non-profit world was better, I advised her to think about positions in non-profits where her finance training would be useful. For example, if she wanted to help low-income women, perhaps she could get involved with an organization that provides financial training to disadvantaged women or manage a non-profit’s endowment. Even though that may not have been her first choice, it would probably be more valuable to society.
So as we sit here wondering, “What the f*** do I do now?”, consider whether your salary allows you to make a substantial donation to the many organizations out there fighting the good fight. If you’re a student, don’t feel like you have to drop everything and become a full-time activist (though you should still call your Congressman once in a while and follow the non-alternative news!). First, sit down and think about how much money you can generate for your favorite organization by not working for them. Alternatively, consider which causes your skills could be useful for – a lawyer going to work for ACLU is a lot more useful than a lawyer going to build houses for Habitat for Humanity.
To be clear, I am not saying that you should take a job you find immoral or incredibly unpleasant. There is ultimately nothing wrong with leaving (or not taking) a high-paying job where you don’t feel like you’re making a difference for a low-paying job where you feel like you do. And of course we need people actually working at organizations like ACLU or Planned Parenthood (yes, I’m shamelessly promoting my favorite ones). But these organizations need money too, and if you face a high opportunity cost of joining them full-time (i.e., your salary is or will be high), consider giving them your money instead. You might not get the same pat on the back from your activist friends, but I promise you that you will be making a big difference!
Most people are taught in introductory economics courses that they should ignore "sunk costs" (costs that cannot be recovered) in their introductory economics courses. But sometimes decisions involving sunk costs can be tricky, as a recent example I came across illustrates.
I have a "friend" who booked a spring break trip to the Dominican Republic with her two kids and husband nine months in advance. About two weeks before they were scheduled to leave, she realized they didn't have a passport for their younger son. They didn't forget, they didn't procrastinate, they didn't think he DIDN'T need a passport (their older daughter had one) - they just didn't think about it. It's one of those inexplicable things like a woman who doesn't realize she's pregnant until she goes into labor. On top of that, American changed their flight itinerary in a way that would make them have to spend a night in Chicago on the way there and probably on the way back. So they were not happy.
As it turned out, it was possible to get a passport on such short notice, but it would involve driving to Chicago (which is 2.5 hours away) during a work day with the son. But the question is, should they do that? Or should they cancel their vacation and stay home? Unsurprisingly, the answer depends on whether or not the airfare and hotel are refundable.
If the trip is not refundable (= sunk cost), it should make my friend MORE likely to go on this vacation than if it's refundable. Why? Because then the (marginal) cost of the nice vacation becomes the time and effort to drive to Chicago, the two nights' hotel stay in Chicago, and the unpleasantness of dealing with all that. That's not too bad for a fancy week-long vacation in the Dominican Republic. On the other hand, if the trip is fully refundable, then you have the costs already listed PLUS the money paid for the trip (because you could get the money back, it's as though it's still sitting in your bank account and you're considering whether to buy the trip). And while my friend was willing to buy the trip when the flights were easier and the document issue was not on her mind, adding these things into the mix tipped the balance against the trip.
As it turns out, the trip was almost fully refundable, so the family decided to cancel the trip. The moral of the story is that whether or not a cost is sunk changes the marginal cost of doing something and you should consider that next time you forget that you need a passport to go to Canada and find yourself without one.
And I'll be staying in Champaign over spring break...
Vaccines prevent diseases and that's awesome because even relatively mild diseases like measles and chickenpox kill or disable a minor share of the people they infect. So it is without a doubt that vaccines have prevented billions of sick days and saved many many lives (don't even go there, anti-vaxxers).
But there's another important benefit of vaccines that I've never seen highlighted - the time they save the parents by reducing the amount of time they have to spend taking care of sick children (I had this epiphany last week when both of my children were sequentially sick and home from daycare). And if your child is constantly home sick, even with routine childhood diseases, it makes it difficult to hold down a job. Undoubtedly, in the pre-vaccines era, this burden would have fallen largely on women. Of course, children still get sick, but my guess is that vaccines made it significantly easier for women with children to hold steady jobs. So the research question is, "What is the effect of vaccines on female labor supply?" Potential title of paper: "Vaccines and Female Labor Supply". You're welcome.