Tag: stats 101

Why the Claims of Election Fraud aren’t based on Statistical Evidence

I’ve been following all the lawsuits trying to overturn the election results in states that Biden won. I’m far from a legal expert so I don’t pay much attention to the legalese but I do like to examine the “evidence” of “irregularities” in the election. I’m highly disappointed in what qualifies as an expert or evidence in most of these lawsuits. It is shocking to see people who claim to be experts make statements that occasionally could be refuted by an undergraduate who took a stats class and a political science class.

If you are going to suggest that our democracy was violated and that there was a massive conspiracy to rig the election you need to have solid evidence. That is a life-altering claim to the health of our democracy. It’s a serious allegation that needs serious evidence. You need to provide an analysis created by someone that has genuine expertise in both political science and statistics. To do a high-quality analysis you probably have to be on the level of a statistics Ph.D. student in the last part of their studies. You need methods commonly used in the literature for similar problems. And you need to approach the analysis knowing that statistical tests aren’t the best equipped to detect fraud and rule out more innocent explanations of the irregularities.

But as I’ve examined the “evidence” of voter fraud I’ve been highly disappointed in the rigor of the arguments. Voter fraud is rare, and the level of voter fraud necessary to change the presidential outcome has to thousands of times higher than normal levels. At best the conclusion of these analyses is that mail-in votes counted later did not come from the same distribution as the in-person votes or that the vote change from 2016. But everyone should have known that because the method of voting was highly partisan and mail-in votes take longer to process. And every election year there are commonly shifts from previous results. There isn’t evidence of massive fraud but people including President Trump are acting like there is. These are highly damaging allegations that are not based on facts.

Now I will break down what’s wrong with the analysis in six lawsuits. These lawsuits were filed by Republicans. I’ve addressed all of this on Twitter, but I wanted to summarize and show that there are many cases with many mistakes.

The Texas Expert who would have failed Stats 101

The recent Texas lawsuit (expert starts on pg. 22) says Michigan, Wisconsin, Georgia, and Pennsylvania had unfair elections and the electors chosen shouldn’t count. The expert has a Ph.D. in economics. Some economists are great at political science and statistics, but this expert doesn’t seem to be good at either. He claimed that the odds of Biden outperforming Clinton was 1 in a quadrillion. But he used a test that doesn’t apply because it assumes that you have a sample from ideally less than 10% of the population and that the votes are counted in random order and all the ballots have the same probability of being for Biden. Additionally, he used the wrong formula. And this test was something I taught undergraduate students in the most basic stats class at Texas A&M. I had final exam questions testing the ability to perform this procedure and this work would deserve a failing grade. I am struggling to express how horrible this analysis is. The analysis doesn’t make a clear conclusion other than these results are unexpected and the expert was used in the lawsuit to show that it is practically impossible for Trump to have lost. But the analysis can’t be used to make that conclusion even if it was appropriate and done correctly. This test can only detect differences in populations but it can’t explain why the difference occurred or if the difference was fraud.

The Michigan analysis that Used Minnesota Data

In another lawsuit, an expert claimed that there were more votes than people in certain precincts. This would obviously be a red flag and if true would likely show fraud. But these calculations combined voting results from Michigan using population data from Minnesota. So the analysis was completely wrong.

The Arizona Lawsuit that said audits by Precinct would be better than Audits by voting center

Another lawsuit in AZ claimed that the audit needed to be done by Precinct instead of by voting center to be accurate. But how you do a sample like that doesn’t have a large effect and sorting the data to Precincts from voting centers would be highly time-consuming. There wasn’t a need to do an audit by Precinct from a statistical standpoint

The Pennslyvania Lawsuit that wanted to assume illegal votes had the same distribution as legal ones

A lawsuit in PA wanted to take a sample of the mail-in ballot envelopes, examine them to see if they were fraudulent, and then assume Biden won the same percentage of illegal votes as legal ones. This is a bad idea because how can you know what the distribution of illegal votes is? Additionally, any analysis of the ballots would likely label some legal ballots as illegal. It’s hard to identify illegal votes unless you have a double vote, someone stealing a ballot, or lying about their eligibility. A signature mismatch isn’t always going to be fraud. You must consider that your analysis would have sampling error if not every ballot was examined. So you need more complex analysis and a highly designed sample. So a fishing expedition to find enough invalid ballots to change the winner is a bad idea.

The Nevada Lawsuit that said if ballots are rejected by the machine at high rates the machine must be missing illegal votes

A lawsuit in Nevada mentioned that the signature verification machine rejected a lot of ballots. Those ballots were reviewed by humans and accepted most of the time. The lawsuit claimed that since the machine rejected a lot of ballots it must have accepted multiple fraudulent ballots. But the machine rejecting a legitimate ballot is a unique process than accepting a fraudulent one. You can’t immediately conclude that the machine accepted fraudulent ballots. The relationship between managing false negatives and false positives often is an inverse relationship. So if this machine was rejecting ballots, those it accepted were likely signature matches.

The Georgia Lawsuit that said you had more voters than eligible people

A lawsuit in GA claimed there were registered voters than eligible voters and that there are many people who voted only for Biden. But the analysis that found “extra” voters used census data without accounting for a the relatively large margin of error on county-level estimate of eligible voters. Basically in a state like GA that has automatic registration, and therefore high registration the census data is often imprecise enough to show over 100% registration. The census data is also pulled over years and known to struggle in areas rapidly growing in population. Also turnout in these counties was much lower than the eligible voters estimates. There might be some extra people on the rolls, but they aren’t voting, and the problem is likely exaggerated in this analysis. Also someone else did really bad math that assumed Biden overperforming the Democratic senate candidate meant people only voted for President and no other candidates. But in reality this was just split ticket voting which is common especially among Republicans in the Trump era.

So in summary, lots of the statistical claims that there is evidence of voter fraud are statistically unsound. I fully believe that the results are correct, and that no massive fraud occurred.

The Nuance of Polling

I’m an election modeler, and my entire dissertation is focusing on analyzing public opinion polling data in one form or another. I love polling. Often on this blog or on twitter I’m cautious about a new poll or what an election model can actually tell us. So I thought perhaps I should explain why polling is important even if it may not tell us who is going to be the next President.

I feel there is an imbalance on how polling is viewed. Some approach it as being completely certain and if it is outside the margin of error it is impossible. Others dismiss polling because they can’t understand how one thousand people can tell us what the entire country thinks or that 2016 showed polling was a failure. But neither of these views is accurate.

The truth is polling remains our only rigorous and mathematically grounded tool to estimate public opinion. Elections can be forecast using economic and other data but that is only because the true proportion voting for a candidate is eventually known. But polling can tell us what percentage of individuals approve a certain policy or unravel how an individual’s policy preferences to prevent terrorism are related to their risk assessment of future terrorist attacks (as I’ve done in a recent project). We can understand how and when people’s opinions do and don’t change.

Polling isn’t a magic problem solver. The results from a poll can not be treated as 100% correct. Polling has error. Sometimes that error puts us in positions where all we know is that a race is too close to call or that the country is evenly split in its support for a policy. We have to acknowledge that margin of error won’t solve all our problems and that polling is hard work. It’s not easy to predict who is a likely voter or decide between an expensive phone poll or a larger internet panel or try to determine why someone left a question blank.

It is possible for polling to be very important because it signals to our government what the people want and that sometimes polling doesn’t give us a clear answer. It’s possible for polling to “be wrong” just by random chance. But it is also possible it gives us a clear answer. Often, it gives us something to point to as important for the government to act on in a way that is far more representative than calls to a congressman or your friend’s opinions or social media comments. If followed by leaders, polling could be a pathway for a more direct democracy without forcing every citizen to give opinions on every issue.

This election, it’s important to embrace the nuance in polling. Every poll is unique and needs to be interpreted holistically considering when and how it was conducted. Every poll on the same issue or election should have different results and that’s expected and ok. Polling is usually going to be off by a handful or two of percentage points, but sometimes the message is clear because the support is so strong or weak. But polling can give us answers when nothing else will, and for that, it will always be valuable.

Unraveling Polling Error (with GIFs and no math)

You can’t understand what polls mean until you understand how they work. The most common misunderstanding people have is about what margin of error means and what type of error it covers. The margin of error doesn’t tell the whole story. Polling error has many different components. Some types of errors are easy to predict, but others can be impossible to guess. I’m going to talk about the three most important sources of error in polling broadly but focusing on election polling. I want to do this so that you better understand why margin of error is not the only type of error and that polls work well all things considered. This post is not about how polls are wrong, it is about how they are right in the midst of numerous challenges.

Three Types of Error:

  1. Sampling Variation: effects of using only a subset of a population
  2. Incorrect Responses and Changes in Opinion: respondent to the survey intentionally or unintentionally does give the correct response or later changes their mind
  3. Miscoverage: the population that was sampled was not the population of interest

Sampling Variation

Possibilities GIF - Possibilities EndlessPossibilities SpongeBob GIFs

One concept people struggle with in statistics is variation. Statistics involves math but there is no longer one solution. If I solve an algebra equation again over and over again my answer shouldn’t change. Statistics is based on samples that are subsets of a population. If a collect a sample over and over again the numbers will be slightly different almost every time. In most cases, my estimate for the sample will not exactly match the true population. You can see this by flipping a coin a bunch of times. A US coin is going to be split relatively evenly, but when you start flipping the coin yourself you might not always get exactly 50% heads and 50% tails. This is because coin flips are random. Polls work in a similar way.

Sampling variation is a huge driver of polling error. It is completely mathematically expected that a sample of a few hundred to a few thousand people isn’t going to tell us the exact true proportion of support for a candidate or policy. If we make some assumptions and adjustments we can calculate a quantity called the margin of error that gives us an estimate of how much randomness to expect. Margin of error only includes error from sampling variation and not the other two types of error I will talk about later.

Typically you are going to make the following assumptions:

  1. People who were sampled but didn’t participate in the survey are not any different than those that did after you control for demographic variables. This is something that is obviously impossible to verify directly.
  2. You have a decent estimate of the target population. The target population is who you want to poll, and the main target populations are likely voters (people thought likely to vote), registered voters (people actively registered to vote), and all adults (including people that can’t vote). The census gives us a pretty good idea about the population characteristics of all adults, and you can use this information combined with other information to get estimates for registered voters. Likely voters are hard to identify because the respondent isn’t always the best predictor of if they will vote, and turnout varies from year to year.

Assumptions are very common in statistics and sometimes it’s difficult to assess how reasonable an assumption is. You may have some doubts about both assumptions being valid. To a certain extent, we know that these assumptions aren’t completely true, but there is a concept in statistics called robustness. Robustness says that under certain conditions (usually large sample sizes) small violations in assumptions are ok, but it has to be acknowledged that violations in assumptions can affect results.

Depending on details about the poll the margin of error is usually about 2-5 points. I’ll omit the calculation because it gets complicated in practice because most surveys have complicated procedures (but statistically valid) to use the estimate of the population characteristics to adjust the poll to match it (this is called raking or weighting involves a lot of math). Now what does this margin of error mean? Typically you add or subtract the margin of error from each estimated quantity and this gives you a range of probable values. Theoretically, if no other types of errors exists (they do) and the assumptions hold and you had dozens upon dozens of polls, 95% of those intervals should can the true population proportions. But in election polls, it’s common to have undecided voters and while it is important to track undecided voters, they complicate things since undecided isn’t really a ballot option. A workaround in election polling is to look at the difference between the margin of candidates, double the margin of error and see if that interval contains 0, and if it does the election is too close to call based on this poll.

Incorrect Responses and Changes in Opinion

Glee Brittany Pierce GIF - Glee BrittanyPierce IsItTheTruth GIFs

People makes mistakes in all aspects in life, and polls are no exception. Additionally you can lie to a pollster. Unfortunately in most polls you don’t know if the answer from the respondent was accurate. If we didn’t have to ask the respondent to answer the question because we knew it already there wouldn’t be much reason for polling. There is no mathematical formula that tells us how to exactly adjust for lying or mistakes in surveys. Respondent mistakes and lying is not in the margin of error because it is too hard to exactly estimate. You can test the respondents ability to follow directions by telling them what response to pick on test questions (i.e. Select True for the next question) and if they get the test questions wrong you could throw them out. Individuals who later changed there mind fall into this category. Early in an election, there are going to be undecided individuals and individuals who later change their minds. Undecided voters are likely a large driver of polling error because it is unknown whether they will vote and if so for whom. Sometimes the people who decide later in the campaign vote differently than other deciders and this is believed to be a large factor in 2016 polling error. If a poll has a higher percentage of undecided voters thant the difference between the candidates, this indicates that the race can be competitive regardless of the margin of error.

Miscoverage

Make It Fit Tight Shoes GIF - MakeItFit TightShoes GIFs

There are a few ways to get a sample. Ideally, your sample is random but there are only a few ways to recruit survey respondents: call random phone numbers, select random addresses, or “randomly” recruit people on the internet, or for exit polls stand outside of polling locations and ask every nth voter. All of these sources of data are not exactly representative of the American voter. Selecting random addresses can be random and representative of the American adults but people move and mail samples are typically used to recruit people for future phone or internet polls because mail is slow. Not everyone has a phone or internet access. Statistical theory often assumes you have a perfect source to draw a sample from. We don’t have a perfect source, so we have a little bit more error. And standing outside of every single polling place is obviously impractical and it’s hard to select a subset that will be completely representative. When we use a sample that doesn’t exactly fit we don’t always get perfect results.

Conclusion

What Does It Mean What Does It All Mean GIF - WhatDoesItMean WhatDoesItAllMean GIFs

I haven’t listed every source of polling error but I wanted to give a few examples of types of error that help explain why margin of error doesn’t always match up with actual survey error.

You may be wondering if margin of error doesn’t provide the whole predict what do we know about polling error? Are polls reliable? What is the difference between margin of error and predicted error on average?

Thankfully we have a great large database from Huffington Post’s Pollster that can help us answer these questions. We have over 5,000 state-level presidential polls from 2008-2016 and over 3,000 have a listed margin of error. In most cases we care about the difference between the democratic and republican candidates as the main metric because it tells us who is leading. We know that the margin of error is double the standard margin of error because margin of error is for one candidate. For about 4 in 5 polls the true election margin was in the interval in polls the last 60 days before the election. But in 1 in 20 polls the observed error was 5 points higher than the margin of error. A key thing about this data set is the polls are not evenly distributed across years or states.

Statistical models like the ones I commonly build can help to predict a polling error given information about where, when, and how the poll was collected. While these models are helpful, they also have uncertainty. Usually polls can be used as signals of what races are competitive, but can’t always predict winners. It is helpful to take a conservative approach when looking at a poll and acknowledging the potential sources of error.

The key thing to remember is that polling is one of our best tools for evaluating public opinion in general. Sometimes people use other types of models to predict elections, but for non-election public opinion questions about policy or presidential approval, polling data is required for statistical analysis.

How to Interpret Election Polls

This is the first of my approximately weekly posts I’m planning about the 2020 election.

As election day approaches, polls are going to become more prominent. It’s important we carefully interpret polls. I suggest you stick to focusing on polls from poll aggregators (like FiveThirtyEight or Real Clear Politics) or those tied to prominent news organizations. Polls brought up by polling experts (myself included) are typically going to be good sources. But you can encounter polls out in the wild that are complete garbage and you should be skeptical of a poll from a website you have never heard of. I’m going to briefly talk about three things you have to always consider when you analyze polls:

  1. Margin of error
  2. Polls are not predictions
  3. Outliers happen

Margin of Error

The margin of error is probably one of the most misunderstood polling concepts. The margin of error comes from a statistical formula that the natural randomness that comes from estimating a proportion for an entire population with a small sample. The margin of error is meant to be added and subtracted to a single candidate’s support. When we are talking about US elections, we normally care about the difference between the democratic and republican candidates (sometimes called the margin and written as Trump +x or Biden +y) and to examine that we must double the error. The reason for doubling the error is because in a poll with a margin of error of three points, Biden could be underestimated by three points, and Trump could be overestimated by three points, which leads to a six-point gap. The margin of error doesn’t cover the rare scenario where a respondent lies or makes a mistake. The margin of error calculation assumes the individuals who respond to a poll are not that much different than the population we are aiming to poll, and we can reach every member of that population, which isn’t exactly the case. The margin of error underestimates the polling error. It’s hard to quantify before an election how much margin of error underestimates the real error, but it is typically less than one percentage point in from my analysis I did on polling error (details will come later). If the difference between two candidates is less than double the margin of error, the poll does not provide enough information about who is winning, and this signals the race is too close to predict from just that poll.

Polls are not Predictions

Polls are not designed to predict elections. Polls are designed to estimate the percent of voters who support each candidate and what percentage of voters are undecided when they are conducted and not necessarily election day. The margin of error estimates the error between the poll and the true support for the candidates while the poll is conducted. People change their minds occasionally, and it’s hard to predict what direction undecided voters will go. A good guideline from research (taken from this book and replicated my own analysis) is that polls are not predictive of the election day result until labor day weekend. The predictiveness of polls improves over time. For example, my model will start collecting data on September 6th and will hopefully be fit by September 22nd, which is about 45 days before the election.

Outliers happen

Occasionally we will see a poll somewhere that is different from other polls. Strange poll results do happen, and you can’t say there has been a change in the race until multiple polls from different pollsters have similar findings. You should look at multiple polls when you check the state of the race. I like to look at two poll aggregators: FiveThirtyEight and RealClearPolitics. FiveThirtyEight is where I am planning to get my data from for my model. The tricky part about comparing two polls is you have to add both their margin of errors together to compare single candidates and double that number if you want to look at the difference between two candidates. Consider Poll A had Trump at 45 and Biden at 48 with a 3 point margin of error, and Poll B had Trump at 48 and Biden at 42 with a 4 point margin of error. The difference between Trump and Biden in Poll A is -3 with a margin of error of 3*2=6, and Poll B is +6 with a margin of error of 8. The margin of error to compare these polls would be 6+8=14, which means if the difference between the polls is less than 14, we would say that the polls aren’t showing statistically different results and the difference we observe could be explained by random sampling error. A lot of times, outliers aren’t really outliers after you adjust margin of error to fit your comparison.

Now you have learned some basic tools to critically analyze polls on your own to follow the races that matter to you. There are many more things you can do with polling, and the analysis I do is far more complicated than this. I’ll continue to write about polls and the election if you want to learn more.

The Polls Might be Wrong on Tuesday, but Here’s Why Thats Ok.

I’m going to preface this by saying I am writing this on the Friday before the election.  I don’t know if the polls are going to be wrong on Tuesday,  but I want to be proactive.  After 2016,  I learned that there were people who didn’t understand the uncertainty about polling and election models.  I also watched the attacks on many of the leaders of my field for their alleged partisan bias that caused them to underestimate Trump.  I can’t speak for other people political motivations, but the models are polls built using sound statistically methodology.

The fact is that polls have uncertainty.  They can be wrong and sometimes will be wrong for a few reasons.  Polls have huge nonresponse rates,  for example in the New York Times live polls you can see that usually only 2-4% of the people called answer.  And since those that don’t answer can be different than those who answer the polls the polls can be biased based on nonresponse.  Nonresponse could be easily fixed if more people answered their political calls or completed online surveys that they are chosen to participate.

Secondly, the structure of polls relies on assuming that individual people favor the candidates at similar rates and that one voter is not affected by other voters.  This assumption is for convenience because if you don’t have this assumption is practically impossible to estimate a margin of error.  So polls usually make this assumption,  which that an interpretation of 95% of all polls contain the real result in the margin of error is an overestimation of the certainty.

A heuristic I like to use is doubling the margin of error because that roughly represents the true error of polls.  One thing you will notice in this election is that a lot of the polls are close.  This means that we can not be sure who runs in quite a lot of races.  In the Senate, about four races (ND, NV, MO, FL) are to close for the polls to predict the winner with a high degree of certainty.

I expect that my model will have an average error of about 3-4 points.   Some of the error is going to come from bad estimates in noncompetitive races with limited polling,  but in the competitive races, I should be off (hopefully) by 2-3 points.  Which means it would not be surprising for me to incorrectly call 2 to 3 races,  but on the other hand I could be completely right or miss four races and not be surprised.

Election prediction is an inexact science,  and while pollsters try our best, since elections have uncertainty,   we will be wrong sometimes.  But for me at least,  I predict because I love the challenge and trying to make sense of a complicated event.  I will be wrong sometimes,  but when I’m right its a great feeling to have defied the uncertainty that makes my job difficult.

What a Pulmonary Embolism Taught Me About Statistics

On May 3rd, 2017, I was released from the hospital following an overnight stay for the treatment of a pulmonary embolism.  I am now almost fully recovered.  I think this experience is a great opportunity to teach statistics through a real-life example. I learned three things from this experience.

Vastly different fields can have the same underlying statistical processes

So far I have worked almost exclusively with political science data. My research is about how to estimate a proportion from a sample and how to compare it to other proportions.  When my doctor told me I might have a pulmonary embolism,  I wanted to see the data for myself.  So I read the journal articles, FDA case reports, and any data I could find to try to get an estimate for the chance I had a pulmonary embolism.  What I quickly realized is that the data about adverse drug reactions had similarities with the political science data I was familiar working with.   The data had issues with nonresponse bias and limitations due to unideal sample sizes.  Although political science and pharmacology are very different fields the share similar kinds of statistical problems.

Bayesian statistics is a powerful tool in many fields

Through this process, I saw how Bayesian statistics could help solve a difficult and important problem.  My doctor came by and saw me during my brief hospital stay.  She talked about how while she knew that it was unlikely any random woman in her twenties would have a pulmonary embolism,  but the details of my case suggested that the probability I had a pulmonary embolism was significant.  In short, the Bayesian mindset is about incorporating your prior beliefs and adapting them in the presence of additional information.  I don’t think my doctor used Bayes Theorem (the formal formula for estimating a probability given prior information), but she used Bayesian reasoning.  She had initial beliefs about the cause of my symptoms, and she updated her beliefs when she got new information  (like lab results).  This is probably normal reasoning for a doctor trying to diagnose a patient, but it showed me how Bayesian statistics could be applied to other fields.   A more formal use of Bayesian statistics would provide even better information to estimate probabilities.  I always knew Bayesian statistics could be useful in other cases besides politics, but this experience showed me a new area I am interested in researching.

 I am interested in other fields to apply statistics to besides politics

I wish I could have discovered my interest in biostatistics without a life-threating medical event, but I am glad.  I was exposed to a problem that is important and would use some of the same techniques I was exposed to during my work on political science.  While I still love political science statistics, I feel like I have now answered the question on what I can research in years where is no major election.  I enjoyed reading clinical trials and studies and analyzing their statistics.  Maybe someday I can even study how to improve statistical methods to prevent and diagnosis pulmonary embolisms like mine.

Six months after returning home from the hospital,  I am grateful that God has found a way to use my PE for good.

 

Data Sharing

Last semester I took a research ethics class.  I wrote a paper on preregistration and data sharing in academic research. I decided to modify the paper into two blog posts. Here is the first part on data sharing.

Statistics is the study of uncertainty.   Any research study not involving the entire population of group will not be able to provide a definite conclusion with 100% certainty.   Conclusions can be made with a high degree of certainty (95-99%) but false positives and false negatives are inevitable in any large statistical analysis.  This means that studies can fail to make the right call, and after multiple replications the original conclusion may be overturned.

One way to improve the statistical integrity of research is to have a database of the data from non-published studies.  Ideally, this database would be accessible to all academic researchers.   A research would then be able to see the data from other similar studies.   The research would then be able to compare his data with the data from the other studies.  At a significance level of .05,  approximately 1 in 20 studies that were statistically significant were a false positive.    This number applies to theoretically perfect studies that meet all the statistically assumptions used.   Any modelling error increases that rate.  With each external replication of a study the probability of a false positive or a false negative greatly decreases.   Grants from the National Science Foundation1, and the National Institute of Health2 currently require that data from the funded studies be made available to the public after the study was completed.  But not all grants and funding sources require this disclosure.    Without an universal requirement for data disclosure, it can be difficult to confirm that the study and the results are legitimate.

Advocates of open data say that data sharing saves time and reduces false positives and false negatives.  A research can look at previously conducted studies and try to replicate the results.   The results of the data can then be recalculated by another research to confirm accuracy.   In a large study with lots of data it is very easy to make a few mistakes.  These mistakes could cause the results to be misinterpreted.   Open data can even help discover fraudulent studies.  There are methods to estimate the probability the data is fraudulent by looking at the relative frequency of the digits.   The distributions of the digits should be pretty uniform and in one case the data didn’t look quite right.  In 2009, Strategic Vision (a polling company) came under fire from potentially falsifying polls, after a Five Thirty Eight analysis3  discovered that something didn’t look quite right.  This isn’t an academic example, but open access data could prevent fraudulent studies from being accepted as fact as in the infamous vaccines cause autism study.  The statistical analysis of the randomness isn’t definite, but they can raise questions that prompt further investigations of the data.   Open data makes replication easier. False positives and false negatives can cause harm in some cases.  Easier replication can help confirm findings quicker.

 

Works Cited

[1] Public Access To the Results of NSF-Funded Research. (n.d.). Retrieved April 28, 2017, from https://www.nsf.gov/news/special_reports/public_access/

[2] NIH’s Commitment to Public Accountability. (n.d.). Retrieved April 28, 2017, from https://grants.nih.gov/grants/public_accountability/

 

[3] Silver, N. (2014, May 07). Strategic Vision Polls Exhibit Unusual Patterns, Possibly Indicating Fraud. Retrieved April 28, 2017, from https://fivethirtyeight.com/features/strategic-vision-polls-exhibit-unusual/

We Don’t Live in Statsland

Statsland is a magical world that exists only in (certain) Statistics textbooks. In Statsland,  statistics is easy.  We can invoke Central Limit theorem and use the normal distribution when n is larger than 30.   In Statsland we either know or can easily determine the correct distribution.  In Statsland 95% confidence intervals have a 95% chance of containing the real value.  But we don’t live in Statsland.

The point of doing statistics is that it would be too difficult (or impossible) to find the true value of a population.  You aren’t likely to find  the exact value, but you can be pretty close.   In a statistics textbook problem, you probably have enough information to do a good job of estimating the desired value. But in applied statistics you may not have as much information.  If you know the mean and standard deviation of a population you do not need to do much (if any) statistics.  Any time you have to estimate or substitute information, your model will not perform as well as a theoretically perfect model.

Statistics never was and never will be an exact science.   In most cases, your model will be wrong.  There are no perfect answers.  Your confidence intervals will rarely perform as they theoretically should.  The requisite sample size to invoke Central Limit Theorem is not clear cut.  Your approach should vary on the individual problem.   There is no universal formula to examine data.   Applied Statistics should be flexible and instead of rigid.   The world is not a statistics textbook problem, and should never be treated as such.

 

Coincidences: A Lesson in Expected Value

As I followed the election I noticed the frequent mentions counties (or cities) that have been known “predict” the presidential election winner. The idea is that a the winner of a certain county has matched the winner of the election for multiple elections. Let’s look at county A for an example. To simplify things lets assume the odds of predicting a winner in a presidential election are 50-50. This would mean that the probability of getting 8 elections right would be 1 in 256. This means that it is unlikely that county A would predict the election by chance. But what about the rest of the counties in America? There are over 3,000 counties in America (according to an economist article found here: http://www.economist.com/blogs/economist-explains/2016/11/economist-explains), so we can expect on average for about 12 of these counties would have “predicted” the winner of the presidential election for eight elections.

Rare events happen all the time. Rare is not impossible. Let’s say that there is a (hypothetical) free sweepstakes with a 1 in 100 chance of winning $100. It may not be likely that you specifically win, but if all your Facebook friends enter the contest someone you know is probably going to win. If you have at least 99 Facebook friends it is likely that you or someone you know will win the sweepstakes. You may think its a coincidence or luck, but it is really math. Expected value can’t tell you who is going to win, but it can tell you someone you know is likely to win. Now expected value is not a magic bullet. You may have 0 friends win or 2 friends win, but the most likely event is that someone will win. Unfortunately (legit) sweepstakes like this don’t exist, but it is a good example of how your perception of probability may not match reality. Another example is it probably going to rain 1 in 10 days where the probability of rain is 10%, but it is easy to pretend like it never rains when the probability of rain is 10%.

You may wonder why expected value matters. But it’s actually quite important when looking at everyday events. Sometimes it is easy to underestimate the chance that something odd or rare would happen. You may think it’s odd that runs when the meteorologist says the chance of that happening is 10%. Or that it only takes 23 people to have a 50% chance of there being, two people with the same birthday (details here). It is easy to forget that once in a lifetime event do happen once in a lifetime. How you think about probability is important. So before you yell at the TV meteorologist that said there was a 10% chance of rain but it rained, try to remember that unlikely does not equal impossible.

Models May Fail but Statistics Matters Anyway

The 2016 presidential election brought attention to the limitations of Statistics.  Most models predicted a Clinton win but Trump will most likely be the president (the results are currently unofficial and recounts are in progress but most experts believe that Trump will be officially elected president). However all models are not 100% certain and the goal of statistics is to find the most likely event.  I have spent the last few weeks reflecting on the results and what this means for the field of political science statistics.  Recently I read a book by David Salsburg called: The Lady Tasting Tea: How Statistics Revolutionized Science in the Twentieth Century. It’s a history of sorts of how the field was developed and then applied to science.  While an exact date of the beginning of statistics is hard to pinpoint the first journals and departments were founded in the early twentieth century.  Statistics is a young field and is constantly growing and evolving as more data and situations are studied.  In the beginning some of the problems may have been trivial, but it is important to try to understand the world around us. Collecting data from an entire population is incredibly difficult and sometimes impossible, so methods of estimation were created.  You may wonder why prediction is necessary or helpful.  After all eventually the election happens and the president is chosen, so why do we care about knowing this in advance?  Why does prediction matter?  Statistics models and research is not just about what is being studied but about creating better ways to understand the world around us.   We can begin to better understand things like the opinions of the people, development of diseases,  and the economy.  Statistics can create better government, better medicine, and better education, and a better world.  If we can understand how polls measure the voting habits of the American people, then we may be able to get a better picture of citizens views on multiple issues and candidates.  If we can help understand how diseases like cancer behave, then we can create better more individualized medicine.  If we can understand how individual students learn and what they know, then we can create a better educational system.  Statistics isn’t perfect.  Statisticians can disagree and still both have valid models and reasoning.  The data may be imperfect and incomplete.  The model may be wrong.  The experiment may seem trivial and unimportant. But there is so much potential for the field of Statistics to change our world.  Just because prominent statisticians like Nate Silver may not have seen a Trump presidency as the most likely event doesn’t mean that the field should be discounted.