How To Build An Election Forecast – Chaz Nuttycombe

Chaz Nuttycombe:

For someone who is looking at state legislative districts, I love internal polling because there's no such thing as state legislative district public polling.

Eric Wilson:

Welcome to the Campaign Trend podcast, where you are joining in on a conversation with the entrepreneurs, operatives, and experts who make professional politics happen. I'm your host, Eric Wilson. Our guest today is Chaz Nutty comb, director of CN analysis, an election forecasting website, and CN analysis is the first ever American election outlet that created forecasts for state legislative districts in chambers across the country, and remains the sole group dedicated to this work. I know I found his forecast in Virginia particularly very helpful to following our elections here. In this episode, we discuss the process behind building election forecasts, what they mean and how they're being used on the campaign trail. Alright, so Chaz, I want to tie one arm behind your back. Okay. If you had to make a forecast, it could only know one piece of information, what would it be and why would you choose that one?

Chaz Nuttycombe:

I would go with just pretty much the previous election results. Polls obviously can have misses. I think polling is still informative, especially on the state legislative district level. I think it can be pretty informative and seeing what trends are going on and whatnot. But the main thing that our forecasts look at, especially on the state legislative level, is how did these districts vote, not just in the legislative vote, but how did it vote for every single statewide election in 20 20, 20 22? Not just how it voted for president, how did they vote for auditor? How did they vote for attorney general? Governor? That's the most important thing, is seeing, okay, for every single race on the statewide level for this district, this is how it voted. And taking the average for a year, creating a trend from one year to the other to create trends. I mean, there's a whole bunch of things that you can do with actual election data, especially on the district level for statewide elections on the district level.

Eric Wilson:

So it sounds like past performance is the best indicator of future results.

Chaz Nuttycombe:

Yes. Yeah, absolutely. Obviously, the legislative vote is the most important, but say for example, you have a, say Frank Burns Pennsylvania State representative who is in a dog fight this year, who represents Johnstown, Pennsylvania. He Pennsylvania, he represents a very red district and he's a Democrat. If Frank Burns retired, then that would be a safe Republican pickup. How do we know that? Because not even Josh Shapiro was able to win that district in the race for governor and only even think Tom Wolf was able to win his district in the race for governor. Previous legislative vote is pretty important. Loss of incumbency or the incumbent running is pretty important as well.

Eric Wilson:

Got it. So obviously you've got a lot more at your disposal. So just walk us through the other data that you take into account when you're building your forecast, and maybe give us a sense of how much weight you give that there's polling, there's campaign finance. What are the things that you're looking at and how important are they?

Chaz Nuttycombe:

Sure. So obviously election data carries most weight. A previous performance carries most weight. Other data that we would look at is if there is internals that sometimes, depending on the state, I can get my hands, hands on, especially if I'm able to get it across all districts and I can compare it with public polling, then that really helps our forecast accuracy. So internal data that is pretty useful and obviously incumbency or lack thereof. We've already touched on campaign finance. We look at that qualitatively just because it would take some research and time to figure out, okay, how much in each state is campaign finance really having some weight? Especially because we only really track candidate campaign finance in all the states. We look at it how much has each candidate raised? Not independent expenditures and whatnot, because a little harder to track, especially on a state by state basis, but we collect that.

There are other things as we research for candidate quality or has there been a scandal with this incumbent or the challenger? And then how much, if I'm able to talk, talk about the qualitative side as I talk with people here are working on these campaigns, if I can say, how about this district? And they say, oh, we're not playing for that. And the other side says, yeah, we're not worried about that. Then that shows like, okay, this is, even though this district might say it's competitive, in instance, this, it's not really that competitive because neither side is really worried about that district or really putting much resource into that district, and thus it's more likely to go with a natural partisan lien. So those are the things off the top of the dome that I can think of that we put into forecasting for what kind of data we look at.

Eric Wilson:

So you touched on a topic that's pretty important right now, which is public polling, usually from universities compared to the campaign's own internal polling. Why do you prefer the internals over what you're seeing from public polls? Well, I didn't say I prefer,

Chaz Nuttycombe:

But I will say obviously. Okay, sorry. I will say

Eric Wilson:

That I prefer them.

Chaz Nuttycombe:

That's fair. But internal polling is no better than public polling. I mean, it's still the same margin of error and everything. I think that there is this kind of, especially in DC or just people who get their hands on internals think they're so, so much better than public polling when really there's no data to back that up. But for someone in my position, for someone who is looking at state legislative districts, I love internal polling because there's no such thing as state legislative district public polling. And it sucks. And that's something you didn't get to mention at the top of the forecast is that we're starting up an organization called State Navigate pending 5 0 1 C3 nonprofit status that wants to do the forecasting, but also getting a whole bunch of other stuff, including trying to conduct polling and public opinion research on the state and even district level because yeah, I mean, when I talk to people off the record on both sides and I'm able to factor in on the back end for this internal polling, that's great because there's no such thing as public slave district polling.

And then of course I'm able to, after I've looked at, okay, what is the average bias using five 30 eights pollster rankings or old pollster rankings had mean reverted biases. Having that in and then being able to compare to public polling to see like, okay, is there a great disparity here? I take into account all polling. I think all polling is great, but to me specifically the polling that I really need is internal polls because there's no such thing as state legislative district polling because unfortunately, this is a level of government that is very much overlooked in my opinion.

Eric Wilson:

So Chaz, one thing that jumps out to me in our conversation, obviously I've been following you since you started doing these legislative raises back in 2017. I was working on Ed Gillespie's gubernatorial campaign at the time, and so obviously we know that that was not the outcome that certainly I had hoped for, and I've always known you to be a data guy. But what I'm hearing is that there's a lot of almost like journalistic reporting where you're talking to your sources on these campaigns. Expand on that a little bit. Do you think of yourself more as a data scientist or a reporter or some blend of both?

Chaz Nuttycombe:

I would say that mainly a data scientist. I obviously love quantitative and qualitative research. I think it's very, very important. I think you need to take into account both. I don't think you can just do numbers, right? Because I mean, especially on the state level, on the state legislative district level, there can be things that can't be quantitatively factored. How are you going to quantitatively factor Mark Robinson calling himself a black Nazi? You can't. We can look at the polls and say, okay, Robinson is down by 15 or 16. And I would also say something, I also consider myself as a recent graduate of Virginia Tech with a bachelor's in political science. I'm a political scientist, so taking my education and knowing that, okay, Robinson is down 15 or 16. However, because of biases like this in Robinson's race where you probably have a significant amount of respondents who are not going to say they're voting for Robinson because they are for whatever reason, afraid of being judged by the pollster for voting for a tainted kind of candidate with a scandal.

I wouldn't say I'm really so much a journalist. I do consider myself as someone who tries to educate the public on these state legislative races. This is my passion and this is a level of government that I really want to bring up to the forefront, especially on for its elections, for its incumbents. One of the things we're also trying to do at State Navigate is calculating the ideology of state legislators, seeing how likely someone was going to vote on a bill based on all the legislative votes, how likely was it that this person is going to vote for the bill? So I would say more so data science and political science. We can say a touch of journalism, but I don't really publish the conversations that I have with the people that are working on both sides. It's all off the record. It's all for my own sake of making sure that we have the most accurate forecasts that we can possibly create.

Eric Wilson:

So how are you balancing the vibes versus the numbers in your forecast?

Chaz Nuttycombe:

Sure. I mean, obviously we take much more weight into quantitative data than qualitative, but I think that it depends on what kind of thing that we're weighing and when it comes to qualitative factors, for example, a scandal tainted candidate on the state legislative level, the impact of that scandal is going to be a lot. It is going to be much bigger in a midterm year than in a presidential year. Why turnout is higher, people are looking at the presidential race. You have so many of these voters who don't really come out in midterms who are coming out for presidential, who are just going to default to whoever they're probably voting for president. The best example of this I love to give, is in 2020 when Rick Rover, who was accused of molesting his own children by his own children, won a competitive race in the Missouri House that Trump won maybe by three points and then over won by two when there was all this local coverage of the scandal come 2022 in a South Dakota Senate race, there is a similar situation, literally pretty much the exact same situation for a Republican candidate in a open Trump won competitive seat, but that district voted overwhelmingly democratic.

The Democrats also had a little bit of a good candidate there, a state rep and all that. But still, it was a massive over performance. So when it comes to all kind of qualitative factors, if you want to use the term vibes, it depends on what I try and not put so much emphasis on what people on the campaign are, I guess feeling. If you want to touch on vibes, it's really what data do they have.

Eric Wilson:

We're seeing a lot of yard signs. Sure.

Chaz Nuttycombe:

Right? Yeah, yeah, exactly right.

Eric Wilson:

I had one of those conversations this week and I was like, oh, okay.

Chaz Nuttycombe:

Yeah, exactly. I mean, that's not useful data. It's never useful data. But what could be useful is if they say something unusual dynamic of the race. I remember in the Prince William County race where all the early vote looks very good for Republicans in Northern Prince William County in District 21 and House of Delegates. But what we were seeing on both sides was that the Republicans were pretty much eating into their election day vote, and that the demo I was talking to, it just mentioned how in a district like that there might be local factors like a big minority population that's not really going to buy into ev, right? So these are things that you can, I wouldn't call vibes, but are important kind of just factors to keep in mind when doing data analysis that I think are really important. So yeah,

Eric Wilson:

Chaz, I'm hoping you can explain for us the various numbers and percentages in your forecast. So for example, today when we're recording this, you've got Kamala Harris in the presidential race at 52.5%. What does that mean and how should we think about that number? Sure.

Chaz Nuttycombe:

So if I recall correctly, the number is 10,000 simulations that Jack Ting the co forecaster at C Analysis runs our forecast through. And so based on the ratings that we have, which are handmade by me, Nevada is a tossup. Arizona, Georgia, North Carolina, Pennsylvania, Michigan, Wisconsin, all everything else goes pretty much to one party or the other. Some might be a little competitive like Texas or Florida. Each rating is made on a kind of 10 point scale. So Iowa has a one in 10 chance of going to the Democrats right now in our forecast, obviously toss up is a 50 50. So taking all the ratings and just pretty much running them through the model, the simulations and saying, okay, pretty much in 52.5% of outcomes have Kamala Harris winning the presidency in the electoral college. That's pretty much how it works.

Eric Wilson:

You're listening to the Campaign Trend podcast. I'm speaking with Chaz Nutty, director of CN analysis. We're talking about forecasting and everyone's impatient want to know the results, but Chaz, we're talking about some of the polling, the different factors on the ground, how much of an elections outcome is already baked in aside from the candidates and their campaigns. You obviously mentioned there's past performance that you take into account. There are macro factors like the environment. How much do you think is dependent on the candidate versus the seat?

Chaz Nuttycombe:

Sure. So there are thousands, almost 6,000 state legislative seats that are up this year in the 2024 elections. Only 1100 of them are at least somewhat competitive. We'll get most of them right, 70, 80 something percent of those competitive seats, hopefully knocking on wood here higher pretty much most districts as polarization is increasing, especially on the state legislative level, a lot of the outcomes are pretty much defaulting to how the district votes in its presidential performance or maybe just state level performance, right? For districts where the down ballot lien might be different from the presidency IE Connecticut or West Virginia, right? West Virginia, very democratic down ballot, Connecticut, very Republican down ballot.

When these incumbents either lose reelection or retire, the districts pretty much are, well depending on the district, depending on the district, I should say. So for example, there was, I think it was Pennsylvania House District one 50 or 1 51 in the Philly suburbs, and you had this Republican who just barely lost reelection in 2022, who was able to win this Biden plus 20 C. He won in 2020, and the lines are pretty similar. Come 2022, obviously they changed a bit, but now that district is safe democratic, so you don't really have as many of these incumbents who are having these huge over performances, and you have people pretty much just defaulting to how they're voting for, we can say what their baseline is, the partisan baseline of the district, and a lot of that is with the presidency, how they're voting for president.

So yeah, most district outcomes and state legislatures are easy to predict even on the seat level because people are just voting for how they would vote in a partisan basis. Now, obviously there are still very interesting things on the state legislative district level compared to the congressional when it comes to ticket splitters, and you do have, I think what's really fun about state legislatures is also state legislators oftentimes don't really have a comms team or just PR people or what have you. And so you'll have just, I mean, there is a Idaho Freedom Caucus state senator who said, go back to where you came from at a Native American running in his district wasn't his opponent because in Idaho they had the same districts, but it was someone running for State House, he's running for state senate, but point being, they don't have a whole bunch of great pr. And I think there's a lot more interesting on the state legislative level, like congressional, even though we're seeing state legislatures become more and more closely aligned with how they're voting for president, and this has been going on since the beginning of time. This is not a recent phenomenon, and Steven Rogers at the University of St. Louis has written about that. I mean, he wrote on it in, I think just right before the 2010 midterms when Alabama Democrats lost their majority in the Alabama legislature.

So it's no surprise that things were coming more closely aligned at the presidential level.

Eric Wilson:

Chaz, how do you account for unexpected or last minute changes in your forecast? So those October surprises or black swan events, is that something you factor in or you just say, Hey, look, I'm going to have to update the forecast now that we have this information?

Chaz Nuttycombe:

It depends. I think a good example of this could be last year in Virginia when Tara Durant had audio released, I think about a week or just under a week before the election, to where she said she would want stricter abortion laws of some sort. I don't remember what she said exactly, but that was something where I chose Tara Durant as the winner, and all the data that I had seen public and privately had shown that she was in a good position to win. And I didn't think that that was going to make the rounds enough with so little time before the election that that will have an effect on the outcome. But I do have something known as the Chris Hurst rule, and Chris Hurst was the delegate for Blacksburg Go Hokies. He was the delegate for Blacksburg, and the day the night before the election, he got caught by the Radford Sheriff for stealing his opponent's signs with his girlfriend at the time.

And so we moved that a column over. So yes, I always like to say these are the final forecast. I put 'em out usually the day before the election, and I will note the Christopher Rule, like, Hey, if there's anything last minute to where this is going to be playing on local news and whatnot all throughout election day, then yeah, there might be a last minute rating change. So it all depends on how salient, and this is just in my opinion, in my experience as someone who looks at say, let us save elections, how salient is this? We can say October surprise going to be. How many people are going to be actually aware of it? And this kind of goes back to what I was saying earlier about the Rick Rover thing or just state legislative scandals this year are not going to be as important as they were in the midterm. So we're not weighing that as heavily, obviously in a toss up district that might be able to persuade in the votes. And if it's especially a district where in Idaho they don't have as much obviously investment in the presidential race, and so you have people as a result more likely to pay attention to their, they're say, legislative race, especially if it's a small town district like this where I was talking about Idaho State Senate District six.

So it all depends on where, it all depends on when is the long and short answer to that.

Eric Wilson:

As we wrap up, Chaz, is there anything that the general public gets wrong about election forecasting and the work that you do that you wish could get cleared up?

Chaz Nuttycombe:

It's a good question. I think two things are big misconceptions about my work and my profession. One is that I think that people in the media and people in general think forecasters must be the best of the best, smartest of the smart, and they know what the outcome is going to be because they have some sort of magical crystal ball when that's not really the case, right? Larry Saboto has a crystal ball, but the crystal ball has been wrong before, even though it's a little

Eric Wilson:

Cloudy. Yeah,

Chaz Nuttycombe:

Sometimes, but still, they got 49 out of 50 in 2020, so they do good work, but they had the Hillary Clinton winning in 2016. Forecasters can be wrong. I mean, I've gotten some races dead wrong. Obviously it's quite different. I look at the state legislative level, and most of my resources are focusing on state legislative level, especially when I'm looking at so many districts. There can be big misses, especially in kind of smaller states. So I think that's one of the thing, and I think one of the things that I think forecasters need to be wary about, especially if they have a good year, and I'll say this as someone who said, yeah, I think the 2023 Virginia election was easy to predict, and I laid out why I thought that. I didn't say that. Oh, I knew what the outcome was going to be this entire time. It was pretty much like, well, I had a whole bunch of data and a whole bunch of experience, and this felt like an easy election to predict. That doesn't mean I think every election is easy to predict.

Eric Wilson:

I'll just say, I've got a lot of confidence in your ability to figure this out because you've obviously got something that people rely on and that they want. We'll figure out the product market fit, as we say. So I want to say thanks to Chaz Netcom for a great conversation. You can keep following his forecast@cnanalysis.com for the next few months, and then keep an eye on State navigate for their upcoming launch. You can learn more about Chaz and his work at the links in our show notes. If this episode made you a little bit smarter or gave you something to think about, all we ask is that you share it with a friend or a colleague. You look smarter in the process, more people hear about the show. It's a win-win all around. Remember to subscribe to the Campaign Trend podcast wherever you get your podcast, so you never miss an episode. You can visit our website@campaigntrend.com for even more. With that, I'll say thanks for listening. We'll see you next time. The Campaign Trend Podcast is produced by Advocacy Content Kitchen, a media production studio. I.

‍