A Workshop Series On Using Text As Data In Policy Analysis (Part 15)

>> Steven Davis: Welcome, everyone, to the Hoover Institution workshop on using text as data in policy analysis. We are delighted you could join us for our first show in 2023. My name is Steven Davis. Justin Grimmer and I moderate the workshop and select speakers. Tara Mahon makes the show run smoothly.

And Kathy Campitelli keeps us organized. Today's workshop features a paper on immigrant narratives by Kai Gehring, Joop Adema, and Panu Poutvaara. Kai is at the Wyss Academy at the University of Bern. Joop and Panu are at the IFO Institute and the Leibniz Institute for Economic Research at the University of Munich.

Here's our format. The authors will present the paper over the course of about 30 minutes. Well, Joop will be doing that, I think. Then we'll turn to discussion and Q&A. If you have a question or comment, put it into the Q&A box. And depending on the flow, we may combine questions, paraphrase, or ask you to state your remarks directly.

We'll run for about 60 minutes. After that, we'll turn off the recording and continue with a more informal discussion for anyone who wants to stick around. Joop, the floor is yours.

>> Joop Adema: So thank you very much for the invitation. I'm very happy to present my paper at your workshop.

I think I didn't know the workshop before, and I think it would have been nice to know. And I think you had some nice speakers that work in also narrative themes, such as Peter Andre, who was there in November or December, I think. And I think it's nice to kind of contrast different approaches.

And think I will focus a bit more on the methodology, but feel free to ask any questions. So the project is called Immigrant Narratives, and the aim is really to study narratives throughout space and time in Germany in the past 10, 15, 20 years. So a joint work with Panu, who's here, and Kai Gehring, who is at the University of Bern.

So the main motivation behind it is that immigration has become really a very polarizing issue in many societies in the past years, and especially so in Germany during the large refugee wave. And a lot of people have tried to study, okay, what are the determinants of attitudes towards migration and concerns about immigration and xenophobia?

So in the economics literature, a lot of people have tried to look at, okay, what happens when immigrants come to someone's locality? Do people update their attitudes? And this is kind of testing this contact hypothesis. So basically, do people become more favorable towards immigration and immigrants in their own country when they come into contact with people?

And a lot of these studies find relatively small effects of a few percentage points. However, this happened really during a wave which led to a large backlash towards immigration in general, but also to far right voting. And if you kind of compare these estimates from this contact literature to basically the huge surge in the negative attitudes towards migration, there are a lot we can't explain.

And one obvious candidate, because attitudes started worsening already before the largest inflow of immigrants in Germany, is to basically look at what happens in media. Because media probably kind of informs people about what is about to happen. And so just to give a little bit of motivation, the attitudes towards migration really worsened a lot.

So basically, there's a question in the German socioeconomic panel which basically asks, do you have large concerns, some concerns, or no concerns about immigration? And basically, the share of people answering that they have large concerns increased a lot. So basically, we study narratives about immigrants in Germany, which is an interesting setting because of the large inflow of immigrants recently, but also because of the strong regional newspaper markets.

And if we look at the economics literature, people have studied a lot of different aspects of migration, and the major one is, of course, labor market effect. But if we actually look at surveys, a lot of different aspects actually affect people's attitudes towards migration, such as concerns about culture, but also about compositional amenities, so schools and things like that.

So basically, these newspaper articles also consist of these different themes, and we basically try to capture this with our methodology. And to do this, we really take an approach which is quite different from other papers that look at narratives. Because they look often at different phenomena and try to figure out how do people explain this and what do people find compelling explanations?

And basically, we try to classify the narratives in broader themes and with what sentiment these themes are covered. And importantly, we do that at the sentence level because it's the smallest understandable unit of text. So we largely connect to this narrative economics literature. And I think you might have heard the motivation already when Poutvaara was giving his talk.

But basically, one of the main motivations is that narratives do shape, but also reflect society. So there have been a lot of papers that have kind of studied how narratives basically affect human behavior, but often these studies really want to study a single aspect or a single narrative.

And nowadays, I think people try to kind of build a bit more comprehensive approaches. So basically, these narratives about, for example, narratives about the macroeconomy, there's really kinds of, we have a phenomenon and how do people explain it? So, as I said, we don't want to look at a certain phenomenon, but we really want to look at, what narratives about other groups in the society prevail, so we want to study that.

And we want to build a methodology that can do that relatively well, and that can do it on a large set of data, because we have this regional component in Germany. But we also want to really look over longer periods of time and with a high frequency as possible.

So what we do is that we build a text-as-data method, which is on one hand built on custom-made dictionaries that fit within the themes. And I will explain in a bit what themes we identified. And we combine this with several natural language processing methods that are available in the Python package, which is called Spacy, which allows for named entity recognition, negation in sentences and other things, but we will come to that.

And if you ask me, okay, why are you looking at regional newspapers? Well, Germany is a bit of a specific case compared to especially other European countries, because there's a strong regional newspaper. So in some cities, the main regional newspaper really has market shares of more than 50%, which allows for a very kind of fine-grained component.

And later, if we want to link up this to, for example, attitudes towards migration, I think that's very useful because we can kind of study what are the local effects of local coverage. And we do this on a set of 70 newspapers, from which we obtained more than 100,000 articles.

And I think I will quickly skip over this, because we connect to a lot of literature on attitudes towards migration, which especially has been prominent in the past years. And it had been previously really not the domain of economists, but more of the domain of sociologists and media scientists, which really did structured readings.

And interestingly, we can also replicate some of Of the findings they find. So that, for example, specific immigrant groups are overrepresented in the media, as well as that there's interesting geographic patterns in Germany, which I'll come to later. I just want to quickly mention that there's kind of two aspects that we want to contribute to in the economics literature.

That's the determinants of regional news coverage. So there's a lot of papers that want to study, okay, how do newspapers actually, and other media outlets, what kind of stories they publish, and what is the demand of people for these stories? And another aspect of it is the effects of media coverage about immigrants.

So some people have studied what is the effect of TV coverage on attitude towards immigration. And from a paper from France, it has been found that, basically, the more the salience of the topic of immigration increases, the more polarized attitudes towards immigration become. And some of you are familiar with the paper of Zulalova, which basically studies the ban by the Associated Press in the US to use the term illegal immigrants.

And she finds that this has strong effects on attitudes towards immigration and towards immigration policy. So I will now dive into our method. So, our method consists roughly of three main steps. So, basically, we obtain newspaper articles, and then, basically, we go to the sentence level. And then at the sentence level, we want to determine, is the sentence about an immigrant or not?

So, basically, we need to have some kind of relevant keyword. So here we try to kind of show what kind of keywords we include. So it either includes a word from a specific dictionary, a foreign name, or whether someone is from a specific country, or whether it contains theme-specific words related towards immigration.

And one of the things that we can do with the python package, paci, is basically, we can track pronouns throughout sentences. So, basically, when sentence A talks about an immigrant, someone from a specific country, and the next sentence uses the pronoun he, we can basically realize that the next sentence is also about that vocal person.

And then as a second step, we basically want to use these theme-specific dictionaries to basically classify the sentences that are about immigrants into specific themes. So here you see the five main themes that we identified. The first is economy, which we further subdivide in work, welfare, and entrepreneurship, which are very distinct aspects of migration, which might be highlighted in different newspaper articles with different sentiments.

The second is foreign religion, which are all foreign religions, except for Christianity and Judaism. The third is cultural integration, which is broadly about integration aspects such as cultural integration, specifically, but also educational integration, which is kind of distinct from the economy perspective. The fourth is the immigrant criminality, which basically is crimes committed by immigrants.

And the last category is immigrants as victims, which is crime committed against immigrants, and other statements that are clearly anti immigrants. So basically, for each of these five themes, we construct dictionaries with words that are specific to the German context. And I think this is a strong approach, but it's also very labor-intensive.

And one thing that we do when we construct these dictionaries, we asked arrays to basically to each of these theme-specific words to basically assign a positive, a neutral, or negative sentiment. And that's our third step. And in assigning the sentiments, we really kind of abstain from using this kind of standard approaches, which are based on keywords, counting and counting the sentiment of specific keywords.

But we really want to make sure that we have as little bias as possible. So for example, we take into account negation in sentences, or we take into account weakening words. And I think with the python package paci, that's very well-to-do. And we really see and later that we will show our validation that we can do much better than traditional approaches.

So I would quickly want to kind of highlight the maintain reasonings behind why we chose this approach, because there's many approaches around that kind of big data approaches of text such as topic modeling or supervised machine learning. And I think one important thing that speaks for our method and against topic modeling approaches is that we define themes that we deem relevant to driving attitudes towards immigration and in the broader sense of the economics literature on effects of immigration.

For example, some of these themes that we identify, such as entrepreneurship, are relatively rare. So if you would do some topic modeling approach, we would not get entrepreneurship as one of the main topics. The second issue is that we look at sentences rather than articles. And if I have time, I can show you some descriptives later that the narratives conveyed at the sentence level are different from the article level.

So, for example, foreign religion is not very prevalent as the main issue of articles, but is prevalent in many articles that are about other main topics. And I can show you later if you want. And then, which I've already mentioned, I think it is assigning sentiment based on this very specific words, I think is very important.

And German is a specific case because there's a lot of composite words that would not show up in sentiment dictionaries, but that we, in our case, hard code. So that can be like specific to migration, and we can assign specific sentiments to those. So the dataset that we use is the following.

So we basically use pactiva. We download newspaper articles from all newspapers available in factiva, which is when you prevent double counting, and is approximately 70 newspapers that you cover for at least the year 2019. And basically, we download articles that go through a certain filter. So there needs to be at least some term related to immigration or immigrants or foreigners, and as well, a location in Germany.

So we really want to capture narratives about immigrants in Germany, but not in other countries or potentially coming to Germany. And because we have access to the university account, we basically have a limited sampling approach. So we downloaded 1,500 newspaper articles per 10-year period. So that's approximately a newspaper article every two or three days, for every newspaper.

And then we make use of a data set called IVW, which basically allows us for each of these newspapers to see how many readers there are in each of the 10,000 municipalities in Germany. So that's nice, because then we can also link this regional aspect very well. For example, we can link narratives about immigrants to how many immigrants from specific immigrant groups are living in these areas.

And in 2019, for the last year of our data, we have very good coverage. So we have at least one regional newspaper in more than 70% of municipalities and 45% of all sales. So we really capture a large part of the regional newspaper market. So, first, I will just give a few descriptives of the temporal averages of how prevalence immigration is.

The salience of specific narrative themes, and the theme-specific sentiments aggregated over all newspapers in the data set. So we focus on the balanced sample from 2005-2019 onwards. And I think this picture will not surprise many people. So, basically, approximately one and a half percent of all newspaper articles contained narratives about immigrants in Germany.

But in 2015, this rose rapidly to almost 3% when the refugee crisis started. And in all these articles that we have downloaded, we basically find that 9% of the sentences in these articles contain narratives about immigration. So we can assign them to any of those themes that we have.

And if we look at the pattern of these, the relative prevalence of these narratives over time, we basically get the following picture. So, you see basically the share of total narratives aggregated by Mon. And basically what we see here is that in the early time periods, the narratives about foreign religion and cultural integration dominate.

And this does not really take much over the whole time period, but we see that during and after the migrant crisis, narratives about immigrant crime became much more prevalent. And there's also some notable events in Germany which show up as strong peaks in the theme share of immigrant crime narrative.

And for example, here you see also a very pronounced peak in welfare concerns and in work concerns. That's when Romanians and Bulgarians were allowed to enter the German labor market unconditionally so that's January 1st 2014. And then we look at the average sentiment. So basically, we assign these positive, neutral, or negative sentiments, and basically we can aggregate that up for each of the month and we can show it over time.

So, if we look at the aggregated sentiment across all of these narrative themes, we basically see that the average is around zero, but becomes more negative during the migrant crisis. And if we look at the separate themes, we see basically that most of the narrative themes, the sentiment is relatively flat, apart from two.

So, sentiments about work have improved over time. And sentiments about foreign religion have been worsened because there was also a large debate in Germany about the immigrants being of Muslim background. And this is really visible in our results. And we see that sentiment in these newspaper articles worsened a lot.

And then I go on to the Human Validation. So basically, we built this approach and we basically built this team specific dictionaries because we had something in mind of what these themes convey. So, basically, we build up coding instructions for human coders to code it in a way as similar as possible.

So we hired 16 RAs, which were not involved in the first step of constructing these initial dictionaries, and we give them approximately 1%. We give them approximately 1% of our sample, and we let each sentence be coded by four human coders. Now, one thing that we found when we gave these instructions to people is that basically, and I think I can quickly show you that there was a large disagreement among human coders, what narrative theme specific sentence belongs to.

So, here on the left, we basically see the share of sentences classified. And as already mentioned, our algorithm, that's the cross here, identifies approximately 9% of all sentences as conveying one of these immigrant narratives. And on average, this is similar for the students. But some students code only 2 to 3% of sentences, whereas some other students code much more.

So, there's a large heterogeneity, and we see similar heterogeneity when we go to teams. However, we see that the cross here is always reasonably within the bound of the students. So, basically, the classification rates per narrative theme don't differ that much between students and our approach. And then basically, we do our approach, and we can compare it to how a dictionary-based approach would perform, and we can kind of add these separate steps that we do in our approach.

So the basic thing that we do is basically just dictionaries. And then we basically see, with only these dictionaries, we only pick simple dictionaries with common migrant terms. We only pick 29% of the sentence that our final algorithm picks up. But this get better when we add the dictionaries.

And the last step, we add the pronoun function, which is able to track. In practice, we found the pronoun function was relatively hard to use, and so only 2.8% of sentences that end up as classified narrative about immigrants is contributed due to the pronoun function. And basically on the bottom, we basically see how our approach does compare to simple dictionary approach.

So on the left, we basically see the share of sentences that are correctly classified where we use basically the sample where all four human coders agree as a ground truth. So, basically, our approach selects almost 97% of sentences correctly, whereas the dictionary-based approach picks up 87%. And if we then look at what we call the complete misalignment rate, which is actually being how many sentences are picked up that are coded as not to be about narratives about immigrants by all four human coders, we basically see that we only pick up a bit more than 2% of sentences, whereas a simple dictionary approach would pick up much more.

And this is maybe because there is some. So basically this could be sentences that are about immigration, but are not about any of the themes. So there's a lot of false counts. And I think this kind of speaks for our method, because basically you could consider this as a false positive rate.

And if you look at the sentiment, we also see that our approach does better. So here on the X axis, we see the absolute deviation in sentiment. So the sentiment of minus 1 0 or 1, of our human approach, compared to the sample where all human coders agree on whether the sentiment is minus 1 0 or plus 1.

And basically we see that we are correct in 67% of cases, whereas these dictionary-based approaches, Santws and text blob, which are specific for German language, only agree with these humans in a third of the cases. So now I want to quickly turn to a few applications, which we can do with basically our data set that we constructed, right?

So we can aggregate these, Prevalence of immigrant articles. We can aggregate what share of all immigrant narratives are about the specific themes, and we have the theme specific sentence, and we can correlate that to specific factors. So first, we just aggregate it up according to newspaper Broad Geographic Group.

We identified four geographic groups. First of them is the five national newspapers. The other one is, I think, four newspapers that are based in Berlin and serving the Berlin markets. The rest newspapers that serve the rest of eastern Germany, and all the other newspapers, which is the biggest group that serve the western German newspaper.

And one thing, what we see on the left, which is basically the share of articles about immigrants in Germany. We basically see that in the beginning, there is already a stark difference between those newspapers from former eastern Germany to the newspapers from other groups, because they talk much less about immigrants.

So approximately 0.5% of all newspaper articles are about immigrants in east region east German newspapers, whereas this is almost 3% in the first period already. So this is 2012. And we see that this gap even widens a lot during the refugee crisis, where 6% of articles in Berlin and national newspapers are about immigrants in Germany, whereas this is still less than 2% in these eastern German newspapers.

And this is especially interesting if you consider that concerns about immigration are higher in eastern Germany, measured by all conventional measures, such as these attitudes towards migration in the socioeconomic panel surveys. And so we kind of try to understand what else is going on here. And we basically, when we look at the relative prevalence of narrative themes, we basically see that in the eastern German newspaper, much more of the narratives are actually about the economy and much less are about foreign religion.

And if we look at the right most panel, we basically graph what happens to the average sentiment in these four newspaper groups over time. And we also see that the average sentiment is not that negative during a migrant crisis in eastern German newspapers. This could, of course, be partially driven because narratives about work are much more positive, on average, than narratives about foreign religion.

So, basically, now we can go back or to a little bit finer level, where we look at the newspaper level. So basically, every dot in this graph is a newspaper. On the X-axis, we see the local share of immigrants in the coverage area. So weighted by newspaper readership of the respective newspapers, and on the Y-axis, we see the share of articles about immigrants.

And basically, the more immigrants are somewhere locally, the more newspaper articles about immigrants. I think this is not a surprise we find it. And if we are gonna look at specific narrative themes, we find something similar. So, basically, the vast majority of narratives about foreign religion are about Muslim immigrants.

So, basically, if we correlate the share of Muslim immigrants locally, which varies. So this is data from 2019, which varies from approximately 2% to 8%, where we define Muslim immigrants as foreign born citizens originating from Muslim countries. And we basically see that the more Muslim immigrants there are locally, the more the relative prevalence of narratives about foreign religion are.

And here in this graph, you see the red denotes newspapers from the former eastern Germany, the blue denotes newspapers from the former western Germany. So, basically, in terms of the number of immigrants, but also the number of Muslim immigrants, there's much less immigrants in former eastern Germany. And that partially explains that there's more newspaper articles about other themes in these eastern German regions.

And then I think I don't have that much time, but I think I will quickly go through this. So, basically, we are interested in what happens during the migrant crisis, and I think there are several interesting events that we can identify. So on the upper left, we just see the total number of articles per month, throughout all the articles that we have in a balanced sample from 2012 to 2020, and we basically see that there are a few very pronounced speaks.

So, for example, the labor market opening did not lead to many more articles about immigrants. But this peak in the beginning of 2015 coincides with when the refugee crisis really became apparent. There became groups against, like a group called Pegida, which had a strong anti immigrant sentiment, strong anti Muslim sentiment, as well as the terrorist attack on Charlie Hebdo in January.

And another event that we can identify is an event in Cologne during the evil 2016. And I think this is a very interesting event, first of all because it was strongly geographically bound. But also because this really shifted people's opinion about immigrants in Germany and led to a strong increase in narratives about crime, but later also about foreign religion.

And basically on the lower left, you can see how the aggregated sentiment develops over time. And I think a few things become very clear. So during the start of the migrant crisis, there was a huge drop in sentiment, as well as in early 2016, during this new year's event, as well as during the end of 2016, when there was a terrorist attack in Berlin.

And we can basically decompose this change in sentiment into our narrative themes and our theme specific sentiments. So we can ask ourselves the questions, okay, sentiments worsen in what kind of dimension does this happen? So, basically, we try to decompose this, keeping the share of narrative themes fixed as well as the theme specific sentiments.

And then we can kind of look what would have happened if, for example, the shares of narrative themes would have been fixed and the sentiment would have been worse. So basically, we can kind of do a decomposition analysis. And we basically find that a large part, so these are the pink purplish bars here a large part of the deterioration sentiment during a migrant crisis is driven by shifts between themes.

So basically, themes that are more negatively displayed become more prevalent, which is this foreign religion in architecture. So I think my time is almost up, so I will kind of conclude, but feel free to ask questions out about all the specific aspects that I've covered. So one major thing that we do in this work is to really build a methodology that allows us to kind of study narratives about groups in society and a group where we are interested in.

Because we're migration scholars, are immigrants in Germany, and we try to apply this to a large set of regional newspapers. And our main findings are that foreign religion and cultural integration, if you go to the sentence level, are really the themes that prevail. And one surprising finding that we.

Find that what we try, which we try and later work to dive a bit deeper into, is how why newspapers in eastern Germany seem to talk less about immigration, but also talk more positively about immigration. Another thing is that we can basically use this dataset to study how nationwide shocks are absorbed.

So I haven't showed you before now, but I can show you later, is that basically local conditions matter. The more Muslim immigrants are somewhere, if a nationwide shock happens, there's a differential response of newspapers with a low share of Muslim immigrants compared to newspapers with a high share of Muslim immigrants.

This is still work in progress. So we basically kind of have several ideas which we would like to pursue. But I think my time is up, so I can talk about that later in the discussion.

>> Steven Davis: Great. Thanks a lot, Job. Let me start with a few comments and questions.

One basic comment is it would be great to see an index that combines average sentiment with the volume of coverage. Unless I missed it, you don't show that. But that's kind of, you go back to the very first picture you started with, the evolution of attitudes towards immigration among Germans from survey based data.

Presumably that is affected by a combination of coverage, volume and average sentiment. It looks as if average sentiment is negatively correlated with volume, but it's hard. That's just what I infer from combining multiple pictures. Anyway, that's just a basic comment. You can think about how best to do that.

You could just take the product of volume average sentiment. One idea. Second, you know, back on, I think there's more to be done with your human coders in a few respects. First, you noted there's a rather remarkable range of human classifications of just articles at the first stage.

Are they about immigrant narratives or not? Then it's like 2% to 8% or something like that, very wide range. One possibility is some of your coders are really outliers and making errors at a higher rate than others. I would just as a robustness exercise, I would compute each coder's average pair wise agreement where with all the other coders, and I would drop the tails.

I take you have like 16 or 20 coders, if I remember right. I just dropped the two coders from each extreme, I guess. No, in this case, you just want to drop the coders who are, who tend to disagree extensively with other coders and then just redo your analysis.

And do you, for example, get better performance metrics on your automated methods? That's a very simple robustness check. Then a different point about your human coders. You sort of present things as the disagreement among coders is a bug of some sort, can also be a feature, and you could use it to assess whether there's some, whether it's.

The error isn't on the human side, it's just that there's ambiguity in the actual language. They could try to identify what are the characteristics of ambiguous language. So one way to do that would be to construct for each sentence, the average pairwise agreement rate among the coders who looked at that sentence.

And you have four coders per sentence, as I understand it. Typically you can construct the average. You get it. You get basically a metric of agreement among coders per sentence. Then you could take a number of approaches, but a simple one would be to try to ask whether a supervised machine learning algorithm could replicate, could do a good job, test a test sample and then a performance sample in identifying sentences that have high disagreement rates among coders.

If they can, that suggests there's something about the structure of the sentences themselves which is ambiguous. You could tell us about what that ambiguous. What are the characteristics of a sentence that gives rise to ambiguity in human readings or it could be ambiguity or heterogeneity in readings? That's true, heterogeneity.

Politicians are often accused of trying to speak to multiple audiences in the same sentence with dog whistles or coded language. So you have the opportunity, given that you've got four coders per sentence and thousands of these things, to actually try to characterize language that is either ambiguous or deliberately designed to send different messages to different groups.

Maybe that's another paper, but you've got the raw material there to do that, and it seems worth exploring. I'll pause there and see what reactions you have and then maybe turn it over to Justin.

>> Justin Grimmer: Yeah, thank you very much. I think the last point is actually very interesting, something we haven't thought about.

I mean, first we just saw this huge disparity and I think we gave them quite clear, we gave students quite clear instructions, but still we saw that there's a lot of variety. But I think it would be great to kind of look at, maybe we can just simply look at kind of what words prevail in the sentences where two or three humans agree, one, two, three humans agree, but not all of them and not none of them.

So I think that's very good approach. One aspect of what you mentioned we can probably not do because we have very homogeneous group of people, namely university students. I think it would be relatively hard to kind of see the different narratives, are they judged differently by different people?

I think that's, unfortunately not what we can do related to the.

>> Steven Davis: You can still make a start on that. You have about half of your coders are men, half are women. You can do see whether there's systematic differences between men and women. I think you said you have three immigrants among your coders.

Do they stand out in some way in their classifications from the other coders? That's an easy thing to check. It's a small sample. But if something noteworthy does emerge from that, then the next time you run an experiment like this. You might want to deliberately select for many immigrants among your coders to explore it in a more effective way.

 

>> Joop Adema: Okay, I have one more issue. So this is actually something I did today. But basically we linked up our data sets to the German socioeconomic panel. Then you get what basically correlates to attitudes towards migration. So there's this three point question in the socioeconomic panel. Do we have large concerns?

Do we have some concerns or do we have no concerns about immigration that here I use about as an outcome group. Basically it's just a regression on the time series where we basically use the lagged value of the newspaper statistics. So basically, the top is basically this percentage of articles about immigrants.

Then we basically look at the different theme shares. The omitted category is the economy theme share and the sentiments. Of the narrative themes, and what we basically see here is that percentage of articles about immigrants is very strongly correlated to concerns about immigration. But we also see that some of these themes, such as the foreign religion, cultural integration, immigrants and victims, seem to be negatively correlated with concerns about immigration.

And also we see sentiments about the economy become more positive, people are more likely to state that they have concerns about immigration. So basically, I mean, from this graph, the correlation between the prevalence of articles and attitudes towards migration is quite strong. But I agree with your point that kind of looking at this aggregate index or looking at the interaction between total volume of negative sentences and the total volume of positive sentences would be kind of something sensible.

So, I mean, in the end, we basically want to merge our data sets on a much finer level to the socioeconomic panel. So at the spatial level, where we basically can study, okay, those local coverage of newspapers actually correlates to attitudes towards immigrants.

>> Steven Davis: Justin?

>> Justin Grimmer: Yeah, definitely Steve and I should have crossed notes, because I wanna focus on the human coders as well, I'm gonna do my best not to repeat what Steve was saying.

So some of the analyses subset to where there's four coders to agree, and I just wanna say that subsets to the set of cases where there is a sort of clear answer. And very much building off what Steve said, I would really embrace the ambiguity here. I think that's a lot of what's interesting, and I would push it just a little bit further to think about, particularly with the sentiment classification, just as an example from American politics.

I forget who said it, but there was a political figure who said something like, if this immigration continues, there'll be a taco truck on every corner if a Democrat gets elected. And that was meant to have a negative sentiment, but I think many of us thought sounded awesome.

And you can imagine this sort of variability is a thing that you almost wanna embrace in the measure, so if there's a development, that development elicits very different reactions from different components of the population. That would have a lot of importance for understanding the effect of these narratives, but also for what's being conveyed in the reader's experience.

So I would almost push you to see if you could identify these sorts of ambiguous sentences from the coders and then almost run something like a survey experiment or a survey where you elicited reaction from a larger population. And see how that variation differed by, I would think, attitude towards immigrants as reported in the same survey.

I would be very interested in carrying out Steve's idea about using some sort of procedure to identify what is it about that sentence that you could then map in to get a sense of what that variation is? Cuz to me, that that's where a lot of the interesting parts of the debate pop up, some folks embrace immigration, other folks don't.

So the same sentence about an individual starting a business could be met negatively or positively, and I don't think we wanna average that out cuz the point is that variation. In a much more, last thing I'll say, much more mechanical, just thinking about disagreement. Agree 100% on calculating pairwise intercoder agreement, that's a thing you definitely wanna see.

If you look at the sort of these underlying themes from the immigrant classification, it seems like there's always someone who's at the tail. And I'm just a little worried about someone, maybe there's a different coder for every category who kind of gets fixated on it, and that happens a lot with coders.

And that would be a thing I'd wanna investigate, so someone could have relatively high agreement but still be an outlier just CUZ they get fixated. And I would wanna dive into that just a little bit more to know what's going on with that coder.

>> Joop Adema: So, yeah, no, I mean, we have looked into this, but we do, our focus was really to kind of build this kind of aggregate measures that we can then see what happens in the whole German debate.

But I think the point related to this, so we looked a little bit at this intercoder agreement. I mean, we don't really, it's a bit limiting because four coders is not that much, but we don't see that there's coders that stand out that much in terms of classification, but mostly in the propensity to classify.

So I think there are some coders that classify 20% of sentences, but they classify a lot of all of the things. So they kind of have, I'm not so sure how to kind of understand this because some people, just when they had saw something vaguely related to culture, they flagged it.

While others only flagged it when that was super explicitly said, this is when it's about a cultural aspect or even the explicit word mention of the word culture. So I think this is an interesting aspect, and I really kind of. Yeah, there can be very interesting things in this, but we have looked at it for now to a very limited extent, but I think it would be nice to do.

 

>> Steven Davis: Great, thanks. Hey, Tara, Elizabeth Elder has a question.

>> Tara Mahon: Hi, can you hear me?

>> Steven Davis: Yeah you're a little bit soft, Elizabeth.

>> Tara Mahon: Okay, I'll speak up. So thanks so much, I have a kind of applied question, I'm interested in this pronoun function you mentioned. It's not something I come across before, and I was curious about the fact it didn't seem like it maybe added too much to your predictions, you said maybe it was a little bit messy.

So I'd love to just hear a little more about where you land on that, why do you think it might have not added that much to your predictions? And do you think that's still, that method is worth doing in analyses like this going forward?

>> Joop Adema: Yes, thank you very much.

So I have to disappoint you on one part because our third co-author is really a specialist on the use of spacy. But I can quickly, so you can kind of try to set the sensitivity of the propensity to select a pronoun. And this is, I'm also not sure.

So this is specifically for German language, and people have cross checked these methods, so I think for English, it actually works better. And our method only, we found that it only selected the most, accurately, the pronouns that are most obvious. And sometimes so German has maybe a little bit more complicated sentence structure during English, so more complicated sentences, it didn't really pick it up.

And I think this is really a limitation, especially when you have these, what we saw that we have these kind of more like human interests based newspaper articles when, after they introduced the focal person. They don't talk about this vocal person explicitly anymore, but we refer in pronouns, so this sometimes makes it a little bit hard to capture this.

I think there is potential to it, but I think that in our method, it works okay because we add more sentences that we found to be relevant to, but it's not that much sentence. And especially when we just kind of looked at the articles that are really these kind of human interest articles, there were a lot of sentences that our method did not classify.

We looked also what the humans did in that case. And we see that some humans picked it up well, but some humans, and it might also be partly in attentiveness, did not do so. So I think there's actually, in specific types of newspaper articles, there's kind of room of improvement using pronoun functions.

 

>> Steven Davis: That was helpful. I could imagine that the usefulness of the pronoun function might vary a lot with writing style, including possibly between German and English, for as you suggested. But can I ask you about your immigrants as victims category?

>> Joop Adema: Yes.

>> Steven Davis: You didn't talk much about it in the talk, but I was puzzled by it.

Your discussion in the paper, it seemed to. Maybe I misunderstood, but it seemed to combine two very different concepts. One, that immigrants were sometimes victims, and it was put in a way as to perhaps foster sympathy for immigrants. But some of your criteria for inclusion in that category also seem to encompass people just making negative remarks about immigrants in a way to not elicit sympathy for immigrants, but to actually elicit hostility or anxiety about immigrants.

They both seem to be in this category. Or did I misunderstand what you're capturing? So maybe you can just first tell us how you classified sentences into this category, and whether my characterization of what your method is doing is correct.

>> Joop Adema: Yes. First of all, I completely agree, this is a weakness.

So when we did this approach, so basically, we really initially want to keep these things separate. So basically, using the natural language processing tools, we kind of, for all the crimes we want to figure out who did what to whom, and if the person who is subject of the crime, if the victim is the immigrant, then we basically kind of want to capture that team.

We figured out that this works relatively poorly, and we decided. So basically, among these sentences that are about immigrants as victims and things that are more explicitly related to anti immigrant statements or discrimination whatsoever, then basically these sentences are much more prevalent. So, basically, this is something that we want to improve upon.

And that's why we also in most of the analysis, which goes into sentiments, for example, we don't talk too much about immigrants as victims. But I completely agree. These are two distinct things that could really very differently drive attitudes towards migration. And one aspect is maybe just kind of reflecting what happens.

And the other one, namely statements about immigrants, is really the societal response in sense of what is being written in newspapers. And yeah, I think this is hard. We should improve on this.

>> Steven Davis: I take it from what you say that currently this category is not really fit for purpose, right?

 

>> Joop Adema: So it includes both these victims, but it's conflated with something else? Yes. So, I mean, yeah, no, I completely agree. I think in terms of discrimination about immigrants, maybe immigrants as victims is not the correct term.

>> Steven Davis: Okay.

>> Panu Poutvaara: Perhaps if I can add something so that, first of all, I should highlight that when we calculate aggregate sentiment, when we do not include immigrants, so that the problems from that category do not carry out to the analysis on aggregate sentiment.

But I would defend the term somewhat. So in a sense that this is capturing how a society treats immigrants or responds to immigrants. So even though it combines both crimes against immigrants and discrimination, it is still something which is interesting and I would say also informative, although I do agree that it does combine these two different things.

 

>> Steven Davis: Yeah. No, no, I'm not getting at the underlying concepts. I agree is quite interesting. It just seems like, yeah, I think we're in agreement. There's two very different concepts that are currently captured in this category. And I guess I gather you're working to figure out how to disentangle them.

 

>> Justin Grimmer: Do you have a sense of the amount of editorial discretion within a region? And so you can imagine what's going on here. There's events that clearly were driving coverage, and basically, newspaper feels obligated to cover these high profile events. But other times, there are stories that's just about how you assign your reporters, whether you're assigning a reporter to the police beat to see who's being arrested vis a vis, like who's opening businesses.

And you do have this interesting regional variation. I was curious, within the region, are you seeing a pro or anti immigrant newspaper pop up?

>> Joop Adema: Yeah, so there's huge variations. So I think there's a lot of variation across newspapers. So there's this typical tabloid newspapers, which are much more negative about immigrants.

And I think one thing that you're hinting at is, okay, to what extent are newspapers just covering the bigger news or what do they kind of put into their own effort, right? I mean, one way you could potentially look at this is to look at articles in different sections of newspapers.

So if you go to opinion pages or editorials or newsletters, letters to the newspaper, I think that would be actually very great to identify because then you also see a bit more a different margin of how newspapers want to cover immigration. And with factiva database, unfortunately, for some newspapers there's identifiers in which sections newspaper articles appeared, but for many, there's not.

And I think this would be very interesting to see, kind of. So when we showed this picture of newspapers in eastern Germany are much more positive. A lot of people said, okay, this is from kind of a backlash of journalists against a large negative trend in these eastern German societies towards immigrants.

I think it would be very great to kind of be able to look at, okay, what is the discretion of the editor. And I think this kind of newspaper session would be very interesting way to look into this. One way that we do have is maybe to kind of see the locality of the newspaper article.

So basically, if the locality of the newspaper, if it mentions places that are in the coverage paper of the newspaper, and there we basically see that these articles across the board are much more positive. So I think there's different ways we can kind of look at what newspapers just take from, say, newspaper agencies and what they basically actually produce themselves in concurrent.

 

>> Steven Davis: In a similar vein, it would be useful to just rank newspapers by their average sentiment, the average sentiment of their articles about immigrants, and then take that as something you want to try to explain not just in terms of location, but also the characteristics of their readership, the known political orientation of the newspaper and so on, at least in the US.

I don't know if this is true in Germany. News newspaper outlets differ greatly in their willingness to say negative or the propensity to say positive things about immigrants. I don't know, is the same thing true in Germany? It would be good to document that and quantify it.

>> Joop Adema: So I think one thing is very different.

So there is some newspapers that are single newspaper covering. So I think first of all, it's very nice to look at the determinants of this I think we should do this. The lack is a little bit of, we have a little bit of lack of information about, say, political stance.

And one reason behind this is that these local newspapers are really kind of focusing on delivering local news. And basically that makes them kind of not completely apolitical, but many of them are the single cover of a market. So I might expect that they want to kind of be around the median voter.

What we do see is that the most positive and the most negative newspapers in our data set are both newspapers that are actually in places where there is more competition. And I think this is kind of interesting and something we want to explore. So currently we have the 65 regional newspapers, which is like 40% of the bit bigger regional newspapers.

But if we get the whole, I think, from some other newspaper databases, we may be able to get almost all regional newspapers. And I think then we can nicely study this kind of newspaper market story about the determinants of dance against the words.

>> Steven Davis: Yeah, that'd be useful to do.

Okay, any other comments or questions, Justin or anyone else on the floor. If not, I'll let you wrap up and then we'll turn off the recording and go to the informal session. Any final words, Joe?

>> Joop Adema: No. I mean, thank you very much for the comments. I think these are very, very useful, especially the aspect of the interhuman coder.

I mean, that's not our aim, but I think there's quite interesting things to look into there.

>> Steven Davis: Okay, thank you so much. This was super interesting. I really enjoyed it and look forward to seeing how this work progresses.