What We Teach About Race And Gender | A Workshop Series On Using Text As Data In Policy Analysis

>> Stephen Davis: Welcome, everyone, to the Hoover Institution workshop on using text as data and policy analysis. My name is Stephen Davis. Justin Grummer and I moderate the workshop and select speakers. Tara Mahan runs the show in the background, and Cecilia Chen keeps us organized. After a long summer break, we are delighted to be back in action today.

We've got a paper titled what we teach about race and representation in images and text of children's books by Anjali Adukia, Alex Eble, Emileigh Harrison, Hakizumwami Birali Runesha, and Teodora Szasz. I hope I didn't massacre anybody's name too badly. I think Anjali is going to present. So just for the audience, so they know where we're going.

The authors have about 30 minutes to present the paper, then we'll turn to a discussion in Q and A. The videotape portion of this will run about 60 minutes. Then we'll turn off the camera and the taping. And for anybody who wants to stick around, we'll have a more informal discussion for another 15 or 20 minutes afterwards.

With that, Anjali, please take it away. The floor is yours.

>> Anjali Adukia: Wonderful. Thank you so much for having us, and good morning, everyone. It's a pleasure to be with you. As Steve said, my name is Anjali Adukia, and my other co-authors are here. Alex Eble, Emileigh Harrison. Hakizumwami Birali Runesha, and Teodora Szasz, will be joining us a little bit later.

And we'll be talking about representation in children's books. So the process of education and its associated books and curricular materials, necessarily and by design, transmit the values of society and whose space it is. And the presence and absence of different identities send messages which can contribute to how children view their own potential and the potential of others, which can then shape subconscious defaults.

There's been this sense that representation matters and these various calls for representation over time, but we don't really know, has this happened? And how do you even measure it? And to even say, well, what is representation? There's two kinds of representation as we see it. There's the present of different identities.

So whether various identities are there, so essentially in numerical accounting, but there's also how different identities are being portrayed. So, you know, you may have, you know, a book full of underrepresented identities, but if they are portrayed in stereotypical or reductive ways, that might be counterproductive to the goals of the person who's actually curating the content.

And, you know, there is a rich literature of traditional content analysis where people go page by page and do intensive manual labelling where they can code images and text. And what's nice about that is you can get both the whether and the how. But the problem is it's extremely labor intensive and costly, and it can be very slow.

You can also imagine leveraging a lot of these advances in artificial intelligence-based content analysis. So thinking about computer vision, where you convert images to data, and natural language processing, where you convert text to data. And the nice thing is, whether you have 100 pages or 100,000 pages, it doesn't really matter, because once you've trained your initial models, you actually can apply your models systematically throughout.

Now, of course, one of the problems here is that it's harder to measure complexity. And also it's important to remember that AI is only human. It necessarily reflects the biases of the humans who coded the data, trained the models. But this is the case for all content analysis, and that these approaches are complementary, that there may be some questions that are better for traditional approaches and other questions that are better for an AI based approach now, because there really is a very rich literature that uses these traditional methods.

What we want to do is really advance the machine-based content analysis. And in particular, we want to develop and apply new tools that convert images to data. I mean, as social scientists, we're often leaving a lot of data on the table by not thinking about what the content and the messages that are being sent through images are.

Especially if we think that a picture is truly worth a thousand words. And we'll use them alongside established text analysis methods, and we're going to apply them to a century of award-winning children's books. And we're going to look at three different aspects of identity. We're going to look at the construct of race.

Why do we say the construct of race? It's because, you know, race is this very ill-defined way of categorizing humans. And, you know, we're going to look at it in terms of three ways. We're going to look at skin color, we're going to look at putative race, which is the societal value of race that is placed upon people, and we're going to look at birthplace, so where people come from.

We're going to look at gender identity. This is necessarily limited because we're binarizing gender. So we'll look at females and males. But this is an example where traditional content analysis does a much better job of accounting for non-binary or genderfluid identities. And then we'll look at age. So our younger people versus people or older characters represented in other work.

We're actually looking at the portrayal of different identities. We've done some work using text analysis in particular, to look at the association between different identities and societal domains and, you know, the sentiments that are associated with different identities. If there's time, we're happy to talk through some of those.

So just to tell you a bit about the data that we use. So we are drawing upon books that were recognized by awards featured by the Association for Library Service to Children. This is a division of the American Library Association, a very influential group. They help to really determine a lot of the books that do show up in classrooms and school libraries and homes, for that matter.

And in particular, they are the arbiters of two major book awards, the Newbery Awards and the Caldecott Awards. And these are considered to be books that receive these awards are considered to be extremely distinguished, either on the literary side or the artistic side. They're considered to be the oldest children's book awards, certainly in the US.

You know, they rarely go out of print. And just to kind of give you a picture here. So, anytime we say mainstream, you know, these are the Newbery Awards and the Caldecott Awards, we're going to be depicting by pink. They go back to the 1920s and the 1930s.

Now, up until 1970s, there hadn't been African Americans who had actually received any of these awards. So in came the Coretta Scott King Awards. And those were specifically put into place to recognize books that centered the African American experience. And they were followed by a whole host of other awards that were meant to center books or books that centered, you know, underrepresented identities.

So the Stonewall Awards, which highlight the LGBTQIA+ experience, you know, the Arab American Awards, the American Indian youth literature awards, and all of these other awards are going to be depicted by this darker blue. And then what we wanted to understand is how smaller sub-collections, how they think about representation, so those book awards that focused on people of color are depicted by this orange.

You know, those for African Americans are depicted by green. You know, those that acknowledge and center people with disabilities are depicted by gray. Those that depict females are depicted. By this light blue, and then those that really center the LGBTQ plus experience are depicted by yellow. And part of the reason that we think about these diversity book awards is, you know, we're very clear that we are saying nothing about optimal representation, but it's useful to know what is an upper bound of representation that might be out there in society.

And because these are, you know, books that have been scoured by these committees to really think about those that are exemplars in terms of centering particular voices, we think of these as potentially close to the upper bound of what representation might exist in a particular year. And just to kind of highlight how important these Newbery and Caldecott awards really are, remember that the pink always depicts the mainstream awards.

And here the solid line are those that actually won the award, and the dashed line are those that got the honors or the runners up. And you can see that around since the time of the award announcement, there is this uptick in terms of the number of library checkouts, so this is from Seattle Public Library checkout data for mainstream awards.

And this pattern continues like this, this increased checkout on a daily basis continues years to come. And even if you extend this x axis out, you see this, whereas where you see for the blue, which is the diversity awards, or this orange, which actually here depicts all other children's book checkouts, there isn't this corresponding uptick.

You can also look at.

>> Stephen Davis: There's a big uptick, obviously, before the awards announced. Is that leakage of the information, or is it they're actually picking books that people think are high quality even before the award?

>> Anjali Adukia: Yeah, it's a good question. Sometimes in the months before, there is a lot of whispering from different big library organizations, school library journal, and people saying, we think these might be.

And so you can imagine that there are people potentially on those committees or people who are kind of in the know as to which ones might be high quality. So it could be that there's leakage, it could be that there's a general agreement as to, well, these are important, and so therefore they get on those lists.

It's unclear why there's necessarily this uptick ahead of time. We also separately have data from the numerator data set, thanks to the Kilt center at the University of Chicago Booth school. And what that has is receipt level data, and so we can look at book purchases over time.

And what we see is that there are more copies sold on average per title for mainstream books than diversity books and all other children's books. And you can see that also the mainstream books are just much more likely to be less expensive per book compared to the diversity book awards as well.

So how do we convert these books into the data that we're wanting? So we start with the images data pipeline. So we start with a digitized scan of each page in each book, and what we want to even start with is just detecting the face. So note that we're not looking at the character, we're not looking at interactions between characters.

We just wanted to start somewhere, and we thought, we'll be able to use a state of the art face detector that's out there. And that was, at that time, Google Vision API. And so we thought, great, let's put a subset of the Caldecott books, because Caldecott is specifically given to books because of their artwork through this face detector and see what comes through.

And that's depicted by FDGV, Face Detection using Google Vision. And as you can see, we only got eight faces. And then we realized, we are going to have to actually train our own models, because this state of the art face detector is actually trained off of photographs. But about 80% of the images in our books, they're children's books, after all, are illustrations or some sort of artwork.

And so then we actually had to go through and do our own, and you can see, and that's kind of depicted by this FDAI, so Face Detection using the AutoML on Illustrations, we were able to detect many more faces. So now we've detected about 57,000 faces. How do we actually get skin color, race, gender, and age classifications?

Well, to start with skin color, we first use what's called a fully connected convolutional neural network. Which all that's doing, right, whenever we think about a neural net, we're really trying to think about, how does the computer think like a human and look at a face as if it's a human?

This is doing a process called skin segmentation, and it's isolating the parts of the face that we really care about. So it's gonna identify the edges of the face, the eyes, the eyebrows, and you can see how it's creating these dots in those spots. And then what we're gonna do is we're actually gonna create a convex hull, which connects all the outer dots.

And you can see it does a fine job, but it also gets some of the background, which we don't want, so we now refine that through what's called a conditional random field module. And what that's doing is it's basically, it's gonna start with a pixel, and it's gonna look at all the pixels around that and say, is that skin or not skin?

Is that skin or not skin Is that skin or not skin? And go all the way around. And based off of those estimates, it's going to predict whether that spot is indeed skin or facial area of interest or not. And it's called a continuous CRF because it continuously applies this for every pixel that's there.

And you can see what that does then, is it helps to refine the actual face area that we're wanting to look at. Now, how do we think about skin color? It's not that you can just take one pixel. Is my actual skin color what's in this light or in the shadow, in the morning, in the night?

In some ways, the ground truth of skin color is somewhat of a philosophical question. But what we really want is when we look at this, we want to take a weighted average of all of the pixels that are there, but we also don't want to have the hair and the eyes, so we apply something called k-means clustering.

And so what that does is it partitions every pixel into the kth closest color. In our case, we're using five colors. So just to give you a very clear picture of what that looks like, you start with a face. It's gonna cluster every pixel into the fifth closest group, so you can see even the eyes like this, the white turns into this beige.

The dark is gonna come to this darker color. But what you can also see is that we don't want the eyes and we don't want the hair, and that's typically the smallest two clusters. So we drop the smallest two clusters, we take a weighted average of the remaining three clusters, and then that weighted average is going to give us an estimate of the remaining perceptual tint that is there.

And that gives us the representative skin color for every single face that we detect in our sample. And there we're gonna actually get two different measures of skin color. We're going to get this continuous measure from zero to 100. But we're also, you know, when people look at faces, they don't say, I think you're a 62 or a 38.

They're really thinking about it in, you think about the emoji scale, okay, is it light, is it dark, is it medium? And because there are various ways you could think about breaking this, where you put the thresholds, for this very first approach, we said we would take an agnostic approach and just break the distribution into terciles.

But in other work We're taking more data driven approach. And so that gives us these coarse bins of are these faces darker? Are they medium? Are they lighter? And so you can actually see what the data look like. So each of these rows is a different collection. Every dot that you see is the classified face color, the color of the faces that we identify.

The y axis is the standard deviation of the RGB values of every face. And so the closer the RGB and B are, the more likely, the less vibrant the color is. And so the more likely they are to be monochromatic. And so, we realized that after doing a bootstrapping exercise that anything under this standard deviation of 17, we were gonna label that as monochromatic.

But then that left us with, you know, two kinds of polychromatic faces. There's humans, but there's also the blue robot or the green alien. And something actually, Emily realized she, it was a photographer in another life and knows a lot about color theory is when r is greater than g is greater than b, then that is more likely to be a human face.

So we applied that rule and said that we're gonna call those human skin colors, and anybody else will be non typical skin colors. So you can see how that does. Here it's mostly black and white or gray. Here you're gonna get more of the browns and the mahogany's and the non typical skin colors.

You're going to get those blues and greens and magentas. And so this is how we do our analysis. Then, separately, we actually run a feature classification model that predicts the probability that a face represents a given identity. In this case, we're gonna look at putative race, at gender and age, and we use the majority prediction to classify what these features potentially are.

And I just want to kind of emphasize that every single component of this pipeline represents an opportunity for further work, right? Really just the beginning. And, you know, there's a lot of computer scientists who've been thinking about this, but it would be great to get more social scientists who are thinking about these questions seriously, because skin color is one of the first ways that we categorize humans, you know, from the age of two months.

And so if we can actually understand the role of skin color and kind of its connection to inequality in life, these are tools that can actually potentially help us do that in a more systematic fashion. Now, separately, how do we convert text into data again, we start with a digitized scan of each page in each book.

We then use optical character recognition to convert the text into a language that the computer can understand. Now, how do we think about getting race, gender, and age from text data? Well, first we want to start with names. So we run it through named entity recognition, or NER which extracts name information from unstructured text.

We then match these names to a dataset called Pantheon 2.0, which is based off of Wikipedia data, and that has a list of famous people, and it's gone through and predicted the gender, their birthplace, actually their occupation as well. And so we can match that. And someone like Rosa Parks, we now get their gender as female.

Their birthplace is the United States. These are real people. So we can then actually manually code their race. So then that gives us a measure of gender, birthplace, and race. No of course there's other characters. Steve went to the store, Justin went to the office. And so how do we get those individual character names?

We take the remaining names that came out of our ner output. We take the first name that's there. We match it to Social Security data, and then based off of Social Security data, so the name Cameron, for example, 91% of the people who are named Cameron identify as male in the Social Security data.

So we then, classify that name as being male. And so then that gives us another measure of gender. Now, of course, there's other words like pronouns and tokens. So tokens in computer science are just a continuous string of characters. In our case, it's just a word. So there's other words like aunt or queen or girl or boy that have gender components.

And so what we do is we pull those out, and then that gives us gender terms, but also words like aunt and queen and girl and boy. Those have age components to them, right? So you can think about aunt and queen as being older words, whereas girl and boy as being younger words.

And so then that gives us a measure of age. And then this is how we get our overall measures of race, gender, and age from images and text. Again, remember the images, we're only looking at the faces. I'll show you some of the results that we have. This is looking at the distribution of skin colors, just in terms of human skin colors.

And the way to read these figures is here. On the left, it means that there are darker skin colors. On the right, it means that there are lighter skin colors. Remember that the pink always is referring to mainstream books. This darker blue is always referring to books from the diversity collection.

And what you can see is that the mainstream books are just much more likely to be shifted to the right, that they're more likely to show characters with lighter skin tints as opposed to the diversity collection. You might think, well, how do we think about this in terms of those coarser bins?

And as you might expect, the diversity collection has more characters in the darkest skin bin and the medium skin bin. But the mainstream collection actually is to do the other way, where it's more likely to show characters in the lighter skin bin and again, in the medium skin binna.

Now, if you wanna kind of think about it in terms of those sub collections that we showed you, again, the people of color in the African American collection are more likely to have characters in those darkest skin bins, but so is the female collection. And so what that might suggest is that they may be more likely to take an intersectional lens where they're not only centering female voices, but they're also centering people of color in terms of what they're showing to.

If you want to think about over time, these are looking at, again, those skin bins. Over time, what you see for the mainstream collection is that there has been this increase in terms of the proportion of characters who are in that darkest skin bin. But there's also this corresponding uptick in terms of the characters who are in the lightest skin bin.

And where they've kind of ended up is where the diversity collection started out in the 1970s. Remember that it's truncated. The diversity collection is truncated in the 1970s because that's where, or where it's censored there, because that's when the Greta Scott King awards first came on the scene.

And then when we look at the patterns for monochromatic or non typical, we actually see very similar patterns. But actually potentially more interesting is when you think about, well, how do these skin color patterns look when you look within race? And so, remember that we separately use a feature classification model to predict what the race of these characters, of these character faces are.

And so, what these quadrants are. This is characters who are all characterized as or predicted to be Asian. These are characters who are predicted to be black. These are characters who are predicted to be latina or others. And these are characters who are predicted to be White. And what you see in each of them is that the mainstream collection is more likely to depict characters of a given race with lighter skin than characters in.

In the diversity collection of a given race. So just to, you know, make it very clear, just focus on this upper left quadrant. You know, these are characters who. All of these are characters who have been classified as being asian. The mainstream collection is more likely to classify characters who are predicted to be asian with lighter skin than characters who are predicted to be asian in the diversity collection.

And to give you a picture, these are both pictures or images from the mainstream collection, the diversity collection. And all the characters here are actually classified or predicted to be black. But you can see there's more likely to be highlighting of the skin in the mainstream collection relative to the diversity collection, where they're more likely to show darker skin colors.

Both of these books are set somewhere in Africa, but there are these differences because also when you are using illustrations, you can make more decisions as to where you're highlighting and where you're not. A vast majority of those characters who are pictured or predicted to be pictured are white individuals.

Even in the people of color and african american collections, where you might expect to see more people of color depicted, another way that you can think about looking at race by gender is actually looking at famous people, because, again, you know, these are real people with real human categorizations.

And similarly, across the board, most of the famous people mentioned are white men. The only place where you actually see more men of color, for example, are black men in the african american collection. And so that's about a little less than a percentage point difference. But in almost all of the categories, you see that men within a racial category are more likely to be present than women as well.

And just to give, again, a very concrete set of examples, if you look at the top ten appearances of famous people, you see that in the mainstream collection, they're all men. You have to get to number 16 before you get to a woman, that's Eleanor Roosevelt. In this case, it's a white woman.

And you get to number 43 before you get to a woman of color. And that's Rosa Parks. In the top ten in the diversity collection, we see that the only woman that's there is Rosa Parks. The rest are men who are depicted. As I said, we're saying nothing about optimal representation, but you might be curious about, well, how does this compare to the population?

And so here, the orange line refers to the share of whoever's in the column header, the share of their population according to the US Census. And so what you can see is that people who are white have been overrepresented relative to their share in the population. Since the 1930s, people who are black were underrepresented, except until the last three decades.

People who are Latino have just not been represented at all, despite a growing increase of their population share over time. We can't do this for skin colors, because we don't actually have a baseline of. We have no idea of what the distribution of skin colors are in the general population, which is, I think, an interesting place for further work as well.

Another way of potentially categorizing humans is actually where they're born. What we see in the mainstream collection is that most of the famous people who are there. And why do we say famous people matter is, in some ways, they're salient. They can serve as role models. There's a lot of literature that talks about these aspirational effects, about people who you're exposed to.

And so, again, who are you exposed to, and what are their stories? And here we see that most of the famous people are in the US and Europe, whereas there's much more geographic heterogeneity in the diversity collection. And when we break it up by gender, we see that most of the geographic heterogeneity that's there is primarily among men.

Now, separately, if you just wanted to look at gender alone, what this is showing you is the distribution of all female words across collections. When we say all female words, that's thinking about pronouns, other gender terms like queen or boy, the individual character name classifications, and the famous people.

And on the left side, it means that there's 0% female words as a percent of all gendered words, and on the right, it's 100%. And what this is showing you is actually the mainstream collection is more male centered than the other collections. Now, the collection that you might expect to be the most female centered is indeed the female collection.

And you do see that it is the most female centered of all of them, but it's less female centered than the mainstream collection is male centered. Now, you do see this increase in terms of the share of female words over time. It's only the female collection that ever goes above 50%.

You might also say, well, when I was growing up, gender neutral pronouns included he and him. So, of course, you're going to see more male representation because of this gender neutrality. So then what we do is we actually break it up by the type of word. And we see, regardless of the type of word, none of the collections, except the female collection ever get above 50% in terms of female representation on average over time.

Now, one thing you can look at is this famous figure. Figure is looking at, for example, let's say Rosa Parks shows up three times in a book. Then here she gets counted for all three of those mentions. You might say, well, what about unique famous people who kids are potentially being exposed to?

When you look at unique people, then you actually see that 63% of those famous people in the famous collection who are uniquely mentioned are actually men. And indeed, in the female collection, the top famous person who's mentioned is actually John F Kennedy. Now, if we want to look at how images and text compare to each other, so on the y axis, this is female faces as a percent of all faces.

And on the x-axis is female words as a percent of all gendered words. There's a circle for every decade that a particular collection shows up. That's why there are more pink dots than others. And what you see is that most of the dots show up above the 45 degree line, which indicates that there are more female representation in images rather than text, which is consistent with the maxim that women should be seen more than heard.

And it really suggests that maybe there's more symbolic representation in pictures rather than substantive inclusion in the actual stories. Now, where you might expect to see more representation of children, if anything, is potentially when you're looking at adults and children. And here the green is showing. The upper is showing the representation of adults versus children and images.

And down below is in text. So the green here is showing adults, and the purple is showing children. Down below, the upper two are showing younger words, and the lower two are showing older words. So you think about girl, boy, mother, father, and you can see in, whether it's images or in text, adults are just much more likely to be represented than their younger counterparts, even relative to their share in the population.

Now, I think probably, for me, one of the most surprising results and something which I don't think we could have seen very easily with traditional content analysis approaches, is looking at the relationship. Between children or age and skin color. So remember that the green is gonna be showing the distribution of skin color for adults, and the purple is going to be showing the distribution of skin colors for children.

And you can see across these collections that children are more likely to be depicted with lighter skin than adults. And these differences are statistically significant. And so it just raises these questions and these interesting questions about, well, what are the messages that are implicitly being sent to kids?

What are the patterns of skin color over time? It's unclear that we should be seeing this pattern, necessarily, because, if anything, melanin breaks down over time, which means that people might be getting lighter over time. Now, is it because people are working out in the fields? Well, that would mean a very narrow, particular set of stories.

Is it because there's increased mixed race couples? Well, if anything, that we should just tighten distributions. It shouldn't actually shift to distributions necessarily. But I think these are all empirical questions to be asked. And what's nice about these tools is that they can allow us to dig deeper into all of these.

And I'll just quickly talk about some of our portrayal results. So this is where we're actually using word embeddings to see how different identities relate to various domains in society. And we see that females are more likely to be associated with appearance versus competence, and they're also more likely to be associated with roles in the family versus business.

When we try to look at this by race, by gender, and various domains in society, we see that black females and black males are more likely to be associated with struggle. Black females, in particular, are more likely to be associated with the performance arts, as are white females.

And white males, actually that's the dark blue. They're more likely to be associated generally, but especially with power and business overall. You could separately use sentiment analysis techniques. And here, this is showing you. So the blue here is the sentiment of females on average, and the orange is the sentiment of males on average.

And this goes back to the 1920s. We can see is that they were always, on average, talked about positively, but there was a big difference between males and females, and that difference has really converged over time. But what that difference somewhat masks is that there are more white women who are portrayed in mainstream books.

And white women are more likely to be portrayed positively, but black women are more often to be portrayed in a negative light relative to their counterparts. And then finally, we'll end by just kind of thinking about what might be driving the supply of different representation or the demand of representation or the patterns that we see.

And so, actually, again, drawing from that numerator data that we mentioned earlier that we got from the kilt center, what we see is that male purchasers are more likely to buy children's books with less female representation. So whether it's words or names or faces, people with daughters are more likely to purchase books with more features.

Yes.

>> Stephen Davis: I wanna jump in here.

>> Anjali Adukia: Yes.

>> Stephen Davis: I want you to tell us what the magnitudes of these associations are. If I'm interpreting the regressions correctly, these are tiny effects. They're estimated with good precision, but they're pretty small.

>> Anjali Adukia: Yeah, Emily.

>> Stephen Davis: Males are slightly more likely to buy male oriented books than females, and so on.

 

>> Anjali Adukia: I think that's it. Emily, do you want to hop in here?

>> Emileigh Harrison: Yeah, so our dependent variable is between 0 and 1, not 0 and 100. So here, the way that we would interpret this is men buy books with about 1.5% fewer female words on average. And the baseline is 38% of books have.

 

>> Stephen Davis: Yeah, okay, so exactly. So that, to me, is a small effect. And so you've documented, I think, pretty persuasively, all this evidence of differences in representation, and it's correlated in the direction, one might think, with some demand side factors that might be pushing that way. But these are small effects, or do you disagree with that interpretation?

So that kind of begs the question of, well, why? Why is this happening? And this is a very natural avenue of exploration. There's something to it, but it's a pretty small part of the story. As I read these evidence, we completely agree that we think of this as descriptive, as exploratory, as just trying to think about why might these effects be there?

 

>> Anjali Adukia: But this is just suggestive, interesting to look at that.

>> Stephen Davis: But why do you think they're there? What's your best assessment at this point?

>> Alex Eble: So if I could jump in, I think it's true that they're small. But if you go back to our results on the representation of gender and race in the books.

It's also true that even in books that are trying to highlight the African-American experience, the experience of females, or the experience of LBTQIA individuals. It's still very often the case that white individuals are dominant and males are dominant. And so I think there's also a case where there is, like, in the support of possible configurations of representation in each of these books.

There actually aren't that many options where it's all black individuals or all females. And so this movement is within kind of a constrained part of the support of possible values of representation. And so I think part of it is, another part of it is that the set of choices is smaller.

 

>> Stephen Davis: Two comments on that. First, a given individual can choose within the full range of what you've specified. So even though on average, across books that are in your collections, there may be a limited range among the books that are in the diversity collection here, what you're doing is you're looking at individual purchase decisions and relating to individual characteristics of the buyer and their children.

So there's plenty of scope for substitution across different representation frequencies in this sample, as I understand it. Second thing, the reason I'm pushing on this is, and this is not something you guys are doing, but there will be a set of readers of your work, I suspect, who will take this and say, look, there's these unequal representations.

They're both unequal in kind of the raw arithmetic sense, but also unequal in the sense that you also showed earlier that they differ from the population proportions for those groups. And some people will take that and say, this is prima facie evidence that something nefarious is happening, or that there are racist sentiments in society, and so on, or sexist sentiments.

And it looks to me like whatever the story is, it must be more nuanced and complicated than that.

>> Anjali Adukia: But it could also be utility from homophily, right? So when we look at even just purchaser ethnicity, we see that people who are black and Latino are more likely to purchase books where the average skin tint is a bit darker, people who are white.

So there could be just people are interested. And similarly, when we look at, you know, the percent of famous mentions by race, people are just more likely. Probably to purchase books.

>> Stephen Davis: Yeah, but again, okay, I'm glad you put this table up. This is the other one I had in mind.

If I am understanding this table correctly, every column except column one, the effects are tiny. And so we're back to this, then. And they do. They do go in the direction of preference for homophily. And with respect to skin tint, the effects are pretty big. These are large effects.

But all the other effects, they're tiny. They're estimated with good precision. They're tiny. So the preference for homophily effect, except on the skin tint dimension, which I would have thought the preference for homophily effects would be big. They look like they're pretty small.

>> Justin Grimmer: The homophily for black purchasers is pretty big.

 

>> Stephen Davis: Isn't that-

>> Justin Grimmer: Twelve percentage points?

>> Stephen Davis: 12 percentage points.

>> Justin Grimmer: Over a 7.8% baseline.

>> Stephen Davis: Okay, so then the left column, I should interpret that as 64 percentage points. I'm a little confused. You got six point in the column. I'm just trying to understand the magnitude of the effect.

Is it 12% or 0.12%?

>> Emileigh Harrison: It's 12%, nut the first column we're not looking, the dependent variable is not in percentage terms.

>> Stephen Davis: Okay, that's very helpful. Okay, so great. So, yeah, 12%, that's a sizable effect. But again, I'm a little. I was expecting to see bigger effects on this side.

And it looks like the homophily effects are there, but with a couple exceptions, they're pretty modest in size, but I think if you divide it by the constant.

>> Alex Eble: So for Asians, it's a 60% increase in the frequency of Asian people in the books. And for black people, it's 150% increase.

For Latinx people, it's a 50%. No, it's a doubling. And so I think relative to the representation, it's actually quite large. So on average, asian people make up less than 1% of asian people in books. And if it's an asian person buying a book, on average, the books that they buy are gonna have almost twice as many asian people.

If it's a black person buying a book, on average, the books in our sample have roughly 8% of the people in the books being black people. And then if it's a black person buying the book, their books, on average, are going to have 20% on average black people.

So I think relative to the baseline, it's large.

>> Stephen Davis: Okay, I know I've interrupted you here, but I want to have time for Q, and I'm going to let Justin, unless there's something else you really want to get on the table before we go.

>> Anjali Adukia: No, in fact, that was it.

We were then, to summarize.

>> Stephen Davis: So let me see. Justin's probably got some comments or questions here.

>> Justin Grimmer: Yeah, I have some questions. First off, this is super exciting work to see and very interesting and thought provoking. And it was fun to think about children's books again, it's been a minute for me.

So one of the things I was thinking about is sort of reading this and going back through some of my kids books are depictions of animals within books and the ways in which animals and allegories get depicted. So let me just throw some books out there. I don't think they're in their sample, but I think they work.

For example, so Frog and Toad are friends. I know that that's a story just about a frog and a toad who are friends, but within the gay community, that sometimes they get portrayed as perhaps being more than friends. There's the Butter Battle book from Doctor Seuss, which involves animals, I believe they're cats or something, like many Doctor Seuss books, but that's ostensibly about things happening in the world right now.

And so in general, there's this sort of explicit versus implicit representation, the way in which some communities can be represented either literally by putting the person in the book, or through implication. And I'm just wondering how you think about that. Has explicit versus implicit representation changed over time?

Were there more animals in the past or now, or how would we go about assessing them?

>> Anjali Adukia: So one result that we saw early on that we actually don't have in the paper, that when we kind of look. So we trained a model to try to predict whether it was human versus nonhuman.

And what you see in terms of predominance is, you see white men are most likely to show. Show up white females, all non humans, and then all people of color. So you're more likely to have a non human than you are to have a person of color. Now, how that's changed over time, I actually don't know.

I don't think we actually looked at the time trends. We were planning to kind of push on that in further work. But we said in this, all non humans and humans are being considered together at least as the first pass. But then separately, you can actually look at them separately.

But, yeah, I mean, they think these implicit representations are really interesting to try to understand and dig more deeply into.

>> Justin Grimmer: It also sort of gets into how different groups are represented. So another sort of representation I was thinking about while reading this actually has to do with the hyenas in the Lion King, which I don't know if follow this controversy at all.

It's like many years ago now, but there was some controversy around the hyenas being depicted as being portrayed as perhaps individuals who may have been or representing black individuals, but in the Lion King, they're obviously hyenas. And so there was some groups that raised objections to that. Anyway, just super fascinating.

A sort of second way of thinking about representation and you sort of get at this with some of the text in the paper, in your analysis of the text, is doing something like a Bechdel test. And so I know that you have the sort of female words and female images which sort of gets close, but I'm curious about any sort of analyses you can do about dialog.

Are female characters doing something other than advancing the plot around male characters? Are they the ones who are actually sort of exchanging words? Obviously, the Bechdel test is this very low bar to clear, but you can imagine doing many, many other things given the awesome data that you've assembled now in something that would be more advanced than a Bechdel test.

 

>> Alex Eble: Which one of you wants to plug our future work?

>> Emileigh Harrison: I love this question. We're really excited about moving into the direction of how women are talked about. And in particular, right now, we've just done some analyses on the actions that women take relative to men. And some of our preliminary findings are that women are more likely to take.

The actions that women take are more positive than the actions that men take, but less active and less powerful using by matching actions to this dictionary, which scores words on these three dimensions. And so that's like moving the needle a little bit towards what you're hoping for, which is, how are they furthering the plot and interacting with others?

So we're working on it. We're pushing there. But we also think this is a space for a lot of people to join, and we're excited to see how these questions are furthered in other research, not just our own.

>> Justin Grimmer: Okay, I have one last question, and it's sort of about the bifurcation of books that I think used to be much more present in children's books and perhaps are now.

So you can imagine, I think when I was a kid, many years ago, there would be boy books and girl books in a way that's not clearly represented now. At least it's my impression of children's books now. Curious if you can detect something like that bifurcation where there are, like, themes that appear only with male characters and themes that only appear with female characters.

And whether that bifurcation has gone away or if it persists or because that also, I think, begins to speak to some of the markets. Like some parents wanna buy their son a book that has a knight or something, and just to carry through the stereotype, like buy their daughter a book that has Barbie dolls doing something or something like that.

You know what I mean? Whether that persists or not, I don't know, but I imagine certainly there was a market for that 40 years ago.

>> Anjali Adukia: Well, and also, I mean, the idea of what the girl book doesn't necessarily apply. That same rule wouldn't necessarily apply to boy books.

So, let's say you're buying a book for a friend's kid. You don't know what the gender is. You might be happy to buy the little prince regardless, but you may not buy the little princess, right? And so that also goes to this notion of status quo bias, but it is an interesting question to see.

And also, how do you algorithmically determine who the protagonist in a book is? We thought, maybe it's the person mentioned the most or those that are pictured the most. But actually, sometimes there may be more male characters in the background, the baker, the cobbler, the whatever, who are actually getting more speaking roles.

So actually, you get more male voices, even though there's a female who's the hero of the story. So, it's actually kind of also hard to at least algorithmically determine. We could go through and try to, by hand, say, well, this is who we think that the protagonist is.

 

>> Stephen Davis: Justin was thinking about animals and how frequently they appear. I'm thinking about famous persons and also what kind of famous person. So is a large share of the characters with any kind of name in these books. Are they famous persons or nothing? If they are famous persons, are they not?

Because then, like the rest of my question is not so interesting if they're not, I just. Here's the concern I have or not? It's not a concern. It's just a question. If famous persons show up a lot in these books and famous persons is mainly people who were political figures and people had historic impact, then you're gonna tend to get an underrepresentation of individuals from groups that were historically subjugated just because they had less opportunity to become president or whatever.

Is that an issue here or is it just a tiny thing.

>> Anjali Adukia: But also remember that there's still a choice because there's famous people all around the world. If it were just heads of state, there are heads of state across all countries around the world. So, there's still choices as to who.

 

>> Stephen Davis: Of course, but then we're back to what is it parents want from, in general, from the kid, the books they present to their kids. Do they want them to have a good time? Do they want them to learn about the world as it is? Presumably, if it's the latter, then they're more interested in their own country than other countries.

So just some background information on what are these books providing? I really don't know. Is it just princess and prince stories and you can grow up and be wonderful, or it's something else, or is it like, what's the world of work like? I don't know.

>> Alex Eble: So, I think we don't have, off the top of our head, how many of the people that are named are famous versus not.

We do try really hard to say, okay, like the normal econ, whack a mole robustness stuff, where we say, okay, let's drop the famous people and see what it looks like. Let's do only pronouns. Let's do only non famous names. And our results all go through if you kind of lop off any.

 

>> Stephen Davis: So just focusing on non-famous names, you get very similar.

>> Alex Eble: Exactly right. We had, we have. We've got some questions, and one was related to the one I just asked, but slightly different. It's you know what. How are you defining famous persons? Does it include celebrities, for example?

Or is it people who had some impact outside the arena of celebrity hood? I'll just leave it at that. So that's one question.

>> Emileigh Harrison: Yeah, so our definition of a famous person is a person who is in the Wiki data dataset. And so they have their own definition of a famous person, which is a person who has a page on Wikipedia that has been translated into, I think, at least twelve different languages.

So there's some definition that someone else has chosen, and we're using their data and matching their data with the names in our text.

>> Stephen Davis: Okay.

>> Anjali Adukia: That can absolutely include celebrities. That can include.

>> Stephen Davis: Yeah, presumably, especially in more recent times, it would include celebrities and maybe mostly celebrities, I'm not sure.

Let's see. We've got, make sure I'm not missing the, get some Q and A. Yeah, Liz, can we get, Elizabeth do you wanna pose your question? We can give Elizabeth elder an opportunity to speak. Are you still with us, Elizabeth?

>> Elizabeth: Yeah, sure. Happy to speak if you can hear me.

Okay, so I was curious about this thought raised by Justin's question, and also this made me think a lot about. We think about political representation of diversity, both descriptive representation in terms of characters of a particular race and versus kind of narratives or kind of viewpoints being represented.

And I certainly, from publishers' perspective, can imagine that stories about prejudice or discrimination or some of these kind of themes that we often think about as adults in a racial context might be easier for parents or for publishers to think kids would understand coming from an animal or coming from kind of an imaginary creature.

So I wonder if it's possible to take themes that maybe are coming up in some of these books that are about people of color, that these kind of less common books, and see whether those narratives are coming up often in these kind of non human character settings. And whether that might be kind of displacing some of those stories onto a different kind of protagonist as maybe a reaction to the market or to something else.

 

>> Anjali Adukia: That would be super interesting to look at. We haven't, but that's. Yeah, it would be, especially because we see this pattern of mostly non humans and then people of color. Is it that that's actually just displacing? That's a great question. Be really interesting to look at.

>> Stephen Davis: We've got a question about just what populations are you drawing from your data?

Is it all us data? Or are you looking elsewhere? And maybe have you done work on other populations? Middle Eastern, Latin American, asian. It's not like you. You haven't been slackers. So, this person's asking for similar analyses in other cultures and countries.

>> Anjali Adukia: That'd be super interesting. I mean, getting, it'd be a question of getting the data.

I mean, the book receipt data is from the numerator data is, you know, we're looking at. I mean, there are Asians and there are Latin Americans who want to end this table, but they're in the US. That you're part of the US, simple.

>> Stephen Davis: But it's very. This is the exercise you've done is very doable in other cultures, and it would be.

There's lots of fascinating questions about how representation of majority groups differs and minority groups differs across countries and cultures. Along many dimensions, and in some places, the religion, religious belief will be a very interesting dimension along which to look. I'm just going. Feel free to jump in here, Justin.

 

>> Justin Grimmer: I just have, I think this is more of an idea cuz I know you haven't done it. But I was trying to think about the sort of experiment to get at the regression that you're running, and obviously, that's extremely challenging. But there is interesting comparisons to be made with the American Girl doll series, because over time, they've, in that series, introduced, I think, more diversity in the stories that they're telling.

My impression was earlier on, it was a lot of historical eras that involved white little girls, but now there's much more diversity around that. I have no idea what sales looks like, who purchases different american girl dolls, who purchases different american girl doll books, but it all seems very closely related to issues of representation.

 

>> Anjali Adukia: Yeah, it's like the Queen of Katwe paper, where suddenly, like, how do people think about tests in particular areas? And some people say, well, suddenly, Elsa, is she actually an asexual character? And what does that do for kids feeling? And she's wearing blue instead of pink. And it used to be that girls would wear blue, then they wore pink, and does that shift things, or does it at least create?

I think there's a lot of questions about, especially people or characters that get a lot of extra press and that much more salience. What is the impact that they have on society more generally? And going to your question or your kinda your note about what we would ideally do.

Look, what we really would wanna be able to do is, these questions that Steve was asking, what you're asking, Justin, about what is the impact of representation on outcomes and on beliefs and on the decisions that people make? In some ways, it's hard to even know what is the ideal experiment here because people are exposed.

Kids are exposed to so much, they're not just exposed to the books that they see in the library or at home. They have YouTube, or they didn't have YouTube back in the day, but they have lots of different influences. And so you really wanna think about the dosage that you're giving people of any particular identity.

So you could try to think about doing A/B testing and saying, okay, you get all of these female scientists of color, and you get all these male scientists who are white. And what does that do, and how does that change? But the thing is, you might see something immediate.

But do you even see something two weeks later? What really changes norms and what really changes beliefs? I don't know if an experiment like that or something as just like the books that you have on your library alone, maybe they shift things in certain directions, but it's hard to know in terms of that overall dosage to really be able to systematically look at this, because you need a first stage.

 

>> Justin Grimmer: Yeah, I guess I was thinking way more so. I totally agree with what you just said about the dosage. I was thinking much more narrowly, like, if I were a VP of a children's book company, how would I think about the sort of portfolio that's being allocated?

And ideally, from a purely instrumental, I'm charged with selling books, I'd be interested in knowing who should be in the books and how are they gonna be purchased. Or should I market different versions of the books in different places? Something like that.

>> Emileigh Harrison: It's interesting to think about the publishers decision making in the age of so much digital media.

Because it used to be that there were large fixed costs to publishing books, but now if you're publishing the book online, the fixed costs are much lower. So I think even seeing how a representation has changed in response to a decrease in costs of publishing would be interesting as well.

 

>> Stephen Davis: Yep.

>> Anjali Adukia: And also when you kind of look at. So I can even show you, we kinda went by this fast, but when you look at the average number of copies sold per title, yes, there are more mainstream books sold per title. But there's also more books from the LGBTQIA+ collection that are being sold per title, even though they're much more expensive.

So you could imagine, so if I were the head of a publishing company or trying to think about the market out there, I would find that quite compelling to say, well, clearly there is demand. Potentially, this might at least lead me to think there might be some demand out there that's actually not being met, that we might be able to, you know, consider in terms of the inventory that we put out there.

 

>> Stephen Davis: Right, there's lots more to discuss here. I know I've got more questions, but we're gonna stop the recording, thank the authors. This was fabulous work, a fabulous presentation. There's room for an enormous amount of additional work in the same direction. So this is likely to become a landmark paper in the study of representation.

And I congratulate the authors for writing it, and I thank them for sharing with us. So with that, we're going to turn off the recording. But stick around if you want to talk more, we'll continue this discussion.

>> Anjali Adukia: Thank you for having us.