You analyze a dataset and figure out that political polarization has been steadily rising in UK for the last 10 years.
You tell this to your colleague who looks surprised: apparently, in her analysis, polarization has been decreasing for the last 10 years! And she used exactly the same dataset as you.
What is going on?
What we know about measurements
i.e. recap of the last post on measurements
In our last post we discussed that, in the social sciences, quantities are defined in a general way. This is different in fields like physics where they employ “operative definitions” which tell you exactly how to measure a specific quantity.
Of course, we have many more interesting details in that post, but the short story is: different people will end up measuring different things (and these measurements are usually non-comparable).
We used the example of counting potato chips and noticed that everyone will count a “full chip” as one. However, everyone will have a different approach when counting broken chips. For example, one may count each fragment as a separate chip, while someone else may not consider them as “chips”.
End of the recap!
What about the dynamic behavior?
The big question now is: we know these measurements are non-comparable, but should they all produce the same dynamic behaviour?
For example, if I add a chip to the bag, all the measurements will increase by one. And if I remove 10 chips from the bag, all measurements will decrease by 10, right?
Yes, indeed, in some cases all the measurements will behave nicely and produce the same behaviour. However, what happens if we start crushing some chips in the bag?
Now, the measurement which counts only the full chips will tell you the number of chips is going down (we are losing a lot of full chips) but the measurement counting each fragment separately will register an increase! We are observing two opposite dynamic behaviours even if we are still measuring the same “macro-concept” (i.e. the number of chips)
Why we have this, again?
As discussed last time, we should be careful with the difference between “macro-concept” and the measurement itself (i.e. “operative definition”). Indeed, while the macro-concept may sound clear and intuitive (“we are just counting how many chips we have in the bag”) it is actually hiding the fact that we can perform this operation in many different ways. And, unfortunately, every way consists of measuring something different.
But maybe you are still not seeing exactly what is causing this problem, nor how this can impact the social sciences. Well, in the next paragraphs we will address this using political polarization and a few elephants.
What is an elephant?
There is a nice story about some blind men trying to figure out what an elephant actually is. One grabs its trunk (i.e. the nose) and thinks that elephants are like snakes; while another one grabs one leg and thinks that elephants are like trees, etc. Maybe this story may teach you something important about life, but we are not interested in life, but in science, measurements and definitions!
So, let us replace the blind men with 3 researchers wanting to write an article on Nature Elephants. They also ask independently their assistants to put a sensor on the elephant to track its movements. One assistant puts the sensor on the tip of the nose, one on the leg and the last one on the chest. However, the researchers are unaware of these differences; they only know that their sensors are on the elephant (and nothing more).
When the elephant is moving, all three measure roughly the same behaviour (i.e. horizontal motion). But when it stops walking, they start observing completely different dynamics.
One, measuring the chest, start observing a rhythmic movement. The one on the leg measure almost complete stillness, while the sensor on the nose shows big irregular movements in different directions.
The three eventually start arguing and debating. Should they use more precise sensors? Or maybe run an experiment with 1,000 elephants? Should they use some statistical technique for removing big or small movements?
This may sound very silly, but unfortunately, it is not too different from the way we discuss when we face more abstract concepts, such as political polarization…
Is polarization increasing?
Some times ago, I gave a talk in which I showed how, with a method we are currently developing, we could explore different aspects of political polarization. At the end of my talk, a person asked me: “so, practically, is polarization increasing or not in the US?“
If are not familiar with the term, polarization represents how “split in two” a certain society is. As you may guess, this is a quite general concept, which incorporates almost endless measurements. Let us see a couple of examples. Polarization can be thought as:
- How many people are at the “extremes” (e.g. most people hold neutral attitudes VS most peple hold more extreme attitudes)
- How extremists are the ideas (e.g. in society 1 extremists think we should not accept immigrants; in society 2 extremists think violence against immigrants is acceptable).
- How “flat” the society is (e.g. a society in which if you are a left-winger you also have specific food, music, and other habits which you share with he other left-wingers)
Take also into account that you can also combine the previous definitions. For example, you can produce a measurement that is mostly sensitive to effect (1) and only a little to effect (2). This means that we have pretty much infinite possible measurements!
Ok, but this was not the initial question. The question was: “is polarization increasing?” And it is clear that the person who asked wants a pretty simple answer, such as “yes” or “no.”
But, before answering it, let us warm up testing it on the elephant.
The elephant game
Let us play the following game: I will give you a situation and a question. You can answer only “yes” or “no.” Ready?
SITUATION: The elephant is running.
QUESTION: Is the elephant moving?
Wow, you are pretty good at this game!
SITUATION: The elephant is standing while eating.
QUESTION: Is the elephant moving?
Ok…
SITUATION: The elephant is sleeping.
QUESTION: Is the elephant moving?
Interesting…
We could go on, but you probably got the point: a simple “yes” or “no” does not fully represent what is going on. Actually, it does not represent it at all.
And this is the reason why I totally disappoint the person asking me the question initial question, by answering her: “Polarization is increasing or decreasing depending on what you mean by polarization.”
What about the models?
As you know, I am very interested in the dynamic modelling of social phenomena. Specifically, in agent-based models. So, how does this knowledge related to modelling?
1. Making the model
The first important thing to consider is how you will build the model. Some people build a model directly from data and experiments, other from more general theories and frameworks. Whatever you do, you will also select a specific definition of the phenomenon you want to model (even if you are not aware of this choice).
So, make sure you know which choice you are making, and that you keep being coherent with that choice. For example, you may mix two theories, or experiments, but make sure they refer to the same specific phenomenon.
2. Using the data
It is probable that one day you will use data with your model. Maybe it will be for calibrating it, or for validation, or whatever. The thing that you have to keep in mind is that there will be many types of data all under the same name.
To stick with the elephant example, you will find data on the nose’s movement, on the ear’s movement, etc. But, probably, they will all be labelled as “elephant’s movement.”
So, when using data with your model, you will have to make sure that both model and data refer to the exact same phenomenon.
3. What can go wrong?
The short answer is: everything. Here, just as an example, enjoy the following:
- The case in which your model makes totally different predictions depending on the measurement used for collecting the data [link].
- The case in which you can actually transform one model into another just by changing the measurement scale [link].
This does not fully explain it!
As you may see, one solution is the simple “just stick to one measurement.” However, this is a little limiting. Indeed, whenever we stick to one single concept of “polarization” we are also neglecting all the others. Similarly, if we track only the elephant’s nose, we lose all the information about its ears, eyes, tail, etc. What to do?
A common thing that people say is something along the lines of “this problem is just too complex because humans are infinitely complex. We should not even attempt to solve this problem. Enjoy life while it lasts.”
While I agree (especially with enjoying life), I think there are a series of things we can do to make things better while doing our research:
- Being as clear as possible about definitions, models and measuring process
- Study the phenomenon using different definitions and measurement (but being clear about these differences!)
- Trying to summarize the results and look for similarities and differences between different measurements and models
Will this solve all the problems? Definitely not, but I am pretty sure it will allow us to make some pretty amazing discoveries.
What’s next?
In these posts we analyzed this problem on a very general level (i.e. broad definitions vs operative definitions).
But it is time to start facing a little more in detail how measurements are made in the social world.
Coming up next, ordinal scales and measurement distortions!