Leverage Research
Exploring the mind and society

Making Introspection More Reliable


Concerns about reliability

Introspection is meant to give you information about your mind. How accurate is that information? How much noise is there? Are there systematic distortions? What can you do about this?

These are critical concerns. In fact, they're more than that. The entire practice of introspection is centered around trying to get a clearer and more accurate view of the mind. When you are learning to introspect, the entire point is to improve the reliability of the process.

This means that if you are serious about learning to introspect, you should not dismiss your concerns about the reliability of the processes you are using. You should be especially attuned to them, and you should seek whatever evidence you can that will help you to sharpen your practice.

Concerns from the scientific literature

One important source of concern about the reliability of introspection is the scientific literature. Experimental psychologists have studied introspection, and it's important to try to learn what we can from their papers.

What does experimental psychology say about introspection? There's a popular story that says that experimental psychology has demonstrated that people cannot introspect reliably on their mental states, and that anything people say when they introspect is liable to be post hoc rationalization.

Luckily for all of us, that's not true. The paper most people cite is Nisbett and Wilson's 1977 paper entitled "Telling More than We Can Know: Verbal Reports on Mental Processes". In their paper, Nisbett and Wilson survey dozens of experimental psychology studies. Each study purports to show that people can be led astray in judgments about their own mental processes in various cleverly designed situations. They eventually reach a stark conclusion: "In summary, it would appear that people may have little ability to report accurately about their cognitive processes..." (p. 246)

That would be a problem. But as they themselves point out just a paragraph earlier: "We also wish to acknowledge that the studies do not suffice to show that people could never be accurate about the processes involved. To do so would require ecologically meaningless but theoretically interesting procedures such as interrupting a process at the very moment it was occurring, alerting subjects to pay careful attention to their cognitive processes, coaching them in introspective procedures, and so on. What the studies do indicate is that such introspective access as may exist is not sufficient to produce accurate reports about the role of critical stimuli in response to questions asked a few minutes or seconds after the stimuli have been processed and a response produced." (p. 246)

We're not sure what they mean by "ecologically meaningless", but if you look through the studies they discuss, it does seem like researchers are studying the introspective reports of untrained introspectors introspecting about past events. It's not surprising it doesn't work.

What happens when you study people introspecting on their current experience? Ericsson and Simon are two researchers who have studied this, exploring the conditions under which verbal reports are more likely to be accurate, and developing a method called Protocol Analysis. Protocol Analysis asks subjects to speak aloud each conscious thought as soon as they become aware of it. The sessions are recorded, transcribed, and analyzed, and the analyses are used to build models of the person's underlying problem solving process. These models can then be used to generate testable predictions about problem solving performance under varying circumstances. For example, in one experiment (paper, p. 236), a subject reported that one of their primary strategies for remembering 3-digit numbers was to convert them to running times (“358” would be remembered as 3 minutes and 58 seconds). The experiment found that this subject’s performance could be reliably increased by giving them sets of numbers that could all be converted into track times (such as “247”) and decreased by giving them sets of numbers that could not be converted into track times (such as “498”).

Now, these papers cover only a hundred studies or so, just a fraction of what has been published on the reliability of introspection. And one has very good reason to be skeptical of anything that comes out of experimental psychology. But the general lesson should be clear: you can't just introspect in any way you want and expect it to work. You need to put in serious effort to improve your practice and make your introspective processes more reliable.

Sharpening your practice

Here are some basic tips for sharpening your introspective practice, and thereby increasing its reliability:

(a) You can't introspect about the past. People sometimes try to introspect now to see what they thought 10 minutes or 10 years ago. Don't do this. You can introspect on current memories, which may be accurate or inaccurate records of what happened in the past, but you can't introspect directly on the past.

(b) You can't introspect on causal processes. People sometimes try to introspect on causal processes. Try all you'd like, though, you just won't ever be aware of causation in your experience. You can be aware of one thing and then another. You just can't be aware of the causation. (You can be aware of your beliefs about causal processes, and these beliefs are sometimes true.)

(c) Don't confuse your theory of X with X. Frequently when people try to understand themselves, they query their theories about themselves rather than querying themselves directly. Both are fine, if that's what you intend. You can introspect on your theories to learn about your theories and you can introspect on other things in your mind to learn about them. Just don't introspect on your theories to learn about things other than your theories... unless, of course, you have good reason to believe that your theories are actually true. (Frequently they're not!)

(d) Maintain your intention. People frequently set out to introspect on some aspect of their mind and end up changing intentions midstream. For instance, you might set out to articulate some felt sense... and then become impatient and then say some words aimed at capturing the gist. That's fine also, it's just not the same thing as articulating the original felt sense.

These tips will help increase the reliability of your introspective practices. There are obviously many, many more things to take into account. The best way to learn introspection and increase your reliability is extensive practice, good written instructions from others, and direct instruction by someone who knows how to introspect and knows how to teach introspection.

Leverage Staff

Compilation piece, written or inspired by several researchers. Opinions expressed represent common views inside Leverage, not necessarily unanimity.

More posts on