PISA - OECD Programme for International Student Assessment

Comparison of students in the OECD countries was in the news today, with the latest report on science and math among 15-year olds. I want to present a contrary view, that "individual" performance that matters today, now, this decade, has little to do with math and science, and a lot to do with how we interact with and relate to each other. That's where we should be looking.

A typical AP news item is here "Other Countries' Students Surpass the U.S. on Tests" and the actual Programme for International Student Assessment report is here. US scores for reading are not included, since, ironically, the printed test in the US was not proofread adequately and was offset a page, making all the references to "the diagram on the facing page" nonsensical.

The report itself is 350 pages, and even the executive summary is 56 pages long. Results for the US are rather dismal, and you can read those yourself. A few highlights that I thought were interesting related to differences between males and females, which I quote further below.

I object to the entire test, not as being "wrong" so much as being misleading and supporting the efforts to "teach to the test" we see in the US, and certainly here in Michigan.

Local news shows that the areas called "social studies", "civics", group music, literature, history, government, and humanities in general, as well as team sports, are being neglected or eliminated in order to improve school rankings and funding. Ratings of schools in New York City recently have created a hornet's nest of debate over "corrective action" required for "bad schools."

I've thought about these issues most of my life. Early in my life, I taught one year at the high-school level in trade school, 10th-12th grade "problem students" from a tri-county area in upstate New York. I taught MBA's for two years at Cornell's Johnson Graduate School of Management. And I've spent most of my adult life in a large university setting, paying attention to educational issues. At Cornell University, at one point, I was the Director of Institutional Planning and Analysis, and very focused on long-term directions in education. I'm currently involved with work involving "leadership training" and quality improvement among teams and executives.

And, my undergraduate major was Physics, heavy on the Math, with a lot of Computer Science thrown in, at which I did very well, so I'm not speaking as one biased against something I can't do, motivated by "sour grapes."

But I have to say that, after a lifetime watching this issue, I don't believe that an increased focus on "science, math, and reading" is where we need to be focusing our attention.

There are two crucial assumptions behind this focus on science and math in this multi-level world we live in. One assumption, on the individual level, is that a child in the US, say, can get the best future for himself or herself by seeking to become excellent in these areas, and in fact to become competitive with international students for good jobs. The other assumption is that, if many or most students succeeded at this goal, things would be much better for the country as a whole -- economically, on a corporate level, militarily, and in terms of the quality of our health and daily lives.

Both assumptions seem patently false to me. They represent the worst of denial, "tooth-fairy" wishful thinking, and unexamined models of how things work.

Regarding the first assumption, the US is so far in the hole at this point, after years of neglect and unjustified self-confidence, that the odds we can make our students individually competitive in a global marketplace are very low, in my judgment. We have a whole generation, if not two, of teachers in place who don't really understand their own subject areas. We have an ethic and status quo of speaking and reading only one language, and being proud of not doing very well at that.

And, we have a hugely anti-intellectual culture in many places, where students attempting to excel are punished by their peers. In fact, the only thing US students seem to rank highest on, based on research studies, is their self-esteem. In other words, there is a disconnect between how they actually perform and how they think of themselves as performing. At that, the USA seems to excel.

This has been a long time coming, and the resistance to the perception of a problem is deep, despite vivid descriptions of this by educators, industrial leaders like Bill Gates and Ross Perot, etc. In the late 1980's, the B-school at Cornell considered setting up an outreach program in Europe, and did a survey of alums living there to get a sense of what we should be teaching. The answers were uniform and startling. Basically, they were a mix between laughter and scorn that the US management education had anything to offer Europe. We weren't even in the running.

Meanwhile, our graduate schools have been making it in science and engineering only by a huge influx of foreign students, in some cases 70% of the graduate student body, because we couldn't find qualified American students. In the life sciences, the ratio seems more like 90% from walking around and peering into labs, but I don't know the exact numbers. It seems clear that, if China stopped sending graduate students and post-docs, the Life Science mission in Michigan would be essentially gutted.

So, the top 10% of our students might be able to compete in that marketplace, but it's just not clear to me that the other 90% of them will stand much of a chance, in the next decade, of catching up. The odds would shift if we could use the power of television and marketing to market multi-lingual, multi-cultural education, familiarity with the world, and an ethic of hard work and putting off present pleasure to invest in the future benefits. Instead, despite whip-cracking from above and jawboning about a need for education, the country seems to be heading towards increased parochialism and isolationism, rejection of science, rejection of reading, and focusing on instant gratification and distracting "entertainment" as the end-point of life.

This seems to me part of a consistent and predictable pattern of denial of inconvenient truth, combined with a helpless/hopeless syndrome, combined with a startling inability, at every level, to carry on reasoned discussion and reach agreement on "hard issues". As I'm writing this, the Michigan government still hasn't sorted out the budget, due last October 1st. The Federal Government is ten days from either a shutdown, or a bitter "showdown", knock-down, screaming public battle over the budget, also due last October 1st. Social security is probably out of control. Health care costs are clearly out of control.

Oh, and the credit markets and home mortgages are out of control. The US debt passed $9 trillion, almost half of which was incurred during the current administration and is accelerating in the wrong direction, out of control, and for that matter almost entirely out of sight of most citizens who either don't know, don't care, or care but feel helpless to do anything about it. It is unmentioned in the political debates. Whatever is going on in the middle East isn't impressing anyone either.

So, I have to ask, what fraction of that mess would be improved if all the participants were simply better at math and science?

Right. Essentially none of it. We are not dying, as a country, because of a lack of math and science. It is something else entirely.


We are dying, I would suggest, from an inability to work together and to reason together and to make hard choices together.

So, now we get to the second assumption I refer to well above, that if only we had more technology, oh boy, then things would be fine for the country, you betcha.

Not.

I'm not a Luddite. We don't have to return our technology to the store and get a refund since it didn't work as advertised, although that's an interesting thought. This "better life" we citizens were promised seems to be less and less likely. The promise of having world control and dominion through advanced technology and weaponry also seems to be surprisingly distant.

Maybe, it is not true that "technology will save us." Either individually or collectively.

If more of the same results in more of the same, I'd say we're going the wrong direction.

So, before we rush off to focus what's left of our national treasure on a solution to our problems, we need to have a serious look at whether we're addressing the right problem, and, if so, are we doing it with an intervention that has any chance at all of working as designed. We need to be alert to signs of wishful thinking and denial and avoiding hard-choices and painful subjects, or subjects which produce high-emotions and social conflict.

What about the argument that if we just had more math and science, that level of logical thinking would fix everything? I have to look at our centers of academic wisdom, our universities, and ask how good a job they do at dealing with internal conflicts and making hard choices. I am not impressed. They don't make the choice that a world composed almost entirely of highly trained people, who are strong in math and science, seems to be any better than the rest of us at making hard choices and managing its own affairs.

I referred in an earlier post to the University of Oxford, which has been in business for over 1100 years now. If there was any place that highly educated people should have had a chance to sort out these issues, you'd expect to find it there. Yet what I read in their on-line documents is that they have spent years arguing over what e-mail system to use, and cannot reach any resolution on the issue.

Would more math and science knowledge help them out? I doubt it. Do they just "need more time" on the exam? I doubt it.

Frankly, I'd suggest doing this. Take technology off the table entirely. Imagine for a moment, as I have, that the problem is not that we have insufficient math and science and engineering to be able to thrive and prosper and be healthy and happy as individuals and as companies and as a society.

Then, hmm, what is the problem? Where is this process of prosperity-reaching breaking down?

Whether it is a symptom or a root-cause, it's clear that our ability to get together, put our heads together, and come to grips with our own problems is not very good. In fact, it probably deserves a failing grade, based on how well the annual budgets are coming along and how civil and rational the discussion is about selection of the next US president.

So, back to my opinion of this whole PISA - Programme for International Student Assessment.

It appears to me that the mental model of that assessment is that only one level matters of the multi-leveled biological world of life that we live in - and that is the "individual." And even there, I find the conclusions unhelpful.

I think we need to take that level off the table as well, keeping technology off it, and say, yes, but what else is needed to make this baby fly?

Very bright individuals, by themselves, can deal with some issues -- fewer these days than you'd think, after accounting for not only internal roadblocks like depression and unexplained fatigue and ill-health, but for external roadblocks in getting almost anything done - which is like trying to get anywhere on an airplane this holiday season in the short 2-3 hours that you'd think a jet aircraft could deliver.

In fact, that's a perfect example. We have jet aircraft, that can go 550 miles per hour, and cross the country, potentially, in 4 hours. Yet, to travel from Detroit to Miami, say, for Christmas, it would be good to leave, oh, 1-3 days for the trip.

Would it help if we had even faster planes? Nope. We've run out of what we can buy with faster individual planes, and need to look at how the system of many planes, interacting, behaves.

This is exactly what's going on with education. We've run out of what we can by with brighter individuals, and need to look at the system of how many individuals, interacting, behaves.

Pouring our remaining bank-account into making faster planes or brighter individuals have equally likely chances of fixing the problems, which are not at the individual level but at the "system" level.

Now, this is nuanced and subtle. The behavior at the system level is dependent on what sort of internal decision-making and behavior-generating rules individuals use. If you change the nature of the individuals, you will definitely change the nature of the emergent system behavior. All research in complex systems shows that.

So, I come full circle. The "problem" has to do with something "wrong" inside individuals that our educational system should strive to make "right' -- but the something has close to nothing to do with math and science, per se.

Yes, maybe if science advanced another 200 years at the speed it's going, you could get to the root problem that direction, but we don't have 200 years, so that's not very helpful.

We need to leap ahead, pull our heads out of the box, climb a tree, and see where this path goes.

It seems to me that a closer first-approximation to where the "wrongness" is that we need to address is suggested by words like: morality, integrity, honesty, humility, sincerity, trust, compassion, civility. In fact, a really big word is this one: maturity.

As a society, in the US, the term "adult" or "mature" seems to have been hijacked to mean "old enough to buy pornography and alcohol and cigarettes and drive a car and generally behave like an adolescent jerk and not have to listen to anyone anymore or take advice about anything."

What the educational situation highlights instead is that the US students are the world leaders in what could be termed pride, arrogance, self-esteem, or conceit. Which means they are also world leaders in unawareness of reality or denial of same.

Which means they are breaking precisely the key feedback loop required for any adaptive entity to, well, adapt to changed conditions in its environment -- namely, it has to be aware of the gap between where it is and where it should be, and it has to be responding to that awareness.

Of course, awareness of a gap produces internal conflict, that seeks to be resolved. It will, in fact, be resolved, come what may. So, if the gap cannot be closed, then the awareness of it will generally be shut off instead. That seems to be what has happened here. We don't want to be so far behind, we don't like being behind, it doesn't reflect well on our leadership to be behind, we can't deal with being behind, so we will effectively agree to ignore it and act as if it doesn't matter one bit to us. And maybe it will go away.

Not.
Summary so far:

• Houston, we have a serious problem here.
• It's not going away.
• In fact, it seems to be getting worse.
• Pouring another decade of math and science into it looks unlikely to help, on any level.
• Technology will not save us this time.

and

• We are going to have to grow up and learn how to act like adults and work with each other and with hard-choices and sacrificing some short-term desires for long-term needs, and some personal desires for some social requirements of continued existence.
  

In other words, the "individual" performance that matters today, now, this decade, has little to do with math and science, and a lot to do with how we interact with and relate to each other.

That's what we need to develop metrics and tests for, and that's what we need to go back to the drawing board and figure out how to do, since we obviously don't know it now.

And, incidentally, that's why so much of this weblog is devoted to "religion" and "social feedback".

There are things outside science that we need to get good at, whether science can help us with them or not. If you get past the high-profile single-issue religious zealots, a large fraction of what religious people are trying to tell scientists is the above message.

I care less about whether life was created in 7 days or 7 billion years than I do whether people have some basis on which to grow up and deal with each other that doesn't involve killing the other party as the only imagined "solution." All major religions attempt to provide such a basis.

Scientists say that atheistic philosophy can serve the same purpose, but that's not obviously true based on any society that has thrived based on technology, absent some embedding religion.

The problems we face as a society are not "in the box" or "in the org chart" that science is good at dealing with. They are in the context, the "white space" between boxes, where, historically, science and mathematics have not gone. To be fair, science is increasingly aware that there is something important going on in this "complex adaptive living system" space.

Our solutions involve things like "norms" and the word "should", which are areas, again, that science, trying to fight off a legacy of "proof by intimidation or inquisition" thought it could get away from by being scrupulously "objective" and "distant" .

While there can be a lot of debate about which "norm" is "right", I'd suggest that norms which result in the destruction of all life on the planet, or of our own society, are probably worth re-examining. It would seem to be a wiser choice to select our norms from the set of norms that actually generate a future for us.

That issue can be thought about scientifically, maybe even at some point using mathematics or simulation and animation to gain insight about the implications, downstream, of processes too complex for our finite brains to comprehend.

These advances require focusing attention on how people relate, how they work together, what works and what doesn't, what's worked out how in the past, what works in different cultures and countries, etc. But, those are the humanities, the subjects that "science and math" fixations are driving out of our school systems.

We have to address how we "should" treat each other. It's a hard issue. It's a bootstrap issue, because our inability to face and deal well with conflict and emotion and differing perspectives makes it hard to deal with precisely those issues.

So, it's a feedback loop. We need to spiral it towards better mutual understanding, not away from that. We need to take on pain to do that. And face reality. Both of which we cannot do in a single step, but these have to be approached slowly, hesitantly, but insistently.

I saw a bumper sticker the other day that seems relevant. "I will speak the truth, even if my voice shakes."

The answer is over there, I think.

==========
I promised some interesting information from the PISA executive summary, so here it is.

Males and females showed no difference in average science performance in the majority of countries, including 22 of the 30 OECD countries. In 12 countries, females outperformed males, on average, while males outperformed females in 8 countries. Most of these differences were small. In no OECD country was the gender difference larger than 12 points on the science scale.

This is different from reading and mathematics where significant gender differences were observed.

However, similarities in average performance mask certain gender differences: In most countries, females were stronger in identifying scientific issues, while males were stronger at explaining phenomena scientifically. Males performed substantially better than females when answering physics questions.

Reading is the area with the largest gender gaps. In all OECD countries in PISA 2006 , females performed better in reading on average than males. In twelve countries, the gap was at least 50 score points.

Photo credits:
Amish barn raising (Swarming) by heyburn3 (click on it to go there).
Team crossing stream photo credit: Ollieda
Houston Graphic by the author.