New York City Schools get graded

Today the New York City School systems are getting done to them what all our schools been doing to students - ranking them and assigning grades -- and they don't like it one bit.

I've posted before on how some things, like the "magic dice", have no "best" and cannot be put into some kind of rank-order.

The New York Times today has an article "Schools brace to be graded" that runs into this problem head-on and is producing a great deal of social conflict.

The point is an important one and I want to mention it again. I keep on seeing cases where people, absolutely sure that there must be some way to do this, valiantly try new and more complicated ways to "get it right" and rank something.

It is a dangerous concept and we need to grow up and get over it. It is a damaging concept. The whole idea is one of the pillars of intense competition between people, cultures, and nations and one of the ultimate causes of outright warfare - to be "best", to be "number one" - our people are killing themselves or going into deep depression over a quest that can never possibly be achieved because the idea is meaningless.

First they try one measure, which everyone knows is incomplete. Then they try a variety of different measures, which are also incomplete. Then, that's complex and confusing to have some high and some low scores, and they know nothing about "unity in diversity", so they try everything they can to "combine" all that information into a uniform single number or letter.

So, first they'll compute an average, then maybe a "weighted average", then something like the square root of the sum of the squares, then even more complex calculations that raise up so much dust that no one can figure out what they did, like the New York Schools, probably trying to be more "accurate" or possibly hoping that no one can challenge what they can't understand or explain.

But I challenge it, on fundamental grounds, that have nothing to do with how it was "computed" at all. It doesn't matter how it was computed - something as multidimensional and complex as a person or a school cannot be meaningfully reduced to a single number, period. The whole concept is flawed.

There is a direct analog in physics, which we can be confident is simpler than society and life as a whole. Scientists have a concept called "rank", but it is the nature of the beast that some set of measurements can be reduced to, and that is as far as you can reduce it.

So, yes, a few things can be reduced to "scalars", which are single numbers, like "temperature", that don't depend on context or what the observer is doing at the time.

Most physical things, however are more complex than that. The next more complex thing than a scalar is a "vector", which you may vaguely recall from school - a directed arrow kind of thingie, like "velocity". Velocity is different from speed, in that speed can be reduced to a number, like 85 miles per hour, but velocity includes a direction as well, such as "85 miles per hour heading due North." And, of course, physical things have those pesky "units" or "dimensions" that are somehow attached, so that talking about a number without units doesn't get the answer right.

So, most easy classical mechanics requires these "vectors" to write down the equations at all and solve them. You really can't even begin to solve the problems using just scalars, period.

That, however, is just the beginning of complexity. Scalars are the first of a long series of types of things called "tensors", and an be described with a single number. Vectors are the the next one up, described with "arrows", and cannot be reduced to single numbers, period.

Another example would be the torque you want to apply to something. You can't say you want a torque of "2", and skip the direction part. Clockwise or counter-clockwise? It matters a lot!!

The complex part cannot be left off just to make your math "simpler" or because you never felt like doing the work required to learn how to do the math correctly. It will not "come to you" -- you have to go to it.

Then there are things physicists deal with daily that cannot be reduced to vectors, even. An example is the "electromagnetic field". This field requires the next higher level tensor,. a second-dimensional one, to capture it correctly, which needs a matrix of numbers and rules for how it changes depending on where you stand and how you're looking at it.

If you use that math, it is actually relatively "easy" to describe in equations, and you get the "Maxwell equations", and can correctly figure out what's going on. If you don't use that math, you can't get answers that match reality and should stop trying.

Things get more complex than that fairly quickly. To describe gravity requires a mix of tensors up to a 4-dimensional thingie called the Riemann space-time curvature tensor.

Once you stop kicking and screaming in protest and accept that you have to use complex tensors, not scalars, and figure out how to do that, the equations suddenly get much easier, actually. It's like why scientists use metric not feet and pounds -- not because it's more sophisticated, but because it's easier.

Alfred Einstein stated that "All physical equations are tensor equations." That's it. You can't get away from this, if you believe Einstein.

And, the equation for space-time, in that formalism works out to this:

R= zero

Cool. Once you start putting mass and planets in there, it gets messy fast, but in a way you can manage with just careful bookkeeping. If you use tensors, you can write simple equations, and solve the problems. If you don't use tensors, and try to use scalars, forget it.

So, that's our social dilemma here. The description of people, schools, sports-teams, presidential candidates, etc. all require a level of math that we wish wasn't true, so we just go on pretending that we can say something meaningful without going to all that effort.

And, we end up assigning "grades" to students, then trying to aggregate different "grades" into single overall "grades" (grade point averages), and then trying to make meaningful decisions based on those single composite numbers, like rank students or schools -- and discover that we get absurd results.

Then, we punish those with "low scores" and apply pressure for them to "shape up" or "teach to the test", and have a mess on our hands.

The core problem here is that the physical objects we are trying to study - school systems -- are not intrinsic "scalars" but are probably at least level 3 or higher "tensors".

Actually physicists and mathematicians squabble over exactly what kind of complexity is required and whether it should be "tensors" or something else -- but they all would agree instantly that you can't reduce the world to a set of equations using just "scalars", like grades.

The first question we should have asked is "What is the smallest rank tensor we can use to meaningfully capture the complexity of this thingie?" and it would immediately be clear that scalar numbers ("grades") are too simplistic.

We don't like that answer, so we just go on doing the wrong thing, then we wonder why we have so much conflict, and why some well-loved schools end up getting low grades. Then we set social policy, public policy, and feedback based on those "grades".

Evaluate, yes. Try to reduce to single scores using the axe of some magical computation? NO!
New York City Schools

What I learned at Johns Hopkins last week

Well, I saw something completely unexpected yesterday.

I wasn't posting here for most of last week because on Friday I completed a course, "Social and Behavioral Aspects of Public Health", at Johns Hopkins School of Public Health. I thought it was a good course and covered many key ideas, although I did wish it had gone into them in a little more depth.


But, I am a finishing 3rd year student, (my last class! Hooray!). and most of the class had just started two weeks ago, so I could understand the need to not overwhelm people with new concepts. And that's what I thought was happening, but now I'm not so sure.

This is like those scenes in the movies where the music changes and everyone knows that the monster is approaching but our hero and heroine happily play on, oblivious.

During lectures, sometimes we would have a simple summary slide with content such as "Poverty is a carcinogen." We were supposed to evaluate that assertion, tease it apart, sort out what portions were true and how you could tell. This is part of a debate that's been raging for at least 400 years.

Many of these lectures were met with a startling silence by the students, who often had no questions at all. This surprised me as I thought there would at least be a heated discussion. Well, I thought, they're tired from working half the night on their classwork, or don't want to ask "dumb questions."

Still it was eerie to have the professor ask something and the room of 100 or so just sit there.

After the class, in the big blue shuttle to Baltimore - Washington BWI airport, I discovered something I wish I'd known the first day, as it would have totally changed my behavior.

I chanced to ride with another MPH student I recognized and asked her what she thought of the class we'd just had. I hit a nerve. She had thought the class was a total waste of time and money, and put up with it just because it was required. She thought, basically, that the lessons the class taught were stupid, wrong-headed, wrong, soft, politically-motivated, you name it, and she had already discarded all of her notes. She was just livid.

Wow. None of that had come out in class. And, obviously, "my mileage varied." I liked the course and I don't think I'm an easy sell. I'm used to executive education programs where "students", often CEO's of companies, wouldn't hesitate a second to challenge something they disagreed with.

Apparently I had fallen into the common trap of interpreting stony silence as agreement, or consent. In point of fact, it was total disagreement and scorn, suppressed by a need to just complete the required course, hold one's breath, and put up with all this "psycho-babble" for two weeks. (She didn't say "psycho-babble", but could have.)

So we had missed a tremendous teaching opportunity to get this debate and dispute out on the table and have at it. What a great opportunity to get our feet wet on what it means to assert that "A causes B", and how we "prove" things, and what level of skepticism is expected, and what the burden of proof is on someone asserting some new claim, and how to meet that burden, etc.

It would have been a perfect chance to show a snippet of Crime Scene Investigator's CSI TV show where CSI Head Gil Grissom could lecture us all on the need to suspend our suspicions and "let the data talk." We could have viewed a few cases where it was way too easy to believe that Mr. Jones obviously "did it" when, in fact, it was Miss Smith, in the Kitchen, with a lead pipe.

We could have talked about how civilized grown ups in the field disagree with each other's conclusions while remaining cordial and committed to careful ways to defend against being too gullible (a "type I error") or too skeptical (a "type II error").

But, at least for this one student, that chance was missed. She had interpreted this class as just one more of those annoying things in life where a person in authority states or does something stupid and the best thing to do is just shut up and pretend you agree. In fact, silent and sullen obedience is the expected and demanded and rewarded behavior.

I guess it was rewarded here too, because I think she "passed." Hmm.

Way too many years ago, before I had taught in trade school or taught MBA's, a book came out titled "Summerhill", I think. It described a school in England that I actually went to go visit because of the book. The school challenged the prevailing "infectious disease" notion that I can recall quite well:

Courses are something like the measles. They are something you "have", and then, since you've "had it" you don't need to "have it again."

Again, wow. I had thought that concept had died in the 60's. It seems to be resurgent. Or maybe it never left and I'm just finally looking up and noticing it.

Now, I'm the first to agree that I went into undergraduate Engineering at Cornell, after reading C.P. Snow's Two Cultures, because I just couldn't figure out how to deal with classes where the teacher would ask "What did Hemmingway mean when he said X?" and I had no idea what to say next after I offered an opinion and the teacher told me I was "wrong". What the heck? What's with that?

At least in Engineering, if you say something should work and someone else says "No, it shouldn't" you can just both happily go down to the lab machine-shop and build one and just see whether it flies or not. No one ever wastes time talking about the "true nature of causality."

We'd just happily compute what size resistor to put at this point in a circuit without losing sleep over what the meaning was of "resistance" and if we could actually be "certain" that changing the value would have the desired impact on the radio receiver actually working. If in doubt, put in a variable resistance potentiometer ("a pot") and turn the screw to change the value while watching the output on an oscilloscope, and when you got it where you wanted it, Bingo, pull out the "pot"and measure what resistance it was set to and solder a permanent resistor of that size into the circuit and go play volleyball. No big deal.

Maybe it's because I'm looking at social issues more than I used to, or maybe it's because society is changing, but that sort of way of gaining an answer to a question seems to be vanishing as the expected behavior of people.

Without some training and skill in the tools of Public Health, or other rigorous but often qualitative fields, we've reverted back to the Middle Ages where causality is either magical or determined by which "authority" one follows blindly.

Again, wow.

So, if I hold out my pencil and release it, and it falls to the ground, and I ask "Why does that happen?" I'm as likely to hear "God made it move" as "Gravity."

So, hmm. Is this an "either/or" question or an "and" question or what? Personally, I prefer to think that "gravity" made the pencil move, and allow that, if you like, you can add "... and God made gravity." At least with the "theory of gravity" I can write some equations, design equipment, know exactly how fast something will fall, plot trajectories, etc. It's a "theory with meat on the bones" that I can rely on to build stuff that works. I don't get much "predictive value" out of "God made it move."

But, I guess if you never had the math, and never did "get" introductory Physics, and the concept of "potential energy" baffled you, so the equations never gave you any insight or power, then it's pretty much equivalent to you to say "God made it move" or "magic made it move" or "gravity made it move." They're all invisible anyway, right?

I was busy raising children and missed the whole 80's and 90's trend towards cultural relativism applied to everything, including physical laws, so that "your idea of how gravity works is no better than anyone else's" and we should agree to let all three just get along - physics, magic, or God."

Besides, frankly, hey, when you get right down to it, I can't "see" gravity anyway. All I can "see" is the pencil. Your invisible force against my invisible force, it's a tie, right?

All of which gets me back to class. I guess we might need to add an introductory class that we never needed before, to socialize students to an accepted way of challenging assertions and assumptions and accepted ways of meeting the burden of proof without being blindly stubborn or gullible about it. We need to know when and how it's appropriate to raise our hand and say "How can you prove that?" in a neutral, polite, but insistent tone.

As about a zillion (technical term) of my previous posts discussed, a key requirement for a "high-reliability" culture is "mindfulness," which requires the ability and sensed-permission and sensed-expectation that you will surface questions you have, not submerge and suppress them.

If we can't have that discussion first, all the rest of this business with models and hypothesis testing and "p-values" and study design and statistical tests is, indeed, just magical rituals that you have to go through for some stupid legacy reason in order to get published. All this demand for "evidence-based" practice is just a waste of time, then. No wonder students are baffled by it.

Well, we all know the rule that "All Indians walk single-file .... at least the one I saw did."

So, I'm extrapolating to an entire entering class of students from observed puzzling behavior of stony silence and from one accidentally chosen student's opinion in a cab on the way to the airport. That suggests an underlying teaching opportunity that maybe I'm imagining or maybe is real.

How would we decide which it is?

I'm concerned that not a single student challenged the teachings and yet clearly, from this and other conversations, many others I checked with also disagreed -- in complete silence.

Again, wow. And these are all students with undergraduate degrees and at least two years work experience and decent GRE scores. Maybe a third of them are already Medical Doctors. (MD's)

What have we done? Can we trace this defect back upstream and find out where it's coming from? (You can click on this next image to zoom up to a readable scale).


And how can we undo it? And how could we measure our impact and know whether we had succeeded or not?

Those are good Public Health questions that deserve some time on the agenda. They're also major business problems that directly short-circuit techniques like "The Toyota Way" that I've discussed, that require that everyone should work with their eyes open and with permission, and even expectation, that they'll spot things that need to be changed and announce them.

An army of silent, obedient, sullen, blind robot lemmings is not a very solid basis on which to build a competitive economy or a good public health infrastructure that actually works, or an army that works, or anything that works, instead of one that everyone pretends works.

What have we done to our children?