Zen of agile programming

There's an interesting thing called the "Agile Manifesto" which is a set of guidelines about how to write software that actually works and is actually helpful to customers.

I love the overlap with my favorite guru of systems, John Gall, in his book Systemantics: How Systems Fail, and his principle #15:

A complex system that works is invariably found to have evolved from a simple system that works.

This is a good start of an idea,  but like "Lean" and Six-Sigma and many other principles of good software design, there is a subtle but fatal flaw in it, IMHO.

With "Lean" process improvement,  this flaw is encapsulated in usually a few short sentences that say something like "For this to work, a culture change is required".   As if that's a rather trivial thing, like buying a toaster.

As I view it, Agile development, correctly done, works by using the immense power of a closed-feedback loop, giving you, in technical terms, an IIR system not an FIR system,  which is, indeed, infinitely more powerful.   (For details see my prior post Systems Thinking and the Emergence of New Life   and "Gentle introduction to feedback loops" )


Also, it only works by plugging into the power of the living social context and staying very closely connected to it, again, as we would expect any new life form to do, ala a developing fetus in the womb.

Still,  the key aspect to making this whole thing function is a "loop invariant" -- something that is destroyed in the middle of each loop, but is restored to its state before the loop exits, so that it appears to be "constant" viewed from outside the loop.   In Agile Programming,  this loop invariant has to include cleaning up all of the loose ends and "technical debt" -- everything that needs to be done to make the program "simple" again.      In other words, after each step of increased complexity, the change has to be digested,  and fully incorporated, and the whole shebang re-diagonalized back to "simple" again.  This cannot be "left for later", critically.


Sadly,  like "a culture change is required",   "clean up after each step" is much easier said than done.
Others have seen this problem:

There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult.  (Tony Hoare)

Je n'ai fait celle-ci plus longue que parce que je n'ai pas eu le loisir de la faire plus courte.

• I would have written a shorter letter, but I did not have the time. (Blaise Pascal, circa 1650).

In my understanding, this issue gets down to a core issue that is the subtle flaw behind most good advice, such as "To keep your expenses under control, develop a budget and just live within it!." The word "just" in such sentences is a red flag that the complexity and difficulty of this step has been grossly understated.    Similarly "Why can't you just have a place for everything and keep it in its place!"

The problem here is that, in the creation of new life,  it is absolutely unavoidable to go uphill in a thermodynamic sense, from a disordered state to a more ordered state.    And the laws of thermodynamics forbid such a thing from occurring.  Closed systems do not move uphill. Period.

Author T.S. Eliot captures the wished for state:

They constantly try to escape
From the darkness outside and within
By dreaming of systems so perfect that no one will need to be good.
But the man that is shall shadow
The man that pretends to be.
(T.S. Eliot Choruses from the Rock)

In short,  the work of our hands must always reflect the state of our heart, and if we have a chaotic spirit,  there is no way, regardless of tools, that we can build unchaotic systems, let alone give them a breath of life.

Well, more correctly, there is no way as a "closed system" we can do that.  Only an open system, drawing in energy and revitalizing life from outside (as from a womb) can generate actual new life, actual steps up the entropy chain,  actual "value" for the customer.

Unless the LORD builds the house, its builders labor in vain. Unless the LORD watches over the city, the watchmen stand guard in vain.  (Psalm 127, Solomon Song of Ascent)

Summarizing this so far, for agile to realize the potential power, three things must be true:

• Only a truly elegantly "simple" system can evolve in a sustainable way,  so that it will not ultimately crash under the weight of unresolved internal complexity or divergence from external life in which it is embedded.  (All complex systems will suffer tissue rejection.)
• Each cycle of the loop,  true simplicity must be rebuilt, at surprisingly great effort and expense.
•  
• All efforts to rebuild simplicity will fail unless they are spiritually grounded in outside life, drawing heavily on it to move up the entropy curve.

 TECHNICAL DISCUSSION OF THE UNDERLYING MATHEMATICS

If the system is restored to simple  (imagine a diagonalized matrix), THEN it is true mathematically that any single small step (from that SIMPLE state) will be linear, ie, straight-forward and not filled with contradictory cross-terms.  This much is basic matrix algebra and finding the eigenvalues and eigenvectors.

But ONLY one step can be taken, and then the whole matrix needs to be re-diagonalized again, because the underlying Riemann metric in which you are "hill-climbing" is only approximately flat,  not actually flat.   The approximation is only valid for one small step.

In less accessible but far more profound accuracy,  the tensor you are working with, connected into physical and social reality, must be "rejuvenated",  and returned to elegant simplicity by giving up something, by contraction across one covariant and one contravariant index, setting them equal.

The very choice of this word by a mathematician is no accident.   A discussion of basic tensor algebra* is required to understand the precise meaning of these terms.  ( We need to understand that concept, the step of "rejuvenation",  to truly guide our insight into how agile development must occur to succeed.  

By repeating the rejuvenation step,  the algorithm becomes truly recursive, and at each step a profoundly simple system can take one step, and only one step, safely, before everything must be recomputed and rejuvenated again.   Expand and contract.  Differentiate and reintegrate. There is no shortcut, no other path, no way around this thermodynamic truth.

But, because the algorithm is, in fact, completely recursive,  you can get better and better at it, faster and faster, over time.  The algorithm never changes, even though the software it is generating is, in fact, being continually migrated along a path in curved metric space towards some objective. 

This ratcheted hill-climbing algorithm will, in fact, work -- but only on its terms, not ours. 

At least, that's how I understand it.

-----
*  See, for example,  as easy a discussion as you can find for non-mathematicians of tensor algebra in Introduction to General Relativity by Adler, Bazin, and Schiffer, chapter 1.  (McGraw Hill, 1965.)    
A less lucid reference, but one that is at least online, is Mathematics of classical and quantum physics, by Byron and Fuller,