One Bite At A Time: Partitioning Complexity
Sufficient unto the day is the evil thereof
–Matthew 6:34, Christian Bible, King James Translation
I was the third best programmer on our senior practicum team of five. Bret and Kevin could both code circles around me. They could both hold ridiculous amounts of complexity in their heads. I especially remember the time Kevin showed me thousands of lines of 6502 assembler hand-written on a roll of teletype paper (it was cheap). If I did that I’d just be lost.
In seven years at Facebook I’ve been surrounded by complexity mavens the whole time. Sometimes when I programmed in our code I just drowned in complexity. Yes, sometimes, I made steady progress. Overall, though, I started to doubt my programming skills, retreating into people-y stuff where I had a (relative) advantage.
A recent programming project of my own reminded me that just because I can’t handle lots of complexity at once, it doesn’t mean I can’t program. I can program, but because of my weakness I use a style that partitions complexity instead of consuming it whole. Here are the elements of that style.
One Problem At A Time
- Run/Right/Fast. The classic problem partitioning strategy, taught to me by my dear departed Pappy on the first day I started programming, is to make the code work then make is clean then make it fast (if necessary). Solving the “correct/clean/performant code” problem all at once can be overwhelming. Don’t be overwhelmed. Solve one at a time.
- Structure/behavior. If I’m changing the structure of the code (refactoring), then I don’t ever change it’s behavior. If, as often happens, I see something wrong with the behavior while I’m refactoring (“How the hell did this ever work?”), then either I finish refactoring (and commit) before adding a test and changing the logic (and commit again), or I abandon the refactoring, change the logic, and refactor again from scratch.
- Interface/implementation. If I’m changing the interface by which some logic is invoked, I never change the logic itself at the same time.
- Test/code. TDD is a “one problem at a time” strategy. Even if I don’t know how to implement a feature, I can almost always figure out how to test for it. (If I can’t figure out how to test for it, maybe it’s just not time for that feature yet.) I put my fears about implementation to one side and just write the test. If I’m implementing and I don’t see how to make the implementation general, that’s just the next test.
Code Structure
- Composed Method. Functions are composed of calls to functions at the next level down in abstraction. This lets me read an outline of what is supposed to happen (encoded in the name), before reading one level deeper of how it is supposed to happen. If I’m curious, I can dig further, but I never have to confront a big twisty bundle of state and logic all at once.
- Mutable core/immutable leaves. I hate, loathe, and despise aliasing errors. To avoid them I compose the leaves of my object graph from immutable objects referred to by mutable objects towards the center of the graph.
- Move logic to data. When I see one object send multiple messages to another object and then combine the results, I move that logic to the other object. When I deal with complexity I want the smallest possible scope. Moving logic to data gives me a smaller scope.
- One pile. When I see ugliness scattered around, I shovel it up and put it in one pile, even if I don’t have a clean way to handle it. I’d rather have ugly code in one file/class/module/element and not in ten others. Most of the time I don’t have to deal with the complexity.
Workflow
- Constant/compute. If I need to compute a value in a complex way and then use it in a complex way, I will hard-wire a constant just good enough for the current test to pass, use it well enough to get the test to pass, then replace the constant with an expression. Sometimes this requires migrating the constant to where it can be computed.
- Predict test results. Before I run tests, I predict the results out loud. “Green”, “Null pointer”, “Index out of bounds”, “Green”. If my prediction is correct, this sets me up for the next step. If my prediction is wrong, I know to stop and think. If I let the test results be a surprise, I have to deal with my implicit expectation and the actual results at the same time.
- Red/red/revert. If I write a failing test, fix the code, and then the code isn’t fixed I sometimes just revert to the state with the failing test. Sometimes I revert to the last known green. I think I do this when I have tried something complex, I know I’m being speculative, and it didn’t work out. Reverting is like downshifting. Now I go forward in smaller steps. I sometimes debug forward, though.
- Red/red/red/green/revert/red/green/red/green. If I debug forward and eventually get the test to pass, I will sometimes revert and redo an hour or two’s work in smaller steps. If I want to learn to work in smaller steps, replaying what seemed like a complicated step is a great teacher.
- Waiting. Massimo Arnoldi taught me this one. When I have ugly code but I don’t know what to do about it, I let it get uglier first. I’m not ignoring the complexity. Ignoring is suppressing appropriate attention. I’m waiting. Waiting is appropriate inactivity in the face of uncertainty.
- Stories. Stories separate thinking about an intended effect from thinking about how to achieve that effect. What change do we hope to see in users’ behavior (or, for infrastructure, the system’s behavior)? How will we measure it? How will we notice unintended consequences? Okay, with that in mind, how are we going to change the system?
Themes
Partitioning complexity is not a tidy process (irony much?) Sometimes I have to step off into excess complexity before I know it’s too much. Thank goodness for reversibility. Sometimes I create complexity, then partition it (conquer and divide). Sometimes I can see complexity coming and head it off (divide and conquer). Sometimes when I head off complexity I end up creating more complexity and I’m back to being thankful for reversibility.
Many of the strategies above sacrifice short-term efficiency in order to keep my feet moving. What really slows me down is not programming slowly, it is getting overwhelmed, losing my confidence, and not programming at all.
My final observation is just how frequently I fall off the complexity wagon. I want to be a genius programmer. I ought to be able to handle this. This time it’s different. Then I look up, realize I’ve been two hours without green tests, revert, take a walk, sit down, and go back to eating one bite at a time.