The more complex the world gets, the more we need models to simplify it. One of the models I return to often is fitness landscapes, which can help solve problems, design better experiences, and explain the world around us.
Imagine you and a group of friends are on a team playing a game.
The game takes place on a huge playing field with rough, mountainous terrain, like the Himalayas or Alps. The only goal is to increase your team’s average altitude. This seems easy enough, but there are a few catches: (1) any player can only see a few feet ahead of them, (2) the terrain slowly changes over time, and (3) if a player drops below a certain altitude for long enough, they’re eliminated. Given these rules, what strategies would your team use to find the highest peaks?
This is a metaphor for the “game” that species must play to survive in an ecosystem.
The terrain is a fitness landscape representing a library or design space of every possible variation of organism, spread out over a nearly infinite surface. The closer together on the surface, the more similar the genotype. This means single species would be clustered together. Dogs would be near wolves, far from fish, and even farther from fungi.
Altitude indicates the fitness of the organism — or how likely it is to survive in a particular environment. The higher it is on the landscape, the better the design and more fit the organism. Below a certain threshold, organisms can’t survive and species go extinct.
As a model, landscapes can help show us visually and mathematically how to find the best designs. The original concept was developed by evolutionary theorist Sewell Wright in 1931, and focused only on biological entities. But a design space could represent almost any set of possibilities — as long as it has building blocks or variables that combine into many variations, each with a value (or fitness level) that can be assigned. This means it could apply to design spaces of problems, equations, technologies, strategies, memes, or even sets of LEGOs.
Features of landscapes
A vast majority of the variations on a typical landscape are bad designs. These are oceans of low fitness, below the surface of which organisms are incapable of survival or reproduction.
But certain regions — springing out of the oceans like islands or continents — are full of a range of potential variations, all with some usable level of fitness. The basic features of these regions of terrain are:
Local peaks or plateaus — A point or area of high fitness where all surrounding paths go down.
Global peak — The highest peak in the region. The fittest entity in the area. The best design of all similar variations.
Valleys — Flatter areas of low fitness adjacent to hills and mountain ranges.
Pits or Crevasses — Deep holes of low fitness below the “sea-level” of survival.
Peaks are good. Pits are bad. And crossing valleys is very risky: you could find higher fitness, but likely not.
The unconscious process of evolution drives genotypes uphill over time, finding and settling on peaks of fitness until the landscape shifts or some other factor forces a move. More on this later.
Making progress — in society, a team, or life — isn’t straightforward most of the time. Knowing where you want to go is generally the first step, but the destination can be very broad. And even if there’s a specific goal, the path to get there may be very indirect.
As strategy transitions into execution, it’s important to understand how these attributes affect progress. If an effort is managed or guided the wrong way, it may be doomed to failure no matter how difficult it is.
Project management as a discipline works great, but not with everything. Managing large, more uncertain endeavors in particular has been problematic recently. The wide-scale ongoing effort to fight COVID-19 and return the world to normalcy brings this challenge front and center.
Why are certain efforts harder than others and how do we navigate them? How were we able to accomplish such large scale collaborative efforts such as the Apollo program or Manhattan Project, but can’t do the same thing for curing cancer?
If we want to build, we need to understand the answers to these questions. The following is a framework for classifying efforts by the certainty of both their objectives and the paths to achieve them. Knowing which “mode” an effort is in is critical to understanding and managing its progress.
How do we classify efforts into modes? The best paradigm I’ve come across is the how/what quadrants.
In his 1994 book “All Change!”, Eddie Obeng described 4 different types of projects along with the difficulties and peculiarities of each: quests, movies, painting by numbers, and fog. It turns out putting a project on both the know how and the know what scales tells you a lot about how it should be managed.
Venkatesh Rao explored the concept much further in his essay on the “Grand Unified Theory of Striving”, pulling in other ideas like convergent thinking, critical paths, and lean methodology. Venkat’s visualization of the critical paths and point frontiers of each quadrant is a particularly insightful way to think about the concept.
Defining the dimensions and modes
Here’s how I’d describe the axes of the 2×2:
“Why”-axis — Know what vs. don’t know what. Do you know what the goal is? How specific is the desired outcome? Not knowing the goal (or having a very broad idea) is in the realm of divergent thinking: there are many potential solutions and progress can be non-linear. It’s the exploration phase of the explore vs. exploit tradeoff, searching for goals or areas of value.
Knowing what and why is in the realm of convergent thinking: there is a single “correct” solution or destination. It squarely aligns with Peter Thiel’s deterministic approach of viewing the future: “There is just one thing—the best thing—that you should do.”
Horizontal-axis — Know how vs. don’t know how. Is it known how to accomplish the goal? Are the bottlenecks or resource-sensitive parts generally understood? When you know exactly how to accomplish something, there is a clear critical path1 (the red lines in the diagram below). Other paths of effort may still be required, but they are oblique with more slack, running parallel to the critical path.
Knowing how allows you to operate lean because you can—in theory—use the least amount of resources necessary to get the job done. In the fat mode of operation, you don’t really know how to reach your goal. You can’t be efficient because you don’t know how to be, and there will be a lot of slack in the system. The path is determined opportunistically as you go, with critical paths only in smaller subsections.
“An old story tells of a visitor who encounters three stonemasons working on a medieval cathedral and asks each what he is doing. ‘I am cutting this stone to shape,’ says the first, describing his basic actions. ‘I am building a great cathedral,’ says the second, describing his intermediate goal. ‘And I am working for the glory of God,’ says the third, describing his high-level objective. The construction of architectural masterpieces required that high objectives be pursued through lesser, but nonetheless fulfilling, goals and actions.”
John Kay, Obliquity
All efforts, from daily personal projects to global collaborative endeavors, fit in a webbed hierarchy of abstraction.
Understanding the full hierarchy of an effort is critical to accomplishing it, along with its higher-level objectives in the long-term. Not understanding it can result in bad planning, mismanagement, and failed expectations.
Ladder of Abstraction
“…the most powerful way to gain insight into a system is by moving between levels of abstraction.” — Bret Victor
The ladder represents a top-level concept or domain, with each rung a subset of the one above it. The rungs move from abstract at the top, to concrete at the bottom. The lower down, the more detailed and specific. The higher up, the broader and more abstract the concept.
The model is very simple, and can be applied to almost any discipline with a hierarchy of nested groups. This includes applying it to efforts.
First of all — what do I mean by effort?
An effort is the active search for the best outcome of an objective. It encompasses both the objective and the pursuit of that objective — both of which are not fixed and can evolve over time. The objective always has some boundaries, but otherwise can be very broad (“solving climate change”) or very narrow (“double next-month’s sales volume”).
All efforts are multi-scale and nested.1 This means we can put them on a ladder of abstraction, each rung with an objective or method that’s a prerequisite of the one above it. Lower-level goals are nested in higher-level purposes. Good project managers do this intuitively when breaking an objective down into tasks and sub-goals, mapping their dependencies.
Because efforts can have many dependencies and relationships aren’t just one-to-one, they exist in more of a webbed hierarchy of abstraction than a ladder. A simple one-dimensional ladder of abstraction is just a slice of the larger hierarchy.
Here’s an example of a hierarchy of abstraction for the efforts relating to Covid-19:
What’s the best way to determine the hierarchy of abstraction for an effort?
A simple way to move up and down the ladder is the Why/How Chain. To move up, ask “Why?”; to move down, ask “How?”. Many know this technique from the Toyota Production System’s method of asking 5 Why’s to find the root cause of an issue.
You can start by finishing the phrase: In what ways might we ___? This method can work on almost everything, from large-scale efforts to small-scale jobs-to-be-done:
⬆️ Why? To make my home look good.
⬆️ Why? To hang a picture.
❇️ In what ways might we drill a hole?
⬇️ How? Use a drill.
In the Covid-19 example, you could start at whatever level is most relevant to you.
❇️ In what ways might we provide better medical care for COVID patients?
⬆️ Why? To stop people from dying and reopen the economy.
⬇️ How? Protect medical workers from getting sick.
⬇️ How? Source and distribute PPE.
⬇️ How? Contact regional manufacturers.
There will always be multiple “how”s, which is the essence of breaking a goal down into sub-goals. There can be multiple “why”s as well, especially the further you go down the ladder. But high up in the hierarchy the whys and hows become more and more vague. This means you have to approach them in a completely different way.
Abstraction = Obliquity
Knowing where an effort fits on the hierarchy is the first step. Now we need to understand how the different levels of scale need to be treated.
This is where John Kay’s concept of obliquity, from his book of the same title, comes in.
To solve a problem obliquely is to solve it through experiment and adaptation. In general, the bigger the scope and complexity of an objective, the more indirect the path is to achieve it.
The ladder of abstraction is a proxy for obliquity. The higher on the ladder, the more adaptive the problem should be solved. John Kay: “High-level objectives — live a fulfilling life, create a successful business, produce a distinguished work of art, glorify God — are almost always too imprecise for us to have any clear idea how to achieve them.” In the process of making progress on these objectives, we don’t only learn how improve, but “about the nature of the objectives themselves.” You’re wayfinding, rather than following a prescribed path.
The lower on the ladder, the more direct. “Directness is only appropriate when the environment is stable, objectives are one-dimensional and transparent and it is possible to determine when and whether goals have been achieved.”
The following table compares the different aspects of both ends of abstraction.
Clear and simple
Loosely defined and multidimensional
Most outcomes are intended
Outcomes arise through complex processes with no simple cause and effect
Interactions with others
Limited and predictable
Dependent on many variables, including interpretation of them
Range of available options is fixed and known
Only a subset of options are available from successive limited comparison
Can be described probabilistically
Uncertain: Range of what might happen is not known
Insists on consistency: always treating the same problem the same way
Consistency is minor and possibly dangerous — rare that same problem is encountered twice
Conscious maximization of objectives
Adapt continuously to changing circumstances
Consistency is vital when you’re low on the ladder, not so much higher up. “The oblique decision maker, the fox,” John Kay remarks, “is not hung up on consistency and frequently holds contradictory ideas simultaneously.”
But the real power of solving an oblique problem lays in adaptation: “If the environment is uncertain, imperfectly understood and constantly changing, the product of a process of adaptation and evolution may be better adapted to that environment than the product of conscious design. It generally will be.” There is no map, so instead you have to wayfind and look for clues in front of you, making your way with the tools you have at hand.
Keep in mind again that this is a scale — it’s rare that an effort would completely check all the above boxes for either Direct or Oblique. The point is that efforts always fall somewhere on the scale and that this determines the best methods to pursue them.
Consequences of Misplaced Directness
I’ll try to keep this section short, as whole books have been written on the consequences of misplaced directness. See Nassim Taleb’s Incerto for example.
Attempting to approach a large, complex effort too directly almost always leads to failure — or at the very least a failure to meet initial expectations. Directness is only appropriate when the objective is one-dimensional and the path to achieve it is known.
The intention was to create a new Brazilian capital from scratch that was truly unique and modern, paying special attention to cars and traffic flow. (This was the same time period the U.S. began building out the intercontinental highway system.)
As time went on, unforeseen circumstances in the messiness of the real world intervened. Overpopulation drove traffic congestion, slums, and general inequality. The focus on form over function from the top-down design caused alienation and poor quality of life. This is exactly why any such an effort can’t be planned with precision. Not only are the details of the true goal not understood, but the methods to achieve it involve unpredictable complexity. They’re in the world of “extremistan” as Taleb would say.
“The structures in this artificial capital are impressive,” read an FT article on one of the architects, “yet few want to walk its barren streets. Politicians leave as soon as possible to return to grittier, but livelier, Brazilian cities.”
The Brasilia master plan was partly based on architect Le Corbusier’s misplaced utopian vision of creating the ideal city. Corbusier’s work also included the Indian city of Chandigarh with similar consequences. This was, in the words of John Kay, “the hope that rational design by an omniscient planner could supersede practical knowledge derived from a process of adaptation and discovery.”
Many overfunded startups suffer from the same fate. When you have access to seemingly unlimited resources, it’s easy to be fooled into thinking you can build your exact vision into reality. But these visions generally exist in a complex world of human culture, desires, and economic feedback loops.
Quibi — the $1.8 billion funded short-form media startup — is case in point. Ultimate success or failure remains to be seen but results have yet to come close to expectations. They had the resources to build a particular product and business model into reality — a reality where most customers don’t seem to want what they’re selling. A destination was chosen on a map that couldn’t be seen.
Magic Leap has an amazing vision of seamless AR glasses to enable a digital layer on top of the real world. The actual objective in this case might actually be the right destination. But the complexity of the problem means the approach still can’t be direct. Currently their product has seen minimal success as they struggle to find a sustainable business model.
WeWork seemed to start at the right level of abstraction, then an ambitious “visionary” founder is given unlimited funds, and a direct approach is applied to a still-oblique problem.
Summary: The Right Strategy for the Right Level
All efforts, and the efforts within them, can be placed on a ladder of abstraction. The higher up you go, the less concrete the objectives and less straightforward the methods to achieve them.
Where the effort falls on the scale is critical to the strategies for making progress on them. Direct, methodical approaches are only appropriate at lower, smaller-scale levels. This is where it’s good to plan details, use processes, and keep things consistent.
You can still have a grand, abstract vision. You just need to wayfind to get there: working from the bottom up, adapting and evolving the path while shaping and refining the details of the goal. Keep things flexible, adaptive, and opportunistic at the top.
Organizations: Given the definition of an effort above, what about coordinated groups of efforts or goals? ↩︎
In the book The Origin of Wealth, Eric Beinhocker describes organizations as “goal directed, boundary-maintaining, and socially constructed systems of human activity. . . . There is a boundary distinguishing the inside world from the outside world, and the goals of the organization drive activities that lower entropy inside the organizational system.” This is the best description I’ve come across given its abstract nature. But I’d like to propose simpler, yet still compatible definition.
An organization is a group of people pursuing one or more ongoing efforts, generally with the same high-level objective. This means an organization can be anything from seven-person hunting parties, to fleets of exploratory vessels, to philanthropies, to a multinational conglomerate.
In April, Marc Andreessen put out the call to build. It was in response to our failure to control and mitigate the effects of Covid-19 — institutions on every level were unprepared for the pandemic, and have continued to show their inability to quickly find and scale solutions.
But more than anything it was in response to our failure to build in general. We chose not to build, he claims. “You see it throughout Western life, and specifically throughout American life.” The problem isn’t a lack of resources or technical ability — it’s with supply and demand of desire. Demand is limited by our ambition and will to build. Supply is limited by the people and organizations holding it back.
Andreessen is generally an optimist, which is why I see his essay as positive in overall tone. But it was also somewhat of a mea culpa. Andreessen has for years been on the other side of Peter Thiel’s view of modern technical stagnation.
Thiel’s view may be too pessimistic, but there’s a kernel of truth to it. If you’re familiar with the history of tech and innovation, something feels different. The late-1800s to mid-1900s had explosions of innovation in fields from medicine to consumer products, transportation, energy, communication, computing, food, and more.1
This is the introduction to a series of ongoing essays centered around the question:
What frameworks can help us build more, better?
And further attempting to investigate the answers to the following:
What are the best ways to approach solving big, complex problems?
Why are certain efforts so much harder to achieve than others?
How are these efforts best managed at every level?
How do we build things faster? (Without sacrificing quality or safety.)
What is holding us back from building more?
How do we overcome these barriers?
Many of these lessons apply not just to “building” in the physical sense, but for solving problems, scientific discoveries, improving systems, and making progress overall. Building in a way is symbolic. It represents making big, necessary changes to move humanity and our planet forward. This can be building something physical or digital, pushing the boundaries of fundamental research, or trying new uncertain ways to solve problems.
It doesn’t even have to be anything new or innovative per se. Andreessen gives many examples of expanding existing tech: housing, infrastructure, education, manufacturing. Even preservation and restoration — in many ways opposites of building — can still apply. In the early 1900s, President Teddy Roosevelt established over 230 million acres of public lands and parks. This added an incalculable amount of value to future generations. I would love to see E.O. Wilson’s Half-Earth Project executed at scale. This is in the spirit of building: making progress and pushing humanity toward a better future.
Here’s a preview of some of the specific topics I want to explore in the series: Ladders of Abstraction (why/how chains), Oblique vs. direct approaches, Modes of effort (why/how quadrants), traversing fitness landscapes, the explore vs. exploit tradeoff, the role of trust in building things fast, forcing functions, and the specific methods we used to accomplish large-scale collaborative efforts such as the Apollo program, the Manhattan Project, etc.
Wayfinding Through the Web of Efforts[8 minutes] — Putting goals on a ladder or hierarchy of abstraction. Defining efforts and their multi-scale nature. Determining the hierarchy of efforts using a why/how chain. The difference between making progress directly and obliquely, and the consequences of misplaced directness.
Managing Modes of Effort[10 minutes] — A framework for understanding how managing progress differs across scales of effort. Classifying efforts into four modes on the how/what quadrants. Defining the modes and how they fit on the hierarchy of abstraction. A Covid-19 case study. How to manage an effort based on its mode.
What was different about this era? The following is a good rundown from Vaclav Smil’s book “Creating the Twentieth Century” on the remarkable attributes of the pre-WWI technical era:
The impact of the late 19th and early 20th century advances was almost instantaneous, as their commercial adoption and widespread diffusion were very rapid. A great deal of useful scientific input that could be used to open some remarkable innovation gates was accumulating during the first half of the 19th century. But it was only after the mid-1860s when so many input parameters began to come together that a flow of new instructions surged through Western society.
The extraordinary concentration of a large number of scientific and technical advances.
The rate with which all kinds of innovations were promptly improved after their introduction—made more efficient, more convenient to use, less expensive, and hence available on truly mass scales.
The imagination and boldness of new proposals. So many of its inventors were eager to bring to life practical applications of devices and processes that seemed utterly impractical, even impossible, to so many of their contemporaries.
The epoch-making nature of these technical advances. Most of them are still with us not just as inconsequential survivors or marginal accoutrements from a bygone age but as the very foundation of modern civilizations. ↩︎
Many of these numbers are rough estimates. Figures adjusted for inflation after 1900 that weren’t already. Any figure before 1900 was adjusted via per capita GDP to more accurately reflect the scale of the undertaking.
If it were possible, the best metric to compare the scale of projects would be something like “Man-years + Value of Raw Materials (possibly in ounces of gold)“. This is especially true for projects like the Great Pyramid, the Suez Canal, the Great Wall of China, or the Manhattan Project which used mostly unpaid or low-paid labor.
Hard Drive — 3rd reading of the best bio of Bill Gates & Microsoft’s early years.
The Elements of Computing Systems — I never had formal CS education so this was a great practical explainer, from translating binary to assembly, to how an OS works.
A Mind at Play — always been a huge fan of Claude Shannon’s work, mind, and humility.
Turing’s Cathedral — a little long in places, but great overall history of computing & early people who shaped it.
Softwar — Reading now. Interesting insights about early Oracle, also gives me new appreciation for Ellison. [Update: I would not recommend this book. First part is good but last half rambles on, fawning over Ellison with random stories. “The Difference Between God and Larry Ellison” is much better.]
Let’s start with a test: Do you have any opinions that you would be reluctant to express in front of a group of your peers? If the answer is no, you might want to stop and think about that. If everything you believe is something you’re supposed to believe, could that possibly be a coincidence? Odds are it isn’t. Odds are you just think whatever you’re told. ― Paul Graham
If you’ll laugh about something one day, you may as well start now. ― Paul Graham
Be patient, calm, compassionate. Know that existence is fleeting.― Ettore Sottsass
A person’s success in life can usually be measured by the number of uncomfortable conversations he or she is willing to have. — Tim Ferriss
We start from the presumption that our people are talented and want to contribute. We accept that, without meaning to, our company is stifling that talent in myriad unseen ways. Finally, we try to identify those impediments and fix them. — Ed Catmull
Adventurous men enjoy shipwrecks, mutinies, earthquakes, conflagrations, and all kinds of unpleasant experiences. They say to themselves, for example, ‘So this is what an earthquake is like,’ and it gives them pleasure to have their knowledge of the world increased by this new item. — Bertrand Russell
Reality provides us with facts so romantic that imagination itself could add nothing to them. — Jules Verne
1) Don’t sell anything you wouldn’t buy yourself, 2) Don’t work for anyone you don’t respect, 3) Work only with people you enjoy. — Charlie Munger
The game of life is the game of everlasting learning. At least it is if you want to win. – Charlie Munger
Christian’s aim was not to offer discrete accounts of each period so much as to integrate them all into vertiginous conceptual narratives, sweeping through billions of years in the span of a single semester. . . . In the worldview of “Big History,” a discussion about the formation of stars cannot help including Einstein and the hydrogen bomb; a lesson on the rise of life will find its way to Jane Goodall and Dian Fossey.
“Most kids experience school as one damn course after another; there’s nothing to build connections between the courses that they take,” says Bob Bain.
“This course is a fundamental shift in how you deliver something. But there’s so many factors in American education that work against it.”
A new visual “grammar” will have to be discovered by filmmakers through trial and error (i.e. no fast cuts, super close-ups, etc.). Parts of the legacy film industry will rebel at first, as they have over the last 100 years since storytelling evolved from live performances to filmed, pre-recorded stories.
Just like audiences were frightened at the sight of a train barreling towards them in early theaters, there will be a learning curve for immersive experiences. Early players of demo games for the Oculus Rift have been scared to the point of ripping their headsets off. Dome cinemas could be the social alternative to VR headsets. (If you ever been on Disney’s Soarin’ Over California ride that’s an example.)
Technology-wise, I feel a complete 360 field-of-view (FOV) like this Jaunt setup won’t be the way to go. There has to be some direction to the audience’s attention. A complete FOV is too immersive and incompatible with users’ prior experiences. Maybe at some point down the road. Something like a 180-220 degree FOV + 180 up and down to allow some freedom of motion (immersion) but still directed view with surround sound.
There is lots of experimentation ahead in the near future in both technology and storytelling grammar. I look forward to both observing and participating.
Fitness Landscapes are used to visualize the relationship between genetic makeup (genotype) and evolutionary fitness (the ability to survive and reproduce). A fitness landscape is a vast landscape divided into a grid of billions of squares. Each square represents a genotype—some squares represent birds; some fish; some humans; with the majority being all the variations of genetic possibility that couldn’t survive in reality. Each square is very similar to its neighbors: two of the same species with a small variation, or two different but related species. The closer the squares, the more similar the genotype, and the further the squares, the more different. The fitness of each genotype is represented by its height on the landscape. Valleys represent low fitness, mountain peaks high fitness.
Over time, species tend to move up the landscape to the nearest peak (A), where all future paths of variation lead downward. The peak that a genotype “settles” on is most likely to be a local optimum, which is not necessarily the highest peak in the landscape (a global optimum). This is because selection pushes fitness towards nearby peaks (what is called a basis of attraction), but lacks the foresight to select the highest peak.
To get to a higher peak, a species may have to reduce its fitness in the near term (C) as it slowly traverses across a valley in order to improve fitness in the long term. In order to make this shift, there has to be sufficient instability or challenge; otherwise, an organism will not opt to leave the intermediate peak and suffer the unknown prospects of the valley. If the valley is too low or the higher peak too far away, it may be unreachable as the low fitness hurdle can’t be overcome. (An example is the lack of wheeled animals, which although beneficial is inaccessible due to the valley of low fitness genotypes around it.)
Evolution usually moves in small steps, but occasionally it takes wild leaps—a single mutation might give a creature an extra pair of legs or another radically different feature. Most of the time these leaps result in much lower fitness (B), and therefore don’t last. But other times it allows the genotype to jump to a higher peak without the slow process of going down before going up.
Every landscape has different terrain that can be on a scale from flat to rugged. A rugged or coarse landscape has many local peaks and deep valleys, while a flat landscape has only very small hills (all genotypes have about the same success rates).
Landscapes don’t remain static—they shift over time due to either environmental changes or adjustments as organisms move across it. The movement can vary from being stable (relatively flat and slow to change) to roiling (likely rugged and changing quickly). Given the likelihood of ever-shifting landscapes, the evolutionary mix of small steps and occasional wild leaps is the best possible way to adapt to the environment.