Five friends decide to have a contest – whoever builds their house at a higher altitude wins.
Andy builds his house in an ordinary town that has other qualities he likes, and ends up a few hundred feet above sea level.
Brandon finds a city that’s quite high up – say, La Paz, nearly twelve thousand feet above sea level, and builds a house there, still with access to the amenities of civilization, but without much choice around the culture or community he’s part of.
Charlie goes to Everest and builds a home at Base Camp, almost eighteen thousand feet above sea level. He’s near some other people, but lacking a lot of amenities, and always cold.
Daniel climbs up Everest until he finds the highest stable spot on which he can build a tiny one-room cottage. He has no space and needs to climb down the mountain if he needs anything, but is living twenty-three thousand feet above sea level.
Finally, Ethan climbs to the very peak of Everest, and does his best to lay a tent there. The wind constantly tries to blow the tent away despite his best efforts, so aside from barely being able to breath or endure the cold, he also cannot rest, as he must spend his (likely to be short) life constantly adjusting how he’s tying the tent down so it doesn’t blow away. But hey, his house is over twenty-nine thousand feet above sea level.
Who won the competition?
When you decide to optimize something, you first define it. You might choose something simple and measurable – like height. Or you might contrive some complicated mix of factors that attempts to approximate something nuanced like quality of life. But whatever you pick, once you plug it in and turn the optimization engine ‘on’, you’re now optimizing for that – and therefore, by definition, not optimizing for anything else.
This is a general problem. Most things in life are complicated; and it’s impossible in practice to define your optimization target well enough to capture everything, outside of the simplest scenarios.
No matter how much you want a house that’s high up, it turns out you also probably care about warmth, sleep, and breathable air.
No matter how much you want to earn more money, it turns out you also probably care about being physically well enough to appreciate it and having the time to spend it on things you enjoy.
No matter how much you want to believe in moral principles like “freedom” or “equal opportunity” from behind the veil of ignorance, it turns out you also probably care about preventing people from using that freedom to hurt you and yours, and you suppose you wouldn’t mind a bit of an edge if the competition is actually that fierce.
A true optimization engine doesn’t care. It just optimizes what it was told to optimize. You might initially get a house that’s at a pleasing altitude, or an exciting uptick in your bank account, or a feeling of moral satisfaction, but sooner or later the engine has you sleepless atop a mountain, or poisoning well water for a profit, or forced to tell a murderer where your friend is hiding instead of telling a lie to save them.
But, of course, human beings can always exercise judgement and Do Something Else Which Is Not That. Which is why it’s good we haven’t built any runaway optimization systems that materially impact out lives.
We haven’t done that, have we?
Oh dear.
I wonder which system you thought of just then.
Many of you would have assumed I was building up to talk about artificial intelligence.
Others might have assumed this essay was about the perils of capitalism or innovation-at-all-costs.
Those focused on technology might have thought of social media prioritizing clicks and outrage, or online platforms vacuuming up data for ever-more-precise predictive modeling.
The politically-minded of you might have made a connection with elections, especially in regions like the US with lots of complexity to exploit by a rogue optimizer (where “most votes” is decreasingly coupled to the originally intended “most actually representative of the will of the people.”)
The truth is “all of them, and more”, or perhaps “it doesn’t matter.” Our lives are shaped by a number of social technologies we deployed without understanding we were building an optimizer – and without building in brakes or levers to adjust that optimization as we pleased.
I want to be clear – that doesn’t mean all these forces are evil.
Capitalism has been an immense force for good. It’s eradicated diseases, achieved great beauty and discovery, and given us conveniences and experiences and opportunities we never dreamed of.
Social media has enabled moments of incredible unity, and fostered creativity and understanding in new ways.
US electoral processes might produce a lot of outrage and deadlocks and increasingly radical polarization today – but they resisted that decay for over two hundred years of beforehand, in which the US was a dominant world power able to do amazing things.
This is the challenge, the riddle, the paradox. Without these optimizers, we would all unequivocally be much worse off. Yet it seems inevitable that with these tools, we’re not so much buying a happier future as a temporarily happier future before the gap between “what we actually wanted” and “what we managed to tell the optimizer we wanted” starts to cause real pain, and decay, and ultimately leave us in a miserable state. That temporary period of bliss might last as long as centuries – or as short as years or even days – depending on various factors. But shy of defining everything that matters to us – a project as eternal as it is impossible – it’s a guarantee that eventually, the optimization will stop generating improvements, and start spending its energy on things we did not actually want.
So what do we do?
First, some definitions.
The thing the optimizer is pursuing? That’s the target.
The thing we actually wanted? That’s the vision.
The gap between them? That’s the chasm caused by using the target as a proxy for the much-too-complicated-to-actually-pin-down vision; so it’s called the proxy gap. And it’s important to understand that this is measured against actions taken, not just measured distance at any given point in time.
Think about two people carpooling from Boston, MA cross-country. The driver is headed to San Francisco, CA (the ‘target’) but the passenger is aiming to get to San Diego (the ‘vision’). For most of the drive, both people are happy, because the best move for the driver (drive west) is also the best move for the passenger. But as they get closer to their destinations, the move that best benefits the driver (stay towards the north) no longer best benefits the passenger (who needs to be south). Eventually, the passenger actively would prefer to get out of the car than to continue to carpool, because even though the carpool got them much closer than they started at, now every additional minute they stay in the car, they get further from their destination.
In this example, San Francisco and San Diego never changed their distance from each other – in that sense, one might argue, the target and vision were always unchanged, so the ‘proxy gap’ was always the same. But this is a misunderstanding. The real ‘proxy gap’ was in how the action that approached the target (San Francisco) did or did not actually approach the vision (San Diego). Early on, the actions that moved towards the target also approached the vision, so the proxy gap was small. Later, actions that benefited the target turned out to do less to reach the vision; and eventually became actively counter to the vision.
That period where the proxy gap is small; where capitalism is making everyone happier and richer; where US elections are generally functioning to elect effective and accurate representatives of the people; where AIs are helping us with menial tasks but not yet hacking their way out of their own testing? That’s the grace period.
The period after that, when the optimization is no longer helpful and possibly actively hurtful to the vision, the real intent of the people that actually matter? That’s the runaway phase.
Ideally, we’d have a way to turn on an optimization engine, run it through the grace period, identify when the grace period is ending, and shift to something else rather than continuing into the runaway phase.
But in practice, this is really hard. Not just because these systems aren’t managed by any single person. Let’s assume “should we pause and reset capitalism” was the kind of thing you could vote on. Even then, it would be hard, because different people will disagree on what the vision was in the first place, how big the proxy gap is, and whether we’re in the runaway phase.
Let’s take a real-world example. Right now, AI has been proven to be superhuman at finding software vulnerabilities. Some people argue this is terrifying, and means software is about to become much less secure. Others argue that it’s great, because even though we humans won’t be able to stop rogue AIs, we’ll just build defensive AIs which will stop the rogue ones.
Here, there’s a debate about what the exact vision is. If it’s “humans can all fully understand and control the security of their applications”, then the proxy gap is large – but it probably was already pretty large twenty years ago, if we’re being honest. On the flip side, if it’s simply “humans know that their important applications are secured”, then (so say the defensive-AI-advocates) the proxy gap is quite small.
The same debate occurs for capitalism (are ‘we’ ‘better off’ if CEOs continue to optimize for stock price above all else?), for voting structures (see the last ten years of US political discussions for basically one non-ending debate about which elected officials ‘actually’ ‘represent’ ‘real’ ‘voters’), and even for AI (is it good for AI to be replacing jobs? Is AI helping us with more access to information, or stifling our ability to think for ourselves? etc).
But this debate assumes that the only way in which an optimizer can ‘fail’ is to have a large proxy gap. And I think this is missing a much more important – and in some ways, more straightforward – debate we should be having.
So let’s talk about precariousness.
Let’s start by returning to our five friends and the altitudes of their homes. Ethan was the…let’s say “enterprising”…one at the very top of Everest. His problems weren’t simply the lack of oxygen or the cold up there. He also had to perpetually put in work to keep his ‘house’ intact, fighting an eternal battle against the winds to keep things balanced.
His house is obviously not a functional house – the proxy gap between “actual house that’s at a high altitude” and “this stupid tent atop a mountain” is quite large. But that doesn’t even capture how bad this situation is. Because even the currently-quite-bad situation is also precarious. One wrong move, and he ends up not even with a crappy freezing tent, but a tent-shaped parachute he’s desperately trying to MacGyver into something useful before he, and it, hit the ground in a rather final way.
This is the basic idea. Precariousness isn’t the same as a large proxy gap or being in the runaway phase. It’s simply a term for when only a narrow set of actions, which require a continued unsustainable investment, can preserve the status quo – and any other actions result in a precipitous decline.
Critically, discussing precariousness doesn’t require agreement on the vision, because you can assess precariousness even just against the target itself.
This might feel abstract, so let’s take some more concrete examples, starting with the cybersecurity example from above. There, we had a debate about what the real vision is, and whether humans need to fully understand what’s going on as part of it.
I say – the debate about the vision doesn’t matter, because regardless, this situation is precarious. It’s not just that few entities know how to protect a given system, so any single outage is increasingly hard to recover from. It’s that as the AIs become smarter, it becomes increasingly necessary for the defensive AI to be constantly training, constantly improving – and therefore its ‘handlers’ must be constantly pouring in more and move effort. Just as Ethan is barely able to rest because he’s constantly re-tying his tent down, here we have another situation where we’re only able to preserve our “high score” by pouring in more and more energy unsustainably – and where one wrong move can lead to devastation.
Fine, you might argue. That’s true, but only because security is relative. But capitalism produces things – like cancer cures – whose value is absolute. How would this concept apply there?
Well, capitalism’s target is “have more money.” We could debate how well or how closely that target hews to the vision – related to human prosperity, but as mentioned above, that’s interminable. Instead, imagine that we, as a society, had agreed “precarious things are bad even if they are still growing.” Then how would we assess whether capitalism is still working for us?
Well, precariousness is when maintaining one’s target-level requires continuous, near-perpetual effort, and a single slip-up can result in the target value going doing significantly.
So capitalism would be precarious if most people found that the amount of work they needed to put in to keep their wealth – usually, income – stable was increasing significantly, and that not putting in that effort or even taking a single one-time unfortunate blow – say, losing your job through no fault of your own – could be disastrous and immensely difficult to recover from, to the point that the odds would suggest the individual would keep falling rather than manage to recover.
Which might sound a bit familiar.
This isn’t meant as a knock-down argument. I’m not arguing “all optimizations are precarious” – that’s obviously untrue. Nor am I arguing that the specific examples I bring up are necessarily precarious, though I believe many of them are or are fast approaching it.
What I’m arguing is that most people implicitly agree that precariousness is bad, but I see an absence of this language in public discourse, and I think it would help us all to state it clearly. I’m arguing that debating “this system isn’t optimizing for the right vision of human flourishing, we should find a way to improve it” tends to entrench the people who benefit from the current system and therefore does not lead to change; but debating “is this system too precarious” aligns those who are currently benefiting (and want to preserve their benefits) and those who are currently losing out (and worry about how much worse things can get if it starts to collapse.)
The work of agreeing on a vision, and refining a target to better match that vision, is the work of years or decades or centuries, and careful debate between respectful contributors. A proxy gap that’s large is important to fix – but not always critical to fix urgently. Ideally, we’d have the time to do it right.
But when a system becomes precarious, that’s when there’s a real urgency to do something. So if we could all agree that, at minimum, if a system is precarious, we should all urgently come together to fix it, then that would at least buy time for the larger debates around what exactly to optimize to take place.
In other words, optimizing for figuring out exactly what the vision and target should really be, is itself a habit that leaves us in a precarious position. We don’t need to fix everything immediately. We just need to shore up our foundations and ensure that what we’ve built will last until we’re ready to build something else.
Leave a Reply