Exported: Blind Goaltenders: Unproductive Disagreements

(Exported: Copying posts out from Lesserwrong, since I have totally lost confidence in it.)

If you’re worried about an oncoming problem and discussing it with others to plan, your ideal interlocutor, generally, is someone who agrees with you about the danger. More often, though, you’ll be discussing it with people who disagree, at least in part.

The question that inspired this post was “Why are some forms of disagreement so much more frustrating than others?” Why do some disagreements feel like talking to a brick wall, while others are far more productive?

My answer is that some interlocutors are ‘blind goaltenders’. They not only disagree about the importance of your problem, they don’t seem to understand what it is you’re worried about. For example, take AI Safety. I believe that it’s a serious problem, most likely the Most Important Problem, and likely to be catastrophic. I can argue about it with someone who’s read a fair chunk of LessWrong or Bostrom, and they may disagree, but they will understand. Their disagreement will probably have gears. This argument may not be productive, but it won’t be frustrating.

Or I could talk to someone who doesn’t understand the complexity of value thesis or orthogonality thesis. Their position may have plenty of nuances, but they are missing a key concept about our disagreement. This argument may be just as civil – or, given my friends in the rationalsphere, more civil – but it will be much more frustrating, because they are a blind goaltender with respect to AI safety. If I’m trying to convince them, for example, not to support an effort to create an AI via a massive RL model trained on a whole datacenter, they may take into account specific criticisms, but will not be blocking the thing I care about. They can’t see the problem I’m worried about, and so they’ll be about as effective in forestalling it as a blind goalie.

Things this does not mean

  • Blind goaltenders are not always wrong. Lifelong atheists are often blind goaltenders with respect to questions of sin, faith, or other religiously-motivated behavior.
  • Blind goaltenders are not impossible to educate. Most people who understand your pet issue now were blind about it in the past, including you.
  • Blind goaltenders are not stupid. Much of the problem in AI safety is that there are a great deal of smart people working in ML who are nonetheless blind goaltenders.
  • Goaltenders who cease to be blind will not always agree with you.

Things this does mean

Part of why AI safety is such a messy fight is that, given the massive impact if the premises are true, it’s rare to understand the premises, see all the metaphorical soccer balls flying at you, and still disagree. Or at least, that’s how it seems from the perspective of someone who believes that AI safety is critical. (Certainly most people who disagree are missing critical premises.) This makes it very tempting to characterize people who are well-informed but disagree, such as non-AI EAs, as being blind to some aspect. (Tangentially, a shout-out to Paul Christiano, who I have strong disagreements with in this area but who definitely sees the problems.)

This idea can reconcile two contrasting narratives of the LessWrong community. The first is that it’s founded on one guy’s ideas and everyone believes his weird ideas. The second is that anyone you ask has a long list of their points of disagreement with Eliezer. I would replace them with the idea that LessWrong established a community which understood and could see some core premises; that AI is hard, that the world is mad, that nihil supernum. People in our community disagree, or draw different conclusions, but they understand enough of the implications of those premises to share a foundation.

This relates strongly to the intellectual turing test, and its differences with steelmanning. Someone who can pass the ITT for your position has demonstrated that they understand your position and why you hold it, and therefore are not blind to your premises. Someone who is a blind goaltender can do their best to steelman you, even with honest intentions, but they will not succeed at interpreting you charitably. The ITT is both a diagnostic for blindness and an attempt to cure it; steelmanning is merely a more lossy diagnostic.

Advertisements

Exported: Personal Model of Social Energy

(Exported: Copying posts out from Lesserwrong, since I have totally lost confidence in it.)

Epistemic Status: This is a model I have derived from my own experience, with a fair amount of very noisy data to back it up. It may not generalize to anyone else. However, it seems like a framework that might be useful, so I’m sharing it here.

The excessively simple model of social energy is the introvert/extravert dichotomy. Introverts lose energy from social situations, extraverts gain energy. This is then elaborated into the I/E scale, where the sign of your energy change in social situations is mapped to an integer. This is clearly more descriptive of reality, but as many have pointed out, still imperfect.

I find that for me there are separate sets of factors that determine energy gain and energy loss.

For energy gain, it is a positive-slope, negative-curvature function of the number of people present. There is energy in a room, and I do pick up some of it. (Something like sqrt(n), or possibly 10-10/n

For energy loss, it is a function of how much I trust the person in the room I trust least; f(min(trust(p) for p in room)). This grows much faster than the number of people present. Trust also seems to be a function of my pre-existing mood (that part I expect won’t generalize).

Naively, I would have expected this to be a weighted average of my trust of people in the room, where five people I trust very much and one I trust very little would feel very different from five I trust somewhat and one I trust very little. I have difficulty arranging that test, but preliminary data suggests that expectation was wrong; one person who I cannot relax around spoils things thoroughly. (‘Trust’, here, is very much a System 1 thing; feeling safe/open around someone, rather than feeling/thinking that they are trustworthy/upstanding/honest.)

The predictions made by this model are that you should choose your social gatherings carefully, even if extroverted, as the benefits of size can be wiped out by one or two poorly-chosen guests.

More broadly, I think that considering gain and loss separately will clarify the feelings toward socializing of many self-identified introverts. Since it seems quite plausible that ‘true introverts’ who never get energy from social interaction are not actually a thing, I expect this would help improve the day to day lives of many.

Exported: UnTAPed Learning:

(Exported: Copying posts out from Lesserwrong, since I have totally lost confidence in it.)

(For this one, the content worth preserving is mainly the comment thread, so I have copied that as well.)

[Epistemic Status: Request for Proposal]

For anyone who is familiar with CFAR’s teaching program, the concept of the TAP will be very familiar. It’s an acronym that expands either to Trigger-Action Pattern (for naturally-occurring instances) or Trigger-Action Plan (for purposefully-implemented instances); the latter is known academically as “implementation intentions”. While there is room for contest for the title of “Most Important CFAR Skill” in terms of which one has the most impact of regular use, in terms of driving adoption of techniques, TAPs are the foundation of the entire curriculum. From simple skills to complex ones, the default flow of practicing a skill is “install a TAP to practice this”, and the default call to action of most CFAR classes is “spend five minutes brainstorming TAPs to install”.

They’re extremely useful…if they work at all.

For me, they don’t. I struggled to find any example at all of a trigger-action patterns in my own life, and even following the extreme repetition, hard-mode path to installing a TAP, found that it vanished as soon as I stopped consciously thinking about keeping it up.

I liked most of the techniques I learned at my workshop. Many seemed quite useful to apply. I remember approximately none of them, because, deprived of the core tool to maintain them, every piece of practice looked like

It’s a strong foundation, for most people. But it’s still a foundation; if it doesn’t hold, nothing else is any use. So:

What other things have people devised to turn a technique from a sketch of ideas to something used often enough that it’s a proper skill?

As a starting point, things that I use in my particular case:

Anki.

Beeminder.

Calendar reminders.

These are all reasonably good at their particular, time-based niches. But none of them work well for irregularly-occurring opportunities or for fuzzier, less specific types of practice.

An alternate question, which seems to me to be similar in applicability, is:
How would you build a daily or weekly routine 
from scratch, with literally no existing example to piggyback on?

Continue reading

Exported: On Inconvenient Truth

(Exported: Copying posts out from Lesserwrong, since I have totally lost confidence in it.)

What are your politics?

What frameworks have you acquired for structuring your interaction with the world?

What facts support them?

What possibilities would undermine them? What significant counterexamples could exist, and which of them could prove to be fact?


Ludwig Wittgenstein said:

If there were a verb meaning “to believe falsely,” it would not have any significant first person, present indicative.

I assert that this is not as true as it seems. Yes, few if any people are willing or able to admit that they hold any specific false belief. If you don’t count beliefs in belief or aliefs, it’s speculative whether it’s logically possible. But I, for one, do believe falsely. I falsely believe something. I am quite sure of it.


The world is wide, and beliefs are complex. Systems of beliefs, like politics, ideology, frameworks, even more so. A complex set of beliefs has many premises it rests on; perhaps none is individually a crux, but propositions that, if false, would cast the rest into doubt in a serious way.

And because there are so many, it is extremely likely, near-certain, that at least one of them is wrong. For almost any political position, there is at least one inconvenient fact.

You may not know it. Perhaps it turns out that, contrary to your ideals of rational actors and self-determination, people born with strawberry-blonde hair are inherently dangerously biased toward pyromania and risk-seeking behavior that they do not endorse, and no one has proposed this, let alone investigated it. Maybe there is a curiously-specific unified field theory that proves that blue is the best color.

But somewhere, the fact is out there. The world is not politically convenient; whatever you wish to believe, there is, somewhere, a fact that will cast doubt on it.

So what should you do?

You could take a strong stance of epistemic and moral modesty, and never take a position with confidence.

You could reject it and embrace views you know are probabilistically ill-founded.

You could try to bite bullets and believe the inconvenient facts.

You could try to find the facts and change your politics to fit.

As a rationalist, I feel committed to the last. I have had some success, but I don’t truly believe there are no remaining inconvenient facts. If I’m lucky, it is something that suits my less-endorsed instincts, like an elitist “pure democracy and sane government are fundamentally incompatible” that pushes in the direction of something possibly palatable like oligarchy. If I’m less lucky, it’s something like “the CEV of humanity includes only very small terms for intellectual exploration”.

But in any case, I think it would benefit everyone, in political and cultural arguments, to remember that somewhere, the Inconvenient Fact exists. For every position modern humans hold, there is some fact that calls it into question. This is true for your opponents and also for you.

The Magic Great Designer Search 3

The Great Designer Search is a once-a-decade or so event run by Wizards of the Coast, makers of Magic: the Gathering, to recruit new designers from the general public. It started because Mark “Maro” Rosewater, head designer of Magic, was told he could hire an intern and could use whatever process he wanted. He said “Can I do a game show?”, and then did. It’s now on its third iteration, which I attempted but did not make the cut in. The basic structure for the preliminaries is

First, a set of 10 essay questions, 250-350 words per answer. This is first to limit the number of entrants, since it’s the most labor-intensive and aversive piece. 3000 people, about half of those who expressed costless interest, submitted this.

Second, everyone who submitted valid entries for the essay takes a multiple choice test. The first test had 35 question, the second 50, and the most recent one 75. The passing score is picked to have about 100 entrants advance (It was 73; I got 70.)

Third, a design challenge, which has had the details change substantially over the three iterations. This one is not public knowledge yet (though will be sometime today). 8 people advance from this, and are the contestants on the ‘game show’ portion.

I’m fairly proud of my essay responses, so I’m reproducing them here, excepting only the first question, which was a personal statement.

2. An evergreen mechanic is a keyword mechanic that shows up in (almost) every set. If you had to make an existing keyword mechanic evergreen, which one would you choose and why?

Priority consideration goes to conceptually simple abilities without unintuitive gotchas in their implementation, to avoid large mindspace requirements. Most evergreens are combat-based or suitable for spells; that should continue. It would be aesthetically off to have exactly one evergreen keyword with a built-in cost, which adds a bias away from those. Evergreening scry violated a similar aesthetic rule, though, so that obviously shouldn’t be decisive. (While aesthetics are not incredibly important in themselves, consistent aesthetics aid in internalizing rules like resonance, and so should be maintained if practical.)

The core problem here is that most potentially-evergreen keywords overlap heavily with existing ones. For example, Exalted is resonant and simple, but works best in the colors that least need more evergreens. Shadow has strictly worse gameplay than flying in most ways. Detain duplicates flickering for no great benefit. For combat mechanics, the best that come to mind are Flanking and Bushido. They’re simple and fairly versatile, but both have flaws and neither stands far apart from existing keywords. Something like Prowl is tempting for a spell evergreen that encourages combat, but the actual implementation would have to change so much that it would hardly be the same ability.

Ultimately, my choice would be Exert. As “super-tapping”, it is nearly as versatile as tapping itself. It is not inherently combat-focused, but as Amonkhet showed goes well with attacking. It would enable a wider middle ground between repeatable abilities and one-time sacrifice abilities. It does have its problems; memory issues are potentially a factor, possibly more so when it is more scarce. But it is simple and versatile, usable across all permanent types and for many effects, so it would be a good choice for frequent reuse.

3. If you had to remove evergreen status from a keyword mechanic that is currently evergreen, which one would you remove and why?

My answer is a bit of a cheat, but: defender. Removing any of the evergreen creature keywords entirely, no longer printing anything functionally equivalent, would be a fairly significant handicap. It’s a pretty parsimonious set, and while some the game could live without (Trample could be sidelined in favor of green menace), design would feel the pinch. Defender, however, could be removed without much impact on design space.

Examine the types of cards with defender in Standard today. They come in two kinds: Creatures with zero power (ex. Guardians of Meletis), and creatures which have defender with a conditional way to circumvent it (ex. Hightide Hermit). For the former, they would play the same without defender in >90% of cases. Yes, they could represent a combat trick or trigger Raid, but on their own they would sit and block anyway. For the latter, most could easily be reworded to reverse the restriction. (ex. “Hightide Hermit can’t attack unless you pay EE,” or, for an older example, “Colossus of Akros can’t attack unless it’s monstrous.”)

Going somewhat further back, we do see cards like Archers of Qarsi or Clinging Anemones which would be difficult to replicate without defender. But these are rare, showing up only once or twice per block, on average. Replacing them with “CARDNAME can’t attack.” would take up a little more space in the textbox and a little more mindspace in player’s heads, but it would be a pretty cheap cost.

It’s actually a cheap enough cost that it’s a little strange it hasn’t happened already. Conceptual space in player’s heads is valuable, and as a vocabulary word defender is not earning its keep. While it’s almost as intuitive a term as flying, it is not effortless, and while I’m not sure it should be gone, I think it should be fighting for its life.

4. You’re going to teach Magic to a stranger. What’s your strategy to have the best possible outcome?

One part of teaching is easy; I would grab a set of welcome decks. One of each color, because talking about the philosophies and styles of the colors is a good way to get someone invested, especially since it gives them a real choice early on. I’ll flip through the contents of the decks beforehand to see what effects they include, and particularly the flashy bombs like Shivan Dragon. For example, it would just be confusing to explain counterspells to a new blue player if they aren’t in the deck at all.

Different people like learning different ways, so it might or might not be a good idea to jump straight into playing. My personal preference is to have a good handle on most of the rules before I start playing a game, so I’d start there, but be willing to jump ahead to turn one if my student was impatient.

I’d start by grabbing another deck that plays differently. Then I’d deal a hand off the top, face-up, and explain what they’d see. Start with the mana cost, and then point out lands as how you pay them. I’d make sure to walk through the full card of at least a couple creatures and a spell, including something that targets. Explain the basics of attacking and blocking, and that you bring someone from 20 life to zero through attacking, and we’d have enough to shuffle and start.

Ideally the first game would be _two_ new players, not just one, with me helping both of them along. If that’s not possible, I’d suggest we play our first game open-handed, so that I can walk through why I’m doing things, and what options they have. In either case it’s possible to turn “giving advice” into “making their choices for them”, ruining the fun, so I wouldn’t want to dictate that. Most other game concepts are straightforward to mention as they come up in game, so as long as I could answer questions as we played it would be a good start.

5. What is Magic’s greatest strength and why?

Magic’s greatest strength is its depth. The largest piece of that is its ability to be many games in one, but that’s not enough by itself (though it does make Magic breed game designers like rabbits). It also is the reason why it can sustain the “crispy hash brown” feeling indefinitely, and why its rules can handle any situation that players (or even, usually, designers) throw at it.

The depth starts with the foundation in the mana system, color wheel, and the TCG itself. But that isn’t enough by itself; science marches on, so younger games have surpassed it, though usually not in large ways. The solidification of the rules and wording, in 6th Edition and later, built a strong frame that made experimenting with strange effects possible without the game becoming incomprehensible for players. But the real key is two decades of full-time design work.

It’s not just that this makes the subtleties and crannies of Magic better-understood than any other game. It also leaves a trail of many different views of how the game ought to be, set by set and block by block. If Magic production stopped today, there would still be crispy hash browns available, because going through its history uncovers many different ways the game can be.

Not all players will be interested in exploring this depth. If you stick to Standard, you will have a good experience, but only age’s benefits for design quality will show. But the unmatchable strength of Magic is that for anyone who wants to go looking for something new and different, there is an archive of progress and visions emphasizing every piece they might find fascinating. You can make your own game within it, but it’s deep enough that often past designers already did.

6. What is Magic’s greatest weakness and why?

Magic’s greatest weakness is, of course, its complexity. The flipside of the long history that makes it unusually robust and deep is that to fully understand it is a bigger undertaking than any other game, excepting some of the larger simulationist wargames. If not compensated for, it’s likely to discourage most people introduced to Magic from ever really picking up the game, and without some adaptations Magic made to formats, could easily lose many active players over time.

People choose to play strategy games because they want to think during their fun, but that doesn’t mean that they want to think as hard as they would during a college exam. Complexity is a problem because, unchecked, it can easily make trying to win a game at least that difficult. It’s a hard problem because the core nature of Magic as the game that breaks its own rules, and its two-decade history of experimenting, mean that it’s too complicated to function simply.

This problem became obvious long ago, so there are a lot of standard tools the game has to deal with it. One is Standard (also Limited, Block, and Modern); formats where the card pool is smaller and the history is shorter. There are fewer kinds of interactions to learn about and so the game has less necessary complexity. Another is temporarily hiding the complexity. As a player gets used to the game, things that initially felt extremely complex become second nature, and so if the complexities of cards and interactions become visible slowly the players can handle more of it in total. I’d say more, but you asked for whys, not hows.

The important point is that, as long-time players of Magic and other games, designers and veterans are almost always going to underrate how out of their depth a comparatively new player will feel if confronted with a large chunk of Magic’s possibilities out of the gate. Complexity is Magic’s greatest weakness in part because it’s the one that the people who play it are worst equipped to fight or fix.

7. What Magic mechanic most deserves a second chance (aka which had the worst first introduction compared to its potential)?

The mechanic I think most deserves a second chance is Prowl. I’ll add the caveat that I know Prowl the keyword is probably not coming back; it needs Tribal to go on noncreatures, and is pretty narrow without that. However, I think the basic structure of the mechanic is very solid, and it deserved better than it got. Strip it of the tribal trappings and have it trigger on any creature (or any creature of a specific characteristic, depending), and it could be a much bigger player; probably not the marquee mechanic of a set like energy, but a second focused mechanic or a solid faction mechanic.

Prowl has many of the virtues that make Raid a versatile mechanic. Encouraging attacking: check. Goes on many effects: check. Works for creatures and spells: check. It also shares the drawback of not working on effects that help you get attackers, and somewhat more severely since it doesn’t synergize with effects that help an attacked get through for damage. But any non-instant spell with Raid could work with Prowl.

Having space for both cost reduction and bonus effects is usually a recipe for players finding something very attractive. They didn’t, for prowl. I would attribute that to a couple factors. First, for competitive players, it coexisted with Fairies, which was in its colors and was far more rewarding. When the best enabler you have available is the marquee card in an oppressively powerful deck, you will be outshined. Second, prowl didn’t fit well with its set. The tribal component felt tacked-on, and the dependence on creatures kept it from having much contrasting appeal for those who didn’t care for tribal. Third, Morningtide was incredibly crowded. It got only nine cards, not even as much as a limited-scope faction mechanic like cipher.

Strip it of that baggage and give it more space to shine, and I think it could be a very popular, successful mechanic.

8. Of all the Magic expansions that you’ve played with, pick your favorite and then explain the biggest problem with it.

My favorite set is Planar Chaos, and the problems with it are obvious. It is the set with the highest barrier to entry, in the block with the highest barrier to entry, of anything in Magic. To appreciate Time Spiral Block, you needed a high tolerance for complexity, familiarity with old mechanics and their subtleties, and, for full appreciation, a curator’s approach to all of Magic’s history, down to the individual cards. This was wonderful for the people who had it — it’s my favorite block, and that’s shared by most of my friends —but terrible for the bulk of the players who were left out.

Planar Chaos adds to that by requiring knowledge of and fascination with the color pie. Even someone who has followed the Magic story since before Weatherlight and digs through Gatherer searches of old cards just for fun, won’t appreciate Healing Leaves if they don’t care about the color pie. And while much of the set is great just for feeling slightly off but still Magic, it is even more confusing if you don’t care enough about the color pie to understand the difference between the mechanical expression present in normal Magic and the Platonic philosophical version; otherwise the different mechanical instantiation will feel random and disconnected. So the aspects that made me love it are also the direct cause of its crippling flaws.

It’s no coincidence or surprise that it divided the broad player base from the competitive scene; tournament players are not just Spikes but heavily-invested Spikes, and that investment gives the most reason and ability to get the context needed. We see something similar with Masters and Conspiracy sets; strong appeal to a narrow audience that can get past the barrier to entry, with the product scoped to them in advance. That gives me some hope that there could be a supplemental set like Planar Chaos in the future, but I know that even for a Masters set it would be inaccessible.

9. Of all the Magic expansions that you’ve played with, pick your least favorite and then explain the best part about it.

My least favorite set is Innistrad. (My objections, in brief, are that it was designed too much for the present and too little for the future, and that it was excessively one-note.) It has many good qualities. I never played Limited at the time, so while I hear that its draft format was one of the best ever made, I can’t really comment on why. What I can talk about is its other very strong quality, which is its embedding of flavor throughout the set.

Innistrad has a lot of really vivid designs. More than any set except maybe Unglued, the cardfile is full of the environment it is portraying and it shows in every pack. Graveyard themes, particularly morbid, was excellent at making everything all about death and the threat and consequences of death. Transform sold werewolves very well, and did some neat one-off effects to add a little flesh to the set. The creative treatment was very good at portraying a world where everyone was desperate and under siege (The symbology of the Church of Avacyn was particularly good at conveying something with Catholicism’s role in gothic horror without treading on its toes too closely for player’s comfort.) Almost anything you can point to in Innistrad is very clearly some horror element or other. That has set an example for all future sets, defining how much it is possible to suffuse a design with a particular theme and aesthetic. And while I have mixed feelings about following that lead, I can’t deny that it’s an achievement that’s responsible for a lot of Innistrad’s fans.

10. You have the ability to change any one thing about Magic. What do you change and why?

“Make one change to Magic.” Well, what’s my goal? To make it more fun for me specifically? Then amp the complexity back up. But no, I should do the best for the game. However, that means nothing big should be changed. Magic is doing well, and that’s not a time for drastic change. I’d do something with a contained potential impact that seemed positive in expectation. As I mentioned in an earlier question, unkeywording Defender seems about right.

But I wouldn’t even do that, right now. I don’t like refusing the premise of the question, but the best action in expectation is the status quo. To make a better decision, it is necessary to have better priorities, better information, or better reasoning ability.  Presumably I have a different perspective about how Magic ought to be, but any differences in goals are subtle or nonexistent. I certainly have less information than R&D. And R&D’s reasoning is many more people spending significantly more time on the question. So I have the same goals, less information, and worse reasoning. It would be surprising if I could find something better.

That being said: If I had to propose an impactful change instead of a harmless one, it would be to impose my ideas about set concepting. It is my considered opinion that every set should have at least two significant themes, one mechanical and one flavorful. There should also be a significant back-and-forth, where the mechanics inform the flavor, which push back and change the mechanical emphasis, and vice versa. A process like this would enforce tighter connection of flavor and mechanics, and create richer, more lifelike settings. Ravnica and Zendikar, two of Magic’s best worlds, were created like this, and I think it would be a good policy.

So if I was going to argue for a change on my first day in the Pit, that would be my position. But as it is, I don’t know enough, so my preferred change is “no-op.”

Reflection

Since I formed the plan to post these, Maro has posted an article about the questions and his answers. For several, I like my answers better. Removing defender is better than removing prowess, cycling is fairly redundant with scry and exert is not, and meld seems like a weak choice of mechanic to return. I also think that he’s wrong about Magic’s strength and weakness. The final question, on a change to magic, I think his answer is better, though I still like both my out-there proposal and “no-op”.

Off the Point and Over the Fence: Thomas Schelling

[This is published unfinished and nearly a year after reading the book, prompted by another reader’s post on the same topic.]

Ha! I’m back. And I’ve read another book. Thomas Schelling’s The Strategy of Conflict. In this book, he produced a rigorous grounding for the Cold War Balance of Terror, described how weakness in bargaining is often strength, and along the way introduced both the Schelling Point and the Schelling Fence (though, not having an ego the size of John Von Neumann, he did not call them by those names).

Setting the stage for the book: Thomas Schelling observed that game theory existed (thanks, Von Neumann) and that Mutual Assured Destruction existed (again, thanks John) but that they didn’t really seem to be strongly tied from a theoretical standpoint; the theory underlying them took as an axiom that war is inherently zero-sum, which he considered be ill-founded.
He decided to construct an extension of game theory that dealt with variable-sum games and implicit communication; what rules arise in games and negotiations when the participants – by fiat or practicalities – can’t communicate, and how this affects optimal strategies. Since nations at war find it understandably difficult to communicate reliably and believably, this seemed potentially important, and due to the streetlight effect, having a theory for negative-sum, variable-sum games was important to making the situation stable.

us_presidential_hotline-300x218

He was also a big influence on Kennedy during the Berlin Airlift and Cuban Missile Crisis, and was influential in getting the hotline created.

While it’s still historically interesting, it’s lost some relevance since we now care almost as much about loose nukes being used by nonstate actors as about states launching nuclear wars. The theory has broader applications, though, and the underlying principles are fascinating.


The basic idea that underlies the whole book is that, in rationalist terms, the shared map can be as important as the territory or even more so. Because human minds tend to settle on the same things as the “most natural” option, and this is common knowledge, communication can happen implicitly. And this is more and more likely the more culture is shared between participants; an alien or de novo AI might share very little, but two men from the same town in Iowa will share quite a lot. The simplest way to express this is a Schelling Point in cooperative games. For example, imagine you and another paratrooper drop and land at points X and Y on this map:

schellingmap

You don’t know where each other landed and cannot communicate by radio. You do have copies of the map. Where do you meet?


While you think a bit, consider another example. If you’re going to meet someone in Times Square, New York City tomorrow, but you can’t communicate and forgot to pick a time, when do you meet? You could pick dawn, dusk, noon, midnight, or many other times. Humans will generally pick noon, though; we’re diurnal and it’s a clear single point.

That’s a fact about humans, though, not all minds. Intelligence-uplifted cats wouldn’t pick noon; they’re naturally crepuscular, awake at dawn and dusk, so noon is inconvenient and not a natural time to pick. Their coordination points are weaker, and not quite the same as we are.

But back to the map question.


If you said “the bridge”, you understand the principle. MOREMOREMOREMORE


Schelling was not principally concerned with these, though. They were important, but only as a stepping stone to his thoughts about bargaining, and how a coordination point becomes significant as a bargaining position. Consider a coordination game where two people need to pick a dollar value between $2419.37 and $2693.01. If they pick the same number they each get that much money. Naturally, they will settle on $2500.00.

For the exact same reason, if two people are haggling over a nice TV and the seller says “I can’t possibly accept anything less than$2693.01”, the buyer will think That’s totally fake. And the same if he “can’t possibly accept” less than $2419.37. But if he says he can’t sell for under $2500, that is believable in a way the other two aren’t. If he drops to $2499.95, the buyer will smell blood and push for lower, until it falls to a more natural place like $2450 or $2400. The seller knows this, and knows that the buyer knows it, and knows that the buyer knows that he knows it, and so on, and vice versa. Therefore, even if the real value that is the true lowest price with positive profit is $2419.37 and highest price the seller would pay is$2693.01, the actual result is most likely $2500 or no sale. (This concept was later rederived and labeled the Schelling Fence by Yvain/Scott Alexander.)

This has profound implications for stability of negotiations, arguments, and wars, and puts an exclamation point on the truly alien nature negotiating with a very-slightly-different type of mind would have. For example, national borders are generally stable because they are a fence; pushing a few miles into national territory and then stopping is implausible, so a border will be defended as though it failing indicates the beginning of a long, deep-striking offensive. A pair of militaries lining up on the map above would draw their battle lines at the river if they both wanted to avoid an immediate fight. The Dow Jones crossing 20,000 could be very important if the optimistic artificial milestone influences behavior. The political status quo, even if arbitrary, can be stable even if no faction is satisfied with it. And so on.

p. 163-172: experiments

somewhere: agreements depend on the shared map, not just the territory

p. 71: The Schelling Fence

p. 109: the eye-tracking machine game

p. 55, p.60-63: Schelling Games


This is the central insight, but not the only one. The largest other discussion is about commitment mechanisms and how they affect bargaining. The critical piece here is that weakness – decisions you can’t make, control given up, outcomes which are worse for you than they could be – can create a much stronger bargaining position. This is a critical piece of the study of commitments; unilaterally making your position worse or your opponent’s position stronger can get you better outcomes. Commitments also can very quickly render even simple games computationally intractable to analyze in full explicit form.

MORE HERE

p. 153: commitment mechanisms and the explosion of options


The last piece I found interesting is a collection of insights about the nature of brinksmanship, surprise attacks, and understanding the balance of terror in terms of much more primitive negotiation tactics.

Threatening to pull the other side off a cliff only works if you can slip. You lose too, so threatening to jump isn’t credible.

p. 199-200: theory of brinksmanship

Model the conflict as a chance of unwise attack plus the ability to attack strategically with some success probability. Intuition suggests that you should update your chance of strategic attack based on the likelihood of enemy attack; the more likely they are to launch a first strike, the more likely you should be to preempt them. This would generate a feedback loop; you believe them to be more likely, so you become more likely to attack, which they know, thus they become more likely to attack, so you must as well, and it spirals to doom. This could be even faster if your chance of unwise attack depends on how threatened you are and so it rises as well.

Instead, even under the strong assumptions, unless the baseline random chance of attack is sufficient to incite a voluntary first strike right off the bat (a calculation that depends on how likely you are to succeed at an attempted one-sided war and exactly how good and bad the continued peace and MAD options look), there is no incentive to strategically start a war. Obeying the principle of ultrafinite recursion, the infinite regress vanishes and is replaced by a simple all or nothing decision.

p. 209-216: surprise attacks, anti-Petrov errors, and ultrafinite recursion

Bomb back to the stone age / Stone Age (well, medieval) tactic of exchanging hostages for good behavior.

p. 239: the balance of terror as hostage-taking

 

Moral Precepts and Suicide Pacts

You may recently have seen this blog post floating around, brought to prominence by some neo-Nazis getting more open than recently normal. Tolerance is not a Moral Precept, which says:

Tolerance is not a moral absolute; it is a peace treaty. Tolerance is a social norm because it allows different people to live side-by-side without being at each other’s throats. It means that we accept that people may be different from us, in their customs, in their behavior, in their dress, in their sex lives, and that if this doesn’t directly affect our lives, it is none of our business.

And this far, I basically agree. This is why tolerance is important, and why it is necessary. But then the author goes on to say: “It is an agreement to live in peace, not an agreement to be peaceful no matter the conduct of others. A peace treaty is not a suicide pact.” And here I strongly disagree. For example, we have no peace treaty with the state of North Korea. This does not mean that we are legally or morally licensed to resume hostilities from the fifty-year ceasefire whenever we like. We could have, many times, destroyed the country; it would have been advantageous and simple in the past, though since they’ve recently acquired a credible nuclear threat to parts of the mainland US, it might not be now.

If we consider this in the frame given here, that a peace treaty is only a promise to be peaceful to those who are peaceful unto you, then we have committed a wrong in not having long since crippled the Kim dynasty and its military. It would be better for us, for our allies, for world stability, and for everyone except the at-the-time citizens of North Korea. But we did not, and that was correct.

Peace treaties are good, but they are not enough. They are one means of credibly committing to live together in peace, but, like Bismarck’s Prussia as it grew into Germany, the mere existence of a peace treaty is not a great reassurance if you cannot otherwise trust the other signatories. A person or country who is champing at the bit for war, but under a treaty to stay at peace, is less trusted to remain peaceful than a country with no treaties but also no prominent militarist factions. To be trusted to remain peaceful, you must be the kind of person who remains peaceful.

And to be a peaceful person and earn the trust placed in you, you must be peaceful even when you have every right to fight. If you aren’t, and take every opportunity to pick a fight when a half-decent excuse is available, your peace treaty is almost worthless, since anyone can see that you will break the spirit as soon as you can find a loophole in the letter.

It’s the same with tolerance. If you want to shut up your argumentative opponents and vigorously retaliate when your opponents show signs of intolerance, you will not be trusted to be tolerant to others who are tolerant, even those who basically agree with you. If you instead treat intolerance tolerantly, other tolerant people will know you are a trustworthy ally to influence your community to remain tolerant and only retaliate in any way when collective action and collective decisions deem it appropriate.

It is what you do in the hardest, most tempting cases – dealing with explicit Nazi rhetoric, eyeing the underdefended soft underbelly of France with your newly-unifying Germany – that draw the contours of your policy of trustworthy peace. Had newly-formed Germany not been so rapacious when it had the opportunity, England might not have felt the need to protect Belgium against the advance of the German army through it on to France, and the First World War could have gone much better for the Central Powers. Similarly, how tolerant you are of hateful fringe religious groups – i.e. the Westboro Baptist Church – determines how well you can be trusted to be tolerant of minority religions which aren’t outright intolerant but have strongly-held convictions you find enraging. That’s critical if you want religious help with a cause you share an interest in.


Another, similar concept is “The Constitution is not a suicide pact.” Starting from Thomas Jefferson and passing through Lincoln and a number of Supreme Court justices, this is the idea that the protections of the Constitution, while strong, can be suspended in extremis, replaced with undeclared martial law and states of emergency when a threat is great enough. As you might guess, I don’t like this idea. I will give its advocates this: the law is not a suicide pact.
Many laws on the books are important to stick to normally, but if following the letter would lead to opening yourself up to attack, then yes, it is legitimate to suspend the law until the danger has passed or the law has been amended to be safer. To give an example: many states have an “Open Meeting Law”, which says that any group of three or more elected officials must announce the time and place of their meeting in advance, and publish all proceedings afterward. If the Weather Underground has been attacking government officials in your area, it would be dangerous to comply with the law for meetings where you discuss how to deal with the problem and formulate a plan to catch them. Since the alternative is for the government to grind to a halt, unable to do business due to deadly threats, it is clearly the best thing to do. Circumstances have demanded the law bend, and it is correct for us to make them do so.

But that doesn’t apply to the Constitution. Unlike the informal British constitution, ours is very explicit. It sets the “guardrails of democracy”. It is, explicitly, the places where the law cannot and must not bend to fit circumstance. Someone may be publicly advocating sabotage-by-inaction to make it more difficult to prosecute an active war, which does pose a danger, but their right to free speech protects them anyway. People may meet to discuss whether the police are their enemies (perhaps for racial reasons) and what to do to resist law enforcement if so: this damages the rule of law and is obviously a conspiracy to make illegality easier, but the right to free assembly protects them anyway.

This is important, because being able to rely on those protections even if you are doing something blatantly hostile to the government’s interests is a strong demonstration that the protections are not fickle. Make rare exceptions, and what speakers and groups won’t wonder whether they’re the next exception? Maybe you trust the government today at the time, and think “This is a clear case; Al Qaeda are violent enemies of the state and its people, and advocating for them should not be protected.” But having made an exception once, and established a procedure for doing so, what else might use that procedure? Will Black Lives Matter be listed as a terrorist group? Environmentalist protestors trying to disrupt the power plants that, as the Saudi Arabia of coal, are the USA’s most secure power supply? In the past, this would all be an academic thought experiment, but today we live in the America At the End of All Hypotheticals. Give away the power to cross the guardrails once in a specific spot, and someone you trust not at all less may get that power and use it to attack someone you like and agree with, who can’t fight back effectively.


The unifying principle behind these is not easy to articulate. Pieces of it are: Ben Franklin’s self-paraphrase, “They who can give up essential Liberty to obtain a little temporary Safety, deserve neither Liberty nor Safety”, is an important piece. The Kantian notion of universalizable rules is another. (If that connection seems a little tenuous, I suggest reading Ozymandias on The Enemy Control Ray.) But the unifying principle, to me, is Thomas Schelling’s thoughts on bargaining, applied to political and societal norms.

Summarized, those are this: A rule never to do X, with no exceptions, is more stable and more credible than ‘never do X unless it’s also Y because Y is beyond the moral pale’. It’s easy to say “never restrict speech”, and it’s easy to check if someone else is holding to that rule. If your rule is “never restrict speech unless it’s hateful”, there will always be room for argument about what counts as hateful. If 100% of people agree about hatefulness 100% of the time, then it’s still stable. But that’s never true; almost no one is the villain of their own story, and almost no one ever feels that their beliefs and opinions are disproportionate. They may admit that they dislike their enemy a little more than is justified, but they will insist that they didn’t start it, that they are only reacting. So if you say that their speech is hate speech, they will disagree, as will many others like them. From their perspective, you are banning perfectly reasonable, justified speech that disagrees with you politically, so even if they respect the non-escalation principle, from their perspective the rule you are following is “Restrict speech if it’s loud and expresses inconvenient truths or disagrees with me politically.”, and that’s the rule they’ll follow if they get power.

So: tolerate everyone, even the intolerant. “An it harm none, do what ye will”, and only when someone has done harm, in a way everyone can recognize, is it permissible to punish them for deviation. Live and let live, even those you hate and who hate you. Their right to swing their fist ends where your nose begins, and so does their right to free speech, free assembly, free worship. Up until you’re defending your elegant schnozz from blunt force trauma, you are bound, by a suicide pact and a moral precept, not to retaliate.