A corporation, made up of both humans and other ingredients like rules, procedures, information systems, and physical infrastructure, can act as a single agent that can process information, plan, make decisions, and take actions, potentially at a greater rate, scope, and effectiveness than any individual person; no individual person fully understands — let alone could build — a modern phone or even microprocessor,1 as a corporation effectively can. Despite containing thousands of employees, billions in infrastructure, terabytes of data, and piles of rules, it generally makes sense to say that a corporation is controlled by its CEO and Board.
What does that mean exactly? First, we can discard simple prescriptions like “if it misbehaves just unplug it” or “we can put it in a box” that are sometimes used to trivialize the issue of control of AI systems: neither of these is a viable way to control a corporation.2 But with the possible exception of emergency shutdown, it’s not hard to see that corporations are set up so that CEOs do have the five control capabilities listed above. It is difficult to get right, but corporations do not typically “go rogue.” To properly get a feel for the superintelligence challenge, however, we must dial up the difficulty level.
Consider yourself as a CEO who becomes afflicted by an unusual disability, so that you can only operate at 1/50th the speed of everyone else in your corporation. You experience about one minute to the corporation’s hour, an hour to its week, a week to its year. You run a fast-growing startup. You go to sleep, during which two months pass for the startup. When you awaken, it has evolved from 102 to 253 employees with many changes: updated management structure, new IT system, new financing, and somewhat of a pivot in business model. Your staff has worked hard to push the company forward with great success, doing everything possible that doesn’t strictly require your signature. But you have 6736 emails and 178 items awaiting approval. During the day you spend processing these, the startup grows to 518 employees. There are requests to approve working with oil companies (something you, a staunch environmentalist, pledged never to do), and several related policy changes have been made provisionally to secure financing. You stay up late pushing back.
By the next morning, your staff is at 764 personnel. Business is booming, but you’re not quite sure what your business is anymore. The petrochemical division is thriving, you read with alarm. The lawyers have determined that corporate bylaws allow your C-suite to make CEO-level decisions during “extended absence” — your sleep counted as five weeks of company time. You send an angry email; the (instant) reply is so convincing that even you doubt whether the company should wait for you on timely matters. While you read it, 17 reports and 23 new decisions pile up; by the time you finish, 18 have been made without you.
By day 6 on your clock, everyone recognizes that you are the central obstacle to efficiency and success. While the Board remains loyal, your diligent (albeit increasingly resentful) staff has many avenues available. They’ve already induced you to delegate most decisions, and sneaked a number of policy changes through long documents crafted by clever lawyers (rather than waiting until their obvious merits can be explained to you). Initially this happened organically as individual staff simply wanted things to happen. But now, and all in the interest of the company, they start coordinating deliberately around transferring power from you to everywhere else. They engineer decisions that feel critical but aren’t, spin reports to tell you what you want to hear, and create crises where you get to feel you’ve won. As long as you’re happy and good decisions finally get made, why not? (Of course, discussion of these topics happens circumspectly, in coded language about “making things simple and easy for the CEO.”)
From your perspective, by day 6 things feel much better: you’re on top of things, have won key battles, and are seeing growth. On day 7, IT mentions they’ve changed passwords to several key systems due to a “security incident.” You request access, but by the time it’s restored hours later, IT has upgraded core systems and you get access to the new ones instead. You set to work learning this intuitive new software — it seems even better than before…
Have you still got meaningful control of your company? Obviously not. You’ve got a facade rather than real comprehension of what’s going on, you cannot modify the company’s goals, or place behavioral boundaries on it, or override its decisions. And if you tried to shut it down, then, well, the Board3 would probably finally step in — and stop you instead.
Footnotes
-
That said, we absolutely should have a way to reliably turn off powerful AI systems at the hardware level. This is highly nontrivial: as described below, in the situations in which we’d want to turn them off, they would be actively trying to prevent or avoid it, with more speed, strategy, and cleverness than those operating the switch. See Appendix B for a bit more discussion. ↩
-
Representing, in this analogy, a humanity that generally wants human control, but also contains elements with large economic and shareholder interests in things proceeding in a maximally productive way. ↩