May 4, 2017

When sudo is too dangerous

On May 4th, Amazon Web Services (AWS) had a major service disruption on the East Coast. It was so bad that Amazon couldn't even notify it's customers about what was happeneing. You can read the post mortem here, but the crux of the story is that a authorized tech mistyped a command. He was supposed to remove a small number of servers from the S3 accounting subsystem, but he accidentially removed a much, much larger set of servers.

These things happen. Heck, I've mistyped several things in this post already! Of course, my typos are usually pretty easy to recover from. But let's be clear: as a developer, I've fat fingered my way into several critical errors, including truncating a table in a production database.

The U.S. military has already solved this problem with the two-man rule. In summary, two people are required to independently authenticate and initiate a nuclear missle launch. We should do the same with mission critical commands in linux by modifying the sudo command.

What I propose is that when a user runs the sudo command, the program will check the config file to see if the command being invoked is in the two-man (or dual-auth) list. If it's in the list, sudo grabs the timeout value from the config (30 second time out as the default.) Sudo then checks the log file to see if the exact same command had been issued by a different user within the timeout period. If it has, sudo runs the command. If not, sudo notifies the user that a second authorized user must execute the same command within the timeout window.

Tags: Security