In November of 2019, the USS Abraham Lincoln’s routine six month deployment to the Mediterranean Sea was extended indefinitely. The 5,000 sailors and Marines aboard were unexpectedly facing the holidays confined to ship, far from family. Just days before Thanksgiving, overwhelmed with reports from the crew and their families of Amazon order attempts failing, the officers of the USS Abraham Lincoln sent an “all hands” message to the crew assuring them that everything possible was being done to restore access to Amazon orders.

The crew of these ships rely on Amazon for everything from basic supplies to holiday decorations and gifts, making Amazon orders important to quality of life aboard ship and critical over the holidays. In November of 2019, every time the crew and their families attempted to place an order with the ship’s addresses, they received the message, “This item requires special handling and cannot be shipped to your selected location.” When they called Amazon customer service, the representatives explained that some items can’t be shipped to a military address and were advised to choose different items. This is always true, and from the customer service representatives’ perspective, there was no reason to suspect anything unusual.

Without a mechanism to look at aggregate ordering experiences of thousands of customers with different, but similar addresses, no one had a clear view of the problem. The ship’s officers were investigating stacks of complaints from their sailors and Marines, and while they knew something was wrong, they didn’t know what the issue was or who to report it to. The ship’s Executive Officer sent a frustrated email to all officers aboard the ship, “Do we know anyone that has the juice to get a higher level of attention from Amazon?”

My friend Andrew, an officer aboard the USS Abraham Lincoln, sent me an email in early December asking for help. At the time, I was an engineering manager in the AWS Cryptography organization. That’s a long way from retail ordering, but ownership is an Amazon leadership principle. “Leaders are owners. […] They act on behalf of the entire company, beyond just their own team. They never say ‘that’s not my job.’” This is a mechanism that reminds me to help my barista apply for a job at Amazon and to help my uber/lyft driver with her tire order.

Amazon also has consistent mechanisms for escalation across the company that make it relatively seamless to interact with different teams. I contacted the appropriate leaders, cut a high severity ticket to the oncall, and together, we found and fixed the root cause.

Aircraft carriers are like small military bases, and they have their own zip code. Those zip codes are a function of each carrier’s home port, so moving a carrier to a new port means changing the zip code. The originating port of the USS Abraham Lincoln was on the east coast, and the carrier was returning to a new port in San Diego, requiring a change to the ship’s zip code. 

The United States Navy notified the United States Postal Service (USPS) to expire the old zip code on the originally planned return date. When the ship’s deployment was extended, no one remembered to update the USPS. When the United State Postal Service notified Amazon to expire the USS Abraham Lincoln’s zip code, we blocked items from being shipped to it.

Everything worked exactly the way it was designed to! But when USPS was not informed of the extended use of the original zip code, there was no mechanism for identifying the issue. While I was able to order Andrew’s foot powder 36 hours after he emailed me, the challenge wasn’t truly resolved until there was a mechanism in place to eliminate the need for anyone to have “juice” to resolve a similar problem.

A mechanism is a complete process that uses tools, adoption, and inspection to deliver specific output across complex systems of people. A mechanism is a cycle that reinforces and improves itself as it operates through inspection and iteration. Amazon customer service representatives are a famously great mechanism for delivering an amazing customer experience, and the case of the USS Abraham Lincoln was an opportunity to inspect and iterate on that mechanism.

To design a mechanism, I work backwards from the desired output I want the mechanism to create. Output could be better employee retention, fewer code bugs, or a great customer experience. In the case of the USS Abraham Lincoln, the output I wanted was an alert, review, and resolution if any single military zip code had unusual error rates. 

Once there’s a clear definition of the desired result, the next step is to identify the minimum input required to produce that output. Inputs will be used by the mechanism, and I only want to invest time collecting what’s necessary to get my results. Mechanism inputs could include data that illustrate the business problem or opportunity and resources like real estate, hardware, and talent (people). In the case of the USS Abraham Lincoln, the inputs are customer service calls and messages, customer service agent software, and system data about order rates. 

To transform the inputs to outputs, a mechanism needs tools, adoption, and inspection. The tool is the implementation, and it can be simple or complex. For the USS Abraham Lincoln zip code issue, we considered tools like system alerts when a zip code change by the USPS impacts a customer address and updated training for customer service agents. We eliminated most of those tool options because of their associated cost, time, and risk to adoption. Part of the calculation in choosing the right tool is adoption.

The most effective tools are rendered ineffective without adoption by stakeholders. For example, unnecessary or excessive alerts transition from signal to noise, and humans ignore noise. An alert that isn’t adopted as a cause for action is no alert at all. For the USS Abraham Lincoln zip code problem, we chose a very simple tool and process precisely because it was easy to guarantee immediate adoption.

The last ingredient in a mechanism is inspection. We have to audit our outputs for success criteria and iterate on both the tool and adoption when we’re not achieving the desired results. An example of a mechanism that needs inspection is a recurring business meeting that no one finds valuable. Frequently asking questions about the output a mechanism is intended to achieve and inspecting the quality of that result has to be part of the mechanism.

There’s a leadership wall that a lot of people hit at different points in their careers, and it always sounds something like, “I have been super successful, and my company is investing in me. My team has grown x%, but we are not delivering as well as we were before the growth.”

What those leaders are describing is a failure to scale themselves. The mechanisms that were successful for them with a smaller team are failing them now. Mechanisms are devices we use to create deterministic results across systems of people. Leaders frequently struggle, or fail, to course correct, when they have borrowed rather than intentionally designed their mechanisms. They are missing the inspection and iteration required to deliver the desired output as things change.

As a Marine Corps wife, I am proud to have been able to serve the sailors and Marines of the USS Abraham Lincoln. Those sailors and Marines stayed at sea for 290 days, the longest carrier deployment since the end of the cold war. And when Andrew called with a similar report a month later, I was able to confidently say that there were no problems. Sometimes, it is just that “…[an] item requires special handling and cannot be shipped to your selected location.”