CodeSOD: Crappy Wiring

Ellen had just finished washing her hands when her phone buzzed. It vibrated itself off the sink, so there was a clumsy moment when it clattered to the restroom floor, and Ellen tried to pick it up with wet hands.

After retrieving it and having another round of hand washing, Ellen read the alert: the UPS was in overload.

That wasn't good. Their power supply was scaled for future growth- there was more than enough capacity to run all their servers off the UPS. So that meant something was very wrong. Ellen dashed out of the bathroom, conveniently located just off the server room, expecting to see something shorting out and sparking and an ugly cloud of smoke.

But everything was fine. Everything in the server room continued to hum along nicely. Barry, the other server room tech, had already checked the UPS management console: something was drawing 2000W above the baseline load. "Something in here must be drawing extra power," Ellen said. Barry agreed, and they went off searching.

While they searched, their phones pinged again: the UPS was no longer in overload.

"Weird," Barry said. "Do you think it's the health monitoring going off?"

"I mean, it could be," Ellen admitted. "But reporting an overload? I suppose we should open a ticket with the manufacturer."

So Ellen did that, while Barry trotted off to make his trip to the restroom. A few hours later, the UPS manufacturer sent their reply, which in short was something like: "You clearly have misconfigured it. Check this wiki article on how to set alert thresholds. Closing this ticket."

Frustrated, and full of coffee, Ellen traipsed back to the restroom to relieve herself. This time, she was halfway out the restroom door when her phone buzzed. The UPS was in overload again.

Once again, there was nothing obvious going wrong, but the management console reported that something was drawing 2000W above the normal load. "Some device in here has to be going wildly wrong," Ellen said. "Did somebody plug in a fridge, or a space heater or something?"

"I dunno," Barry said. "Do you think we're going to need to do an inventory?"

"I suppose so," Ellen said. She checked her phone. "There are a few hours left in the day, let's see if there's some device we don't expect, or something."

"Sure, but ah, gotta hit the head first," Barry said. He went to the restroom while Ellen pulled up the inventory of the server room and started walking the racks, looking for anything out of place.

That's how they spent the rest of the day, searching through the server room, trying to figure out what might be drawing a whopping 2000W of power off the UPS. There was nothing unusual or out of place, however.

Frustrated and annoyed, Ellen took one last trip through the restroom before she left for the day. And, like clockwork, she was just drying her hands on a paper towel when her phone pinged. The UPS was in overload again.

Once is chance. Twice is coincidence. Three times is a pattern. When Ellen used the bathroom, the alert went off. When Barry used it, it didn't.

Ellen looked around the bathroom. It was a spartan, functional room, without much in it. Lights. Exhaust fan. Sink. Sink. The sink used hot water. On the wall, under the sink, was a petite five liter instant-on hot water heater.

"Oh, they didn't."

The next morning, they got an electrician in. Within a few minutes, Ellen was able to confirm that they, in fact, had: they had wired the bathroom circuit to the server room UPS, not building mains. Every time Ellen washed her hands, the UPS went into overload.

The electrician was able to rewire the bathroom, which lifted that additional load off the UPS. That solved the obvious problem, and made everyone happy.

Nothing could fix the other problem, however. Ellen couldn't bring herself to shake Barry's hands anymore. Maybe he just used cold water… maybe.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

This post originally appeared on The Daily WTF.

Leave a Reply

Your email address will not be published.