Ever feel like it'd all fall to pieces if you so much as turned your head? In the comments section of our article seeking your seasonal horror stories, Wayne shared a holiday WTF of a different sort that's too good not to share:
Not a holiday problem, but a me being on holiday problem.
I was the sole SQL Server database admin back in the '90s for a pretty good-sized police department. Everything ran pretty darn smooth, and I took a week off, probably to go to ComicCon. I left VERY specific instructions to call me if there was a problem.
We had a consultant working with us, I was never clear on what he was doing, but he sat in his office every day typing away and looking busy. Apparently he was writing mods to our payroll pre-process system. Police payroll is complicated because pretty much every rank has a different union, and contract, that changes how things are paid out: if you hold or are paid overtime, how sick leave works, it's all pretty weird. Or was at the time. It was wonderful getting the Peoplesoft programmer saying "We can't do that!" when they wanted us to put our preprocess into their system.
ANYWAY, this system ran an extract every week which produced two files, overtime and leave, which was then sneakernetted down the street to the mainframe. On pay weeks, there was an absolute time window for when that 3.5" floppy had to be there as it could hold up mainframe jobs. We ran the extract at about noon and the disk was there by 1. Very solid.
Until I went on vacation.
Consultant in his infinite wisdom pushed his changes to our payroll database on an extract day. And crashed the database, making it impossible to run the extract. And it was payroll week. AND they didn't call me. He spent 4-5 hours MANUALLY UNDOING HIS CHANGES. And holding up the mainframe processing schedules.
Had they called me, I could have told them that the system automatically backed itself up, and they could have done a rollback to undo his changes and reverted to a good state and run the extract.
Come to think of it, I never did find out what his modifications were supposed to do.
It's baffling why Wayne was never called, but I assume it's because the consultant had assured everyone that he had the situation under control. He was either in a mad, panicked rush to fix what he'd broken, or, he was calm and fully confident in his own abilities.
Doth pride goeth before the missed paycheck?
[Advertisement]
Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.
This post originally appeared on The Daily WTF.