When I was a baby DBA I got a job that I still talk about all the time, even more than 20 years later.
The big, stand-out story for me is how they refused to do database maintenance. They had never done maintenance.
This wasn’t a small shop, either. They had a huge document scanning operation – a big warehouse filled with these big ultra-fast IBM scanners. The operation scanned boxes of mortgage applications and other important paperwork…literally 5 million records per hour, 24 hours a day, 7 days a week.
I’ll say it again: that shop had never done any maintenance. No index maintenance, no checking for data corruption, nothing. In retrospect, that fact is a little terrifying.
I started at the new job, set them up with everything they needed for maintenance, and got their performance straightened out.
Then I said, OK. We’ve got to have database maintenance every Sunday.
I’m sure this will go well…
And the general manager – let’s call him Bob – said, We can’t afford that. We got stuff to do! We just can’t do it.
Me: Well, you’re going to have to.
Bob: Well, we can’t.
Me: OK, right.
Even back then, I knew to have that conversation in email.
Remember, kids: cover your bases and keep things in writing.
I wasn’t very reserved back then, so I did express to Bob some version of Skipping maintenance is fine up until something breaks. Then, everything will go to hell and you’ll blame me for being down four times as long as the maintenance would have been.
Anyway. That first Sunday came, and we didn’t do maintenance.
How a data pro says “I told you so”
Monday followed Sunday, and Tuesday after that. Tuesday afternoon, the system slowed to an absolute, unusable crawl.
Bob came over and said, Dude, we can’t do anything.
And I said, Yeah, I know. I told you this would happen. You can’t skip database maintenance.
This wasn’t a case of, “Oh, there’s a new DBA. He made changes and we crashed.” The system performance was killing them. It got consistently worse as the volume of data grew, and as the number of orders went up. They had a backlog bad enough that they’d been missing SLAs, and were paying $40,000 a month in fees.
But, back to that Tuesday: we were down in the middle of a week, and Bob took an emergency maintenance window. That got things working again.
Lesson learned? Lesson learned.
The next weekend rolled around. Bob sent out an e-mail saying, Remember everybody, we’re doing maintenance this Sunday.
One of the floor supervisors replied, Dude, we can’t afford this. We got tons of stuff to do.
Bob: We’re not missing maintenance.
That was the last I ever heard from him about it.
This was where I learned one of my big enterprise concepts:
You’re not special. Your server is not special.
I don’t care how busy you are, or how busy your marketing or sales departments think you are. You can’t change the math of reindexing. You can’t wish away corruption.