A disk is one of those server components that you know will eventually fail. In a datacenter with thousands of servers, that means there are always going to be disks that need to be replaced.
At a company I previously worked with, we had multiple datacenters, and one of the teams was responsible for keeping track of server health, including disk-related issues.
Normally, when we found a disk problem, we would ask the datacenter engineering team to replace the affected disk. Once the disk was replaced, the RAID array would start rebuilding.
So, what happens during a rebuild?
The servers used several RAID configurations, including RAID 1, RAID 5, and RAID 10. The whole point of RAID is to keep the data available even when one or more disks fail.
When a failed disk is replaced, the RAID controller starts reconstructing the missing data using the remaining disks. That's what we usually referred to as a disk rebuild.
The problem is that a rebuild can take a while. Sometimes it finishes in a few minutes, but depending on the amount of data and the condition of the array, it can also take hours or even days.
During that time, someone from the team still had to keep checking the server to see whether the rebuild was done.
The annoying part
We already had a Bash script that could check the disk rebuild status, so the actual check wasn't difficult.
The annoying part was everything around it.
First, we had to find the server's IP address, then run the script, then check the result. The whole thing took at least around 10 seconds.
Then we had to do it again later.
And again.
And again, until the rebuild finished.
There wasn't even a fixed interval for it. Some teammates checked every hour, some every 30 minutes, and some checked more frequently depending on the situation.
On a typical day, there were around 150 servers that needed disk rebuild monitoring.
That made me realize that the 10 seconds wasn't really the problem. The problem was doing those 10 seconds over and over again, across a lot of servers and a lot of people.
How much time were we actually spending?
We can roughly calculate it with:
Manual checking time = Number of team members × Time per check × Number of checks
And:
Number of checks = Rebuild duration ÷ Check interval
For example, let's say five people are monitoring a server, the rebuild takes 24 hours, and everyone checks it every 30 minutes.
5 × 10 × (24 ÷ 0.5)
= 2,400 seconds
= 40 minutesSo just one server can cost us around 40 minutes of combined human time in this example.
Now imagine doing that for a large number of servers, every day.
At some point I started thinking: why are we doing this manually in the first place?
The idea
The solution I came up with was a server tracker.
The idea was pretty simple: instead of manually checking the same server every 30 minutes, we register it once and let the system handle the checks from there.
When someone registers a server, the backend first checks whether it currently has a rebuilding disk. If it does, we add that server to the tracker.
After that, the user doesn't have to keep checking the server manually anymore.
Letting the cronjob do the boring part
Every 30 minutes, a cronjob goes through all the servers currently being tracked and performs the same disk rebuild check.
For each server, we check whether the rebuild is complete, still in progress, or whether something went wrong while checking it.
Once the cronjob finishes checking all the servers, it sends a summary of the results to the team.
So instead of having multiple people repeatedly checking servers throughout the day, we get one automated summary of what's happening.
Before and after
Before the tracker, it was basically:
Check server
↓
Wait 30 minutes
↓
Check server again
↓
Wait 30 minutes
↓
Check server again
↓
...After the tracker:
Register server
↓
System keeps checking it
↓
Receive summaryTechnically, the system wasn't doing anything particularly fancy. The Bash script already existed, and the actual check only took a few seconds.
What changed was that we stopped spending human time on it.
One small automation
I like this kind of automation because it wasn't really about building something complicated. We already knew exactly what needed to be done, and we already had a script that could do it.
The problem was simply that we were asking people to repeatedly perform the same predictable task.
Ten seconds doesn't sound like a lot. But when you have around 150 servers that may need monitoring, and each one can take hours or days to rebuild, those ten seconds start adding up.
Sometimes automation isn't about making a process faster.
Sometimes it's just about looking at a boring task and asking:
Does a human really need to do this?