This weekend I finally decided it was time to retire my old QNAP chassis and move my Proxmox server into a brand new Jonsbo N5.
Like every hardware migration, I figured it would be a simple afternoon project.
You already know how that turned out.
While moving everything into the new case, I also swapped my LSI 9211-8i HBA with my video card so each would sit in a different PCIe slot. Once everything was buttoned up, I powered the server back on.
Immediately something looked wrong.
One of my drives wasn’t showing up.
Since I had just moved the HBA, my brain immediately blamed the PCIe slot.
That assumption sent me down a rabbit hole that lasted for hours.
The hardware
The server currently consists of:
- Jonsbo N5
- Proxmox VE
- LSI 9211-8i SAS HBA
- 8 × Seagate IronWolf 4 TB drives
- 4 × WD Red 4 TB drives
- 2 × Crucial P3 1 TB NVMe SSDs
- Samsung SSD boot drive
Nothing had changed except the case and the PCIe slot where the HBA lived.
Or so I thought.
First question: What does Linux see?
The first thing I always do is ask Linux what hardware it sees.
lsblk -d -o NAME,SIZE,MODEL,SERIAL
Then I wanted a complete picture.
lsblk -o NAME,SIZE,TYPE,MODEL,SERIAL,FSTYPE,MOUNTPOINT
Something still wasn’t adding up.
Verify the HBA
Maybe moving the card caused a problem.
First I checked whether Linux could see the controller.
lspci | grep -Ei "sas|raid|storage|lsi|broadcom|avago"
Then I verified the driver.
lspci -nnk | grep -A5 -Ei "sas|raid|storage|lsi|broadcom"
Everything looked perfect.
The controller was detected.
The correct driver was loaded.
No obvious problems.
Verify the driver
To be sure, I confirmed the kernel module was loaded.
lsmod | grep mpt
Still looked healthy.
Read the kernel logs
Next I started reading the kernel logs.
dmesg | grep -Ei "mpt|sas|port enable|error|fail"
The logs looked great.
The firmware initialized.
Every SAS PHY came online.
Eventually it reported:
port enable: SUCCESS
Looking back, that should have been my clue.
The HBA wasn’t the problem.
Check every storage device
I listed every storage device.
lsscsi
Then every persistent disk identifier.
ls /dev/disk/by-id
Then only the physical SATA drives.
ls /dev/disk/by-id | grep "^ata-"
Again, everything looked healthy.
Look for errors
Maybe something was failing after boot.
dmesg | grep -Ei "error|failed|timeout|reset|offline"
Nothing.
No controller errors.
No timeouts.
No link failures.
No resets.
Verify SMART
Finally I checked every drive individually.
for d in /dev/sd?; do
echo "===== $d ====="
smartctl -i "$d"
done
Every connected drive reported healthy SMART information.
The drives weren’t complaining.
Linux wasn’t complaining.
The HBA wasn’t complaining.
I was officially confused.
When nothing makes sense
At this point I started questioning everything.
Was the motherboard disabling PCIe lanes?
Did moving the HBA change something in the BIOS?
Was the firmware acting up?
Was Linux hiding a drive?
Was Proxmox doing something strange?
Eventually I even asked ChatGPT because nothing I was seeing made sense anymore.
Ironically, all the logs were telling me exactly what I needed to know.
The HBA was healthy.
Linux was healthy.
The controller was healthy.
I hadn’t found the actual problem yet.
The real problem
Eventually I decided to pull the front drive cage back out of the Jonsbo N5.
I wanted to inspect everything without removing the power supply again.
That’s when I found it.
While moving everything from the old QNAP chassis into the new case, I had accidentally pulled the SATA power cable loose from the backplane while removing the drive cage.
One drive wasn’t receiving power.
That was it.
Hours of troubleshooting.
Dozens of Linux commands.
Reading logs.
Questioning PCIe lane assignments.
Questioning firmware.
Questioning Linux.
Questioning Proxmox.
The actual problem was a SATA power cable that I had unplugged myself.
The part I’m not proud of came next.
By that point I was frustrated. I kept trying to get a better look at the backplane without taking everything apart again, and I pushed a little harder than I should have.
I ended up cracking part of the plastic housing on the SATA power connector.
Thankfully it was only the plastic.
The actual connector survived, the backplane wasn’t damaged, and everything worked once I plugged the cable back in.
I got lucky.
That could have turned an annoying afternoon into replacing hardware.
The bigger lesson
The real mistake wasn’t unplugging the cable.
Mistakes happen.
The bigger mistake was trying to troubleshoot while I wasn’t in the right state of mind.
I’ve been working in IT and infrastructure for well over twenty years.
I’ve built servers.
I’ve built networks.
I’ve managed virtualization platforms.
I’ve worked with enterprise storage.
I’ve spent thousands of hours troubleshooting hardware and software.
And I still made one of the oldest mistakes in the book.
I was running on almost no sleep.
I hadn’t eaten all day.
I hadn’t even taken a shower.
I kept telling myself I was one more command away from figuring it out.
Instead of stepping away, I kept digging deeper.
I convinced myself the problem had to be something complicated because I had just changed hardware.
It wasn’t.
It was a loose power cable.
Experience doesn’t make you immune to simple mistakes.
Sometimes experience works against you because your brain starts looking for complicated answers before checking the obvious ones.
We’re all human.
No matter how many years we’ve been doing this, we all have days where we’re tired, hungry, frustrated, and miss something sitting right in front of us.
My new troubleshooting checklist
Before I blame Linux…
Before I blame Proxmox…
Before I blame firmware…
Before I blame drivers…
I’m going to follow this order.
- Check power.
- Check every cable.
- Reseat everything I recently touched.
- Verify the BIOS sees the hardware.
- Verify Linux sees the hardware.
- Only then start investigating software.
Simple problems happen every day.
Sometimes the hardest part is believing the answer is that simple.
One final thought
I’ve been doing IT and infrastructure work for over two decades.
If there’s one thing this experience reminded me, it’s that experience doesn’t prevent mistakes.
Sometimes it makes them easier to make because we start looking for complicated problems before checking the simple ones.
The best engineers I’ve worked with aren’t the ones who never make mistakes.
They’re the ones who recognize when they’re no longer thinking clearly, step away, reset, and come back with a fresh perspective.
This time I got lucky.
I lost several hours.
I cracked the plastic on a SATA power connector.
I learned a lesson I’ll probably remember for the rest of my career.
The next time I’m convinced Linux, Proxmox, or my hardware is broken, I’m going to stop and ask myself one simple question.
“When was the last time I ate?”
Because sometimes the best troubleshooting tool isn’t another command.
Sometimes it’s a good meal, a hot shower, and a full night’s sleep.