Truenas disaster #2

Random or Off Topic
1

Decided to drop the jj thread hijack and start my own.

Disclaimer: This is in no way an issue with truenas. It works great when the hardware is reliable.

My truenas stopped working quietly a few days ago. I thought power glitch perhaps and restarted it. Lights came on. Nothing happened though. Built this box a year ago with new processor, ram, motherboard, power supply, and 4 matched ironwolf drives into a case that had a working but older system in it.

Tested ram independently, swapped power supply then motherboard and it came back up after switching to vga output on the newer one, so swapped power supply back. No problems. Must be a bad motherboard chipset, bios or something… Guess these things happen.

Get it put back together and start reconfiguring the network adapter and making sure it works. Then it starts to randomly reboot once, twice, then goes into a boot loop and then stops responding… In a matter of about a minute. Now responds just like the other one. Power only activates fans. Motherboard #2 (brand new) is toast.

Since the truenas was using the case, i had this old system running linux on my workbench without a case. Used it to reprogram the arduino when the 3dchameleon had issues. I reinstalled the original system motherboard and power supply (a little dusty compared with the nearly new one) with all the drives and it is back up and running.

So i have 2 cooked motherboards and a bad power supply and some unused ram now. I bought this second board when we made the traditional christmas time pilgrimage to microcenter planning to gift it to my son for his birthday in a few months. Not happy about this.

@jono035 was most likely right that it was the power supply. I’ve built many computers over the years and never seen this. I built one over christmas several years back (core 2 duo) with all new parts and had the brand new power supply give it up on the first power switch. After replacing it and getting a warranty refund, the computer worked fine for 12 years with the newer power supply.

Well, that’s the story of consumer grade hardware. Going to try and find another AM4 socket board before May and hope the old hardware holds up. Frustrating.

2 Likes
2

Ah, that sucks man, sorry to hear that.

That’s a real danger with power supplies, they’re often a component that doesn’t get much thought but a bad power supply can really ruin your entire day. A moderately bad power supply can be incredibly frustrating to diagnose (unreliable rails, lots of voltage ripple, dropping out under load briefly etc.). A very bad power supply can not only kill the hardware that’s connected to it but also the replacement hardware during the troubleshooting stage… It sucks that there isn’t a better way to keep an eye on them.

1 Like
3

Is this more likely with cheaper no-name PSUs? Or is it a concern with even the well-known brands like Corsair and such?

I’m pondering building out a NAS and/or home server box, as I’ve currently got an old Dell micro PC running Proxmox that’s handling my Home Assistant and Jellyfin installs, but that machine is very limited on storage, as it has a single M.2 and one SATA HDD slot.

I’m definitely interested in avoiding any pitfalls on hardware that’s more likely to fail, so if I’m better off avoiding cheapo power supplies, that would be good to know.

2 Likes
4

I’d say it’s possible with any brand of PSU but significantly more likely with cheaper products or products with a brand that isn’t necessarily as worried about bad press. The best option (and what I typically go for) would be mid-tier and above power supplies from a well-known brand. Doesn’t need to be a top $ enthusiast level unit, but I’d also avoid the most economy options.

There have been known problems with PSUs from some brands (Gigabyte was one, I believe), but the nice thing about issues from known brands is that the issues actually get press and people put pressure on the brand to make the situation right. That’s more the value of known brands, in my opinion, rather than there being any inherent outright quality difference. I doubt most of these companies are actually designing their power supplies from scratch, anyway.

I’d say to try keep the cost in-line with the other components. If you’re throwing together a machine from bits that are scrounged up, I wouldn’t bother with a super-great power supply for them. On the other hand if I were putting together something full of expensive hardware and wanted reliability, I’d be making sure to spend a bit extra on the power supply.

Gamers Nexus has done a ton of surprisingly in-depth power supply testing in the past so they’ve got some good recommendations there.

This is a video from a couple of years ago that has a few recommendations listed. If you go forward to the ~17m mark they cover the power supply protections working as expected. Something with reasonably close tolerances on the over-current and over-power protections is probably more likely to be built in such a way that it’s not going to do anything nasty under most failure conditions. It’s not a guaranteed thing because they’re testing the behaviour of the power supply, not the actual design for issues that could cause shorts between rails or other things that will cause catastrophic hardware damage.

Edit: Actually in that video they comment that all the power supplies in the round-up come from one of 2 manufacturers, which is what I’d expect at the lower end. They’re not being designed by the brands. At best they may be a semi-custom version with the features the brand has requested or quality adjusted up and down to fit a specific price point. At worst they may be a cookie-cutter rebadging effort.

That’s a weirdly in-depth conversation about power supply stuff, too. It’s GN talking to a power supply guy from Corsair. I think at some point they talk about the differences of some of the Corsair power supplies which probably gives some good insights into which is worth picking up.

3 Likes
5

@jono035 Super-helpful! Thanks so much!

1 Like
6

The first one that fried on initial power up was a rosewill and those were the big deal at the time. The one that failed is a 600w off-brand one I got… Likely off amazon. It is a Apevia and is gold rated 80 plus (power efficiency?). Not a quality rating.

7

Yeah, the 80plus is just an efficiency rating. Good to have, for sure, but everything comes at a cost. A higher efficiency rating at the same price might mean that there are corners cut elsewhere like EMI filtering, inrush protection, lower quality DC link capacitors etc. Or it might just be that it’s a higher volume product or a brand selling it at lower markup, etc. etc.

I think the current PSU in my gaming PC is a Corsair 850W.

1 Like
8

This system runs 4 hard drives and one ssd. No video card, no monitor.

9

Ok so this is an arena i have been in for a verrrrry long time! Do you have a list of hardware? Most motherboard mfrs will take the board back under warranty, if new. Am4 so this is amd. What were the boards, what size power supply? 4 spindles and amd this system needs a good power supply!

1 Like
10

Repurposed case

Powersupply (deemed bad)
Apevia ATX-PR600W prestige power 80+ gold bought Dec 2023 from Amazon.

PXL_20250122_023514704
PXL_20250122_0235147041920Ă—1446 179 KB

Motherboard:
Gigabyte B450M DS3H WiFi bought Dec 2023 from microcenter.

PXL_20250122_023456352
PXL_20250122_0234563521920Ă—1446 182 KB

Ram:
TForce vulcanZ 32 GByte 2 Dimm kit DDR4-3200 bought Dec 2023 microcenter

Processor:
AMD Ryzen 5000 (w/graphics)

10 port sata adapter expansion card

Hard Drives:
4 ea 4TB Seagate Ironwolf bought from amazon Dec 2023

Ssd: sandisk 128 GB repurpose, exact specs not readily available

New motherboard:
ASRock A520M-HDV bought Dec 2024 at microcenter.

If i would have left the swapped power supply in with the new motherboard, this wouldn’t have been an issue.

Processor and ram are fine… I think.

11

Burn in, burn in, burn in.

Somewhere around here I have an ATX load board that I use when building a system (which I seldom do these days).

I run new PS on that for at least 3 days to try and verify things are stable. I gave away my monitor board (basically a set of embedded DVMs to monitor the voltage rails. Useful stuff.

Out of curiosity, what brand of PS did you use?

Did you have a reason to suspect a power glitch? E.g. an outage? Was this sysetem on a UPS?

12

No ups. No power outage of record. In all honesty, it was a budget build with reasonable parts and a cheap power supply. The power supply didnt need to be a fancy modular with leds or 1200W output. I sure hope there isnt more to this story other than happily ever after

13

The old reliable died this morning. Similar behavior. No post with power. Different everything except the hard drives and ssd. Removed everything but ssd, no go. Removed ssd and it posts. Put in previous hd from this older system, no post. Move to hd to sata channel 1 from channel 0 and system boots.

Bad ssd and sata0 on board. Sandisk x110 128gb sata drive that was the nas boot drive is done. Sata channel 0 on the motherboard does not work now either.

With the Synology nas example at such a lower watt rating, would it be worth considering to get an sbc and a 4 disk usb das box and use that for a nas to save power, cost and complexity?

Going to get another boot os drive set up. While this nas is actually a remote mirror of another system, that other system also hosts a backup of this system, so no data was harmed in the destruction of this hardware.

1 Like
14

I would imagine you’d pay a hefty performance penalty once you move from integrated. SATA and/or M.2 to USB for the drives, but I guess that’s only a big deal for certain use cases.

15

It’s a NAS. if you don’t have >1Gbps Ethernet then faster disk throughput won’t help you.

1 Like
16

Fair enough.

I think part of it for me is that my brain hasn’t fully adjusted to the improvements in USB speeds, so I still see that as the bottleneck.

Technically, my router hardware supports link aggregation up to 2gbps, but that is probably not supported on Fresh Tomato, so might be limited to 1gbps on Ethernet.

17

You definitely could. I’m not sure what the performance would be like. There are also other lower power options like a 2nd hand Intel NUC. I’ve had friends that have repurposed old laptops with broken screens or shagged batteries for that task, too.

I think I’ve seen some SBCs out there that have PCIE slots, so you could always add a hardware RAID card if the USB is a concern. Personally I’m not a huge fan of connecting critical peripherals via USB, I’ve spent too much time dealing with failures to enumerate correctly or things enumerating in a different order and mucking everything up. That may be a thing of the past, though.

Edit: There are also M.2 to PCIE adapters which could allow an even wider variety of CBS to work. Anything with an M.2 slot with PCIE lanes run to it could in theory be adapted out to a full size PCIE slot and then used to run a raid card.

That made me wonder if there are M.2 RAID cards and, sure enough:
https://www.newegg.com/orico-pm2ts6-bp-pci-express-to-m-2-card/p/17Z-0003-00027
Which is a $30 card that fits an M.2 slot and has 6x SATA 6Gbps ports. Zero idea if it’s any good at all, though.

1 Like
20

When was the last time you changed your password, how strong is it, and is it unique to these forums?

1 Like
22

Jeff Geerling built a Pi 5 NAS using the PCIE slot (and then also tweaked it to add a PCIE switch and 2.5GB Ethernet HAT, and tested both ways). He was getting pretty decent performance, even with just the native 1GB Ethernet on the Pi.

No idea of how expensive the 8GB drives he used were at the time of filming, but apparently they’ve gone up substantially, to the point where this rig would cost more than $2K to pull off.

But it probably represents the upper limit of what you could expect from a Pi-based NAS. Might be able to get a little more juice from another SBC, particularly one with native 2.5GB Ethernet, but probably not much more.

2 Likes
23

Turns out the SSD on the truenas bit the dust. It took out sata channels on old reliable, but channels 3 and 4 work fine with another drive. The offending drive is no more. I got a pair of 256GB SSDs for the truenas OS and have started to reconfigure.

In a further fit of desperation last night. I was thinking if it was really just the SSD that gave it up, is there a possibility that the boards might still be alive… But with bad data channels? So with the “bad” replacement board, i powered it on. Nothing. But i didnt just flip the os switch off. I held the power button down for like 10 seconds sort of resigning to defeat… I didnt buy the extended warranty, microcenter only covers it for 15 days and the mfgr wont cover customer induced damage. During the soft power down, the post screen from the motherboard flickered on for a second. I then started boot cycling the morherboard and pressing keys to see if it would give access into the cmos settings and it did. Set the boot drive and it is running. I did the same thing with the original, but because it had hdmi out only. I dropped in an ancient video card and it was also anle to recover.

Post mortem analysis: nothing concrete. My suspicion is the browning out of the SSD wiped something on the motherboard that had to be reset that pulling the cmos battery did not do (tried that). In addition to frying a sata channel on the old drive.

1 Like