Device reordering has been the bane of sysadmins existence for a long
time. I still remember when a Sun firmware (with a hardware intervention
by the vendor) update changed the order it scanned for scsi devices and we
got different c0t0's in the c0t0d0s0 addresses. I had to learn Forth to
work around that at scale, *sigh* (yeah, it not only changed the
controller, but the cable plug order). That's why these days my vm's don't
talk directly to disk, they talk to some form of abstraction like a NAS.
Made it so the thousands of vm's don't really care and you only have to
worry about it with the dozens or so of hosts. And for them you can spring
for a couple of extra disks to run the host root on its own controller and
eventually figure out a topography that recovers in some sane fashion.
Doesn't help you, here, though and I feel your pain.
The issue I ran into with udev rules is you don't know the new disk's ID
before you plug it into something. I played around a bit with whitelisting
devices, etc so it only loaded the disks I told it about and then I added
the new device to the whitelist when I had it configured and only then did
it get mknoded. This was with lvm, not mdadm. But that level of
intervention is just asking for trouble. You CAN do it, with the lower
level commands, but will you do it right when it's xmas eve and it blew the
magic smoke at 2am? Or if you have to talk a junior through it because
you're not near a terminal? Always good to have the knowledge in your back
pocket, but I wouldn't call that a solid recovery plan. I just could never
count on any SAS/SATA ordering disks consistently like we could almost
usually count on with the old controller, cable, order system we could with
scsi or
I've been looking at an old xen system where I did use mdadm, and what we
did was define the /dev/md0 to use an array UUID(ARRAY /dev/md0 UUID=<uuid>,
restricted mdadm to just look in /dev/disk/by-uuid, and then had to
manually fix when we "hotswapped". We had 2 disks set aside for host root,
and each xen server had its own grabbag of how it raided the vm volumes.
Probably not what you're looking for, but might prevent your mdadm -scan
from giving it device paths instead of uuids.
#get uuid
mdadm --detail /dev/md0
# set it to use uuid in mdadm.conf
ARRAY /dev/md0 uuid=<your-uuid>
#remove all the disk paths from mdadm.conf
# then prevent mdadm from scanning for and readding those non-uuid paths in
mdadm.conf
DEVICE /dev/disk/by-uuid/*
# add the following to ask mdadm to please try not to recreate everything
again and use metadata
HOMEHOST <full hostname>
#reboot and pray
This is just me going through a very old cent5 xen server, so... YMMV.
Going through the 15 y/o instrux(for NEW builds!!! Do not do this if you
wish to preserve anything!!! This is being run off boot media!) for these
guys, they did a lot of bypassing the --scan with things like mdadm
--assemble --no-device /dev/md0, etc. Dunno how much you're willing to
rebuild, but the main steps are:
#i honestly don't know how to even do this one since md0 doesn't exist yet,
but I'm not an mdadm person
mdadm --create /dev/md0 --level=1 --raid-devices=2 --uuid=<your uuid>
/dev/disk/by-uuid/<disk1> /dev/disk/by-uuid/<disk2>
# test with
mdadm --assemble /dev/md0 --uuid=<your uuid>
#and to get the madm.conf happy
mdadm --detail --scan | grep UUID > /etc/mdadm.conf
# add DEVICE and HOMEHOST. it'll look like:
DEVICE /dev/disk/by-uuid/*
HOMEHOST yourhostname.localdomain
ARRAY /dev/md0 --uuid=<your uuid>
# at this point it has you do another --assemble and see what it says for
itself.
# and also do the udev rules at this point to bind scsi addresses to disk
uuids
# there's a warning here about mdadm trying to autoassemble at start, so
make sure DEVICE is set
# and also rerun dracut -f so your initrd has your new settings. (or
whatever you use to rebuild your initrd i guess. i have a note about
mkinitrd with a bunch of machine-specific flags)
# post stuff troubleshooting, logs (or journalctl systemd-udevd these days)
and mdadm --examine /dev/md0
I tore these guys down pretty quickly and replaced them with NAS-backed
with fakeraid boot disks because that's what I do at work and it was 15
years ago, so my memory isn't very clear. The "NAS" in this case has been
another group of disks, an eSATA, of all things a share off a windows
machine, and am currently running a very very basic ZFS running on a cheap
little NUC. For me it's about being substitutable, that the vm's don't
care, they are presented with a simple device. For home lab.
Sorry to be long-winded, just wanted to give you all the ammo I could if
you wanted to dig into it. At the end of my notes I've written down
something about FS and disk uuids being different and make sure to check
for conflicts? I don't remember adding the note, but hey, 15 years.
I can say that while i was running this thing, it DID use uuids and never
tried to use the scsi paths. And the ones I saw doing this in the wild
were consistently ok unless someone overwrote mdadm.conf and that's where
we always started troubleshooting at. "someone" also being autoscan which
happened a couple times due to booting to recovery media.
On Mon, May 19, 2025 at 4:26 AM Ryan Petris via PLUG-discuss <
plug-discuss@lists.phxlinux.org> wrote:
> Yeah, I powered off the drive bay to see what would happen :).
>
>
> And I'm guessing it was in-use when you powered it off, i.e. part of a
> running mdadm array.
>
> but I'm not rebooting for hot-pluggable
> drives
>
>
> The problem is that if hot-pluggable drives are still in use at the time
> you remove them, it's likely that it will fall into an invalid state where
> it seemingly doesn't exist and yet still somehow exists. I've had this
> happen in particular with not-so-great NVME to USB controllers where lots
> of activity causes the drive to fall offline.
>
> My hunch is that you needed to fail/remove the drive before it would get
> cleaned up, and you'd also have to zero the superblock on the drive you
> removed before you can add it as a "new" drive. See here:
> https://wiki.archlinux.org/title/RAID#Removing_devices_from_an_array
>
> I need it to work if I remove the disks to do something, then put
> them back in.
>
>
> You shouldn't be doing this... if you're removing disks with the intent of
> adding them back and the array just continuing like nothing happened, you
> need to unmount then stop/disassemble the array first or just shut the
> machine down, and then when you're done power it back on or reassemble and
> mount the array again.
>
> On Mon, May 19, 2025, at 12:01 AM, der.hans wrote:
>
> Am 18. May, 2025 schwätzte Ryan Petris via PLUG-discuss so:
>
> moin moin,
>
> >> It picked up the wrong devices when powering the drive bay down and up
> >> again. mdadm was showing and trying to use the orginal sd names rather
> >> than the newly assigned names.
> >
> > That doesn't sound right, something else must have been going on.
> >
> > What likely happened is if the drive was lost unexpectedly, the old
> > device names were never cleaned up properly, so when it got plugged back
>
> Yeah, I powered off the drive bay to see what would happen :).
>
> > in there were in effect duplicates which may have confused mdadm. If you
> > were to have restarted it would have likely been fine.
>
> Rebooting will likely fix it, but I'm not rebooting for hot-pluggable
> drives. I need it to work if I remove the disks to do something, then put
> them back in.
>
> I'm certain I can make it work, but I'd like it to be less annoying.
>
> I'm also trying to refresh my memory a bit and learn the current state of
> "make it work like I want".
>
> ciao,
>
> der.hans
>
> > ..."fine" assuming that the array immediately went offline and then
> assembled properly when the lost device existed again. However if the array
> stayed online and continued to work then mdadm would likely reject the
> drive if it came back online anyway and you'd have to "remove" and re-add
> it for it to rebuild.
> >
> > On Sat, May 17, 2025, at 9:54 PM, der.hans wrote:
> >> Am 17. May, 2025 schwätzte Ryan Petris via PLUG-discuss so:
> >>
> >>> You can't, unfortunately. The only thing in /dev you can technically
> >>> rename are network interfaces; the rest you can only add symlinks for,
> >>> which is what for instance /dev/disk/by-partlabel/* is.
> >>
> >> A shame, I like long path names :)
> >>
> >>> Why do you care what mdadm shows for device names anyway? If you're
> >>> concerned that it will pick up the wrong disk on reboot, it won't
> >>> because internally it's using its own identifiers to find the right
> >>> drives, and will snow you whatever /dev/sd* it happens to end up
> >>
> >> It picked up the wrong devices when powering the drive bay down and up
> >> again. mdadm was showing and trying to use the orginal sd names rather
> >> than the newly assigned names.
> >>
> >> I will experiment more with mdadm. It's been a while since I used it, so
> >> was expecting some reacquaintance exercises to be necessary.
> >>
> >> ciao,
> >>
> >> der.hans
> >>
> >>> on when running mdadm commands. If you want to figure out which
> >>> /dev/sd* device belongs to your disk label, you can just run `readlink
> >>> /dev/disk/by-partlabel/raid...`.
> >>
> >>
> >>>
> >>> On Sat, May 17, 2025, at 6:46 PM, der.hans via PLUG-discuss wrote:
> >>>> moin moin,
> >>>>
> >>>> I'm using a USB JBOD enclosure to build a RAID set.
> >>>>
> >>>> Restarting the enclosure ends up with the drives on new names, e.g.
> sda,
> >>>> sdb and sdc come back sdd, sde and sdf.
> >>>>
> >>>> I added labels to my disk partitions and was hoping to use them, e.g.
> >>>> /dev/disk/by-partlabel/raid{0,1,2}, but mdadm turned them back to
> >>>> /dev/sd{d,e,f} names as members of the array
> >>>>
> >>>> Anyone know how I can either get /dev/sd names not to change for
> removable
> >>>> media or get mdadm to accept the names I want to use?
> >>>>
> >>>> I'd rather the latter.
> >>>>
> >>>> ciao,
> >>>>
> >>>> der.hans
> >>>> --
> >>>> # https://www.SpiralArray.com https://www.PhxLinux.org
> >>>> # <arclight> Delicious red tape, like a Twizzler but flat. And
> adhesive.
> >>>> ---------------------------------------------------
> >>>> PLUG-discuss mailing list: PLUG-discuss@lists.phxlinux.org
> >>>> To subscribe, unsubscribe, or to change your mail settings:
> >>>> https://lists.phxlinux.org/mailman/listinfo/plug-discuss
> >>>>
> >>>
> >>
> >> --
> >> # https://www.SpiralArray.com https://www.PhxLinux.org
> >> # "Luckily, this is a comic book, for which no idea is too complex."
> >> # -- Larry Gonick from The Cartoon History of the United States
> >
>
> --
> # https://www.SpiralArray.com https://www.PhxLinux.org
> # "Lie detector eyeglasses perfected: Civilization collapses."
> # -- Richard Powers
>
>
> ---------------------------------------------------
> PLUG-discuss mailing list: PLUG-discuss@lists.phxlinux.org
> To subscribe, unsubscribe, or to change your mail settings:
> https://lists.phxlinux.org/mailman/listinfo/plug-discuss
>
--
James McPhee
jmcphe@gmail.com
---------------------------------------------------
PLUG-discuss mailing list:
PLUG-discuss@lists.phxlinux.org
To subscribe, unsubscribe, or to change your mail settings:
https://lists.phxlinux.org/mailman/listinfo/plug-discuss
(text/plain)
