Linux – dynamically add/remove scsi from linux1. SCSI Device Addressing
A four-part addressing scheme is used to define the location of SCSI devices within a system. The attributes include:
<H>ost: Instance of hostadapter to which device is attached
<B>us: SCSI Bus or Channel on the hostadapter
<T>arget: SCSI Id assigned to an individual device
<L>un: Logical unit number on the device
Each attribute, <H> <B> <T> <L>, refers to a part of the device location, similar to how number, street, suburb and state all form an address.
References to device addresses are readily visible from system logs and various command output, though the availability of certain commands or utilities depends on the distribution and operating system version used. The lsscsi(8) utility, for example, is natively available on Enterprise Linux 5.
The following denotes the use of several commands to describe a relatively simple SCSI system.
# dmesg
...
SCSI subsystem initialized
libata version 2.21 loaded.
ata_piix 0000:00:07.1: version 2.12
scsi0 : ata_piix
scsi1 : ata_piix
ata1: PATA max UDMA/33 cmd 0x000101f0 ctl 0x000103f6 bmdma 0x0001ffa0 irq 14
ata2: PATA max UDMA/33 cmd 0x00010170 ctl 0x00010376 bmdma 0x0001ffa8 irq 15
ata1.00: ATA-5: IC35L040AVVA07-0, VA2OA51A, max UDMA/100
ata1.00: 78165360 sectors, multi 8: LBA
ata1.01: ATA-5: WDC WD400BB-32CLB0, 05.04E05, max UDMA/100
ata1.01: 78165360 sectors, multi 8: LBA
ata1.00: configured for UDMA/33
ata1.01: configured for UDMA/33
ata2.00: ATA-5: QUANTUM FIREBALLP AS60.0, A1Y.1500, max UDMA/100
ata2.00: 117266688 sectors, multi 8: LBA
ata2.01: ATAPI: JLMS DVD-ROM XJ-HD166, DD05, max UDMA/33
ata2.00: configured for UDMA/33
ata2.01: configured for UDMA/33
scsi 0:0:0:0: Direct-Access ATA IC35L040AVVA07-0 VA2O PQ: 0 ANSI: 5
sd 0:0:0:0: [sda] 78165360 512-byte hardware sectors (40021 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 0:0:0:0: [sda] 78165360 512-byte hardware sectors (40021 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sda: sda1 sda2 sda3
sd 0:0:0:0: [sda] Attached SCSI disk
scsi 0:0:1:0: Direct-Access ATA WDC WD400BB-32CL 05.0 PQ: 0 ANSI: 5
sd 0:0:1:0: [sdb] 78165360 512-byte hardware sectors (40021 MB)
sd 0:0:1:0: [sdb] Write Protect is off
sd 0:0:1:0: [sdb] Mode Sense: 00 3a 00 00
sd 0:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 0:0:1:0: [sdb] 78165360 512-byte hardware sectors (40021 MB)
sd 0:0:1:0: [sdb] Write Protect is off
sd 0:0:1:0: [sdb] Mode Sense: 00 3a 00 00
sd 0:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sdb: sdb1 sdb2
sd 0:0:1:0: [sdb] Attached SCSI disk
scsi 1:0:0:0: Direct-Access ATA QUANTUM FIREBALL A1Y. PQ: 0 ANSI: 5
sd 1:0:0:0: [sdc] 117266688 512-byte hardware sectors (60041 MB)
sd 1:0:0:0: [sdc] Write Protect is off
sd 1:0:0:0: [sdc] Mode Sense: 00 3a 00 00
sd 1:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 1:0:0:0: [sdc] 117266688 512-byte hardware sectors (60041 MB)
sd 1:0:0:0: [sdc] Write Protect is off
sd 1:0:0:0: [sdc] Mode Sense: 00 3a 00 00
sd 1:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sdc: sdc1
sd 1:0:0:0: [sdc] Attached SCSI disk
scsi 1:0:1:0: CD-ROM JLMS DVD-ROM XJ-HD166 DD05 PQ: 0 ANSI: 5
...
# lspci | grep -i ide
00:07.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)
# lsscsi
[0:0:0:0] disk ATA IC35L040AVVA07-0 VA2O /dev/sda
[0:0:1:0] disk ATA WDC WD400BB-32CL 05.0 /dev/sdb
[1:0:0:0] disk ATA QUANTUM FIREBALL A1Y. /dev/sdc
[1:0:1:0] cd/dvd JLMS DVD-ROM XJ-HD166 DD05 /dev/sr0
# cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: ATA Model: IC35L040AVVA07-0 Rev: VA2O
Type: Direct-Access ANSI SCSI revision: 05
Host: scsi0 Channel: 00 Id: 01 Lun: 00
Vendor: ATA Model: WDC WD400BB-32CL Rev: 05.0
Type: Direct-Access ANSI SCSI revision: 05
Host: scsi1 Channel: 00 Id: 00 Lun: 00
Vendor: ATA Model: QUANTUM FIREBALL Rev: A1Y.
Type: Direct-Access ANSI SCSI revision: 05
Host: scsi1 Channel: 00 Id: 01 Lun: 00
Vendor: JLMS Model: DVD-ROM XJ-HD166 Rev: DD05
Type: CD-ROM ANSI SCSI revision: 05
# grep host /etc/modprobe.conf
alias scsi_hostadapter ata_piix
# ls -ld /sys/class/scsi_host/host*/
drwxr-xr-x 2 root root 0 2008-08-06 17:25 /sys/class/scsi_host/host0/
drwxr-xr-x 2 root root 0 2008-08-06 17:25 /sys/class/scsi_host/host1/
# cat /sys/class/scsi_host/host[0-1]/proc_name
ata_piix
ata_piix
Above, it’s evident that four devices are attached to two hostadapters, both of type ata_piix i.e. host0: sda sdb, host1: sdc sr0.
Note too, there is sufficient overlap between various command output that device addresses and naming can be easily identified and mapped.
2. SCSI Device Naming
The name assigned to a SCSI device is completely independent to it’s SCSI address. In fact, taking the linux 2.6 kernel as an example, the device naming system used, udev(8), dynamically allocates device names upon each boot.
During system initialisation, hardware is scanned and devices are named according to their discovery order. This means, however, that the same device may not always be assigned the same name. This may have implications on some systems, especially those that solely rely on device names such as file /etc/fstab for mounting filesystems.
For such cases, explicit udev(8) configuration may be required to guarantee cross-reboot persistency of device naming. Alternatively, other methods, such as mount-by-label (where supported), may be employed to ensure that only intended devices are selected and used, regardless of their arbitrary device name. udev(8), though mentioned here, is not described in any detail – it’s relevance, however, becomes more apparent later. Refer to the udev(8) and mount(8) man pages and references below for more information.
3. Scanning, Adding and Removing Devices
From time to time, it may be necessary to add, remove, replace or even reorganise SCSI devices in a system. Broadly speaking, there are two main approaches to how this can be achieved – offline and online.
The method one might choose to add or remove devices usually depends on several factors, such as:
distribution, operating system and version
hardware type, driver version/capability
system availability requirements
storage availability requirements
availability of backup/redundant systems
familiarity of system, I/O stack and storage
acceptance of associated risk
Following are several methods of device addition and removal. Regardless of which method you use, always ensure to perform thorough testing before use within a production environment.
3.1 System reboot
Adding and removing devices when a system is shutdown is considered the simplest and safest method. Clean filesystem unmount and ordered shutdown of all of components/drivers involved in the I/O path avoids the potential risks associated with online (dynamic) device removal. On boot, ordered driver load allows newly added or removed devices to be correctly discovered and identified. This method, obviously, necessitates total system unavailability.
3.2 Reload hostadapter driver
The installation (loading) of common host bus adapter (HBA) driver modules initiates a scan (or rescan) of the associated device, resulting in the (re)discovery of any newly added/removed devices. Whilst performed with the system online, to be able to reload the hostadapter module, it must first be unloaded. This, therefore, means that any (all) filesystem, volume or array on devices associated with the driver must first be offlined (in the case of arrays or logical volumes) and/or unmounted. In fact, depending on the complexity of the I/O stack involved, this method is likely to also require the shutdown of related storage services and the unloading of other related or dependent modules e.g. multipathing.
3.3 procfs /proc/scsi/scsi (2.4 kernel)
Linux provides the ability to dynamically interact with the running kernel via the /proc (procfs) filesystem. Dynamically adding or removing devices can be accomplished via the /proc/scsi/scsi interface i.e.:
To remove a specific device:
# echo "scsi remove-single-device <H> <B> <T> <L>" > /proc/scsi/scsi
where <H> <B> <T> <L> refers to Host, Bus, Target and Lun
To add a specific device:
# echo "scsi add-single-device <H> <B> <T> <L>" > /proc/scsi/scsi
where <H> <B> <T> <L> refers to Host, Bus, Target and Lun
This method allows for all filesytems, volumes and arrays, except those immediately involved in specific device removal, to remain online and mounted. However, this method carries a high element of risk. Strong knowledge of the system and all storage layers/devices involved in the I/O path is required. Removal of an incorrect device or premature removal of an intended device may result in volume corruption or, in the presence of Clusterware (e.g. Oracle Cluster File System 2 (OCFS2) or Real Application Clusters (RAC)), could result in node eviction.
Use this method as required if using a 2.4 kernel. The 2.6 kernel provides an improved sysfs interface (/sys, described below) for managing devices.
The following example illustrates the removal, then addition of a SCSI device:
# lsscsi
[0:0:0:0] disk ATA IC35L040AVVA07-0 VA2O /dev/sda
[0:0:1:0] disk ATA WDC WD400BB-32CL 05.0 /dev/sdb
[1:0:0:0] disk ATA QUANTUM FIREBALL A1Y. /dev/sdc
[1:0:1:0] cd/dvd JLMS DVD-ROM XJ-HD166 DD05 /dev/sr0
# echo "scsi remove-single-device 1 0 0 0" > /proc/scsi/scsi
# lsscsi
[0:0:0:0] disk ATA IC35L040AVVA07-0 VA2O /dev/sda
[0:0:1:0] disk ATA WDC WD400BB-32CL 05.0 /dev/sdb
[1:0:1:0] cd/dvd JLMS DVD-ROM XJ-HD166 DD05 /dev/sr0
# echo "scsi add-single-device 1 0 0 0" > /proc/scsi/scsi
# lsscsi
[0:0:0:0] disk ATA IC35L040AVVA07-0 VA2O /dev/sda
[0:0:1:0] disk ATA WDC WD400BB-32CL 05.0 /dev/sdb
[1:0:0:0] disk ATA QUANTUM FIREBALL A1Y. /dev/sdd
[1:0:1:0] cd/dvd JLMS DVD-ROM XJ-HD166 DD05 /dev/sr0
Note that the removed device, originally known as /dev/sdc, was added back to the system, but as /dev/sdd. This highlights the potential risk associated with solely relying on arbitrary kernel-assigned device file names.
3.4 sysfs /sys/class/scsi_host/ (2.6 kernel)
The 2.6 kernel provides the /sys (sysfs) interface for interacting and managing system devices. In the case of SCSI devices, the /sys/class/scsi_host/ interface can be used to dynamically rescan a hostadapter, as well as add or remove specific devices.
To rescan a hostadapter:
# echo '- - -' > /sys/class/scsi_host/host<H>/scan
where <H> refers to the hostadapter or the instance of hostadapter where multiple (of the same type) exist on the system
To remove a specific device:
# echo 1 > /sys/class/scsi_host/host<H>/device/target<H>:<B>:<T>/<H>:<B>:<T>:<L>/delete
where <H> <B> <T> <L> refers to Host, Bus, Target and Lun
To add a specific device:
# echo "<B> <T> <L>" > /sys/class/scsi_host/host<H>/scan
where <H> <B> <T> <L> refers to Host, Bus, Target and Lun
Like the /proc/scsi/scsi interface, the /sys/class/scsi_host/ interface similarly allows for all filesytems, volumes and arrays, except those immediately involved in specific device removal, to remain online and mounted. Again, this method carries a high level of risk and requires a strong knowledge of the system and all storage layers/devices involved in the I/O path.
Where Fiber Channel (FC) Host Bus Adapters (HBA) are used, separate procfs and/or sysfs entries are created in various locations depending on Operating System and driver type and version used. In such cases, FC HBA driver level re-scan should precede SCSI Bus rescan. For example, for QLogic (qla2xxx):
Enterprise Linux 4:
# echo "scsi-qlascan" >> /proc/scsi/qla2xxx/<H>
# echo "scsi add-single-device <H> <B> <T> <L>" > /proc/scsi/scsi
Enterprise Linux 5:
# echo 1 > /sys/class/fc_host/host<H>/issue_lip
# echo '- - -' > /sys/class/scsi_host/host<H>/scan
The following example illustrates (non-FC HBA) SCSI hostadapter rescan, then removal and addition of a device:
# echo '- - -' > /sys/class/scsi_host/host1/scan
# lsscsi
[0:0:0:0] disk ATA IC35L040AVVA07-0 VA2O /dev/sda
[0:0:1:0] disk ATA WDC WD400BB-32CL 05.0 /dev/sdb
[1:0:0:0] disk ATA QUANTUM FIREBALL A1Y. /dev/sdc
[1:0:1:0] cd/dvd JLMS DVD-ROM XJ-HD166 DD05 /dev/sr0
[root@toxic ~]# echo 1 > /sys/class/scsi_host/host1/device/target1:0:0/1:0:0:0/delete
# lsscsi
[0:0:0:0] disk ATA IC35L040AVVA07-0 VA2O /dev/sda
[0:0:1:0] disk ATA WDC WD400BB-32CL 05.0 /dev/sdb
[1:0:1:0] cd/dvd JLMS DVD-ROM XJ-HD166 DD05 /dev/sr0
[root@toxic /]# echo '0 0 0' > /sys/class/scsi_host/host1/scan
# lsscsi
[0:0:0:0] disk ATA IC35L040AVVA07-0 VA2O /dev/sda
[0:0:1:0] disk ATA WDC WD400BB-32CL 05.0 /dev/sdb
[1:0:0:0] disk ATA QUANTUM FIREBALL A1Y. /dev/sdd
[1:0:1:0] cd/dvd JLMS DVD-ROM XJ-HD166 DD05 /dev/sr0
Once again, note that the name of the re-added device (/dev/sdd) differs to it’s original name (/dev/sdc).
The entries and names beneath /sys/class/scsi_host/ may vary depending on the operating system, kernel version and type of devices in use on the system. For example, if using iSCSI devices, additional directory entry session exists:
# lsscsi
[0:0:0:0] disk ATA WDC WD1600JS-75N 10.0 /dev/sda
[2:0:0:0] disk IET VIRTUAL-DISK 0 /dev/sdb
[3:0:0:0] disk IET VIRTUAL-DISK 0 /dev/sdc
[7:0:0:0] disk IET VIRTUAL-DISK 0 /dev/sdd
[10:0:0:0] disk IET VIRTUAL-DISK 0 /dev/sde
# ls -ld /sys/class/scsi_host/host*/device/session*/target*/[0-9]*
drwxr-xr-x 3 root root 0 Aug 7 22:35 /sys/class/scsi_host/host10/device/session8/target10:0:0/10:0:0:0
drwxr-xr-x 3 root root 0 Aug 7 22:35 /sys/class/scsi_host/host2/device/session0/target2:0:0/2:0:0:0
drwxr-xr-x 3 root root 0 Aug 7 22:35 /sys/class/scsi_host/host3/device/session1/target3:0:0/3:0:0:0
drwxr-xr-x 3 root root 0 Aug 7 22:35 /sys/class/scsi_host/host7/device/session5/target7:0:0/7:0:0:0
3.5 Host Adapter Vendor Supplied Scripts
Some hostadapter vendors supply their own scripts that can be used to scan, add and remove devices. These scripts may be used in favour of manual device interaction via native kernel interfaces. The scripts should, however, be thoroughly tested before reliance in production. Naturally, where provided by a third-party vendor, any issues arising from their use should be referred to the originating supplier.
Examples of vendor supplied scripts include:
ql-dynamic-tgt-lun-disc.sh (QLogic)
rescan-scsi-bus.sh (QLogic)
qlun_disc.sh (QLogic)
lun_scan (Emulex)
hp_rescan (HP)
proprietary and custom others
Contact your hostadapter vendor for latest available and recommended scripts.
4. Removing a Multipath Device by Example
Let’s look at a more complex, yet typical scenario involving multipathing (device-mapper-multipath).
In this instance, explicitly white-listed iSCSI-served partitioned target devices (LUNs) are multipathed on the initiator using user-defined names; ocr1, voting1, etc.
# dmsetup ls | sort
ocr1 (253, 5)
ocr1p1 (253, 9)
ocr2 (253, 6)
ocr2p1 (253, 10)
ocr3 (253, 7)
ocr3p1 (253, 11)
voting1 (253, 0)
voting1p1 (253, 3)
voting2 (253, 1)
voting2p1 (253, 4)
voting3 (253, 2)
voting3p1 (253, 8)
# multipath -ll
ocr3 (149455400000000000000000001000000ca0200000d000000) dm-7 IET,VIRTUAL-DISK
[size=980M][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
\_ 1:0:0:10 sdn 8:208 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 1:0:0:11 sdo 8:224 [active][ready]
ocr2 (149455400000000000000000001000000ed0200000d000000) dm-6 IET,VIRTUAL-DISK
[size=980M][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
\_ 1:0:0:8 sdl 8:176 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 1:0:0:9 sdm 8:192 [active][ready]
ocr1 (149455400000000000000000001000000e80200000d000000) dm-5 IET,VIRTUAL-DISK
[size=980M][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
\_ 1:0:0:6 sdj 8:144 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 1:0:0:7 sdk 8:160 [active][ready]
voting3 (149455400000000000000000001000000e30200000d000000) dm-2 IET,VIRTUAL-DISK
[size=965M][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
\_ 1:0:0:4 sdh 8:112 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 1:0:0:5 sdi 8:128 [active][ready]
voting2 (149455400000000000000000001000000de0200000d000000) dm-1 IET,VIRTUAL-DISK
[size=965M][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
\_ 1:0:0:2 sdf 8:80 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 1:0:0:3 sdg 8:96 [active][ready]
voting1 (149455400000000000000000001000000d90200000d000000) dm-0 IET,VIRTUAL-DISK
[size=965M][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
\_ 1:0:0:1 sde 8:64 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 1:0:0:0 sdd 8:48 [active][ready]
# cat /proc/partitions
major minor #blocks name
8 0 6291456 sda
8 1 5735173 sda1
8 2 554242 sda2
8 16 2097152 sdb
8 17 2096451 sdb1
8 32 2097152 sdc
8 33 2096451 sdc1
8 48 987966 sdd
8 49 987681 sdd1
8 64 987966 sde
8 65 987681 sde1
8 80 987966 sdf
8 81 987681 sdf1
8 96 987966 sdg
8 97 987681 sdg1
8 112 987966 sdh
8 113 987681 sdh1
8 128 987966 sdi
8 129 987681 sdi1
8 144 1004031 sdj
8 145 1003873 sdj1
8 160 1004031 sdk
8 161 1003873 sdk1
8 176 1004031 sdl
8 177 1003873 sdl1
8 192 1004031 sdm
8 193 1003873 sdm1
8 208 1004031 sdn
8 209 1003873 sdn1
8 224 1004031 sdo
8 225 1003873 sdo1
253 0 987966 dm-0
253 1 987966 dm-1
253 2 987966 dm-2
253 3 987681 dm-3
253 4 987681 dm-4
253 5 1004031 dm-5
253 6 1004031 dm-6
253 7 1004031 dm-7
253 8 987681 dm-8
253 9 1003873 dm-9
253 10 1003873 dm-10
253 11 1003873 dm-11
Below, unused multipath device ocr3 is dynamically removed from the system, as are its associated underlying devices. The devices were verified to no longer be in use or required by any program or service before their removal.
Note, whether the /sbin/multipath or /sbin/dmsetup command is used, the result is the same. However, when using the dmsetup command to remove partitioned multipathed devices, the multipath aliases for all of its partitions must first be removed before the multipath alias of the device itself can be removed, otherwise the command fails with a ‘device or resource busy error’ message. When using the multipath command to remove partitioned devices, removal of the multipath alias of the device will automatically remove all multipath aliases for all of its partitions.
# multipath -ll ocr3
ocr3(149455400000000000000000001000000ca0200000d000000) dm-7 IET,VIRTUAL-DISK
[size=980M][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
\_ 1:0:0:10 sdn 8:208 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 1:0:0:11 sdo 8:224 [active][ready]
# dmsetup ls | sort
ocr1 (253, 5)
ocr1p1 (253, 9)
ocr2 (253, 6)
ocr2p1 (253, 10)
ocr3 (253, 7)
ocr3p1 (253, 11)
voting1 (253, 0)
voting1p1 (253, 3)
voting2 (253, 1)
voting2p1 (253, 4)
voting3 (253, 2)
voting3p1 (253,
# cat /proc/partitions | grep -e 'sdo\|sdn\|dm-7\|dm-11'
8 208 1004031 sdn
8 209 1003873 sdn1
8 224 1004031 sdo
8 225 1003873 sdo1
253 7 1004031 dm-7
253 11 1003873 dm-11
# multipath -f ocr3
OR
# dmsetup remove ocr3p1
# dmsetup remove ocr3
# multipath -ll ocr3
#
# dmsetup ls | sort
ocr1 (253, 5)
ocr1p1 (253, 9)
ocr2 (253, 6)
ocr2p1 (253, 10)
voting1 (253, 0)
voting1p1 (253, 3)
voting2 (253, 1)
voting2p1 (253, 4)
voting3 (253, 2)
voting3p1 (253,
# cat /proc/partitions | grep -e 'sdo\|sdn\|dm-7\|dm-11'
8 208 1004031 sdn
8 209 1003873 sdn1
8 224 1004031 sdo
8 225 1003873 sdo1
# echo 1 > /sys/class/scsi_host/host1/device/session0/target1:0:0/1:0:0:10/delete
# echo 1 > /sys/class/scsi_host/host1/device/session0/target1:0:0/1:0:0:11/delete
# cat /proc/partitions | grep -e 'sdo\|sdn\|dm-7\|dm-11'
#
Note that the device-mapper devices previously associated with multipath device ocr3 (/dev/dm-7, /dev/dm-11) are removed, however the underlying device paths (/dev/sdn, /dev/sdo) remain until explicitly removed from the system.
