使用multipath创建ASM磁盘的奇怪现象

clevernby

操作系统 RHEL 6.2 x86-64
Grid 11.2.0.3.0 Linux x86-64
Oracle 11.2.0.3.0 Linux x86-64
存储设备为 dell eqlogic iscsi

此为新接手的一套RAC环境，使用multipath配置ASM设备，但有些地方很奇怪
ll /dev/mapper/
oracle oinstall asmdisk1 -> ../dm-7
oracle oinstall asmdisk1p1 -> ../dm-11
oracle oinstall asmdisk2 -> ../dm-8
oracle oinstall asmdisk2p2 -> ../dm-12
如上，使用了符合连接
ll /dev/dm*
...
oracle oinstall dm-7
oracle oinstall dm-8
...
oracle oinstall dm-11
oracle oinstall dm-12
...

目前由于iscsi不稳定，经常会断，受其影响/dev/dm设备的ctime会变成当前时间，属组变成root:root，最终导致DBMS异常，比如ORA-01115.

我的问题是multipath配置ASM设备为什么会是符号链接，这是再哪里设置的？

qq7343696

6开始的权限修改可以通过udev来实现，可以不能chown的方式。

Maclean Liu(刘相兵

Linux 6 之后需要用udev 来固化权限

orafans

oracle oinstall asmdisk1 -> ../dm-7这个是正常现象的，Redhat中，内部使用的dm-X来识别多路径的盘的，但是使用的话，是使用的/dev/mapper下的设备。

Maclean Liu(刘相兵

参考这一段：

Setting up Device Mapper Multipath

The purpose of Device Mapper Multipath is to enable multiple I/O paths to improve performance and provide consistent naming. Multipathing accomplishes this by combining your I/O paths into one device mapper path and properly load balancing the I/O. This section will provide the best practices on how to setup your device mapper multipathing within your Dell PowerEdge server.Verify that your device-mapper and multipath driver are at least the version shown below or higher:

rpm -qa | grep device-mapper-multipath

device-mapper-multipath-0.4.9-46.el6

Identify your local disks i.e. /dev/sda. Once your local disk is determined run the following command to obtain its scsi_id:
scsi_id --page=0x83 --whitelisted --device=/dev/sda
360026b900061855e000007a54ea53534

Open the /etc/multipath.conf file and locate and comment out the section below:
#blacklist {
# devnode “*”
#}

Once the scsi_id of your local disk has been retrieved, you must blacklist this scsi_id from being used as a multipath device within /etc/multipath.conf file and locate,uncomment,and modify the section below:
blacklist {

                wwid <enter your local disk scsi_id here>
                devnode "^(ram|raw|loop|fd|md|dm-|sr||scd|st)[0-9]*"
                devnode "^hd[a-z]"
}

Uncomment your defaults section within your /etc/multipath.conf:

defaults {
udev_dir /dev
polling_interval 10
selector "round-robin 0"
path_grouping_policy multibus
getuid_callout "/sbin/scsi_id -g -u -s/block/%n"
prio_callout /bin/true
path_checker readsector0
rr_min_io 100
max_fds 8192
rr_weight priorities
failback immediate
no_path_retry fail
user_friendly_names yes
}

Locate the multipaths section within your /etc/multipath.conf file. In this section you will provide the scsi_id of each LUN/volume and provide an alias in order to keep a consistent naming convention across all of your nodes. An example is shown below:
multipaths {
           multipath {
                            wwid <scsi_id of volume1>
                            alias alias_of_volume1
           }
           multipath {
                            wwid <scsi_id of volume2>
                            alias alias_of_volume2
           }
}

Restart your multipath daemon service using:
service multipathd restart

Verify that your multipath volumes alias are displayed properly:
multipath -ll

Make sure iSCSI service starts upon boot using the command:
chkconfig multipathd on

Repeat steps 1-9 for all nodes.
[top]

Partitioning the Shared Disk

This section describes how to use Linux’s native partition utility fdisk to create  a single partition on a volume/virtual disk that spans the entire disk.

To use the fdisk utility to create a partition:

At the command prompt, type one of the following:
#> fdisk –cu /dev/<block_device>
#> fdisk –cu /dev/mapper/<multipath_disk>
Where, <block_device> is the name of the block device that you are
creating a partition on. For example, if the block device is /dev/sdb, type: fdisk –cu /dev/sdb

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x89058e03.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
The device presents a logical sector size that is smaller than
the physical sector size. Aligning to a physical sector (or optimal
I/O) size boundary is recommended, or performance may be impacted.
Command (m for help): n # To create a new partition
Command actione extendedp primary partition (1-4):
P # To create a primary partition
Partition number (1-4): 1
First sector (4096-xxxxxx, default 4096):

Repeat step 1 for all the disks.

Type the following to re-read the partition table and to be able to see the newly created partition(s)

#> partprobe
Or
#> service multipathd restart
Or
#> kpartx –a /dev/mapper/<multipath_disk>

Reboot the system if your newly created  partition is not
displayed properly.

[top]
Identifying ASM Disks to Setup Udev Rules **Applies to RHEL 6 and Oracle Linux 6 (if not using ASMlib)**

Red Hat Enterprise Linux 6/Oracle Linux 6 have the ability to use udev rules to ensure that the system properly manages permissions of device nodes. In this case, we are referring to properly setting permissions for our LUNs/volumes discovered by the OS. It is important to note that udev rules are executed in enumerated order. When creating udev rules for setting permissions, please include the prefix 20- and append .rules to the end of the filename. An example file name is 20-dell_oracle.rules

In order to set udev rules, one must capture the WWIDs of each disk to be used within your ASM device using the scsi_id command.

The command is as follows:

scsi_id --page=0x83 --whitelisted --device=/dev/sdX

where sdX is the name of your block device.

If one must run this command to capture multiple WWIDs, one could use the following for loop to do just that via the shell:

[root@rhel6 ~]# for i in sdb sdc sdd sde; do \ printf "%s %s\n" "$i" \ "$(scsi_id --page=0x83 --whitelisted --device=/dev/$i)"; done

Output:

sdb 360026b900061855e000008a54ea5356a
sdc 360026b9000618571000008b54ea5360b
sdd 360026b900061855e000008a54ea5356a
sde 360026b9000618571000008b54ea5360b

Once the WWIDs have been captured, create a file within the /etc/udev/rules.d/ directory and name it 20-dell_oracle.rules. A separate KERNEL entry must exist for each storage device and will require adding the WWID to the "RESULT==" field.

An example of what needs to be placed in the /etc/udev/rules.d/20-dell_oracle.rules file

#------------------------ start udev rule contents ------------------#

KERNEL=="dm-*", PROGRAM="scsi_id --page=0x83 --whitelisted --device=/dev/%k",RESULT=="360026b9000618571000008b54ea5360b", OWNER:="grid", GROUP:="asmadmin"

KERNEL=="dm-*", PROGRAM="scsi_id --page=0x83 --whitelisted --device=/dev/%k",RESULT=="360026b900061855e000008a54ea5356a", OWNER:="grid", GROUP:="asmadmin"

#-------------------------- end udev rule contents ------------------#

As you can see from the above, the KERNEL command looks at all dm devices, executes the PROGRAM which captures the WWID, if the RESULT of the WWID is matched, appropriately assign the grid user as the OWNER and the asmadmin group as the GROUP.

abao2000521

Configuring non-raw Multipath Devices for Oracle Clusterware 11g (11.1.0, 11.2.0) on RHEL6/OL6－－看一下这个文章

xifenfei

你的有两个问题：
1. 需要使用/dev/mapper下面设备
2. 权限不能直接这样授权，需要使用udev或者rc.local 或者md本身的权限设置（参考http://www.xifenfei.com/5066.html）

harryzhang

手动chmod修改权限要小心，因为 udevd 后台进程若在的情况下，可能2分钟后，权限又修改回去了。
so，最好还是udev来搞~