Oracle 10g RAC Installation on NetApps Raw Devices with ASM: Part 3

Installing and Configuring Oracle ASMlib

Oracle ASMlib is software provided to ease the configuration of Disk Devices.

Please download the library files from http://www.oracle.com/technology/tech/linux/asmlib/index.html

The library are available with respect to your OS and OS version. The current version of ASMlib is 2.0. With respect to my operating system I downloaded:

oracleasm-support-2.0.3-1.x86_64.rpm

oracleasmlib-2.0.2-1.x86_64.rpm

oracleasm-2.6.9-55.ELsmp-2.0.3-1.x86_64.rpm

oracleasm-2.6.9-55.EL-2.0.3-1.x86_64.rpm

Installation of these binaries:

rpm –Uvh oracleasm-support-2.0.3-1.x86_64.rpm

rpm –Uvh oracleasm-2.6.9-55.EL-2.0.3-1.x86_64.rpm

rpm –Uvh oracleasmlib-2.0.2-1.x86_64.rpm

rpm –Uvh oracleasm-2.6.9-55.ELsmp-2.0.3-1.x86_64.rpm

Install these binaries on all RAC nodes.

Initialize oracleasm:

# /etc/init.d/oracleasm configure

Default user to own the driver interface []: oracle

Default group to own the driver interface []: dba

Start Oracle ASM library driver on boot (y/n) [n]: y

Fix permissions of Oracle ASM disks on boot (y/n) [y]

…

Note: Please make sure to enter default user and group to own the driver interface.

Creation of Disk Partitions and Oracle ASM Disk:

We are using 1 lun of 2GB to create partitions. We will create partitions for OCR, Voting Disk, and other database files.

You can also create separate luns for OCR, Voting Disk and DB files.

As we are using iSCSI devices we do not need to create disk partitions we can directly use /dev/sd* devices or as we are using multipath devices, we can directly use /dev/mpath/mpath*

For example sake I am creating disk partitions for this installation:

[root@oradbdev01 mpath]# fdisk /dev/dm-8 à LUN (multipath device)

Command (m for help): p

Disk /dev/dm-8: 2147 MB, 2147483648 bytes

67 heads, 62 sectors/track, 1009 cylinders

Units = cylinders of 4154 * 512 = 2126848 bytes

Device Boot Start End Blocks Id System

Command (m for help): n

Command action

e extended

p primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-1009, default 1):

Using default value 1

Last cylinder or +size or +sizeM or +sizeK (1-1009, default 1009): +128M àOCR FILE

Command (m for help): p

Disk /dev/dm-8: 2147 MB, 2147483648 bytes

67 heads, 62 sectors/track, 1009 cylinders

Units = cylinders of 4154 * 512 = 2126848 bytes

Device Boot Start End Blocks Id System

/dev/dm-8p1 1 61 126666 83 Linux

Command (m for help): n

Command action

e extended

p primary partition (1-4)

p

Partition number (1-4): 2

First cylinder (62-1009, default 62):

Using default value 62

Last cylinder or +size or +sizeM or +sizeK (62-1009, default 1009): +128M àOCR FILE

Command (m for help): p

Disk /dev/dm-8: 2147 MB, 2147483648 bytes

67 heads, 62 sectors/track, 1009 cylinders

Units = cylinders of 4154 * 512 = 2126848 bytes

Device Boot Start End Blocks Id System

/dev/dm-8p1 1 61 126666 83 Linux

/dev/dm-8p2 62 122 126697 83 Linux

Command (m for help): n

Command action

e extended

p primary partition (1-4)

p

Partition number (1-4): 3

First cylinder (123-1009, default 123):

Using default value 123

Last cylinder or +size or +sizeM or +sizeK (123-1009, default 1009): +1000M àData File

Command (m for help): p

Disk /dev/dm-8: 2147 MB, 2147483648 bytes

67 heads, 62 sectors/track, 1009 cylinders

Units = cylinders of 4154 * 512 = 2126848 bytes

Device Boot Start End Blocks Id System

/dev/dm-8p1 1 61 126666 83 Linux

/dev/dm-8p2 62 122 126697 83 Linux

/dev/dm-8p3 123 593 978267 83 Linux

Command (m for help): n

Command action

e extended

p primary partition (1-4)

e

Selected partition 4

First cylinder (594-1009, default 594):

Using default value 594

Last cylinder or +size or +sizeM or +sizeK (594-1009, default 1009):

Using default value 1009 à Creating extended partition of the remaing space.

Command (m for help): p

Disk /dev/dm-8: 2147 MB, 2147483648 bytes

67 heads, 62 sectors/track, 1009 cylinders

Units = cylinders of 4154 * 512 = 2126848 bytes

Device Boot Start End Blocks Id System

/dev/dm-8p1 1 61 126666 83 Linux

/dev/dm-8p2 62 122 126697 83 Linux

/dev/dm-8p3 123 593 978267 83 Linux

/dev/dm-8p4 594 1009 864032 5 Extended

Command (m for help): n

First cylinder (594-1009, default 594):

Using default value 594

Last cylinder or +size or +sizeM or +sizeK (594-1009, default 1009): +20M àVoting Disk

Command (m for help): n

First cylinder (604-1009, default 604):

Using default value 604

Last cylinder or +size or +sizeM or +sizeK (604-1009, default 1009): +20M àVoting Disk

Command (m for help): n

First cylinder (614-1009, default 614): +20M

Value out of range.

First cylinder (614-1009, default 614):

Using default value 614

Last cylinder or +size or +sizeM or +sizeK (614-1009, default 1009): +20M àVoting Disk

Command (m for help): n

First cylinder (624-1009, default 624):

Using default value 624

Last cylinder or +size or +sizeM or +sizeK (624-1009, default 1009):

Using default value 1009 à Remaining space for Database files

Command (m for help): p

Disk /dev/dm-8: 2147 MB, 2147483648 bytes

67 heads, 62 sectors/track, 1009 cylinders

Units = cylinders of 4154 * 512 = 2126848 bytes

Device Boot Start End Blocks Id System

/dev/dm-8p1 1 61 126666 83 Linux àOCR FILE

/dev/dm-8p2 62 122 126697 83 Linux àOCR FILE

/dev/dm-8p3 123 593 978267 83 Linux àDatabase

/dev/dm-8p4 594 1009 864032 5 Extended

/dev/dm-8p5 594 603 20739 83 Linux àVoting Disk

/dev/dm-8p6 604 613 20739 83 Linux àVoting Disk

/dev/dm-8p7 614 623 20739 83 Linux àVoting Disk

/dev/dm-8p8 624 1009 801691 83 Linux àDatabase

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 22: Invalid argument.

The kernel still uses the old table.

The new table will be used at the next reboot.

Syncing disks.

Reboot all nodes to read the partition table.

The devices will look like this:

[oracle@oradbdev01 ~]$ ls -lrt /dev/mpath/mpath8*

lrwxrwxrwx 1 root root 7 Apr 10 15:02 /dev/mpath/mpath8 -> ../dm-0

lrwxrwxrwx 1 root root 7 Apr 10 15:02 /dev/mpath/mpath8p2 -> ../dm-4

lrwxrwxrwx 1 root root 7 Apr 10 15:02 /dev/mpath/mpath8p6 -> ../dm-8

lrwxrwxrwx 1 root root 7 Apr 10 15:02 /dev/mpath/mpath8p1 -> ../dm-2

lrwxrwxrwx 1 root root 7 Apr 10 15:02 /dev/mpath/mpath8p5 -> ../dm-7

lrwxrwxrwx 1 root root 7 Apr 10 15:02 /dev/mpath/mpath8p7 -> ../dm-9

lrwxrwxrwx 1 root root 8 Apr 10 15:02 /dev/mpath/mpath8p8 -> ../dm-10

lrwxrwxrwx 1 root root 7 Apr 10 15:02 /dev/mpath/mpath8p3 -> ../dm-6

Now add database disk devices to oracle asm:

[root@oradbdev01 mpath]# /etc/init.d/oracleasm createdisk DISK3 /dev/mpath/mpath8p3;

Marking disk “/dev/mpath/mpath8p3″ as an ASM disk: [ OK ]

[root@oradbdev01 mpath]# /etc/init.d/oracleasm createdisk DISK4 /dev/mpath/mpath8p8;

Marking disk “/dev/mpath/mpath8p8″ as an ASM disk: [ OK ]

Run the command “/etc/init.d/oracleasm scandisks” on all other nodes so that ASM disks will show up on all other nodes.

[root@oradbdev02 ~]# /etc/init.d/oracleasm scandisks

Scanning system for ASM disks: [ OK ]

Run the command /etc/init.d/oracleasm listdisks and /etc/init.d/oracleasm querydisk on all nodes to verify the disks has been marked as ASM disks. For example:

[oracle@oradbdev01 ~]$ ssh oradbdev02 /etc/init.d/oracleasm listdisks

DISK1

DISK2

DISK3

DISK4

[oracle@oradbdev01 ~]$ /etc/init.d/oracleasm querydisk DISK3

Disk “DISK3″ is a valid ASM disk on device [8, 19]

[oracle@oradbdev01 ~]$ ssh oradbdev02 /etc/init.d/oracleasm querydisk DISK3

Disk “DISK3″ is a valid ASM disk on device [8, 19]

[oracle@oradbdev01 ~]$ /etc/init.d/oracleasm querydisk DISK4

Disk “DISK4″ is a valid ASM disk on device [8, 24]

[oracle@oradbdev01 ~]$ ssh oradbdev02 /etc/init.d/oracleasm querydisk DISK4

Disk “DISK4″ is a valid ASM disk on device [8, 24]

[root@oradbdev01 ~]# /etc/init.d/oracleasm querydisk /dev/mpath/mpath8p3

Disk “/dev/mpath/mpath8p3″ is marked an ASM disk with the label “DISK3″

[root@oradbdev01 ~]# /etc/init.d/oracleasm querydisk /dev/mpath/mpath8p8

Disk “/dev/mpath/mpath8p8″ is marked an ASM disk with the label “DISK4″

To find out the exact disk reflected in the output of oracleasm, use the entries in /proc/partitions file.

Configuring ASMlib on Multipath Disk:

[root@oradbdev01 ~]# vi /etc/sysconfig/oracleasm

#

# This is a configuration file for automatic loading of the Oracle

# Automatic Storage Management library kernel driver. It is generated

# By running /etc/init.d/oracleasm configure. Please use that method

# to modify this file

#

# ORACLEASM_ENABELED: ‘true’ means to load the driver on boot.

ORACLEASM_ENABLED=true

# ORACLEASM_UID: Default user owning the /dev/oracleasm mount point.

ORACLEASM_UID=oracle

# ORACLEASM_GID: Default group owning the /dev/oracleasm mount point.

ORACLEASM_GID=dba

# ORACLEASM_SCANBOOT: ‘true’ means fix disk perms on boot

ORACLEASM_SCANBOOT=true

# ORACLEASM_CLEARBOOT: ‘true’ means clean old disk perms on boot

ORACLEASM_CLEARBOOT=true

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning

ORACLEASM_SCANORDER=”dm” ##Updated it to “dm”

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan

ORACLEASM_SCANEXCLUDE=”sd” ##Updated it to “sd”

Make these changes in all the nodes.

Preparing & Configuring OCR and Voting Disk on Raw devices using Multipath or Udev:

On Linux platform, the LUNs presented from NAS or SCSI devices are identified with the device name (or path) /dev/sdx (/dev/sda, dev/sdb, etc). A common issue is that the device names (including multipath device name like /dev/mpath/mpath0 if multipath is enabled) can be changed when some disks are removed or added (as planned or due to I/O failures), SCSI or FC is reconfigured (like adding or removing ISCSI channels) or even host is rebooted. Even these devices are bound to raw with the consistent name (like /dev/raw/raw1) and assigned to OCR and voting disks, starting CRS can still fail because of the content of the device is not the OCR or the voting disk. To address this issue, you can use either multipath software or udev utility to setup devices with consistent name for OCR and voting disks.

multipath – multipath is a native Linux dynamic multipathing/failover software. An I/O path generally consists of an initiator port, target port, and LUN. Each permutation of this I/O is considered an independent path. Dynamic multipathing/failover tools (software or hardware) aggregate these independent paths into a single logical path. This path virtualization provides I/O load-balancing and non-disruptive failover across the host bus adapters (HBAs) or network interface card (NIC). Multipath software also can assign a consistent device name to LUNs as described in next section.

More info: http://en.wikipedia.org/wiki/Multipath_I/O

udev – udev is a new feature introduced in Linux 2.6. It provides a dynamic device directory containing only the files for actually present devices. It creates or removes device node files usually located in the /dev directory, or it renames network interfaces. As part of the hotplug subsystem, udev is executed if a kernel device is added or removed from the system. On device creation, udev reads the sysfs directory of the given device to collect device attributes like label, serial number or bus device number. These attributes may be used as keys to determine a unique name for the device.

More info: http://en.wikipedia.org/wiki/Udev

Configuring OCR & Voting Disk:

1. Choose the multipath device which will hold the ocr & voting disk files.

e.g /dev/mpath/mpath8

[root@oradbdev01 dev]# multipath -d -l mpath8

mpath8 (360a9800043346d77674a486332443238)

[size=2 GB][features="1 queue_if_no_path"][hwhandler="0"]

\_ round-robin 0 [active]

\_ 0:0:0:1 sdb 8:16 [active]

2. Rediscover all multipath devices on all nodes:

Note: Please shutdown all databases and ASM running on the affect nodes, as this will remove all multipath devices.

[root@oradbdev01 dev]# multipath –F

[root@oradbdev01 dev]# multipath –v2

As we have already created paritions, for probing these new partition devices on all the other nodes run:

[root@oradbdev02 dev]# partprobe

Instead you can also do:

/etc/init.d/iscsi reload

3. Make these changes to the /etc/sysconfig/rawdevices file, these are for OCR & Voting Disk.

[root@oradbdev01 dev]# more /etc/sysconfig/rawdevices

# This file and interface are deprecated.

# Applications needing raw device access should open regular

# block devices with O_DIRECT.

# raw device bindings

# format:

#

# example: /dev/raw/raw1 /dev/sda1

# /dev/raw/raw2 8 5

/dev/raw/raw1 /dev/mpath/mpath8p1

/dev/raw/raw2 /dev/mpath/mpath8p2

/dev/raw/raw3 /dev/mpath/mpath8p3

/dev/raw/raw8 /dev/mpath/mpath8p8

/dev/raw/raw4 /dev/mpath/mpath8p5

/dev/raw/raw5 /dev/mpath/mpath8p6

/dev/raw/raw6 /dev/mpath/mpath8p7

4. Bind the disk devices to the raw devices on all nodes as follow:

/sbin/service rawdevices restart

5. Run the following command as root on all nodes so that system automatically binds the devices listed in the rawdevices file when it reboots.

/sbin/chkconfig –level 345 rawdevices on

6. Setup the owner, group and permissions on the OCR and voting disks as root on all nodes as follow:

For OCR:

chown root:dba /dev/raw/raw1

chown root:dba /dev/raw/raw2

chmod 660 /dev/raw/raw1

chmod 660 /dev/raw/raw2

For voting disks:

chown oracle:dba /dev/raw/raw4

chown oracle:dba /dev/raw/raw5

chown oracle:dba /dev/raw/raw6

chmod 660 /dev/raw/raw4

chmod 660 /dev/raw/raw5

chmod 660 /dev/raw/raw6

7. For RHEL4, add the following lines in /etc/rc.d/rc.local as root on all nodes so that system will setup the owner, group and permissions properly after reboot.

# For OCR:

chown root:dba /dev/raw/raw1

chown root:dba /dev/raw/raw2

chmod 660 /dev/raw/raw1

chmod 660 /dev/raw/raw2

# For voting disks:

chown oracle:dba /dev/raw/raw4

chown oracle:dba /dev/raw/raw5

chown oracle:dba /dev/raw/raw6

chmod 660 /dev/raw/raw4

chmod 660 /dev/raw/raw5

chmod 660 /dev/raw/raw6

8. Initialize the OCR and voting disk as root on one node as follow.

dd if=/dev/zero of=/dev/mpath/mpath8p1 bs=125829120 count=1

dd if=/dev/zero of=/dev/mpath/mpath8p2 bs=125829120 count=1

dd if=/dev/zero of=/dev/mpath/mpath8p5 bs=20971520 count=1

dd if=/dev/zero of=/dev/mpath/mpath8p6 bs=20971520 count=1

dd if=/dev/zero of=/dev/mpath/mpath8p7 bs=20971520 count=1

10. Now most importantly setting up udev permission.

Incase you do not make this changes during CRS installation you will get error.

When you would specify the OCR device location in the CRS installation page it will give error that “The specified shared raw partition /dev/raw/raw1 may not have correct permission. Verify that the partition is owned by Oracle User.”

This is a misleading error, and will make us do changes to the actual permission of the devices. I had struggled with this error for a couple of day before I realized that I need to create a file /etc/udev/permissions.d/40-udev.permissions.

[oracle@oradbdev01 logs]$ more /etc/udev/permissions.d/40-udev.permissions

# dm devices

dm-*:root:dba:0660

# ocr devices –inserted by apun

raw/raw1:root:dba:660

raw/raw2:root:dba:660

# voting disk –inserted by apun

raw/raw4:oracle:dba:660

raw/raw5:oracle:dba:660

raw/raw6:oracle:dba:660

Also comment out all entries for these devices in /etc/udev/permissions.d/50-udev.permissions file. The OS & oracle needs to refer to only /etc/udev/permissions.d/40-udev.permissions file.

Make these changes on all the RAC servers.

And reboot all the nodes.

This should complete all steps prior to starting any installation. Now you can download oracle crs and database software and start installation.

Important links to refer for more information;

http://www.oracle.com/technology/tech/linux/asmlib/index.html

Please refer to metalink notes: 465001.1 (for more info on udev)

Oracle 10g RAC Installation on NetApps Raw Devices with ASM: Part 2

Start the iSCSI service on the hosts:

If you are using dm-multipath, be sure you have configured multipathing before starting iSCSI.

Use the following the command to start iSCSI on all hosts:

sudo /etc/init.d/iscsi start

Add the iSCIS service to the boot sequence by entering the following commands:

sudo chkconfig –add iscsi

sudo chkconfig –list iscsi

sudo chkconfig iscsi on

To view the LUN information from the host:

[root@oradbdev02 ~]# iscsi-ls -l

***************************************************************************

SFNet iSCSI Driver Version …4:0.1.11-6(03-Aug-2007)

***************************************************************************TARGET NAME : iqn.1992-08.com.netapp:sn.101198981

TARGET ALIAS :

HOST ID : 0

BUS ID : 0

TARGET ID : 0

TARGET ADDRESS : 10.66.81.75:3260,1000

SESSION STATUS : ESTABLISHED AT Tue Apr 8 13:45:30 IST 2008

SESSION ID : ISID 00023d000001 TSIH 37

DEVICE DETAILS:

—————

LUN ID : 0

Vendor: NETAPP Model: LUN Rev: 0.2

Type: Direct-Access ANSI SCSI revision: 04

page83 type3: 60a9800043346d77674a484d2f6e7758

page80: 43346d77674a484d2f6e77580a

Device: /dev/sda

***************************************************************************

The output show that device /dev/sda is a LUN (LUN ID: 0) on the Netapps storage with IP Address 10.66.81.75.

If iSCSI Initiator Support Kit has been installed on the host, you can run the command “sanlun” to get a better view on LUNs.

[root@oradbdev02 ~]# sanlun lun show

filer: lun-pathname device filename adapter protocol lun size lun state

silo: /vol/vol1/ractest.lun /dev/sda host0 iSCSI 2g (2147483648) GOOD

Discovering new LUNs

If you create a new LUN and map it to the Linux host, you can discover the LUN by reloading the iscsi service on the host. To reload the iscsi service, enter the following command on the hosts:

/etc/init.d/iscsi reload

View multipath devices:

[root@oradbdev02 ~]# multipath -d -l

mpath6 (360a9800043346d77674a484d2f6e7758)

[size=2 GB][features="1 queue_if_no_path"][hwhandler="0"]

\_ round-robin 0 [active]

\_ 0:0:0:0 sda 8:0 [active]

Oracle 10g RAC Installation on NetApps Raw Devices with ASM: Part 1

Setting Up storage:

For performance benefits while using ASM we would be using iSCSI devices instead of virtual devices on NFS filesystem. We would also use Multipath software along with multiple SCSI Channels or fibre channels between the hosts and storage. This is to avoid single point of fault in the I/O channel and will also enhance performance.

For more info in Multipath: http://en.wikipedia.org/wiki/Multipath_I/O

Hardware setup for the scope of this document:

-sh-3.00$ more /etc/redhat-release

Red Hat Enterprise Linux ES release 4 (Nahant Update 5)

-sh-3.00$ more /proc/meminfo

MemTotal: 8164980 kB

-sh-3.00$ more /proc/cpuinfo |grep processor

processor : 0

processor : 1

processor : 2

processor : 3

Filer: NetApps FS 3050

Install iSCSI software on all the host used for RAC setup:

Download the latest package of iSCSI daemon and utility program from redhat and install it on all the hosts.

sudo rpm –Uvh iscsi-initiator-utils-4.0.3.0-6.x86_64.rpm

Install and configure Linux multipath software on all the hosts:

We are using Linux dm-multipath support for our setup:

Download the latest rpm from redhat and install it on all the hosts.

sudo rpm –Uvh device-mapper-multipath-0.4.5-27.RHEL4.x86_64.rpm

Load the dm_multipath kernel module:

modprobe dm_multipath

Edit the configuration file /etc/multipath.conf as follow:

Comment out the following line:

devnode_blacklist { devnode “*” }

Create a device-specific section at the end of the file for the storage system as follow:

devices {

device {

vendor “NETAPP”

product “LUN”

path_grouping_policy multibus

features “1 queue_if_no_path”

path_checker readsector0

failback immediate

}

}

Example of a /etc/multipath.conf file:

# This is an example configuration file for device mapper multipath.

# For a complete list of the default configuration values, see

# /usr/share/doc/device-mapper-multipath-0.4.5/multipath.conf.defaults

# For a list of configuration options with descriptions, see

# /usr/share/doc/device-mapper-multipath-0.4.5/multipath.conf.annotated

# Blacklist all devices by default. Remove this to enable multipathing

# on the default devices.

#devnode_blacklist {

# devnode “*”

#}

## Use user friendly names, instead of using WWIDs as names.

defaults {

user_friendly_names yes

}

## By default, devices with vendor = “IBM” and product = “S/390.*” are

## blacklisted. To enable mulitpathing on these devies, uncomment the

## following lines.

#devices {

# device {

# vendor “IBM”

# product “S/390 DASD ECKD”

# path_grouping_policy multibus

# getuid_callout “/sbin/dasdview -j -f /dev/%n”

# path_checker directio

# }

#}

##

## This is a template multipath-tools configuration file

## Uncomment the lines relevent to your environment

##

#defaults {

# udev_dir /dev

# polling_interval 10

# selector “round-robin 0″

# path_grouping_policy multibus

# getuid_callout “/sbin/scsi_id -g -u -s /block/%n”

# prio_callout /bin/true

# path_checker readsector0

# rr_min_io 100

# rr_weight priorities

# failback immediate

# no_path_retry fail

# user_friendly_name yes

#}

##

## The wwid line in the following blacklist section is shown as an example

## of how to blacklist devices by wwid. The 3 devnode lines are the

## compiled in default blacklist. If you want to blacklist entire types

## of devices, such as all scsi devices, you should use a devnode line.

## However, if you want to blacklist specific devices, you should use

## a wwid line. Since there is no guarantee that a specific device will

## not change names on reboot (from /dev/sda to /dev/sdb for example)

## devnode lines are not recommended for blacklisting specific devices.

##

#devnode_blacklist {

# wwid 26353900f02796769

# devnode “^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*”

# devnode “^hd[a-z]“

#}

#multipaths {

# multipath {

# wwid 3600508b4000156d700012000000b0000

# alias yellow

# path_grouping_policy multibus

# path_checker readsector0

# path_selector “round-robin 0″

# failback manual

# rr_weight priorities

# no_path_retry 5

# }

# multipath {

# wwid 1DEC_____321816758474

# alias red

# }

#}

#devices {

# device {

# vendor “COMPAQ “

# product “HSV110 (C)COMPAQ”

# path_grouping_policy multibus

# getuid_callout “/sbin/scsi_id -g -u -s /block/%n”

# path_checker readsector0

# path_selector “round-robin 0″

# hardware_handler “0″

# failback 15

# rr_weight priorities

# no_path_retry queue

# }

# device {

# vendor “COMPAQ “

# product “MSA1000 “

# path_grouping_policy multibus

# }

#}

devices {

device {

vendor “NETAPP”

product “LUN”

path_grouping_policy multibus

features “1 queue_if_no_path”

path_checker readsector0

failback immediate

}

}

Start the multipath daemon by entering the following command:

sudo /etc/init.d/multipathd start

Add the multipath service to the boot sequence by entering the following commands:

chkconfig –add multipathd

chkconfig –list multipathd

chkconfig multipathd on

Install iSCSI Initiator Support Kit on all the RAC Nodes:

To have a better view on iSCSI devices from hosts, you should download and install iSCSI support Kit from NetApp as follow:

Download iSCSI Linux Initiator Support Kit from http://now.netapp.com/NOW/download/software/kit_iscsi/Linux/ to a working directory on the hosts

Untar the file like

tar -xvf netapp_linux_tools_1_5.tar.gz

When I downloaded the file from this website, I found that the file has a .Z extension but when I tried to unzip the file it gave me error. The I just untar the file and it worked.

cd to the subdirectory netapp_linux_tools_1_5 and run install

./install

Modify iSCSI configurations on all RAC servers:

/sbin/iscsi-iname command will give to the iSCSI initiator name, which can be used to update the file /etc/initiatorname.iscsi. I have added the node name to the initiator name for node reference.

Edit the configuration file /etc/initiatorname.iscsi on all host to contain single line:

[oracle@oradbdev02 ~]$ sudo more /etc/initiatorname.iscsi

Password:

## DO NOT EDIT OR REMOVE THIS FILE!

## If you remove this file, the iSCSI daemon will not start.

## If you change the InitiatorName, existing access control lists

## may reject this initiator. The InitiatorName must be unique

## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.

InitiatorName=iqn.1987-05.com.cisco:01.45b2c1c2e7:oradbdev02

Edit the configuration file /etc/iscsi.conf on all hosts as follow:

If you are using dm-multipath support, remove the comment of the ConnFailTimeout line in the Session Timeout Settings section and set the value to

ConnFailTimeout=5.

Add the following line for any one iSCSI-enabled interface on each storage system that you will use for iSCSI LUNs:

DiscoveryAddress=storage_system_IPaddress

Where storage_system_Ipaddress is the IP address of an Ethernet interface on the storage system. Specify an interface that will be used for iSCSI communication. For example:

# ============================================================================

# iSCSI Configuration File Sample – see iscsi.conf(5)

# ============================================================================

#

# All of the configuration parameters described in this file are applied

# globally to all targets, unless they are overridden by a local setting. The

# three types of local categories that can override the global settings are:

#

# Target Name (i.e., TargetName)

# Network (i.e., Subnet or Address)

# SCSI Routing Instance (i.e., DiscoveryAddress)

#

# All parameters that are localized to one of the categories above must be

# indented by at least one white space or a tab character. If the parameter is

# not indented, it will be interpreted as a global parameter (see examples for

# each parameter).

#

# The parameters are global if they appear before any DiscoveryAddress

# or TargetName or Subnet entries. The global entries need to

# start at column 1. See the iscsi.conf man page’s Note section for more

# details

#

# If more that one entry exists for any given parameter (either global or

# local), the last entry has precedence.

#

# If a parameter setting under the network category conflicts with a different

# setting of the same parameter under the discovery address or target name

# category (for the same target), the network setting will have precedence.

#

# If a parameter is not specified in the iscsi.conf file, the default setting is

# used. The default values for all parameters can be found in the readme file.

#

# In the sample settings shown below, the following definitions apply:

# = any alpha-numeric text string

# = any numeric text string

#

= valid IP address of the form a.b.c.d[/e]

#
= valid portal address of the form a.b.c.d[:e]

#

# ============================================================================

# PARAMETERS

# ============================================================================

#

# ———————–

# AUTHENTICATION SETTINGS

# ———————–

# To globally configure a CHAP username and password for initiator

# authentication by the target(s), uncomment the following lines:

#

#OutgoingUsername=

#OutgoingPassword=

#

# The maximum length for both the password and username is 256 characters.

# An example username and password would be:

#

#OutgoingUsername=alice

#OutgoingPassword=nty57nbe

#

# To globally configure a CHAP username and password for target(s)

# authentication by the initiator, uncomment the following lines:

#

#IncomingUsername=

#IncomingPassword=

#

# The maximum length for both the password and username is 256 characters.

# An example username and password would be:

#

#IncomingUsername=bill

#IncomingPassword=ghot67

#

# The global authentication settings can be overridden on a per discovery

# address basis. An example of a unique username and password for all targets

# found at address 192.168.10.94 would be:

#

#DiscoveryAddress=192.168.10.94

# OutgoingUsername=fred

# OutgoingPassword=uhyt6h

# and/or

#

#DiscoveryAddress=192.168.10.94

# IncomingUsername=mary

# IncomingPassword=kdhjkd9l

#

# —————

# DIGEST SETTINGS

# —————

# To globally enable CRC32C digest checking for the header part of

# iSCSI PDUs, uncomment the following line:

#

#HeaderDigest=always

#

# To globally disable digest checking for the header data part of

# iSCSI PDUs, uncomment the following line:

#

#HeaderDigest=never

#

# To globally allow the targets to control the setting of the digest checking,

# with the initiator requesting a preference of enabling the checking, uncomment

# the following line:

#

#HeaderDigest=prefer-on

#

# To globally allow the targets to control the setting of the digest checking,

# with the initiator requesting a preference of disabling the checking,

# uncomment the following line:

#

#HeaderDigest=prefer-off

#

# The global digest settings can be overridden on a per target name basis. An

# example of enabling header digest checking just for target iqn.1987-

# 05.com.cisco:00.0d1d898e8d66.t0 would be:

#

#TargetName=iqn.1987-05.com.cisco:00.0d1d898e8d66.t0

# HeaderDigest=always

#

# It should also be noted that if the initiator and the target have incompatible

# settings (e.g., target set for “always” and initiator set for “never”), the

# login will fail.

#

# ———————-

# ENABLE/DISABLE TARGETS

# ———————-

# To globally enable/disable group of targets use the following option.

#

# TargetNames mentioned after the below entry will be enabled by default.

#Enabled=yes

#

# TargetNames mentioned after the below entry will be disabled by default.

#Enabled=no

#

# To specifically enable/disable a target, use the following entry

# under Targetname.

#

#TargetName=iqn.1987-05.com.cisco:00.0d1d898e8d66.t0

# Enabled=yes

#

# —————————

# CONNECTION TIMEOUT SETTINGS

# —————————

# To globally specify the time to wait for a login PDU to be received from

# the target in response to a login request sent by the initiator before failing

# the connection attempt, uncomment the following line:

#

#LoginTimeout=

#

# where is in seconds. A setting of “0″ will result in commands never

# being timed out.

#

# IdleTimeout and ActiveTimeout determine when a ping has to be sent to check

# the status of the connection when there is no traffic on the connection from

# the target.

#

# To globally specify the time to wait on a connection with no tasks active in

# the session, uncomment the following line:

#

#IdleTimeout=

#

# where is in seconds. A setting of “0″ will result in a ping never

# being sent.

#

# To globally specify the time to wait on a connection with tasks active in the

# session, uncomment the following line:

#

#ActiveTimeout=

#

# where is in seconds. A setting of “0″ will result in a ping never

# being sent.

#

# To globally specify the time to wait for a ping response after a ping has been

# sent to a target before failing the existing connection and initiating a new

# one, uncomment the following line:

#

#PingTimeout=

#

# where is in seconds. A setting of “0″ will result in the ping command

# never timing out.

#

# The global connection timeout settings can be overridden on a per target name,

# discovery address or IP address basis. An example of setting the

# “LoginTimeout” value to 12 seconds for just target iqn.1987-

# 05.com.cisco:00.0d1d898e8d66.t0 would be:

#

#TargetName=iqn.1987-05.com.cisco:00.0d1d898e8d66.t0

# LoginTimeout=12

#

# An example of setting the “PingTimeout” value to 8 seconds for just all

# targets found at address 192.168.10.94 would be:

#

#DiscoveryAddress=192.168.10.94

# PingTimeout=8

#

# An example of setting the “IdleTimeout” value to 3 seconds for just all

# targets found on subnet 192.168.10.94 would be:

#

#Subnet=192.168.10.0/24

# IdleTimeout=3

#

# ————————

# SESSION TIMEOUT SETTINGS

# ————————

# To globally specify the length of time to wait for session re-establishment

# before failing SCSI commands back to the application, uncomment the

# following line:

#

ConnFailTimeout=5

#

# where is in seconds. A setting of “0″ will result in commands never

# being failed back due to connection failure.

#

# The global session timeout settings can be overridden on a per target name

# basis. An example of setting the “ConnFailTimeout” value to 5 seconds for

# just target iqn.1987-05.com.cisco:00.0d1d898e8d66.t0 would be:

#

#TargetName=iqn.1987-05.com.cisco:00.0d1d898e8d66.t0

# ConnFailTimeout=5

#

# If a third party multipathing application is being used,

# then the “ConnFailTimeout” should be set to smaller value

# such as 15. This value is just a guideline so the actual value will be

# dependent on the users operating environment.

#

# ——————————-

# ERROR HANDLING TIMEOUT SETTINGS

# ——————————-

# To globally specify the length of time to wait for an abort command to

# complete before declaring the abort command has failed, uncomment the

# following line:

#

#AbortTimeout=

#

# where is in seconds. A setting of “0″ will result in commands never

# being timed out.

#

# To globally specify the length of time to wait for a reset command to complete

# before declaring that the reset command has failed, uncomment the following

# line:

#

#ResetTimeout=

#

# where is in seconds. A setting of “0″ will result in commands never

# being timed out.

#

# The global error handling timeout settings can be overridden on a per target

# name or per IP address basis. An example of setting the “AbortTimeout” value

# to 10 seconds for just target iqn.1987-05.com.cisco:00.0d1d898e8d66.t0 would

# be:

#

#TargetName=iqn.1987-05.com.cisco:00.0d1d898e8d66.t0

# AbortTimeout=10

#

# An example of setting the “ResetTimeout” value to 6 seconds for just all

# targets found on portal 192.168.10.94 would be:

#

#Subnet=192.168.10.0/24

# ResetTimeout=6

#

# —————————–

# CONTINUOUS DISCOVERY SETTINGS

# —————————–

# To globally specify that all discovery sessions be kept open, uncomment the

# following line:

#

#Continuous=yes

#

# To globally specify that all discovery sessions be closed once discovery is

# completed, uncomment the following line:

#

#Continuous=no

#

# The global continuous discovery setting can be overridden on a per target

# basis. An example of setting “Continuous” to “no” for just target iqn.1987-

# 05.com.cisco:00.0d1d898e8d66.t0 would be:

#

#TargetName=iqn.1987-05.com.cisco:00.0d1d898e8d66.t0

# Continuous=no

#

# ———————————

# ASYNC EVENT NOTIFICATION SETTINGS

# ———————————

# To globally specify that the initiator wants to receive vendor specific async

# events from the target(s), uncomment the following line:

#

#SendAsyncText=yes

#

# To globally specify that the initiator does not want to receive vendor

# specific async events from the target(s), uncomment the following line:

#

#SendAsyncText=no

#

# The SendAsyncText key can be specified for a particular Discovery Address.

#

# The global async event notification setting can be overridden on a per target

# basis. An example of setting “SendAsyncText” to “no” for just target iqn.1987-

# 05.com.cisco:00.0d1d898e8d66.t0 would be:

#

#TargetName=iqn.1987-05.com.cisco:00.0d1d898e8d66.t0

# SendAysncText=no

#

# ————————————

# iSCSI OPERATIONAL PARAMETER SETTINGS

# ————————————

# To globally enable R2T flow control (i.e., the initiator must wait for an R2T

# command before sending any data), uncomment the following line:

#

#InitialR2T=yes

#

# To globally disable R2T flow control (i.e., the initiator has an implied

# initial R2T of “FirstBurstLength” at offset 0), uncomment the following line:

#

#InitialR2T=no

#

# To globally enable immediate data (i.e., the initiator sends unsolicited data

# with the iSCSI command packet), uncomment the following line:

#

#ImmediateData=yes

#

# To globally disable immediate data (i.e., the initiator does not send

# unsolicited data with the iSCSI command PDU), uncomment the following line:

#

#ImmediateData=no

#

# To globally specify the maximum number of data bytes the initiator can receive

# in an iSCSI PDU from a target, uncomment the following line:

#

#MaxRecvDataSegmentLength=

#

# where is the number of bytes in the range of 512 to (2^24-1)

#

# To globally specify the maximum number of unsolicited data bytes the initiator

# can send in an iSCSI PDU to a target, uncomment the following line:

#

#FirstBurstLength=

#

# where is the number of bytes in the range of 512 to (2^24-1)

#

# To globally specify the maximum SCSI payload that the initiator will negotiate

# with the target for, uncomment the following line:

#

#MaxBurstLength=

#

# where is the number of bytes in the range of 512 to (2^24-1)

#

# To globally specifiy the maximum number of bytes that can be sent over a TCP

# connection by the initiator before receiving an acknowledgement from the

# target, uncomment the following line:

#

#TCPWindowSize=

#

# where is the number of bytes in the range of 512 to (2^24-1)

#

# The global iSCSI operational parameter setting can be overridden on a per

# target basis. An example of setting multiple parameters for just target

# iqn.1987-05.com.cisco:00.0d1d898e8d66.t0 would be:

#

#TargetName=iqn.1987-05.com.cisco:00.0d1d898e8d66.t0

# InitialR2T=no

# ImmediateData=no

# MaxRecvDataSegmentLength=128 * 1024

# FirstBurstLength=262144

# MaxBurstLength=(16 * 1024 * 1024) – 1024

# TCPWindowSize=262144

#

# The global “TCPWindowSize” setting can also be overridden on a per portal

# basis. An example of setting the “TCPWindowSize” for just subnet 10.77.13.0/16

# would be:

#

#Subnet=10.77.13.0/16

# TCPWindowSize=262144

#

# —————-

# Network Category

# —————-

# To localize parameters to targets on a particular network (i.e., to

# override the global settings), you need to use either the “Subnet” or

# “Address” settings. The format for the “Subnet” setting is a.b.c.d/e. An

# example of these settings would be:

#

#Subnet=10.4.100.0/24

#

# The format for “Address” is a.b.c.d. An example of these settings would be:

#

#Address=10.4.100.0

#

# The following parameters can be specified using the network category:

#

# 1) Connection Timeout Settings

# 2) Error Handling Timeout Settings

# 3) TCP Settings

#

# ————————–

# Discovery Address Category

# ————————–

# To localize parameters to targets found on a particular discovery address

# (i.e., to override the global settings), you need to use the

# “DiscoveryAddress” setting. The format for the “DiscoveryAddress” setting is

# a.b.c.d, a.b.c.d:e (where e is a TCP port number), or an instance name. An

# example of these settings would be:

#

#DiscoveryAddress=10.4.100.0

# or

#DiscoveryAddress=10.4.100.1:3260

# or

#DiscoveryAddress=scisrouter1

#

# The following parameters can be specified using the discovery address

# category:

#

# 1) Authentication Settings

# 2) ConnectionTimeout Settings

# 3) Continuous Discovery settings

# 4) AsyncEvent Notification Settings

#

# ——————–

# Target Name Category

# ——————–

# To localize parameters to targets identified by a particular target name

# (i.e., to override the global settings), you need to use the

# “TargetName” setting. The format for the “TargetName” setting is

# either the ‘iqn’ or ‘eui’ format. An example of these settings would be:

#

#TargetName=iqn.1987-05.com.cisco:00.0d1d898e8d66.t0

#

# The following parameters can be specified using the target name category:

#

# 1) CRC Settings

# 2) iSCSI Operational Parameter settings

# 3) Connection Timeout Settings

# 4) Session Timeout Settings

# 5) Error Handling Timeout Settings

# 6) TCP Settings

# 7) Enable/Disable targets

#

DiscoveryAddress=10.66.81.75

Configure the storage system for iSCSI:

Perform the following tasks on the storage system console:

I would recommend taking help from your storage team to make these changes instead of trying it yourself. These steps don’t fall in the scope of an Oracle DBA.

You can ask your storage admin to create luns of the size you require for your oracle database. Also mention that you would like these luns to be visible from all your RAC nodes.

Enter the following command to check whether iSCSI service is running:

iscsi status

iSCSI service is not running

If it is not running, enter the following command to start it and check it again:

iscsi start

iscsi status

iSCSI service is running

Create iSCSI initiator group with the following command:

igroup create -i -t linux

Example:

igroup create -i -t linux ractest

Add iSCSI initiators to the iSCSI initiator group with the following command:

igroup add

where initiator name is defined in the host’s configuration file /etc/initiatorname.iscsi.

Example:

igroup add ractest iqn.1987-05.com.cisco:01.45b2c1c2e7:oradbdev02

Use the following command to see the initiators and initiator groups

igroup show

Example:

igroup show

igroup1 (iSCSI) (ostype: linux):

iqn.1987-05.com.cisco:01.45b2c1c2e7:oradbdev02 (not logged in)

Create volumes that contain LUNs with the following commands:

vol create size {k|m|g|t}

vol options nosnap on # disable snapshots

snap reserve 0 # disable snapshot reservation

snap sched 0 0 0 # disable the automatic snapshot schedule

For example, the following commands create an 2 GB volume /vol/vol1 on the aggregate aggr1:

vol create ractest aggr1 size 2g

vol options vol1 nosnap on

snap reserve vol1 0

snap sched vol1 0 0 0

Create LUNs with the following command:

lun create -s size -t linux lun_path

where lun_path is the LUN’s path name that includes the volume and qtree.

For example, the following commands create a 500 GB LUN /vol/vol1/ractest.lun in the volume /vol/vol1:

lun create -s 2g -t linux /vol/vol1/ractest.lun

You can use the following command to see all LUNs (From OS):

[oracle@oradbdev02 ~]$ sudo sanlun lun show

Password:

filer: lun-pathname device filename adapter protocol lun size lun state

silo: /vol/vol1/ractest.lun /dev/sda host0 iSCSI 2g (2147483648) GOOD

Map the LUN to an igroup with the following command:

lun map lun_path initiator_group

Example:

lun map /vol/vol1/ractest.lun ractest

How To Change the location of OCR in 10gR2 Oracle RAC

OCR:

1.) Login as root.

2.) Start CRS on all nodes:

root@oracle-server:/home/crs/bin[bin]# ./crsctl start crs

Attempting to start CRS stack

The CRS stack will be started shortly

root@oracle-server:/home/crs/bin[bin]# ./crsctl check crs

Failure 1 contacting CSS daemon

Cannot communicate with CRS

Cannot communicate with EVM

root@oracle-server:/home/crs/bin[bin]# ./crsctl check crs

CSS appears healthy

CRS appears healthy

EVM appears healthy

3.) Check OCR configuration/backups:

root@oracle-server:/home/crs/bin[bin]# ocrcheck

Status of Oracle Cluster Registry is as follows :

Version : 2

Total space (kbytes) : 262144

Used space (kbytes) : 2812

Available space (kbytes) : 259332

ID : 429958626

Device/File Name : /mnt/prdaudit_crs/crs

Device/File integrity check succeeded

Device/File Name : /mnt/prdaudit_crs/crs_mirror

Device/File integrity check succeeded

Cluster registry integrity check succeeded

root@oracle-server:/home/crs/bin[bin]# ocrconfig -showbackup

ac4-irauditdb-002 2008/04/10 23:47:02 /oracle/product/10.2.0/crs/cdata/ac4-iraudit-crs

ac4-irauditdb-002 2008/04/10 19:47:03 /oracle/product/10.2.0/crs/cdata/ac4-iraudit-crs

ac4-irauditdb-002 2008/04/10 15:47:04 /oracle/product/10.2.0/crs/cdata/ac4-iraudit-crs

ac4-irauditdb-002 2008/04/08 19:47:17 /oracle/product/10.2.0/crs/cdata/ac4-iraudit-crs

ac4-irauditdb-002 2008/04/03 15:47:53 /oracle/product/10.2.0/crs/cdata/ac4-iraudit-crs

4.) Copy the OCR mirror to the new location:

root@oracle-server:/mnt/prdaudit_crs[root@oracle-server prdaudit_crs]# ls -rlt

mtotal 38916

-rw-r–r– 1 oracle dba 10240000 Apr 11 02:27 vd2.old

-rw-r–r– 1 oracle dba 10240000 Apr 11 02:27 vd3.old

-rw-r—– 1 root dba 3121152 Apr 11 02:59 crs_mirror

-rw-r—– 1 root dba 3121152 Apr 11 03:22 crs

-rw-r–r– 1 oracle dba 10240000 Apr 11 03:23 vd1

root@oracle-server:/mnt/prdaudit_crs[root@oracle-server prdaudit_crs]# cp crs_mirror /mnt/prdaudit_crs_mirror1

Check the permission, it should be root:dba

5.) Change the OCR location in the CRS:

root@oracle-server:/home/crs/bin[bin]# ./ocrconfig -replace ocrmirror /mnt/prdaudit_crs_mirror1/crs_mirror

root@oracle-server:/home/crs/bin[bin]# ocrcheck

Status of Oracle Cluster Registry is as follows :

Version : 2

Total space (kbytes) : 262144

Used space (kbytes) : 2812

Available space (kbytes) : 259332

ID : 429958626

Device/File Name : /mnt/prdaudit_crs/crs

Device/File integrity check succeeded

Device/File Name : /mnt/prdaudit_crs_mirror1/crs_mirror

Device/File integrity check succeeded

Cluster registry integrity check succeeded

root@oracle-server:/home/crs/bin[bin]# exit

How To Change the location of Voting Disk in 10gR2 Oracle RAC

How To Change the location of Voting Disk in 10gR2 Oracle RAC

Voting Disk:
1.) Login as root
2.) Shutdown CRS on all nodes:
root@oracle-server:/home/crs/bin[bin]# ./crsctl stop crs
Stopping resources.
Successfully stopped CRS resources
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
root@oracle-server:/home/crs/bin[bin]# ./crsctl check crs
Failure 1 contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM

3.) Check the the location of the voting disk:
root@oracle-server:/home/crs/bin[bin]# ./crsctl query css votedisk
0. 0 /mnt/prdaudit_crs/vd1
1. 0 /mnt/prdaudit_crs/vd2
2. 0 /mnt/prdaudit_crs/vd3

4.) Copy the voting disk to the new location:
root@oracle-server:/home/crs/bin[bin]# cp /mnt/prdaudit_crs/vd2 /mnt/prdaudit_crs_mirror1/
root@oracle-server:/home/crs/bin[bin]# cp /mnt/prdaudit_crs/vd3 /mnt/prdaudit_crs_mirror2/

5.) Change the location in CRS:
root@oracle-server:/home/crs/bin[bin]# ./crsctl add css votedisk /mnt/prdaudit_crs_mirror1/vd2
Now formatting voting disk: /mnt/prdaudit_crs_mirror1/vd2
successful addition of votedisk /mnt/prdaudit_crs_mirror1/vd2.
root@oracle-server:/home/crs/bin[bin]# ./crsctl add css votedisk /mnt/prdaudit_crs_mirror2/vd3
Now formatting voting disk: /mnt/prdaudit_crs_mirror2/vd3
successful addition of votedisk /mnt/prdaudit_crs_mirror2/vd3.

6.) Delete the old Voting Disks:
root@oracle-server:/home/crs/bin[bin]# crsctl delete css votedisk /mnt/prdaudit_crs/vd2 -force
successful deletion of votedisk /mnt/prdaudit_crs/vd2.
root@oracle-server:/home/crs/bin[bin]# crsctl delete css votedisk /mnt/prdaudit_crs/vd3 -force
successful deletion of votedisk /mnt/prdaudit_crs/vd3.

7.) Query CRS to see if the changes where proper:
root@oracle-server:/home/crs/bin[bin]# ./crsctl query css votedisk
0. 0 /mnt/prdaudit_crs/vd1
1. 0 /mnt/prdaudit_crs_mirror2/vd3
2. 0 /mnt/prdaudit_crs_mirror1/vd2

PS: The permission to the voting disk should be root:dba, while copying and creating voting disk as we are using the root user, the permission will change to root:root, which will give error when starting crs.