Setting Up storage:

For performance benefits while using ASM we would be using iSCSI devices instead of virtual devices on NFS filesystem. We would also use Multipath software along with multiple SCSI Channels or fibre channels between the hosts and storage. This is to avoid single point of fault in the I/O channel and will also enhance performance.

For more info in Multipath: http://en.wikipedia.org/wiki/Multipath_I/O

Hardware setup for the scope of this document:

-sh-3.00$ more /etc/redhat-release

Red Hat Enterprise Linux ES release 4 (Nahant Update 5)

-sh-3.00$ more /proc/meminfo

MemTotal: 8164980 kB

-sh-3.00$ more /proc/cpuinfo |grep processor

processor : 0

processor : 1

processor : 2

processor : 3

Filer: NetApps FS 3050

Install iSCSI software on all the host used for RAC setup:

Download the latest package of iSCSI daemon and utility program from redhat and install it on all the hosts.

sudo rpm –Uvh iscsi-initiator-utils-4.0.3.0-6.x86_64.rpm

Install and configure Linux multipath software on all the hosts:

We are using Linux dm-multipath support for our setup:

Download the latest rpm from redhat and install it on all the hosts.

sudo rpm –Uvh device-mapper-multipath-0.4.5-27.RHEL4.x86_64.rpm

Load the dm_multipath kernel module:

modprobe dm_multipath

Edit the configuration file /etc/multipath.conf as follow:

Comment out the following line:

devnode_blacklist { devnode “*” }

Create a device-specific section at the end of the file for the storage system as follow:

devices {

device {

vendor “NETAPP”

product “LUN”

path_grouping_policy multibus

features “1 queue_if_no_path”

path_checker readsector0

failback immediate

}

}

# This is an example configuration file for device mapper multipath.

# For a complete list of the default configuration values, see

# /usr/share/doc/device-mapper-multipath-0.4.5/multipath.conf.defaults

# For a list of configuration options with descriptions, see

# /usr/share/doc/device-mapper-multipath-0.4.5/multipath.conf.annotated

# Blacklist all devices by default. Remove this to enable multipathing

# on the default devices.

#devnode_blacklist {

# devnode “*”

#}

## Use user friendly names, instead of using WWIDs as names.

defaults {

user_friendly_names yes

}

## By default, devices with vendor = “IBM” and product = “S/390.*” are

## blacklisted. To enable mulitpathing on these devies, uncomment the

## following lines.

#devices {

# device {

# vendor “IBM”

# product “S/390 DASD ECKD”

# path_grouping_policy multibus

# getuid_callout “/sbin/dasdview -j -f /dev/%n”

# path_checker directio

# }

#}

##

## This is a template multipath-tools configuration file

## Uncomment the lines relevent to your environment

##

#defaults {

# udev_dir /dev

# polling_interval 10

# selector “round-robin 0″

# path_grouping_policy multibus

# getuid_callout “/sbin/scsi_id -g -u -s /block/%n”

# prio_callout /bin/true

# path_checker readsector0

# rr_min_io 100

# rr_weight priorities

# failback immediate

# no_path_retry fail

# user_friendly_name yes

#}

##

## The wwid line in the following blacklist section is shown as an example

## of how to blacklist devices by wwid. The 3 devnode lines are the

## compiled in default blacklist. If you want to blacklist entire types

## of devices, such as all scsi devices, you should use a devnode line.

## However, if you want to blacklist specific devices, you should use

## a wwid line. Since there is no guarantee that a specific device will

## not change names on reboot (from /dev/sda to /dev/sdb for example)

## devnode lines are not recommended for blacklisting specific devices.

##

#devnode_blacklist {

# wwid 26353900f02796769

# devnode “^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*”

# devnode “^hd[a-z]“

#}

#multipaths {

# multipath {

# wwid 3600508b4000156d700012000000b0000

# alias yellow

# path_grouping_policy multibus

# path_checker readsector0

# path_selector “round-robin 0″

# failback manual

# rr_weight priorities

# no_path_retry 5

# }

# multipath {

# wwid 1DEC_____321816758474

# alias red

# }

#}

#devices {

# device {

# vendor “COMPAQ “

# product “HSV110 (C)COMPAQ”

# path_grouping_policy multibus

# getuid_callout “/sbin/scsi_id -g -u -s /block/%n”

# path_checker readsector0

# path_selector “round-robin 0″

# hardware_handler “0″

# failback 15

# rr_weight priorities

# no_path_retry queue

# }

# device {

# vendor “COMPAQ “

# product “MSA1000 “

# path_grouping_policy multibus

# }

#}

devices {

device {

vendor “NETAPP”

product “LUN”

path_grouping_policy multibus

features “1 queue_if_no_path”

path_checker readsector0

failback immediate

}

Start the multipath daemon by entering the following command:

sudo /etc/init.d/multipathd start

Add the multipath service to the boot sequence by entering the following commands:

chkconfig –add multipathd

chkconfig –list multipathd

chkconfig multipathd on

Install iSCSI Initiator Support Kit on all the RAC Nodes:

To have a better view on iSCSI devices from hosts, you should download and install iSCSI support Kit from NetApp as follow:

Download iSCSI Linux Initiator Support Kit from http://now.netapp.com/NOW/download/software/kit_iscsi/Linux/ to a working directory on the hosts

Untar the file like

tar -xvf netapp_linux_tools_1_5.tar.gz

When I downloaded the file from this website, I found that the file has a .Z extension but when I tried to unzip the file it gave me error. The I just untar the file and it worked.

cd to the subdirectory netapp_linux_tools_1_5 and run install

./install

Modify iSCSI configurations on all RAC servers:

/sbin/iscsi-iname command will give to the iSCSI initiator name, which can be used to update the file /etc/initiatorname.iscsi. I have added the node name to the initiator name for node reference.

Edit the configuration file /etc/initiatorname.iscsi on all host to contain single line:

[oracle@oradbdev02 ~]$ sudo more /etc/initiatorname.iscsi

## DO NOT EDIT OR REMOVE THIS FILE!

## If you remove this file, the iSCSI daemon will not start.

## If you change the InitiatorName, existing access control lists

## may reject this initiator. The InitiatorName must be unique

## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.

Edit the configuration file /etc/iscsi.conf on all hosts as follow:

If you are using dm-multipath support, remove the comment of the ConnFailTimeout line in the Session Timeout Settings section and set the value to

ConnFailTimeout=5.

Add the following line for any one iSCSI-enabled interface on each storage system that you will use for iSCSI LUNs:

DiscoveryAddress=storage_system_IPaddress

# ============================================================================

# iSCSI Configuration File Sample – see iscsi.conf(5)

# ============================================================================

#

# All of the configuration parameters described in this file are applied

# globally to all targets, unless they are overridden by a local setting. The

# three types of local categories that can override the global settings are:

#

# Target Name (i.e., TargetName)

# Network (i.e., Subnet or Address)

# SCSI Routing Instance (i.e., DiscoveryAddress)

#

# All parameters that are localized to one of the categories above must be

# indented by at least one white space or a tab character. If the parameter is

# not indented, it will be interpreted as a global parameter (see examples for

# each parameter).

#

# The parameters are global if they appear before any DiscoveryAddress

# or TargetName or Subnet entries. The global entries need to

# start at column 1. See the iscsi.conf man page’s Note section for more

# details

#

# If more that one entry exists for any given parameter (either global or

# local), the last entry has precedence.

#

# If a parameter setting under the network category conflicts with a different

# setting of the same parameter under the discovery address or target name

# category (for the same target), the network setting will have precedence.

#

# If a parameter is not specified in the iscsi.conf file, the default setting is

# used. The default values for all parameters can be found in the readme file.

#

# In the sample settings shown below, the following definitions apply:

# = any alpha-numeric text string

# = any numeric text string

#

#
= valid portal address of the form a.b.c.d[:e]

#

# ============================================================================

# PARAMETERS

# ============================================================================

#

# ———————–

# AUTHENTICATION SETTINGS

# ———————–

# To globally configure a CHAP username and password for initiator

# authentication by the target(s), uncomment the following lines:

#

#OutgoingUsername=

#OutgoingPassword=

#

# The maximum length for both the password and username is 256 characters.

# An example username and password would be:

#

#OutgoingUsername=alice

#OutgoingPassword=nty57nbe

#

# To globally configure a CHAP username and password for target(s)

# authentication by the initiator, uncomment the following lines:

#

#IncomingUsername=

#IncomingPassword=

#

# The maximum length for both the password and username is 256 characters.

# An example username and password would be:

#

#IncomingUsername=bill

#IncomingPassword=ghot67

#

# The global authentication settings can be overridden on a per discovery

# address basis. An example of a unique username and password for all targets

# found at address 192.168.10.94 would be:

#

#DiscoveryAddress=192.168.10.94

# OutgoingUsername=fred

# OutgoingPassword=uhyt6h

# and/or

#

#DiscoveryAddress=192.168.10.94

# IncomingUsername=mary

# IncomingPassword=kdhjkd9l

#

# —————

# DIGEST SETTINGS

# —————

# To globally enable CRC32C digest checking for the header part of

# iSCSI PDUs, uncomment the following line:

#

#HeaderDigest=always

#

# To globally disable digest checking for the header data part of

# iSCSI PDUs, uncomment the following line:

#

#HeaderDigest=never

#

# To globally allow the targets to control the setting of the digest checking,

# with the initiator requesting a preference of enabling the checking, uncomment

# the following line:

#

#HeaderDigest=prefer-on

#

# To globally allow the targets to control the setting of the digest checking,

# with the initiator requesting a preference of disabling the checking,

# uncomment the following line:

#

#HeaderDigest=prefer-off

#

# The global digest settings can be overridden on a per target name basis. An

# example of enabling header digest checking just for target iqn.1987-

# 05.com.cisco:00.0d1d898e8d66.t0 would be:

#

#TargetName=iqn.1987-05.com.cisco:00.0d1d898e8d66.t0

# HeaderDigest=always

#

# It should also be noted that if the initiator and the target have incompatible

# settings (e.g., target set for “always” and initiator set for “never”), the

# login will fail.

#

# ———————-

# ENABLE/DISABLE TARGETS

# ———————-

# To globally enable/disable group of targets use the following option.

#

# TargetNames mentioned after the below entry will be enabled by default.

#Enabled=yes

#

# TargetNames mentioned after the below entry will be disabled by default.

#Enabled=no

#

# To specifically enable/disable a target, use the following entry

# under Targetname.

#

#TargetName=iqn.1987-05.com.cisco:00.0d1d898e8d66.t0

# Enabled=yes

#

# —————————

# CONNECTION TIMEOUT SETTINGS

# —————————

# To globally specify the time to wait for a login PDU to be received from

# the target in response to a login request sent by the initiator before failing

# the connection attempt, uncomment the following line:

#

#LoginTimeout=

#

# where is in seconds. A setting of “0″ will result in commands never

# being timed out.

#

# IdleTimeout and ActiveTimeout determine when a ping has to be sent to check

# the status of the connection when there is no traffic on the connection from

# the target.

#

# To globally specify the time to wait on a connection with no tasks active in

# the session, uncomment the following line:

#

#IdleTimeout=

#

# where is in seconds. A setting of “0″ will result in a ping never

# being sent.

#

# To globally specify the time to wait on a connection with tasks active in the

# session, uncomment the following line:

#

#ActiveTimeout=

#

# where is in seconds. A setting of “0″ will result in a ping never

# being sent.

#

# To globally specify the time to wait for a ping response after a ping has been

# sent to a target before failing the existing connection and initiating a new

# one, uncomment the following line:

#

#PingTimeout=

#

# where is in seconds. A setting of “0″ will result in the ping command

# never timing out.

#

# The global connection timeout settings can be overridden on a per target name,

# discovery address or IP address basis. An example of setting the

# “LoginTimeout” value to 12 seconds for just target iqn.1987-

# 05.com.cisco:00.0d1d898e8d66.t0 would be:

#

#TargetName=iqn.1987-05.com.cisco:00.0d1d898e8d66.t0

# LoginTimeout=12

#

# An example of setting the “PingTimeout” value to 8 seconds for just all

# targets found at address 192.168.10.94 would be:

#

#DiscoveryAddress=192.168.10.94

# PingTimeout=8

#

# An example of setting the “IdleTimeout” value to 3 seconds for just all

# targets found on subnet 192.168.10.94 would be:

#

#Subnet=192.168.10.0/24

# IdleTimeout=3

#

# ————————

# SESSION TIMEOUT SETTINGS

# ————————

# To globally specify the length of time to wait for session re-establishment

# before failing SCSI commands back to the application, uncomment the

# following line:

#

ConnFailTimeout=5

#

# where is in seconds. A setting of “0″ will result in commands never

# being failed back due to connection failure.

#

# The global session timeout settings can be overridden on a per target name

# basis. An example of setting the “ConnFailTimeout” value to 5 seconds for

# just target iqn.1987-05.com.cisco:00.0d1d898e8d66.t0 would be:

#

#TargetName=iqn.1987-05.com.cisco:00.0d1d898e8d66.t0

# ConnFailTimeout=5

#

# If a third party multipathing application is being used,

# then the “ConnFailTimeout” should be set to smaller value

# such as 15. This value is just a guideline so the actual value will be

# dependent on the users operating environment.

#

# ——————————-

# ERROR HANDLING TIMEOUT SETTINGS

# ——————————-

# To globally specify the length of time to wait for an abort command to

# complete before declaring the abort command has failed, uncomment the

# following line:

#

#AbortTimeout=

#

# where is in seconds. A setting of “0″ will result in commands never

# being timed out.

#

# To globally specify the length of time to wait for a reset command to complete

# before declaring that the reset command has failed, uncomment the following

# line:

#

#ResetTimeout=

#

# where is in seconds. A setting of “0″ will result in commands never

# being timed out.

#

# The global error handling timeout settings can be overridden on a per target

# name or per IP address basis. An example of setting the “AbortTimeout” value

# to 10 seconds for just target iqn.1987-05.com.cisco:00.0d1d898e8d66.t0 would

# be:

#

#TargetName=iqn.1987-05.com.cisco:00.0d1d898e8d66.t0

# AbortTimeout=10

#

# An example of setting the “ResetTimeout” value to 6 seconds for just all

# targets found on portal 192.168.10.94 would be:

#

#Subnet=192.168.10.0/24

# ResetTimeout=6

#

# —————————–

# CONTINUOUS DISCOVERY SETTINGS

# —————————–

# To globally specify that all discovery sessions be kept open, uncomment the

# following line:

#

#Continuous=yes

#

# To globally specify that all discovery sessions be closed once discovery is

# completed, uncomment the following line:

#

#Continuous=no

#

# The global continuous discovery setting can be overridden on a per target

# basis. An example of setting “Continuous” to “no” for just target iqn.1987-

# 05.com.cisco:00.0d1d898e8d66.t0 would be:

#

#TargetName=iqn.1987-05.com.cisco:00.0d1d898e8d66.t0

# Continuous=no

#

# ———————————

# ASYNC EVENT NOTIFICATION SETTINGS

# ———————————

# To globally specify that the initiator wants to receive vendor specific async

# events from the target(s), uncomment the following line:

#

#SendAsyncText=yes

#

# To globally specify that the initiator does not want to receive vendor

# specific async events from the target(s), uncomment the following line:

#

#SendAsyncText=no

#

# The SendAsyncText key can be specified for a particular Discovery Address.

#

# The global async event notification setting can be overridden on a per target

# basis. An example of setting “SendAsyncText” to “no” for just target iqn.1987-

# 05.com.cisco:00.0d1d898e8d66.t0 would be:

#

#TargetName=iqn.1987-05.com.cisco:00.0d1d898e8d66.t0

# SendAysncText=no

#

# ————————————

# iSCSI OPERATIONAL PARAMETER SETTINGS

# ————————————

# To globally enable R2T flow control (i.e., the initiator must wait for an R2T

# command before sending any data), uncomment the following line:

#

#InitialR2T=yes

#

# To globally disable R2T flow control (i.e., the initiator has an implied

# initial R2T of “FirstBurstLength” at offset 0), uncomment the following line:

#

#InitialR2T=no

#

# To globally enable immediate data (i.e., the initiator sends unsolicited data

# with the iSCSI command packet), uncomment the following line:

#

#ImmediateData=yes

#

# To globally disable immediate data (i.e., the initiator does not send

# unsolicited data with the iSCSI command PDU), uncomment the following line:

#

#ImmediateData=no

#

# To globally specify the maximum number of data bytes the initiator can receive

# in an iSCSI PDU from a target, uncomment the following line:

#

#MaxRecvDataSegmentLength=

#

# where is the number of bytes in the range of 512 to (2^24-1)

#

# To globally specify the maximum number of unsolicited data bytes the initiator

# can send in an iSCSI PDU to a target, uncomment the following line:

#

#FirstBurstLength=

#

# where is the number of bytes in the range of 512 to (2^24-1)

#

# To globally specify the maximum SCSI payload that the initiator will negotiate

# with the target for, uncomment the following line:

#

#MaxBurstLength=

#

# where is the number of bytes in the range of 512 to (2^24-1)

#

# To globally specifiy the maximum number of bytes that can be sent over a TCP

# connection by the initiator before receiving an acknowledgement from the

# target, uncomment the following line:

#

#TCPWindowSize=

#

# where is the number of bytes in the range of 512 to (2^24-1)

#

# The global iSCSI operational parameter setting can be overridden on a per

# target basis. An example of setting multiple parameters for just target

# iqn.1987-05.com.cisco:00.0d1d898e8d66.t0 would be:

#

#TargetName=iqn.1987-05.com.cisco:00.0d1d898e8d66.t0

# InitialR2T=no

# ImmediateData=no

# MaxRecvDataSegmentLength=128 * 1024

# FirstBurstLength=262144

# MaxBurstLength=(16 * 1024 * 1024) – 1024

# TCPWindowSize=262144

#

# The global “TCPWindowSize” setting can also be overridden on a per portal

# basis. An example of setting the “TCPWindowSize” for just subnet 10.77.13.0/16

# would be:

#

#Subnet=10.77.13.0/16

# TCPWindowSize=262144

#

# —————-

# Network Category

# —————-

# To localize parameters to targets on a particular network (i.e., to

# override the global settings), you need to use either the “Subnet” or

# “Address” settings. The format for the “Subnet” setting is a.b.c.d/e. An

# example of these settings would be:

#

#Subnet=10.4.100.0/24

#

# The format for “Address” is a.b.c.d. An example of these settings would be:

#

#Address=10.4.100.0

#

# The following parameters can be specified using the network category:

#

# 1) Connection Timeout Settings

# 2) Error Handling Timeout Settings

# 3) TCP Settings

#

# ————————–

# Discovery Address Category

# ————————–

# To localize parameters to targets found on a particular discovery address

# (i.e., to override the global settings), you need to use the

# “DiscoveryAddress” setting. The format for the “DiscoveryAddress” setting is

# a.b.c.d, a.b.c.d:e (where e is a TCP port number), or an instance name. An

# example of these settings would be:

#

#DiscoveryAddress=10.4.100.0

# or

#DiscoveryAddress=10.4.100.1:3260

# or

#DiscoveryAddress=scisrouter1

#

# The following parameters can be specified using the discovery address

# category:

#

# 1) Authentication Settings

# 2) ConnectionTimeout Settings

# 3) Continuous Discovery settings

# 4) AsyncEvent Notification Settings

#

# ——————–

# Target Name Category

# ——————–

# To localize parameters to targets identified by a particular target name

# (i.e., to override the global settings), you need to use the

# “TargetName” setting. The format for the “TargetName” setting is

# either the ‘iqn’ or ‘eui’ format. An example of these settings would be:

#

#TargetName=iqn.1987-05.com.cisco:00.0d1d898e8d66.t0

#

# The following parameters can be specified using the target name category:

#

# 1) CRC Settings

# 2) iSCSI Operational Parameter settings

# 3) Connection Timeout Settings

# 4) Session Timeout Settings

# 5) Error Handling Timeout Settings

# 6) TCP Settings

# 7) Enable/Disable targets

#

Configure the storage system for iSCSI:

Perform the following tasks on the storage system console:

I would recommend taking help from your storage team to make these changes instead of trying it yourself. These steps don’t fall in the scope of an Oracle DBA.

You can ask your storage admin to create luns of the size you require for your oracle database. Also mention that you would like these luns to be visible from all your RAC nodes.

Enter the following command to check whether iSCSI service is running:

iscsi status

iSCSI service is not running

If it is not running, enter the following command to start it and check it again:

iscsi start

iscsi status

iSCSI service is running

Create iSCSI initiator group with the following command:

igroup create -i -t linux

Example:

igroup create -i -t linux ractest

Add iSCSI initiators to the iSCSI initiator group with the following command:

igroup add

where initiator name is defined in the host’s configuration file /etc/initiatorname.iscsi.

Example:

igroup add ractest iqn.1987-05.com.cisco:01.45b2c1c2e7:oradbdev02

Use the following command to see the initiators and initiator groups

igroup show

Example:

igroup show

igroup1 (iSCSI) (ostype: linux):

iqn.1987-05.com.cisco:01.45b2c1c2e7:oradbdev02 (not logged in)

Create volumes that contain LUNs with the following commands:

vol create size {k|m|g|t}

vol options nosnap on # disable snapshots

snap reserve 0 # disable snapshot reservation

snap sched 0 0 0 # disable the automatic snapshot schedule

For example, the following commands create an 2 GB volume /vol/vol1 on the aggregate aggr1:

vol create ractest aggr1 size 2g

vol options vol1 nosnap on

snap reserve vol1 0

snap sched vol1 0 0 0

Create LUNs with the following command:

lun create -s size -t linux lun_path

where lun_path is the LUN’s path name that includes the volume and qtree.

For example, the following commands create a 500 GB LUN /vol/vol1/ractest.lun in the volume /vol/vol1:

lun create -s 2g -t linux /vol/vol1/ractest.lun

You can use the following command to see all LUNs (From OS):

[oracle@oradbdev02 ~]$ sudo sanlun lun show

Password:

filer: lun-pathname device filename adapter protocol lun size lun state

silo: /vol/vol1/ractest.lun /dev/sda host0 iSCSI 2g (2147483648) GOOD

Map the LUN to an igroup with the following command:

lun map lun_path initiator_group

Example:

lun map /vol/vol1/ractest.lun ractest