Manual : Oracle 10g RAC on Red Hat Linux Enterprise 4 - ASM mode

 Oracle 10g

 

RAC

(Real Application Cluster)

 

RHEL4

 

ASM mode

 

 

 

Download Software

 

1. Minimum of 40GB Spare Disk

2. VM-Ware Work Station 8.0

3. RHEL4-U3-i386-AS-disc1~4.iso

4. Oracle 10g database package zip file

5. Oracle 10g clustware package zip file

6. Oracle 10g 1024 patch

 

 

 

 

Rac 1 Server Operating System Installation 

Rac1 Server

OS : Linux

Version : RHEL 4

 

Virtual Machine Setting

location : C:\Virtual Machines\rac1

Memory : 1024

Harddisk : 20G scsi type

Network : NAT, Host Only

 

RHEL 4 Setting

Partition : / - 5000mb

   /boot - 100mb

   /var - 1000mb

   swap - 1500mb

   /home - rest all

 

Network Devices : eth0(NAT), eth1(host-only)

Host Name : rac1

 

 

Set the Virtual Server name and location

 

 

Select "Store vitual disk as a single file"

 

 

Remove non required devices; printer, usb and etc. 

Add Host-only network.

 

Use 'RHEL-4-U3-i386~-disk1.iso'

 

 

Run RAC1 Server.

 

 

Select 'Manually Partition'

 

 

Partitioning disk by shown green label above

 

Click 'Next' on "BOOT LOADER CONFIGURATION"

 

 

Set both 'eth0' and 'eth1' as Network Devices and name it as 'rac1'

 

 

Disable Firewall and SELinux.

Then Click 'Proceed'

Select Language

Set Time Zone

Set root password on 'rac1' server

 

 

Select 'Cusomize software package'

 

 

Select 'Editors', 'Graphical Internet' and 'Text-based Internet' for Applications package.

 

 

Select only 'Server Configuration Tools' for Servers package and uncheck 'Web Server' and 'Windows File Server'

 

 

 

Select all packages for Development

 

 

Select 'Administartion Tools' and 'System Tools

Click 'Details' on System Tools package.

 

 

Select 'sysstat - The sar and iostat system monitoring commands', which is the required rpm on oracle 10g.

Then Click 'Ok'

Click 'Next'

 

Click 'Next' to start install

 

 

Insert 'Red Hat Enterprise Linux AS disc from 1 to 4 disc' when noticed

 

Skip registartion part.

Click 'Next' by giving default options.

Then Installation has done

 

Login as 'root'

 

 

Change display setting to work on proper size of grapical interface later.

 

 

Set as 'LCD Panel 1024x768'

Click 'Ok'

 

 

Click tab 'Settings'

Click 'Resolution'

Select '1024x768'

Then Click 'Ok'

 

 

It will adjust to x-window after reboot, but don't reboot now.

You'll do it later on anyway.

 

 

Transfer required Oracle 10g patch using WinSCP software

 

Early patch transferation will save our time.

Patch :

 

Trasfer to rac1 server in /home/oracle/pkg directory

]#mkdir -p /home/oracle/pkg

]#ifconfig

=>rac1 ip : 192.168.230.131

 

Open WinSCP software

 

Enter rac1 IP address

Login as root

 

Click 'Yes'

 

 

Copy 3 zip files to /home/oracle/pkg

 

 

Turn off unneccesory Services

# chkconfig --level 123456 xinetd off
# chkconfig --level 123456 sendmail off
# chkconfig --level 123456 cups off
# chkconfig --level 123456 cups-config-daemon off
# chkconfig --level 123456 smartd off
# chkconfig --level 123456 isdn off
# chkconfig --level 123456 pcmcia off
# chkconfig --level 123456 iptables off

 

Setting up the prerequisites for RAC 1

 

1. Set RAC1 Server ip and 'hosts' file

2. Install rpm packages

3. Set process range values; 'sysctl.conf'file and 'limits.conf' file

4. Set server time; 'modprobe.conf' file and 'rc.local' file 

5. Create user and group to manage RAC install

6. Set up root user home directory, '.bash.profile' file

7. Set up new created user profile, '.bash.profile' file

 

 

1. RAC1 IP and '/etc/hosts' file

#ifconfig

#netstat -nr

#neat

 

'eth0' will be used as public ip

'eth1' will be used as private one. You can set this ip whatever you want within 255.255.255.0 'C' class range because it is for internal use only between rac1 and rac2 server.

=> eth0 = public ip = rac1 = 192.168.230.131

=> eth1 = private ip = rac1-priv = 192.168.195.131

if eth1 has different ip value from eth0, such as 192.168.230.131 as eth0 and 192.168.195.140 as eth1, it is also okay. Just keep that mind eth0 is for public use so that should have external accesibility, but eth1 is for internal use between rac1 and rac2 server as private one, meaning set rac1-priv and rac2-priv in same submask range.

 

 

=> Gateway = 192.168.230.2

 

 

Set 'eth0' IP addresses

 

 

Set 'eth1' IP addresses

No gateway addresses on 'eth1'

 

 

Set Hostname as 'rac1' 

DNS as '168.126.63.1'

 

 

Save the setting

 

 

Restart the network

]# service network restart

 

Set up '/etc/hosts' file

]# vi /etc/hosts

 

Add following lines on 'hosts'file

#public

192.168.230.131    rac1

192.168.230.132    rac2

 

#private

192.168.195.131    rac1-priv

192.168.195.132    rac2-priv

 

#vip

192.168.230.31    rac1-vip

192.168.230.32    rac2-vip

 

 

Remove 'rac1'

 Test IP

]#ping rac1

]#ping rac1-priv

]#ping 168.126.63.1

 

'rac2', 'rac2-priv' and 'vip' won't work for now.

 

2. Check required rpm packages

 

binutils-2.15.92.0.2-21

compat-db-4.1.25-9

compat-gcc-32-3.2.3-47.3

compat-gcc-32-c++-3.2.3-47.3

compat-libstdc++-33-3.2.3-47.3

compat-libgcc-296-2.96-132.7.2

control-center-2.8.0-12.rhel4.5

cpp-3.4.6-3

gcc-3.4.6-3

gcc-c++-3.4.6-3

glibc-2.3.4-2.25

glibc-common-2.3.4-2.25

glibc-devel-2.3.4-2.25

glibc-headers-2.3.4-2.25

glibc-kernheaders-2.4-9.1.98.EL

gnome-libs-1.4.1.2.90-44.1

libaio-0.3.105-2

libstdc++-3.4.6-3

libstdc++-devel-3.4.6-3

make-3.80-6.EL4

openmotif-2.2.3-10.RHEL4.5

openmotif21-2.1.30-11.RHEL4.6

pdksh-5.2.14-30.3

setarch-1.6-1

sysstat-5.0.5-11.rhel4 

xscreensaver-4.18-5.rhel4.11

 

Presense of required rpm package can be vary by version of OS. For RHEL 4, there is nothing to do since we have added 'sysstat-5.0.5-11.rhel4' on software package selection.

For other version CENT OS or else, check if correct rpm is installed.

]# rpm -qa | grep rpmpackage*

 

Since our linux version is RHEL4, we have all required rpm package set. Nothing to do.

 

3. Set Processor Values

]# vi /etc/sysctl.conf

add following lines on the content

 

kernel.shmall = 2097152

kernel.shmmax = 2147483648

kernel.shmmni = 4096

kernel.sem = 250 32000 100 128

fs.file-max = 65536

net.ipv4.ip_local_port_range = 1024 65000

net.core.rmem_default = 4194304

net.core.rmem_max = 4194304

net.core.wmem_default = 262144

net.core.wmem_max = 262144

 

]# vi /etc/security/limits.conf

add following lines on the content

 

oracle soft nproc 2047

oracle hard nproc 16384

oracle soft nofile 1024

oracle hard nofile 65536

 

]# vi /etc/pam.d/login

add following lines on the content

 

session required pam_limits.so

 

 

4. Set Sever Time

]# vi /etc/modprobe.conf

add following lines on the content

 

options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180

 

]# modprobe -v hangcheck-timer

check system timer

 

 

 

]# vi /etc/rc.local

add following lines on the content

 

/sbin/modprobe hangcheck-timer rdate -s time.bora.net

 

 

5. Create User and Group

]# groupadd -g 5000 dba

]# useradd -g dba oracle

]# passwd oracle

 

 

 

6. Setting up root profie and oracle user profile

 

]# vi ~/.bash_profile

add following lines on content

 

PATH=$PATH:$HOME/bin:/home/oracle/product/10g/crs/bin

 

 

 

 

]# vi /home/oracle/.bash_profile

add following lines on content

 

export EDITOR=vi

export LD_ASSUME_KERNEL=2.4.19

export ORACLE_BASE=/home/oracle

export ORA_CRS_HOME=$ORACLE_BASE/product/10g/crs

export ORACLE_HOME=$ORACLE_BASE/product/10g/db

export ORA_ASM_HOME=$ORACLE_BASE/product/10g/asm

export ORACLE_SID=rac1

export LANG=ko_KR.UTF-8

export ORACLE_TERM=xterm

export NLS_LANG=AMERICAN_AMERICA.KO16MSWIN949

export ORA_NLS33=$ORACLE_HOME/ocommon/nls/admin/data

export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib:/usr/local/lib

export PATH=$PATH:$ORACLE_HOME/bin:/$ORA_CRS_HOME/bin

export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

 

 

Cautions that copy this content will automatically add '?' on the last line. It should be removed.

=> export CLASSPATH~ 

 

 

 

Create Shared Disks on 'storage' DB

Shutdown to work on creating disks

]#init 0

 

Route : C:\virtual disk\storage\'new shared disks'

Disks :

ocr1.vmdk    300M    1:0 (SCSI)

ocr2.vmdk    300M    1:1 (SCSI)

vote1.vmdk   300M    1:2 (SCSI)

vote2.vmdk   300M    1:3 (SCSI)

vote3.vmdk   300M    1:4 (SCSI)

asm1.vmdk    5G       1:5 (SCSI)

asm2.vmdk    5G       1:6 (SCSI)

asm3.vmdk    5G       1:8 (SCSI)

 

 

Make new folder and name it 'storage'

 

 

Edit virtual machine setting

Add Hard Disk, then click 'Next'

Create a new virtual disk, then click 'Next'

 

 

Select 'Persistent' as 'Independent' mode

Click 'Next'

 

Set size 0.3GB ( 300MB ) for 'ocr1.vmdk' disk

Select 'Allocate all disk space now'

Select 'Store virtual disk as a single file'

Then click 'Next'

 

Browse disk file

Set as \virtual machine\ storage

Name disk file as 'ocr1.vmdk'

Click 'Open'

 

 

Click 'Finish'

 

 

Click 'Advanced'

Set device node to 'SCSI 1:0'

Click 'Ok'

 

Repeat the previous hard disk creation steps on following disks

ocr2.vmdk 300M 1:1 (SCSI)

vote1.vmdk 300M 1:2 (SCSI)

vote2.vmdk 300M 1:3 (SCSI)

vote3.vmdk 300M 1:4 (SCSI)

asm1.vmdk 5G 1:5 (SCSI)

asm2.vmdk 5G 1:6 (SCSI)

asm3.vmdk 5G 1:8 (SCSI)

 

 

When done with disks creation on storage, you'll see similar to above results.

New Hard Disk 2~9

asm1~3.vmdk, ocr1~2.vmdk and vote1~3.vmdk on 'storage' folder.

 

'Storage' disk and 'rac1' disk must be seperated because they are different server.

 

Edit 'rac1.vmx' file on 'rac1' folder using text editor.

 

Open 'rac1.vmx' file with notepad or and kinds of text editor.

add following lines on the content

 

disk.locking = "FALSE"

diskLib.dataCacheMaxSize = "0"

scsi1.sharedBus = "virtual“

 

scsi1:0.deviceType = "disk“

scsi1:1.deviceType = "disk“

scsi1:2.deviceType = "disk“

.

.

. 

scsi1:8.deviceType = "disk“ 

 

 

Save the contents

 

Start the virtual machine

 

If you have just copy and paste above contents, then you will see this error pop up. Open the 'rac1.vmx' file again, and make sure any misstype on new added lines, such as " end in every lines or wrong space.

 

 

Enter any key to continue

 

 

Enter 'Configure'

 

 

Wait

 

 

Proceed clicking 'Next' and skip registartion part.

 

Login as root user

 

    Partitioning new created shared disks

 

new disks :

sdb, sdc, sdd, sde, sdf, sdg, sdh, sdi

 

 

new partition as primary disk :

sdb1, sdc1, sdd1, sde1,sdf1, sdg1, sdh1, sdi1

 

]# fdisk /dev/sdb

Command : n

Command Action : p (primary partition)

Partition number : 1

First cylinder : enter

Last cylinder : enter

Command : w 

 

.

.

.

.

.

.

repeat above partitioning steps on following disks

 sdc, sdd, sde, sdf, sdg, sdh, sdi

 

new partition as primary disk :

sdb1, sdc1, sdd1, sde1,sdf1, sdg1, sdh1, sdi1

 

 

 

Creating raw devices and change permission

 

All RAC nodes must use the same raw devices.

 

]# vi /etc/sysconfig/rawdevices

add following lines on the content

 

/dev/raw/raw1 /dev/sdb1

/dev/raw/raw2 /dev/sdc1

/dev/raw/raw3 /dev/sdd1

/dev/raw/raw4 /dev/sde1

/dev/raw/raw5 /dev/sdf1

/dev/raw/raw6 /dev/sdg1

/dev/raw/raw7 /dev/sdh1

/dev/raw/raw8 /dev/sdi1

 

Restart the rawdevice service

]# service rawdevices restart

]# ll /dev/raw/

 

 

 

    Chaging permissions on the raw devices

 

]# vi /etc/udev/permissions.d/50-udev.permissions

modify following contents

 

line-113: raw/*:root:disk:0660 

        =>  raw/*:oracle:dba:0660   

 

 

Restart rawdevice

]# service rawdevices restart

]# ll /dev/raw

 

 

Shutdown

]# init 0

 

 

Cloning rac1 to rac2

 

Create 'rac2' folder in C:\virtual machine\

 

 

 

Copy 'rac1.vmdk' and 'rac1.vmx' in rac1 folder.

Paste in C:\virtual machine\rac2

 

 

 

Change diplay name to rac2 and also edit the content of 'rac1.vmx' file

 

edit the contents only

 

Open the virtual machine of 'rac1.vmx' in rac2 folder

 

 

Then Run it.

 

Setting up rac2 server

 

 

Click 'I copied it'

 

Enter any key

 

Enter 'Remove Configuration'

Enter 'Remove Configuration' again

 

 

Enter 'Configure'

 

Press space bar to select 'use dynamic IP~ (DHCP)'

Enter 'Ok'

Enter 'Configure'

 

Again..

Press space bar to select 'use dynamic IP~ (DHCP)' again

Enter 'Ok' again

Enter 'Configure'again

 

Login as root

 

 

Setting up rac2 network

 

]# cat /etc/hosts

192.168.230.132    rac2

192.168.195.132    rac2-priv

 

]# neat

eth0

ip : 192.168.230.132

subnet : 255.255.255.0

gateway : 192.168.230.2

 

eth1

ip : 192.168.195.132

subnet : 255.255.255.0

 

DNS

hostname : rac2

dns : 168.126.63.1

 

 

 

 

]# service network restart

 

Power on 'rac1' virtual machine and login as oracle user

]# ping rac1

]# ping rac1-priv

 

 

 

 

Changing rac2 profile 

 

 ]# vi /home/oracle/.bash_profile

modify following line

 

export ORACLE_SID=rac2

 

 

Reboot 'rac1' virtual machine and 'rac2' virtual machine

Login as oracle user

 

Configure SSH on each rac

Perform following tasks on each rac 

 

On rac1 and rac2 as oracle user

]$ mkdir .ssh

]$ ssh-keygen -t rsa

Enter file in which to save the key

(/home/oracle/.ssh/id_rsa) : enter

Enter passphrase : enter

Enter same passphrase again : enter

Identification has saved in 'id_rsa.' file

Pulic key has saved in 'id_rsa.pub.'

 

Permit each rac to access server without password

 

Sourcing the public key of 'id_rsa.pub' to 'authorized_keys' file on each rac

Copy 'authorized_keys' file to each rac authorize access without password

 

On rac1 and rac2 

Perform the following tasks on each node
]$cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

]$scp ~/.ssh/authorized_keys rac2:~/.ssh/authorized_keys (on rac1)

Are you sure you want to continue connecting (yes/no) ? yes

oracle@rac2 's password : type password

 

]$scp ~/.ssh/authorized_keys rac1:~/.ssh/authorized_keys (on rac2)

 

Equivalance Check

 

Perform following tasks on each node

If password is not asked to input, it is working out well.

 

]$ssh rac1 date
]$ssh rac2 date
]$ssh rac1-priv date
]$ssh rac2-priv date

 

Rac 1

Succeed

Rac 2

Succeed

Setting up patch before running Clustware

 

]$su -

]#chown oracle.dba /home/oracle/pkg

]#chown oracle.dba /home/oracle/pkg/*

]#su - oracle

]$chmod 755 pkg/*

 

Unzip patch package in /home/oracle/pkg

]$unzip 10201_clusterware_linux32.zip && unzip 10201_database_linux32.zip

&& unzip p6810189_10204_LINUX-x86.zip

]$rm -rf *.zip

 

]$cd /home/oracle/pkg/clusterware

]$./runInstaller

If error occurs regarding permission, do this

]$chmod 755 /home/oracle/pkg/clusterware/*

]$chmod 755 /home/oracle/pkg/clusterware/install/*

]$chmod 755 /home/oracle/pkg/clusterware/install/.oui