Oracle 11g in CentOS 6

I’ve installed a basic Oracle server in a VirtualBox image, following the guide on their site and a bunch of other guides, without prior knowledge about the process. This is a basic setup, using CentOS 6, minimal install. The idea is to bring it to a working state, operating system all set up, so that the database administrator can take over without having to worry about changing things outside the Oracle environment. What follows is the steps I took, so I don’t have to spend a few days again searching.

0. System prerequisites

1. Software packages

2. Kernel parameters

3. User and shell limits

4. Install directories

5. Run the installer

6. Post-install


0. System prerequisites:

How should the OS be installed, partitions size, RAM demands, etc.

RAM: 1GB minimum, recommended 2GB or more.

SWAP: depends on the amount of RAM in the system. General guidelines:

Amount of RAM Recommended SWAP
between 1 GB and 2 GB 150% the amount of RAM
between 2 GB and 16 GB the same size as RAM
more than 16 GB 16 GB

 

TMPFS: ( aka shmfs aka /dev/shm) depends on init params MEMORY_TARGET, MEMORY_MAX_TARGET. SGA and PGA mem sizes. Those should default to zero, so whatever size tmpfs is should be fine. Basically it should be set to at least the larger value of MEMORY_TARGET or MEMORY_MAX_TARGET, otherwise oracle will error on startup with “ORA-00845: MEMORY_TARGET not supported on this system” or “ORA-01078: Failure in processing system parameters”. More info here.

DISK SPACE: /tmp must have 1 GB at least. If the current value is too low and you can’t increase it (i.e. OS already installed and no more free space), the environment variables TMP and TMPDIR of the user running oracle (usually “oracle”) can be set to somewhere else. Maximum free space needed by bin + data files is 7 GB, rounded up, for Oracle Enterprise. Depending on instalation type, less is still fine probably. Details. Also, this is just the base install. It’s recommended to keep the production databases, backups, etc. on different file systems, those should require a lot more space, so don’t allocate the whole disk to one partition. Also the database files are usually large files, so fewer inodes and larger blocks on the file system is better.

1. Software packages

The installer is a graphical installer, as in runs in X, as in X forwarding over SSH should work in order to install remotely from an X server. This would be the minimum amount of X-related packages installed on the system, rather than having a full X server. The package needed is xorg-x11-xauth with dependencies, of course, so..

yum install xorg-x11-xauth

To install the rest of the needed packages in one go, do..

yum install \
        binutils \
        compat-libstdc++-33 \
        elfutils-libelf \
        elfutils-libelf-devel \
        elfutils-libelf-devel-static \
        gcc \
        gcc-c++ \
        glibc \
        glibc-common \
        glibc-devel \
        glibc-headers \
        kernel-headers \
        libaio \
        libaio-devel \
        libgcc \
        libgomp \
        libstdc++ \
        libstdc++-devel \
        make \
        sysstat \
        ksh

Some of the scripts used by the installer are written for ksh, so that’s also needed. You might also want unixODBC drivers, they are in the base repository, version 2.2.11 or higher.

2. Kernel parameters

Short and fast version, add these lines to /etc/sysctl.conf then reboot the system:

# ephemeral port range as recommended by IANA
net.ipv4.ip_local_port_range = 49152 65535
# concurrent outstandin requests
fs.aio-max-nr = 1048576
# max number of file handles
fs.file-max = 6815744
# shared mem segment sizes
kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
# semaphores
kernel.sem = 250 32000 100 128
# net packets
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048586

Long version follows. Jump to next section if you want to skip it.

EPHEMERAL PORT RANGE

must be above 9000 for oracle, IANA suggests 49152-65535 for dynamic or private ports, increase if needed

To check the current value To make it permanent
cat /proc/sys/net/ipv4/ip_local_port_range echo “net.ipv4.ip_local_port_range = 49152 65535” >> /etc/sysctl.conf

The values here are the IANA values, although Oracle recommends setting them to 9000-65000.

# /etc/init.d/network restart

will load the new values.

SEM – semaphores

To check the current value To make it permanent
/sbin/sysctl -a | grep sem echo “kernel.sem = 250 32000 100 128” >> /etc/sysctl.conf
cat /proc/sys/kernel/sem

These are recommended values. If the current (default) value of any of the four parameters is higher, keep the higher one.

SHM – shared memory

To check the current value To make it permanent
/sbin/sysctl -a | grep shm
cat /proc/sys/kernel/shmall echo “kernel.shmall = 2097152” >> /etc/sysctl.conf
cat /proc/sys/kernel/shmmax echo “kernel.shmmax = 2147483648” >> /etc/sysctl.conf
cat /proc/sys/kernel/shmmni echo “kernel.shmmni = 4096” >> /etc/sysctl.conf

 

FILE-MAX – maximum number of file handles

Basically, how many open files can there be at one time

To check the current value To make it permanent
/sbin/sysctl -a | grep file-max echo “fs.file-max = 6815744” >> /etc/sysctl.conf
cat /proc/sys/fs/file-max

 

AIO-MAX-NR – concurrent outstanding requests

should be set to avoid I/O subsystem failures

To check the current value To make it permanent
/sbin/sysctl -a | grep aio-max-nr echo “fs.aio-max-nr = 1048576” >> /etc/sysctl.conf
cat /proc/sys/fs/aio-max-nr

 

RMEM_DEFAULT

rmem and wmem are settings related to the network stack, namely the sizes of the receive and send window buffers in TCP/IP

To check the current value To make it permanent
/sbin/sysctl -a | grep rmem_default echo “net.core.rmem_default = 262144” >> /etc/sysctl.conf
cat /proc/sys/net/core/rmem_default

 

RMEM_MAX

To check the current value To make it permanent
/sbin/sysctl -a | grep rmem_max echo “net.core.rmem_max = 4194304” >> /etc/sysctl.conf
cat /proc/sys/net/core/rmem_max

 

WMEM_DEFAULT

To check the current value To make it permanent
/sbin/sysctl -a | grep wmem_default echo “net.core.wmem_default = 262144” >> /etc/sysctl.conf
cat /proc/sys/net/core/rmem_max

 

WMEM_MAX

To check the current value To make it permanent
/sbin/sysctl -a | grep wmem_max echo “net.core.wmem_max = 1048586” >> /etc/sysctl.conf
cat /proc/sys/net/core/wmem_max

 

3. User and shell limits

At this point the OS should be set up properly and we should create a new user, usually “oracle”, that will actually run the whole thing, listener and database and all. This user needs certain shell limits changed/increased from the defaults, like the number of processes it can run and the number of files it can open. Also some environment variables need to be set, that will be done in .bash_profile.

Here’s a small script that will do some of that. It creates the user and the necessary groups then checks if the shell limits are properly set. if they aren’t it’ll output the lines to add to /etc/security/limits.conf. Also checks for user “nobody”, but it doesn’t create it if it’s not there. It should be, in most linux installs, certainly in CentOS 6. Note that there are a number of ways to set the shell limits, but limits.conf works on CentOS 6 and it’s probably the easiest to do.

#!/bin/sh

# adds the necessary users and groups if they don't exist already
#     will ask for password
#
# check if user 'nobody' exists, it should, warns if it doesn't
#
# check shell limits for USRORACLE. Starting values should be:
# Resource shell limit        Resource    Soft limit    Hard limit
# Open file descriptors        nofile        >= 1024        >= 65536
# Processes available        nproc        >= 2047        >= 16384
# Stack segment per process    stack        >= 10240 KB    >= 10240 KB && <= 32768 KB

# groups and user to be created. change these if needed
GRPOINSTALL="oinstall"
GRPDBA="dba"
USRORACLE="oracle"

# check if script is run as root, if it isn't, exit
if [ $(id -u) -ne 0 ]; then
    echo "You need to be root to add users"
    exit 1
fi

addgrp ()
# check for group $1, add if it doesn't exist
{
echo -n "Adding group $1... "
grep "^$1:" /etc/group > /dev/null
if [ $? -eq 0 ]; then
    echo "[EXISTS]"
else
    groupadd $1
    if [ $? -eq 0 ]; then
        echo "[OK]"
    else
        echo "[ERROR]"
        exit 2
    fi
fi
}

addusr ()
# check for user $1, add if it doesn't exist
# ask for password if $2 > 0
{
echo -n "Adding user $1... "
grep "^$1:" /etc/passwd > /dev/null
if [ $? -eq 0 ]; then
    echo "[EXISTS]"
else
    useradd -g $GRPOINSTALL -G $GRPDBA -m $1
    if [ $? -eq 0 ]; then
        echo "[OK]"
        passwd $1
    else
        echo "[ERROR]"
        exit 2
    fi
fi
}

addgrp $GRPOINSTALL
addgrp $GRPDBA
addusr $USRORACLE
echo "Done adding groups/users"

# check for 'nobody'
echo -n "Checking for user nobody... "
grep "^nobody:" /etc/passwd > /dev/null
if [ $? -ne 0 ]; then
    echo "[FAIL - user should exist, add it by hand]"
else
    echo "[OK]"
fi

# check shell limits for user $USRORACLE
echo
LIMITS=""
echo -n "Soft limit for file descriptors... "
ULIMIT=`su - oracle -c "ulimit -Sn"`
if [ $ULIMIT -ge 1024 ]; then
    echo "[OK - $ULIMIT]"
else
    echo "[FAIL - $ULIMIT]"
    LIMITS="$LIMITS oracle        soft    nofile    1024 \n"
fi
echo -n "Hard limit for file descriptors... "
ULIMIT=`su - oracle -c "ulimit -Hn"`
if [ $ULIMIT -ge 65536 ]; then
    echo "[OK - $ULIMIT]"
else
    echo "[FAIL - $ULIMIT]"
    LIMITS="$LIMITS oracle        hard    nofile    65536 \n"
fi
echo -n "Soft limits for number of processes... "
ULIMIT=`su - oracle -c "ulimit -Su"`
if [ $ULIMIT -ge 2047 ]; then
    echo "[OK - $ULIMIT]"
else
    echo "[FAIL - $ULIMIT]"
    LIMITS="$LIMITS oracle        soft    nproc     2047 \n"
fi
echo -n "Hard limits for number of processes... "
ULIMIT=`su - oracle -c "ulimit -Hu"`
if [ $ULIMIT -ge 16384 ]; then
    echo "[OK - $ULIMIT]"
else
    echo "[FAIL - $ULIMIT]"
    LIMITS="$LIMITS oracle        hard    nproc     16384 \n"
fi
echo -n "Soft limits for stack setting... "
ULIMIT=`su - oracle -c "ulimit -Ss"`
if [ $ULIMIT -ge 10240 ]; then
    echo "[OK - $ULIMIT]"
else
    echo "[FAIL - $ULIMIT]"
    LIMITS="$LIMITS oracle        soft    stack     10240 \n"
fi
echo -n "Hard limits for stack setting... "
ULIMIT=`su - oracle -c "ulimit -Hs"`
if [ $ULIMIT == "unlimited" ] ; then
    echo "[OK - $ULIMIT]"
elif [ $ULIMIT -ge 10240 ] ; then
    echo "[OK - $ULIMIT]"
else
    echo "[FAIL - $ULIMIT]"
#    Not sure about this one
#    LIMITS="$LIMITS oracle      hard    stack   10240 \n"
fi

if [ -n "$LIMITS" ]; then
    echo
    echo " Some limits aren't set to minimal values"
    echo " Add the following lines to /etc/security/limits.conf"
    echo -e $LIMITS
    echo " In order to enforce all the limits anyway"
    echo " even those that are correct, add"
    echo " oracle              soft    nproc   2047
 oracle              hard    nproc   16384
 oracle              soft    nofile  1024
 oracle              hard    nofile  65536
 oracle              soft    stack   10240"
fi

Sample output:

Adding group oinstall... [EXISTS]
Adding group dba... [EXISTS]
Adding user oracle2... [OK]
Changing password for user oracle2.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
Done adding groups/users
Checking for user nobody... [OK]

Soft limit for file descriptors... [OK - 1024]
Hard limit for file descriptors... [FAIL - 1024]
Soft limits for number of processes... [FAIL - 1024]
Hard limits for number of processes... [FAIL - 11838]
Soft limits for stack setting... [OK - 10240]
Hard limits for stack setting... [OK - unlimited]

 Some limits aren't set to minimal values
 Add the following lines to /etc/security/limits.conf
 oracle2 hard nofile 65536
 oracle2 soft nproc 2047
 oracle2 hard nproc 16384 

 In order to enforce all the limits anyway
 even those that are correct, add
 oracle2              soft    nproc   2047
 oracle2              hard    nproc   16384
 oracle2              soft    nofile  1024
 oracle2              hard    nofile  65536
 oracle2              soft    stack   10240

Notice the last lines, you can just add these settings to limits.conf and be done with it.

4. Install directories

This is easy enough, create the direcotries where you want it installed and set the apropriate permissions. Something like

# mkdir -p /ora/app/oracle
# chown -R oracle:oinstall /ora/app/oracle
# chmod -R 775 /ora/app/oracle

5. Run the installer

Log in as “oracle”, set the environment variables and start the installer. ORACLE_BASE needs to be set to the base directory created above and ORACLE_SID to the name of the database. ORACLE_HOME and TNS_ADMIN need to be unset. Also, if /tmp has less than 1 GB free space, set TMP and TMPDIR to some other directory. Assuming bash:

$ export ORACLE_BASE=/ora/app/oracle
$ export ORACLE_SID=sales
$ unset ORACLE_HOME
$ unset TNS_ADMIN

Use setenv and unsetenv in a C shell. You will have a chance to change the location during install anyway. Now run the installer, X forwarding needs to be enabled if you’re doing it over ssh.

$ cd /whatever/dir/it/was/unpacked/in/
$ ./runInstaller

The program will also check if the system was properly set up, kernel parameters et al, so you can still fix things if it says something is wrong. Note that it will complain about the ephemeral ports, you can change it if you think more connections are needed or just don’t like it being there.

6. Post-install

If everything went according to plan oracle should now be started. Check for the listener by doing

$ ps -aux | grep tnslsnr

If Oracle Enterprise Manager was also installed it should be running on port 1158 on https (https://hostname:1158/em/)

There’s a few more things to do, setting the environment variables for the user and setting up a script to automatically start oracle on next system boot. The installer does no such thing. Assuming the user is “oracle” and oracle’s shell is bash, add a few lines like the following to /home/oracle/.bash_profile :

export ORACLE_HOME=/ora/app/oracle/product/11.2.0/sales
export ORACLE_SID=sales
export ORACLE_UNQNAME=sales

PATH=$PACTH:$ORACLE_HOME/bin
export PATH

ORACLE_HOME depends on the version of oracle and the database name, ORACLE_SID and ORACLE_UNQNAME contain the database name.

Now for the startup script. In /etc/oratab there’s a list of instances (databases) that oracle has. On a fresh install there will be only one, named “sales” in this document. Change the last field from “N” to “Y” for the instances that you want to start automatically. “W” means that dbstart auto-starts the database only after the Automatic Storage Management instance is started, if you have such a thing. If in doubt, use “Y”. After that a script should be added to /etc/init.d to actually start the listener and whatnot. There’s a number of them, I’m using this one:

#!/bin/sh
# chkconfig: 345 99 10
# description: Oracle auto start-stop script.
#
# Set ORA_HOME to be equivalent to the $ORACLE_HOME
# from which you wish to execute dbstart and dbshut;
#
# Set ORA_OWNER to the user id of the owner of the
# Oracle database in ORA_HOME.
ORA_HOME=/ora/app/oracle/product/11.2.0/sales
ORA_OWNER=oracle

ORACLE_HOME=/ora/app/oracle/product/11.2.0/sales
ORACLE_SID=sales
ORACLE_UNQNAME=sales

if [ ! -f $ORA_HOME/bin/dbstart ]
then
    echo "Oracle startup: cannot start"
    exit
fi

case "$1" in
'start')
    # Start the Oracle databases:
    # The following command assumes that the oracle login
    # will not prompt the user for any values
    su - $ORA_OWNER -c "$ORA_HOME/bin/emctl start dbconsole"
#   su - $ORA_OWNER -c "$ORA_HOME/bin/lsnrctl start"
    su - $ORA_OWNER -c "$ORA_HOME/bin/dbstart $ORA_HOME"
        touch /var/lock/subsys/dbora
    ;;
'stop')
    # Stop the Oracle databases:
    # The following command assumes that the oracle login
    # will not prompt the user for any values
    su - $ORA_OWNER -c "$ORA_HOME/bin/emctl stop dbconsole"
#   su - $ORA_OWNER -c "$ORA_HOME/bin/lsnrctl stop"
    su - $ORA_OWNER -c "$ORA_HOME/bin/dbshut $ORA_HOME"
        rm -f /var/lock/subsys/dbora
    ;;
esac

Simple and good enough. I think i got it from here, although not sure. It also starts the Enterprise Manager, comment it out if it’s not there. Change the variables as needed, copy the thing in /etc/init.d/dbora, set it owned by root and executable

# chown root:dba /etc/init.d/dbora
# chmod 750 /etc/init.d/dbora

then set it to “on” when the server boots

# chkconfig --level 345 dbora on

Try stopping/starting the database using the script, reboot the system to check that everything’s in order, done.