Upgrade ODA

Below the steps to Upgrade the ODA Baremetal from 2.10. to, 
probably for earlier version the steps should be the same...

1.) Upgrade to (20690087)

*** Pre-Requisites:
  • Passwords:
    • ROOT: 
    • SYSASM: 
    • ILOM:
    • SYSDBA: 
    • ORACLE: 
    • GRID:
 Check: ODA Supplemental Pre-Patch Heath Check (Doc ID 2140853.1)
oakcli manage cleanrepo --ver <Cleaning the OLD Versions>
oakcli validate -c <NODE NAME> -prechecks
oakcli validate -a
  • Disable cron jobs
     cd ~
       crontab -l > crontab.'date "+%Y%m%d"'
       ls -latr crontab.*
       crontab < /dev/null
 Backup the .ssh directory 
  • Blackout EM agents on each ODA node
 export ORACLE_HOME=/u01/app/oracle/product/agent12c/core/
    cd /u01/app/oracle/product/agent12c/core/
   ./emctl status agent
   ./emctl start blackout ODAPatch_${HOSTNAME} -nodeLevel
   ./emctl stop agent
   ./emctl status agent
  • Checking the Master/Slave:
[root@SERVER1 ~]# oakcli show ismaster
OAKD is in Slave Mode

[root@SERVER2 ~]# oakcli show ismaster
OAKD is in Master Mode

On this case we need restart OAK on node 2:

oakcli restart oak
  • Confirm that both nodes version are up to date
  • identical versions on each node (minus ASR)
         # oakcli validate -d -f /tmp/$HOSTNAME-OAKvalidate.log
  • ASM Volume : the volume DATAFSVOL mounted on /odadatafs must exist
    check: select * from V$ASM_ACFSVOLUMES;
    -AS root:
    cd /
    mkdir odadatafs
    chmod 770 odadatafs
    chown oracle:asmdba odadatafs
    As grid:
    sqlplus / as sysasm
    ALTER DISKGROUP RECO ADD VOLUME DATAFSVOL SIZE 10G; select volume_name, volume_device from v$asm_volume; /dev/asm/datafsvol-20 – RCW/dev/asm/datafsvol-190 - RCS Exit; -   Check the volume name on:ls –tlr /dev/asm/ or select * from v$asm_volumeAs root:/sbin/mkfs -t acfs -b 4k /dev/asm/datafsvol-190 -n "DATAFSVOL"  // ASMFSVOL -   Register FS/sbin/acfsutil registry -f -a /dev/asm/datafsvol-190 /odadatafs
    The ASM Cluster File System should now be mounted on all RAC nodes. 
    If it not, then issue the following command on each node to mount it. 
    As root
    /bin/mount -t acfs /dev/asm/datafsvol-20 /odadatafs Check the FS
    as grid:
    crsctl stat res ora.reco.datafsvol.acfs
    In case of the Filesystem not registered on the CRS, please execute the command below:
    As root:
    /sbin/acfsutil registry -d /odadatafs  // Delete the registry on CRS
    /u01/app/ add filesystem -d /dev/asm/datafsvol-190 -v DATAFSVOL -g RECO -m /odadatafs
    As root:
    /u01/app/ start res ora.reco.datafsvol.acfs/u01/app/ stat res ora.reco.datafsvol.acfsNAME=ora.reco.datafsvol.acfsTYPE=ora.acfs.typeTARGET=ONLINE                  , ONLINESTATE=ONLINE on SERVER1, ONLINE on SERVER2         -  Check AUDIT files on $GRID_HOME/rdbms/audit, if you have a lot of files, you should delete those files before start the upgrade on Grid Infrastructure. "opatch auto" or "rootcrs.pl -unlock" or "roothas.pl -unlock" or addNode.sh Hangs Due to Many Audit Files in GRID_HOME (Doc ID 1352884.1)                
    A.) Check BACKUPS for the database list below:
    B.) Stop Apps / Blackout on OEM12c
    C.) Execute a clear shutdown / startup. To check if there is some 
    issue before start the upgrade;
    D.) Check Alert.log for all databases after startup; (BOTH NODES)
    E.) Copy the files below to servers (ROOT) both nodes:
           oakcli unpack -package /tmp/p20690087_121230_Linux-x86-64_1of2.zip
           oakcli unpack -package /tmp/p20690087_121230_Linux-x86-64_2of2.zip
    If face some problems with space, please check the document below:
    ODA (Oracle Database Appliance): Cleanup trace and log files 
    Using purgeODALog Tool (Doc ID 2081655.1)
    F.) After UNPACK the files on both nodes, you can remove as ROOT on both nodes:
    rm /tmp/p20690087_121230_Linux-x86-64_1of2.zip
    rm /tmp/p20690087_121230_Linux-x86-64_2of2.zip
    Note: After oakcli -unpack, you need to have at least 3GB free.
    G.) Get Linux Version, to check after, what has been changed:
           cat /etc/issue
           uname -a
           - Check the components that will be Patched (both nodes)
           oakcli update -patch --verify
           - Validate PATCH
           oakcli validate -c ospatch -ver
           - Check Space
           Before the patching, please check your system to ensure you have 
    enough space for /root, /tmp, /u01, /opt to avoid unable to 
    uncompressing patching file.
    H.) *** ILOM / BIOS (In case of failure on NORMAL process --INFRA)
    ILOM Oracle Integrated Lights Out Manager (Doc ID 1424493.1)
    Patch 20073310: Oracle Sun Server X4-2 SW 1.2.1 - FIRMWARE PACK
    New ILOM Version: r94217
                 - Current:
                 [root@SERVER1 ~]# ipmitool sunoem cli
                 Connected. Use ^D to exit.
                 -> version
                 SP firmware
                 SP firmware build number: 88185
                 SP firmware date: Fri Mar 28 11:23:54 CST 2014
                 SP filesystem version: 0.1.23
                 -> show /SYS/MB/BIOS
                        show /SYS/MB/BIOS
                                     type = BIOS
                                     ipmi_name = MB/BIOS
                                     fru_name = SYSTEM BIOS
                                     fru_manufacturer = AMERICAN MEGATRENDS
                                     fru_version = 25010604
                                     fru_part_number = APTIO
                 - Check Version:
                 oakcli update -patch --verify
                 - Stop All resources:
                 - List all homes
                 oakcli show dbhomes
    Oracle Home Name      Oracle Home version                  Home Location                          
    ----------------      -------------------                  ------------                           
    OraDb11204_home1,18031740)       /u01/app/oracle/product/
    OraDb11204_home2,18031740)       /u01/app/oracle/product/
                 - Stop all resources
                   grid# crsctl stop res -all
    - Stop SCAN_LISTENER on both nodes
                 srvctl status scan_listener
                 srvctl stop scan_listener
                 - Check date/time on both nodes
                 ssh SERVER1 date
                 ssh SERVER2 date
                 - Check NTP on both Nodes
                 service ntpd status
                 - Check/Stop CRS on both nodes;
                 crsctl check crs
                 crsctl disable crs
                 crsctl stop crs
                 - From the ILOM : stop the operating system (both nodes)
                        -> stop /SYS
                        Are you sure you want to stop /SYS (y/n)? y
                        Stopping /SYS
                        * Wait until power_stat=Off
                        -> show /SYS
                 - Patch ILOM:
                 ILOM Version: r94217
                 load -source scp://<OSUSER>@<IP ADDRESS><PATH>/ILOM-3_2_4_20_a_r94217-Sun_Server_X4-2.pkg
                 - Verify:
                        ipmitool sunoem cli
                        show /SYS/MB/BIOS
                 - Start OS
                        start /SYS
                 - Start the Grid Infrastructure ( root@odabase )
                        [root]# export ORACLE_HOME=/u01/app/
                        [root]# $ORACLE_HOME/bin/crsctl enable crs
                        [root]# $ORACLE_HOME/bin/crsctl start crs
                        Check both nodes:
                        crsctl check crs
  • Apply the patch to the Infra components (1o. this, if face some problems do ILOM manually)
             - Stop all resources
               grid# crsctl stop res -all
             - As root from the first node:
             oakcli update -patch --verify

              - Take the root password for both nodes *****
             - As root from the first node: (TAKE THE TIME)
             oakcli update -patch --infra

              - Verify pos-upgrade (TAKE THE TIME):
             oakcli update -patch --verify

       J.) Apply the Patch to the GI components. 
             **CRS must to be running
             - Verify the cluster status (TAKE THE TIME):
               grid# crsctl stat res -t
             - Stop all resources
               Check if all resources are stopped. IF NOT
  grid# crsctl stop res -all

             - Verify the cluster status
               grid# crsctl stat res -t
             - As root from the first Node:
               oakcli update -patch --gi

              - Verify pos-upgrade (TAKE THE TIME): Check the GI_HOME
               oakcli update -patch --verify
             - Check NTP on both Nodes
               service ntpd status
       K.) Apply the Patch to the Database. 
       Dataguard: Oracle Database Appliance Patching with Standby (Doc ID 1900412.1)

              - As root from the first node: (TAKE THE TIME)
             oakcli update -patch --database

              - Verify pos-upgrade (TAKE THE TIME): Check the DB_HOME
             oakcli update -patch --verify           

             - Start all instances again:
             grid# crsctl start res -all     

- Check SCAN_LISTENER on both nodes
             srvctl status scan_listener
             - In case of not started:
             srvctl start scan_listener
- Check PSU applied for each database: (EXCEPT DATAGUARDS)

Col comments format a40;
   select comments, version, bundle_series
from sys.registry$history
where bundle_series = 'PSU'
order by action_time;

              Current Version:
COMMENTS                                 VERSION                        BUNDLE_SERIES
---------------------------------------- ------------------------------ ------------------------------
PSU                                        PSU

In case of not applied the CATBUNDLE must be applied manually.
  • Upgrade Database in case of requested:
pscp p19520042_121230_Linux-x86-64.zip root@SERVER1:/tmp
oakcli unpack -package /tmp/p19520042_121230_Linux-x86-64.zip
oakcli create dbhome --version
Upgrade DB:
Crsctl start res –all
As root:
oakcli upgrade database --db <DB NAME>  -to OraDb12102_home1 ß HOME NAME
After upgrade the database please check the TIMEZONE:
Scripts to automatically update the RDBMS DST (timezone) version in an 11gR2 or 12cR1 database . (Doc ID 1585343.1)

       L.) Kerberos (Check First)
       - After patching the Infra, the Kerberos configuration is deactivated.
       To set it active again, type the following commands on both nodes connected as root on the ODA_BASE
                   /etc/init.d/autofs start
                    chkconfig autofs on

       M.) Check Dataguards, if needed


shut immediate;
*** References:

ODA DOM0 Lost Network Communications Between Nodes While running 
'oakcli clone vm' or Copy (cp) Commands of Large Files over NFS (Doc ID 1665055.1)
Oracle Database Appliance - 12.1.2 and 2.X Supported ODA Versions & Known Issues (Doc ID 888888.1)
Oracle Database Appliance Patching with Standby (Doc ID 1900412.1)
https://updates.oracle.com/Orion/Services/download?type=readme&aru=18299059Oracle Sun Server X4-2 Platform Software Release 1.2.1 
 Firmware Pack

ILOM Oracle Integrated Lights Out Manager (Doc ID 1424493.1)


How to upgrade the firmware using the ILOM CLI , for SPARC CMT Servers (Doc ID 2101665.1)
*** Know issues
After failed patch: BEFORE RE-EXECUTE
oakcli update -patch –clean
After a failed –gi installation, you can find an issue with:
Oakcli show ismaster
[root@SERVER1 log]# oakcli show ismaster

OAKD is in Slave Mode

[root@SERVER2 oracle]# oakcli show ismaster
OAKD is in Slave Mode
Both nodes are is SLAVE MODE

To fix in both nodes:

Vi /opt/oracle/oak/install/oakdrun
[root@SERVER1 log]# cat /opt/oracle/oak/install/oakdrun


cp /opt/oracle/oak/install/oakdrun /opt/oracle/oak/install/oakdrun_20160820.bkp
cp /opt/oracle/oak/install/oakdrun_20160820.bkp /opt/oracle/oak/install/oakdrun

Change to : 
oakcli stop oak
oakcli start oak
oakcli show ismaster

In case of errors on update: Check this directory

oakcli update -patch --database
After update database check the catalog:
prompt "11g"
select * from registry$history;

prompt "12c"
select * from registry$sqlpatch;
oakcli delete dbhome -oh <oracle_home>

To connect on ILOM:
After you have the IP address, replace it in the next URL:

https://<IP Address>/iPages/i_login.asp - Node 1
https://<IP Address>/iPages/i_login.asp - Node 2

 Restart ODA: Is better do using ILOM
2 Options (ILOM or Server)
reset /SYS , restart the server
start /SP/Console, to see the reboot status

Shutdown –r now

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s