Upgrade ODA

Below the steps to Upgrade the ODA Baremetal from 2.10. to 12.1.2.3.0, 
probably for earlier version the steps should be the same...

1.) Upgrade to 12.1.2.3.0 (20690087)
SERVER1
SERVER2


*** Pre-Requisites:
  • Passwords:
    • ROOT: 
    • SYSASM: 
    • ILOM:
    • SYSDBA: 
    • ORACLE: 
    • GRID:
 Check: ODA Supplemental Pre-Patch Heath Check (Doc ID 2140853.1)
oakcli manage cleanrepo --ver 2.10.0.0.0 <Cleaning the OLD Versions>
oakcli validate -c <NODE NAME> -prechecks
oakcli validate -a
  • Disable cron jobs
     cd ~
       crontab -l > crontab.'date "+%Y%m%d"'
       ls -latr crontab.*
       crontab < /dev/null
 Backup the .ssh directory 
  • Blackout EM agents on each ODA node
 export ORACLE_HOME=/u01/app/oracle/product/agent12c/core/12.1.0.4.0
    cd /u01/app/oracle/product/agent12c/core/12.1.0.4.0/bin
   ./emctl status agent
   ./emctl start blackout ODAPatch_${HOSTNAME} -nodeLevel
   ./emctl stop agent
   ./emctl status agent
  • Checking the Master/Slave:
[root@SERVER1 ~]# oakcli show ismaster
OAKD is in Slave Mode

[root@SERVER2 ~]# oakcli show ismaster
OAKD is in Master Mode

On this case we need restart OAK on node 2:

oakcli restart oak
 
  • Confirm that both nodes version are up to date
  • identical versions on each node (minus ASR)
Example:
         # oakcli validate -d -f /tmp/$HOSTNAME-OAKvalidate.log
  • ASM Volume : the volume DATAFSVOL mounted on /odadatafs must exist
    https://oracle-base.com/articles/11g/acfs-11gr2
    http://docs.oracle.com/cd/E11882_01/server.112/e18951/asmviews.htm#OSTMG94187
    https://gjilevski.com/2010/08/28/asm-cluster-file-system-acfs-in-oracle-11gr2/
    http://www.solvemyissue.com/2014/08/oda-how-to-create-and-delete-oracle.html5
    
    check: select * from V$ASM_ACFSVOLUMES;
    -AS root:
    
    cd /
    mkdir odadatafs
    chmod 770 odadatafs
    chown oracle:asmdba odadatafs
    
    As grid:
    sqlplus / as sysasm
    
    ALTER DISKGROUP RECO ADD VOLUME DATAFSVOL SIZE 10G; select volume_name, volume_device from v$asm_volume; /dev/asm/datafsvol-20 – RCW/dev/asm/datafsvol-190 - RCS Exit; -   Check the volume name on:ls –tlr /dev/asm/ or select * from v$asm_volumeAs root:/sbin/mkfs -t acfs -b 4k /dev/asm/datafsvol-190 -n "DATAFSVOL"  // ASMFSVOL -   Register FS/sbin/acfsutil registry -f -a /dev/asm/datafsvol-190 /odadatafs
    The ASM Cluster File System should now be mounted on all RAC nodes. 
    If it not, then issue the following command on each node to mount it. 
    As root
    
    /bin/mount -t acfs /dev/asm/datafsvol-20 /odadatafs Check the FS
    as grid:
    crsctl stat res ora.reco.datafsvol.acfs
    
    In case of the Filesystem not registered on the CRS, please execute the command below:
    
    As root:
    
    /sbin/acfsutil registry -d /odadatafs  // Delete the registry on CRS
    /u01/app/11.2.0.4/grid/bin/srvctl add filesystem -d /dev/asm/datafsvol-190 -v DATAFSVOL -g RECO -m /odadatafs
    As root:
    /u01/app/11.2.0.4/grid/bin/crsctl start res ora.reco.datafsvol.acfs/u01/app/11.2.0.4/grid/bin/crsctl stat res ora.reco.datafsvol.acfsNAME=ora.reco.datafsvol.acfsTYPE=ora.acfs.typeTARGET=ONLINE                  , ONLINESTATE=ONLINE on SERVER1, ONLINE on SERVER2         -  Check AUDIT files on $GRID_HOME/rdbms/audit, if you have a lot of files, you should delete those files before start the upgrade on Grid Infrastructure. "opatch auto" or "rootcrs.pl -unlock" or "roothas.pl -unlock" or addNode.sh Hangs Due to Many Audit Files in GRID_HOME (Doc ID 1352884.1)                
    
    A.) Check BACKUPS for the database list below:
    Db1
    Db2…
    
    B.) Stop Apps / Blackout on OEM12c
    C.) Execute a clear shutdown / startup. To check if there is some 
    issue before start the upgrade;
    D.) Check Alert.log for all databases after startup; (BOTH NODES)
    E.) Copy the files below to servers (ROOT) both nodes:
    
           p20690087_121230_Linux-x86-64_1of2.zip
           p20690087_121230_Linux-x86-64_2of2.zip
           /tmp
           oakcli unpack -package /tmp/p20690087_121230_Linux-x86-64_1of2.zip
           oakcli unpack -package /tmp/p20690087_121230_Linux-x86-64_2of2.zip
    
    If face some problems with space, please check the document below:
    
    ODA (Oracle Database Appliance): Cleanup trace and log files 
    Using purgeODALog Tool (Doc ID 2081655.1)
    
    F.) After UNPACK the files on both nodes, you can remove as ROOT on both nodes:
    
    rm /tmp/p20690087_121230_Linux-x86-64_1of2.zip
    rm /tmp/p20690087_121230_Linux-x86-64_2of2.zip
    
    Note: After oakcli -unpack, you need to have at least 3GB free.
    
    G.) Get Linux Version, to check after, what has been changed:
           cat /etc/issue
           uname -a
           - Check the components that will be Patched (both nodes)
           oakcli update -patch 12.1.2.3.0 --verify
           - Validate PATCH
           https://docs.oracle.com/cd/E68623_01/doc.121/e68637/GUID-571A2AB5-2C2F-48E6-B5EE-6F3EB724B8D1.htm#CMTAR1355
           oakcli validate -c ospatch -ver 12.1.2.3.0
           - Check Space
           Before the patching, please check your system to ensure you have 
    enough space for /root, /tmp, /u01, /opt to avoid unable to 
    uncompressing patching file.
    
    H.) *** ILOM / BIOS (In case of failure on NORMAL process --INFRA)
    
    ILOM Oracle Integrated Lights Out Manager (Doc ID 1424493.1)
    Patch 20073310: Oracle Sun Server X4-2 SW 1.2.1 - FIRMWARE PACK
    
    New ILOM Version: 3.2.4.20.a r94217
    
                 - Current:
    
                 [root@SERVER1 ~]# ipmitool sunoem cli
                 Connected. Use ^D to exit.
                 -> version
                 version
                 SP firmware 3.1.2.30.c
                 SP firmware build number: 88185
                 SP firmware date: Fri Mar 28 11:23:54 CST 2014
                 SP filesystem version: 0.1.23
     
                 -> show /SYS/MB/BIOS
                        show /SYS/MB/BIOS
     
                         /SYS/MB/BIOS
                               Targets:
     
                               Properties:
                                     type = BIOS
                                     ipmi_name = MB/BIOS
                                     fru_name = SYSTEM BIOS
                                     fru_manufacturer = AMERICAN MEGATRENDS
                                     fru_version = 25010604
                                     fru_part_number = APTIO
     
                 - Check Version:
                 oakcli update -patch 12.1.2.3.0 --verify
    
                 - Stop All resources:
                 - List all homes
                 oakcli show dbhomes
     
    Oracle Home Name      Oracle Home version                  Home Location                          
    ----------------      -------------------                  ------------                           
    OraDb11204_home1      11.2.0.4.2(18031668,18031740)       /u01/app/oracle/product/11.2.0.4/dbhome_1
    OraDb11204_home2      11.2.0.4.2(18031668,18031740)       /u01/app/oracle/product/11.2.0.4/dbhome_2
    
                 - Stop all resources
                   grid# crsctl stop res -all
     
    - Stop SCAN_LISTENER on both nodes
                 srvctl status scan_listener
                 srvctl stop scan_listener
     
                 - Check date/time on both nodes
                 ssh SERVER1 date
                 ssh SERVER2 date
     
                 - Check NTP on both Nodes
                 service ntpd status
                 - Check/Stop CRS on both nodes;
                 crsctl check crs
                 crsctl disable crs
                 crsctl stop crs
                 - From the ILOM : stop the operating system (both nodes)
                        -> stop /SYS
                        Are you sure you want to stop /SYS (y/n)? y
                        Stopping /SYS
     
                        * Wait until power_stat=Off
                        -> show /SYS
                        ...
                                     power_state=Off
                        ...
                 - Patch ILOM:
                 ILOM Version: 3.2.4.20.a r94217
                 load -source scp://<OSUSER>@<IP ADDRESS><PATH>/ILOM-3_2_4_20_a_r94217-Sun_Server_X4-2.pkg
     
                 - Verify:
                        ipmitool sunoem cli
                        version
                        show /SYS/MB/BIOS
    
                 - Start OS
                        start /SYS
     
                 - Start the Grid Infrastructure ( root@odabase )
                        [root]# export ORACLE_HOME=/u01/app/12.1.0.2/grid
                        [root]# $ORACLE_HOME/bin/crsctl enable crs
                        [root]# $ORACLE_HOME/bin/crsctl start crs
     
                        Check both nodes:
                        crsctl check crs
    
    
  • Apply the patch to the Infra components (1o. this, if face some problems do ILOM manually)
 
             - Stop all resources
               grid# crsctl stop res -all
 
             - As root from the first node:
             oakcli update -patch 12.1.2.3.0 --verify

              - Take the root password for both nodes *****
             pwd:
 
             - As root from the first node: (TAKE THE TIME)
             oakcli update -patch  12.1.2.3.0 --infra

              - Verify pos-upgrade (TAKE THE TIME):
             oakcli update -patch  12.1.2.3.0 --verify

       J.) Apply the Patch to the GI components. 
             **CRS must to be running
             - Verify the cluster status (TAKE THE TIME):
               grid# crsctl stat res -t
 
             - Stop all resources
               Check if all resources are stopped. IF NOT
  grid# crsctl stop res -all

             - Verify the cluster status
               grid# crsctl stat res -t
 
             - As root from the first Node:
               oakcli update -patch  12.1.2.3.0 --gi

              - Verify pos-upgrade (TAKE THE TIME): Check the GI_HOME
               oakcli update -patch  12.1.2.3.0 --verify
 
             - Check NTP on both Nodes
               service ntpd status
 
       K.) Apply the Patch to the Database. 
       Dataguard: Oracle Database Appliance Patching with Standby (Doc ID 1900412.1)

              - As root from the first node: (TAKE THE TIME)
             oakcli update -patch  12.1.2.3.0 --database

              - Verify pos-upgrade (TAKE THE TIME): Check the DB_HOME
             oakcli update -patch  12.1.2.3.0 --verify           

             - Start all instances again:
             grid# crsctl start res -all     


- Check SCAN_LISTENER on both nodes
             srvctl status scan_listener
             - In case of not started:
             srvctl start scan_listener
 
- Check PSU applied for each database: (EXCEPT DATAGUARDS)


Col comments format a40;
   select comments, version, bundle_series
from sys.registry$history
where bundle_series = 'PSU'
order by action_time;

              Current Version:
COMMENTS                                 VERSION                        BUNDLE_SERIES
---------------------------------------- ------------------------------ ------------------------------
PSU 11.2.0.4.2                           11.2.0.4                       PSU

In case of not applied the CATBUNDLE must be applied manually.
https://updates.oracle.com/Orion/Services/download?type=readme&aru=18604497#BHAGIFCG
  • Upgrade Database in case of requested:
pscp p19520042_121230_Linux-x86-64.zip root@SERVER1:/tmp
oakcli unpack -package /tmp/p19520042_121230_Linux-x86-64.zip
oakcli create dbhome --version 12.1.0.2.3
Upgrade DB:
As GRID:
Crsctl start res –all
As root:
oakcli upgrade database --db <DB NAME>  -to OraDb12102_home1 ß HOME NAME
After upgrade the database please check the TIMEZONE:
Scripts to automatically update the RDBMS DST (timezone) version in an 11gR2 or 12cR1 database . (Doc ID 1585343.1)


       L.) Kerberos (Check First)
       - After patching the Infra, the Kerberos configuration is deactivated.
       To set it active again, type the following commands on both nodes connected as root on the ODA_BASE
                   /etc/init.d/autofs start
                    chkconfig autofs on

 
       M.) Check Dataguards, if needed

       https://oracle-base.com/articles/11g/data-guard-setup-using-broker-11gr2

       Stop:
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;
shut immediate;
     Reactivate:
       STARTUP NOMOUNT;
ALTER DATABASE MOUNT STANDBY DATABASE;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;
*** References:
https://docs.oracle.com/cd/E63431_01/doc.121/e63330/patches.htm#CHDFECJE

ODA DOM0 Lost Network Communications Between Nodes While running 
'oakcli clone vm' or Copy (cp) Commands of Large Files over NFS (Doc ID 1665055.1)
Oracle Database Appliance - 12.1.2 and 2.X Supported ODA Versions & Known Issues (Doc ID 888888.1)
Oracle Database Appliance Patching with Standby (Doc ID 1900412.1)
https://updates.oracle.com/Orion/Services/download?type=readme&aru=18299059Oracle Sun Server X4-2 Platform Software Release 1.2.1 
 Firmware Pack

ILOM Oracle Integrated Lights Out Manager (Doc ID 1424493.1)
https://updates.oracle.com/Orion/Services/download?type=readme&aru=18938721#_Toc419319836

 

How to upgrade the firmware using the ILOM CLI , for SPARC CMT Servers (Doc ID 2101665.1)
*** Know issues
After failed patch: BEFORE RE-EXECUTE
oakcli update -patch 12.1.2.3 –clean
 
After a failed –gi installation, you can find an issue with:
 
Oakcli show ismaster
[root@SERVER1 log]# oakcli show ismaster

OAKD is in Slave Mode

[root@SERVER2 oracle]# oakcli show ismaster
OAKD is in Slave Mode
 
Both nodes are is SLAVE MODE

To fix in both nodes:

Vi /opt/oracle/oak/install/oakdrun
[root@SERVER1 log]# cat /opt/oracle/oak/install/oakdrun

Upgrade

cp /opt/oracle/oak/install/oakdrun /opt/oracle/oak/install/oakdrun_20160820.bkp
cp /opt/oracle/oak/install/oakdrun_20160820.bkp /opt/oracle/oak/install/oakdrun

Change to : 
start
Restart:
oakcli stop oak
oakcli start oak
oakcli show ismaster

In case of errors on update: Check this directory
/opt/oracle/oak/log/SERVER1/patch/12.1.2.3.0

oakcli update -patch  12.1.2.3.0 --database
After update database check the catalog:
prompt "11g"
select * from registry$history;

prompt "12c"
select * from registry$sqlpatch;
OAKCLI CREATE DBHOMES FAILS IN 12.1.0.2.3 (Bug ID 21308909)
oakcli delete dbhome -oh <oracle_home>

To connect on ILOM:
http://www.solvemyissue.com/2014/08/oda-ilom-integrated-lights-out-manager.html
After you have the IP address, replace it in the next URL:

https://<IP Address>/iPages/i_login.asp - Node 1
https://<IP Address>/iPages/i_login.asp - Node 2

 Restart ODA: Is better do using ILOM
2 Options (ILOM or Server)
ILOM:
reset /SYS , restart the server
start /SP/Console, to see the reboot status

Server:
Shutdown –r now
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s