Snapshot Databases in Oracle Database Appliance

 * Defining ORACLE_HOME:

[root@oda01 ~]# oakcli show dbhomes
Oracle Home Name      Oracle Home version                  Home Location
OraDb11204_home1      11.2.0.4.6(20299013,20420937)       /u01/app/oracle/product/11.2.0.4/dbhome_1
OraDb12102_home2      12.1.0.2.3(20299023,20299022)       /u01/app/oracle/product/12.1.0.2/dbhome_2
OraDb12102_home1      12.1.0.2.160719(23054246,23054327)  /u01/app/oracle/product/12.1.0.2/dbhome_1
root@oda01 ~]# oakcli show databases
DB01   RAC        ASM       OraDb12102_home1     /u01/app/oracle/product/12.1.0.2/dbhome_1          12.1.0.2.160719(23054246,23054327)
DB02   RAC        ASM       OraDb12102_home1     /u01/app/oracle/product/12.1.0.2/dbhome_1          12.1.0.2.160719(23054246,23054327)
 [grid@oda01 ~]$ asmcmd lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  NORMAL  N         512   4096  4194304   7372800  1478768           368640          555064              0             Y  DATA/
MOUNTED  NORMAL  N         512   4096  4194304   9796800  6419444           489840         2964802              0             N  RECO/
MOUNTED  HIGH    N         512   4096  4194304    763120   373600           381560           -2653              0             N  REDO/

* Create Test Database:

oakcli create database -db TSTSNAP -oh OraDb12102_home1
Pwd: oracle123
Please select one of the following for Database type  [1 .. 3] :
1    => OLTP
2    => DSS
3    => In-Memory
option: 1
Please select one of the following for Database Deployment  [1 .. 3] :
1    => EE : Enterprise Edition
2    => RACONE
3    => RAC
option: 3

Please select one of the following for Database Class  [1 .. 6] :
1    => odb-01s  (   1 cores ,     4 GB memory)
2    =>  odb-01  (   1 cores ,     8 GB memory)
3    =>  odb-02  (   2 cores ,    16 GB memory)
4    =>  odb-04  (   4 cores ,    32 GB memory)
5    =>  odb-06  (   6 cores ,    48 GB memory)
6    =>  odb-12  (  12 cores ,    96 GB memory)
option: 1
[root@oda01 ~]# oakcli show databases
DB01     RAC        ASM       OraDb12102_home1     /u01/app/oracle/product/12.1.0.2/dbhome_1          12.1.0.2.160719(23054246,23054327)
DB02     RAC        ASM       OraDb12102_home1     /u01/app/oracle/product/12.1.0.2/dbhome_1          12.1.0.2.160719(23054246,23054327)
TSTSNAP  RAC        ACFS      OraDb12102_home1     /u01/app/oracle/product/12.1.0.2/dbhome_1          12.1.0.2.160719(23054246,23054327) <---
 [root@oda01 ~]# srvctl config database -d TSTSNAP
Database unique name: TSTSNAP
Database name: TSTSNAP
Oracle home: /u01/app/oracle/product/12.1.0.2/dbhome_1
Oracle user: oracle
Spfile: /u02/app/oracle/oradata/datastore/.ACFS/snaps/TSTSNAP/TSTSNAP/spfileTSTSNAP.ora
Password file: /u02/app/oracle/oradata/datastore/.ACFS/snaps/TSTSNAP/TSTSNAP/orapwTSTSNAP
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups:
Mount point paths: /u01/app/oracle/oradata/datastore,/u02/app/oracle/oradata/datastore,/u01/app/oracle/fast_recovery_area/datastore
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: racoper
Database instances: TSTSNAP1,TSTSNAP2
Configured nodes: oda01 ,oda02 
Database is administrator managed

—> Pre-Requisites to create a snpshot database

 https://docs.oracle.com/cd/E75550_01/doc.121/e79567/GUID-63DFD214-2B87-40A0-AE1B-C59805F0043E.htm#CMTXH-GUID-8B7849A5-1A3C-469D-9C90-12394364C980
    They must not be a standby or container database
    They must not be running in read-only mode, or in restricted mode, or in online backup mode
    They must be in ARCHIVELOG mode
    They must have all defined data files available and online
    They must not use centralized wallets with Transparent Data Encryption.

–> Archivelog Mode

 SQL> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     3
Next log sequence to archive   4
Current log sequence           4

–> Size of DATABASE

create tablespace ts_test 
datafile '/u02/app/oracle/oradata/datastore/.ACFS/snaps/TSTSNAP/TSTSNAP/datafile/ts_test1.dbf' 
size 1g;
 col file_name format a100
select bytes/(1024*1024) mb, tablespace_name, file_name from dba_data_files;
         MB TABLESPACE_NAME                FILE_NAME
---------- ------------------------------ ----------------------------------------------------------------------------------------------------
       700 SYSTEM                         /u02/app/oracle/oradata/datastore/.ACFS/snaps/TSTSNAP/TSTSNAP/datafile/o1_mf_system_dmlcolr5_.dbf
       600 SYSAUX                         /u02/app/oracle/oradata/datastore/.ACFS/snaps/TSTSNAP/TSTSNAP/datafile/o1_mf_sysaux_dmlcp5g2_.dbf
       305 UNDOTBS1                       /u02/app/oracle/oradata/datastore/.ACFS/snaps/TSTSNAP/TSTSNAP/datafile/o1_mf_undotbs1_dmlcplml_.dbf
       200 UNDOTBS2                       /u02/app/oracle/oradata/datastore/.ACFS/snaps/TSTSNAP/TSTSNAP/datafile/o1_mf_undotbs2_dmlcq436_.dbf
         5 USERS                          /u02/app/oracle/oradata/datastore/.ACFS/snaps/TSTSNAP/TSTSNAP/datafile/o1_mf_users_dmlcq8rx_.dbf
      1024 TS_TEST                        /u02/app/oracle/oradata/datastore/.ACFS/snaps/TSTSNAP/TSTSNAP/datafile/ts_test1.dbf
prompt "Total Size: "
select round(sum(bytes)/(1024*1024*1024),0) GB from
(select sum(bytes) bytes from dba_data_files
union
select sum(bytes) bytes from dba_temp_files);
        GB
----------
         3
 --> Populating the DB
 SQL> create user test identified by test default tablespace ts_test;
User created.
SQL> grant dba to test;
Grant succeeded.
create table test.tb1 as select * from dba_segments;
  1*  select segment_name, bytes / (1024*1024) mb, tablespace_name from dba_segments where segment_name='TB1'
SQL> /
SEGMENT_NAME                 MB TABLESPACE_NAME
-------------------- ---------- ------------------------------
TB1                         .75 TS_TEST
 declare
begin
for x in 1 .. 10 loop
  insert into test.tb1 select * from test.tb1;
end loop;
commit;
end;
/
  select segment_name, bytes / (1024*1024) mb, tablespace_name 
from dba_segments where segment_name='TB1'
SEGMENT_NAME                 MB TABLESPACE_NAME
-------------------- ---------- ------------------------------
TB1                         672 TS_TEST
 select round(sum(bytes)/(1024*1024*1024),2) GB
from dba_free_space;  2
         GB
----------
      1.21
1

—> Creating the Snapshot

Start Time: 2017-06-08 14:12:06
End Time  : 2017-06-08 14:29:22

oakcli create snapshotdb -db SNAP01 -from TSTSNAP
pwd: oracle123 (if the password doens't match try: welcome1)

Please select one of the following for Database Deployment  [1 .. 2] :
1    => RACONE
2    => RAC
Option: 2
Please select one of the following for Database Class  [1 .. 5] :
1    => odb-01s  (   1 cores ,     4 GB memory)
2    =>  odb-01  (   1 cores ,     8 GB memory)
3    =>  odb-02  (   2 cores ,    16 GB memory)
4    =>  odb-04  (   4 cores ,    32 GB memory)
5    =>  odb-06  (   6 cores ,    48 GB memory)
Option: 1
SUCCESS: 
All nodes in /opt/oracle/oak/temp_clunodes.txt are pingable and alive.
......
SUCCESS: All nodes in /opt/oracle/oak/temp_clunodes.txt are 
pingable and alive.
INFO: 2017-06-08 14:16:35: Creating the SNAP 
Database 'SNAP01' from the source Database 'TSTSNAP'
INFO: 2017-06-08 14:16:44: Do not perform any Structural change to 
Database 'TSTSNAP' till SNAP Database 'SNAP01' is created
INFO: 2017-06-08 14:17:05: Taking SNAP of the Database 'TSTSNAP'
INFO: 2017-06-08 14:17:10: Successfully took  the SNAP of database: TSTSNAP
INFO: 2017-06-08 14:20:53: Creating controlfile for database: SNAP01
INFO: 2017-06-08 14:21:39: Successfully created the control file for the database : SNAP01
INFO: 2017-06-08 14:21:39: Adding log files for the second thread for the database : SNAP01
INFO: 2017-06-08 14:21:46: Successfully added the log files for second thread
INFO: 2017-06-08 14:21:52: Recovering the database: SNAP01,  snapshot time : '2017-06-08:14:17:09' , until time : '2017-06-08:14:17:40'
INFO: 2017-06-08 14:21:55: Successfully recovered the database
INFO: 2017-06-08 14:21:55: Opening the database with resetlogs
INFO: 2017-06-08 14:22:26: Successfully opened the database after recovery
INFO: 2017-06-08 14:22:32: Setting the temporary tablespace for database : SNAP01
INFO: 2017-06-08 14:22:47: Successfully set the temporary tablespace for the database : SNAP01
INFO: 2017-06-08 14:23:32: Successfully changed the Database ID
INFO: 2017-06-08 14:25:17: Adding the Database resource to the clusterware
INFO: 2017-06-08 14:26:43: Successfully started the database
INFO: 2017-06-08 14:26:43: Updating the TNS entries for the database SNAP01
INFO: 2017-06-08 14:27:31: Successfully set the RMAN SNAPSHOT control file
INFO: 2017-06-08 14:27:44: Disabling the external references in the database 'SNAP01' inherited from 'TSTSNAP'
INFO: 2017-06-08 14:27:45: Successfully disabled the external references
INFO: 2017-06-08 14:28:08: Run the SQL script '/u01/app/oracle/product/12.1.0.2/dbhome_1/enable_external_refs_SNAP01_r_GY.sql' on the database 'SNAP01' to enable these external references
 Also need to restart the database after running the SQL script
SUCCESS: 2017-06-08 14:29:22: Successfully created the Database 'SNAP01' from 'TSTSNAP'
 2
[oracle@oda01 ~]$ srvctl config database -d SNAP01
Database unique name: SNAP01
Database name:
Oracle home: /u01/app/oracle/product/12.1.0.2/dbhome_1
Oracle user: oracle
Spfile: 
/u02/app/oracle/oradata/datastore/.ACFS/snaps/SNAP01/SNAP01/spfileSNAP01.ora
Password file: 
/u02/app/oracle/oradata/datastore/.ACFS/snaps/SNAP01/SNAP01/orapwSNAP01
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups:
Mount point paths: 
/u01/app/oracle/oradata/datastore,/u02/app/oracle/oradata/datastore,
/u01/app/oracle/fast_recovery_area/datastore
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: racoper
Database instances: SNAP011,SNAP012
Configured nodes: oda01 ,oda02 
Database is administrator managed

. oraenv
SNAP011
 sqlplus / as sysdba
SQL> col file_name format a100
SQL> select bytes/(1024*1024) mb, tablespace_name, file_name 
from dba_data_files;
        MB TABLESPACE_NAME                FILE_NAME
---------- ------------------------------ ----------------------------------------------------------------------------------------------------
       700 SYSTEM                         /u02/app/oracle/oradata/datastore/.ACFS/snaps/SNAP01/SNAP01/datafile/o1_mf_system_dmlcolr5_.dbf
       600 SYSAUX                         /u02/app/oracle/oradata/datastore/.ACFS/snaps/SNAP01/SNAP01/datafile/o1_mf_sysaux_dmlcp5g2_.dbf
       305 UNDOTBS1                       /u02/app/oracle/oradata/datastore/.ACFS/snaps/SNAP01/SNAP01/datafile/o1_mf_undotbs1_dmlcplml_.dbf
       200 UNDOTBS2                       /u02/app/oracle/oradata/datastore/.ACFS/snaps/SNAP01/SNAP01/datafile/o1_mf_undotbs2_dmlcq436_.dbf
         5 USERS                          /u02/app/oracle/oradata/datastore/.ACFS/snaps/SNAP01/SNAP01/datafile/o1_mf_users_dmlcq8rx_.dbf
      1024 TS_TEST                        /u02/app/oracle/oradata/datastore/.ACFS/snaps/SNAP01/SNAP01/datafile/ts_test1.dbf
SQL> select round(sum(bytes)/(1024*1024*1024),0) GB from
(select sum(bytes) bytes from dba_data_files
union
select sum(bytes) bytes from dba_temp_files);
        GB
----------
         3
 ---> Creating the Snapshot from previous Snapshot
Start Time: 2017-06-08 14:41:18
End Time  : 2017-06-08 14:54:10
 oakcli create snapshotdb -db SNAP02 -from SNAP01
pwd: welcome1
*** Note: on this case i changed the type of the database to RACONE
[root@oda01 ~]# oakcli create snapshotdb -db SNAP02 -from SNAP01
INFO: 2017-06-08 14:41:18: Please check the logfile  
'/opt/oracle/oak/log/oda01/tools/12.1.2.8.0/createdb_SNAP02_61544.log' for more details
 Please enter the 'SYS'  password for the Database SNAP01:
Please re-enter the 'SYS' password:
Please select one of the following for Database Deployment  [1 .. 2] :
1    => RACONE
2    => RAC
1
The selected value is : RACONE
Please select one of the following for Database Class  [1 .. 5] :
1    => odb-01s  (   1 cores ,     4 GB memory)
2    =>  odb-01  (   1 cores ,     8 GB memory)
3    =>  odb-02  (   2 cores ,    16 GB memory)
4    =>  odb-04  (   4 cores ,    32 GB memory)
5    =>  odb-06  (   6 cores ,    48 GB memory)
1
The selected value is : odb-01s  (   1 cores ,     4 GB memory)
......
SUCCESS: 
All nodes in /opt/oracle/oak/temp_clunodes.txt are pingable and alive.
......
SUCCESS: 
All nodes in /opt/oracle/oak/temp_clunodes.txt are pingable and alive.
INFO: 2017-06-08 14:44:13: 
Creating the SNAP Database 'SNAP02' from the source Database 'SNAP01'
INFO: 2017-06-08 14:44:22: 
Do not perform any Structural change to Database 'SNAP01' till SNAP Database 'SNAP02' is created
INFO: 2017-06-08 14:44:42: 
Taking SNAP of the Database 'SNAP01'
INFO: 2017-06-08 14:44:47: 
Successfully took  the SNAP of database: SNAP01
INFO: 2017-06-08 14:45:35: 
Creating controlfile for database: SNAP02
INFO: 2017-06-08 14:46:40: 
Successfully created the control file for the database : SNAP02
INFO: 2017-06-08 14:46:46: 
Recovering the database: SNAP02,  snapshot time : '2017-06-08:14:44:47' , until time : '2017-06-08:14:44:57'
INFO: 2017-06-08 14:46:49: Successfully recovered the database
INFO: 2017-06-08 14:46:49: Opening the database with resetlogs
INFO: 2017-06-08 14:47:20: Successfully opened the database after recovery
INFO: 2017-06-08 14:47:25: Setting the temporary tablespace for database : SNAP02
INFO: 2017-06-08 14:47:33: Successfully set the temporary tablespace for the database : SNAP02
INFO: 2017-06-08 14:48:19: Successfully changed the Database ID
INFO: 2017-06-08 14:49:56: Adding the Database resource to the clusterware
INFO: 2017-06-08 14:51:40: Successfully started the database
INFO: 2017-06-08 14:51:46: Updating the TNS entries for the database SNAP02
INFO: 2017-06-08 14:52:27: Successfully set the RMAN SNAPSHOT control file
INFO: 2017-06-08 14:52:38: 
Disabling the external references in the database 'SNAP02' inherited from 'SNAP01'
INFO: 2017-06-08 14:52:39: Successfully disabled the external references
INFO: 2017-06-08 14:53:07: 
Run the SQL script '/u01/app/oracle/product/12.1.0.2/dbhome_1/enable_external_refs_SNAP02_wpY9.sql' on the database 'SNAP02' to enable these external references
 Also need to restart the database after running the SQL script
SUCCESS: 2017-06-08 14:54:10: 
Successfully created the Database 'SNAP02' from 'SNAP01'
 3

 

Please note that the second snapshot took 30% less of the used storage, this is because that only changed blocks are copied during the process.

Advertisements

Upgrade ODA

Below the steps to Upgrade the ODA Baremetal from 2.10. to 12.1.2.3.0, 
probably for earlier version the steps should be the same...

1.) Upgrade to 12.1.2.3.0 (20690087)
SERVER1
SERVER2


*** Pre-Requisites:
  • Passwords:
    • ROOT: 
    • SYSASM: 
    • ILOM:
    • SYSDBA: 
    • ORACLE: 
    • GRID:
 Check: ODA Supplemental Pre-Patch Heath Check (Doc ID 2140853.1)
oakcli manage cleanrepo --ver 2.10.0.0.0 <Cleaning the OLD Versions>
oakcli validate -c <NODE NAME> -prechecks
oakcli validate -a
  • Disable cron jobs
     cd ~
       crontab -l > crontab.'date "+%Y%m%d"'
       ls -latr crontab.*
       crontab < /dev/null
 Backup the .ssh directory 
  • Blackout EM agents on each ODA node
 export ORACLE_HOME=/u01/app/oracle/product/agent12c/core/12.1.0.4.0
    cd /u01/app/oracle/product/agent12c/core/12.1.0.4.0/bin
   ./emctl status agent
   ./emctl start blackout ODAPatch_${HOSTNAME} -nodeLevel
   ./emctl stop agent
   ./emctl status agent
  • Checking the Master/Slave:
[root@SERVER1 ~]# oakcli show ismaster
OAKD is in Slave Mode

[root@SERVER2 ~]# oakcli show ismaster
OAKD is in Master Mode

On this case we need restart OAK on node 2:

oakcli restart oak
 
  • Confirm that both nodes version are up to date
  • identical versions on each node (minus ASR)
Example:
         # oakcli validate -d -f /tmp/$HOSTNAME-OAKvalidate.log
  • ASM Volume : the volume DATAFSVOL mounted on /odadatafs must exist
    https://oracle-base.com/articles/11g/acfs-11gr2
    http://docs.oracle.com/cd/E11882_01/server.112/e18951/asmviews.htm#OSTMG94187
    https://gjilevski.com/2010/08/28/asm-cluster-file-system-acfs-in-oracle-11gr2/
    http://www.solvemyissue.com/2014/08/oda-how-to-create-and-delete-oracle.html5
    
    check: select * from V$ASM_ACFSVOLUMES;
    -AS root:
    
    cd /
    mkdir odadatafs
    chmod 770 odadatafs
    chown oracle:asmdba odadatafs
    
    As grid:
    sqlplus / as sysasm
    
    ALTER DISKGROUP RECO ADD VOLUME DATAFSVOL SIZE 10G; select volume_name, volume_device from v$asm_volume; /dev/asm/datafsvol-20 – RCW/dev/asm/datafsvol-190 - RCS Exit; -   Check the volume name on:ls –tlr /dev/asm/ or select * from v$asm_volumeAs root:/sbin/mkfs -t acfs -b 4k /dev/asm/datafsvol-190 -n "DATAFSVOL"  // ASMFSVOL -   Register FS/sbin/acfsutil registry -f -a /dev/asm/datafsvol-190 /odadatafs
    The ASM Cluster File System should now be mounted on all RAC nodes. 
    If it not, then issue the following command on each node to mount it. 
    As root
    
    /bin/mount -t acfs /dev/asm/datafsvol-20 /odadatafs Check the FS
    as grid:
    crsctl stat res ora.reco.datafsvol.acfs
    
    In case of the Filesystem not registered on the CRS, please execute the command below:
    
    As root:
    
    /sbin/acfsutil registry -d /odadatafs  // Delete the registry on CRS
    /u01/app/11.2.0.4/grid/bin/srvctl add filesystem -d /dev/asm/datafsvol-190 -v DATAFSVOL -g RECO -m /odadatafs
    As root:
    /u01/app/11.2.0.4/grid/bin/crsctl start res ora.reco.datafsvol.acfs/u01/app/11.2.0.4/grid/bin/crsctl stat res ora.reco.datafsvol.acfsNAME=ora.reco.datafsvol.acfsTYPE=ora.acfs.typeTARGET=ONLINE                  , ONLINESTATE=ONLINE on SERVER1, ONLINE on SERVER2         -  Check AUDIT files on $GRID_HOME/rdbms/audit, if you have a lot of files, you should delete those files before start the upgrade on Grid Infrastructure. "opatch auto" or "rootcrs.pl -unlock" or "roothas.pl -unlock" or addNode.sh Hangs Due to Many Audit Files in GRID_HOME (Doc ID 1352884.1)                
    
    A.) Check BACKUPS for the database list below:
    Db1
    Db2…
    
    B.) Stop Apps / Blackout on OEM12c
    C.) Execute a clear shutdown / startup. To check if there is some 
    issue before start the upgrade;
    D.) Check Alert.log for all databases after startup; (BOTH NODES)
    E.) Copy the files below to servers (ROOT) both nodes:
    
           p20690087_121230_Linux-x86-64_1of2.zip
           p20690087_121230_Linux-x86-64_2of2.zip
           /tmp
           oakcli unpack -package /tmp/p20690087_121230_Linux-x86-64_1of2.zip
           oakcli unpack -package /tmp/p20690087_121230_Linux-x86-64_2of2.zip
    
    If face some problems with space, please check the document below:
    
    ODA (Oracle Database Appliance): Cleanup trace and log files 
    Using purgeODALog Tool (Doc ID 2081655.1)
    
    F.) After UNPACK the files on both nodes, you can remove as ROOT on both nodes:
    
    rm /tmp/p20690087_121230_Linux-x86-64_1of2.zip
    rm /tmp/p20690087_121230_Linux-x86-64_2of2.zip
    
    Note: After oakcli -unpack, you need to have at least 3GB free.
    
    G.) Get Linux Version, to check after, what has been changed:
           cat /etc/issue
           uname -a
           - Check the components that will be Patched (both nodes)
           oakcli update -patch 12.1.2.3.0 --verify
           - Validate PATCH
           https://docs.oracle.com/cd/E68623_01/doc.121/e68637/GUID-571A2AB5-2C2F-48E6-B5EE-6F3EB724B8D1.htm#CMTAR1355
           oakcli validate -c ospatch -ver 12.1.2.3.0
           - Check Space
           Before the patching, please check your system to ensure you have 
    enough space for /root, /tmp, /u01, /opt to avoid unable to 
    uncompressing patching file.
    
    H.) *** ILOM / BIOS (In case of failure on NORMAL process --INFRA)
    
    ILOM Oracle Integrated Lights Out Manager (Doc ID 1424493.1)
    Patch 20073310: Oracle Sun Server X4-2 SW 1.2.1 - FIRMWARE PACK
    
    New ILOM Version: 3.2.4.20.a r94217
    
                 - Current:
    
                 [root@SERVER1 ~]# ipmitool sunoem cli
                 Connected. Use ^D to exit.
                 -> version
                 version
                 SP firmware 3.1.2.30.c
                 SP firmware build number: 88185
                 SP firmware date: Fri Mar 28 11:23:54 CST 2014
                 SP filesystem version: 0.1.23
     
                 -> show /SYS/MB/BIOS
                        show /SYS/MB/BIOS
     
                         /SYS/MB/BIOS
                               Targets:
     
                               Properties:
                                     type = BIOS
                                     ipmi_name = MB/BIOS
                                     fru_name = SYSTEM BIOS
                                     fru_manufacturer = AMERICAN MEGATRENDS
                                     fru_version = 25010604
                                     fru_part_number = APTIO
     
                 - Check Version:
                 oakcli update -patch 12.1.2.3.0 --verify
    
                 - Stop All resources:
                 - List all homes
                 oakcli show dbhomes
     
    Oracle Home Name      Oracle Home version                  Home Location                          
    ----------------      -------------------                  ------------                           
    OraDb11204_home1      11.2.0.4.2(18031668,18031740)       /u01/app/oracle/product/11.2.0.4/dbhome_1
    OraDb11204_home2      11.2.0.4.2(18031668,18031740)       /u01/app/oracle/product/11.2.0.4/dbhome_2
    
                 - Stop all resources
                   grid# crsctl stop res -all
     
    - Stop SCAN_LISTENER on both nodes
                 srvctl status scan_listener
                 srvctl stop scan_listener
     
                 - Check date/time on both nodes
                 ssh SERVER1 date
                 ssh SERVER2 date
     
                 - Check NTP on both Nodes
                 service ntpd status
                 - Check/Stop CRS on both nodes;
                 crsctl check crs
                 crsctl disable crs
                 crsctl stop crs
                 - From the ILOM : stop the operating system (both nodes)
                        -> stop /SYS
                        Are you sure you want to stop /SYS (y/n)? y
                        Stopping /SYS
     
                        * Wait until power_stat=Off
                        -> show /SYS
                        ...
                                     power_state=Off
                        ...
                 - Patch ILOM:
                 ILOM Version: 3.2.4.20.a r94217
                 load -source scp://<OSUSER>@<IP ADDRESS><PATH>/ILOM-3_2_4_20_a_r94217-Sun_Server_X4-2.pkg
     
                 - Verify:
                        ipmitool sunoem cli
                        version
                        show /SYS/MB/BIOS
    
                 - Start OS
                        start /SYS
     
                 - Start the Grid Infrastructure ( root@odabase )
                        [root]# export ORACLE_HOME=/u01/app/12.1.0.2/grid
                        [root]# $ORACLE_HOME/bin/crsctl enable crs
                        [root]# $ORACLE_HOME/bin/crsctl start crs
     
                        Check both nodes:
                        crsctl check crs
    
    
  • Apply the patch to the Infra components (1o. this, if face some problems do ILOM manually)
 
             - Stop all resources
               grid# crsctl stop res -all
 
             - As root from the first node:
             oakcli update -patch 12.1.2.3.0 --verify

              - Take the root password for both nodes *****
             pwd:
 
             - As root from the first node: (TAKE THE TIME)
             oakcli update -patch  12.1.2.3.0 --infra

              - Verify pos-upgrade (TAKE THE TIME):
             oakcli update -patch  12.1.2.3.0 --verify

       J.) Apply the Patch to the GI components. 
             **CRS must to be running
             - Verify the cluster status (TAKE THE TIME):
               grid# crsctl stat res -t
 
             - Stop all resources
               Check if all resources are stopped. IF NOT
  grid# crsctl stop res -all

             - Verify the cluster status
               grid# crsctl stat res -t
 
             - As root from the first Node:
               oakcli update -patch  12.1.2.3.0 --gi

              - Verify pos-upgrade (TAKE THE TIME): Check the GI_HOME
               oakcli update -patch  12.1.2.3.0 --verify
 
             - Check NTP on both Nodes
               service ntpd status
 
       K.) Apply the Patch to the Database. 
       Dataguard: Oracle Database Appliance Patching with Standby (Doc ID 1900412.1)

              - As root from the first node: (TAKE THE TIME)
             oakcli update -patch  12.1.2.3.0 --database

              - Verify pos-upgrade (TAKE THE TIME): Check the DB_HOME
             oakcli update -patch  12.1.2.3.0 --verify           

             - Start all instances again:
             grid# crsctl start res -all     


- Check SCAN_LISTENER on both nodes
             srvctl status scan_listener
             - In case of not started:
             srvctl start scan_listener
 
- Check PSU applied for each database: (EXCEPT DATAGUARDS)


Col comments format a40;
   select comments, version, bundle_series
from sys.registry$history
where bundle_series = 'PSU'
order by action_time;

              Current Version:
COMMENTS                                 VERSION                        BUNDLE_SERIES
---------------------------------------- ------------------------------ ------------------------------
PSU 11.2.0.4.2                           11.2.0.4                       PSU

In case of not applied the CATBUNDLE must be applied manually.
https://updates.oracle.com/Orion/Services/download?type=readme&aru=18604497#BHAGIFCG
  • Upgrade Database in case of requested:
pscp p19520042_121230_Linux-x86-64.zip root@SERVER1:/tmp
oakcli unpack -package /tmp/p19520042_121230_Linux-x86-64.zip
oakcli create dbhome --version 12.1.0.2.3
Upgrade DB:
As GRID:
Crsctl start res –all
As root:
oakcli upgrade database --db <DB NAME>  -to OraDb12102_home1 ß HOME NAME
After upgrade the database please check the TIMEZONE:
Scripts to automatically update the RDBMS DST (timezone) version in an 11gR2 or 12cR1 database . (Doc ID 1585343.1)


       L.) Kerberos (Check First)
       - After patching the Infra, the Kerberos configuration is deactivated.
       To set it active again, type the following commands on both nodes connected as root on the ODA_BASE
                   /etc/init.d/autofs start
                    chkconfig autofs on

 
       M.) Check Dataguards, if needed

       https://oracle-base.com/articles/11g/data-guard-setup-using-broker-11gr2

       Stop:
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;
shut immediate;
     Reactivate:
       STARTUP NOMOUNT;
ALTER DATABASE MOUNT STANDBY DATABASE;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;
*** References:
https://docs.oracle.com/cd/E63431_01/doc.121/e63330/patches.htm#CHDFECJE

ODA DOM0 Lost Network Communications Between Nodes While running 
'oakcli clone vm' or Copy (cp) Commands of Large Files over NFS (Doc ID 1665055.1)
Oracle Database Appliance - 12.1.2 and 2.X Supported ODA Versions & Known Issues (Doc ID 888888.1)
Oracle Database Appliance Patching with Standby (Doc ID 1900412.1)
https://updates.oracle.com/Orion/Services/download?type=readme&aru=18299059Oracle Sun Server X4-2 Platform Software Release 1.2.1 
 Firmware Pack

ILOM Oracle Integrated Lights Out Manager (Doc ID 1424493.1)
https://updates.oracle.com/Orion/Services/download?type=readme&aru=18938721#_Toc419319836

 

How to upgrade the firmware using the ILOM CLI , for SPARC CMT Servers (Doc ID 2101665.1)
*** Know issues
After failed patch: BEFORE RE-EXECUTE
oakcli update -patch 12.1.2.3 –clean
 
After a failed –gi installation, you can find an issue with:
 
Oakcli show ismaster
[root@SERVER1 log]# oakcli show ismaster

OAKD is in Slave Mode

[root@SERVER2 oracle]# oakcli show ismaster
OAKD is in Slave Mode
 
Both nodes are is SLAVE MODE

To fix in both nodes:

Vi /opt/oracle/oak/install/oakdrun
[root@SERVER1 log]# cat /opt/oracle/oak/install/oakdrun

Upgrade

cp /opt/oracle/oak/install/oakdrun /opt/oracle/oak/install/oakdrun_20160820.bkp
cp /opt/oracle/oak/install/oakdrun_20160820.bkp /opt/oracle/oak/install/oakdrun

Change to : 
start
Restart:
oakcli stop oak
oakcli start oak
oakcli show ismaster

In case of errors on update: Check this directory
/opt/oracle/oak/log/SERVER1/patch/12.1.2.3.0

oakcli update -patch  12.1.2.3.0 --database
After update database check the catalog:
prompt "11g"
select * from registry$history;

prompt "12c"
select * from registry$sqlpatch;
OAKCLI CREATE DBHOMES FAILS IN 12.1.0.2.3 (Bug ID 21308909)
oakcli delete dbhome -oh <oracle_home>

To connect on ILOM:
http://www.solvemyissue.com/2014/08/oda-ilom-integrated-lights-out-manager.html
After you have the IP address, replace it in the next URL:

https://<IP Address>/iPages/i_login.asp - Node 1
https://<IP Address>/iPages/i_login.asp - Node 2

 Restart ODA: Is better do using ILOM
2 Options (ILOM or Server)
ILOM:
reset /SYS , restart the server
start /SP/Console, to see the reboot status

Server:
Shutdown –r now

ODA – Reconfig SSH user equivalence

In case of problem with SSH User Equivalence, there are some steps to rebuild the equivalence for Oracle, Grid and ROOT, bellow the steps :
Obs. : -> Mandatory reset default password to WELCOME1
  • Login as ROOT on Node 1;
  • Passwd root, change password to welcome1;
  • Passwd oracle, change password to welcome1;
  • Passwd grid, chance password to welcome1;
  • Login as ROOT on Node 2;
  • Passwd root, change password to welcome1;
  • Passwd oracle, change password to welcome1;
  • Passwd grid, chance password to welcome1;
  • On Node 1, execute
    1. cd /opt/oracle/oak/onecmd
    2. ./gridinst.pl -o -s  3 è for root
    3. ./gridinst.pl -o -s  12 è for oracle and grid
  • login as Oracle ;
  • execute a test with ssh between the nodes ;
    1. from Node 1 :
       ssh <node 2> date, the password prompt should not appears
    2. from Node 2 :
       ssh <node 1> date, the password prompt should not appears
  • login as GRID ;
  • execute a test with ssh between the nodes ;
    1. from Node 1 :
       ssh <node 2> date, the password prompt should not appears
    2. from Node 2 :
       ssh <node 1> date, the password prompt should not appears
  • login as ROOT ;
  • execute a test with ssh between the nodes ;
    1. from Node 1 :
       ssh <node 2> date, the password prompt should not appears
    2. from Node 2 :
       ssh <node 1> date, the password prompt should not appears

ODA – Change ROOT password ILOM

The procedure bellow describes how to change the password for Oracle and GRID users on ODA (Oracle Database Appliance).
  • Login as ROOT on ILOM;
  • -> set /SP/users/root password=<PASSWORD>
     set /SP/users/root password=<PASSWORD>
     Changing password for user /SP/users/root...
     Enter new password again: <PASSWORD>
     ********
     New password was successfully set for user /SP/users/root

Obs.: If any problem to connect directly to the ILOM, can connect on ODA_BASE and use the command bellow:
# ipmitool sunoem cli
 Connected. Use ^D to exit.

ODA – Change ROOT Password

The procedure bellow describes how to change the password for ROOT user on ODA (Oracle Database Appliance).
  • On BOTH NODES login as ROOT ;
    1. create a copy of the directory $HOME/.ssh;
  • execute a test with ssh between the nodes ;
  1. from Node 1 :
     ssh <node 2> date, the password prompt should not appears
  2. from Node 2 :
     ssh <node 1> date, the password prompt should not appears
  • passwd root, change password for root user for BOTH NODES ;
  • execute a test with ssh between the nodes ;
    1. from Node 1 :
       ssh <node 2> date, the password prompt should not appears
    2. from Node 2 :
       ssh <node 1> date, the password prompt should not appears

ODA – Change Password for Oracle and GRID

The procedure bellow describes how to change the password for Oracle and GRID users on ODA (Oracle Database Appliance).
  • Login as ROOT on ODA_BASE, node 1 ;
  • create a copy of the directory /home/oracle/.ssh ;
  • create a copy of the directory /home/grid/.ssh ;
  • Login as ROOT on ODA_BASE, node 2 ;
  • create a copy of the directory /home/oracle/.ssh ;
  • create a copy of the directory /home/grid/.ssh ;
  • login as Oracle on Node1;
  • execute a test with ssh between the nodes ;
    1. from Node 1 :
       ssh <node 2> date, the password prompt should not appears
    2. from Node 2 :
       ssh <node 1> date, the password prompt should not appears
  • login as GRID ;
    1. from Node 1 :
       ssh <node 2> date, the password prompt should not appears
    2. from Node 2 :
       ssh <node 1> date, the password prompt should not appears
  • Login as ROOT on ODA_BASE, node 1 and node 2 ;
    1. passwd oracle, change password for oracle user;
    2. passwd grid, change password for grid user;
  • login as Oracle ;
    1. from Node 1 :
       ssh <node 2> date, the password prompt should not appears
    2. from Node 2 :
       ssh <node 1> date, the password prompt should not appears
  • login as GRID ;
    1. from Node 1 :
       ssh <node 2> date, the password prompt should not appears
    2. from Node 2 :
       ssh <node 1> date, the password prompt should not appears