Exadata X7-2 – Patch 18.1.5 – OVM

Quarterly Full Stack Download Patch For
Oracle Exadata (Apr 2018 – 18.0.0.0)

 

 

Version

Software

Notes See Note 1270094.1 for additional fixes that
address critical issues.

18.1.5.0.0

Patch
27503194
Storage server and InfiniBand switch software (18.1.5.0.0.180506)

Patch
27561908
– x86-64 Database server bare metal / domU ULN
exadata_dbserver_18.1.5.0.0_x86_64_base OL6 channel ISO image
(18.1.5.0.0.180506)

Patch
27561909

x86-64 Database server dom0 ULN
exadata_dbserver_dom0_18.1.5.0.0_x86_64_base OVM3 channel ISO image
(18.1.5.0.0.180506) 

Recommended for all Database releases to get the latest
functionality, bug fixes, and security fixes

Supplemental README Note 2362411.1

Install proper ACFS drivers before updating to this
release. See Supplemental README.

 

Note:

The patchmgr used for
updating Exadata storage servers is not the same as the patchmgr used for
applying the Exadata database server software update
.

Summary of Steps:

1.)
Grid Infra – Patch Below

[24 de maio de 2018 17:37:46 UTC+2] Carlos Magno Andrade
Junior (cmagno.andrade@gmail.com):
2.) InfiniBand

[24 de maio de 2018 17:38:00 UTC+2] Carlos Magno Andrade
Junior (cmagno.andrade@gmail.com):
3.) Cells

[24 de maio de 2018 17:38:07 UTC+2] Carlos Magno Andrade Junior
(cmagno.andrade@gmail.com):
4.) Db Node (DOM0 and DOMU)

 

Pre-Requisite:

The Patch below should be applied in each CRS for each VM
before start the Patching Process:

Release Oracle Database 12.2.0.1.180417 ACFSApr2018RU

Patch
27463879
:
TRACKING BUG FOR RECOMPILING USM DRIVERS WITH LATEST GCC

Note: As pre-requisite for the patch above the Grid
Infrastructure must be Patched with: 27468969

So, those 2 Patches must be present in each VM.

Reference: Oracle Exadata
Database Machine Patch Availability Document for CVE-2017-5715, CVE-2017-5753,
and CVE-2017-5754 (Doc ID 2356385.1)

 

Hints:

While executing the
UPGRADE Oracle does not recommend to use ILOM Console, there are some known
issues regarding halting the session. For this case I suggest use the NOHUP in
order to be safe regarding Network issues.

 

In some customer you
can have the SCREEN tool, this is also recommended to use.

 

Example:

[root@exa0021-dom0 dbserver_patch_5.180508]# vi
upgradeDOM0_exa0020-dom0.sh

 

Copy the command below:

./patchmgr -dbnodes /EXAVMIMAGES/procedure/patch_18_1_5/dbserver_patch_5.180508/dbnodes
-upgrade -iso_repo /EXAVMIMAGES/procedure/patch_18_1_5/p27561909_181000_Linux-x86-64.zip
-target_version 18.1.5.0.0.180506

 

[root@exa0021-dom0 dbserver_patch_5.180508]# chmod +x
upgradeDOM0_exa0020-dom0.sh

[root@exa0021-dom0 dbserver_patch_5.180508]# nohup
./upgradeDOM0_exa0020-dom0.sh &

 

è
Patch
27503194
– Storage server and InfiniBand switch software (18.1.5.0.0.180506)

 

Note:
The
patchmgr utility must be launched as the
root user from a
database server in the rack that has
root user SSH
equivalence set up to the
root user on all cells that you want to
update.

 

Note:
The Files that will be create in order to execute commands like DCLI, must uses
always (admin hostnames), example:
cell_group;

è
Procedure Patch Directory: /EXAVMIMAGES/procedure/patch_18_1_5/,
all files should be inside of this path.

è
Patchmgr comes inside of this patch: (Patch 27503194Storage server
and InfiniBand switch
software (18.1.5.0.0.180506)
) and will be
used to patch SWITCHES and CELLS.

 

Task

Description

Patching InfiniBand Switches

Create ibswitches.lst

Log
in as the
root user to a database server on
Oracle Exadata Database Machine that has
root user SSH access to the switches. The database server must be on the same InfiniBand network as the switches.

 

Unzip
the Patch 27503194

Change
to the
patch_release.date directory.

 

#
ENTER SWITCH INFO TO A FILE as ibswitches.lst ON THE FIRST DB NODE

[root@exadbadm01:~]$
cat ibswitches.lst

exasw-iba01

exasw-ibb01

 

Check Current Version

connect ib
switches with root user and run “version” command

Checking Subnet Manager

Connect in IB
Switches and execute :

getmaster -l

Checking SSH Connection

ssh <ib
switch> (Should not ask password)

 

If still asking
password follow the procedure below:

1.)
Check
if already exist:
~/.ssh/id_rsa.pub

2.)
If
YES, copy to IB Switch:
ssh-copy-id -i ~/.ssh/id_rsa.pub remote-host

3.)  Test the SSH connection: ssh
<ib switch> date, should not ask password;

 

If the rsa key
doesn’t exist, create a new one in order to copy:

 

Example:

 

jsmith@local-host$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/jsmith/.ssh/id_rsa):[Enter key]
Enter passphrase (empty for no passphrase): [Press enter key]
Enter same passphrase again: [Pess enter key]
Your identification has been saved in /home/jsmith/.ssh/id_rsa.
Your public key has been saved in /home/jsmith/.ssh/id_rsa.pub.
The key fingerprint is:
33:b3:fe:af:95:95:18:11:31:d5:de:96:2f:f2:35:f9 jsmith@local-host

  

Check Pre-Requisites

./patchmgr -ibswitches ibswitches.lst -upgrade -ibswitch_precheck

Apply Patch

./patchmgr -ibswitches ibswitches.lst -upgrade

Check Current Version

connect ib
switches with root user and run “version” command

Patching Storage Cells

EXACHK

Oracle
recommends running the Exachk utility before and after performing planned
maintenance. Running the utility allows you to review and cross reference
collected data against supported release levels and recommended Oracle
Exadata best practices. See My Oracle Support note
1070954.1, “Oracle Exadata
Database Machine Exachk or HealthCheck”
.

https://docs.oracle.com/cd/E75572_01/OEXUG/toc.htm

 

User Connection

Run
all steps as the
root user.

cell_group

Prepare
a file named
cell_group that has one cell host name or
IP address per line for each cell that you want to update.

root SSH
equivalence

Check for existing root SSH equivalence. The following
command should require no password prompts and no interaction. It should
return the list of host names in the
cell_group file.

dcli -g cell_group
-l root ‘hostname -i’

 

SSH root equivalence if not already done

Set up SSH root equivalence if not already done so from the launch server(DN NODE
0). Do not do this step if you already have
root SSH equivalence.

Generate root SSH keys as follows:

ssh-keygen -t rsa

Accept the defaults so the SSH keys are created for the
root user.

Push the SSH keys to set up SSH equivalence using the
following command. Enter the
root
password when prompted.

dcli -g cell_group
-l root -k

 

note 2362411.1

Oracle
recommends reviewing My Oracle Support note
2362411.1, check the issues and
workarounds listed in the note just before applying the update.

performing a
non-rolling update

 

All Oracle
Clusterware components must be offline. If you are performing a non-rolling
update in a configuration running Oracle VM, then you must check the Oracle
Clusterware state in all Oracle VM clusters

 

STOP CRS in all
VMs:

 

crsctl stop cluster -all
crsctl stop crs 
Note: Here is possible use dcli also to execute the commands. 
But check if you ROOT profile Inside of the VM is set to CRS Environment Variables.
 Also you need that SSH Equivalence already OK for all VMs.
 dcli -g VMs.lst -l root "crsctl stop crs" 

Stop Cell Services

Shut down all cell
services on all cells to be updated. This may be done by the
root user on each cell by running cellcli -e 'alter cell shutdown services all'
or by the following
dcli
command to do all cells at the same time:

dcli -g cell_group -l root "cellcli -e alter cell shutdown services all"
dcli -g cell_group -l root "cellcli -e list cell detail"

 

Check the software
release and the installed Oracle Linux package

If the system meets the following conditions, then
follow the instructions in My Oracle Support note
1589868.1 before applying this update to
storage servers and database servers.

  • Oracle Exadata Database Machine hardware uses Sun
    servers.
  • The installed Oracle Exadata Storage Server
    Software release is earlier than release 11.2.2.2.0.
  • Installed Linux ofa package is earlier than 1.5.1-4.0.28.

If the system meets all of the preceding conditions,
then Exadata Storage Servers and database servers running Oracle Linux may
encounter a file system corruption that results in the root file system
mounted as read-only after reboot.

Applying
the Update to Exadata Cells

 

  • Do not use the serial console, or the ILOM
    web-based console to start the patchmgr utility.

There is a known issue of a
system halt on the serial console when a write is attempted to
stderr or stdout. If an update is started from
the serial console, then it may halt.

You are using the serial console
if the output from the following command is
serial.

echo $consoletype

  • Use the ILOM web-based console to monitor the cell
    during the update. The ILOM web-based console is the only way to
    interact with the cell if there is a problem after a failed firmware
    update or if there is a file system problem that has to be dealt with.
    The ILOM web-based console is a requirement during the update process.

To obtain ILOM and serial
console access for the cells, use SSH to the ILOM host name or IP address
as the
root user. Do the following to start the
serial console:

start /SP/console

To stop it press the Escape key
(ESC) followed by
( and then stop /SP/console.

  • Use a fresh log in session for each update or
    rollback procedure. Do not run the rollback procedure from the same
    login session where the update was applied. Do not re-run an update from
    a login session where the rollback procedure was run.
  • Do not interrupt the update process once it is
    initiated on a cell.
  • If you must use a cell as the patchmgr utility
    launch system, then do not use
    /opt/oracle as the staging area for the update. Using /opt/oracle as the staging area causes
    the update to fail and corrupt the cell. Use the
    /tmp directory as the staging
    area, that is, unzip the files for the update in
    /tmp.
  • Cells automatically reboot, as needed, during the
    update process. Do not reboot or power cycle cells while applying the
    update.
  • Do not edit any log file or open log files in writable
    mode. You may use any of the following to view a log:
    view, less, more or tail. You may cause the update
    process to be interrupted by editing the log files during the update.
  • At the end of the patchmgr session, the patchmgr.stdout log file is divided into
    individual cell log files with names in the format of
    cell_name.log. In addition, the /var/log/cellos content from the inactive
    cell partition is copied to the
    /var/log/cellos/inactive_partition directory. To locate the
    inactive partition, use the following command:

imageinfo -inactive -sys

 

Monitoring
Update Activity

 

Monitor update activity
using
less
-rf patchmgr.stdout
from another terminal session or window to see raw log details
from the patchmgr utility.
You may monitor
activity of the cell, by logging in to the serial console or web-based ILOM
console of individual cells being updated 5 minutes after the patchmgr
utility has started. Waiting 5 minutes allows the patchmgr utility time to
reset the ILOM on Sun hardware. Reset of the ILOM disconnects you from the
ILOM web console and serial console. You can reconnect once the ILOM has been
reset. By waiting 5 minutes, you avoid having to reconnect.

You lose the connection
during any ILOM update, and need to reconnect. The ILOM does not show any update
actions.

It is helpful to
monitor the activities of the normal cell boot, reboot, and other activities
to ensure that the process is proceeding correctly.

 

Using
the patchmgr Utility

 

1.)
Log in to a
system that has SSH equivalence set up for the
root user to all cells that are to
be updated. ( DBNODE);

 

2.)
Unzip the
file p27503194_181000_Linux-x86-64.zip (If not Already done in “Patch IB
Switch”);

 

3.)
Change to
the
patch_18.1.5.0.0.180506 directory

 

4.)
You use the
patchmgr update utility for updating Oracle Exadata storage servers. For
Exadata storage server updates the utility is packaged (and shipped) with the
update itself and is available for download from My Oracle Support as storage
server update.

https://docs.oracle.com/cd/E80920_01/DBMMN/updating-exadata-software.htm#DBMMN-GUID-B0A6A060-1C69-43D1-8780-B59B5E585C10

 

5.)  Patchmgr is inside:
../p27503194_181000_Linux-x86-64/patch_18.1.5.0.0.180506

patchmgr

 

6.)
Verify SSH
access to cells that was configured earlier with the following command:

dcli -g cell_group -l root 'hostname -i'

7.)
(Recommended)
Reset the patchmgr state to a known state using the following command:

./patchmgr -cells
cell_group -reset_force

 

Note:

This
step is only done the first time the cells are updated to this release. It
is not necessary to use the command for subsequent cell updates, even after
rolling back the update.

 

8.)
Clean up any
previous patchmgr utility runs using the following command:

 

./patchmgr -cells
cell_group -cleanup

 

Note: Always use the -cleanup option before retrying a failed or halted run of the
patchmgr utility.

 

9.)
Apply

 

Note: Recommended use NOHUP or
SCREEN Linux tool to execute, by this way avoiding any network issue.

 

./patchmgr
-cells cell_group -patch_check_prereq

./patchmgr
-cells cell_group -patch

 

Folow
the process, in another session:

[root@exadbadm01 patch_<your patch number>]$ tail -f patchmgr.stdout

Note:

The
ILOM restarts approximately 5 minutes after starting the patchmgr utility.
When cell update starts, a countdown runs on the session console where the
patchmgr utility was run from. Connections to monitor the console through
ILOM disconnect when ILOM restarts, and must be reconnected to continue to
monitor the console.

 

10.)
 Check image
status and history using the
imageinfo and imagehistory commands on each cell:

 

[root@exadbadm01:~]$ dcli -g cell_group -l root 'imageinfo'
[root@exadbadm01:~]$ dcli -g cell_group -l root 'imagehistory'
[root@exadbadm01:~]$ dcli -g cell_group -l root 'cellcli -e list alerthistory'
[root@exadbadm01:~]$ dcli -g cell_group -l root "grep -i fail /var/log/cellos/validations.log"
[root@exadbadm01:~]$ ibhosts
 

Patching Database Nodes (DOM0 and DOMU)

patch
21634633

Before
starting go this note and download the suitable version of dbserver.patch.
Download dbserver.patch.zip as p21634633_12*_Linux-x86-64.zip, which contains
dbnodeupdate.zip and patchmgr for dbnodeupdate orchestration 

 

Patch
21634633: DBSERVER.PATCH.ZIP ORCHESTRATOR PLUS DBNU – ARU PLACEHOLDER

 

Last Updated

10-May-2018 17:58 (16 days ago)

Product

Oracle Exadata Storage Server Software

(More…)

Release

Oracle Exadata Storage Server 18.1.5.0.0

Platform

Linux x86-64

PATCHING DOM0

Unzip p21634633_181500_Linux-x86-64.zip

Unzip File: p21634633_181500_Linux-x86-64.zip,
inside all DOM-0s (/EXAVMIMAGES/procedure/patch_18_1_5) , in order to be used
as base for Patching Dom0 and DomU.

 

Sequence:

exa0020-dom0
Patches exa0021-dom0

exa0021-dom0
Patches exa0020-dom0

 

dbnodes file

è  exa0020-dom0:

/EXAVMIMAGES/procedure/patch_18_1_5/dbserver_patch_5.180508

 

[root@exa0020-dom0
dbserver_patch_5.180508]# cat dbnodes

exa0021-dom0

 

è  exa0021-dom0:

 

/EXAVMIMAGES/procedure/patch_18_1_5/dbserver_patch_5.180508

 

[root@exa0021-dom0
dbserver_patch_5.180508]# cat dbnodes

exa0020-dom0

 

SSH Equivalence

All DBnodes as
root must not ask for password during SSH connection between then

 

Check:

dcli -g dbnodes -l
root ‘hostname -i’

 

If it’s not
configured the SSH equivalence please do the follow:

 

%
ssh-keygen

% dcli –g
dbnodes –l root –k

% dcli -g dbnodes
-l root ‘hostname -i’

 

Backup /etc/sysconfig/network-scripts

è  All DOM-0s

 

cd /etc/sysconfig/network-scripts

mkdir backup

cp * backup

 

Note: Save this
output for all DOM0s

cd /etc/sysconfig

 

[root@exa0020-dom0
sysconfig]# cat network

NETWORKING=yes

NOZEROCONF=yes

HOSTNAME=exa0020-dom0.domain.client

NETWORKING_IPV6=no

IPV6INIT=no

GATEWAY=10.177.97.1

GATEWAYDEV=vmeth0

 

[root@exa0021-dom0
sysconfig]# cat network

NETWORKING=yes

NOZEROCONF=yes

HOSTNAME=exa0021-dom0.domain.client

NETWORKING_IPV6=no

IPV6INIT=no

GATEWAY=10.177.97.1

GATEWAYDEV=vmeth0

 

Pre-Requites Check

PATCH: p27561909_181000_Linux-x86-64.zip

The patch p27561909_181000_Linux-x86-64.zip
must inside
/EXAVMIMAGES/procedure/patch_18_1_5/

 

cd /EXAVMIMAGES/procedure/patch_18_1_5/dbserver_patch_5.180508

 

./patchmgr
-dbnodes /EXAVMIMAGES/procedure/patch_18_1_5/dbserver_patch_5.180508/dbnodes
-precheck -nomodify_at_prereq -log_dir auto -target_version 18.1.5.0.0.180506
-iso_repo /EXAVMIMAGES/procedure/patch_18_1_5/p27561909_181000_Linux-x86-64.zip

 

 

************************************************************************************************************

NOTE    patchmgr
release: 5.180508 (always check MOS 1553103.1 for the latest release of
dbserver.patch.zip)

NOTE

WARNING Do not
interrupt the patchmgr session.

WARNING Do not
resize the screen. It may disturb the screen layout.

WARNING Do not
reboot database nodes during update or rollback.

WARNING Do not
open logfiles in write mode and do not try to alter them.

************************************************************************************************************

2018-05-29
17:38:13 +0200        :Working: DO: Initiate precheck on 1 node(s)

2018-05-29
17:39:26 +0200        :Working: DO: Check free space and verify SSH
equivalence for the root user to exa0020-dom0

2018-05-29
17:40:14 +0200        :SUCCESS: DONE: Check free space and verify SSH equivalence
for the root user to exa0020-dom0

2018-05-29
17:41:04 +0200        :Working: DO: dbnodeupdate.sh running a precheck on
node(s).

2018-05-29
17:43:15 +0200        :SUCCESS: DONE: Initiate precheck on node(s).

 

 

Upgrade DBNode

./patchmgr
-dbnodes /EXAVMIMAGES/procedure/patch_18_1_5/dbserver_patch_5.180508/dbnodes
-upgrade -iso_repo /EXAVMIMAGES/procedure/patch_18_1_5/p27561909_181000_Linux-x86-64.zip
-target_version 18.1.5.0.0.180506

Checking

[root@exa0021-dom0
~]# imageinfo

 

Kernel
version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018
x86_64

Image kernel version: 4.1.12-94.8.4.el6uek

Image version: 18.1.5.0.0.180506

Image
activated: 2018-05-29 17:15:16 +0200

Image
status: success

System
partition on device: /dev/mapper/VGExaDb-LVDbSys3

 

[root@exa0021-dom0
~]# imagehistory

Version                              : 18.1.4.0.0.180125.3

Image activation date                : 2018-05-08 15:56:04 +0200

Imaging mode                         : fresh

Imaging
status                       : success

 

Version
: 18.1.5.0.0.180506

Image
activation date                : 2018-05-29 17:15:16 +0200

Imaging
mode                         : patch

Imaging
status                       : success

PATCHING DOMU

dbnodes file

 

è  exa0020-dom0:

 

[root@exa0020-dom0
dbserver_patch_5.180508]# xm list

Name
ID   Mem VCPUs      State   Time(s)

Domain-0
0  8011     4     r—–   1034.4

exabdt01.domain.client
1 148483     8     -b—-    384.6

exabdt99.domain.client
2 16387     2     -b—-    275.4

exaint01.domain.client
3 98307     4     -b—-    378.6

exaint03.domain.client
4 102403     4     -b—-    504.1

exaogr03.domain.client
5  2099     2     -b—-    142.2 (OEMCC)

 

/EXAVMIMAGES/procedure/patch_18_1_5/dbserver_patch_5.180508

 

[root@exa0020-dom0
dbserver_patch_5.180508]# cat VMs.lst

exabdt01

exabdt99

exaint01

exaint03

exaogr03

 

 

è  exa0021-dom0:

 

[root@exa0021-dom0
dbserver_patch_5.180508]# xm list

Name
ID   Mem VCPUs      State   Time(s)

Domain-0
0  8030     4     r—–  33965.0

exaint02.domain.client
1 98307     4     -b—-  15708.3

exaogr04.domain.client
2  2099     2     -b—-   2043.8 (OEMCC)

exaprd04.domain.client
3 117763     8     -b—-  18366.4

exaprd05.domain.client
4 133123     8     -b—-  18500.4

 

/EXAVMIMAGES/procedure/patch_18_1_5/dbserver_patch_5.180508

 

[root@exa0021-dom0
dbserver_patch_5.180508]# cat VMs.lst

exaint02

exaprd04

exaprd05

exaogr04

 

Check Connectivity

for
x in `cat VMs.lst`

>
do

>
ping $x

>
done

SSH Equivalence
(All DOM0s -> DOMUs)

dcli -g VMs.lst -l root ‘hostname -i’

 

if still does not
exist the SSH, please execute the commands below:

 

ssh-copy-id -i
~/.ssh/id_rsa.pub exabdt01

ssh-copy-id -i
~/.ssh/id_rsa.pub exabdt99

ssh-copy-id -i
~/.ssh/id_rsa.pub exaint01

ssh-copy-id -i
~/.ssh/id_rsa.pub exaint03

ssh-copy-id -i
~/.ssh/id_rsa.pub exaogr03

 

Check:

 

[root@exa0020-dom0
dbserver_patch_5.180508]# dcli -g VMs.lst -l root ‘hostname -i’

exabdt01:
10.177.97.33

exabdt99:
10.177.97.43

exaint01:
10.177.97.34

exaint03:
10.177.97.36

exaogr03:
10.177.97.41

 

Pre-Requites Check

PATCH: p27561908_181000_Linux-x86-64.zip

Note: As
recommendation avoid to execute ALL VMs in just one shot. Start with 1 VM per
time.

 

File created: DOMU.lstm (contain just one VM)

 

./patchmgr
-dbnodes /EXAVMIMAGES/procedure/patch_18_1_5/dbserver_patch_5.180508/
DOMU.lst -precheck -nomodify_at_prereq
-log_dir auto -target_version 18.1.5.0.0.180506 -iso_repo /EXAVMIMAGES/procedure/patch_18_1_5/
p27561908_181000_Linux-x86-64.zip

 

************************************************************************************************************

NOTE    patchmgr
release: 5.180508 (always check MOS 1553103.1 for the latest release of
dbserver.patch.zip)

NOTE

WARNING Do not
interrupt the patchmgr session.

WARNING Do not
resize the screen. It may disturb the screen layout.

WARNING Do not
reboot database nodes during update or rollback.

WARNING Do not
open logfiles in write mode and do not try to alter them.

************************************************************************************************************

2018-05-30
13:12:16 +0200        :Working: DO: Initiate precheck on 4 node(s)

2018-05-30
13:13:54 +0200        :Working: DO: Check free space and verify SSH
equivalence for the root user to node(s)

2018-05-30
13:17:37 +0200        :SUCCESS: DONE: Check free space and verify SSH
equivalence for the root user to node(s)

2018-05-30
13:18:40 +0200        :Working: DO: dbnodeupdate.sh running a precheck on
node(s).

2018-05-30
13:20:22 +0200        :SUCCESS: DONE: Initiate precheck on node(s).

 

è  Connect in each VM and check the
IMAGEINFO and IMAGEHISTORY

 

[root@exabdt99 ~]#
imageinfo

 

Kernel version:
4.1.12-94.7.8.el6uek.x86_64 #2 SMP Thu Jan 11 20:41:01 PST 2018 x86_64

Image kernel
version: 4.1.12-94.7.8.el6uek

Image version:
18.1.4.0.0.180125.3

Image activated:
2018-05-18 08:51:40 +0200

Image status:
success

System partition
on device: /dev/mapper/VGExaDb-LVDbSys1

 

[root@exabdt99 ~]#
imagehistory

Version
: 18.1.4.0.0.180125.3

Image activation
date                : 2018-05-18 08:51:40 +0200

Imaging
mode                         : fresh

Imaging
status                       : success

 

Note (ERROR):

 

If you found the
ERROR during the pre-check procedure with the message below:

 

exaint01:
Warning: Active network mounts found on this DB node

 

Please umount all
NFS mountpoints inside of the VM, and after the Upgrade you mount again if
it’s needed.

 

umount
/dmp_mig_oda_ZFS

umount /patches

umount /cloudfs

 

Network Config

There is an issue
regarding the routes specific for CUSTOMER environment.

Checking the CRS Service
inside of the VM you may found the NETWORK service down, this occurs because
of the GATEWAY is not reachable.

 

In order to fix it, the
configuration below must be executed.

 

The line below must be
inserted into ifcfg-eth1 / ifcfg-eth2 files. This is needed for LACP – ALL
DOM0s

 

ETHTOOL_OPTS=”speed
10000 duplex full autoneg off”

 

ifup eth1

ifup eth2

 

ethtool eth1

ethtool eth2

 

Note: The link
must be yes

 

After
that check the services inside of the VMs. Everything should be online and
the gateway now is pingable.

 

crsctl
stat res -t

 

Upgrade DOMUs

As
best practice, don’t execute all VMs at the same time, so create a specific
file just with one HOSTANAME.

 

File:

DOMU.lst

 

DOMUs:

– exa0020-dom0:

 

exabdt01

exabdt99

exaint01

exaint03

exaogr03

 

– exa0021-dom0:

 

exaint02

exaprd04

exaprd05

exaogr04

 

 

è
Prepare a
shell script to be used in NOHUP, in case you don’t have screen tool.

 

[root@exa0020-dom0
dbserver_patch_5.180508]# vi patchDOMU.sh

[root@exa0020-dom0
dbserver_patch_5.180508]# cat patchDOMU.sh

./patchmgr
-dbnodes /EXAVMIMAGES/procedure/patch_18_1_5/dbserver_patch_5.180508/DOMU.lst
-upgrade -iso_repo /EXAVMIMAGES/procedure/patch_18_1_5/p27561908_181000_Linux-x86-64.zip
-target_version 18.1.5.0.0.180506

 

[root@exa0020-dom0
dbserver_patch_5.180508]# chmod +x patchDOMU.sh

 

nohup
./patchDOMU.sh &

 

follow
the process:

nohup.out

patchmgr.trc

 

 

************************************************************************************************************

NOTE
patchmgr release: 5.180508 (always check MOS 1553103.1 for the latest release
of dbserver.patch.zip)

NOTE

NOTE
Database nodes will reboot during the update process.

NOTE

WARNING
Do not interrupt the patchmgr session.

WARNING
Do not resize the screen. It may disturb the screen layout.

WARNING
Do not reboot database nodes during update or rollback.

WARNING
Do not open logfiles in write mode and do not try to alter them.

************************************************************************************************************

2018-05-3013:33:59 +0200        :Working: DO: Initiate prepare steps on node(s).

2018-05-3013:34:00 +0200        :Working: DO: Check free space and verify SSH
equivalence for the root user to exabdt99

2018-05-3013:35:04 +0200        :SUCCESS: DONE: Check free space and verify SSH
equivalence for the root user to exabdt99

2018-05-3013:36:24 +0200        :SUCCESS: DONE: Initiate prepare steps on node(s).

2018-05-3013:36:24 +0200        :Working: DO: Initiate update on 1 node(s).

2018-05-3013:36:24 +0200        :Working: DO: dbnodeupdate.sh running a backup on 1
node(s).

2018-05-3013:39:12 +0200        :SUCCESS: DONE: dbnodeupdate.sh running a backup on 1
node(s).

2018-05-3013:39:12 +0200        :Working: DO: Initiate update on node(s)

2018-05-3013:39:18 +0200        :Working: DO: Get information about any required OS
upgrades from node(s).

2018-05-3013:39:29 +0200        :SUCCESS: DONE: Get information about any required OS
upgrades from node(s).

2018-05-3013:39:29 +0200        :Working: DO: dbnodeupdate.sh running an update step on
all nodes.

|||||

2018-05-3013:48:29 +0200        :INFO   : exabdt99 is ready to reboot.

2018-05-3013:48:29 +0200        :SUCCESS: DONE: dbnodeupdate.sh running an update step
on all nodes.

2018-05-3013:48:29 +0200        :Working: DO: Initiate reboot on node(s)

2018-05-3013:48:46 +0200        :SUCCESS: DONE: Initiate reboot on node(s)

2018-05-3013:48:46 +0200        :Working: DO: Waiting to ensure exabdt99 is down before
reboot.

2018-05-3013:49:13 +0200        :SUCCESS: DONE: Waiting to ensure exabdt99 is down
before reboot.

2018-05-3013:49:13 +0200        :Working: DO: Waiting to ensure exabdt99 is up after
reboot.

2018-05-3013:50:37 +0200        :SUCCESS: DONE: Waiting to ensure exabdt99 is up after
reboot.

2018-05-3013:50:37 +0200        :Working: DO: Waiting to connect to exabdt99 with SSH.
During Linux upgrades this can take some time.

2018-05-3013:51:20 +0200        :SUCCESS: DONE: Waiting to connect to exabdt99 with
SSH. During Linux upgrades this can take some time.

2018-05-3013:51:20 +0200        :Working: DO: Wait for exabdt99 is ready for the
completion step of update.

2018-05-3013:51:21 +0200        :SUCCESS: DONE: Wait for exabdt99 is ready for the
completion step of update.

2018-05-3013:51:26 +0200        :Working: DO: Initiate completion step from
dbnodeupdate.sh on node(s)

2018-05-3013:56:36 +0200        :SUCCESS: DONE: Initiate completion step from
dbnodeupdate.sh on exabdt99

2018-05-3013:57:22 +0200        :SUCCESS: DONE: Initiate update on node(s).

2018-05-3013:57:22 +0200        :SUCCESS: DONE: Initiate update on 1 node(s).

 

Command:

./patchmgr
-dbnodes /EXAVMIMAGES/procedure/patch_18_1_5/dbserver_patch_5.180508/DOMU.lst

 -upgrade
-iso_repo /EXAVMIMAGES/procedure/patch_18_1_5/p27561908_181000_Linux-x86-64.zip
-target_version 18.1.5.0.0.180506

Check Pos Upgrade

Inside
of the VM :

exabdt99

  

[root@exabdt99 ~]#
imageinfo

 

Kernel version:
4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64

Image kernel
version: 4.1.12-94.8.4.el6uek

Image version: 18.1.5.0.0.180506

Image
activated: 2018-05-30 13:51:19 +0200

Image status:
success

System partition on device:
/dev/mapper/VGExaDb-LVDbSys1

 

[root@exabdt99
~]# imagehistory

Version
: 18.1.4.0.0.180125.3

Image
activation date                : 2018-05-18 08:51:40 +0200

Imaging
mode                         : fresh

Imaging
status                       : success

 

Version
: 18.1.5.0.0.180506

Image
activation date                : 2018-05-30 13:51:19 +0200

Imaging
mode                         : patch

Imaging
status                       : success

ZFS Mount Points

In
case of UMOUNT ZFS mount points please mount again the MPs inside of the VMs,
if it’s not mounted yet:

 

mount
/cloudfs

mount
/patches

mount
/dmp_mig_oda_ZFS

 

Check Network

This
is just a double comparing the current scripts with the backup done before.
Sometimes the routes are lost.

 

Dom0:

cd
/etc/sysconfig/network-scripts

 

for
x in `ls -ltr | awk ‘{print $9}’`

>
do

>
echo “Checking …” $x

>
diff $x backup/$x

>
echo ————————————————————————

>
done

 

 

Actions into the
VM

ssh
<vm>

 

.
oraenv

+ASM

 

crsctl stat res -t

(check
all resources)

 

Check Oracle_Home
for Databases

Check
if the ORACLE_HOME is using RDS protocol

  

cd
/u01/app/oracle/product/12.1.0.2/dbhome_1/bin

./skgxpinfo
-v

 

Oracle
RDS/IP (generic)

 

è
If the protocol is not the good one, relink the ORACLE_HOME for Databases.

 

Before
stop a resources that uses the ORACLE_HOME that is going to be relinked.

 

    * (as oracle) cd $ORACLE_HOME/rdbms/lib
    * (as oracle) make -f ins_rdbms.mk ipc_rds ioracle

 

 

 

Check SSH and 
LDAP Services
Check inside of each VM if the SSH and LDAP connections 
are OK.


service nslcd status
service nscd status
service sssd status


If the services are down execute the commands below:


chkconfig nslcd on
chkconfig nscd on
chkconfig sssd on

service nslcd start
service nscd start
service sssd start

Useful Commands:

 

è
Shutdown
VM:

xm
shutdown exaint01.domain.client

 

è
Startup
VM:

xm create /EXAVMIMAGES/GuestImages/exaint01.domain.client/vm.cfg

 

è
Check
VMs:

xm list

 

è
Get Console:

xm console exaint01.domain.client

 

è
ETHTOOL
check Ethernet Connections:

ethtool
eth1

 

è
Check
Bridges:

brctl
show

 

References:
https://docs.oracle.com/cd/E80920_01/DBMMN/updating-exadata-software.htm#DBMMN-GUID-5A36B89A-34AB-46D1-95AB-896056C92727
Exadata 18.1.5.0.0 release and patch (27503194) (Doc ID 2362411.1)
Patch 27503194Storage server and InfiniBand switch software (18.1.5.0.0.180506)
Patch 27561908 – x86-64 Database server bare metal / domU ULN exadata_dbserver_18.1.5.0.0_x86_64_base OL6 channel ISO image (18.1.5.0.0.180506)
Patch 27561909 – x86-64 Database server dom0 ULN exadata_dbserver_dom0_18.1.5.0.0_x86_64_base OVM3 channel ISO image (18.1.5.0.0.180506)
Patch 27463879: TRACKING BUG FOR RECOMPILING USM DRIVERS WITH LATEST GCC
Exadata Critical Issues (Doc ID 1270094.1)
Oracle Exadata Database Machine Patch Availability Document for CVE-2017-5715, CVE-2017-5753, and CVE-2017-5754 (Doc ID 2356385.1)
Patch 21634633: DBSERVER.PATCH.ZIP ORCHESTRATOR PLUS DBNU – ARU PLACEHOLDER

 

5 comments

  1. Hi,
    We are relatively new Exadata Users, wondering how you use or configure activities on the module for host_access_control. This seems to run after a patch reboot and changes many of our security settings. Very limited information from Oracle on this topic. Do you have advice on host_access_control? PS. we are apply this new patch 18.0.1.5, as well)
    Thanks in advance
    R Mahoney

    Like

    • Good Morning, i tried find some info about that, and really there is no detailed info to describe the steps..

      Basic these 3 Notes:
      File changes for sshd/pam.d/iptables etc. are lost after Exadata CELL/Database node patching/reboot (Doc ID 2362792.1)
      Oracle Exadata Storage Server Security Configuration Supplement for the United States Department of Defense (Doc ID 2274231.1)
      Guidelines for enhancing the security for an Oracle Database Machine deployment (Doc ID 1068804.1)

      The main message from Oracle:
      “host_access_control keeps Exadata CELL/Database Nodes to comply with security requirements.

      Customer can still update and change files but they will be lost after next reboot/patching – instead host_access_control tool should be used to run all changes so they won’t be lost.

      In some cases even if host_access_control was properly used – some settings can be still reverted back to default values to actually match Exadata security policies and customer will need to get all further information from Exadata Oracle Support Team.

      Please contact Exadata Oracle Support team to obtain further details”

      I didn’t face any issue regarding that on my last intervention. So my advise is work with the Oracle Support.

      If i have any further information i let you know.

      Thanks for follow the Blog.

      Like

  2. Wow, superb blog layout! How long have you been blogging for? you made blogging look easy. The overall look of your website is excellent, let alone the content!

    Like

  3. It’s actually a great and useful piece of information. I’m glad that you just shared this helpful info with us. Please keep us up to date like this. Thank you for sharing.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s