Oracle RAC install and Linux 8 - SSH connectivity issue INS-06006

 Many systems are getting upgraded to Linux version 8 due to the impending end of life for Linux 7.  This includes both RedHat and Oracle Enterprise Linux.  In general most are choosing to do a new build out and migrate their databases from the previous system to the new system.  This is the most supportable model, though not the only.

One issue you may run into is installing either Oracle RAC Grid Infrastructure or Oracle Database in a clustered environment.  During the runInstaller or gridSetup.sh steps you may receive an error about "Passwordless SSH connectivity not setup" even though you have set it up and manually verified it.  This is due to a change in the SSH 8 to deal with a security vulnerability (CVE-2019-6111).  You will receive a message like this:

[FATAL] [INS-06006] Passwordless SSH connectivity not set up between the following node(s): [srvr02].

   CAUSE: Either passwordless SSH connectivity is not setup between specified node(s) or they are not reachable. Refer to the logs for more details.

   ACTION: Refer to the logs for more details or contact Oracle Support Services.

When you manually verify SSH connectivity everything works:

$ for i in srvr01 srvr02; do ssh $i hostname -s ; done

srvr01

srvr02

The fix is to make sure you are applying the RU / patch to the software during the install process.  Basically forcing the usage of --applyRU for runInstaller or gridSetup.sh.  This is also the recommended method for all patching / installs going forward (expect more of this in 23ai version).  For 19c you need to use 19.6 or above RU.

For GI install your command line would look something like:

/u01/app/19/grid_1/gridSetup.sh --applyRU /u01/app/grid/patch/36233126

There is a little caveat for this when it comes to database software install.  Normally you would run the install from the first node of the cluster and the installer would replicate the software to each node, using SSH.  The issue is that the RU for the database does not directly include the fix for SSH (Bug 30159782 - Remote Copy Fails if using openssh-7.2. 7.4, etc. (Doc ID 30159782.8)).  You have to include the OCW (Oracle ClusterWare) patch into the runInstaller steps.  The OCW patch is included in the GI RU not the database RU.

Image 1

The easiest way to deal with this is the use the Grid Infrastructure (GI) RU patch for both the GI home and the database home.  Though the syntax will look slightly different for the database home.  We will use the recent April 2024 RU as an example.  For the 19.23 GI RU we see the following patch contents on the download screen (see image 1).  In the red box's we can see that both the database patch and the OCW patch are included.

Once you have the the patch downloaded and unzipped we can verify that from the directory structure

36233126/
├── 36233263  <- Database RU 19.23
├── 36233343
├── 36240578  <- OCW patch 19.23
├── 36383196
├── 36460248
├── automation
├── bundle.xml
├── README.html
└── README.txt

So the steps for setting up a database home on RAC with Linux 8 would look like this:

1. Stage the patch files E.G. /u01/app/oracle/patch (along with any additional patches)
   
   unzip -qo p36233126_190000_Linux-x86-64.zip -d /u01/app/oracle/patch
   
   
2. Unzip the install media (19.3) into the home E.G. /u01/app/oracle/product/19/dbhome_2
   
   unzip -qo LINUX.X64_193000_db_home.zip -d /u01/app/oracle/product/19/dbhome_2
   
   
3. Update the OPatch version in the new home (be sure to download the latest OPatch from patch 6880880 in MOS)
   
   rm -rf /u01/app/oracle/product/19/dbhome_2/OPatch
   
   unzip -qo p6880880_190000_Linux-x86-64.zip -d /u01/app/oracle/product/19/dbhome_2
   
   
4. Install the software and apply the patches in one command (be sure to updated the highlighted items based on your configuration).  Here we are installing the RU, OCW, OJVM, and Datapump bundle patches.
   
   export CV_ASSUME_DISTID=OEL7.8
   
   /u01/app/oracle/product/19/dbhome_2/runInstaller \
   -silent -ignoreprereqfailure \
   -applyRU /u01/app/oracle/patch/36233126/36233263 \
   -applyOneOffs
   /u01/app/oracle/patch/36233126/36240578,/u01/app/oracle/patch/36420641,/u01/app/oracle/patch/36199232 \
   oracle.install.db.InstallEdition=EE \
   oracle.install.option=INSTALL_DB_SWONLY \
   INVENTORY_LOCATION=/u01/app/OraInventory \
   ORACLE_BASE=/u01/app/oracle \
   UNIX_GROUP_NAME=oinstall \
   oracle.install.db.CLUSTER_NODES=srvr01,srvr02 \
   oracle.install.db.OSDBA_GROUP=dba \
   oracle.install.db.OSOPER_GROUP=oper \
   oracle.install.db.OSBACKUPDBA_GROUP=backupdba \
   oracle.install.db.OSDGDBA_GROUP=dgdba \
   oracle.install.db.OSKMDBA_GROUP=kmdba \
   oracle.install.db.OSRACDBA_GROUP=racdba 
   
   Note: back slashes (\) are used to continue the command on to individual lines.  Also the patch list is comma separated with no spaces, and be sure to export the CV_ASSUME_DISTID which is required for Linux 8 when installing 19c.
   
   
5. After runInstaller finishes, run the root.sh script on every node you installed on.

If you make a mistake or have issues during the runInstaller, just remove the dbhome directory and start over.

Now you should have a new 19.23 home installed on all nodes of the cluster with the OCW patch, JVM, and DataPump patches all ready to go!