Oracle ACFS / ASM Filter Driver and Oracle Enterprise Linux

In the world of Oracle databases, DBA’s and system administrators don’t always see eye to eye.  Mostly I have found that this is really just about coming at challenges from different angles.  Some of the major changes in Oracle since 10g version has been the introduction of Oracle Automatic Storage Management or ASM.  This has led to many features such as ASMLIB, ASM Clustered Filesystem (ACFS), and now the ASM Filter Driver (AFD or ASMFD).  These features require OS specific compatible libraries, and kernel modules.  Information on compatibility can be confusing.

Background References

Let’s start with some background.  For this blog, I’m going to focus on Oracle Enterprise Linux (OEL).  The information here could be assembled for any supported OS.  This information is from a few sources, the main ones being:

• My Oracle Support (MOS) document “ACFS Support On OS Platforms (Certification Matrix). (Doc ID 1369107.1)”
• "Oracle Linux and Unbreakable Enterprise Kernel (UEK) Releases"By Simon Coter, SENIOR MANAGER, ORACLE LINUX AND VIRTUALIZATION PRODUCT MANAGEMENT https://blogs.oracle.com/scoter/oracle-linux-and-unbreakable-enterprise-kernel-uek-releases

Before making any major patching or upgrade decisions please reference these notes and Oracle support in general. 

Oracle does a pretty good job of providing information on Linux and Oracle database / ASM compatibility.  Though, the notes are not always clear as to what the goals is.  For example, I have heard many people say that ASM has stopped working during a Linux upgrade.  Or that ASM drivers are not compatible with Linux changes, or that you can’t patch Linux without breaking ASM / AFD / ACFS.  Well, these are antidotal statements, and the truth is a little more complex.

Linux Kernel Version

Oracle Enterprise Linux (OEL) recommends using the Unbreakable Enterprise Kernel or UEK.  The main reason behind this is that Oracle controls the destiny of the UEK.  As noted above ACFS and AFD are kernel dependent.  So having the ability to set course for both the Linux Kernel and the ASM related kernel modules is important.  For the scope of this discussion, let’s look at OEL releases 5 through 8 and Oracle Database releases 11gR2 through 19c.  First here is a high level view of OEL and UEK releases.

Support Dates	OEL Version
Jun 2007 - Jun 2017	OEL5
Feb 2011 - Mar 2021	OEL6
Jul 2014 - Jul 2024	OEL7
Jul 2019 - Jul 2029	OEL8
	UEK YUM Channel	UEKR1	UEKR2	UEKR3	UEKR4	UEKR5	UEKR6
	Major UEK Version	2.6.32	2.6.39	3.8.13	4.1.12	4.14.35	5.4.17
	Released		Mar, 2012	Oct, 2013	Jan, 2016	Jun, 2018	Mar, 2020

So this is a nice swath of about 10 years of versions.  Please note that the major version number of the kernels does not change if you don’t change UEK channels.  For example, if you move from OEL6 Update 5 to Update 8, but stick with UEKR3 your kernel major number will remain 3.8.13.  The minor numbers beyond the .13 will change, but not the major number.  This is also true for OEL7 and UEKR5 where the major release number is 4.14.35.  Within in this major version, Oracle release new security fixes and patch updates monthly or quarterly for each kernel version that is supported within the support dates.  So for example, OEL6 UEKR2 which was first release in Mar of 2012, just released kernel UEK 2.6.39-400.321.1 in April of 2020.  That is a long history of fixing the same set of code.

ACFS / AFD Linux Kernel Compatibility

Now let's move over to the database support for ACFS / AFD on OEL.  Here is another summary now listing Oracle database versions.

	UEK YUM Channel	UEKR1	UEKR2	UEKR3	UEKR4	UEKR5	UEKR6
	Major UEK Version	2.6.32	2.6.39	3.8.13	4.1.12	4.14.35	5.4.17
DB Version	Release Date		Mar, 2012	Oct, 2013	Jan, 2016	Jun, 2018	Mar, 2020
11.2.0.3	OEL5 - OEL6	2.6.32-100	2.6.39-100
11.2.0.4	OEL5 - OEL6	2.6.32-100	2.6.39-100	3.8.13-118	4.1.12-112.16.4
12.1.0.1	OEL5 - OEL6	2.6.32-100	2.6.39-100
12.1.0.2	OEL6 - OEL7		2.6.39-100	3.8.13-118
12.2.0.1	OEL6 - OEL7		2.6.39-100	3.8.13-118	4.1.12-112.16.4	4.14.35-1902
18c	OEL6 - OEL7		2.6.39-100	3.8.13-118	4.1.12-112.16.4	4.14.35-1902	TBD
19c	OEL7				4.1.12-112.16.4	4.14.35-1902	TBD

Here again I’ve covered about 10 years, and you can see clearly what high-level database versions are compatible with what OEL and UEK versions.  You might also see Oracle’s commitment to long term support for 11.2.0.4.  Similar to the expected support for 19c going until 2024.  Also note that UEKR6 is not yet supported.

Now the information above still high level.  You may assume that every 19c version of Oracle support all UEK4 and UEK5 kernels out of the box.  That is not true.  Database releases and kernel releases are not specifically synchronized.  So perhaps you have tried to test if the ACFS drivers can be installed:

$ acfsdriverstate supported -v
ACFS-9459: ADVM/ACFS is not supported on this OS version: '4.14.35-1902.10.8.el7uek.x86_64'
ACFS-9201: Not Supported

ACFS-9553: Operating System: linux
ACFS-9554: Machine Architecture: x86_64-linux-thread-multi
ACFS-9555: Operating system name and information: Linux srvr03 4.14.35-1902.10.8.el7uek.x86_64 #2 SMP Thu Feb 6 11:02:28 PST 2020 x86_64 x86_64 x86_64 GNU/Linux
ACFS-9556: Release package: oraclelinux-release-7.7-1.0.5.el7.x86_64
ACFS-9557: Version: ADVM/ACFS is not supported on 4.14.35-1902.10.8.el7uek.x86_64
ACFS-9558: Variable _ORA_USM_NOT_SUPPORTED is defined: no

ASM Required Patches

If your combination of grid software and Linux kernel are not supported, you will see the above message “ACFS-9201: Not Supported”.  How can you fix this?  Oracle provides patches at specific versions of GI to support UEK versions.

DB Version	OS Version	UEK Channel	UEK Minimum	Base bug or Base version
11.2.0.3.6	OEL5 - OEL6	UEKR1	2.6.32-100	15986571
11.2.0.3.7	OEL5 - OEL6	UEKR1	2.6.32-100	12983005
11.2.0.4	OEL5	UEKR1	2.6.32-100	Base
11.2.0.4	OEL5 - OEL6	UEKR2	2.6.39-100	Base
11.2.0.4.4	OEL6	UEKR3	3.8.13-118	16318126
11.2.0.4.6	OEL7	UEKR3	3.8.13-35	18321597
11.2.0.4.180717	OEL6 - OEL7	UEKR4	4.1.12-112.16.4	22810422, 28171094, 27463879
12.1.0.1	OEL5 - OEL6	UEKR1	2.6.32-100	Base
12.1.0.1	OEL5 - OEL6	UEKR2	2.6.39-100	Base
12.1.0.2.170718	OEL5 - OEL6	UEKR2	2.6.39-100	Base
12.1.0.2.170718	OEL6	UEKR3	3.8.13-13	Base
12.1.0.2.170718	OEL7	UEKR3	3.8.13-35	18321597
12.1.0.2.181016	OEL6 - OEL7	UEKR4	4.1.12-112.16.4	22810422, 27942938, 28171094, 27942938
12.1.0.2.190716	OEL7	UEKR5	4.14.35-1902	27494830
12.2.0.1	OEL6	UEKR2	2.6.39-100	Base
12.2.0.1	OEL6	UEKR3	3.8.13-13	Base
12.2.0.1.180717	OEL6	UEKR4	4.1.12-112.16.4	27463879, 28171094
12.2.0.1	OEL7	UEKR3	3.8.13-35	Base
12.2.0.1.180717	OEL7	UEKR4	4.1.12-112.16.4	27463879, 28171094, 27463879
12.2.0.181016	OEL7	UEKR5	4.14.35-1902	28069955
18.3.0.0	OEL6	UEKR2	2.6.39-100	Base
18.3.0.0	OEL6	UEKR3	3.8.13-13	Base
18.4.0.0.181016	OEL6	UEKR4	4.1.12-112.16.4	28069955
18.3.0.0	OEL7	UEKR3	3.8.13-35	Base
18.4.0.0.181016	OEL7	UEKR4	4.1.12-112.16.4	27463879, 28171094, 27463879
18.4.0.0.181016	OEL7	UEKR5	4.14.35-1902	27494830
19.3.0.0	OEL7	UEKR4	4.1.12-112.16.4	Base
19.4.190716	OEL7	UEKR5	4.14.35-1902	27494830

The example above on my test system is a fresh install of GI 19.3.0.0 on a OEL 7.8 system using UEKR5.  Based on the list above we can tell that for UEKR5 support I need either 19.4.190716 or I need patch 27494830.  Oracle provides many patches where the title contains “ACFS RU” that you can search for in the patches section of MOS.  I decided to use this patch “MERGE ON ACFS RU 19.6.0.0.0 OF 28531803 30685278 (Patch 31055785)” which includes the following merge of patches:

Patch  31055785     : applied on Fri May 22 14:15:34 CDT 2020
Unique Patch ID:  23448796
Patch description:  "ACFS Interim patch for 31055785"
   Created on 20 Mar 2020, 03:53:18 hrs PST8PDT
   Bugs fixed:
     28531803, 30685278, 27494830, 27917085, 28064731, 28293236, 28321248
     28375150, 28553487, 28611527, 28687713, 28701011, 28740425, 28818513
     28825772, 28844788, 28855761, 28860451, 28900212, 28951588, 28960047
     28995524, 29001307, 29030106, 29031452, 29039918, 29115917, 29116692
     29127489, 29127491, 29167352, 29173957, 29198743, 29229120, 29234059
     29250565, 29264772, 29302070, 29313039, 29318169, 29338628, 29339084
     29350729, 29363565, 29379932, 29391373, 29411007, 29417321, 29428155
     29437701, 29482354, 29484738, 29520544, 29524859, 29527221, 29551699
     29560075, 29586338, 29590813, 29604082, 29643818, 29704429, 29705711
     29721120, 29760083, 29779338, 29791186, 29848987, 29851205, 29862693
     29872187, 29893148, 29929003, 29929061, 29937236, 29941227, 29963428
     30003321, 30032562, 30046061, 30051637, 30057972, 30076951, 30093449
     30140896, 30179531, 30239575, 30251503, 30264950, 30269395, 30275174
     30324590, 30363621, 30655657
     

Which you can see includes the required patch 27494830 (first row, 3rd column). Depending on your version of Oracle, and quarterly bundle patch there are many options of patches available. The quarterly bundle patches for GI generally contain an updated ACFS patch as well.

 

Details of the ASM Kernel Modules 

In 19c, under the GRID_HOME is a directory called usm. On my test system under here I can find this:

$ ls $GRID_HOME/usm/install/Oracle/EL7UEK/x86_64
4.1.12/  4.1.12-112.16.4/  4.14.35-1902/

This is the location of the libraries used to compile the kernel modules for ACFS / AFD.  As expected, there is a directory for each major kernel version that pretty much matches our list above.  The command “amcmd afd_configure -e” is used to install the AFD kernel modules.  The AFD module use the Kernel Application Binary Interface and are installed under the /lib/modules location on your Linux system.  As they are considered weak modules, they are placed in one location and then symbolically linked into each kernel specific directory when the kernel is updated.

[root@srvr03 sbin]# ls -l /lib/modules/4.14.35-1902*/{extra,weak-updates}/oracle
/lib/modules/4.14.35-1902.0.9.el7uek.x86_64/extra/oracle:
total 9824
-rw-r--r-- 1 root root 10055682 Jun  5 13:16 oracleafd.ko

/lib/modules/4.14.35-1902.10.8.el7uek.x86_64/weak-updates/oracle:
total 0
lrwxrwxrwx 1 root root 69 Jun  5 13:17 oracleafd.ko -> /lib/modules/4.14.35-1902.0.9.el7uek.x86_64/extra/oracle/oracleafd.ko

/lib/modules/4.14.35-1902.11.3.1.el7uek.x86_64/weak-updates/oracle:
total 0
lrwxrwxrwx 1 root root 69 Jun  5 13:44 oracleafd.ko -> /lib/modules/4.14.35-1902.0.9.el7uek.x86_64/extra/oracle/oracleafd.ko

Here we can see that the primary module was installed under 4.14.35-1902.0.9.  Then when the kernel 4.14.35-1902.11.3.1 was installed, a symbolic link was created to the original module.

The AFD module is normally loaded by a Linux RC script called afd (/etc/init.d/afd).  You can verify it will be started at boot with the following command as the root user:

[root@srvr03 init.d]# chkconfig --list afd

Note: This output shows SysV services only and does not include native
      systemd services. SysV configuration data might be overridden by native
      systemd configuration.

      If you want to list systemd services use 'systemctl list-unit-files'.
      To see services enabled on particular target use
      'systemctl list-dependencies [target]'.

afd            	0:off	1:off	2:off	3:on	4:off	5:on	6:off

Here you can see that the AFD module will attempt to be loaded at run levels 3 and 5.  You can also check what version of the AFD module you have loaded with the following Linux command:

[oracle@srvr03 ~]$ modinfo oracleafd
filename:       /lib/modules/4.14.35-1902.11.3.1.el7uek.x86_64/weak-updates/oracle/oracleafd.ko
license:        Oracle Corporation
description:    ASM Filter Driver
author:         Oracle Corporation
srcversion:     533BB7E5866E52F63B9ACCB
depends:        
retpoline:      Y
name:           oracleafd
vermagic:       4.14.35-1902.0.9.el7uek.x86_64 SMP mod_unload modversions 
signat:         PKCS#7
signer:         
sig_key:        
sig_hashalgo:   md4
parm:           oracleafd_use_logical_block_size:Non-zero value implies using logical sectorsize (int)
parm:           oracleafd_use_maxio_softlimits:Non-zero value implies using conservative maximum block io limits (int)

Using the lsmod command to see if it is loaded.  If this command returns no lines, then it is not loaded. 

[oracle@srvr03 ~]$ lsmod |grep oracleafd
oracleafd             229376  0 

Conclusion

Ok, so hopefully this has given you a clear picture as to what you need to plan for when using ASM Filter Driver (AFD) or ACFS on your OEL system with UEK.  Once you have a base version of kernel and GI you should have little issues unless you jump UEK channels or do a GI major version change.  As many people are eyeing up 19c upgrades, hopefully the above information is helpful to prepare your Linux OS for the change.  For myself what started as a somewhat simple post about kernel drivers turned into a pretty long delve into the workings of the ASM filter driver.  Please leave your comments and questions.

Oracle 19c – Plan your upgrade now and “Carpe Indicii”

The year is 2020 and things are very turbulent.  Not only from a social and economic standpoint but also from a technical standpoint.  Specifically, we are talking about Oracle database and reaching the end of a decade for the 11g version.

11.2.0.1 came out about 2010 or late 2009 (depending on your platform).  11.2.0.4 patch set was first released in 2013 (again depending on your platform).  Free extended support ended in 2018.  Very much like 9.2.0.8 was widely adapted and used, customers have found 11.2.0.4 to be very stable for a long time.  This has allowed businesses to focus on application rollouts, or business priorities and not tackle the larger task of major database upgrades.  Even though the non-technical environment may not be the friendliest right now, unfortunately database upgrades just cannot wait any more.

You might say, well I’m on 12.1 which was released approximate in 2013 and free extended support ended in 2019.  That’s just barely a better state than 11g by less than a year.

I see three compelling reasons for upgrade:

Cost of support

First financially Oracle has already increase support fees if you want to gain fixes or patches for 11g or 12cR1.  That cost could be spent keeping your environment up to date or the cost could be spent on paying Oracle.  Of which Oracle will not provide value addition on that old version other than security fixes, and not even all of those. 

Cost can also be lost opportunity, what could your teams be using in the new features?  What if instead of fighting constant fires on poor performing out of date hardware or the same old limitations of the old software you were using the newest features?  So, when you think cost, think of all the things you are giving up which includes the bottom financial line as well.

Risk

This is a very broad category, but I’m going to keep it close to just a small area.  Risk of losing your main system.  This could be due to a security breach, this could be due to a hardware issue and not being able to get replacement parts, or it could be due to hitting a software issue that is not fixable in the current environment.  All of these risks go down by keeping somewhat up to date. 

Having software up to date helps reduce the risk of security breaches or hitting unrecoverable bugs.  Having hardware that is current means performing well and can be maintained.  Keeping both hardware and software current means having access to teams and talent that knows what they are doing. 

I worked with a customer a few years back trying to get help on an 8.1 Oracle database and had three different firms refuse to even talk to them.  A product not even 20 years old, and nobody would touch it.  I have also frequently worked on production systems where the only source for broken hardware is eBay.  Do you want to bet your company’s future on used parts off of eBay?  Where will you be in a few years? 

Timing

If not now, then when?  There are very few systems I have worked on where an upgrade did not take well over a year to do.  Why?  Well it starts with the idea, the call to action.  “The database needs to be upgraded”.  That’s great, but now what?  Generally, you need to get approval, which means putting plans together.  Budgeting financial resources, getting teams lined up, and possibly reaching out for external help.  That takes time. 

Then you have to make sure the teams are ready, have they learned what they need to know to do the upgrade?  How will they learn this? How long will it take?  Finally, the process of actually doing the upgrades. Starting in lower environments and moving through the IT lifecycle until finally reaching production.  All of this generally takes at least a year from conception to actualization if not longer. 

So, what if you start now?  What does 2021 or 2022 look like?  If you don’t start now are, we discussing 2022 and 2023 for upgrades?  What will be the climate of business and social economics be then?  I bet looking back you thought 2018 was a tough year, maybe not so much anymore?

I say “carpe indicii” translates to seize the information, but I like to say seize the data!

Start your upgrade plans now, make it a priority.  Spend the time needed to get your teams up to date.  Pull together the plan and funding needed and know where you are headed and why.  Change is hard, but fear of change is worse than the change itself.  Tens of thousands of systems upgraded from 8i to 10g and again to 11g and some even to 12c.  It is all possible, and it all works given the right effort and attention to detail.  If it didn’t work, we wouldn’t have the internet, large digital companies like Amazon, or Zoom meetings. Things are not perfect, but progress is inevitable, and we must keep moving forward.

Oracle 19c is 12.2.0.3 under the covers. Consider it your next 11.2.0.4 long term version.  You will probably be running it for the next 4+ years.   Possibly long after the system you upgrade is someone else’s responsibility. 

In my next couple of blog posts we will cover options for upgrading to 19c and some of the important testing and upgrade factors to maintain performance.

Mac OS X Catalina and Oracle Instant Client

If you have upgraded to the most recent Mac OS X Catalina, you may be seeing some issues trying to install and run the Oracle instant client.  This is part of new security put into OS X in the past few versions.

You may also being seeing this if you are just upgrading your instant client, which maybe you haven't done in a year or two.

Developer Cannot be Verified

After downloading the ZIP files from Oracle and installing the client, you try to run SQLPlus from the command line and you get the following error:

You can go to the Control Panel -> Security & Privacy -> General tab, and you will see a prompt asking if you would like to approve this application:

The problem with this method, is that every library and additional file related to the instant client also has to be approved.  So after you approve this, and then re-run sqlplus; you will get prompted for another file, and then another, and so on.

Extra File Attributes

If you note in the message, the file is being flagged since it was downloaded from the internet.  If we look at the files extra attributes we can see that they have all been flagged with "com.apple.quarantine"

$ ls -lg@

total 484136

-r-xr-xr-x@ 1 staff       5780 Feb 22  2019 BASIC_LICENSE

com.apple.quarantine       57 

-rw-r--r--@ 1 staff       1670 Aug 19 05:57 BASIC_README

com.apple.quarantine       57 

-r-xr-xr-x@ 1 staff       5780 Feb 22  2019 SQLPLUS_LICENSE

com.apple.quarantine       57 

-rw-r--r--@ 1 staff       1676 Sep  6 04:11 SQLPLUS_README

com.apple.quarantine       57 

-rwxr-xr-x@ 1 staff       9888 Aug 19 05:57 adrci

com.apple.quarantine       57 

-rwxr-xr-x@ 1 staff      40368 Aug 19 05:57 genezi

com.apple.quarantine       57 

-r-xr-xr-x@ 1 staff        342 Apr 24  2015 glogin.sql

com.apple.quarantine       57 

lrwxrwxrwx  1 staff         20 Jan 16 08:39 libclntsh.dylib -> libclntsh.dylib.19.1

lrwxrwxrwx  1 staff         20 Jan 16 08:39 libclntsh.dylib.10.1 -> libclntsh.dylib.19.1

lrwxrwxrwx  1 staff         20 Jan 16 08:39 libclntsh.dylib.11.1 -> libclntsh.dylib.19.1

lrwxrwxrwx  1 staff         20 Jan 16 08:39 libclntsh.dylib.12.1 -> libclntsh.dylib.19.1

lrwxrwxrwx  1 staff         20 Jan 16 08:39 libclntsh.dylib.18.1 -> libclntsh.dylib.19.1

-rwxr-xr-x@ 1 staff   98137256 Aug 13 02:10 libclntsh.dylib.19.1

com.apple.quarantine       57 

-rwxr-xr-x@ 1 staff    4803832 Jul  7  2019 libclntshcore.dylib.19.1

com.apple.quarantine       57 

-rwxr-xr-x@ 1 staff    8349944 Aug  6 12:14 libnnz19.dylib

com.apple.quarantine       57 

lrwxrwxrwx  1 staff         18 Jan 16 08:39 libocci.dylib -> libocci.dylib.19.1

lrwxrwxrwx  1 staff         18 Jan 16 08:39 libocci.dylib.10.1 -> libocci.dylib.19.1

lrwxrwxrwx  1 staff         18 Jan 16 08:39 libocci.dylib.11.1 -> libocci.dylib.19.1

lrwxrwxrwx  1 staff         18 Jan 16 08:39 libocci.dylib.12.1 -> libocci.dylib.19.1

lrwxrwxrwx  1 staff         18 Jan 16 08:39 libocci.dylib.18.1 -> libocci.dylib.19.1

-rwxr-xr-x@ 1 staff    1594048 Jul  5  2019 libocci.dylib.19.1

com.apple.quarantine       57 

-rwxr-xr-x@ 1 staff  125518764 Aug 19 05:57 libociei.dylib

com.apple.quarantine       57 

-r-xr-xr-x@ 1 staff     151980 Jul  3  2019 libocijdbc19.dylib

com.apple.quarantine       57 

-rwxr-xr-x@ 1 staff     100492 Aug 13 02:32 liboramysql19.dylib

com.apple.quarantine       57 

-rwxrwxrwx@ 1 staff    1322588 Sep  6 04:05 libsqlplus.dylib

com.apple.quarantine       57 

-r-xr-xr-x@ 1 staff    1659868 Jul  5  2019 libsqlplusic.dylib

com.apple.quarantine       57 

drwxr-xr-x@ 3 staff         96 Aug 19 05:57 network

com.apple.quarantine       57 

-rw-r--r--@ 1 staff    4210510 May 10  2019 ojdbc8.jar

com.apple.quarantine       57 

-rwxr-xr-x@ 1 staff       8528 Sep  6 04:11 sqlplus

com.apple.quarantine       57 

-rw-r--r--@ 1 staff    1680080 May 10  2019 ucp.jar

com.apple.quarantine       57 

-rwxr-xr-x@ 1 staff     150080 Aug 19 05:57 uidrvci

com.apple.quarantine       57 

-rw-r--r--@ 1 staff      74263 Apr 16  2019 xstreams.jar

com.apple.quarantine       57 

The at sign "@" is telling us that there are extra attributes on the files, and the OS security layer is using those attributes to apply rules about what can be run or not.

Simple Fix

So the easier fix is to just remove this attribute that is causing the security system to kick in.
NOTE, make sure you know where you downloaded the files from and that you are absolutely sure you are not introducing malware, ransomware, etc...

1. Some of the files ship without WRITE rights so we will need to fix that first:
   
   $ cd instantclient_19_3
   $ chmod u+w *
2. Next we will remove the extra attribute:
   
   $ cd ..
   $ xattr -r -d -s com.apple.quarantine instantclient_19_3
3. Now SQLPlus should run without any issues:
   
   $ cd instantclient_19_3/
   $ ./sqlplus /nolog
   
   
   SQL*Plus: Release 19.0.0.0.0 - Production on Thu Jan 16 09:01:29 2020
   Version 19.3.0.0.0
   
   
   Copyright (c) 1982, 2019, Oracle.  All rights reserved.
   
   
   SQL> 

Pretty simple fix as apposed to having to go into the control panel and approve every library or tool you want to use in the instant client set.

Gary

Oracle RAC + ASM + NFS home lab

Many people look for ways to emulate enterprise hardware in their home labs in order to get good hands on experience with Oracle products.  One of the harder ones to get working is Oracle RAC due to the need of multiple networks and shared storage.

With the plethora of virtualization software out now, this has gotten much easier.  For my job I needed a two node RAC setup to do some testing of DBCA (Database Creation Assistant) scripts.  I've done 100's of RAC installs, so other than the time, I didn't foresee any issues.  Well, as usual, what can go wrong will go wrong.

Basic RAC Setup

So lets jump to the basics.  For RAC I needed a public network, private network, and shared storage.  For the network pieces I also needed additional virtual IP's for each node as well as a set of three SCAN IP address.

I setup a new virtual private network for my servers.  I added all the new IP's to my DNS server.  Things were going well.  I then did the trick to create shared ASM disks on a shared NFS mount:

dd if=/dev/zero of=/u03/oradata/asm_dsk1 bs=1k count=10000000
dd if=/dev/zero of=/u03/oradata/asm_dsk2 bs=1k count=10000000
dd if=/dev/zero of=/u03/oradata/asm_dsk3 bs=1k count=10000000
dd if=/dev/zero of=/u03/oradata/asm_dsk4 bs=1k count=10000000
dd if=/dev/zero of=/u03/oradata/asm_dsk5 bs=1k count=10000000


I then changed the ownership:

chown grid:asmdba /u03/oradata/asm-dsk?
chmod 660 /u03/oradata/asm-dsk?

That second command, to change the permissions is what I missed.  This was somewhat critical.  It also speaks to a problem I've seen many times over.  How to deal with Oracle access issue when you have role separation setup.  The is particularly common when running Oracle E-Business suite due to the number of assumptions made by the Oracle procedures for cloning an EBS instance.

Troubleshooting Role separation issues.

As mentioned I've done this many times, and the only way to effectively troubleshoot these issues is to methodically go step by step through the setup of Oracle and Grid and ASM.  There are a few notes for help on this in MOS, but not a lot:

• UNIX: Diagnostic C program for ORA-1031 from CONNECT INTERNAL / AS SYSDBA (Doc ID 67984.1)
• ORA­15183 Unable to Create Database on Server using 11.2 ASM and Grid Infrastructure (Doc ID 1054033.1)
• Database Creation on 11.2 Grid Infrastructure with Role Separation ( ORA-15025, KFSG-00312, ORA-15081 ) (Doc ID 1084186.1)
• How To Recompile config.c / Relink Executables Of A Grid Infrastructure Home (Cluster) (Doc ID 1637766.1)
• Connect as SYSDBA on 11.2 Cloned Home Gives "ORA-1031: Insufficient Privileges" Error (Doc ID 1061788.1)

These all sound misleading, but lets go through the issue I faced.

The Issue

I had setup all my hardware and ASM raw disks.  After completing the GRID INFRASTRUCTURE installation, things looked good.  ASM was up and running, and the ASM diskgroup (DATA) looked fine.

I then went on to install the database home, and try to create my RAC database with DBCA.  That is when the trouble started.  During the initial create database statement I would get:
ORA-00200: control file could not be created
ORA-00202: control file: '+DATA'
ORA-15045: ASM file name '+DATA' is not in reference form
ORA-17502: ksfdcre:5 Failed to create file +DATA
ORA-27091: unable to queue I/O
ORA-27041: unable to open file


I jumped into the database alert log and DBCA logs, and garnered the following additional error:

Linux-x86_64 Error: 13: Permission denied

Additional information: 3

ORA-1501 signalled during: CREATE DATABASE "mydb2"

Ok, so time to go back and check everything. For role separation to work there is a OS group that is used to allow for the oracle and grid users to have permissions on the ASM disks.  In my case that group was supposed to be asmadmin.  Somewhere during the install I must have picked the wrong item in a pulldown box, or didn't pay attention to a fixup script that ran.  Either way I ended up with a difference.  So here is a list of steps to check that everything is right for role separation.  You need to check every item.

Role Separation Checklist

You should shutdown CRS before making any changes / updating any of these items.  As the root user run on all nodes: 

$GRID_HOME/bin/crsctl stop crs

1. Check the user id and group membership for the grid user:
   id grid
   (note the id number and group names, numbers the user is in)
2. Do the same check for the oracle user:
   id oracle
3. Make a list of the groups you are using for each role in oracle (ASM access, ASM super user, DB DBA, DB OPER, etc..)
4. As the grid user, check the configuration for the Grid infrastructure ($GRID_HOME/rdbms/lib/conf.c)
   #define SS_DBA_GRP "asmdba"
#define SS_OPER_GRP "asmoper"
#define SS_ASM_GRP "asmadmin"
   If changes are needed, update the config.c and re-link the oracle binary
   cd $GRID_HOME/rdbms/lib
make -f ins_rdbms.mk ioracle
5. As the oracle user do the same check in the oracle database home:
   ($ORACLE_HOME/rdbms/lib/conf.c)
   #define SS_DBA_GRP "dba"
#define SS_OPER_GRP "oper"
#define SS_ASM_GRP "asmadmin"
   (note that the SS_ASM_GRP has to match the grid home setting)
   If you need to make changes, again re-link the oracle binary
   cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk ioracle
6. Check that the oracle binary in the grid home has the right permissions and ownership. The user, group ownership are critical.  Also the sitcky bit has to be set for execution:
   ls -l $GRID_HOME/bin/oracle
-rwsr-s--x. 1 grid asmadmin 291225032 Jun  6 10:15 /u01/app/12.1.0/grid/bin/oracle
   If this is not correct then as the root user run:
   $GRID_HOME/bin/setasmgidwrap $GRID_HOME/bin/oracle
7. Do the same check for the database home oracle binary (again pay close attention to the user and group ownership as well as the sticky bit on the execution flag):
   ls -l $ORACLE_HOME/bin/oracle
-rwsr-s--x. 1 oracle asmadmin 323613264 Jun  7 16:08 /u01/app/oracle/product/12.1.0/dbhome_1/bin/oracle
   If this is not correct, then as the root user run:
   $GRID_HOME/bin/setasmgidwrap $ORACLE_HOME/bin/oracle
8. Check the ownership / permissions on the ASM source disks (in my case this was the raw files I had created on the shared NFS mount):
   ls -l /u03/oradata/asm*
-rw-rw----. 1 grid asmadmin 10240000000 Jun 11  2018 /u03/oradata/asm_dsk1
   (Again note the user and group ownership, as well as file permissions.  I was missing the write option on the group)
9. Check NFS mount settings.  Since I'm on a home lab using NFS, this is also important.  If your in a SAN or Exadata situation, you shouldn't have to check this. 
   I use the following settings, which are based on Oracle's recommended settings with the addition of "_netdev" which tells the startup scripts to wait until the network is started before trying to mount this specific mountpoint:
   rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,vers=3,timeo=600,actimeo=0,_netdev

You can now reboot or restart CRS (as the root user run $GRID_HOME/bin/crsctl start crs).

That should take care of permission issues when working with role separation.  It's important to go through every step, and verify everything.  I spent a few hours repeating these steps over a few times, having missed one thing or another.  So even with years of experience it's easy to miss a step.  What should have taken me about 30 minutes to resolve, ended up taking about 3 hours due to this.

Not the end of the world, but really should have been caught sooner.
Gary