Wednesday, June 17, 2020

Oracle ACFS / ASM Filter Driver and Oracle Enterprise Linux

In the world of Oracle databases, DBA’s and system administrators don’t always see eye to eye.  Mostly I have found that this is really just about coming at challenges from different angles.  Some of the major changes in Oracle since 10g version has been the introduction of Oracle Automatic Storage Management or ASM.  This has led to many features such as ASMLIB, ASM Clustered Filesystem (ACFS), and now the ASM Filter Driver (AFD or ASMFD).  These features require OS specific compatible libraries, and kernel modules.  Information on compatibility can be confusing.

Background References

Let’s start with some background.  For this blog, I’m going to focus on Oracle Enterprise Linux (OEL).  The information here could be assembled for any supported OS.  This information is from a few sources, the main ones being:
Before making any major patching or upgrade decisions please reference these notes and Oracle support in general. 

Oracle does a pretty good job of providing information on Linux and Oracle database / ASM compatibility.  Though, the notes are not always clear as to what the goals is.  For example, I have heard many people say that ASM has stopped working during a Linux upgrade.  Or that ASM drivers are not compatible with Linux changes, or that you can’t patch Linux without breaking ASM / AFD / ACFS.  Well, these are antidotal statements, and the truth is a little more complex.

Linux Kernel Version

Oracle Enterprise Linux (OEL) recommends using the Unbreakable Enterprise Kernel or UEK.  The main reason behind this is that Oracle controls the destiny of the UEK.  As noted above ACFS and AFD are kernel dependent.  So having the ability to set course for both the Linux Kernel and the ASM related kernel modules is important.  For the scope of this discussion, let’s look at OEL releases 5 through 8 and Oracle Database releases 11gR2 through 19c.  First here is a high level view of OEL and UEK releases.
Support DatesOEL Version
Jun 2007 - Jun 2017OEL5      
Feb 2011 - Mar 2021OEL6      
Jul 2014 - Jul 2024OEL7      
Jul 2019 - Jul 2029OEL8      
UEK YUM ChannelUEKR1UEKR2UEKR3UEKR4UEKR5UEKR6
Major UEK Version2.6.322.6.393.8.134.1.124.14.355.4.17
Released Mar, 2012Oct, 2013Jan, 2016Jun, 2018Mar, 2020

So this is a nice swath of about 10 years of versions.  Please note that the major version number of the kernels does not change if you don’t change UEK channels.  For example, if you move from OEL6 Update 5 to Update 8, but stick with UEKR3 your kernel major number will remain 3.8.13.  The minor numbers beyond the .13 will change, but not the major number.  This is also true for OEL7 and UEKR5 where the major release number is 4.14.35.  Within in this major version, Oracle release new security fixes and patch updates monthly or quarterly for each kernel version that is supported within the support dates.  So for example, OEL6 UEKR2 which was first release in Mar of 2012, just released kernel UEK 2.6.39-400.321.1 in April of 2020.  That is a long history of fixing the same set of code.

ACFS / AFD Linux Kernel Compatibility

Now let's move over to the database support for ACFS / AFD on OEL.  Here is another summary now listing Oracle database versions.
UEK YUM ChannelUEKR1UEKR2UEKR3UEKR4UEKR5UEKR6
Major UEK Version2.6.322.6.393.8.134.1.124.14.355.4.17
DB VersionRelease Date Mar, 2012Oct, 2013Jan, 2016Jun, 2018Mar, 2020
11.2.0.3OEL5 - OEL62.6.32-1002.6.39-100    
11.2.0.4OEL5 - OEL62.6.32-1002.6.39-1003.8.13-1184.1.12-112.16.4  
12.1.0.1OEL5 - OEL62.6.32-1002.6.39-100    
12.1.0.2OEL6 - OEL7 2.6.39-1003.8.13-118   
12.2.0.1OEL6 - OEL7 2.6.39-1003.8.13-1184.1.12-112.16.44.14.35-1902 
18cOEL6 - OEL7 2.6.39-1003.8.13-1184.1.12-112.16.44.14.35-1902TBD
19cOEL7   4.1.12-112.16.44.14.35-1902TBD

Here again I’ve covered about 10 years, and you can see clearly what high-level database versions are compatible with what OEL and UEK versions.  You might also see Oracle’s commitment to long term support for 11.2.0.4.  Similar to the expected support for 19c going until 2024.  Also note that UEKR6 is not yet supported.

Now the information above still high level.  You may assume that every 19c version of Oracle support all UEK4 and UEK5 kernels out of the box.  That is not true.  Database releases and kernel releases are not specifically synchronized.  So perhaps you have tried to test if the ACFS drivers can be installed: 
$ acfsdriverstate supported -v
ACFS-9459: ADVM/ACFS is not supported on this OS version: '4.14.35-1902.10.8.el7uek.x86_64'
ACFS-9201: Not Supported

ACFS-9553: Operating System: linux
ACFS-9554: Machine Architecture: x86_64-linux-thread-multi
ACFS-9555: Operating system name and information: Linux srvr03 4.14.35-1902.10.8.el7uek.x86_64 #2 SMP Thu Feb 6 11:02:28 PST 2020 x86_64 x86_64 x86_64 GNU/Linux
ACFS-9556: Release package: oraclelinux-release-7.7-1.0.5.el7.x86_64
ACFS-9557: Version: ADVM/ACFS is not supported on 4.14.35-1902.10.8.el7uek.x86_64
ACFS-9558: Variable _ORA_USM_NOT_SUPPORTED is defined: no


ASM Required Patches

If your combination of grid software and Linux kernel are not supported, you will see the above message “ACFS-9201: Not Supported”.  How can you fix this?  Oracle provides patches at specific versions of GI to support UEK versions.
DB VersionOS VersionUEK ChannelUEK MinimumBase bug or Base version
11.2.0.3.6OEL5 - OEL6UEKR12.6.32-10015986571
11.2.0.3.7OEL5 - OEL6UEKR12.6.32-10012983005
11.2.0.4OEL5UEKR12.6.32-100Base
11.2.0.4OEL5 - OEL6UEKR22.6.39-100Base
11.2.0.4.4OEL6UEKR33.8.13-11816318126
11.2.0.4.6OEL7UEKR33.8.13-3518321597
11.2.0.4.180717 OEL6 - OEL7UEKR44.1.12-112.16.422810422, 28171094, 27463879
12.1.0.1OEL5 - OEL6UEKR12.6.32-100Base
12.1.0.1OEL5 - OEL6UEKR22.6.39-100Base
12.1.0.2.170718OEL5 - OEL6UEKR22.6.39-100Base
12.1.0.2.170718OEL6UEKR33.8.13-13Base
12.1.0.2.170718OEL7UEKR33.8.13-3518321597
12.1.0.2.181016OEL6 - OEL7UEKR44.1.12-112.16.422810422, 27942938, 28171094, 27942938
12.1.0.2.190716OEL7UEKR54.14.35-190227494830
12.2.0.1OEL6UEKR22.6.39-100Base
12.2.0.1OEL6UEKR33.8.13-13Base
12.2.0.1.180717OEL6UEKR44.1.12-112.16.427463879, 28171094
12.2.0.1OEL7UEKR33.8.13-35Base
12.2.0.1.180717OEL7UEKR44.1.12-112.16.427463879, 28171094, 27463879
12.2.0.181016OEL7UEKR54.14.35-190228069955
18.3.0.0OEL6UEKR22.6.39-100Base
18.3.0.0OEL6UEKR33.8.13-13Base
18.4.0.0.181016OEL6UEKR44.1.12-112.16.428069955
18.3.0.0OEL7UEKR33.8.13-35Base
18.4.0.0.181016OEL7UEKR44.1.12-112.16.427463879, 28171094, 27463879
18.4.0.0.181016OEL7UEKR54.14.35-190227494830
19.3.0.0OEL7UEKR44.1.12-112.16.4Base
19.4.190716OEL7UEKR54.14.35-190227494830

The example above on my test system is a fresh install of GI 19.3.0.0 on a OEL 7.8 system using UEKR5.  Based on the list above we can tell that for UEKR5 support I need either 19.4.190716 or I need patch 27494830.  Oracle provides many patches where the title contains “ACFS RU” that you can search for in the patches section of MOS.  I decided to use this patch “MERGE ON ACFS RU 19.6.0.0.0 OF 28531803 30685278 (Patch 31055785)” which includes the following merge of patches: 
Patch  31055785     : applied on Fri May 22 14:15:34 CDT 2020
Unique Patch ID:  23448796
Patch description:  "ACFS Interim patch for 31055785"
   Created on 20 Mar 2020, 03:53:18 hrs PST8PDT
   Bugs fixed:
     28531803, 30685278, 27494830, 27917085, 28064731, 28293236, 28321248
     28375150, 28553487, 28611527, 28687713, 28701011, 28740425, 28818513
     28825772, 28844788, 28855761, 28860451, 28900212, 28951588, 28960047
     28995524, 29001307, 29030106, 29031452, 29039918, 29115917, 29116692
     29127489, 29127491, 29167352, 29173957, 29198743, 29229120, 29234059
     29250565, 29264772, 29302070, 29313039, 29318169, 29338628, 29339084
     29350729, 29363565, 29379932, 29391373, 29411007, 29417321, 29428155
     29437701, 29482354, 29484738, 29520544, 29524859, 29527221, 29551699
     29560075, 29586338, 29590813, 29604082, 29643818, 29704429, 29705711
     29721120, 29760083, 29779338, 29791186, 29848987, 29851205, 29862693
     29872187, 29893148, 29929003, 29929061, 29937236, 29941227, 29963428
     30003321, 30032562, 30046061, 30051637, 30057972, 30076951, 30093449
     30140896, 30179531, 30239575, 30251503, 30264950, 30269395, 30275174
     30324590, 30363621, 30655657
Which you can see includes the required patch 27494830 (first row, 3rd column). Depending on your version of Oracle, and quarterly bundle patch there are many options of patches available. The quarterly bundle patches for GI generally contain an updated ACFS patch as well.

 

Details of the ASM Kernel Modules 

In 19c, under the GRID_HOME is a directory called usm. On my test system under here I can find this: 
$ ls $GRID_HOME/usm/install/Oracle/EL7UEK/x86_64
4.1.12/  4.1.12-112.16.4/  4.14.35-1902/
This is the location of the libraries used to compile the kernel modules for ACFS / AFD.  As expected, there is a directory for each major kernel version that pretty much matches our list above.  The command “amcmd afd_configure -e” is used to install the AFD kernel modules.  The AFD module use the Kernel Application Binary Interface and are installed under the /lib/modules location on your Linux system.  As they are considered weak modules, they are placed in one location and then symbolically linked into each kernel specific directory when the kernel is updated. 
[root@srvr03 sbin]# ls -l /lib/modules/4.14.35-1902*/{extra,weak-updates}/oracle
/lib/modules/4.14.35-1902.0.9.el7uek.x86_64/extra/oracle:
total 9824
-rw-r--r-- 1 root root 10055682 Jun  5 13:16 oracleafd.ko

/lib/modules/4.14.35-1902.10.8.el7uek.x86_64/weak-updates/oracle:
total 0
lrwxrwxrwx 1 root root 69 Jun  5 13:17 oracleafd.ko -> /lib/modules/4.14.35-1902.0.9.el7uek.x86_64/extra/oracle/oracleafd.ko

/lib/modules/4.14.35-1902.11.3.1.el7uek.x86_64/weak-updates/oracle:
total 0
lrwxrwxrwx 1 root root 69 Jun  5 13:44 oracleafd.ko -> /lib/modules/4.14.35-1902.0.9.el7uek.x86_64/extra/oracle/oracleafd.ko
Here we can see that the primary module was installed under 4.14.35-1902.0.9.  Then when the kernel 4.14.35-1902.11.3.1 was installed, a symbolic link was created to the original module.

The AFD module is normally loaded by a Linux RC script called afd (/etc/init.d/afd).  You can verify it will be started at boot with the following command as the root user: 
[root@srvr03 init.d]# chkconfig --list afd

Note: This output shows SysV services only and does not include native
      systemd services. SysV configuration data might be overridden by native
      systemd configuration.

      If you want to list systemd services use 'systemctl list-unit-files'.
      To see services enabled on particular target use
      'systemctl list-dependencies [target]'.

afd            	0:off	1:off	2:off	3:on	4:off	5:on	6:off

Here you can see that the AFD module will attempt to be loaded at run levels 3 and 5.  You can also check what version of the AFD module you have loaded with the following Linux command: 

[oracle@srvr03 ~]$ modinfo oracleafd
filename:       /lib/modules/4.14.35-1902.11.3.1.el7uek.x86_64/weak-updates/oracle/oracleafd.ko
license:        Oracle Corporation
description:    ASM Filter Driver
author:         Oracle Corporation
srcversion:     533BB7E5866E52F63B9ACCB
depends:        
retpoline:      Y
name:           oracleafd
vermagic:       4.14.35-1902.0.9.el7uek.x86_64 SMP mod_unload modversions 
signat:         PKCS#7
signer:         
sig_key:        
sig_hashalgo:   md4
parm:           oracleafd_use_logical_block_size:Non-zero value implies using logical sectorsize (int)
parm:           oracleafd_use_maxio_softlimits:Non-zero value implies using conservative maximum block io limits (int)

Using the lsmod command to see if it is loaded.  If this command returns no lines, then it is not loaded.  

[oracle@srvr03 ~]$ lsmod |grep oracleafd
oracleafd             229376  0


Conclusion

Ok, so hopefully this has given you a clear picture as to what you need to plan for when using ASM Filter Driver (AFD) or ACFS on your OEL system with UEK.  Once you have a base version of kernel and GI you should have little issues unless you jump UEK channels or do a GI major version change.  As many people are eyeing up 19c upgrades, hopefully the above information is helpful to prepare your Linux OS for the change.  For myself what started as a somewhat simple post about kernel drivers turned into a pretty long delve into the workings of the ASM filter driver.  Please leave your comments and questions.


Posted by at 1 comment:
Labels: ,
Subscribe to: Posts (Atom)