In the world of Oracle databases, DBA’s and system administrators don’t always
see eye to eye. Mostly I have found that this is really just about
coming at challenges from different angles. Some of the major changes in
Oracle since 10g version has been the introduction of Oracle Automatic Storage
Management or ASM. This has led to many features such as ASMLIB, ASM
Clustered Filesystem (ACFS), and now the ASM Filter Driver (AFD or
ASMFD). These features require OS specific compatible libraries, and
kernel modules. Information on compatibility can be confusing.
Background References
Let’s start with some background. For this blog, I’m going to focus on
Oracle Enterprise Linux (OEL). The information here could be assembled
for any supported OS. This information is from a few sources, the main
ones being:
Before making any major patching or upgrade decisions please reference these
notes and Oracle support in general.
Oracle does a pretty good job of providing information on Linux and Oracle
database / ASM compatibility. Though, the notes are not always clear as
to what the goals is. For example, I have heard many people say that ASM
has stopped working during a Linux upgrade. Or that ASM drivers are not
compatible with Linux changes, or that you can’t patch Linux without breaking
ASM / AFD / ACFS. Well, these are antidotal statements, and the truth is
a little more complex.
Linux Kernel Version
Oracle Enterprise Linux (OEL) recommends using the Unbreakable Enterprise
Kernel or UEK. The main reason behind this is that Oracle controls the
destiny of the UEK. As noted above ACFS and AFD are kernel
dependent. So having the ability to set course for both the Linux
Kernel and the ASM related kernel modules is important. For the scope
of this discussion, let’s look at OEL releases 5 through 8 and Oracle
Database releases 11gR2 through 19c. First here is a high level view
of OEL and UEK releases.
| Support Dates | OEL Version | | | | | | |
| Jun 2007 - Jun 2017 | OEL5 | | | | | | |
| Feb 2011 - Mar 2021 | OEL6 | | | | | | |
| Jul 2014 - Jul 2024 | OEL7 | | | | | | |
| Jul 2019 - Jul 2029 | OEL8 | | | | | | |
| UEK YUM Channel | UEKR1 | UEKR2 | UEKR3 | UEKR4 | UEKR5 | UEKR6 |
| Major UEK Version | 2.6.32 | 2.6.39 | 3.8.13 | 4.1.12 | 4.14.35 | 5.4.17 |
| Released | | Mar, 2012 | Oct, 2013 | Jan, 2016 | Jun, 2018 | Mar, 2020 |
So this is a nice swath of about 10 years of versions. Please note
that the major version number of the kernels does not change if you don’t
change UEK channels. For example, if you move from OEL6 Update 5 to
Update 8, but stick with UEKR3 your kernel major number will remain
3.8.13. The minor numbers beyond the .13 will change, but not the
major number. This is also true for OEL7 and UEKR5 where the major
release number is 4.14.35. Within in this major version, Oracle
release new security fixes and patch updates monthly or quarterly for each
kernel version that is supported within the support dates. So for
example, OEL6 UEKR2 which was first release in Mar of 2012, just released
kernel UEK 2.6.39-400.321.1 in April of 2020. That is a long history
of fixing the same set of code.
ACFS / AFD Linux Kernel Compatibility
Now let's move over to the database support for ACFS / AFD on OEL.
Here is another summary now listing Oracle database versions.
| UEK YUM Channel | UEKR1 | UEKR2 | UEKR3 | UEKR4 | UEKR5 | UEKR6 |
| Major UEK Version | 2.6.32 | 2.6.39 | 3.8.13 | 4.1.12 | 4.14.35 | 5.4.17 |
| DB Version | Release Date | | Mar, 2012 | Oct, 2013 | Jan, 2016 | Jun, 2018 | Mar, 2020 |
| 11.2.0.3 | OEL5 - OEL6 | 2.6.32-100 | 2.6.39-100 | | | | |
| 11.2.0.4 | OEL5 - OEL6 | 2.6.32-100 | 2.6.39-100 | 3.8.13-118 | 4.1.12-112.16.4 | | |
| 12.1.0.1 | OEL5 - OEL6 | 2.6.32-100 | 2.6.39-100 | | | | |
| 12.1.0.2 | OEL6 - OEL7 | | 2.6.39-100 | 3.8.13-118 | | | |
| 12.2.0.1 | OEL6 - OEL7 | | 2.6.39-100 | 3.8.13-118 | 4.1.12-112.16.4 | 4.14.35-1902 | |
| 18c | OEL6 - OEL7 | | 2.6.39-100 | 3.8.13-118 | 4.1.12-112.16.4 | 4.14.35-1902 | TBD |
| 19c | OEL7 | | | | 4.1.12-112.16.4 | 4.14.35-1902 | TBD |
Here again I’ve covered about 10 years, and you can see clearly what
high-level database versions are compatible with what OEL and UEK
versions. You might also see Oracle’s commitment to long term support
for 11.2.0.4. Similar to the expected support for 19c going until
2024. Also note that UEKR6 is not yet supported.
Now the information above still high level. You may assume that every
19c version of Oracle support all UEK4 and UEK5 kernels out of the
box. That is not true. Database releases and kernel releases are
not specifically synchronized. So perhaps you have tried to test if
the ACFS drivers can be installed:
$ acfsdriverstate supported -v
ACFS-9459: ADVM/ACFS is not supported on this OS version: '4.14.35-1902.10.8.el7uek.x86_64'
ACFS-9201: Not Supported
ACFS-9553: Operating System: linux
ACFS-9554: Machine Architecture: x86_64-linux-thread-multi
ACFS-9555: Operating system name and information: Linux srvr03 4.14.35-1902.10.8.el7uek.x86_64 #2 SMP Thu Feb 6 11:02:28 PST 2020 x86_64 x86_64 x86_64 GNU/Linux
ACFS-9556: Release package: oraclelinux-release-7.7-1.0.5.el7.x86_64
ACFS-9557: Version: ADVM/ACFS is not supported on 4.14.35-1902.10.8.el7uek.x86_64
ACFS-9558: Variable _ORA_USM_NOT_SUPPORTED is defined: no
ASM Required Patches
If your combination of grid software and Linux kernel are not supported, you
will see the above message “ACFS-9201: Not Supported”. How can you fix
this? Oracle provides patches at specific versions of GI to support
UEK versions.
| DB Version | OS Version | UEK Channel | UEK Minimum | Base bug or Base version |
| 11.2.0.3.6 | OEL5 - OEL6 | UEKR1 | 2.6.32-100 | 15986571 |
| 11.2.0.3.7 | OEL5 - OEL6 | UEKR1 | 2.6.32-100 | 12983005 |
| 11.2.0.4 | OEL5 | UEKR1 | 2.6.32-100 | Base |
| 11.2.0.4 | OEL5 - OEL6 | UEKR2 | 2.6.39-100 | Base |
| 11.2.0.4.4 | OEL6 | UEKR3 | 3.8.13-118 | 16318126 |
| 11.2.0.4.6 | OEL7 | UEKR3 | 3.8.13-35 | 18321597 |
| 11.2.0.4.180717 | OEL6 - OEL7 | UEKR4 | 4.1.12-112.16.4 | 22810422, 28171094, 27463879 |
| 12.1.0.1 | OEL5 - OEL6 | UEKR1 | 2.6.32-100 | Base |
| 12.1.0.1 | OEL5 - OEL6 | UEKR2 | 2.6.39-100 | Base |
| 12.1.0.2.170718 | OEL5 - OEL6 | UEKR2 | 2.6.39-100 | Base |
| 12.1.0.2.170718 | OEL6 | UEKR3 | 3.8.13-13 | Base |
| 12.1.0.2.170718 | OEL7 | UEKR3 | 3.8.13-35 | 18321597 |
| 12.1.0.2.181016 | OEL6 - OEL7 | UEKR4 | 4.1.12-112.16.4 | 22810422, 27942938, 28171094, 27942938 |
| 12.1.0.2.190716 | OEL7 | UEKR5 | 4.14.35-1902 | 27494830 |
| 12.2.0.1 | OEL6 | UEKR2 | 2.6.39-100 | Base |
| 12.2.0.1 | OEL6 | UEKR3 | 3.8.13-13 | Base |
| 12.2.0.1.180717 | OEL6 | UEKR4 | 4.1.12-112.16.4 | 27463879, 28171094 |
| 12.2.0.1 | OEL7 | UEKR3 | 3.8.13-35 | Base |
| 12.2.0.1.180717 | OEL7 | UEKR4 | 4.1.12-112.16.4 | 27463879, 28171094, 27463879 |
| 12.2.0.181016 | OEL7 | UEKR5 | 4.14.35-1902 | 28069955 |
| 18.3.0.0 | OEL6 | UEKR2 | 2.6.39-100 | Base |
| 18.3.0.0 | OEL6 | UEKR3 | 3.8.13-13 | Base |
| 18.4.0.0.181016 | OEL6 | UEKR4 | 4.1.12-112.16.4 | 28069955 |
| 18.3.0.0 | OEL7 | UEKR3 | 3.8.13-35 | Base |
| 18.4.0.0.181016 | OEL7 | UEKR4 | 4.1.12-112.16.4 | 27463879, 28171094, 27463879 |
| 18.4.0.0.181016 | OEL7 | UEKR5 | 4.14.35-1902 | 27494830 |
| 19.3.0.0 | OEL7 | UEKR4 | 4.1.12-112.16.4 | Base |
| 19.4.190716 | OEL7 | UEKR5 | 4.14.35-1902 | 27494830 |
The example above on my test system is a fresh install of GI 19.3.0.0 on a
OEL 7.8 system using UEKR5. Based on the list above we can tell that
for UEKR5 support I need either 19.4.190716 or I need patch 27494830.
Oracle provides many patches where the title contains “ACFS RU” that you can
search for in the patches section of MOS. I decided to use this patch
“MERGE ON ACFS RU 19.6.0.0.0 OF 28531803 30685278 (Patch 31055785)” which
includes the following merge of patches:
Patch 31055785 : applied on Fri May 22 14:15:34 CDT 2020
Unique Patch ID: 23448796
Patch description: "ACFS Interim patch for 31055785"
Created on 20 Mar 2020, 03:53:18 hrs PST8PDT
Bugs fixed:
28531803, 30685278, 27494830, 27917085, 28064731, 28293236, 28321248
28375150, 28553487, 28611527, 28687713, 28701011, 28740425, 28818513
28825772, 28844788, 28855761, 28860451, 28900212, 28951588, 28960047
28995524, 29001307, 29030106, 29031452, 29039918, 29115917, 29116692
29127489, 29127491, 29167352, 29173957, 29198743, 29229120, 29234059
29250565, 29264772, 29302070, 29313039, 29318169, 29338628, 29339084
29350729, 29363565, 29379932, 29391373, 29411007, 29417321, 29428155
29437701, 29482354, 29484738, 29520544, 29524859, 29527221, 29551699
29560075, 29586338, 29590813, 29604082, 29643818, 29704429, 29705711
29721120, 29760083, 29779338, 29791186, 29848987, 29851205, 29862693
29872187, 29893148, 29929003, 29929061, 29937236, 29941227, 29963428
30003321, 30032562, 30046061, 30051637, 30057972, 30076951, 30093449
30140896, 30179531, 30239575, 30251503, 30264950, 30269395, 30275174
30324590, 30363621, 30655657
Which you can see includes the required patch 27494830 (first row, 3rd column). Depending on your
version of Oracle, and quarterly bundle patch there are many options of patches
available. The quarterly bundle patches for GI generally contain an updated ACFS
patch as well.
Details of the ASM Kernel Modules
In 19c, under the GRID_HOME is a directory called usm. On my test system under
here I can find this:
$ ls $GRID_HOME/usm/install/Oracle/EL7UEK/x86_64
4.1.12/ 4.1.12-112.16.4/ 4.14.35-1902/
This is the location of the libraries used to compile the kernel modules
for ACFS / AFD. As expected, there is a directory for each major
kernel version that pretty much matches our list above. The command
“amcmd afd_configure -e” is used to install the AFD kernel
modules. The AFD module use the Kernel Application Binary Interface
and are installed under the /lib/modules location on your
Linux system. As they are considered weak modules, they are placed
in one location and then symbolically linked into each kernel specific
directory when the kernel is updated.
[root@srvr03 sbin]# ls -l /lib/modules/4.14.35-1902*/{extra,weak-updates}/oracle
/lib/modules/4.14.35-1902.0.9.el7uek.x86_64/extra/oracle:
total 9824
-rw-r--r-- 1 root root 10055682 Jun 5 13:16 oracleafd.ko
/lib/modules/4.14.35-1902.10.8.el7uek.x86_64/weak-updates/oracle:
total 0
lrwxrwxrwx 1 root root 69 Jun 5 13:17 oracleafd.ko -> /lib/modules/4.14.35-1902.0.9.el7uek.x86_64/extra/oracle/oracleafd.ko
/lib/modules/4.14.35-1902.11.3.1.el7uek.x86_64/weak-updates/oracle:
total 0
lrwxrwxrwx 1 root root 69 Jun 5 13:44 oracleafd.ko -> /lib/modules/4.14.35-1902.0.9.el7uek.x86_64/extra/oracle/oracleafd.ko
Here we can see that the primary module was installed under
4.14.35-1902.0.9. Then when the kernel 4.14.35-1902.11.3.1 was
installed, a symbolic link was created to the original module.
The AFD module is normally loaded by a Linux RC script called afd
(/etc/init.d/afd). You can verify it will be started at boot with
the following command as the root user:
[root@srvr03 init.d]# chkconfig --list afd
Note: This output shows SysV services only and does not include native
systemd services. SysV configuration data might be overridden by native
systemd configuration.
If you want to list systemd services use 'systemctl list-unit-files'.
To see services enabled on particular target use
'systemctl list-dependencies [target]'.
afd 0:off 1:off 2:off 3:on 4:off 5:on 6:off
Here you can see that the AFD module will attempt to be loaded at run levels 3 and 5. You can also check what version of the AFD module you have loaded with the
following Linux command:
[oracle@srvr03 ~]$ modinfo oracleafd
filename: /lib/modules/4.14.35-1902.11.3.1.el7uek.x86_64/weak-updates/oracle/oracleafd.ko
license: Oracle Corporation
description: ASM Filter Driver
author: Oracle Corporation
srcversion: 533BB7E5866E52F63B9ACCB
depends:
retpoline: Y
name: oracleafd
vermagic: 4.14.35-1902.0.9.el7uek.x86_64 SMP mod_unload modversions
signat: PKCS#7
signer:
sig_key:
sig_hashalgo: md4
parm: oracleafd_use_logical_block_size:Non-zero value implies using logical sectorsize (int)
parm: oracleafd_use_maxio_softlimits:Non-zero value implies using conservative maximum block io limits (int)
Using the lsmod command to see if it is loaded. If this
command returns no lines, then it is not loaded.
[oracle@srvr03 ~]$ lsmod |grep oracleafd
oracleafd 229376 0
Conclusion
Ok, so hopefully this has given you a clear picture as to what you need to
plan for when using ASM Filter Driver (AFD) or ACFS on your OEL system
with UEK. Once you have a base version of kernel and GI you should
have little issues unless you jump UEK channels or do a GI major version
change. As many people are eyeing up 19c upgrades, hopefully the
above information is helpful to prepare your Linux OS for the
change. For myself what started as a somewhat simple post about
kernel drivers turned into a pretty long delve into the workings of the
ASM filter driver. Please leave your comments and questions.