Accessing DBFS Using Oracle Wallet Fails With Input/output error (Doc ID 1358194.1)

After configuring DBFS to use Oracle Wallet, to store the password, via “Configuring DBFS on Oracle Database Machine (Doc ID 1054431.1)” or by following DBFS documentation in Oracle® Database SecureFiles and Large Objects Developer’s Guide 11g Release 2 (11.2), Part Number E18294-01 , the /dbfs directory mounts but when trying to access directory in any way you get “Input/output error”, i.e.

# df -k /dbfs
df: `/dbfs’: Input/output error

# ls -dl /dbfs
ls: /dbfs: Input/output error

Bringing dbfs_mount online, which will mount the file system on all nodes, via “crsctl start resource dbfs_mount” from any cluster node shows no errors.

(oracle)$ crsctl start resource dbfs_mount
CRS-2672: Attempting to start ‘dbfs_mount’ on ‘host05’
CRS-2672: Attempting to start ‘dbfs_mount’ on ‘host06’
CRS-2676: Start of ‘dbfs_mount’ on ‘host06’ succeeded
CRS-2676: Start of ‘dbfs_mount’ on ‘host05’ succeeded

The “crsctl stat res dbfs_mount -t” confirms that dbfs is online

# oracle)$ crsctl stat res dbfs_mount -t
——————————————————————————–
NAME TARGET STATE SERVER STATE_DETAILS
——————————————————————————–
Local Resources
——————————————————————————–
dbfs_mount
ONLINE ONLINE host05
ONLINE ONLINE host06

By setting trace parameter as part of the mount command we find that accessing /dbfs directory results in “ORA-01017: invalid username/password; logon denied”.

To turn on tracing, modify /etc/fstab to have following mount command and remount /dbfs via “crsctl stop/start resource dbfs_mount” command

/sbin/mount.dbfs#/@dbfs_mount.local /dbfs fuse rw,user,direct_io,allow_other,wallet,trace_file=/tmp/dbfs_mount.trc,trace_level=1,trace_size=0,noauto 0 0

After running “df -k /dbfs” command and getting error you will see following in /tmp/dbfs_mount.trc

[5bbf7940 09/09/11 11:24:42.818800 LcdfLock.cpp:230 ] DEBUG: Global Lock – unlock
[5bbf7940 09/09/11 11:24:42.818806 LcdfLock.cpp:242 ] DEBUG: Global Lock (R) rdwr:0
[45bec940 09/09/11 11:24:42.857258 LcdfDBPool.cpp:398 ] ERROR: ERROR 1017 – ORA-01017: invalid username/password; logon denied

[45bec940 09/09/11 11:24:42.857287 LcdfJob.cpp:304 ] DEBUG: Exit LcdfJob::DoJobInt job:0x2aaaac0bc0e0 op(1)