Fails With ENOENT In Open(2) System call with A Ja_JP.PCK Filename And O_CREAT

Symptoms
In ZFS, open(2) system call with O_CREAT could fail with the error ENOENT as a result of having error in u8_strcmp(3C) despite of the description.


This issue occurs if

1. When ZFS file system is used. The problem will not occur on UFS file system.
2. A file is created with the non-UTF-8 and non-ASCII filename such as the ja_JP.eucJP or ja_JP.PCK string.
3. At the same time, Someone is creating or unlinking a file is created or unlinked under the same directory where action 2 above is work on.


This problem will happen on Solaris 10, 11 and 11.1

The following test case is to duplicate the issue

$ cat open.sjis.c
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>

#include <unistd.h>

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

const char* file1 = "\223\372\226{\214\352\203t\203@\203C\203\213";
const char* file2 = "file";

int
main(void)
{
        int fd;
        const char* file = file1;

        
        while (1) {
                fd = open(file, O_WRONLY|O_CREAT|O_TRUNC, 0666);
                if ( fd < 0 ) {
                        perror("Failed to open(2)");
                        exit(1);
                }
                close(fd);
                unlink(file);
        }
        return(0);
}


$ cc -o open open.sjis.c
"open.sjis.c", line 29: warning: statement not reached
$
$ ./open &
[1] 8632
$
$ ./open
Failed to open(2): No such file or directory
$

Customer would usually find this problem when migrating from UFS to ZFS.



Cause
 Ths is actually the bug 16490464.

According to the u8_number_of_bytes[] table in unicode/u8_textprep.c, a int8_t value ranging from 0x0-0x7f is an ascii char. from 0x80-0xC1 is an illegal char, from 0xC2-0xF4 is a UNICODE standard char and from 0xF5-0xFF, it is out of range. When u8_* specific functions encounter a char in illegal char or out of range char, it sets an error EILSEQ.

However EILSEQ is not probably handled on zfs_dirent_lock() kernel routine during u8_strcmp().

Solution
 Please contact Oracle support to get the fix.