Open control FD in gofer server (and directfs gofer client) with O_PATH.

[ayushr2]

Description

          > mountpoint_s3::fuse: open failed: inode error: inode 3 (full key "synmon/test_file.txt") is not writable while being read

The issue is that gVisor's gofer client (pkg/fsimpl/gofer/) opens the file read-only first and then opens it for writing, but mountpoint_s3::fuse refuses to support that. The gofer client first caches the dentry for the file being opened. It holds a "control FD" for all the dentries it caches. This control FD is opened with O_RDONLY|O_NONBLOCK:

gvisor/pkg/sentry/fsimpl/gofer/directfs_dentry.go

Lines 41 to 50 in f67e10c

	// tryOpen tries to open() with different modes in the following order:
	// 1. RDONLY | NONBLOCK: for all files, directories, ro mounts, FIFOs.
	// Use non-blocking to prevent getting stuck inside open(2) for
	// FIFOs. This option has no effect on regular files.
	// 2. PATH: for symlinks, sockets.
	func tryOpen(open func(int) (int, error)) (int, error) {
	flags := []int{
	unix.O_RDONLY | unix.O_NONBLOCK,
	unix.O_PATH,
	}

I think we should change that logic to open regular files with O_PATH only. Directories and FIFOs can continue to be opened with O_RDONLY.

Originally posted by @ayushr2 in awslabs/mountpoint-s3#862 (comment)

[ayushr2]

This work may be more involved. The control FD is used to perform various operations on a file (see usages). Some of those operations will fail with EBADF if the control FD is a O_PATH FD. For instance: fchmod(2), fchown(2) and fgetxattr(2). See Linux kernel source code for these syscalls; they use fdget(fd) => __fget_light(fd, FMODE_PATH) which returns NULL for O_PATH FDs. So we will need to open all files with O_PATH and upgrade to a readable FD when needed OR use f*at(2) syscall variant with AT_EMPTY_PATH (which effectively bypasses this issue).

Some questions:

• How is fchmod(2) working for symlinks?
  

  gvisor/pkg/sentry/fsimpl/gofer/directfs_dentry.go

  Lines 268 to 271 in 6e61813

  	func (d *directfsDentry) chmod(ctx context.Context, mode uint16) error {
  	if !d.isSocket() {
  	return unix.Fchmod(d.controlFD, uint32(mode))
  	}

• How is fgetxattr(2) working for symlinks and sockets?
  

  gvisor/pkg/sentry/fsimpl/gofer/directfs_dentry.go

  Lines 426 to 430 in 6e61813

  	func (d *directfsDentry) getXattr(name string, size uint64) (string, error) {
  	data := make([]byte, size)
  	if _, err := unix.Fgetxattr(d.controlFD, name, data); err != nil {
  	return "", err
  	}

• Can we open FIFOs with O_PATH?