正文

一个docker镜像中的目录删除不了问题

IP云
IP云V管理员/6 阅读
11/30
提示：扫一扫查出行
【扫一扫了解最新限行尾号】
复制提示

在一个容器中，删除一个目录，失败： 

bash-4.2# pwd

/home/zxcdn/ottcache/tomcat

bash-4.2# uname -a

Linux 3516b6c97679 3.10.-327.22..el7.x86_64 # SMP Fri Sep  :: CST  x86_64 x86_64 x86_64 GNU/Linux

bash-4.2# whoami

root

bash-4.2# ls -alrt bin

total

drwxr-xr-x.  root root  Dec   : .

drwxr-xr-x.  root root  Dec   : ..

bash-4.2# rm -rf bin

bash-4.2# ls -i

 bin

bash-4.2# rm -rf bin

bash-4.2# ls -i

 bin

相关docker版本信息： 

[root@host---- caq]# docker info

Containers:

 Running:

 Paused:

 Stopped:

Images:

Server Version: 1.13.

Storage Driver: overlay2----------存储引擎

 Backing Filesystem: extfs--------底层文件系统

 Supports d_type: true

 Native Overlay Diff: false

Logging Driver: journald

Cgroup Driver: systemd

Plugins:

 Volume: local

 Network: bridge host macvlan null overlay

Swarm: inactive

Runtimes: docker-runc runc

Default Runtime: docker-runc

Init Binary: /usr/libexec/docker/docker-init-current

containerd version:  (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)

runc version: 5eda6f6fd0c2884c2c8e78a6e7119e8d0ecedb77 (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)

init version: fec3683b971d9c3ef73f284f176672c44b448662 (expected: 949e6facb77383876aeff8a6944dde66b3089574)

Security Options:

 seccomp

  WARNING: You're not using the default seccomp profile

  Profile: /etc/docker/seccomp.json

Kernel Version: 3.10.-327.22..el7.x86_64

Operating System: Carrier Grade Server Linux

OSType: linux

Architecture: x86_64

Number of Docker Hooks:

CPUs:

Total Memory: 3.703 GiB

Name: host----

ID: 4CV6:Y3Q4:NYGV:PABH:VG42:3CN7:CKET:SEIV:4SYF:63PI:HYAB:AZR2

Docker Root Dir: /var/lib/docker

Debug Mode (client): false

Debug Mode (server): false

Registry: https://index.docker.io/v1/

WARNING: bridge-nf-call-iptables is disabled

WARNING: bridge-nf-call-ip6tables is disabled

Experimental: false

Insecure Registries:

 0.0.0.0/

 127.0.0.0/

Live Restore Enabled: false

Registries: docker.io (secure)

发现删除不了这个空目录，strace跟踪一下，报错如下： 

fcntl(, F_GETFL)                       = 0x38800 (flags O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_NOFOLLOW)

fcntl(, F_SETFD, FD_CLOEXEC)           =

getdents(, /* 2 entries */, )     =

getdents(, /* 0 entries */, )     =

close()                                =

unlinkat(AT_FDCWD, "bin", AT_REMOVEDIR) = - EINVAL (Invalid argument)

lseek(, , SEEK_CUR)                   = - ESPIPE (Illegal seek)

原来是unlinkat报错，然后内核打点跟踪，堆栈如下： 

Returning from:  0xffffffff811ed500 : vfs_rename+0x0/0x790 [kernel]

Returning to  :  0xffffffffa039860b : ovl_do_rename+0x3b/0xa0 [overlay]

 0xffffffffa0398e4e : ovl_clear_empty+0x27e/0x2e0 [overlay]

 0xffffffffa0398f28 : ovl_check_empty_and_clear+0x78/0x90 [overlay]

 0xffffffffa039999c : ovl_do_remove+0x1ec/0x470 [overlay]

 0xffffffffa0399c36 : ovl_rmdir+0x16/0x20 [overlay]

 0xffffffff811ec738 : vfs_rmdir+0xa8/0x100 [kernel]

 0xffffffff811f16d5 : do_rmdir+0x1a5/0x200 [kernel]

 0xffffffff811f28b5 : SyS_unlinkat+0x25/0x40 [kernel]

 0xffffffff81649909 : system_call_fastpath+0x16/0x1b [kernel]

看下确定是vfs_rename出错了，具体按行号打点： 

probe kernel.statement("vfs_rename@namei.c:4122")

{

    p_my=@cast($old_dir,"struct inode")->i_op;

    iflags=@cast($old_dir,"struct inode")->i_flags;

    printf("line 4122 flags=%u,rename2=%x,iflags=%u\r\n",$flags,@cast(p_my,"struct inode_operations_wrapper")->rename2,iflags);

    print_backtrace();

}

对应的内核源码： 

int vfs_rename(struct inode *old_dir, struct dentry *old_dentry,

           struct inode *new_dir, struct dentry *new_dentry,

           struct inode **delegated_inode, unsigned int flags)

{

。。。。

    rename2 = get_rename2_iop(old_dir);---------------4118行

    if (!old_dir->i_op->rename && !rename2)

        return -EPERM;

    if (flags && !rename2)----------------------------4122行

        return -EINVAL;

。。。。

}

一开始我直接取的rename2，发现不为NULL，按道理进不去4122行，后来经细心的谈虎走查，才发现是进入了如下的判断条件： 

static inline const struct inode_operations_wrapper *get_iop_wrapper(struct inode *inode,

                                     unsigned version)

{

    const struct inode_operations_wrapper *wrapper;

    if (!IS_IOPS_WRAPPER(inode))------------最终是这个条件起作用了

        return NULL;

    wrapper = container_of(inode->i_op, const struct inode_operations_wrapper, ops);

    if (wrapper->version < version)

        return NULL;

    return wrapper;

}

static inline iop_rename2_t get_rename2_iop(struct inode *inode)

{

    const struct inode_operations_wrapper *wrapper = get_iop_wrapper(inode, );

    return wrapper ? wrapper->rename2 : NULL;

}

看起来，该内核版本的overlay存储引擎，对ext3的底层文件系统，兼容性存在一些问题。后来使用device-mapper来解决了该问题。

ext4里面，ext4_iget的时候，对目录操作的时候，inode的i_flags是设置了S_IOPS_WRAPPER属性的，
} else if (S_ISDIR(inode->i_mode)) {
inode->i_op = &ext4_dir_inode_operations.ops;
inode->i_fop = &ext4_dir_operations;
inode->i_flags |= S_IOPS_WRAPPER;

但是ext3没有设置。

相关阅读

目录[+]