Error response from daemon readlink var lib docker overlay2 invalid argument

I saw this error today on a dind 17.11.0-ce on Linuxkit, on XFS. I wiped /var/lib/docker and the issue went away. Not sure the cause offhand.

Description
bash-4.3# docker logs -f kontena-agent
Error response from daemon: readlink /var/lib/docker/overlay2/l: invalid argument
Steps to reproduce the issue:

have overlay2 and extfs4 mounted in /var/lib/docker

run containers

reboot the machine by power cycling it hard enough

Output of docker version:
Docker version 17.09.0-ce, build afdb6d4
Output of docker info:
Docker version 17.09.0-ce, build afdb6d4
bash-4.3# docker info
Containers: 9
 Running: 2
 Paused: 0
 Stopped: 7
Images: 10
Server Version: 17.09.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: v0.13.2 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  Profile: default
 selinux
Kernel Version: 4.13.16-coreos-r2
Operating System: Container Linux by CoreOS 1576.4.0 (Ladybug)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.863GiB
Name: pannu
ID: ZJ6X:DLDI:4MHS:3DIG:WQUY:SFBI:SCUC:7URO:CYDC:4DR6:UI56:XGCJ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
 region=virtualbox
 ephemeral=yes
Experimental: false
Insecure Registries:
 10.81.0.0/16
 127.0.0.0/8
Live Restore Enabled: false
$ cat /etc/lsb-release
DISTRIB_ID="Container Linux by CoreOS"
DISTRIB_RELEASE=1576.4.0
DISTRIB_CODENAME="Ladybug"
DISTRIB_DESCRIPTION="Container Linux by CoreOS 1576.4.0 (Ladybug)"

$ uname
Linux pannu 4.13.16-coreos-r2 #1 SMP Wed Dec 6 04:27:34 UTC 2017 x86_64 Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz GenuineIntel GNU/Linux
Additional environment details (AWS, VirtualBox, physical, etc.):

Virtualbox

I have the same problem on my virtualbox too, Have you find a solution to fix it ?

My error log like this:
«ImageInspectError: Failed to inspect image «grafana/grafana:5.0.0″: rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2/l: invalid argument»

I can not figure out why other containers inspect image succeed and grafana failed…

I am having the same error

bash -c ‘docker-compose -f docker-compose.yml -f docker-compose-host.yml up -d’
ERROR: readlink /var/lib/docker/overlay2/l: invalid argument
Makefile:75: recipe for target ‘dev.up’ failed
make: *** [dev.up] Error 1

Please let me know how to solve this issue

Can you stat the file on the docker host?
Does it exist? Is it a symlink?

@cpuguy83 I seem to be getting the same symptoms and can answer your questions.

root@lumber:~# uname -a
Linux lumber 4.15.0-29-generic #31~16.04.1-Ubuntu SMP Wed Jul 18 08:54:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
root@lumber:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.3 LTS
Release:	16.04
Codename:	xenial
root@lumber:~# docker version
Client:
 Version:           18.09.6
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        481bc77
 Built:             Sat May  4 02:35:27 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.4
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       d14af54
  Built:            Wed Mar 27 18:01:48 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Some selected lines from journalctl:

May 21 14:34:21 lumber kernel: overlayfs: '/var/lib/docker/overlay2/l/BSZ2R3I2SJ63GVCGDYRFGUHFFI' not a directory
May 21 14:34:21 lumber dockerd[1612]: time="2019-05-21T14:34:21.250483116Z" level=error msg="error unmounting /var/lib/docker/overlay2/73c03c8d94a01711bc68fb667cb21e7f8a13e303a28817898304c948ec6d6593/merged: invalid argument" storage-driver=overlay2
May 21 14:34:21 lumber kernel: overlayfs: failed to resolve '/var/lib/docker/overlay2/l/YPK2TM6LOUZ7GRSNYI4LKZAHRO': -2
May 21 14:34:21 lumber dockerd[1612]: time="2019-05-21T14:34:21.258672271Z" level=error msg="error unmounting /var/lib/docker/overlay2/e9712d9c6efc3f7c278996155a3565f78c6c72042a85262bfba07439e9511d3b/merged: invalid argument" storage-driver=overlay2
May 21 14:34:36 lumber dockerd[1612]: time="2019-05-21T14:34:36.043373031Z" level=error msg="Handler for GET /v1.22/containers/06c1d8bbebdf2ecfdf33244658c5825f29ca34dac3efa4b18eb51bd520f2f424/json returned error: readlink /var/lib/docker/overlay2/l/BSZ2R3I2SJ63GVCGDYRFGUHFFI: invalid argument"
May 21 14:34:36 lumber docker-compose[4119]: readlink /var/lib/docker/overlay2/l/BSZ2R3I2SJ63GVCGDYRFGUHFFI: invalid argument

To answer your latest questions:

root@lumber:~# stat /var/lib/docker/overlay2/l/BSZ2R3I2SJ63GVCGDYRFGUHFFI
  File: '/var/lib/docker/overlay2/l/BSZ2R3I2SJ63GVCGDYRFGUHFFI'
  Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
Device: fd00h/64768d	Inode: 2895012     Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-05-21 14:36:45.743698369 +0000
Modify: 2019-04-04 20:42:03.112247693 +0000
Change: 2019-04-04 20:42:03.112247693 +0000
 Birth: -
root@lumber:~# readlink -v /var/lib/docker/overlay2/l/BSZ2R3I2SJ63GVCGDYRFGUHFFI
readlink: /var/lib/docker/overlay2/l/BSZ2R3I2SJ63GVCGDYRFGUHFFI: Invalid argument

BSZ2R3I2SJ63GVCGDYRFGUHFFI is the only item in /var/lib/docker/overlay2/l that isn’t a symlink.

I’ve taken this server out of production and I have it for a couple of days so I can run further commands to investigate. Unfortunately after that I’ll have to wipe it and reinstall.

I’ve got the same? Similar? Problem. For me it’s doing readlink on /var/lib/docker/overlay2 though which doesn’t make sense to me at all:

error msg=»Handler for GET /v1.31/images/xx.dkr.ecr.us-east-1.amazonaws.com/yy/zzz:359fdb695fda85fa92606bf502f38d59e1589bbb/json returned error: readlink /var/lib/docker/overlay2: invalid argument»

Which yes, isn’t a symlink:

readlink -v /var/lib/docker/overlay2
readlink: /var/lib/docker/overlay2: Invalid argument

@discordianfish I see it’s calling API v1.31 (which corresponds with Docker 17.07); what version are you running? I think there have been some fixes in this area, so wondering if you’re running into an issue that was fixed

I’m running coreos 1967.3.0 which comes with 18.06.1-ce. The client in that snippet is kubernetes, hence the older endpoint.

@euank any ideas? (oh! looks from your profile that you’re no longer at CoreOS; do you know who else I should ping?)

Skimming the overlay2 code, there’s a direct call to readlink when it’s looking for lower directories

The errors in this issue could happen if the lower file were corrupt, or if the write to it were short. That file being truncated/not fully written would lead to the errors described here I think.

@discordianfish If you check the contents of the lower files in /var/lib/docker/overlay2/$id/lower, I think you might find one that ends with a : character rather than with a valid path. Running head -n9999 /var/lib/docker/overlay2/*/lower as root should display each «lower» file and its contents. Can you either paste that output somewhere or verify that one of them is erroneous?

Since docker isn’t doing an atomic write+move operation to update those files, there’s numerous ways in which they could end up truncated in such a way. If these errors are the result of corrupt lower files, switching to an atomic update of those files should resolve the issue.

@thaJeztah No worries about the ping; I’m still happy to hack on container-stuff regardless. For CoreOS-related issues, @bgilbert should be able to help with them.

Thanks @euank — much appreciated 👍

lower file were corrupt,

It looks once pulling huge image (>5GB), lower file corrupt happen frequently, @euank what do you mean to «atomic update of those files»?

So… what is one supposed to do when this happens?

Simply clear out all contents of /var/lib/docker/overlay2? Wouldn’t that risk data loss?

So… what is one supposed to do when this happens?

Simply clear out all contents of /var/lib/docker/overlay2? Wouldn’t that risk data loss?

Exactly it’s data loss case. So the acceptable solution should preserve data.

So you’re saying there is no acceptable solution at the moment?

any reason of this problem?

In my case running docker:19.03.5-dind with --volume=${PWD}/.dockercache:/var/lib/docker produces an error, but using named volume --volume=named:/var/lib/docker works.

Given:

HOST MACHINE WINDOWS:

Client:
 Debug Mode: false

Server:
 Containers: 8
  Running: 1
  Paused: 0
  Stopped: 7
 Images: 366
 Server Version: 19.03.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.9.184-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 1.95GiB
 Name: docker-desktop
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 38
  Goroutines: 58
  System Time: 2020-01-25T17:14:58.1639972Z
  EventsListeners: 1
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

CONTAINER MACHINE:

Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 19.03.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.9.184-linuxkit
 Operating System: Alpine Linux v3.11 (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 1.95GiB
 Name: 97331e08f173
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Given a folder:

Dockerfile
config.cnf
docker-entrypoint-initdb.d

FROM mysql:8.0.18
COPY config.cnf /etc/mysql/conf.d/config.cnf
COPY docker-entrypoint-initdb.d docker-entrypoint-initdb.d

WORKING

docker run --rm -it --privileged docker:19.03.5-dind

docker build . (in container)

Status: Downloaded newer image for mysql:8.0.18
 ---> ed1ffcb5eff3
Step 2/3 : COPY config.cnf /etc/mysql/conf.d/config.cnf
 ---> 4af3aa7c9f35
Step 3/3 : COPY docker-entrypoint-initdb.d docker-entrypoint-initdb.d
 ---> 73868723b6cb
Successfully built 73868723b6cb

NOT WORKING

docker run --rm -it --privileged --volume=${PWD}/.dockercache:/var/lib/docker docker:19.03.5-dind

docker build . (in container)

Status: Downloaded newer image for mysql:8.0.18
 ---> ed1ffcb5eff3
Step 2/3 : COPY config.cnf /etc/mysql/conf.d/config.cnf
ERRO[2020-01-25T17:33:16.305955600Z] error unmounting /var/lib/docker/overlay2/fcdafa541cec1a73a49adb85f313e1b4382b19c8bc80a8b377c0ec12a6de9ad7/merged: invalid argument  storage-driver=overlay2
error creating overlay mount to /var/lib/docker/overlay2/fcdafa541cec1a73a49adb85f313e1b4382b19c8bc80a8b377c0ec12a6de9ad7/merged: invalid argument

WORKING DEAMON LOG:

INFO[2020-01-25T17:29:45.091557200Z] Starting up
WARN[2020-01-25T17:29:45.093101500Z] could not change group /var/run/docker.sock to docker: group docker not found
INFO[2020-01-25T17:29:45.094358100Z] libcontainerd: started new containerd process  pid=40
INFO[2020-01-25T17:29:45.094468700Z] parsed scheme: "unix"                         module=grpc
INFO[2020-01-25T17:29:45.094520700Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2020-01-25T17:29:45.094597800Z] ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}] <nil>}  module=grpc
INFO[2020-01-25T17:29:45.094658200Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2020-01-25T17:29:45.115215700Z] starting containerd                           revision=b34a5c8af56e510852c35414db4c1f4fa6172339 version=v1.2.10
INFO[2020-01-25T17:29:45.115624300Z] loading plugin "io.containerd.content.v1.content"...  type=io.containerd.content.v1
INFO[2020-01-25T17:29:45.115772800Z] loading plugin "io.containerd.snapshotter.v1.btrfs"...  type=io.containerd.snapshotter.v1
WARN[2020-01-25T17:29:45.116059700Z] failed to load plugin io.containerd.snapshotter.v1.btrfs  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
INFO[2020-01-25T17:29:45.116103200Z] loading plugin "io.containerd.snapshotter.v1.aufs"...  type=io.containerd.snapshotter.v1
WARN[2020-01-25T17:29:45.120927100Z] failed to load plugin io.containerd.snapshotter.v1.aufs  error="modprobe aufs failed: "ip: can't find device 'aufs'nmodprobe: can't change directory to '/lib/modules': No such file or directoryn": exit status 1"
INFO[2020-01-25T17:29:45.120981100Z] loading plugin "io.containerd.snapshotter.v1.native"...  type=io.containerd.snapshotter.v1
INFO[2020-01-25T17:29:45.121127500Z] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  type=io.containerd.snapshotter.v1
INFO[2020-01-25T17:29:45.121285100Z] loading plugin "io.containerd.snapshotter.v1.zfs"...  type=io.containerd.snapshotter.v1
INFO[2020-01-25T17:29:45.121571000Z] skip loading plugin "io.containerd.snapshotter.v1.zfs"...  type=io.containerd.snapshotter.v1
INFO[2020-01-25T17:29:45.121615500Z] loading plugin "io.containerd.metadata.v1.bolt"...  type=io.containerd.metadata.v1
WARN[2020-01-25T17:29:45.121660700Z] could not use snapshotter aufs in metadata plugin  error="modprobe aufs failed: "ip: can't find device 'aufs'nmodprobe: can't change directory to '/lib/modules': No such file or directoryn": exit status 1"
WARN[2020-01-25T17:29:45.121671800Z] could not use snapshotter zfs in metadata plugin  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin"
WARN[2020-01-25T17:29:45.121681900Z] could not use snapshotter btrfs in metadata plugin  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
INFO[2020-01-25T17:29:45.134158500Z] loading plugin "io.containerd.differ.v1.walking"...  type=io.containerd.differ.v1
INFO[2020-01-25T17:29:45.134223300Z] loading plugin "io.containerd.gc.v1.scheduler"...  type=io.containerd.gc.v1
INFO[2020-01-25T17:29:45.134315700Z] loading plugin "io.containerd.service.v1.containers-service"...  type=io.containerd.service.v1
INFO[2020-01-25T17:29:45.134360800Z] loading plugin "io.containerd.service.v1.content-service"...  type=io.containerd.service.v1
INFO[2020-01-25T17:29:45.134388100Z] loading plugin "io.containerd.service.v1.diff-service"...  type=io.containerd.service.v1
INFO[2020-01-25T17:29:45.134448300Z] loading plugin "io.containerd.service.v1.images-service"...  type=io.containerd.service.v1
INFO[2020-01-25T17:29:45.134506900Z] loading plugin "io.containerd.service.v1.leases-service"...  type=io.containerd.service.v1
INFO[2020-01-25T17:29:45.134560600Z] loading plugin "io.containerd.service.v1.namespaces-service"...  type=io.containerd.service.v1
INFO[2020-01-25T17:29:45.134627000Z] loading plugin "io.containerd.service.v1.snapshots-service"...  type=io.containerd.service.v1
INFO[2020-01-25T17:29:45.134687000Z] loading plugin "io.containerd.runtime.v1.linux"...  type=io.containerd.runtime.v1
INFO[2020-01-25T17:29:45.135010400Z] loading plugin "io.containerd.runtime.v2.task"...  type=io.containerd.runtime.v2
INFO[2020-01-25T17:29:45.135214300Z] loading plugin "io.containerd.monitor.v1.cgroups"...  type=io.containerd.monitor.v1
INFO[2020-01-25T17:29:45.135723400Z] loading plugin "io.containerd.service.v1.tasks-service"...  type=io.containerd.service.v1
INFO[2020-01-25T17:29:45.135806100Z] loading plugin "io.containerd.internal.v1.restart"...  type=io.containerd.internal.v1
INFO[2020-01-25T17:29:45.135932400Z] loading plugin "io.containerd.grpc.v1.containers"...  type=io.containerd.grpc.v1
INFO[2020-01-25T17:29:45.135988700Z] loading plugin "io.containerd.grpc.v1.content"...  type=io.containerd.grpc.v1
INFO[2020-01-25T17:29:45.136006300Z] loading plugin "io.containerd.grpc.v1.diff"...  type=io.containerd.grpc.v1
INFO[2020-01-25T17:29:45.136067800Z] loading plugin "io.containerd.grpc.v1.events"...  type=io.containerd.grpc.v1
INFO[2020-01-25T17:29:45.136131600Z] loading plugin "io.containerd.grpc.v1.healthcheck"...  type=io.containerd.grpc.v1
INFO[2020-01-25T17:29:45.136185800Z] loading plugin "io.containerd.grpc.v1.images"...  type=io.containerd.grpc.v1
INFO[2020-01-25T17:29:45.136222700Z] loading plugin "io.containerd.grpc.v1.leases"...  type=io.containerd.grpc.v1
INFO[2020-01-25T17:29:45.136238400Z] loading plugin "io.containerd.grpc.v1.namespaces"...  type=io.containerd.grpc.v1
INFO[2020-01-25T17:29:45.136253100Z] loading plugin "io.containerd.internal.v1.opt"...  type=io.containerd.internal.v1
INFO[2020-01-25T17:29:45.136481700Z] loading plugin "io.containerd.grpc.v1.snapshots"...  type=io.containerd.grpc.v1
INFO[2020-01-25T17:29:45.136540500Z] loading plugin "io.containerd.grpc.v1.tasks"...  type=io.containerd.grpc.v1
INFO[2020-01-25T17:29:45.136586100Z] loading plugin "io.containerd.grpc.v1.version"...  type=io.containerd.grpc.v1
INFO[2020-01-25T17:29:45.136639600Z] loading plugin "io.containerd.grpc.v1.introspection"...  type=io.containerd.grpc.v1
INFO[2020-01-25T17:29:45.136940600Z] serving...                                    address="/var/run/docker/containerd/containerd-debug.sock"
INFO[2020-01-25T17:29:45.137072900Z] serving...                                    address="/var/run/docker/containerd/containerd.sock"
INFO[2020-01-25T17:29:45.137139300Z] containerd successfully booted in 0.022860s
INFO[2020-01-25T17:29:45.152257600Z] parsed scheme: "unix"                         module=grpc
INFO[2020-01-25T17:29:45.152303900Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2020-01-25T17:29:45.152321800Z] ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}] <nil>}  module=grpc
INFO[2020-01-25T17:29:45.152399200Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2020-01-25T17:29:45.153748300Z] parsed scheme: "unix"                         module=grpc
INFO[2020-01-25T17:29:45.153806800Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2020-01-25T17:29:45.153824400Z] ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}] <nil>}  module=grpc
INFO[2020-01-25T17:29:45.153865300Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2020-01-25T17:29:45.192289700Z] Loading containers: start.
WARN[2020-01-25T17:29:45.201964800Z] Running modprobe bridge br_netfilter failed with message: ip: can't find device 'bridge'
ip: can't find device 'br_netfilter'
modprobe: can't change directory to '/lib/modules': No such file or directory
, error: exit status 1
INFO[2020-01-25T17:29:45.269345600Z] Default bridge (docker0) is assigned with an IP address 172.18.0.0/16. Daemon option --bip can be used to set a preferred IP address
INFO[2020-01-25T17:29:45.311257400Z] Loading containers: done.
INFO[2020-01-25T17:29:45.410423700Z] Docker daemon                                 commit=633a0ea838 graphdriver(s)=overlay2 version=19.03.5
INFO[2020-01-25T17:29:45.410562300Z] Daemon has completed initialization
INFO[2020-01-25T17:29:45.450419200Z] API listen on /var/run/docker.sock

NOT WORKING DEAMON LOG:

INFO[2020-01-25T17:34:31.593017700Z] Starting up
WARN[2020-01-25T17:34:31.594118400Z] could not change group /var/run/docker.sock to docker: group docker not found
INFO[2020-01-25T17:34:31.595102900Z] libcontainerd: started new containerd process  pid=38
INFO[2020-01-25T17:34:31.595194900Z] parsed scheme: "unix"                         module=grpc
INFO[2020-01-25T17:34:31.595232900Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2020-01-25T17:34:31.595255400Z] ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}] <nil>}  module=grpc
INFO[2020-01-25T17:34:31.595292400Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2020-01-25T17:34:31.619489200Z] starting containerd                           revision=b34a5c8af56e510852c35414db4c1f4fa6172339 version=v1.2.10
INFO[2020-01-25T17:34:31.621183600Z] loading plugin "io.containerd.content.v1.content"...  type=io.containerd.content.v1
INFO[2020-01-25T17:34:31.625726200Z] loading plugin "io.containerd.snapshotter.v1.btrfs"...  type=io.containerd.snapshotter.v1
WARN[2020-01-25T17:34:31.627884800Z] failed to load plugin io.containerd.snapshotter.v1.btrfs  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
INFO[2020-01-25T17:34:31.627950300Z] loading plugin "io.containerd.snapshotter.v1.aufs"...  type=io.containerd.snapshotter.v1
WARN[2020-01-25T17:34:31.632611500Z] failed to load plugin io.containerd.snapshotter.v1.aufs  error="modprobe aufs failed: "ip: can't find device 'aufs'nmodprobe: can't change directory to '/lib/modules': No such file or directoryn": exit status 1"
INFO[2020-01-25T17:34:31.632663700Z] loading plugin "io.containerd.snapshotter.v1.native"...  type=io.containerd.snapshotter.v1
INFO[2020-01-25T17:34:31.637093400Z] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  type=io.containerd.snapshotter.v1
INFO[2020-01-25T17:34:31.644923000Z] loading plugin "io.containerd.snapshotter.v1.zfs"...  type=io.containerd.snapshotter.v1
INFO[2020-01-25T17:34:31.645207000Z] skip loading plugin "io.containerd.snapshotter.v1.zfs"...  type=io.containerd.snapshotter.v1
INFO[2020-01-25T17:34:31.645248900Z] loading plugin "io.containerd.metadata.v1.bolt"...  type=io.containerd.metadata.v1
WARN[2020-01-25T17:34:31.647323000Z] could not use snapshotter zfs in metadata plugin  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin"
WARN[2020-01-25T17:34:31.647374700Z] could not use snapshotter btrfs in metadata plugin  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
WARN[2020-01-25T17:34:31.647413300Z] could not use snapshotter aufs in metadata plugin  error="modprobe aufs failed: "ip: can't find device 'aufs'nmodprobe: can't change directory to '/lib/modules': No such file or directoryn": exit status 1"
INFO[2020-01-25T17:34:31.658808600Z] loading plugin "io.containerd.differ.v1.walking"...  type=io.containerd.differ.v1
INFO[2020-01-25T17:34:31.658869400Z] loading plugin "io.containerd.gc.v1.scheduler"...  type=io.containerd.gc.v1
INFO[2020-01-25T17:34:31.658976700Z] loading plugin "io.containerd.service.v1.containers-service"...  type=io.containerd.service.v1
INFO[2020-01-25T17:34:31.659026000Z] loading plugin "io.containerd.service.v1.content-service"...  type=io.containerd.service.v1
INFO[2020-01-25T17:34:31.659066800Z] loading plugin "io.containerd.service.v1.diff-service"...  type=io.containerd.service.v1
INFO[2020-01-25T17:34:31.659084100Z] loading plugin "io.containerd.service.v1.images-service"...  type=io.containerd.service.v1
INFO[2020-01-25T17:34:31.659105200Z] loading plugin "io.containerd.service.v1.leases-service"...  type=io.containerd.service.v1
INFO[2020-01-25T17:34:31.659142400Z] loading plugin "io.containerd.service.v1.namespaces-service"...  type=io.containerd.service.v1
INFO[2020-01-25T17:34:31.659200000Z] loading plugin "io.containerd.service.v1.snapshots-service"...  type=io.containerd.service.v1
INFO[2020-01-25T17:34:31.659303900Z] loading plugin "io.containerd.runtime.v1.linux"...  type=io.containerd.runtime.v1
INFO[2020-01-25T17:34:31.661608400Z] loading plugin "io.containerd.runtime.v2.task"...  type=io.containerd.runtime.v2
INFO[2020-01-25T17:34:31.665546500Z] loading plugin "io.containerd.monitor.v1.cgroups"...  type=io.containerd.monitor.v1
INFO[2020-01-25T17:34:31.665990600Z] loading plugin "io.containerd.service.v1.tasks-service"...  type=io.containerd.service.v1
INFO[2020-01-25T17:34:31.666066600Z] loading plugin "io.containerd.internal.v1.restart"...  type=io.containerd.internal.v1
INFO[2020-01-25T17:34:31.666174600Z] loading plugin "io.containerd.grpc.v1.containers"...  type=io.containerd.grpc.v1
INFO[2020-01-25T17:34:31.666218300Z] loading plugin "io.containerd.grpc.v1.content"...  type=io.containerd.grpc.v1
INFO[2020-01-25T17:34:31.666230600Z] loading plugin "io.containerd.grpc.v1.diff"...  type=io.containerd.grpc.v1
INFO[2020-01-25T17:34:31.666278800Z] loading plugin "io.containerd.grpc.v1.events"...  type=io.containerd.grpc.v1
INFO[2020-01-25T17:34:31.666328100Z] loading plugin "io.containerd.grpc.v1.healthcheck"...  type=io.containerd.grpc.v1
INFO[2020-01-25T17:34:31.666376600Z] loading plugin "io.containerd.grpc.v1.images"...  type=io.containerd.grpc.v1
INFO[2020-01-25T17:34:31.666426200Z] loading plugin "io.containerd.grpc.v1.leases"...  type=io.containerd.grpc.v1
INFO[2020-01-25T17:34:31.666482200Z] loading plugin "io.containerd.grpc.v1.namespaces"...  type=io.containerd.grpc.v1
INFO[2020-01-25T17:34:31.666521100Z] loading plugin "io.containerd.internal.v1.opt"...  type=io.containerd.internal.v1
INFO[2020-01-25T17:34:31.666765200Z] loading plugin "io.containerd.grpc.v1.snapshots"...  type=io.containerd.grpc.v1
INFO[2020-01-25T17:34:31.666816100Z] loading plugin "io.containerd.grpc.v1.tasks"...  type=io.containerd.grpc.v1
INFO[2020-01-25T17:34:31.666858000Z] loading plugin "io.containerd.grpc.v1.version"...  type=io.containerd.grpc.v1
INFO[2020-01-25T17:34:31.666870200Z] loading plugin "io.containerd.grpc.v1.introspection"...  type=io.containerd.grpc.v1
INFO[2020-01-25T17:34:31.667191200Z] serving...                                    address="/var/run/docker/containerd/containerd-debug.sock"
INFO[2020-01-25T17:34:31.667381600Z] serving...                                    address="/var/run/docker/containerd/containerd.sock"
INFO[2020-01-25T17:34:31.667412100Z] containerd successfully booted in 0.053759s
INFO[2020-01-25T17:34:31.689723400Z] parsed scheme: "unix"                         module=grpc
INFO[2020-01-25T17:34:31.689779400Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2020-01-25T17:34:31.689797300Z] ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}] <nil>}  module=grpc
INFO[2020-01-25T17:34:31.689838400Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2020-01-25T17:34:31.695497300Z] parsed scheme: "unix"                         module=grpc
INFO[2020-01-25T17:34:31.695550500Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2020-01-25T17:34:31.695569300Z] ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}] <nil>}  module=grpc
INFO[2020-01-25T17:34:31.695605400Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2020-01-25T17:34:31.783614100Z] Loading containers: start.
WARN[2020-01-25T17:34:31.799928200Z] Running modprobe bridge br_netfilter failed with message: ip: can't find device 'bridge'
ip: can't find device 'br_netfilter'
modprobe: can't change directory to '/lib/modules': No such file or directory
, error: exit status 1
INFO[2020-01-25T17:34:31.931323700Z] Default bridge (docker0) is assigned with an IP address 172.18.0.0/16. Daemon option --bip can be used to set a preferred IP address
INFO[2020-01-25T17:34:31.987528700Z] Loading containers: done.
WARN[2020-01-25T17:34:32.033863200Z] Not using native diff for overlay2, this may cause degraded performance for building images: failed to set opaque flag on middle layer: operation not supported  storage-driver=overlay2
INFO[2020-01-25T17:34:32.036451000Z] Docker daemon                                 commit=633a0ea838 graphdriver(s)=overlay2 version=19.03.5
INFO[2020-01-25T17:34:32.038604500Z] Daemon has completed initialization
INFO[2020-01-25T17:34:32.200449100Z] API listen on /var/run/docker.sock

Can someone help clarify how to reproduce this issue easily?

Clean/Purge data on Docker app desktop

Skimming the overlay2 code, there’s a direct call to readlink when it’s looking for lower directories

The errors in this issue could happen if the lower file were corrupt, or if the write to it were short. That file being truncated/not fully written would lead to the errors described here I think.

@discordianfish If you check the contents of the lower files in /var/lib/docker/overlay2/$id/lower, I think you might find one that ends with a : character rather than with a valid path. Running head -n9999 /var/lib/docker/overlay2/*/lower as root should display each «lower» file and its contents. Can you either paste that output somewhere or verify that one of them is erroneous?

Since docker isn’t doing an atomic write+move operation to update those files, there’s numerous ways in which they could end up truncated in such a way. If these errors are the result of corrupt lower files, switching to an atomic update of those files should resolve the issue.

@thaJeztah No worries about the ping; I’m still happy to hack on container-stuff regardless. For CoreOS-related issues, @bgilbert should be able to help with them.

I also meet this problem in the latest version docker 20.10. And rmi image and pull again or restart dockerd can not slove.

I have traced get more new information. The problem layer link and lower file’s content is empty:

committed  diff  link  lower  work

cat link
cat lower

This will cause Driver.getLower in daemon/graphdriver/overlay2/overlay.go get parentLink with «», and lowers is [l]

lowers := []string{path.Join(linkDir, string(parentLink))}

And parentLower is «», which lead parentLowers get []

parentLowers := strings.Split(string(parentLower), ":")

And finally, lowers will get [l, ]. Driver.getLower function will return l:

lowers = append(lowers, parentLowers...)
...
return strings.Join(lowers, ":"), nil

Hello,
This issue is opened since 2017, I have not seen any progress here…
I am seeing this issue happening on an embedded device very often. In fact, it’s too easy to reproduce, simply set the restart always policy and have you container hang. Forcing such loop will end up with the above issue very fast…
I believe the docker implementation must be bullet proof, being power cut safe should be at the highest priority…
If this issue has been resolved with some docker version would be glad to hear which version….
If not, will try helping as possible to resolve this.

Thanks

Just to add my voice to the crowd, we see this as well on embedded boards — however we haven’t been able to come up with a solid way to reproduce it — as it just seems random. It’s either network connectivity issues or something similar that randomly happens and essentially kills the container from ever running again without running a docker system prune -a -f and reinstalling / pulling all containers again.

@tirankas — Do you have further information on how to reproduce this issue regularly? We’ve only been able to reproduce randomly — normally with zero interaction…

Hi @CRCinAU ,
I believe the best way to reproduce it is with a container that crashes randomly, simply run this container with a restart always and you will see this issue.
It might take time to reproduce it, from minutes to hours…

Thanks

Hi,
we are seeing the same issue regularly. I have not been able to reproduce it so far and am still trying to verify what conditions have to be met in order for this to happen (error during deployment, container crash, power loss, etc).

Once I have a case at hand, I will verify if the problems is identical to the observations made by @zvier in #35747 (comment).

What is not clear to me, is how the lower files can be corrupted during regular operation. I see the pull request #42978 as trying to correct the problem once it happens but would like to understand what operations happen on lower files that could end up with invalid content / empty files.
Especially, since I have seen that occur with base layers, which we haven’t changed in ages.

This comment was marked as off-topic.

Источник

Содержание

docker: Error response from daemon: invalid argument. #32651
Comments
Error response from daemon: Cannot start container : [8] System error: invalid argument #16150
Comments
ImageInspectError — Failed to inspect image — Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument #738
Comments
Is this a BUG REPORT or FEATURE REQUEST?
Versions
What happened?
What you expected to happen?
How to reproduce it (as minimally and precisely as possible)?
Anything else we need to know?
Is this a BUG REPORT or FEATURE REQUEST?
Versions
What happened?
What you expected to happen?
How to reproduce it (as minimally and precisely as possible)?
Anything else we need to know?
«docker cp» not working when bind mounting cgroup dir with btrfs storage driver #38252
Comments
BUG REPORT INFORMATION
[1.9.0-rc4] Containers failing randomly with [8] System error: invalid argument #17577
Comments

docker: Error response from daemon: invalid argument. #32651

Description
Just installed docker and was trying to run the docker/whalesay image. Ran into the following error:

docker run docker/whalesay
docker: Error response from daemon: mkdir /var/lib/docker/overlay/a721883eda8a130b2f6744e89997351689e620c0d8680fa2c119dfa8dff59e74-init/merged/dev/shm: invalid argument.
See ‘docker run —help’.

I’ve seen some defects which talk about overlay. I really don’t understand what this means and neither does it explicitly state how to fix the problem. I’m new to using docker and gitHub so clear explanations would be greatly appreciated.

Output of docker version:

docker version
Client:
Version: 17.03.1-ce
API version: 1.27
Go version: go1.7.5
Git commit: c6d412e
Built: Mon Mar 27 17:05:44 2017
OS/Arch: linux/amd64

Server:
Version: 17.03.1-ce
API version: 1.27 (minimum version 1.12)
Go version: go1.7.5
Git commit: c6d412e
Built: Mon Mar 27 17:05:44 2017
OS/Arch: linux/amd64
Experimental: false

Output of docker info :
[root@ggnlabvm-juper5

]# docker info
Containers: 2
Running: 0
Paused: 0
Stopped: 2
Images: 2
Server Version: 17.03.1-ce
Storage Driver: overlay
Backing Filesystem: xfs
Supports d_type: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-229.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.703 GiB
Name: ggnlabvm-juper5
ID: Q5J3:Z4ZJ:U54L:FKP6:RQGH:EHE5:KK62:YHDC:BMTY:DSZR:O4X2:QEAM
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

The text was updated successfully, but these errors were encountered:

Источник

Error response from daemon: Cannot start container : [8] System error: invalid argument #16150

Description of problem:
Upgraded to 1.7.1 this morning, since then I can’t start any docker containers. RHEL 6.6

Linux [REDACTED] 2.6.32-504.30.3.el6.x86_64 #1 SMP Thu Jul 9 15:20:47 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

Environment details (AWS, VirtualBox, physical, etc.):

How reproducible:
Everytime

Steps to Reproduce:

Expected Results:
New Container Running

Was previously using 1.6.2
Even tried to revert the package and got this:

All ran the check-config script. definitely some issues, but not sure what to do about them:

The text was updated successfully, but these errors were encountered:

I also experienced the same issue, #14386

do you have DOCKER_STORAGE_OPTIONS defined in /etc/sysconfig/docker-storage?

if so, edit /etc/init.d/docker and fix the line
$exec -d $other_args &>> $logfile &
with:
$exec -d $other_args $ &>> $logfile &

Thanks for opening this issue. I’m going to close it though as it’s duplicated. That way we can centralize all the information regarding the same problem under only one issue. Please, feel free to follow up in #14386.

I don’t understand why this was closed. Issue #14386 has an error stating there is no disk space left while this one states an invalid argument. I have this same issue now (or very similar) I upgraded from RHEL 6.6 to RHEL 6.7 and I upgraded to Docker 1.7.1 and now I cannot start any containers as I get «invalid argument» error every time with every image. I can do a lot of other things with docker, like build new images, but I can’t run containers. Personally, I think it has something to do with the kernel supporting Docker 1.7.1 And I suspect I will have to uninstall docker and reinstall 1.6.x . Either that or upgrade my kernel to RHEL 7 I would like any advice or pointers to instruction on how to do either of these scenarios

@mbaudisch if you are using docker-io-1.7.1-2 from epel, try upgrading to the testing version 1.7.1-4 which fixes the devicemapper storage driver setup

@mbaudisch I agree. #14386 gives a different error. I have the same issue here. @truatpasteurdotfr tried that, but it didn’t fix the issue.

poxa o meu erro persiste

FATA[0012] Error response from daemon: Cannot start container 82da267bde763f0a3f8d6452ffbe5252247bb4ed068db1f8ebae0fa1eebea970: [8] System error: mountpoint for devices not found

estou tentando inicializar um app cloud-torrent

Источник

ImageInspectError — Failed to inspect image — Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument #738

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: BUG REPORT

Versions

kubeadm version (use kubeadm version ):

Environment:

Kubernetes version (use kubectl version ):

Cloud provider or hardware configuration:
OS (e.g. from /etc/os-release):

Kernel (e.g. uname -a ):
Linux master1-dev 3.10.0-693.el7.x86_64

kubeadm join on slave node fails preflight checks #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Others:

What happened?

I created a cluster with kubeadm init —pod-network-cidr=10.244.0.0/16 —kubernetes-version 1.9.6 —ignore-preflight-errors=cri

All control plane pods are running Running state
However I cannot start a simple postgres pod on the cluster. kubectl describe pod postgres-664cfc9966-sz9zf shows

What you expected to happen?

How to reproduce it (as minimally and precisely as possible)?

kubeadm init —pod-network-cidr=10.244.0.0/16 —kubernetes-version 1.9.6 —ignore-preflight-errors=cri
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml
kubeadm token create —print-join-command
join minions
kubectl create -f postgres.yaml

Anything else we need to know?

The text was updated successfully, but these errors were encountered:

@dkirrane this is not a kubeadm specific error, but I’ll loop in some other folks to see if they want to comment. It seems like an docker storage or image issue.

I moved to kubeadm and Kubernetes 1.10.0 and works now.

I’m hitting this same issue again. Same CentOS VM. This time I’m using kubeadmn 1.10.3 and docker 18.03.1-ce and same CentOS as above. Can this be re-opened. It seems to be an intermittent issue

@dkirrane report this in the kubernetes/kubernetes repo

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: BUG REPORT

Versions

kubeadm version (use kubeadm version ):

Environment:

Kubernetes version (use kubectl version ):

Cloud provider or hardware configuration:
OS (e.g. from /etc/os-release):

Kernel (e.g. uname -a ):
Linux master1-dev 3.10.0-693.el7.x86_64

kubeadm join on slave node fails preflight checks #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Others:

What happened?

I created a cluster with kubeadm init —pod-network-cidr=10.244.0.0/16 —kubernetes-version 1.9.6 —ignore-preflight-errors=cri

All control plane pods are running Running state
However I cannot start a simple postgres pod on the cluster. kubectl describe pod postgres-664cfc9966-sz9zf shows

What you expected to happen?

How to reproduce it (as minimally and precisely as possible)?

kubeadm init —pod-network-cidr=10.244.0.0/16 —kubernetes-version 1.9.6 —ignore-preflight-errors=cri
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml
kubeadm token create —print-join-command
join minions
kubectl create -f postgres.yaml

Anything else we need to know?

HI @dkirrane I got the same problem too.
Have you find any solution to fix it ?

Источник

«docker cp» not working when bind mounting cgroup dir with btrfs storage driver #38252

BUG REPORT INFORMATION

Description

Since 18.09 version, docker cp is not working at all for container with /sys/fs/cgroup bind mounted and if storage driver is btrfs. By bind mounting another dir, or by using vfs or overlay2, it is working fine.

Steps to reproduce the issue:

docker run -ti —name test —mount type=bind,source=/sys/fs/cgroup,target=/sys/fs/cgroup,readonly centos /usr/bin/echo created
touch foo.file
docker cp foo.file test:/tmp

Describe the results you received:

Error response from daemon: invalid argument

Describe the results you expected:

It should copy foo.file into the test container in /tmp

Additional information you deem important (e.g. issue happens only occasionally):

—mount type=bind,source=/sys/fs/cgroup,target=/sys/fs/cgroup,readonly and BTRFS storage driver are both needed to trigger the bug.

Output of docker version :

Output of docker info :

Additional environment details (AWS, VirtualBox, physical, etc.):
physical laptop

The text was updated successfully, but these errors were encountered:

The error is coming from

if err := mount . MakeRPrivate ( dest ); err != nil <

and is like this (I have added some log statements to the code):

«(*Daemon).mountVolumes: mount.MakeRPrivate(«/var/lib/docker/btrfs/subvolumes/dde19ba26d33cc3cf4398e4b3b80c10823ef6dbec4e86d2b160cddf67f5cc12f/sys/fs/cgroup»): invalid argument»

and the error could have been much more verbose if #38068 is merged.

and the error could have been much more verbose if #38068 is merged.

With the code from that PR, the error message now is

Error response from daemon: remount /var/lib/docker/btrfs/subvolumes/dde19ba26d33cc3cf4398e4b3b80c10823ef6dbec4e86d2b160cddf67f5cc12f/sys/fs/cgroup, flags: 0x44000: invalid argument

This is MS_REC | MS_PRIVATE which is what «rprivate» translates to.

Now, mount binary fails with the same error:

sudo strace mount —make-rprivate /var/lib/docker/btrfs/subvolumes/dde19ba26d33cc3cf4398e4b3b80c10823ef6dbec4e86d2b160cddf67f5cc12f/sys/fs/cgroup
.
mount(«none», «/var/lib/docker/btrfs/subvolumes/dde19ba26d33cc3cf4398e4b3b80c10823ef6dbec4e86d2b160cddf67f5cc12f/sys/fs/cgroup», NULL, MS_REC|MS_PRIVATE, NULL) = -1 EINVAL (Invalid argument)

Nice; I just found out that my set of optimizations for pkg/mount (#38992) accidentally fixes this bug. Specifically, commit a6773f6.

What happens is docker client generates two successive API calls, HEAD /containers/ID/archive and GET (or PUT ) /containers/ID/archive . First call handler calls (*Daemon).ContainerStatPath which mounts all the volumes (daemon.mountVolumes), then lazily unmounts it. Second call handler calls (*Daemon).ContainerExtractToDir (for PUT ) which also mounts all the volumes (daemon.mountVolumes).

It looks like the first unmount does not remove the mount from the /proc/self/mountinfo , when the second call to mount.Mount() does not do anything (since the mount is already there), and the second call to mount.MakePRivate() fails (as the mount is to be unmounted? dunno).

Now, the key is to figure out why after the first unmount the mount is still there.

Tried same with overlay2: the first unmount succeeds. Not sure how btrfs affects things; in theory it should not as it’s only used as a mount point.

@kolyshkin so this issue is fixed for BTRFS but not for overlay2 (or didn’t overlay2 have the issue in the first place?)

so this issue is fixed for BTRFS but not for overlay2 (or didn’t overlay2 have the issue in the first place?)

overlay2 does not have this issue. The above mentioned commit makes the bug go away for btrfs.

I am still looking into it, can’t fully understand why btrfs is acting weird wrt bind mounts on top of it.

I’ve run into a similar issue a few times:

Error response from daemon: remount /var/lib/docker/overlay2/cc7ff474f61db9a2181b40399623a50c21c06e4e5e095d8dd92f08c111785483/merged/my/folder, flags: 0x44000: invalid argument

This seems to happen whenever I try to use Docker cp on a container which has:

a directory that is bind mounted as rslave
nested mounts propagated into the rslave mount

When this happens, the docker container is completely borked, it can’t even be successfully removed without a restart of the host because the mounts are stuck and cannot be unmounted.

It looks like something is going wrong during Docker CP because the Daemon mounts the Container’s mounts, which include the nested mounts, and it fails to unmount them when it’s done, which leads to this perma-stuck state for the container.

Would be really nice if this could be fixed because currently I basically have to unmount/remount all nested mounts around calls to Docker cp.

Источник

[1.9.0-rc4] Containers failing randomly with [8] System error: invalid argument #17577

The bug looks similar to #17527 and, in fact, is the new behaviour after #17527 has been patched, but sure is not its duplicate.

I tried to narrow it down a little and here’s what I’ve found. There are a few potential places in the code where this error could be thrown: usages of libcontainer.SystemError , all of them are in vendor/src/github.com/opencontainers/runc/libcontainer/factory_linux.go . The error is triggered by trying to run concurrent —net=host containers. If you run the repro script w/o —net=host , the issue goes away.

Repro: the easiest is to run the following in the docker development container (the one built by make binary ):

Supplementary info:
docker version :

The text was updated successfully, but these errors were encountered:

I can confirm that I see this on master in make shell .

Let’s close this after merge in docker and confirmation from rancheros.

yes. I didnt know that the issue can be closed cross projects. with the fixes attribute.

@LK4D4 @mavenugo This patch has fixed this issue and RancherOS boots properly. On subsequent boot though we are now getting errors such as

I’ll keep investigating

A bit more info, just grepping for the one of the failed container ids yields

This seems to be related to not having a graceful shutdown in Docker. In rancheros we develop in KVM and as such most of the time when I reboot I just kill the VM. Quite rarely do I shutdown gracefully. If I do gracefully shutdown I don’t see this behavior. I will try to create a simple tests to reproduce.

@mavenugo Here’s a simple way to reproduce. I run this in the root of the docker git repo.

@ibuildthecloud it seems you are trying to run multiple docker daemons with the same docker root ( -g / —graph ). The sandbox cleanup failures I think are expected in that case. Those are error messages and don’t stop the containers from booting.

But the message —
time=»2015-11-02T03:06:43.647720373Z» level=warning msg=»HostsPath set to »/var/lib/system-docker/containers/fbf9d23e25f25a54e40273cf09eaa61fc0942a74213b260cc9e15eeb7feed713/hosts», but can’t stat this filename (err = stat /var/lib/system-docker/containers/fbf9d23e25f25a54e40273cf09eaa61fc0942a74213b260cc9e15eeb7feed713/hosts: no such file or directory); skipping» indicates the missing /etc/hosts for the container, which indicates that multiple docker daemon sharing the same docker root is causing this.

Can you please try the scenario with executing each docker daemon’s with its dedicated -g option ?

@mrjana will have more information to share regarding this.

@mavenugo I don’t think @ibuildthecloud is using the same docker root. In fact he is bind mounting $(pwd)/test as his /var/lib/docker for the inside docker. So outside docker and inside docker are in fact running in different docker root.

@ibuildthecloud I was able to repro the problem using your script. Will send out an analysis after investigation.

The test case launches a daemon, kills it, and then starts it again. I use the same graph dir to simulate a hard reboot, but at no time are two daemons running using the same graph root.

@ibuildthecloud Fixed the problem with your ungraceful restart scenario. Will add the fix as part of the vendoring in libnetwork.

@mrjana Thanks! I’ll test early tomorrow morning.

Источник

#kubernetes

Вопрос:

Сегодня мой kubernetes 1.15 показывает эту ошибку:

 Failed to inspect image "registry.cn-hangzhou.aliyuncs.com/dabai_app_k8s/dabai-fat/soa-illidan-superhub:v1.0.3": rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument
Error: ImageInspectError

Я попытался извлечь изображение на своем локальном компьютере:

 ~ ⌚ 10:57:02
$ docker pull registry.cn-hangzhou.aliyuncs.com/dabai_app_k8s/dabai-fat/soa-illidan-superhub:v1.0.3
v1.0.3: Pulling from dabai_app_k8s/dabai-fat/soa-illidan-superhub
e9afc4f90ab0: Already exists
989e6b19a265: Already exists
af14b6c2f878: Already exists
5573c4b30949: Already exists
fb1a405f128d: Already exists
197b0f525c26: Already exists
f133ed18caca: Already exists
ec53837eaf93: Pull complete
24caf1aa821b: Pull complete
f2d6b0ee2469: Pull complete
Digest: sha256:43caa136d717fcca3a6aad96568c9d29745f3c3d391f29facc2bebeb9c26b5a0
Status: Downloaded newer image for registry.cn-hangzhou.aliyuncs.com/dabai_app_k8s/dabai-fat/soa-illidan-superhub:v1.0.3
registry.cn-hangzhou.aliyuncs.com/dabai_app_k8s/dabai-fat/soa-illidan-superhub:v1.0.3
(base)

работает нормально, и я пытаюсь извлечь изображение на удаленной кластерной машине kubernetes, работает нормально. Похоже, проблема с kubernetes, что я должен сделать, чтобы ее исправить?

Это моя версия kubernetes:

 $ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T23:35:15Z", GoVersion:"go1.14.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
(base)

Ответ №1:

Я предлагаю вам сначала остановить движок docker, systemctl stop docker например, и вручную удалить папку с помощью rm -rf /var/lib/docker . затем вам следует снова запустить docker с systemctl start docker помощью .

ссылка: https://github.com/docker/for-mac/issues/1396

Источник

This comment was marked as off-topic.

docker: Error response from daemon: invalid argument. #32651

Error response from daemon: Cannot start container : [8] System error: invalid argument #16150

ImageInspectError — Failed to inspect image — Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument #738

Is this a BUG REPORT or FEATURE REQUEST?

Versions

What happened?

What you expected to happen?

How to reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

Is this a BUG REPORT or FEATURE REQUEST?

Versions

What happened?

What you expected to happen?

How to reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

«docker cp» not working when bind mounting cgroup dir with btrfs storage driver #38252

BUG REPORT INFORMATION

[1.9.0-rc4] Containers failing randomly with [8] System error: invalid argument #17577

Вопрос:

Комментарии:

Ответ №1:

Читайте также: