内置rbd-hub push image failure

syngna

运行一段时间后，程序构建会出现镜像push失败的问题，重启服务器后恢复正常
Image description

SmallQi

请提供 Rainbond 版本以及 rbd-chaos 日志/

kubectl logs -fl name=rbd-chaos -n rbd-system

syngna

@“Qi Zhang” 版本 5.8.1 基于web界面安装
Image description
time=“2022-11-11T12:24:56+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221111122047 container exit 0 and complete”
time=“2022-11-11T12:24:56+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221111122047 container ec28973068ea1c2c557b34cce114e549-20221111122047 state {Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2022-11-11 12:20:57 +0800 CST,FinishedAt:2022-11-11 12:24:54 +0800 CST,ContainerID:docker://7da1af504b763626b4ce9e7d50282b7a02ec1f84355f45a52540a0e6ce167717,}}”
time=“2022-11-11T12:24:56+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221111122047 container exit 0 and complete”
time=“2022-11-11T12:24:56+08:00” level=info msg=“delete job ec28973068ea1c2c557b34cce114e549-20221111122047 finish”
time=“2022-11-11T12:24:56+08:00” level=info msg=“[Watch] Build job pod ec28973068ea1c2c557b34cce114e549-20221111122047 deleted”
time=“2022-11-11T12:24:56+08:00” level=info msg=“pull image goodrain.me/runner successfully.”
time=“2022-11-11T12:25:24+08:00” level=warning msg=“10.42.0.218:3228 lid[3ec28464e2a9d623] keepAlive err: etcdserver: requested lease not found, try to reset…”
time=“2022-11-11T12:25:29+08:00” level=info msg=“Register a builder server endpoint 10.42.0.218:3228 to cluster”
time=“2022-11-11T12:25:29+08:00” level=info msg=“10.42.0.218:3228 set lid[3ec28464e2a9dae4] success”
time=“2022-11-11T12:34:27+08:00” level=info msg=“Build task 8fc8247112a240ee994b80e2231e1088 is completed”
time=“2022-11-12T08:53:42+08:00” level=info msg=“Build task 703637692ccc4da28a1beb40d67bb9ac in progress”
time=“2022-11-12T08:53:42+08:00” level=info msg=“cache code dir is /cache/source/build/0e9e5a96c2b54fadbfb4df817ff1379b/ef42dbb523fcbd4171022662965349530c95571a for service 2e5fe404ccc1ee2a69f762a2c46d7186”
[mysql] 2022/11/12 08:53:46 packets.go:122: closing bad idle connection: EOF
[mysql] 2022/11/12 08:53:46 connection.go:158: driver: bad connection
time=“2022-11-12T08:54:49+08:00” level=info msg=“start auto release event logger. d75761feb23a437a88f6d4b2ed8ee494”

SmallQi

再次出现镜像不能push的问题，尝试手动 docker push 一下，然后提供 rbd-chaos 日志，现在你这个日志看起来都正常

syngna

SmallQi
嗯下次出问题我在试一下吧，这个日志是重启完之后的，目前正常。当时出问题的时候手动push试过，就是push到三百多兆后会一直卡在那里
Image description

[root@goodrain /]# docker push goodrain.me/0e9e5a96c2b54fadbfb4df817ff1379b-app-09d28186-gr14e549
The push refers to repository [goodrain.me/0e9e5a96c2b54fadbfb4df817ff1379b-app-09d28186-gr14e549]
ce4cb3017ee6: Layer already exists
1f5ff6002921: Layer already exists
cfd05f66e502: Layer already exists
b77622cc8a06: Layer already exists
2230c4e26094: Layer already exists
55cb7408d3fb: Layer already exists
625166966871: Layer already exists
5c61fe70ff7a: Layer already exists
1d1c35119166: Layer already exists
fc01035b11c7: Layer already exists
1c62aed69194: Layer already exists
47f6bd1a4eb2: Layer already exists
f929690eaa16: Layer already exists
f0902bdbab5f: Layer already exists
7bf04ad1d006: Layer already exists
023b7696bd58: Layer already exists
ad37eb38337a: Layer already exists
350bf4dddc59: Layer already exists
554f627f09ee: Layer already exists
20221110083606: digest: sha256:ca39d5caea95afc4e3ced1b905889d3d0fde509004a73d6e112499e4c94bcd40 size: 4302
0f41c74554ee: Layer already exists
217da7270150: Layer already exists
cfd05f66e502: Layer already exists
b77622cc8a06: Layer already exists
2230c4e26094: Layer already exists
55cb7408d3fb: Layer already exists
625166966871: Layer already exists
5c61fe70ff7a: Layer already exists
1d1c35119166: Layer already exists
fc01035b11c7: Layer already exists
1c62aed69194: Layer already exists
47f6bd1a4eb2: Layer already exists
f929690eaa16: Layer already exists
f0902bdbab5f: Layer already exists
7bf04ad1d006: Layer already exists
023b7696bd58: Layer already exists
ad37eb38337a: Layer already exists
350bf4dddc59: Layer already exists
554f627f09ee: Layer already exists
20221111100620: digest: sha256:0f88249d5f732bdaabc5d9688f6b7e107d83086791206bcab35cde5a62d6339e size: 4302
61370eeb4fdd: Pushing [==================================================>] 321.5MB

61370eeb4fdd: Layer already exists
566bdd677d1e: Pushing [==================================================>] 321.5MB
cfd05f66e502: Layer already exists
b77622cc8a06: Layer already exists
2230c4e26094: Layer already exists
55cb7408d3fb: Layer already exists
625166966871: Layer already exists
5c61fe70ff7a: Layer already exists
1d1c35119166: Layer already exists
fc01035b11c7: Layer already exists
1c62aed69194: Layer already exists
47f6bd1a4eb2: Layer already exists
f929690eaa16: Layer already exists
f0902bdbab5f: Layer already exists
7bf04ad1d006: Layer already exists
023b7696bd58: Layer already exists
ad37eb38337a: Layer already exists
350bf4dddc59: Layer already exists
554f627f09ee: Layer already exists
^C

SmallQi

目前无法判断问题，需要看日志结合现象排查

syngna

@“Qi Zhang” 您好，今天又出现了push失败的问题，对应的日志如下：

Image description

time=“2022-11-16T09:32:10+08:00” level=info msg=“Build task 996524c5c2e64c4cb3a84d78cfedca4f in progress”
time=“2022-11-16T09:32:10+08:00” level=info msg=“cache code dir is /cache/source/build/0e9e5a96c2b54fadbfb4df817ff1379b/d71a7802faedee2a17f77bf8931b2d5886ce2684 for service ec28973068ea1c2c557b34cce114e549”
time=“2022-11-16T09:32:21+08:00” level=info msg=“package code for building service ec28973068ea1c2c557b34cce114e549 version 20221116093209 successful, take time 15.949227ms”
time=“2022-11-16T09:32:21+08:00” level=info msg=“set maven setting config fw success”
time=“2022-11-16T09:32:21+08:00” level=info msg=“create build job ec28973068ea1c2c557b34cce114e549-20221116093209 for service ec28973068ea1c2c557b34cce114e549 build version 20221116093209”
time=“2022-11-16T09:32:21+08:00” level=info msg=“[Watch] Build job pod ec28973068ea1c2c557b34cce114e549-20221116093209 created”
time=“2022-11-16T09:32:21+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container ec28973068ea1c2c557b34cce114e549-20221116093209 state {Waiting:&ContainerStateWaiting{Reason:ContainerCreating,Message:,} Running:nil Terminated:nil}”
time=“2022-11-16T09:32:27+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container ec28973068ea1c2c557b34cce114e549-20221116093209 state {Waiting:&ContainerStateWaiting{Reason:ContainerCreating,Message:,} Running:nil Terminated:nil}”
time=“2022-11-16T09:32:29+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container ec28973068ea1c2c557b34cce114e549-20221116093209 state {Waiting:nil Running:&ContainerStateRunning{StartedAt:2022-11-16 09:32:28 +0800 CST,} Terminated:nil}”
time=“2022-11-16T09:32:37+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container ec28973068ea1c2c557b34cce114e549-20221116093209 state {Waiting:nil Running:&ContainerStateRunning{StartedAt:2022-11-16 09:32:28 +0800 CST,} Terminated:nil}”
time=“2022-11-16T09:32:47+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container ec28973068ea1c2c557b34cce114e549-20221116093209 state {Waiting:nil Running:&ContainerStateRunning{StartedAt:2022-11-16 09:32:28 +0800 CST,} Terminated:nil}”
time=“2022-11-16T09:32:57+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container ec28973068ea1c2c557b34cce114e549-20221116093209 state {Waiting:nil Running:&ContainerStateRunning{StartedAt:2022-11-16 09:32:28 +0800 CST,} Terminated:nil}”
time=“2022-11-16T09:33:07+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container ec28973068ea1c2c557b34cce114e549-20221116093209 state {Waiting:nil Running:&ContainerStateRunning{StartedAt:2022-11-16 09:32:28 +0800 CST,} Terminated:nil}”
time=“2022-11-16T09:33:17+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container ec28973068ea1c2c557b34cce114e549-20221116093209 state {Waiting:nil Running:&ContainerStateRunning{StartedAt:2022-11-16 09:32:28 +0800 CST,} Terminated:nil}”
time=“2022-11-16T09:33:27+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container ec28973068ea1c2c557b34cce114e549-20221116093209 state {Waiting:nil Running:&ContainerStateRunning{StartedAt:2022-11-16 09:32:28 +0800 CST,} Terminated:nil}”
time=“2022-11-16T09:33:29+08:00” level=info msg=“start auto release event logger. 6bdac9625a5c46b28512490ff46c4a13”
time=“2022-11-16T09:33:37+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container ec28973068ea1c2c557b34cce114e549-20221116093209 state {Waiting:nil Running:&ContainerStateRunning{StartedAt:2022-11-16 09:32:28 +0800 CST,} Terminated:nil}”
time=“2022-11-16T09:33:47+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container ec28973068ea1c2c557b34cce114e549-20221116093209 state {Waiting:nil Running:&ContainerStateRunning{StartedAt:2022-11-16 09:32:28 +0800 CST,} Terminated:nil}”
time=“2022-11-16T09:33:57+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container ec28973068ea1c2c557b34cce114e549-20221116093209 state {Waiting:nil Running:&ContainerStateRunning{StartedAt:2022-11-16 09:32:28 +0800 CST,} Terminated:nil}”
time=“2022-11-16T09:34:07+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container ec28973068ea1c2c557b34cce114e549-20221116093209 state {Waiting:nil Running:&ContainerStateRunning{StartedAt:2022-11-16 09:32:28 +0800 CST,} Terminated:nil}”
time=“2022-11-16T09:34:17+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container ec28973068ea1c2c557b34cce114e549-20221116093209 state {Waiting:nil Running:&ContainerStateRunning{StartedAt:2022-11-16 09:32:28 +0800 CST,} Terminated:nil}”
time=“2022-11-16T09:34:27+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container ec28973068ea1c2c557b34cce114e549-20221116093209 state {Waiting:nil Running:&ContainerStateRunning{StartedAt:2022-11-16 09:32:28 +0800 CST,} Terminated:nil}”
time=“2022-11-16T09:34:37+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container ec28973068ea1c2c557b34cce114e549-20221116093209 state {Waiting:nil Running:&ContainerStateRunning{StartedAt:2022-11-16 09:32:28 +0800 CST,} Terminated:nil}”
time=“2022-11-16T09:34:43+08:00” level=error msg=“message dequeue failure rpc error: code = Unavailable desc = transport is closing, will retry”
time=“2022-11-16T09:34:47+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container ec28973068ea1c2c557b34cce114e549-20221116093209 state {Waiting:nil Running:&ContainerStateRunning{StartedAt:2022-11-16 09:32:28 +0800 CST,} Terminated:nil}”
time=“2022-11-16T09:34:57+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container ec28973068ea1c2c557b34cce114e549-20221116093209 state {Waiting:nil Running:&ContainerStateRunning{StartedAt:2022-11-16 09:32:28 +0800 CST,} Terminated:nil}”
time=“2022-11-16T09:35:07+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container ec28973068ea1c2c557b34cce114e549-20221116093209 state {Waiting:nil Running:&ContainerStateRunning{StartedAt:2022-11-16 09:32:28 +0800 CST,} Terminated:nil}”
time=“2022-11-16T09:35:17+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container ec28973068ea1c2c557b34cce114e549-20221116093209 state {Waiting:nil Running:&ContainerStateRunning{StartedAt:2022-11-16 09:32:28 +0800 CST,} Terminated:nil}”
time=“2022-11-16T09:35:27+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container ec28973068ea1c2c557b34cce114e549-20221116093209 state {Waiting:nil Running:&ContainerStateRunning{StartedAt:2022-11-16 09:32:28 +0800 CST,} Terminated:nil}”
time=“2022-11-16T09:35:37+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container ec28973068ea1c2c557b34cce114e549-20221116093209 state {Waiting:nil Running:&ContainerStateRunning{StartedAt:2022-11-16 09:32:28 +0800 CST,} Terminated:nil}”
time=“2022-11-16T09:35:47+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container ec28973068ea1c2c557b34cce114e549-20221116093209 state {Waiting:nil Running:&ContainerStateRunning{StartedAt:2022-11-16 09:32:28 +0800 CST,} Terminated:nil}”
time=“2022-11-16T09:35:57+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container ec28973068ea1c2c557b34cce114e549-20221116093209 state {Waiting:nil Running:&ContainerStateRunning{StartedAt:2022-11-16 09:32:28 +0800 CST,} Terminated:nil}”
time=“2022-11-16T09:36:07+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container ec28973068ea1c2c557b34cce114e549-20221116093209 state {Waiting:nil Running:&ContainerStateRunning{StartedAt:2022-11-16 09:32:28 +0800 CST,} Terminated:nil}”
time=“2022-11-16T09:36:07+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container ec28973068ea1c2c557b34cce114e549-20221116093209 state {Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2022-11-16 09:32:28 +0800 CST,FinishedAt:2022-11-16 09:36:07 +0800 CST,ContainerID:docker://b717c28bd2d4e5fa3b39d4a3aefad94bea8fa342351b9f1ffac716ab4178aa26,}}”
time=“2022-11-16T09:36:07+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container exit 0 and complete”
time=“2022-11-16T09:36:08+08:00” level=info msg=“job[ec28973068ea1c2c557b34cce114e549-20221116093209] get log complete”
time=“2022-11-16T09:36:08+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container ec28973068ea1c2c557b34cce114e549-20221116093209 state {Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2022-11-16 09:32:28 +0800 CST,FinishedAt:2022-11-16 09:36:07 +0800 CST,ContainerID:docker://b717c28bd2d4e5fa3b39d4a3aefad94bea8fa342351b9f1ffac716ab4178aa26,}}”
time=“2022-11-16T09:36:08+08:00” level=info msg=“job ec28973068ea1c2c557b34cce114e549-20221116093209 container exit 0 and complete”
time=“2022-11-16T09:36:08+08:00” level=info msg=“delete job ec28973068ea1c2c557b34cce114e549-20221116093209 finish”
time=“2022-11-16T09:36:08+08:00” level=info msg=“[Watch] Build job pod ec28973068ea1c2c557b34cce114e549-20221116093209 deleted”
time=“2022-11-16T09:36:08+08:00” level=info msg=“pull image goodrain.me/runner successfully.”
time=“2022-11-16T09:47:02+08:00” level=error msg=“push image error: context deadline exceeded”
time=“2022-11-16T09:47:04+08:00” level=error msg=“build from source code error: build runner image failure”
time=“2022-11-16T09:47:04+08:00” level=info msg=“Build task 996524c5c2e64c4cb3a84d78cfedca4f is completed”

SmallQi

尝试手动 docker push 一下镜像试试，确保 rbd-hub 正常工作，同时看下 rbd-hub 日志

syngna

@“Qi Zhang” 现在是好的了，这次报错之后再构建就是好的了，估计是网络问题。有台服务器ping的时候丢包比较严重