5.1.9升级到5.1.10后报错,rbd-worker 提示已经创建第三方服务端口异常

请完善如下信息,方便我们协助排查问题

5.1.9升级到5.1.10后报错,查看rbd-worker 后发现提示原来平台已经创建的第三方服务端口异常。请问要如何修复?

Rainbond版本:v5.1.10-release-6ab0b92-2020-01-09-07
操作系统:ubuntu16.04
内核版本:
环境:(云服务商,虚拟机等) vm
节点配置:3master+3computer
安装类型:私有化高可用
如何复现:
尝试解决: 重启各个
相关截图:
是否重新执行安装:否




抄送: @dazuimao1990

rbd-worker 还有除了 endpoint 外的日志吗?
因为图中的 worker 日志, 应该只会影响到第三方服务, 而不会影响到 worker 的健康状态.
另外, 除了 第三方组件, 其他功能正常吗? 比如组件启动, 更新等

grclis upgrade rbd-worker

执行完后报错变了
Jan 10 10:00:04 rainbondmaster1 bash[21807]: time=“2020-01-10T10:00:04+08:00” level=warning msg=“update endpoints for metrics-server: Operation cannot be fulfilled on endpoints “metrics-server”: the object has been modified; please apply your changes to the latest version and try again”

目前主要影响 控制台UI 管理拓扑图那个界面,实例多的超级慢。之前都是秒开。应用实例少的还是秒开。
其他我先测试下 反馈。

rbd-worker 不健康是因为在升级中?

集群监控检查已经通过。


目前是rbd-gateway 组件有大量报错。还有上面的那个问题。rbd-gateway组件应该有问题,实例-端口-对外服务。访问异常。重启或者更新微服务后,绑定的域名都访问不了。

Jan 10 10:46:44 rainbondmaster1 bash[1077]: time="2020-01-10T10:46:44+08:00" level=warning msg="Ingress Key: 61dd6b7160b4434ebde76a3fd5ab26cd/d1c9980d9050e39f810b71251b7cc82f; error parsing set-header: ingress rule without annotations"
Jan 10 10:46:44 rainbondmaster1 bash[1077]: time="2020-01-10T10:46:44+08:00" level=warning msg="Ingress Key: 61dd6b7160b4434ebde76a3fd5ab26cd/7570ab381afa33376e59e0501948b8bb; error parsing set-header: ingress rule without annotations"
Jan 10 10:46:44 rainbondmaster1 bash[1077]: time="2020-01-10T10:46:44+08:00" level=warning msg="Ingress Key: 61dd6b7160b4434ebde76a3fd5ab26cd/5931b2db649fe60ead1453116e748d4a; error parsing set-header: ingress rule without annotations"
Jan 10 10:47:04 rainbondmaster1 bash[1077]: time="2020-01-10T10:47:04+08:00" level=warning msg="wrong endpoint: :6442"
Jan 10 10:47:04 rainbondmaster1 bash[1077]: time="2020-01-10T10:47:04+08:00" level=warning msg="wrong endpoint: :8081"
Jan 10 10:35:31 rainbondmaster1 bash[1077]: 2020/01/10 10:35:31 [error] 5504#5504: *5925733 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.10.182, server: 5000.gr7be8c8.y8cmzbva.bestbooks.cn, request: "GET / HTTP/1.1", upstream: "http://10.0.231.202:65301/", host: "5000.gr7be8c8.y8cmzbva.bestbooks.cn", referrer: "http://192.168.10.212:7070/static/www/weavescope/index.html"
Jan 10 10:35:31 rainbondmaster1 bash[1077]: 2020/01/10 10:35:31 [error] 5504#5504: *5925733 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.10.182, server: 5000.gr7be8c8.y8cmzbva.bestbooks.cn, request: "GET / HTTP/1.1", upstream: "http://10.0.231.202:65301/", host: "5000.gr7be8c8.y8cmzbva.bestbooks.cn", referrer: "http://192.168.10.212:7070/static/www/weavescope/index.html"
Jan 10 10:35:31 rainbondmaster1 bash[1077]: 2020/01/10 10:35:31 [error] 5504#5504: *5925733 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.10.182, server: 5000.gr7be8c8.y8cmzbva.bestbooks.cn, request: "GET / HTTP/1.1", upstream: "http://10.0.231.202:65301/", host: "5000.gr7be8c8.y8cmzbva.bestbooks.cn", referrer: "http://192.168.10.212:7070/static/www/weavescope/index.html"
Jan 10 10:35:31 rainbondmaster1 bash[1077]: 2020/01/10 10:35:31 [error] 5504#5504: *5925733 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.10.182, server: 5000.gr7be8c8.y8cmzbva.bestbooks.cn, request: "GET / HTTP/1.1", upstream: "http://10.0.231.202:65301/", host: "5000.gr7be8c8.y8cmzbva.bestbooks.cn", referrer: "http://192.168.10.212:7070/static/www/weavescope/index.html"
Jan 10 10:35:31 rainbondmaster1 bash[1077]: 2020/01/10 10:35:31 [error] 5504#5504: *5925733 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.10.182, server: 5000.gr7be8c8.y8cmzbva.bestbooks.cn, request: "GET / HTTP/1.1", upstream: "http://10.0.231.202:65301/", host: "5000.gr7be8c8.y8cmzbva.bestbooks.cn", referrer: "http://192.168.10.212:7070/static/www/weavescope/index.html"
Jan 10 10:35:31 rainbondmaster1 bash[1077]: 2020/01/10 10:35:31 [error] 5504#5504: *5925733 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.10.182, server: 5000.gr7be8c8.y8cmzbva.bestbooks.cn, request: "GET / HTTP/1.1", upstream: "http://10.0.231.202:65301/", host: "5000.gr7be8c8.y8cmzbva.bestbooks.cn", referrer: "http://192.168.10.212:7070/static/www/weavescope/index.html"

检查了各个服务 截取有报错或者有警告的服务日志。希望对你们排查有帮助。




重新执行grclis upgrade rbd-worker 后报错信息变了。之前按升级文档操作完后就一直在报错。不是因为正在升级中。