ResultFAILURE
Tests 6 failed / 18 succeeded
Started2020-09-19 03:56
Elapsed1h43m
Work namespaceci-op-g5943rxq
pod12f9e618-fa2c-11ea-a1fd-0a580a800db2
revision1

Test Failures


Cluster upgrade Application behind service load balancer with PDB is not disrupted 1h4m

go run hack/e2e.go -v -test --test_args='--ginkgo.focus=Cluster\supgrade\sApplication\sbehind\sservice\sload\sbalancer\swith\sPDB\sis\snot\sdisrupted$'
Sep 19 05:33:21.153: Failed to wait for service to hasFinalizer=true: timed out waiting for the condition

github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/service.WaitForServiceUpdatedWithFinalizer(0x63a5e40, 0xc001110000, 0xc001eed050, 0x21, 0xc001443480, 0xc, 0x5767b01)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/service/wait.go:115 +0x23b
github.com/openshift/origin/test/e2e/upgrade/service.(*UpgradeTest).Test(0xc001501bf0, 0xc0025b7180, 0xc001f064e0, 0x2)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/e2e/upgrade/service/service.go:139 +0x757
github.com/openshift/origin/test/extended/util/disruption.(*chaosMonkeyAdapter).Test(0xc001915380, 0xc001e85580)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/util/disruption/disruption.go:143 +0x328
github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/chaosmonkey.(*Chaosmonkey).Do.func1(0xc001e85580, 0xc001c50da0)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:90 +0x76
created by github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/chaosmonkey.(*Chaosmonkey).Do
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:87 +0xa7
				from junit_upgrade_1600493601.xml

Filter through log files


Cluster upgrade Cluster frontend ingress remain available 44m36s

go run hack/e2e.go -v -test --test_args='--ginkgo.focus=Cluster\supgrade\sCluster\sfrontend\singress\sremain\savailable$'
Frontends were unreachable during disruption for at least 7m22s of 44m35s (17%):

Sep 19 04:40:38.976 E ns/openshift-authentication route/oauth-openshift Route stopped responding to GET requests over new connections
Sep 19 04:40:38.976 E ns/openshift-authentication route/oauth-openshift Route stopped responding to GET requests on reused connections
Sep 19 04:40:38.976 E ns/openshift-console route/console Route stopped responding to GET requests over new connections
Sep 19 04:40:39.316 I ns/openshift-authentication route/oauth-openshift Route started responding to GET requests over new connections
Sep 19 04:40:39.320 I ns/openshift-console route/console Route started responding to GET requests over new connections
Sep 19 04:40:39.323 I ns/openshift-authentication route/oauth-openshift Route started responding to GET requests on reused connections
Sep 19 04:40:49.976 E ns/openshift-authentication route/oauth-openshift Route stopped responding to GET requests over new connections
Sep 19 04:40:49.976 E ns/openshift-console route/console Route stopped responding to GET requests over new connections
Sep 19 04:40:50.319 I ns/openshift-console route/console Route started responding to GET requests over new connections
Sep 19 04:40:50.976 - 28s   E ns/openshift-authentication route/oauth-openshift Route is not responding to GET requests over new connections
Sep 19 04:40:54.976 E ns/openshift-console route/console Route stopped responding to GET requests on reused connections
Sep 19 04:40:55.976 - 8s    E ns/openshift-console route/console Route is not responding to GET requests on reused connections
Sep 19 04:41:02.976 E ns/openshift-console route/console Route stopped responding to GET requests over new connections
Sep 19 04:41:03.976 - 18s   E ns/openshift-console route/console Route is not responding to GET requests over new connections
Sep 19 04:41:05.314 I ns/openshift-console route/console Route started responding to GET requests on reused connections
Sep 19 04:41:16.976 E ns/openshift-console route/console Route stopped responding to GET requests on reused connections
Sep 19 04:41:17.976 - 4s    E ns/openshift-console route/console Route is not responding to GET requests on reused connections
Sep 19 04:41:20.312 I ns/openshift-authentication route/oauth-openshift Route started responding to GET requests over new connections
Sep 19 04:41:22.703 I ns/openshift-console route/console Route started responding to GET requests on reused connections
Sep 19 04:41:22.704 I ns/openshift-console route/console Route started responding to GET requests over new connections
Sep 19 04:41:48.976 E ns/openshift-authentication route/oauth-openshift Route stopped responding to GET requests on reused connections
Sep 19 04:41:48.976 E ns/openshift-console route/console Route stopped responding to GET requests on reused connections
Sep 19 04:41:48.976 E ns/openshift-authentication route/oauth-openshift Route stopped responding to GET requests over new connections
Sep 19 04:41:48.976 E ns/openshift-console route/console Route stopped responding to GET requests over new connections
Sep 19 04:41:49.976 - 8s    E ns/openshift-authentication route/oauth-openshift Route is not responding to GET requests on reused connections
Sep 19 04:41:49.976 - 40s   E ns/openshift-console route/console Route is not responding to GET requests on reused connections
Sep 19 04:41:49.976 - 40s   E ns/openshift-authentication route/oauth-openshift Route is not responding to GET requests over new connections
Sep 19 04:41:49.976 - 40s   E ns/openshift-console route/console Route is not responding to GET requests over new connections
Sep 19 04:41:59.317 I ns/openshift-authentication route/oauth-openshift Route started responding to GET requests on reused connections
Sep 19 04:42:24.976 E ns/openshift-authentication route/oauth-openshift Route stopped responding to GET requests on reused connections
Sep 19 04:42:25.976 - 1s    E ns/openshift-authentication route/oauth-openshift Route is not responding to GET requests on reused connections
Sep 19 04:42:28.715 I ns/openshift-authentication route/oauth-openshift Route started responding to GET requests on reused connections
Sep 19 04:42:30.784 I ns/openshift-authentication route/oauth-openshift Route started responding to GET requests over new connections
Sep 19 04:42:30.797 I ns/openshift-console route/console Route started responding to GET requests on reused connections
Sep 19 04:42:30.804 I ns/openshift-console route/console Route started responding to GET requests over new connections
Sep 19 04:43:41.359 E ns/openshift-console route/console Route stopped responding to GET requests on reused connections
Sep 19 04:43:41.722 I ns/openshift-console route/console Route started responding to GET requests on reused connections
Sep 19 04:43:48.241 E ns/openshift-authentication route/oauth-openshift Route stopped responding to GET requests over new connections
Sep 19 04:43:48.976 - 9s    E ns/openshift-authentication route/oauth-openshift Route is not responding to GET requests over new connections
Sep 19 04:43:58.976 E ns/openshift-authentication route/oauth-openshift Route stopped responding to GET requests on reused connections
Sep 19 04:43:59.323 I ns/openshift-authentication route/oauth-openshift Route started responding to GET requests on reused connections
Sep 19 04:43:59.334 I ns/openshift-authentication route/oauth-openshift Route started responding to GET requests over new connections
Sep 19 04:44:02.395 E ns/openshift-console route/console Route stopped responding to GET requests over new connections
Sep 19 04:44:02.976 - 9s    E ns/openshift-console route/console Route is not responding to GET requests over new connections
Sep 19 04:44:12.976 E ns/openshift-console route/console Route stopped responding to GET requests on reused connections
Sep 19 04:44:13.327 I ns/openshift-console route/console Route started responding to GET requests on reused connections
Sep 19 04:44:13.328 I ns/openshift-console route/console Route started responding to GET requests over new connections
Sep 19 04:44:14.320 E ns/openshift-console route/console Route stopped responding to GET requests over new connections
Sep 19 04:44:14.976 - 1s    E ns/openshift-console route/console Route is not responding to GET requests over new connections
Sep 19 04:44:16.318 I ns/openshift-console route/console Route started responding to GET requests over new connections
Sep 19 04:44:17.310 E ns/openshift-console route/console Route stopped responding to GET requests over new connections
Sep 19 04:44:17.319 E ns/openshift-console route/console Route stopped responding to GET requests on reused connections
Sep 19 04:44:17.976 - 4s    E ns/openshift-console route/console Route is not responding to GET requests on reused connections
Sep 19 04:44:17.976 - 11s   E ns/openshift-console route/console Route is not responding to GET requests over new connections
Sep 19 04:44:23.688 I ns/openshift-console route/console Route started responding to GET requests on reused connections
Sep 19 04:44:29.320 I ns/openshift-console route/console Route started responding to GET requests over new connections
Sep 19 04:44:30.307 E ns/openshift-console route/console Route stopped responding to GET requests over new connections
Sep 19 04:44:30.976 - 999ms E ns/openshift-console route/console Route is not responding to GET requests over new connections
Sep 19 04:44:32.325 I ns/openshift-console route/console Route started responding to GET requests over new connections
Sep 19 04:45:32.976 E ns/openshift-authentication route/oauth-openshift Route stopped responding to GET requests over new connections
Sep 19 04:45:32.976 E ns/openshift-authentication route/oauth-openshift Route stopped responding to GET requests on reused connections
Sep 19 04:45:33.315 I ns/openshift-authentication route/oauth-openshift Route started responding to GET requests over new connections
Sep 19 04:45:33.322 I ns/openshift-authentication route/oauth-openshift Route started responding to GET requests on reused connections
Sep 19 04:45:36.168 E ns/openshift-authentication route/oauth-openshift Route stopped responding to GET requests over new connections
Sep 19 04:45:36.976 E ns/openshift-authentication route/oauth-openshift Route is not responding to GET requests over new connections
Sep 19 04:45:37.305 I ns/openshift-authentication route/oauth-openshift Route started responding to GET requests over new connections
Sep 19 04:45:39.144 E ns/openshift-authentication route/oauth-openshift Route stopped responding to GET requests over new connections
Sep 19 04:45:39.976 - 2s    E ns/openshift-authentication route/oauth-openshift Route is not responding to GET requests over new connections
Sep 19 04:45:43.311 I ns/openshift-authentication route/oauth-openshift Route started responding to GET requests over new connections
Sep 19 04:45:44.143 E ns/openshift-authentication route/oauth-openshift Route stopped responding to GET requests over new connections
Sep 19 04:45:44.976 E ns/openshift-authentication route/oauth-openshift Route is not responding to GET requests over new connections
Sep 19 04:45:45.311 I ns/openshift-authentication route/oauth-openshift Route started responding to GET requests over new connections
Sep 19 04:45:50.143 E ns/openshift-authentication route/oauth-openshift Route stopped responding to GET requests over new connections
Sep 19 04:45:50.976 E ns/openshift-authentication route/oauth-openshift Route is not responding to GET requests over new connections
Sep 19 04:45:51.304 I ns/openshift-authentication route/oauth-openshift Route started responding to GET requests over new connections
Sep 19 04:46:42.976 E ns/openshift-console route/console Route stopped responding to GET requests on reused connections
Sep 19 04:46:43.316 I ns/openshift-console route/console Route started responding to GET requests on reused connections
Sep 19 04:56:31.976 E ns/openshift-console route/console Route stopped responding to GET requests over new connections
Sep 19 04:56:32.976 - 9s    E ns/openshift-console route/console Route is not responding to GET requests over new connections
Sep 19 04:56:32.976 E ns/openshift-console route/console Route stopped responding to GET requests on reused connections
Sep 19 04:56:33.976 - 9s    E ns/openshift-console route/console Route is not responding to GET requests on reused connections
Sep 19 04:56:34.976 E ns/openshift-authentication route/oauth-openshift Route stopped responding to GET requests over new connections
Sep 19 04:56:35.324 I ns/openshift-authentication route/oauth-openshift Route started responding to GET requests over new connections
Sep 19 04:56:43.922 I ns/openshift-console route/console Route started responding to GET requests over new connections
Sep 19 04:56:43.923 I ns/openshift-console route/console Route started responding to GET requests on reused connections
Sep 19 05:00:27.976 E ns/openshift-authentication route/oauth-openshift Route stopped responding to GET requests over new connections
Sep 19 05:00:28.976 - 9s    E ns/openshift-authentication route/oauth-openshift Route is not responding to GET requests over new connections
Sep 19 05:00:28.976 E ns/openshift-console route/console Route stopped responding to GET requests on reused connections
Sep 19 05:00:28.976 E ns/openshift-console route/console Route stopped responding to GET requests over new connections
Sep 19 05:00:29.976 - 18s   E ns/openshift-console route/console Route is not responding to GET requests on reused connections
Sep 19 05:00:29.976 - 18s   E ns/openshift-console route/console Route is not responding to GET requests over new connections
Sep 19 05:00:38.306 I ns/openshift-authentication route/oauth-openshift Route started responding to GET requests over new connections
Sep 19 05:00:49.342 I ns/openshift-console route/console Route started responding to GET requests over new connections
Sep 19 05:00:49.344 I ns/openshift-console route/console Route started responding to GET requests on reused connections
Sep 19 05:00:51.976 E ns/openshift-authentication route/oauth-openshift Route stopped responding to GET requests over new connections
Sep 19 05:00:52.976 - 9s    E ns/openshift-authentication route/oauth-openshift Route is not responding to GET requests over new connections
Sep 19 05:00:59.976 E ns/openshift-console route/console Route stopped responding to GET requests over new connections
Sep 19 05:01:00.319 I ns/openshift-console route/console Route started responding to GET requests over new connections
Sep 19 05:01:02.313 I ns/openshift-authentication route/oauth-openshift Route started responding to GET requests over new connections
Sep 19 05:04:32.976 E ns/openshift-console route/console Route stopped responding to GET requests on reused connections
Sep 19 05:04:32.976 E ns/openshift-authentication route/oauth-openshift Route stopped responding to GET requests over new connections
Sep 19 05:04:32.976 E ns/openshift-console route/console Route stopped responding to GET requests over new connections
Sep 19 05:04:32.976 E ns/openshift-authentication route/oauth-openshift Route stopped responding to GET requests on reused connections
Sep 19 05:04:33.976 - 18s   E ns/openshift-authentication route/oauth-openshift Route is not responding to GET requests on reused connections
Sep 19 05:04:33.976 - 18s   E ns/openshift-authentication route/oauth-openshift Route is not responding to GET requests over new connections
Sep 19 05:04:33.976 - 23s   E ns/openshift-console route/console Route is not responding to GET requests on reused connections
Sep 19 05:04:33.976 - 23s   E ns/openshift-console route/console Route is not responding to GET requests over new connections
Sep 19 05:04:53.328 I ns/openshift-authentication route/oauth-openshift Route started responding to GET requests on reused connections
Sep 19 05:04:53.331 I ns/openshift-authentication route/oauth-openshift Route started responding to GET requests over new connections
Sep 19 05:04:58.346 I ns/openshift-console route/console Route started responding to GET requests over new connections
Sep 19 05:04:58.347 I ns/openshift-console route/console Route started responding to GET requests on reused connections
Sep 19 05:05:03.976 E ns/openshift-authentication route/oauth-openshift Route stopped responding to GET requests over new connections
Sep 19 05:05:04.309 I ns/openshift-authentication route/oauth-openshift Route started responding to GET requests over new connections
Sep 19 05:05:13.692 E ns/openshift-console route/console Route stopped responding to GET requests over new connections
Sep 19 05:05:13.976 E ns/openshift-console route/console Route is not responding to GET requests over new connections
Sep 19 05:05:14.041 I ns/openshift-console route/console Route started responding to GET requests over new connections
Sep 19 05:05:14.976 E ns/openshift-authentication route/oauth-openshift Route stopped responding to GET requests over new connections
Sep 19 05:05:15.316 I ns/openshift-authentication route/oauth-openshift Route started responding to GET requests over new connections
Sep 19 05:05:18.540 E ns/openshift-console route/console Route stopped responding to GET requests over new connections
Sep 19 05:05:18.541 E ns/openshift-authentication route/oauth-openshift Route stopped responding to GET requests over new connections
Sep 19 05:05:18.879 I ns/openshift-console route/console Route started responding to GET requests over new connections
Sep 19 05:05:18.884 I ns/openshift-authentication route/oauth-openshift Route started responding to GET requests over new connections
				from junit_upgrade_1600493601.xml

Filter through log files


Cluster upgrade Kubernetes and OpenShift APIs remain available 44m36s

go run hack/e2e.go -v -test --test_args='--ginkgo.focus=Cluster\supgrade\sKubernetes\sand\sOpenShift\sAPIs\sremain\savailable$'
API was unreachable during disruption for at least 1m58s of 44m35s (4%):

Sep 19 04:57:58.783 I openshift-apiserver OpenShift API stopped responding to GET requests: Get https://api.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com:6443/apis/image.openshift.io/v1/namespaces/openshift-apiserver/imagestreams/missing?timeout=15s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Sep 19 04:57:58.866 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 04:58:16.289 I openshift-apiserver OpenShift API stopped responding to GET requests: Get https://api.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com:6443/apis/image.openshift.io/v1/namespaces/openshift-apiserver/imagestreams/missing?timeout=15s: unexpected EOF
Sep 19 04:58:16.782 E openshift-apiserver OpenShift API is not responding to GET requests
Sep 19 04:58:16.952 E kube-apiserver Kube API started failing: Get https://api.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com:6443/api/v1/namespaces/kube-system?timeout=15s: dial tcp 44.239.232.208:6443: connect: connection refused
Sep 19 04:58:17.116 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 04:58:17.782 E kube-apiserver Kube API is not responding to GET requests
Sep 19 04:58:17.866 I kube-apiserver Kube API started responding to GET requests
Sep 19 04:58:32.783 I openshift-apiserver OpenShift API stopped responding to GET requests: Get https://api.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com:6443/apis/image.openshift.io/v1/namespaces/openshift-apiserver/imagestreams/missing?timeout=15s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Sep 19 04:58:32.868 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:01:56.866 I openshift-apiserver OpenShift API stopped responding to GET requests: the server is currently unable to handle the request (get imagestreams.image.openshift.io missing)
Sep 19 05:01:57.782 E openshift-apiserver OpenShift API is not responding to GET requests
Sep 19 05:01:57.867 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:01:59.864 I openshift-apiserver OpenShift API stopped responding to GET requests: the server is currently unable to handle the request (get imagestreams.image.openshift.io missing)
Sep 19 05:02:00.782 E openshift-apiserver OpenShift API is not responding to GET requests
Sep 19 05:02:00.868 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:02:02.864 I openshift-apiserver OpenShift API stopped responding to GET requests: the server is currently unable to handle the request (get imagestreams.image.openshift.io missing)
Sep 19 05:02:03.782 E openshift-apiserver OpenShift API is not responding to GET requests
Sep 19 05:02:03.867 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:02:19.783 I openshift-apiserver OpenShift API stopped responding to GET requests: Get https://api.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com:6443/apis/image.openshift.io/v1/namespaces/openshift-apiserver/imagestreams/missing?timeout=15s: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Sep 19 05:02:20.782 - 15s   E openshift-apiserver OpenShift API is not responding to GET requests
Sep 19 05:02:35.866 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:02:39.312 I openshift-apiserver OpenShift API stopped responding to GET requests: the server is currently unable to handle the request (get imagestreams.image.openshift.io missing)
Sep 19 05:02:39.782 - 2s    E openshift-apiserver OpenShift API is not responding to GET requests
Sep 19 05:02:42.470 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:02:45.457 I openshift-apiserver OpenShift API stopped responding to GET requests: the server is currently unable to handle the request (get imagestreams.image.openshift.io missing)
Sep 19 05:02:45.541 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:02:48.529 I openshift-apiserver OpenShift API stopped responding to GET requests: the server is currently unable to handle the request (get imagestreams.image.openshift.io missing)
Sep 19 05:02:48.613 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:02:51.600 I openshift-apiserver OpenShift API stopped responding to GET requests: the server is currently unable to handle the request (get imagestreams.image.openshift.io missing)
Sep 19 05:02:51.782 E openshift-apiserver OpenShift API is not responding to GET requests
Sep 19 05:02:51.868 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:02:57.744 I openshift-apiserver OpenShift API stopped responding to GET requests: the server is currently unable to handle the request (get imagestreams.image.openshift.io missing)
Sep 19 05:02:57.782 E openshift-apiserver OpenShift API is not responding to GET requests
Sep 19 05:02:57.828 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:03:00.816 I openshift-apiserver OpenShift API stopped responding to GET requests: the server is currently unable to handle the request (get imagestreams.image.openshift.io missing)
Sep 19 05:03:00.902 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:03:03.888 I openshift-apiserver OpenShift API stopped responding to GET requests: the server is currently unable to handle the request (get imagestreams.image.openshift.io missing)
Sep 19 05:03:04.782 E openshift-apiserver OpenShift API is not responding to GET requests
Sep 19 05:03:04.867 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:03:06.858 I openshift-apiserver OpenShift API stopped responding to GET requests: Get https://api.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com:6443/apis/image.openshift.io/v1/namespaces/openshift-apiserver/imagestreams/missing?timeout=15s: http2: server sent GOAWAY and closed the connection; LastStreamID=1121, ErrCode=NO_ERROR, debug=""
Sep 19 05:03:07.782 - 14s   E openshift-apiserver OpenShift API is not responding to GET requests
Sep 19 05:03:21.944 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:03:38.783 I openshift-apiserver OpenShift API stopped responding to GET requests: Get https://api.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com:6443/apis/image.openshift.io/v1/namespaces/openshift-apiserver/imagestreams/missing?timeout=15s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Sep 19 05:03:39.782 - 13s   E openshift-apiserver OpenShift API is not responding to GET requests
Sep 19 05:03:53.868 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:04:15.783 I openshift-apiserver OpenShift API stopped responding to GET requests: Get https://api.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com:6443/apis/image.openshift.io/v1/namespaces/openshift-apiserver/imagestreams/missing?timeout=15s: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Sep 19 05:04:16.782 - 13s   E openshift-apiserver OpenShift API is not responding to GET requests
Sep 19 05:04:30.869 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:06:38.783 I openshift-apiserver OpenShift API stopped responding to GET requests: Get https://api.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com:6443/apis/image.openshift.io/v1/namespaces/openshift-apiserver/imagestreams/missing?timeout=15s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Sep 19 05:06:38.867 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:06:55.782 I openshift-apiserver OpenShift API stopped responding to GET requests: Get https://api.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com:6443/apis/image.openshift.io/v1/namespaces/openshift-apiserver/imagestreams/missing?timeout=15s: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Sep 19 05:06:55.867 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:07:02.421 I openshift-apiserver OpenShift API stopped responding to GET requests: the server is currently unable to handle the request (get imagestreams.image.openshift.io missing)
Sep 19 05:07:02.782 E openshift-apiserver OpenShift API is not responding to GET requests
Sep 19 05:07:02.867 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:07:08.565 I openshift-apiserver OpenShift API stopped responding to GET requests: the server is currently unable to handle the request (get imagestreams.image.openshift.io missing)
Sep 19 05:07:08.782 - 2s    E openshift-apiserver OpenShift API is not responding to GET requests
Sep 19 05:07:11.722 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:07:17.781 I openshift-apiserver OpenShift API stopped responding to GET requests: the server is currently unable to handle the request (get imagestreams.image.openshift.io missing)
Sep 19 05:07:17.782 - 3s    E openshift-apiserver OpenShift API is not responding to GET requests
Sep 19 05:07:20.938 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:07:26.997 I openshift-apiserver OpenShift API stopped responding to GET requests: the server is currently unable to handle the request (get imagestreams.image.openshift.io missing)
Sep 19 05:07:27.782 - 5s    E openshift-apiserver OpenShift API is not responding to GET requests
Sep 19 05:07:33.226 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:07:36.213 I openshift-apiserver OpenShift API stopped responding to GET requests: the server is currently unable to handle the request (get imagestreams.image.openshift.io missing)
Sep 19 05:07:36.782 E openshift-apiserver OpenShift API is not responding to GET requests
Sep 19 05:07:36.867 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:07:39.286 I openshift-apiserver OpenShift API stopped responding to GET requests: the server is currently unable to handle the request (get imagestreams.image.openshift.io missing)
Sep 19 05:07:39.782 E openshift-apiserver OpenShift API is not responding to GET requests
Sep 19 05:07:39.867 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:07:42.357 I openshift-apiserver OpenShift API stopped responding to GET requests: the server is currently unable to handle the request (get imagestreams.image.openshift.io missing)
Sep 19 05:07:42.442 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:07:45.430 I openshift-apiserver OpenShift API stopped responding to GET requests: the server is currently unable to handle the request (get imagestreams.image.openshift.io missing)
Sep 19 05:07:45.782 - 24s   E openshift-apiserver OpenShift API is not responding to GET requests
Sep 19 05:08:10.090 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:08:13.077 I openshift-apiserver OpenShift API stopped responding to GET requests: the server is currently unable to handle the request (get imagestreams.image.openshift.io missing)
Sep 19 05:08:13.782 - 4s    E openshift-apiserver OpenShift API is not responding to GET requests
Sep 19 05:08:19.306 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:08:22.293 I openshift-apiserver OpenShift API stopped responding to GET requests: the server is currently unable to handle the request (get imagestreams.image.openshift.io missing)
Sep 19 05:08:22.782 - 4s    E openshift-apiserver OpenShift API is not responding to GET requests
Sep 19 05:08:28.521 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:08:31.510 I openshift-apiserver OpenShift API stopped responding to GET requests: the server is currently unable to handle the request (get imagestreams.image.openshift.io missing)
Sep 19 05:08:31.595 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:08:34.581 I openshift-apiserver OpenShift API stopped responding to GET requests: the server is currently unable to handle the request (get imagestreams.image.openshift.io missing)
Sep 19 05:08:34.666 I openshift-apiserver OpenShift API started responding to GET requests
Sep 19 05:08:37.653 I openshift-apiserver OpenShift API stopped responding to GET requests: the server is currently unable to handle the request (get imagestreams.image.openshift.io missing)
Sep 19 05:08:37.782 - 2s    E openshift-apiserver OpenShift API is not responding to GET requests
Sep 19 05:08:40.810 I openshift-apiserver OpenShift API started responding to GET requests
				from junit_upgrade_1600493601.xml

Filter through log files


openshift-tests Monitor cluster while tests execute 1h4m

go run hack/e2e.go -v -test --test_args='--ginkgo.focus=openshift\-tests\sMonitor\scluster\swhile\stests\sexecute$'
218 error level events were detected during this test run:

Sep 19 04:32:56.409 E ns/openshift-kube-apiserver-operator pod/kube-apiserver-operator-7f668f7cd9-8kh5z node/ip-10-0-145-20.us-west-2.compute.internal container=kube-apiserver-operator container exited with code 255 (Error): go:270] github.com/openshift/client-go/config/informers/externalversions/factory.go:101: watch of *v1.ClusterOperator ended with: too old resource version: 16770 (17139)\nW0919 04:32:52.339765       1 reflector.go:270] github.com/openshift/client-go/config/informers/externalversions/factory.go:101: watch of *v1.APIServer ended with: too old resource version: 14809 (17246)\nW0919 04:32:52.606121       1 reflector.go:270] github.com/openshift/client-go/config/informers/externalversions/factory.go:101: watch of *v1.Authentication ended with: too old resource version: 14807 (17249)\nW0919 04:32:53.196997       1 reflector.go:270] github.com/openshift/client-go/config/informers/externalversions/factory.go:101: watch of *v1.FeatureGate ended with: too old resource version: 14809 (17255)\nW0919 04:32:53.336005       1 reflector.go:270] github.com/openshift/client-go/config/informers/externalversions/factory.go:101: watch of *v1.Image ended with: too old resource version: 14809 (17256)\nW0919 04:32:53.675612       1 reflector.go:270] github.com/openshift/client-go/config/informers/externalversions/factory.go:101: watch of *v1.Infrastructure ended with: too old resource version: 14810 (17261)\nW0919 04:32:53.942247       1 reflector.go:270] github.com/openshift/client-go/config/informers/externalversions/factory.go:101: watch of *v1.Network ended with: too old resource version: 14810 (17264)\nW0919 04:32:54.521375       1 reflector.go:270] github.com/openshift/client-go/config/informers/externalversions/factory.go:101: watch of *v1.Scheduler ended with: too old resource version: 14809 (17271)\nW0919 04:32:54.748471       1 reflector.go:270] github.com/openshift/client-go/operator/informers/externalversions/factory.go:101: watch of *v1.KubeAPIServer ended with: too old resource version: 15385 (17272)\nI0919 04:32:55.332628       1 cmd.go:78] Received SIGTERM or SIGINT signal, shutting down controller.\nF0919 04:32:55.332778       1 leaderelection.go:65] leaderelection lost\nF0919 04:32:55.483787       1 builder.go:217] server exited\n
Sep 19 04:36:05.491 E clusteroperator/cloud-credential changed Degraded to True: CredentialsFailing: 1 of 8 credentials requests are failing to sync.
Sep 19 04:36:26.624 E ns/openshift-apiserver-operator pod/openshift-apiserver-operator-6b5dd676c9-lrs2m node/ip-10-0-145-20.us-west-2.compute.internal container=openshift-apiserver-operator container exited with code 255 (Error): osed the connection; LastStreamID=1495, ErrCode=NO_ERROR, debug=""\nW0919 04:36:23.524024       1 reflector.go:270] k8s.io/client-go/informers/factory.go:132: watch of *v1.Service ended with: too old resource version: 14639 (15299)\nW0919 04:36:23.524078       1 reflector.go:270] k8s.io/client-go/informers/factory.go:132: watch of *v1.Pod ended with: too old resource version: 14662 (15299)\nW0919 04:36:23.524725       1 reflector.go:270] k8s.io/client-go/informers/factory.go:132: watch of *v1.ConfigMap ended with: too old resource version: 17585 (18476)\nW0919 04:36:23.524786       1 reflector.go:270] k8s.io/client-go/informers/factory.go:132: watch of *v1.ConfigMap ended with: too old resource version: 16967 (18476)\nW0919 04:36:23.524840       1 reflector.go:270] k8s.io/client-go/informers/factory.go:132: watch of *v1.ConfigMap ended with: too old resource version: 17270 (18476)\nW0919 04:36:23.524917       1 reflector.go:270] k8s.io/client-go/informers/factory.go:132: watch of *v1.DaemonSet ended with: too old resource version: 14117 (15302)\nW0919 04:36:23.525915       1 reflector.go:270] github.com/openshift/client-go/config/informers/externalversions/factory.go:101: watch of *v1.Image ended with: too old resource version: 17256 (17731)\nW0919 04:36:23.702372       1 reflector.go:270] github.com/openshift/client-go/config/informers/externalversions/factory.go:101: watch of *v1.Ingress ended with: too old resource version: 17263 (18757)\nW0919 04:36:23.792701       1 reflector.go:270] github.com/openshift/client-go/config/informers/externalversions/factory.go:101: watch of *v1.Project ended with: too old resource version: 17269 (18759)\nW0919 04:36:23.819653       1 reflector.go:270] github.com/openshift/client-go/operator/informers/externalversions/factory.go:101: watch of *v1.OpenShiftAPIServer ended with: too old resource version: 18449 (18759)\nI0919 04:36:24.885469       1 cmd.go:78] Received SIGTERM or SIGINT signal, shutting down controller.\nF0919 04:36:24.885546       1 leaderelection.go:65] leaderelection lost\n
Sep 19 04:36:28.223 E ns/openshift-machine-api pod/machine-api-operator-59b4994479-fpwjw node/ip-10-0-145-20.us-west-2.compute.internal container=machine-api-operator container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:36:35.426 E ns/openshift-machine-api pod/machine-api-controllers-75df6f4c8-m4sjm node/ip-10-0-129-244.us-west-2.compute.internal container=controller-manager container exited with code 1 (Error): 
Sep 19 04:36:35.426 E ns/openshift-machine-api pod/machine-api-controllers-75df6f4c8-m4sjm node/ip-10-0-129-244.us-west-2.compute.internal container=nodelink-controller container exited with code 2 (Error): 
Sep 19 04:39:50.283 E ns/openshift-cluster-machine-approver pod/machine-approver-84875498d8-xd8rm node/ip-10-0-145-20.us-west-2.compute.internal container=machine-approver-controller container exited with code 2 (Error): ts?resourceVersion=14116&timeoutSeconds=319&watch=true: dial tcp 127.0.0.1:6443: connect: connection refused\nE0919 04:36:24.270236       1 reflector.go:205] github.com/openshift/cluster-machine-approver/main.go:185: Failed to list *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: dial tcp 127.0.0.1:6443: connect: connection refused\nE0919 04:36:25.271703       1 reflector.go:205] github.com/openshift/cluster-machine-approver/main.go:185: Failed to list *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: dial tcp 127.0.0.1:6443: connect: connection refused\nE0919 04:36:29.220427       1 reflector.go:205] github.com/openshift/cluster-machine-approver/main.go:185: Failed to list *v1beta1.CertificateSigningRequest: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:serviceaccount:openshift-cluster-machine-approver:machine-approver-sa" cannot list resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope\nE0919 04:37:52.713686       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=NO_ERROR, debug=""\nE0919 04:37:52.714041       1 reflector.go:322] github.com/openshift/cluster-machine-approver/main.go:185: Failed to watch *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?resourceVersion=18809&timeoutSeconds=329&watch=true: dial tcp 127.0.0.1:6443: connect: connection refused\nE0919 04:37:53.714454       1 reflector.go:205] github.com/openshift/cluster-machine-approver/main.go:185: Failed to list *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: dial tcp 127.0.0.1:6443: connect: connection refused\n
Sep 19 04:40:05.170 E ns/openshift-authentication pod/oauth-openshift-6f974c5767-f59c7 node/ip-10-0-145-20.us-west-2.compute.internal container=oauth-openshift container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:40:07.523 E ns/openshift-monitoring pod/node-exporter-z8584 node/ip-10-0-138-182.us-west-2.compute.internal container=node-exporter container exited with code 143 (Error): 
Sep 19 04:40:16.218 E ns/openshift-cluster-node-tuning-operator pod/tuned-dwdzg node/ip-10-0-158-47.us-west-2.compute.internal container=tuned container exited with code 143 (Error):  Resync period to pull node/pod labels: 116 [s]\nI0919 04:28:34.515263    2400 openshift-tuned.go:435] Pod (openshift-monitoring/prometheus-k8s-0) labels changed node wide: true\nI0919 04:28:39.509499    2400 openshift-tuned.go:293] Dumping labels to /var/lib/tuned/ocp-pod-labels.cfg\nI0919 04:28:39.512249    2400 openshift-tuned.go:275] Dumping labels to /var/lib/tuned/ocp-node-labels.cfg\nI0919 04:28:39.514659    2400 openshift-tuned.go:326] Getting recommended profile...\nI0919 04:28:39.628557    2400 openshift-tuned.go:523] Active and recommended profile (openshift-node) match.  Label changes will not trigger profile reload.\nI0919 04:28:45.689221    2400 openshift-tuned.go:435] Pod (e2e-k8s-sig-apps-daemonset-upgrade-7990/ds1-86sk2) labels changed node wide: true\nI0919 04:28:49.509545    2400 openshift-tuned.go:293] Dumping labels to /var/lib/tuned/ocp-pod-labels.cfg\nI0919 04:28:49.511726    2400 openshift-tuned.go:326] Getting recommended profile...\nI0919 04:28:49.625991    2400 openshift-tuned.go:523] Active and recommended profile (openshift-node) match.  Label changes will not trigger profile reload.\nI0919 04:30:30.503503    2400 openshift-tuned.go:691] Lowering resyncPeriod to 58\nI0919 04:35:24.571479    2400 openshift-tuned.go:435] Pod (openshift-monitoring/prometheus-operator-89569dd85-2gjhj) labels changed node wide: true\nI0919 04:35:29.509590    2400 openshift-tuned.go:293] Dumping labels to /var/lib/tuned/ocp-pod-labels.cfg\nI0919 04:35:29.511314    2400 openshift-tuned.go:326] Getting recommended profile...\nI0919 04:35:29.624851    2400 openshift-tuned.go:523] Active and recommended profile (openshift-node) match.  Label changes will not trigger profile reload.\nI0919 04:39:14.509504    2400 openshift-tuned.go:275] Dumping labels to /var/lib/tuned/ocp-node-labels.cfg\nI0919 04:39:14.510792    2400 openshift-tuned.go:326] Getting recommended profile...\nI0919 04:39:14.631313    2400 openshift-tuned.go:523] Active and recommended profile (openshift-node) match.  Label changes will not trigger profile reload.\n
Sep 19 04:40:16.405 E ns/openshift-cluster-node-tuning-operator pod/tuned-9b6q9 node/ip-10-0-130-52.us-west-2.compute.internal container=tuned container exited with code 143 (Error): penshift-kube-scheduler/installer-6-ip-10-0-130-52.us-west-2.compute.internal) labels changed node wide: false\nI0919 04:38:54.890590   12319 openshift-tuned.go:435] Pod (openshift-apiserver/apiserver-hvh2r) labels changed node wide: true\nI0919 04:38:56.499687   12319 openshift-tuned.go:293] Dumping labels to /var/lib/tuned/ocp-pod-labels.cfg\nI0919 04:38:56.501101   12319 openshift-tuned.go:326] Getting recommended profile...\nI0919 04:38:56.605093   12319 openshift-tuned.go:523] Active and recommended profile (openshift-control-plane) match.  Label changes will not trigger profile reload.\nI0919 04:38:57.163214   12319 openshift-tuned.go:435] Pod (openshift-kube-controller-manager/revision-pruner-8-ip-10-0-130-52.us-west-2.compute.internal) labels changed node wide: false\nI0919 04:38:58.689809   12319 openshift-tuned.go:435] Pod (openshift-kube-scheduler/openshift-kube-scheduler-ip-10-0-130-52.us-west-2.compute.internal) labels changed node wide: true\nI0919 04:39:01.499686   12319 openshift-tuned.go:293] Dumping labels to /var/lib/tuned/ocp-pod-labels.cfg\nI0919 04:39:01.501137   12319 openshift-tuned.go:326] Getting recommended profile...\nI0919 04:39:01.619104   12319 openshift-tuned.go:523] Active and recommended profile (openshift-control-plane) match.  Label changes will not trigger profile reload.\nI0919 04:39:16.499692   12319 openshift-tuned.go:275] Dumping labels to /var/lib/tuned/ocp-node-labels.cfg\nI0919 04:39:16.500757   12319 openshift-tuned.go:326] Getting recommended profile...\nI0919 04:39:16.601733   12319 openshift-tuned.go:523] Active and recommended profile (openshift-control-plane) match.  Label changes will not trigger profile reload.\nE0919 04:39:38.005974   12319 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=17, ErrCode=NO_ERROR, debug=""\nE0919 04:39:38.008630   12319 openshift-tuned.go:720] Pod event watch channel closed.\nI0919 04:39:38.008720   12319 openshift-tuned.go:722] Increasing resyncPeriod to 130\n
Sep 19 04:40:16.554 E ns/openshift-cluster-node-tuning-operator pod/tuned-bn9g9 node/ip-10-0-138-182.us-west-2.compute.internal container=tuned container exited with code 143 (Error): 791897    2357 openshift-tuned.go:523] Active and recommended profile (openshift-node) match.  Label changes will not trigger profile reload.\nI0919 04:29:01.300598    2357 openshift-tuned.go:435] Pod (e2e-k8s-sig-storage-sig-api-machinery-secret-upgrade-3176/pod-secrets-f0a90273-9bca-4ad1-83be-943571b65214) labels changed node wide: false\nI0919 04:29:12.230237    2357 openshift-tuned.go:435] Pod (e2e-k8s-sig-storage-sig-api-machinery-secret-upgrade-3176/pod-secrets-f0a90273-9bca-4ad1-83be-943571b65214) labels changed node wide: false\nI0919 04:32:51.623974    2357 openshift-tuned.go:691] Lowering resyncPeriod to 120\nI0919 04:34:51.628521    2357 openshift-tuned.go:691] Lowering resyncPeriod to 60\nI0919 04:35:11.295131    2357 openshift-tuned.go:435] Pod (openshift-monitoring/prometheus-operator-85dd8bccf6-57wnx) labels changed node wide: true\nI0919 04:35:11.627971    2357 openshift-tuned.go:293] Dumping labels to /var/lib/tuned/ocp-pod-labels.cfg\nI0919 04:35:11.630162    2357 openshift-tuned.go:326] Getting recommended profile...\nI0919 04:35:11.792113    2357 openshift-tuned.go:523] Active and recommended profile (openshift-node) match.  Label changes will not trigger profile reload.\nI0919 04:39:56.628020    2357 openshift-tuned.go:275] Dumping labels to /var/lib/tuned/ocp-node-labels.cfg\nI0919 04:39:56.629465    2357 openshift-tuned.go:326] Getting recommended profile...\nI0919 04:39:56.741383    2357 openshift-tuned.go:523] Active and recommended profile (openshift-node) match.  Label changes will not trigger profile reload.\nI0919 04:40:07.068869    2357 openshift-tuned.go:435] Pod (openshift-monitoring/prometheus-operator-85dd8bccf6-57wnx) labels changed node wide: true\nI0919 04:40:11.628045    2357 openshift-tuned.go:293] Dumping labels to /var/lib/tuned/ocp-pod-labels.cfg\nI0919 04:40:11.630238    2357 openshift-tuned.go:326] Getting recommended profile...\nI0919 04:40:11.768194    2357 openshift-tuned.go:523] Active and recommended profile (openshift-node) match.  Label changes will not trigger profile reload.\n
Sep 19 04:40:21.383 E ns/openshift-cluster-node-tuning-operator pod/tuned-s2tfg node/ip-10-0-145-20.us-west-2.compute.internal container=tuned container exited with code 143 (Error): hanged node wide: true\nI0919 04:35:15.365261   18780 openshift-tuned.go:293] Dumping labels to /var/lib/tuned/ocp-pod-labels.cfg\nI0919 04:35:15.366548   18780 openshift-tuned.go:326] Getting recommended profile...\nI0919 04:35:15.463765   18780 openshift-tuned.go:523] Active and recommended profile (openshift-control-plane) match.  Label changes will not trigger profile reload.\nI0919 04:36:10.609397   18780 openshift-tuned.go:435] Pod (openshift-kube-scheduler/installer-6-ip-10-0-145-20.us-west-2.compute.internal) labels changed node wide: false\nE0919 04:36:23.260099   18780 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=23, ErrCode=NO_ERROR, debug=""\nE0919 04:36:23.267700   18780 openshift-tuned.go:720] Pod event watch channel closed.\nI0919 04:36:23.267783   18780 openshift-tuned.go:722] Increasing resyncPeriod to 116\nI0919 04:38:19.267961   18780 openshift-tuned.go:187] Extracting tuned profiles\nI0919 04:38:19.270056   18780 openshift-tuned.go:623] Resync period to pull node/pod labels: 116 [s]\nI0919 04:38:19.298842   18780 openshift-tuned.go:435] Pod (openshift-sdn/ovs-87x4r) labels changed node wide: true\nI0919 04:38:24.291503   18780 openshift-tuned.go:293] Dumping labels to /var/lib/tuned/ocp-pod-labels.cfg\nI0919 04:38:24.292752   18780 openshift-tuned.go:275] Dumping labels to /var/lib/tuned/ocp-node-labels.cfg\nI0919 04:38:24.293650   18780 openshift-tuned.go:326] Getting recommended profile...\nI0919 04:38:24.395736   18780 openshift-tuned.go:523] Active and recommended profile (openshift-control-plane) match.  Label changes will not trigger profile reload.\nE0919 04:39:37.999630   18780 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=3, ErrCode=NO_ERROR, debug=""\nE0919 04:39:38.003573   18780 openshift-tuned.go:720] Pod event watch channel closed.\nI0919 04:39:38.003644   18780 openshift-tuned.go:722] Increasing resyncPeriod to 232\n
Sep 19 04:40:21.391 E ns/openshift-image-registry pod/image-registry-5dd85f7db5-nvgjk node/ip-10-0-158-47.us-west-2.compute.internal container=registry container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:40:21.670 E ns/openshift-image-registry pod/image-registry-5dd85f7db5-64z9p node/ip-10-0-129-73.us-west-2.compute.internal container=registry container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:40:23.471 E ns/openshift-image-registry pod/image-registry-5dd85f7db5-4fdks node/ip-10-0-129-73.us-west-2.compute.internal container=registry container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:40:23.928 E ns/openshift-image-registry pod/image-registry-5dd85f7db5-trz2p node/ip-10-0-158-47.us-west-2.compute.internal container=registry container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:40:25.684 E ns/openshift-image-registry pod/image-registry-5dd85f7db5-fx5qh node/ip-10-0-158-47.us-west-2.compute.internal container=registry container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:40:30.046 E ns/openshift-image-registry pod/image-registry-5dd85f7db5-xfwgm node/ip-10-0-138-182.us-west-2.compute.internal container=registry container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:40:30.456 E ns/openshift-image-registry pod/image-registry-5dd85f7db5-qltw2 node/ip-10-0-129-73.us-west-2.compute.internal container=registry container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:40:30.632 E ns/openshift-image-registry pod/image-registry-5dd85f7db5-c499k node/ip-10-0-158-47.us-west-2.compute.internal container=registry container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:40:39.716 E ns/openshift-service-ca-operator pod/service-ca-operator-bfd59dd96-nfsr4 node/ip-10-0-145-20.us-west-2.compute.internal container=operator container exited with code 2 (Error): 
Sep 19 04:40:51.749 E ns/openshift-console-operator pod/console-operator-5bb677dfbb-9sgwl node/ip-10-0-145-20.us-west-2.compute.internal container=console-operator container exited with code 255 (Error): ith: too old resource version: 18753 (20872)\ntime="2020-09-19T04:39:52Z" level=info msg="started syncing operator \"cluster\" (2020-09-19 04:39:52.906863605 +0000 UTC m=+1192.614903273)"\ntime="2020-09-19T04:39:52Z" level=info msg="console is in a managed state."\ntime="2020-09-19T04:39:52Z" level=info msg="running sync loop 4.0.0"\ntime="2020-09-19T04:39:52Z" level=info msg="route ingress 'default' found and admitted, host: console-openshift-console.apps.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com \n"\ntime="2020-09-19T04:39:52Z" level=info msg="service-ca configmap exists and is in the correct state"\ntime="2020-09-19T04:39:52Z" level=info msg="route ingress 'default' found and admitted, host: console-openshift-console.apps.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com \n"\ntime="2020-09-19T04:39:53Z" level=info msg=-----------------------\ntime="2020-09-19T04:39:53Z" level=info msg="sync loop 4.0.0 resources updated: false \n"\ntime="2020-09-19T04:39:53Z" level=info msg=-----------------------\ntime="2020-09-19T04:39:53Z" level=info msg="deployment is available, ready replicas: 2 \n"\ntime="2020-09-19T04:39:53Z" level=info msg="sync_v400: updating console status"\ntime="2020-09-19T04:39:53Z" level=info msg="route ingress 'default' found and admitted, host: console-openshift-console.apps.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com \n"\ntime="2020-09-19T04:39:53Z" level=info msg="sync loop 4.0.0 complete"\ntime="2020-09-19T04:39:53Z" level=info msg="finished syncing operator \"cluster\" (108.295µs) \n\n"\nI0919 04:40:51.294002       1 observer_polling.go:78] Observed change: file:/var/run/configmaps/config/controller-config.yaml (current: "41adc4f67c9486b2d108e39abc8b009b458b16fccaa6c860984b7a3410299dff", lastKnown: "9a9f171084db3a5d8481509ac5a988ce2000ef99d0a85c83a40458eb32dc4bbc")\nW0919 04:40:51.294035       1 builder.go:108] Restart triggered because of file /var/run/configmaps/config/controller-config.yaml was modified\nF0919 04:40:51.294077       1 leaderelection.go:65] leaderelection lost\n
Sep 19 04:41:23.969 E ns/openshift-controller-manager pod/controller-manager-zxdmn node/ip-10-0-145-20.us-west-2.compute.internal container=controller-manager container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:41:27.743 E kube-apiserver failed contacting the API: Get https://api.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com:6443/apis/config.openshift.io/v1/clusterversions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dversion&resourceVersion=25564&timeout=5m31s&timeoutSeconds=331&watch=true: dial tcp 44.237.26.173:6443: connect: connection refused
Sep 19 04:41:28.736 E ns/openshift-service-catalog-apiserver-operator pod/openshift-service-catalog-apiserver-operator-84889bc57d-wlzsx node/ip-10-0-130-52.us-west-2.compute.internal container=operator container exited with code 2 (Error): .30.0.1:443/api/v1/namespaces/openshift-config/secrets?limit=500&resourceVersion=0\nI0919 04:39:39.741633       1 request.go:530] Throttling request took 577.457873ms, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver\nI0919 04:39:39.938720       1 request.go:530] Throttling request took 696.982716ms, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver-operator/secrets?limit=500&resourceVersion=0\nI0919 04:39:40.138744       1 request.go:530] Throttling request took 384.630691ms, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-service-catalog-apiserver\nI0919 04:39:41.466703       1 leaderelection.go:245] successfully renewed lease openshift-service-catalog-apiserver-operator/openshift-cluster-svcat-apiserver-operator-lock\nI0919 04:39:44.568420       1 reflector.go:357] github.com/openshift/client-go/operator/informers/externalversions/factory.go:101: Watch close - *v1.ServiceCatalogAPIServer total 0 items received\nW0919 04:39:44.580569       1 reflector.go:270] github.com/openshift/client-go/operator/informers/externalversions/factory.go:101: watch of *v1.ServiceCatalogAPIServer ended with: too old resource version: 20251 (20678)\nI0919 04:39:45.585864       1 reflector.go:169] Listing and watching *v1.ServiceCatalogAPIServer from github.com/openshift/client-go/operator/informers/externalversions/factory.go:101\nI0919 04:39:51.482385       1 leaderelection.go:245] successfully renewed lease openshift-service-catalog-apiserver-operator/openshift-cluster-svcat-apiserver-operator-lock\nI0919 04:40:01.502193       1 leaderelection.go:245] successfully renewed lease openshift-service-catalog-apiserver-operator/openshift-cluster-svcat-apiserver-operator-lock\nI0919 04:40:10.519046       1 reflector.go:215] k8s.io/client-go/informers/factory.go:132: forcing resync\nI0919 04:40:11.530066       1 leaderelection.go:245] successfully renewed lease openshift-service-catalog-apiserver-operator/openshift-cluster-svcat-apiserver-operator-lock\n
Sep 19 04:41:33.115 E ns/openshift-authentication-operator pod/authentication-operator-6498b57c5-4rmz8 node/ip-10-0-130-52.us-west-2.compute.internal container=operator container exited with code 255 (Error): ason":"AsExpected","status":"True","type":"Upgradeable"}]}}\nE0919 04:40:14.837797       1 controller.go:129] {AuthenticationOperator2 AuthenticationOperator2} failed with: error checking current version: unable to check route health: failed to GET route: net/http: TLS handshake timeout\nI0919 04:40:14.846122       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"cd86b7d7-fa2e-11ea-8901-02f26ee80463", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "" to "RouteHealthDegraded: failed to GET route: net/http: TLS handshake timeout"\nI0919 04:40:16.093690       1 status_controller.go:164] clusteroperator/authentication diff {"status":{"conditions":[{"lastTransitionTime":"2020-09-19T04:23:23Z","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-09-19T04:39:52Z","message":"Progressing: not all deployment replicas are ready","reason":"ProgressingOAuthServerDeploymentNotReady","status":"True","type":"Progressing"},{"lastTransitionTime":"2020-09-19T04:27:35Z","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2020-09-19T04:20:10Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}\nI0919 04:40:16.105491       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"cd86b7d7-fa2e-11ea-8901-02f26ee80463", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "RouteHealthDegraded: failed to GET route: net/http: TLS handshake timeout" to ""\nI0919 04:40:27.785508       1 cmd.go:78] Received SIGTERM or SIGINT signal, shutting down controller.\nF0919 04:40:27.785562       1 leaderelection.go:65] leaderelection lost\n
Sep 19 04:41:35.322 E ns/openshift-console pod/downloads-5c59f467dc-67krp node/ip-10-0-130-52.us-west-2.compute.internal container=download-server container exited with code 137 (Error): 
Sep 19 04:41:35.926 E ns/openshift-monitoring pod/node-exporter-zxb4g node/ip-10-0-130-52.us-west-2.compute.internal container=node-exporter container exited with code 143 (Error): 
Sep 19 04:41:37.913 E ns/openshift-operator-lifecycle-manager pod/olm-operators-g5mw7 node/ip-10-0-130-52.us-west-2.compute.internal container=configmap-registry-server container exited with code 2 (Error): 
Sep 19 04:41:39.507 E ns/openshift-service-ca pod/service-serving-cert-signer-865648474d-866c5 node/ip-10-0-129-244.us-west-2.compute.internal container=service-serving-cert-signer-controller container exited with code 2 (Error): 
Sep 19 04:41:39.521 E ns/openshift-service-ca pod/apiservice-cabundle-injector-7b45d6d55b-mk5zg node/ip-10-0-129-244.us-west-2.compute.internal container=apiservice-cabundle-injector-controller container exited with code 2 (Error): 
Sep 19 04:41:47.258 E ns/openshift-monitoring pod/prometheus-adapter-6c4d49d7b4-g5w4p node/ip-10-0-158-47.us-west-2.compute.internal container=prometheus-adapter container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:41:57.859 E ns/openshift-monitoring pod/node-exporter-xrkmm node/ip-10-0-158-47.us-west-2.compute.internal container=node-exporter container exited with code 143 (Error): 
Sep 19 04:41:59.054 E ns/openshift-marketplace pod/redhat-operators-865d988bb8-8ktcp node/ip-10-0-138-182.us-west-2.compute.internal container=redhat-operators container exited with code 2 (Error): 
Sep 19 04:42:06.181 E ns/openshift-console-operator pod/console-operator-5bb677dfbb-9sgwl node/ip-10-0-145-20.us-west-2.compute.internal container=console-operator container exited with code 255 (Error): came leader\nI0919 04:42:04.823336       1 unsupportedconfigoverrides_controller.go:151] Starting UnsupportedConfigOverridesController\nI0919 04:42:04.825700       1 status_controller.go:187] Starting StatusSyncer-console\nI0919 04:42:04.827096       1 controller.go:54] Starting Console\ntime="2020-09-19T04:42:04Z" level=info msg="started syncing operator \"cluster\" (2020-09-19 04:42:04.933068475 +0000 UTC m=+72.955801730)"\ntime="2020-09-19T04:42:04Z" level=info msg="console is in a managed state."\ntime="2020-09-19T04:42:04Z" level=info msg="running sync loop 4.0.0"\ntime="2020-09-19T04:42:04Z" level=info msg="route ingress 'default' found and admitted, host: console-openshift-console.apps.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com \n"\ntime="2020-09-19T04:42:04Z" level=info msg="service-ca configmap exists and is in the correct state"\ntime="2020-09-19T04:42:04Z" level=info msg="route ingress 'default' found and admitted, host: console-openshift-console.apps.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com \n"\ntime="2020-09-19T04:42:04Z" level=info msg=-----------------------\ntime="2020-09-19T04:42:04Z" level=info msg="sync loop 4.0.0 resources updated: false \n"\ntime="2020-09-19T04:42:04Z" level=info msg=-----------------------\ntime="2020-09-19T04:42:04Z" level=info msg="deployment is available, ready replicas: 2 \n"\ntime="2020-09-19T04:42:04Z" level=info msg="sync_v400: updating console status"\ntime="2020-09-19T04:42:04Z" level=info msg="route ingress 'default' found and admitted, host: console-openshift-console.apps.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com \n"\ntime="2020-09-19T04:42:04Z" level=info msg="sync loop 4.0.0 complete"\ntime="2020-09-19T04:42:04Z" level=info msg="finished syncing operator \"cluster\" (38.472µs) \n\n"\nI0919 04:42:05.232589       1 cmd.go:78] Received SIGTERM or SIGINT signal, shutting down controller.\nF0919 04:42:05.232725       1 leaderelection.go:65] leaderelection lost\nI0919 04:42:05.232963       1 secure_serving.go:156] Stopped listening on 0.0.0.0:8443\n
Sep 19 04:42:16.207 E ns/openshift-console pod/downloads-5c59f467dc-s57n9 node/ip-10-0-145-20.us-west-2.compute.internal container=download-server container exited with code 137 (Error): 
Sep 19 04:43:06.388 E ns/openshift-image-registry pod/image-registry-65f679c459-t9smp node/ip-10-0-158-47.us-west-2.compute.internal container=registry container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:43:14.572 E clusteroperator/monitoring changed Degraded to True: UpdatingPrometheusOperatorFailed: Failed to rollout the stack. Error: running task Updating Prometheus Operator failed: reconciling Prometheus Operator Deployment failed: updating deployment object failed: Deployment.apps "prometheus-operator" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/component":"controller", "app.kubernetes.io/name":"prometheus-operator"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
Sep 19 04:43:16.421 E ns/openshift-authentication pod/oauth-openshift-8588456875-zzjnn node/ip-10-0-130-52.us-west-2.compute.internal container=oauth-openshift container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:43:21.470 E ns/openshift-controller-manager pod/controller-manager-lllsm node/ip-10-0-130-52.us-west-2.compute.internal container=controller-manager container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:43:47.758 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ip-10-0-138-182.us-west-2.compute.internal container=kube-rbac-proxy container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:43:47.758 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ip-10-0-138-182.us-west-2.compute.internal container=prometheus container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:43:47.758 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ip-10-0-138-182.us-west-2.compute.internal container=rules-configmap-reloader container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:43:47.758 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ip-10-0-138-182.us-west-2.compute.internal container=prometheus-config-reloader container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:43:47.758 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ip-10-0-138-182.us-west-2.compute.internal container=prometheus-proxy container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:43:47.758 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ip-10-0-138-182.us-west-2.compute.internal container=prom-label-proxy container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:43:53.050 E ns/openshift-kube-controller-manager-operator pod/kube-controller-manager-operator-9755bcd89-wp2zn node/ip-10-0-145-20.us-west-2.compute.internal container=kube-controller-manager-operator container exited with code 255 (Error): I0919 04:43:39.809826       1 event.go:209] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-controller-manager-operator", Name:"kube-controller-manager-operator", UID:"c5cf246c-fa2e-11ea-8901-02f26ee80463", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NodeTargetRevisionChanged' Updating node "ip-10-0-129-244.us-west-2.compute.internal" from revision 9 to 10 because node ip-10-0-129-244.us-west-2.compute.internal with revision 9 is the oldest\nI0919 04:43:41.553982       1 event.go:209] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-controller-manager-operator", Name:"kube-controller-manager-operator", UID:"c5cf246c-fa2e-11ea-8901-02f26ee80463", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'PodCreated' Created Pod/installer-10-ip-10-0-129-244.us-west-2.compute.internal -n openshift-kube-controller-manager because it was missing\nI0919 04:43:52.110438       1 observer_polling.go:114] Observed file "/var/run/secrets/serving-cert/tls.key" has been modified (old="2476a523d0bd543c25ca5145545e0b649214877c4a3c1bc70402be91e4fe3e7c", new="57e5dfc26bdf47f5927ac615211398e352d867fd846f7b099a1be0c88511408b")\nW0919 04:43:52.110466       1 builder.go:108] Restart triggered because of file /var/run/secrets/serving-cert/tls.key was modified\nI0919 04:43:52.110590       1 observer_polling.go:114] Observed file "/var/run/secrets/serving-cert/tls.crt" has been modified (old="90750887cb194ab0ed872022d63f62815d8f8af6dfa4cc4f53d7082d89ebd4b2", new="2caaecef3c1eb85a5e3ea01f178aa0186874d17c5097645b9c26086e1cf68074")\nI0919 04:43:52.110602       1 config_observer_controller.go:159] Shutting down ConfigObserver\nI0919 04:43:52.110670       1 node_controller.go:172] Shutting down NodeController\nI0919 04:43:52.110776       1 secure_serving.go:160] Stopped listening on 0.0.0.0:8443\nI0919 04:43:52.110817       1 resourcesync_controller.go:227] Shutting down ResourceSyncController\nF0919 04:43:52.110865       1 builder.go:248] stopped\n
Sep 19 04:43:54.037 E ns/openshift-service-catalog-controller-manager-operator pod/openshift-service-catalog-controller-manager-operator-7b44ld9k8 node/ip-10-0-145-20.us-west-2.compute.internal container=operator container exited with code 255 (Error): 1/servicecatalogcontrollermanagers/cluster\nI0919 04:41:35.038019       1 request.go:530] Throttling request took 193.75326ms, request: GET:https://172.30.0.1:443/apis/operator.openshift.io/v1/servicecatalogcontrollermanagers/cluster\nI0919 04:41:35.238021       1 request.go:530] Throttling request took 197.290914ms, request: GET:https://172.30.0.1:443/apis/operator.openshift.io/v1/servicecatalogcontrollermanagers/cluster\nI0919 04:42:02.875815       1 wrap.go:47] GET /metrics: (4.984221ms) 200 [Prometheus/2.7.2 10.129.2.10:37136]\nI0919 04:42:02.877442       1 wrap.go:47] GET /metrics: (1.993197ms) 200 [Prometheus/2.7.2 10.131.0.8:34512]\nI0919 04:42:32.876101       1 wrap.go:47] GET /metrics: (5.222292ms) 200 [Prometheus/2.7.2 10.129.2.10:37136]\nI0919 04:42:32.877637       1 wrap.go:47] GET /metrics: (2.287301ms) 200 [Prometheus/2.7.2 10.131.0.8:34512]\nI0919 04:43:02.922706       1 wrap.go:47] GET /metrics: (51.080085ms) 200 [Prometheus/2.7.2 10.129.2.10:37136]\nI0919 04:43:02.924355       1 wrap.go:47] GET /metrics: (38.743942ms) 200 [Prometheus/2.7.2 10.131.0.8:34512]\nI0919 04:43:32.882416       1 wrap.go:47] GET /metrics: (6.942988ms) 200 [Prometheus/2.7.2 10.131.0.8:34512]\nI0919 04:43:32.884091       1 wrap.go:47] GET /metrics: (13.209313ms) 200 [Prometheus/2.7.2 10.129.2.10:37136]\nI0919 04:43:53.214987       1 observer_polling.go:78] Observed change: file:/var/run/secrets/serving-cert/tls.crt (current: "02f5d958a415da2bb0ec1c76593463c6039825c9e479c00dcf75656eae2d396a", lastKnown: "6ce534c0906aaeeb1b24d7f8624dfc44ad2f35512c6d8af7ba7c229bbef8f3cb")\nW0919 04:43:53.215018       1 builder.go:108] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was modified\nI0919 04:43:53.215068       1 observer_polling.go:78] Observed change: file:/var/run/secrets/serving-cert/tls.key (current: "5e259f664efdaf3721c43cfe8757cb1f5eb73a9cf55a9ef0b41d9ff4850acf7f", lastKnown: "f7adf255779ecf0907a354a4fff126474b65e2a74174f702613b1eee445835d1")\nF0919 04:43:53.215095       1 leaderelection.go:66] leaderelection lost\n
Sep 19 04:43:54.970 E ns/openshift-kube-scheduler-operator pod/openshift-kube-scheduler-operator-584cbb54f-bp6hq node/ip-10-0-129-244.us-west-2.compute.internal container=kube-scheduler-operator-container container exited with code 255 (Error): sion 6; 0 nodes have achieved new revision 7","reason":"Progressing","status":"True","type":"Progressing"},{"lastTransitionTime":"2020-09-19T04:19:30Z","message":"Available: 3 nodes are active; 3 nodes are at revision 6; 0 nodes have achieved new revision 7","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2020-09-19T04:16:49Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}\nI0919 04:43:54.110169       1 event.go:209] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-scheduler-operator", Name:"openshift-kube-scheduler-operator", UID:"c5d7ce81-fa2e-11ea-8901-02f26ee80463", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-scheduler changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready" to "StaticPodsDegraded: nodes/ip-10-0-145-20.us-west-2.compute.internal pods/openshift-kube-scheduler-ip-10-0-145-20.us-west-2.compute.internal container=\"scheduler\" is not ready\nNodeControllerDegraded: All master nodes are ready"\nI0919 04:43:54.283726       1 installer_controller.go:331] "ip-10-0-145-20.us-west-2.compute.internal" is in transition to 7, but has not made progress because static pod is pending\nI0919 04:43:54.847665       1 observer_polling.go:114] Observed file "/var/run/secrets/serving-cert/tls.crt" has been modified (old="822224c9ed448f45b8061fc8f370a1c4432f47738a5f8b9fbd03c1ca687bb856", new="924223aa09a2ae09ad9595faa36a9989aec47cf69d910547192a5bcd012a3fb6")\nW0919 04:43:54.847690       1 builder.go:108] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was modified\nI0919 04:43:54.847736       1 observer_polling.go:114] Observed file "/var/run/secrets/serving-cert/tls.key" has been modified (old="aa6cef2d4cc6242e2a2f5a8be039b4250e791f52efab88602cf8ca415b73fc35", new="f55ef00cbaf076e2585d5384f42533fe47ea8ffbab384a248db41790bbbfee75")\nF0919 04:43:54.847775       1 leaderelection.go:66] leaderelection lost\n
Sep 19 04:43:59.954 E ns/openshift-image-registry pod/node-ca-7875z node/ip-10-0-158-47.us-west-2.compute.internal container=node-ca container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:44:01.497 E ns/openshift-apiserver-operator pod/openshift-apiserver-operator-677c58c6b5-mf2rb node/ip-10-0-130-52.us-west-2.compute.internal container=openshift-apiserver-operator container exited with code 255 (Error):  04:41:27.598344       1 reflector.go:289] github.com/openshift/client-go/config/informers/externalversions/factory.go:101: watch of *v1.Ingress ended with: too old resource version: 18757 (20250)\nW0919 04:41:27.598739       1 reflector.go:289] k8s.io/client-go/dynamic/dynamicinformer/informer.go:90: watch of *unstructured.Unstructured ended with: too old resource version: 19789 (20250)\nW0919 04:41:27.598866       1 reflector.go:289] github.com/openshift/client-go/config/informers/externalversions/factory.go:101: watch of *v1.Project ended with: too old resource version: 18759 (20251)\nW0919 04:41:27.599159       1 reflector.go:289] k8s.io/client-go/informers/factory.go:133: watch of *v1.Service ended with: too old resource version: 18536 (19639)\nW0919 04:41:27.634517       1 reflector.go:289] k8s.io/client-go/informers/factory.go:133: watch of *v1.DaemonSet ended with: too old resource version: 23274 (26594)\nW0919 04:41:27.705654       1 reflector.go:289] k8s.io/client-go/informers/factory.go:133: watch of *v1.ConfigMap ended with: too old resource version: 19712 (24556)\nW0919 04:41:27.723808       1 reflector.go:289] github.com/openshift/client-go/operator/informers/externalversions/factory.go:101: watch of *v1.OpenShiftAPIServer ended with: too old resource version: 19789 (20250)\nI0919 04:44:00.458410       1 observer_polling.go:78] Observed change: file:/var/run/secrets/serving-cert/tls.crt (current: "d034a8cad1c92612b01ac7434f0a320c3de75070ac0ae9ea11ca83f8d5a312a4", lastKnown: "856370537d91f3e8be5c602fe9b17285948b505e3e6397874f86098ffefd38d6")\nW0919 04:44:00.458435       1 builder.go:108] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was modified\nI0919 04:44:00.458483       1 observer_polling.go:78] Observed change: file:/var/run/secrets/serving-cert/tls.key (current: "39cbbb6ff1ecb6ecb987236e287abb0a639385e1134eae8700d8116f4f5be9ec", lastKnown: "20f84319792fae826dfdf2336884629d93532d58ec1d636628ba18659b6ed2ec")\nF0919 04:44:00.458489       1 leaderelection.go:66] leaderelection lost\n
Sep 19 04:44:03.507 E ns/openshift-console pod/console-bfffb56f5-8cfw4 node/ip-10-0-130-52.us-west-2.compute.internal container=console container exited with code 2 (Error): 2020/09/19 04:25:07 cmd/main: cookies are secure!\n2020/09/19 04:25:07 cmd/main: Binding to 0.0.0.0:8443...\n2020/09/19 04:25:07 cmd/main: using TLS\n2020/09/19 04:40:38 auth: failed to get latest auth source data: request to OAuth issuer endpoint https://oauth-openshift.apps.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com/oauth/token failed: Head https://oauth-openshift.apps.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n2020/09/19 04:40:44 auth: failed to get latest auth source data: request to OAuth issuer endpoint https://oauth-openshift.apps.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com/oauth/token failed: Head https://oauth-openshift.apps.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n2020/09/19 04:43:51 http: TLS handshake error from 10.128.2.21:37624: remote error: tls: error decrypting message\n2020/09/19 04:43:56 http: TLS handshake error from 10.128.2.21:37714: remote error: tls: error decrypting message\n2020/09/19 04:44:01 http: TLS handshake error from 10.128.2.21:37774: remote error: tls: error decrypting message\n
Sep 19 04:44:03.799 E ns/openshift-monitoring pod/prometheus-adapter-6c7dd56d64-7hw88 node/ip-10-0-158-47.us-west-2.compute.internal container=prometheus-adapter container exited with code 2 (Error): I0919 04:41:53.884341       1 adapter.go:93] successfully using in-cluster auth\nI0919 04:41:54.963890       1 secure_serving.go:116] Serving securely on [::]:6443\n
Sep 19 04:44:04.003 E ns/openshift-service-catalog-apiserver-operator pod/openshift-service-catalog-apiserver-operator-f8b76df5d-mtrrs node/ip-10-0-145-20.us-west-2.compute.internal container=operator container exited with code 255 (Error):  ]\n}'\nI0919 04:43:18.454530       1 workload_controller.go:325] No service bindings found, nothing to delete.\nI0919 04:43:18.461506       1 workload_controller.go:179] apiservice v1beta1.servicecatalog.k8s.io deleted\nI0919 04:43:27.938711       1 leaderelection.go:258] successfully renewed lease openshift-service-catalog-apiserver-operator/openshift-cluster-svcat-apiserver-operator-lock\nI0919 04:43:37.946969       1 leaderelection.go:258] successfully renewed lease openshift-service-catalog-apiserver-operator/openshift-cluster-svcat-apiserver-operator-lock\nI0919 04:43:38.454370       1 request.go:1145] body was not decodable (unable to check for Status): Object 'Kind' is missing in '{\n  "paths": [\n    "/apis",\n    "/metrics",\n    "/version"\n  ]\n}'\nI0919 04:43:38.454511       1 workload_controller.go:325] No service bindings found, nothing to delete.\nI0919 04:43:38.461618       1 workload_controller.go:179] apiservice v1beta1.servicecatalog.k8s.io deleted\nI0919 04:43:47.954695       1 leaderelection.go:258] successfully renewed lease openshift-service-catalog-apiserver-operator/openshift-cluster-svcat-apiserver-operator-lock\nI0919 04:43:57.962263       1 leaderelection.go:258] successfully renewed lease openshift-service-catalog-apiserver-operator/openshift-cluster-svcat-apiserver-operator-lock\nI0919 04:44:03.562353       1 observer_polling.go:78] Observed change: file:/var/run/secrets/serving-cert/tls.crt (current: "d0d002fb199116d7dd10882f0b9872c38e43154a322f713bd34fb4f3b6bfe5da", lastKnown: "0feacdf977e0e76b8e0911d20534ef7756f04d55b812d3427d672cc3f73ef66d")\nW0919 04:44:03.562379       1 builder.go:108] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was modified\nI0919 04:44:03.562422       1 observer_polling.go:78] Observed change: file:/var/run/secrets/serving-cert/tls.key (current: "1cf567230d4d494d8c054a44165075b9b84d5fdc3ddd8aa37db6de235ca0a689", lastKnown: "5f18a183656893d4c3c071b0a3db87f26be1ddd432d2a193efaefd1a34debc8c")\nF0919 04:44:03.562439       1 builder.go:217] server exited\n
Sep 19 04:44:06.003 E ns/openshift-authentication-operator pod/authentication-operator-66ffd6c4c7-w5bfv node/ip-10-0-129-244.us-west-2.compute.internal container=operator container exited with code 255 (Error): perator/authentication diff {"status":{"conditions":[{"lastTransitionTime":"2020-09-19T04:23:23Z","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-09-19T04:43:49Z","reason":"AsExpected","status":"False","type":"Progressing"},{"lastTransitionTime":"2020-09-19T04:27:35Z","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2020-09-19T04:20:10Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}\nI0919 04:43:49.272461       1 event.go:209] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"cd86b7d7-fa2e-11ea-8901-02f26ee80463", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Progressing changed from True to False ("")\nI0919 04:44:05.892630       1 observer_polling.go:88] Observed change: file:/var/run/secrets/serving-cert/tls.key (current: "20aa1a708ed2ab796f52d561a02c53ca0ae86105e825b68bebd5f3d68a5fd514", lastKnown: "da1b3f91b7e2ecc0508319686feb3fec3acc0d7be4e296c0d301bf84af54a171")\nW0919 04:44:05.892654       1 builder.go:108] Restart triggered because of file /var/run/secrets/serving-cert/tls.key was modified\nF0919 04:44:05.892710       1 leaderelection.go:66] leaderelection lost\nI0919 04:44:05.904811       1 resourcesync_controller.go:227] Shutting down ResourceSyncController\nI0919 04:44:05.904906       1 controller.go:214] Shutting down RouterCertsDomainValidationController\nI0919 04:44:05.904945       1 logging_controller.go:92] Shutting down LogLevelController\nI0919 04:44:05.904961       1 unsupportedconfigoverrides_controller.go:161] Shutting down UnsupportedConfigOverridesController\nI0919 04:44:05.904969       1 management_state_controller.go:111] Shutting down management-state-controller-authentication\nI0919 04:44:05.904976       1 remove_stale_conditions.go:83] Shutting down RemoveStaleConditions\nF0919 04:44:05.904916       1 builder.go:248] stopped\n
Sep 19 04:44:07.021 E ns/openshift-controller-manager-operator pod/openshift-controller-manager-operator-69479fc684-kfzjl node/ip-10-0-145-20.us-west-2.compute.internal container=operator container exited with code 255 (Error): 04:44:01.271032       1 request.go:530] Throttling request took 196.769186ms, request: GET:https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/namespaces/openshift-controller-manager/rolebindings/prometheus-k8s\nI0919 04:44:01.304978       1 status_controller.go:165] clusteroperator/openshift-controller-manager diff {"status":{"conditions":[{"lastTransitionTime":"2020-09-19T04:16:48Z","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-09-19T04:44:01Z","reason":"AsExpected","status":"False","type":"Progressing"},{"lastTransitionTime":"2020-09-19T04:17:28Z","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2020-09-19T04:16:48Z","reason":"NoData","status":"Unknown","type":"Upgradeable"}]}}\nI0919 04:44:01.311445       1 event.go:209] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-controller-manager-operator", Name:"openshift-controller-manager-operator", UID:"c61bddfa-fa2e-11ea-8901-02f26ee80463", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/openshift-controller-manager changed: Progressing changed from True to False ("")\nI0919 04:44:06.203104       1 observer_polling.go:78] Observed change: file:/var/run/secrets/serving-cert/tls.crt (current: "058fb4a630830071950c9b605038da1077e7a9a38b69894849b1e38d0e893c5b", lastKnown: "0e402e0034c5b712499e239866a1bcd21f555b89b861335618921662cdd34371")\nW0919 04:44:06.203138       1 builder.go:108] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was modified\nI0919 04:44:06.203185       1 observer_polling.go:78] Observed change: file:/var/run/secrets/serving-cert/tls.key (current: "d0d5c227f8b491c752357b70f4840555626084685a3c414deb06f94e17cf1edf", lastKnown: "39a3e7c9aaed88ff6406659996cb7f06ceee58de7969a6096617513925d78bc7")\nF0919 04:44:06.203193       1 leaderelection.go:66] leaderelection lost\nI0919 04:44:06.206923       1 config_observer_controller.go:159] Shutting down ConfigObserver\n
Sep 19 04:44:15.533 E ns/openshift-console-operator pod/console-operator-7d5f9cbd6c-lss7t node/ip-10-0-130-52.us-west-2.compute.internal container=console-operator container exited with code 255 (Error): eploymentAvailable: 3 replicas ready at version 4.2.36" to "DeploymentAvailable: 2 replicas ready at version 4.2.36"\nE0919 04:43:32.462544       1 status.go:71] SyncLoopRefreshProgressing InProgress Working toward version 4.2.36\nE0919 04:43:32.462640       1 status.go:71] DeploymentAvailable FailedUpdate 2 replicas ready at version 4.2.36\nI0919 04:44:02.388660       1 status_controller.go:165] clusteroperator/console diff {"status":{"conditions":[{"lastTransitionTime":"2020-09-19T04:43:09Z","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-09-19T04:44:02Z","reason":"AsExpected","status":"False","type":"Progressing"},{"lastTransitionTime":"2020-09-19T04:44:02Z","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2020-09-19T04:20:02Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}\nI0919 04:44:02.395160       1 event.go:209] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"5833d1c3-fa2f-11ea-8f7e-0603b42f4bc7", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/console changed: Progressing changed from True to False (""),Available changed from False to True ("")\nI0919 04:44:14.603194       1 observer_polling.go:78] Observed change: file:/var/run/secrets/serving-cert/tls.crt (current: "466c380d105c82eeaefc91bea104e2fc4dc5b2b5a0dc12ffd19c348259d4c586", lastKnown: "6f011dea389d8162786e0ed00f86e23179a96df2e7f1c8367ac4d1a6e503ec29")\nW0919 04:44:14.603228       1 builder.go:108] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was modified\nI0919 04:44:14.603275       1 observer_polling.go:78] Observed change: file:/var/run/secrets/serving-cert/tls.key (current: "2d4a081d1dab62c0ec98b4dadb4b5e37e72dbbe1bd9cc8507595de06d7d1a684", lastKnown: "82afea717dfc8e251bb357e8e4cfc0e4475971382a4583193086200bd052a9c7")\nF0919 04:44:14.603284       1 leaderelection.go:66] leaderelection lost\n
Sep 19 04:44:16.684 E ns/openshift-monitoring pod/prometheus-adapter-6c7dd56d64-s2lvv node/ip-10-0-129-73.us-west-2.compute.internal container=prometheus-adapter container exited with code 2 (Error): I0919 04:41:03.580083       1 adapter.go:93] successfully using in-cluster auth\nI0919 04:41:04.401353       1 secure_serving.go:116] Serving securely on [::]:6443\n
Sep 19 04:44:22.627 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ip-10-0-138-182.us-west-2.compute.internal container=prometheus container exited with code 1 (Error): =info ts=2020-09-19T04:44:12.407Z caller=main.go:332 fd_limits="(soft=1048576, hard=1048576)"\nlevel=info ts=2020-09-19T04:44:12.407Z caller=main.go:333 vm_limits="(soft=unlimited, hard=unlimited)"\nlevel=info ts=2020-09-19T04:44:12.408Z caller=web.go:448 component=web msg="Start listening for connections" address=127.0.0.1:9090\nlevel=info ts=2020-09-19T04:44:12.408Z caller=main.go:652 msg="Starting TSDB ..."\nlevel=info ts=2020-09-19T04:44:12.415Z caller=main.go:667 fs_type=XFS_SUPER_MAGIC\nlevel=info ts=2020-09-19T04:44:12.415Z caller=main.go:668 msg="TSDB started"\nlevel=info ts=2020-09-19T04:44:12.415Z caller=main.go:738 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml\nlevel=info ts=2020-09-19T04:44:12.415Z caller=main.go:521 msg="Stopping scrape discovery manager..."\nlevel=info ts=2020-09-19T04:44:12.415Z caller=main.go:535 msg="Stopping notify discovery manager..."\nlevel=info ts=2020-09-19T04:44:12.415Z caller=main.go:557 msg="Stopping scrape manager..."\nlevel=info ts=2020-09-19T04:44:12.415Z caller=main.go:531 msg="Notify discovery manager stopped"\nlevel=info ts=2020-09-19T04:44:12.415Z caller=main.go:517 msg="Scrape discovery manager stopped"\nlevel=info ts=2020-09-19T04:44:12.416Z caller=main.go:551 msg="Scrape manager stopped"\nlevel=info ts=2020-09-19T04:44:12.416Z caller=manager.go:776 component="rule manager" msg="Stopping rule manager..."\nlevel=info ts=2020-09-19T04:44:12.416Z caller=manager.go:782 component="rule manager" msg="Rule manager stopped"\nlevel=info ts=2020-09-19T04:44:12.417Z caller=notifier.go:602 component=notifier msg="Stopping notification manager..."\nlevel=info ts=2020-09-19T04:44:12.417Z caller=main.go:722 msg="Notifier manager stopped"\nlevel=error ts=2020-09-19T04:44:12.417Z caller=main.go:731 err="error loading config from \"/etc/prometheus/config_out/prometheus.env.yaml\": couldn't load configuration (--config.file=\"/etc/prometheus/config_out/prometheus.env.yaml\"): open /etc/prometheus/config_out/prometheus.env.yaml: no such file or directory"\n
Sep 19 04:44:32.093 E ns/openshift-kube-apiserver-operator pod/kube-apiserver-operator-57547b4d45-px7l8 node/ip-10-0-129-244.us-west-2.compute.internal container=kube-apiserver-operator container exited with code 255 (Error): figMap/revision-status-8 -n openshift-kube-apiserver: cause by changes in data.status\nI0919 04:41:49.162135       1 event.go:209] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"c5cf4d35-fa2e-11ea-8901-02f26ee80463", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-apiserver changed: Degraded message changed from "StaticPodsDegraded: nodes/ip-10-0-130-52.us-west-2.compute.internal pods/kube-apiserver-ip-10-0-130-52.us-west-2.compute.internal container=\"kube-apiserver-8\" is not ready\nNodeControllerDegraded: All master nodes are ready" to "NodeControllerDegraded: All master nodes are ready"\nI0919 04:41:51.487731       1 event.go:209] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"c5cf4d35-fa2e-11ea-8901-02f26ee80463", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'PodCreated' Created Pod/revision-pruner-8-ip-10-0-130-52.us-west-2.compute.internal -n openshift-kube-apiserver because it was missing\nI0919 04:44:31.388054       1 observer_polling.go:114] Observed file "/var/run/secrets/serving-cert/tls.crt" has been modified (old="17cd38092c1de82d742290c73fc0e3166b3f491fbd83fd55865b4e13119108e2", new="4c5be2f6fef18b1261f934ced83e2ea8e98f3f31644503286e2744468e431d74")\nW0919 04:44:31.388165       1 builder.go:108] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was modified\nI0919 04:44:31.388214       1 observer_polling.go:114] Observed file "/var/run/secrets/serving-cert/tls.key" has been modified (old="49d08ed6a5fc239711a3106c83336dec8613c67f8517bd977c1a311de99c8d62", new="dd721aab47bafc10cd3bfa8c59aa0870f222411acd7862d05b5179f5b291b4e7")\nF0919 04:44:31.388259       1 leaderelection.go:66] leaderelection lost\nI0919 04:44:31.403945       1 backing_resource_controller.go:148] Shutting down BackingResourceController\n
Sep 19 04:44:52.143 E ns/openshift-controller-manager pod/controller-manager-wxlrg node/ip-10-0-129-244.us-west-2.compute.internal container=controller-manager container exited with code 137 (Error): 
Sep 19 04:44:57.274 E ns/openshift-sdn pod/sdn-fvmnh node/ip-10-0-130-52.us-west-2.compute.internal container=sdn container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:45:03.255 E ns/openshift-monitoring pod/prometheus-k8s-0 node/ip-10-0-158-47.us-west-2.compute.internal container=prometheus container exited with code 1 (Error): =info ts=2020-09-19T04:44:52.592Z caller=main.go:332 fd_limits="(soft=1048576, hard=1048576)"\nlevel=info ts=2020-09-19T04:44:52.592Z caller=main.go:333 vm_limits="(soft=unlimited, hard=unlimited)"\nlevel=info ts=2020-09-19T04:44:52.594Z caller=main.go:652 msg="Starting TSDB ..."\nlevel=info ts=2020-09-19T04:44:52.594Z caller=web.go:448 component=web msg="Start listening for connections" address=127.0.0.1:9090\nlevel=info ts=2020-09-19T04:44:52.605Z caller=main.go:667 fs_type=XFS_SUPER_MAGIC\nlevel=info ts=2020-09-19T04:44:52.606Z caller=main.go:668 msg="TSDB started"\nlevel=info ts=2020-09-19T04:44:52.606Z caller=main.go:738 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml\nlevel=info ts=2020-09-19T04:44:52.606Z caller=main.go:521 msg="Stopping scrape discovery manager..."\nlevel=info ts=2020-09-19T04:44:52.606Z caller=main.go:535 msg="Stopping notify discovery manager..."\nlevel=info ts=2020-09-19T04:44:52.606Z caller=main.go:557 msg="Stopping scrape manager..."\nlevel=info ts=2020-09-19T04:44:52.606Z caller=main.go:531 msg="Notify discovery manager stopped"\nlevel=info ts=2020-09-19T04:44:52.606Z caller=main.go:517 msg="Scrape discovery manager stopped"\nlevel=info ts=2020-09-19T04:44:52.606Z caller=manager.go:776 component="rule manager" msg="Stopping rule manager..."\nlevel=info ts=2020-09-19T04:44:52.606Z caller=manager.go:782 component="rule manager" msg="Rule manager stopped"\nlevel=info ts=2020-09-19T04:44:52.606Z caller=main.go:551 msg="Scrape manager stopped"\nlevel=info ts=2020-09-19T04:44:52.609Z caller=notifier.go:602 component=notifier msg="Stopping notification manager..."\nlevel=info ts=2020-09-19T04:44:52.609Z caller=main.go:722 msg="Notifier manager stopped"\nlevel=error ts=2020-09-19T04:44:52.613Z caller=main.go:731 err="error loading config from \"/etc/prometheus/config_out/prometheus.env.yaml\": couldn't load configuration (--config.file=\"/etc/prometheus/config_out/prometheus.env.yaml\"): open /etc/prometheus/config_out/prometheus.env.yaml: no such file or directory"\n
Sep 19 04:45:13.300 E ns/openshift-sdn pod/sdn-controller-prpdx node/ip-10-0-145-20.us-west-2.compute.internal container=sdn-controller container exited with code 137 (Error): I0919 04:16:01.174833       1 leaderelection.go:205] attempting to acquire leader lease  openshift-sdn/openshift-network-controller...\nE0919 04:16:01.178530       1 leaderelection.go:270] error retrieving resource lock openshift-sdn/openshift-network-controller: Get https://api-int.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com:6443/api/v1/namespaces/openshift-sdn/configmaps/openshift-network-controller: dial tcp 10.0.158.154:6443: connect: connection refused\nE0919 04:16:11.497419       1 leaderelection.go:270] error retrieving resource lock openshift-sdn/openshift-network-controller: Get https://api-int.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com:6443/api/v1/namespaces/openshift-sdn/configmaps/openshift-network-controller: dial tcp 10.0.158.154:6443: connect: connection refused\nE0919 04:28:25.800317       1 leaderelection.go:270] error retrieving resource lock openshift-sdn/openshift-network-controller: configmaps "openshift-network-controller" is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot get resource "configmaps" in API group "" in the namespace "openshift-sdn"\n
Sep 19 04:45:25.063 E ns/openshift-monitoring pod/alertmanager-main-2 node/ip-10-0-138-182.us-west-2.compute.internal container=config-reloader container exited with code 2 (Error): 2020/09/19 04:43:54 Watching directory: "/etc/alertmanager/config"\n
Sep 19 04:45:25.063 E ns/openshift-monitoring pod/alertmanager-main-2 node/ip-10-0-138-182.us-west-2.compute.internal container=alertmanager-proxy container exited with code 2 (Error): 2020/09/19 04:43:54 provider.go:117: Defaulting client-id to system:serviceaccount:openshift-monitoring:alertmanager-main\n2020/09/19 04:43:54 provider.go:122: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token\n2020/09/19 04:43:54 provider.go:310: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.\n2020/09/19 04:43:54 oauthproxy.go:200: mapping path "/" => upstream "http://localhost:9093/"\n2020/09/19 04:43:54 oauthproxy.go:221: compiled skip-auth-regex => "^/metrics"\n2020/09/19 04:43:54 oauthproxy.go:227: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:alertmanager-main\n2020/09/19 04:43:54 oauthproxy.go:237: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> refresh:disabled\n2020/09/19 04:43:54 http.go:106: HTTPS: listening on [::]:9095\n2020/09/19 04:44:03 server.go:3012: http: TLS handshake error from 10.131.0.15:44858: remote error: tls: error decrypting message\n2020/09/19 04:44:08 server.go:3012: http: TLS handshake error from 10.131.0.15:44934: remote error: tls: error decrypting message\n2020/09/19 04:44:13 server.go:3012: http: TLS handshake error from 10.131.0.15:45026: remote error: tls: error decrypting message\n2020/09/19 04:44:19 server.go:3012: http: TLS handshake error from 10.131.0.15:45144: remote error: tls: error decrypting message\n2020/09/19 04:44:24 server.go:3012: http: TLS handshake error from 10.131.0.15:45226: remote error: tls: error decrypting message\n2020/09/19 04:44:29 server.go:3012: http: TLS handshake error from 10.131.0.15:45296: remote error: tls: error decrypting message\n
Sep 19 04:45:26.416 E ns/openshift-sdn pod/sdn-6d685 node/ip-10-0-158-47.us-west-2.compute.internal container=sdn container exited with code 255 (Error): ck.go:62] SDN healthcheck unable to reconnect to OVS server: dial unix /var/run/openvswitch/db.sock: connect: connection refused\nI0919 04:45:24.751574    2443 healthcheck.go:62] SDN healthcheck unable to reconnect to OVS server: dial unix /var/run/openvswitch/db.sock: connect: connection refused\nI0919 04:45:24.842713    2443 proxier.go:367] userspace proxy: processing 0 service events\nI0919 04:45:24.842743    2443 proxier.go:346] userspace syncProxyRules took 84.800426ms\nI0919 04:45:24.851565    2443 healthcheck.go:62] SDN healthcheck unable to reconnect to OVS server: dial unix /var/run/openvswitch/db.sock: connect: connection refused\nI0919 04:45:24.952223    2443 healthcheck.go:62] SDN healthcheck unable to reconnect to OVS server: dial unix /var/run/openvswitch/db.sock: connect: connection refused\nI0919 04:45:24.956102    2443 ovs.go:169] Error executing ovs-ofctl: ovs-ofctl: /var/run/openvswitch/br0.mgmt: failed to open socket (Connection refused)\nE0919 04:45:24.956136    2443 metrics.go:132] failed to dump OVS flows for metrics: exit status 1\nI0919 04:45:25.051617    2443 healthcheck.go:62] SDN healthcheck unable to reconnect to OVS server: dial unix /var/run/openvswitch/db.sock: connect: connection refused\nI0919 04:45:25.151558    2443 healthcheck.go:62] SDN healthcheck unable to reconnect to OVS server: dial unix /var/run/openvswitch/db.sock: connect: connection refused\nI0919 04:45:25.251557    2443 healthcheck.go:62] SDN healthcheck unable to reconnect to OVS server: dial unix /var/run/openvswitch/db.sock: connect: connection refused\nI0919 04:45:25.351619    2443 healthcheck.go:62] SDN healthcheck unable to reconnect to OVS server: dial unix /var/run/openvswitch/db.sock: connect: connection refused\nI0919 04:45:25.351686    2443 healthcheck.go:62] SDN healthcheck unable to reconnect to OVS server: dial unix /var/run/openvswitch/db.sock: connect: connection refused\nF0919 04:45:25.351700    2443 healthcheck.go:78] SDN healthcheck detected unhealthy OVS server, restarting: timed out waiting for the condition\n
Sep 19 04:45:32.187 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ip-10-0-138-182.us-west-2.compute.internal container=prometheus container exited with code 1 (Error): =info ts=2020-09-19T04:44:12.407Z caller=main.go:332 fd_limits="(soft=1048576, hard=1048576)"\nlevel=info ts=2020-09-19T04:44:12.407Z caller=main.go:333 vm_limits="(soft=unlimited, hard=unlimited)"\nlevel=info ts=2020-09-19T04:44:12.408Z caller=web.go:448 component=web msg="Start listening for connections" address=127.0.0.1:9090\nlevel=info ts=2020-09-19T04:44:12.408Z caller=main.go:652 msg="Starting TSDB ..."\nlevel=info ts=2020-09-19T04:44:12.415Z caller=main.go:667 fs_type=XFS_SUPER_MAGIC\nlevel=info ts=2020-09-19T04:44:12.415Z caller=main.go:668 msg="TSDB started"\nlevel=info ts=2020-09-19T04:44:12.415Z caller=main.go:738 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml\nlevel=info ts=2020-09-19T04:44:12.415Z caller=main.go:521 msg="Stopping scrape discovery manager..."\nlevel=info ts=2020-09-19T04:44:12.415Z caller=main.go:535 msg="Stopping notify discovery manager..."\nlevel=info ts=2020-09-19T04:44:12.415Z caller=main.go:557 msg="Stopping scrape manager..."\nlevel=info ts=2020-09-19T04:44:12.415Z caller=main.go:531 msg="Notify discovery manager stopped"\nlevel=info ts=2020-09-19T04:44:12.415Z caller=main.go:517 msg="Scrape discovery manager stopped"\nlevel=info ts=2020-09-19T04:44:12.416Z caller=main.go:551 msg="Scrape manager stopped"\nlevel=info ts=2020-09-19T04:44:12.416Z caller=manager.go:776 component="rule manager" msg="Stopping rule manager..."\nlevel=info ts=2020-09-19T04:44:12.416Z caller=manager.go:782 component="rule manager" msg="Rule manager stopped"\nlevel=info ts=2020-09-19T04:44:12.417Z caller=notifier.go:602 component=notifier msg="Stopping notification manager..."\nlevel=info ts=2020-09-19T04:44:12.417Z caller=main.go:722 msg="Notifier manager stopped"\nlevel=error ts=2020-09-19T04:44:12.417Z caller=main.go:731 err="error loading config from \"/etc/prometheus/config_out/prometheus.env.yaml\": couldn't load configuration (--config.file=\"/etc/prometheus/config_out/prometheus.env.yaml\"): open /etc/prometheus/config_out/prometheus.env.yaml: no such file or directory"\n
Sep 19 04:45:32.187 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ip-10-0-138-182.us-west-2.compute.internal container=rules-configmap-reloader container exited with code 2 (Error): 2020/09/19 04:44:17 Watching directory: "/etc/prometheus/rules/prometheus-k8s-rulefiles-0"\n
Sep 19 04:45:32.187 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ip-10-0-138-182.us-west-2.compute.internal container=prometheus-config-reloader container exited with code 2 (Error): ts=2020-09-19T04:44:16.75393656Z caller=main.go:85 msg="Starting prometheus-config-reloader version '0.33.0'."\nlevel=error ts=2020-09-19T04:44:16.755712516Z caller=runutil.go:88 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post http://localhost:9090/-/reload: dial tcp [::1]:9090: connect: connection refused"\nlevel=error ts=2020-09-19T04:44:21.75567857Z caller=runutil.go:88 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post http://localhost:9090/-/reload: dial tcp [::1]:9090: connect: connection refused"\nlevel=info ts=2020-09-19T04:44:26.86640079Z caller=reloader.go:286 msg="Prometheus reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml rule_dirs=\nlevel=info ts=2020-09-19T04:44:26.866493925Z caller=reloader.go:154 msg="started watching config file and non-recursively rule dirs for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=\n
Sep 19 04:45:36.436 E ns/openshift-authentication-operator pod/authentication-operator-66ffd6c4c7-w5bfv node/ip-10-0-129-244.us-west-2.compute.internal container=operator container exited with code 255 (Error): aws.dev.rhcloud.com on 172.30.0.10:53: read udp 10.129.0.56:46737-\u003e172.30.0.10:53: i/o timeout","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-09-19T04:45:11Z","message":"Progressing: not all deployment replicas are ready","reason":"ProgressingOAuthServerDeploymentNotReady","status":"True","type":"Progressing"},{"lastTransitionTime":"2020-09-19T04:27:35Z","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2020-09-19T04:20:10Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}\nE0919 04:45:27.415848       1 controller.go:129] {AuthenticationOperator2 AuthenticationOperator2} failed with: error checking current version: unable to check route health: failed to GET route: dial tcp: lookup oauth-openshift.apps.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com on 172.30.0.10:53: read udp 10.129.0.56:46737->172.30.0.10:53: i/o timeout\nI0919 04:45:27.420970       1 event.go:209] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"cd86b7d7-fa2e-11ea-8901-02f26ee80463", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "" to "RouteHealthDegraded: failed to GET route: dial tcp: lookup oauth-openshift.apps.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com on 172.30.0.10:53: read udp 10.129.0.56:46737->172.30.0.10:53: i/o timeout"\nI0919 04:45:36.184176       1 observer_polling.go:88] Observed change: file:/var/run/configmaps/trusted-ca-bundle/ca-bundle.crt (current: "c594b2d8cd9a41518152c0dadeff734c173705a8708ae3561b9f88714f09add6", lastKnown: "")\nI0919 04:45:36.184201       1 cmd.go:111] exiting because "/var/run/configmaps/trusted-ca-bundle/ca-bundle.crt" changed\nF0919 04:45:36.184239       1 leaderelection.go:66] leaderelection lost\nI0919 04:45:36.190797       1 logging_controller.go:92] Shutting down LogLevelController\n
Sep 19 04:45:51.479 E ns/openshift-sdn pod/sdn-controller-nvwtb node/ip-10-0-129-244.us-west-2.compute.internal container=sdn-controller container exited with code 137 (Error): server could not find the requested resource\nE0919 04:40:33.074922       1 memcache.go:141] couldn't get resource list for scheduling.k8s.io/v1: the server could not find the requested resource\nE0919 04:40:33.076147       1 memcache.go:141] couldn't get resource list for coordination.k8s.io/v1: the server could not find the requested resource\nE0919 04:40:33.077356       1 memcache.go:141] couldn't get resource list for node.k8s.io/v1beta1: the server could not find the requested resource\nE0919 04:41:03.145844       1 memcache.go:141] couldn't get resource list for networking.k8s.io/v1beta1: the server could not find the requested resource\nE0919 04:41:03.152879       1 memcache.go:141] couldn't get resource list for scheduling.k8s.io/v1: the server could not find the requested resource\nE0919 04:41:03.154189       1 memcache.go:141] couldn't get resource list for coordination.k8s.io/v1: the server could not find the requested resource\nE0919 04:41:03.155536       1 memcache.go:141] couldn't get resource list for node.k8s.io/v1beta1: the server could not find the requested resource\nE0919 04:41:27.532609       1 reflector.go:237] k8s.io/client-go/informers/factory.go:132: Failed to watch *v1.Node: Get https://api-int.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com:6443/api/v1/nodes?resourceVersion=27488&timeout=6m27s&timeoutSeconds=387&watch=true: dial tcp 10.0.133.160:6443: connect: connection refused\nW0919 04:41:27.601834       1 reflector.go:256] github.com/openshift/client-go/network/informers/externalversions/factory.go:101: watch of *v1.HostSubnet ended with: too old resource version: 18757 (20522)\nW0919 04:44:38.576450       1 reflector.go:256] github.com/openshift/client-go/network/informers/externalversions/factory.go:101: watch of *v1.HostSubnet ended with: too old resource version: 20522 (32101)\nW0919 04:44:38.973424       1 reflector.go:256] github.com/openshift/client-go/network/informers/externalversions/factory.go:101: watch of *v1.NetNamespace ended with: too old resource version: 20623 (32103)\n
Sep 19 04:45:51.546 E ns/openshift-sdn pod/sdn-hlqd5 node/ip-10-0-129-73.us-west-2.compute.internal container=sdn container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:45:51.554 E ns/openshift-sdn pod/sdn-m7xqj node/ip-10-0-129-244.us-west-2.compute.internal container=sdn container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:45:54.474 E ns/openshift-multus pod/multus-dtlkm node/ip-10-0-145-20.us-west-2.compute.internal container=kube-multus container exited with code 137 (Error): 
Sep 19 04:45:56.980 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ip-10-0-138-182.us-west-2.compute.internal container=prometheus container exited with code 1 (Error): =info ts=2020-09-19T04:45:52.496Z caller=main.go:332 fd_limits="(soft=1048576, hard=1048576)"\nlevel=info ts=2020-09-19T04:45:52.496Z caller=main.go:333 vm_limits="(soft=unlimited, hard=unlimited)"\nlevel=info ts=2020-09-19T04:45:52.498Z caller=main.go:652 msg="Starting TSDB ..."\nlevel=info ts=2020-09-19T04:45:52.498Z caller=web.go:448 component=web msg="Start listening for connections" address=127.0.0.1:9090\nlevel=info ts=2020-09-19T04:45:52.503Z caller=main.go:667 fs_type=XFS_SUPER_MAGIC\nlevel=info ts=2020-09-19T04:45:52.504Z caller=main.go:668 msg="TSDB started"\nlevel=info ts=2020-09-19T04:45:52.504Z caller=main.go:738 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml\nlevel=info ts=2020-09-19T04:45:52.504Z caller=main.go:521 msg="Stopping scrape discovery manager..."\nlevel=info ts=2020-09-19T04:45:52.504Z caller=main.go:535 msg="Stopping notify discovery manager..."\nlevel=info ts=2020-09-19T04:45:52.504Z caller=main.go:557 msg="Stopping scrape manager..."\nlevel=info ts=2020-09-19T04:45:52.504Z caller=main.go:531 msg="Notify discovery manager stopped"\nlevel=info ts=2020-09-19T04:45:52.504Z caller=main.go:517 msg="Scrape discovery manager stopped"\nlevel=info ts=2020-09-19T04:45:52.504Z caller=manager.go:776 component="rule manager" msg="Stopping rule manager..."\nlevel=info ts=2020-09-19T04:45:52.504Z caller=manager.go:782 component="rule manager" msg="Rule manager stopped"\nlevel=info ts=2020-09-19T04:45:52.504Z caller=main.go:551 msg="Scrape manager stopped"\nlevel=info ts=2020-09-19T04:45:52.505Z caller=notifier.go:602 component=notifier msg="Stopping notification manager..."\nlevel=info ts=2020-09-19T04:45:52.505Z caller=main.go:722 msg="Notifier manager stopped"\nlevel=error ts=2020-09-19T04:45:52.506Z caller=main.go:731 err="error loading config from \"/etc/prometheus/config_out/prometheus.env.yaml\": couldn't load configuration (--config.file=\"/etc/prometheus/config_out/prometheus.env.yaml\"): open /etc/prometheus/config_out/prometheus.env.yaml: no such file or directory"\n
Sep 19 04:46:01.371 E ns/openshift-monitoring pod/telemeter-client-d5798f6df-4n4bv node/ip-10-0-129-73.us-west-2.compute.internal container=telemeter-client container exited with code 2 (Error): 
Sep 19 04:46:01.371 E ns/openshift-monitoring pod/telemeter-client-d5798f6df-4n4bv node/ip-10-0-129-73.us-west-2.compute.internal container=reload container exited with code 2 (Error): 
Sep 19 04:46:01.767 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-129-73.us-west-2.compute.internal container=config-reloader container exited with code 2 (Error): 2020/09/19 04:43:50 Watching directory: "/etc/alertmanager/config"\n
Sep 19 04:46:01.767 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-129-73.us-west-2.compute.internal container=alertmanager-proxy container exited with code 2 (Error): 2020/09/19 04:43:55 provider.go:117: Defaulting client-id to system:serviceaccount:openshift-monitoring:alertmanager-main\n2020/09/19 04:43:55 provider.go:122: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token\n2020/09/19 04:43:55 provider.go:310: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.\n2020/09/19 04:43:55 oauthproxy.go:200: mapping path "/" => upstream "http://localhost:9093/"\n2020/09/19 04:43:55 oauthproxy.go:221: compiled skip-auth-regex => "^/metrics"\n2020/09/19 04:43:55 oauthproxy.go:227: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:alertmanager-main\n2020/09/19 04:43:55 oauthproxy.go:237: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> refresh:disabled\n2020/09/19 04:43:55 http.go:106: HTTPS: listening on [::]:9095\n2020/09/19 04:44:03 server.go:3012: http: TLS handshake error from 10.131.0.15:60774: remote error: tls: error decrypting message\n2020/09/19 04:44:08 server.go:3012: http: TLS handshake error from 10.131.0.15:60846: remote error: tls: error decrypting message\n2020/09/19 04:44:13 server.go:3012: http: TLS handshake error from 10.131.0.15:60940: remote error: tls: error decrypting message\n2020/09/19 04:44:18 server.go:3012: http: TLS handshake error from 10.131.0.15:32828: remote error: tls: error decrypting message\n2020/09/19 04:44:23 server.go:3012: http: TLS handshake error from 10.131.0.15:32912: remote error: tls: error decrypting message\n2020/09/19 04:44:28 server.go:3012: http: TLS handshake error from 10.131.0.15:32984: remote error: tls: error decrypting message\n
Sep 19 04:46:24.749 E ns/openshift-monitoring pod/prometheus-k8s-0 node/ip-10-0-158-47.us-west-2.compute.internal container=prometheus container exited with code 1 (Error): =info ts=2020-09-19T04:46:14.074Z caller=main.go:332 fd_limits="(soft=1048576, hard=1048576)"\nlevel=info ts=2020-09-19T04:46:14.074Z caller=main.go:333 vm_limits="(soft=unlimited, hard=unlimited)"\nlevel=info ts=2020-09-19T04:46:14.082Z caller=main.go:652 msg="Starting TSDB ..."\nlevel=info ts=2020-09-19T04:46:14.082Z caller=web.go:448 component=web msg="Start listening for connections" address=127.0.0.1:9090\nlevel=info ts=2020-09-19T04:46:14.089Z caller=main.go:667 fs_type=XFS_SUPER_MAGIC\nlevel=info ts=2020-09-19T04:46:14.089Z caller=main.go:668 msg="TSDB started"\nlevel=info ts=2020-09-19T04:46:14.089Z caller=main.go:738 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml\nlevel=info ts=2020-09-19T04:46:14.089Z caller=main.go:521 msg="Stopping scrape discovery manager..."\nlevel=info ts=2020-09-19T04:46:14.089Z caller=main.go:535 msg="Stopping notify discovery manager..."\nlevel=info ts=2020-09-19T04:46:14.089Z caller=main.go:557 msg="Stopping scrape manager..."\nlevel=info ts=2020-09-19T04:46:14.089Z caller=main.go:531 msg="Notify discovery manager stopped"\nlevel=info ts=2020-09-19T04:46:14.089Z caller=manager.go:776 component="rule manager" msg="Stopping rule manager..."\nlevel=info ts=2020-09-19T04:46:14.089Z caller=main.go:517 msg="Scrape discovery manager stopped"\nlevel=info ts=2020-09-19T04:46:14.089Z caller=manager.go:782 component="rule manager" msg="Rule manager stopped"\nlevel=info ts=2020-09-19T04:46:14.089Z caller=main.go:551 msg="Scrape manager stopped"\nlevel=info ts=2020-09-19T04:46:14.090Z caller=notifier.go:602 component=notifier msg="Stopping notification manager..."\nlevel=info ts=2020-09-19T04:46:14.090Z caller=main.go:722 msg="Notifier manager stopped"\nlevel=error ts=2020-09-19T04:46:14.090Z caller=main.go:731 err="error loading config from \"/etc/prometheus/config_out/prometheus.env.yaml\": couldn't load configuration (--config.file=\"/etc/prometheus/config_out/prometheus.env.yaml\"): open /etc/prometheus/config_out/prometheus.env.yaml: no such file or directory"\n
Sep 19 04:46:33.250 E ns/openshift-monitoring pod/grafana-fcbdf6fb7-srnk5 node/ip-10-0-138-182.us-west-2.compute.internal container=grafana-proxy container exited with code 2 (Error): 
Sep 19 04:46:41.834 E ns/openshift-multus pod/multus-p6bq8 node/ip-10-0-130-52.us-west-2.compute.internal container=kube-multus container exited with code 137 (Error): 
Sep 19 04:46:43.633 E ns/openshift-sdn pod/sdn-controller-dsq5p node/ip-10-0-130-52.us-west-2.compute.internal container=sdn-controller container exited with code 137 (Error): I0919 04:17:29.806666       1 leaderelection.go:205] attempting to acquire leader lease  openshift-sdn/openshift-network-controller...\n
Sep 19 04:46:44.189 E ns/openshift-monitoring pod/alertmanager-main-0 node/ip-10-0-158-47.us-west-2.compute.internal container=config-reloader container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:46:44.189 E ns/openshift-monitoring pod/alertmanager-main-0 node/ip-10-0-158-47.us-west-2.compute.internal container=alertmanager-proxy container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:46:44.189 E ns/openshift-monitoring pod/alertmanager-main-0 node/ip-10-0-158-47.us-west-2.compute.internal container=alertmanager container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:46:44.356 E ns/openshift-sdn pod/ovs-bqm4m node/ip-10-0-138-182.us-west-2.compute.internal container=openvswitch container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:46:49.384 E ns/openshift-sdn pod/sdn-ppq68 node/ip-10-0-138-182.us-west-2.compute.internal container=sdn container exited with code 255 (Error): robin.go:310] LoadBalancerRR: Setting endpoints for openshift-monitoring/alertmanager-operated:mesh-udp to [10.128.2.27:9094 10.129.2.24:9094]\nI0919 04:46:42.459771   67000 roundrobin.go:240] Delete endpoint 10.131.0.18:9094 for service "openshift-monitoring/alertmanager-operated:mesh-udp"\nI0919 04:46:42.459785   67000 roundrobin.go:310] LoadBalancerRR: Setting endpoints for openshift-monitoring/alertmanager-operated:mesh-tcp to [10.128.2.27:9094 10.129.2.24:9094]\nI0919 04:46:42.459797   67000 roundrobin.go:240] Delete endpoint 10.131.0.18:9094 for service "openshift-monitoring/alertmanager-operated:mesh-tcp"\nI0919 04:46:42.461255   67000 roundrobin.go:310] LoadBalancerRR: Setting endpoints for openshift-monitoring/alertmanager-main:web to [10.128.2.27:9095 10.129.2.24:9095]\nI0919 04:46:42.461285   67000 roundrobin.go:240] Delete endpoint 10.131.0.18:9095 for service "openshift-monitoring/alertmanager-main:web"\nI0919 04:46:42.604968   67000 proxy.go:334] hybrid proxy: mainProxy.syncProxyRules complete\nI0919 04:46:42.667112   67000 proxier.go:367] userspace proxy: processing 0 service events\nI0919 04:46:42.667131   67000 proxier.go:346] userspace syncProxyRules took 62.141943ms\nI0919 04:46:42.667141   67000 proxy.go:337] hybrid proxy: unidlingProxy.syncProxyRules complete\nI0919 04:46:42.667153   67000 proxy.go:331] hybrid proxy: syncProxyRules start\nI0919 04:46:42.815928   67000 proxy.go:334] hybrid proxy: mainProxy.syncProxyRules complete\nI0919 04:46:42.878730   67000 proxier.go:367] userspace proxy: processing 0 service events\nI0919 04:46:42.878748   67000 proxier.go:346] userspace syncProxyRules took 62.801163ms\nI0919 04:46:42.878757   67000 proxy.go:337] hybrid proxy: unidlingProxy.syncProxyRules complete\nI0919 04:46:48.304939   67000 ovs.go:169] Error executing ovs-ofctl: ovs-ofctl: /var/run/openvswitch/br0.mgmt: failed to open socket (Connection refused)\nF0919 04:46:48.305002   67000 healthcheck.go:82] SDN healthcheck detected OVS server change, restarting: OVS reinitialization required: plugin is not setup\n
Sep 19 04:46:55.904 E ns/openshift-dns pod/dns-default-bxkkb node/ip-10-0-158-47.us-west-2.compute.internal container=dns container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:46:55.904 E ns/openshift-dns pod/dns-default-bxkkb node/ip-10-0-158-47.us-west-2.compute.internal container=dns-node-resolver container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:47:22.557 E ns/openshift-multus pod/multus-5n556 node/ip-10-0-129-73.us-west-2.compute.internal container=kube-multus container exited with code 137 (Error): 
Sep 19 04:47:24.799 E ns/openshift-apiserver pod/apiserver-zqf47 node/ip-10-0-129-244.us-west-2.compute.internal container=openshift-apiserver container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:47:38.248 E ns/openshift-sdn pod/ovs-77h7h node/ip-10-0-130-52.us-west-2.compute.internal container=openvswitch container exited with code 137 (Error): deletes)\n2020-09-19T04:46:48.337Z|00502|bridge|INFO|bridge br0: deleted interface veth074d0b2a on port 65\n2020-09-19T04:46:48.344Z|00503|bridge|WARN|could not open network device veth47b65d2a (No such device)\n2020-09-19T04:46:48.347Z|00504|bridge|WARN|could not open network device veth465cbb40 (No such device)\n2020-09-19T04:46:48.352Z|00505|bridge|WARN|could not open network device veth47b65d2a (No such device)\n2020-09-19T04:46:48.359Z|00506|bridge|WARN|could not open network device veth465cbb40 (No such device)\n2020-09-19T04:46:48.394Z|00507|bridge|WARN|could not open network device veth47b65d2a (No such device)\n2020-09-19T04:46:48.396Z|00508|bridge|WARN|could not open network device veth465cbb40 (No such device)\n2020-09-19T04:46:48.486Z|00509|bridge|WARN|could not open network device veth47b65d2a (No such device)\n2020-09-19T04:46:48.489Z|00510|bridge|WARN|could not open network device veth465cbb40 (No such device)\n2020-09-19T04:47:05.785Z|00511|bridge|WARN|could not open network device veth47b65d2a (No such device)\n2020-09-19T04:47:05.787Z|00512|bridge|WARN|could not open network device veth465cbb40 (No such device)\n2020-09-19T04:47:05.798Z|00513|bridge|WARN|could not open network device veth47b65d2a (No such device)\n2020-09-19T04:47:05.803Z|00514|bridge|WARN|could not open network device veth465cbb40 (No such device)\n2020-09-19T04:47:05.809Z|00515|bridge|WARN|could not open network device veth47b65d2a (No such device)\n2020-09-19T04:47:05.812Z|00516|bridge|INFO|bridge br0: added interface veth2a6d7ead on port 66\n2020-09-19T04:47:05.814Z|00517|bridge|WARN|could not open network device veth465cbb40 (No such device)\n2020-09-19T04:47:05.820Z|00518|bridge|WARN|could not open network device veth47b65d2a (No such device)\n2020-09-19T04:47:05.822Z|00519|bridge|WARN|could not open network device veth465cbb40 (No such device)\n2020-09-19T04:47:05.842Z|00520|connmgr|INFO|br0<->unix#1009: 5 flow_mods in the last 0 s (5 adds)\n2020-09-19T04:47:05.878Z|00521|connmgr|INFO|br0<->unix#1012: 2 flow_mods in the last 0 s (2 deletes)\n
Sep 19 04:47:47.275 E ns/openshift-sdn pod/sdn-zrwcb node/ip-10-0-130-52.us-west-2.compute.internal container=sdn container exited with code 255 (Error): 37.060395   73062 proxy.go:331] hybrid proxy: syncProxyRules start\nI0919 04:47:37.226673   73062 proxy.go:334] hybrid proxy: mainProxy.syncProxyRules complete\nI0919 04:47:37.284381   73062 proxier.go:367] userspace proxy: processing 0 service events\nI0919 04:47:37.284403   73062 proxier.go:346] userspace syncProxyRules took 57.708529ms\nI0919 04:47:37.284412   73062 proxy.go:337] hybrid proxy: unidlingProxy.syncProxyRules complete\nI0919 04:47:39.841100   73062 ovs.go:169] Error executing ovs-vsctl: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (Connection refused)\nI0919 04:47:39.844553   73062 ovs.go:169] Error executing ovs-vsctl: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (Connection refused)\nI0919 04:47:39.853951   73062 ovs.go:169] Error executing ovs-vsctl: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (Connection refused)\nI0919 04:47:39.890973   73062 ovs.go:169] Error executing ovs-vsctl: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (Connection refused)\nI0919 04:47:46.223553   73062 ovs.go:169] Error executing ovs-vsctl: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (Connection refused)\nI0919 04:47:46.229100   73062 ovs.go:169] Error executing ovs-vsctl: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (Connection refused)\nI0919 04:47:46.245817   73062 ovs.go:169] Error executing ovs-vsctl: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (Connection refused)\nI0919 04:47:46.283362   73062 ovs.go:169] Error executing ovs-vsctl: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (Connection refused)\nI0919 04:47:47.121353   73062 ovs.go:169] Error executing ovs-ofctl: ovs-ofctl: /var/run/openvswitch/br0.mgmt: failed to open socket (Connection refused)\nF0919 04:47:47.121394   73062 healthcheck.go:82] SDN healthcheck detected OVS server change, restarting: OVS reinitialization required: plugin is not setup\n
Sep 19 04:48:26.388 E ns/openshift-service-ca pod/apiservice-cabundle-injector-6f46f46b89-p756w node/ip-10-0-130-52.us-west-2.compute.internal container=apiservice-cabundle-injector-controller container exited with code 255 (Error): 
Sep 19 04:48:34.069 E ns/openshift-sdn pod/ovs-5k9zr node/ip-10-0-129-244.us-west-2.compute.internal container=openvswitch container exited with code 137 (Error): last 0 s (2 deletes)\n2020-09-19T04:47:22.882Z|00494|connmgr|INFO|br0<->unix#1140: 4 flow_mods in the last 0 s (4 deletes)\n2020-09-19T04:47:22.904Z|00495|bridge|INFO|bridge br0: deleted interface veth6713b580 on port 52\n2020-09-19T04:47:22.911Z|00496|bridge|WARN|could not open network device veth4f052031 (No such device)\n2020-09-19T04:47:22.923Z|00497|bridge|WARN|could not open network device veth4f052031 (No such device)\n2020-09-19T04:47:22.963Z|00498|bridge|WARN|could not open network device veth4f052031 (No such device)\n2020-09-19T04:47:23.065Z|00499|bridge|WARN|could not open network device veth4f052031 (No such device)\n2020-09-19T04:47:28.804Z|00500|connmgr|INFO|br0<->unix#1146: 2 flow_mods in the last 0 s (2 deletes)\n2020-09-19T04:47:28.835Z|00501|connmgr|INFO|br0<->unix#1149: 4 flow_mods in the last 0 s (4 deletes)\n2020-09-19T04:47:28.874Z|00502|bridge|INFO|bridge br0: deleted interface veth3dcff9ec on port 71\n2020-09-19T04:47:28.880Z|00503|bridge|WARN|could not open network device veth4f052031 (No such device)\n2020-09-19T04:47:28.886Z|00504|bridge|WARN|could not open network device veth4f052031 (No such device)\n2020-09-19T04:47:28.934Z|00505|bridge|WARN|could not open network device veth4f052031 (No such device)\n2020-09-19T04:47:29.010Z|00506|bridge|WARN|could not open network device veth4f052031 (No such device)\n2020-09-19T04:47:41.516Z|00507|bridge|WARN|could not open network device veth4f052031 (No such device)\n2020-09-19T04:47:41.534Z|00508|bridge|WARN|could not open network device veth4f052031 (No such device)\n2020-09-19T04:47:41.549Z|00509|bridge|INFO|bridge br0: added interface vethfaa699e4 on port 76\n2020-09-19T04:47:41.551Z|00510|bridge|WARN|could not open network device veth4f052031 (No such device)\n2020-09-19T04:47:41.556Z|00511|bridge|WARN|could not open network device veth4f052031 (No such device)\n2020-09-19T04:47:41.577Z|00512|connmgr|INFO|br0<->unix#1152: 5 flow_mods in the last 0 s (5 adds)\n2020-09-19T04:47:41.613Z|00513|connmgr|INFO|br0<->unix#1155: 2 flow_mods in the last 0 s (2 deletes)\n
Sep 19 04:48:40.575 E ns/openshift-multus pod/multus-pndtt node/ip-10-0-138-182.us-west-2.compute.internal container=kube-multus container exited with code 137 (Error): 
Sep 19 04:48:45.098 E ns/openshift-sdn pod/sdn-vdbj5 node/ip-10-0-129-244.us-west-2.compute.internal container=sdn container exited with code 255 (Error): endpoints for openshift-sdn/sdn:metrics to [10.0.129.244:9101 10.0.129.73:9101 10.0.130.52:9101 10.0.138.182:9101 10.0.145.20:9101 10.0.158.47:9101]\nI0919 04:47:56.883692   74893 roundrobin.go:240] Delete endpoint 10.0.130.52:9101 for service "openshift-sdn/sdn:metrics"\nI0919 04:47:56.883739   74893 proxy.go:331] hybrid proxy: syncProxyRules start\nI0919 04:47:57.025168   74893 proxy.go:334] hybrid proxy: mainProxy.syncProxyRules complete\nI0919 04:47:57.118248   74893 proxier.go:367] userspace proxy: processing 0 service events\nI0919 04:47:57.118269   74893 proxier.go:346] userspace syncProxyRules took 93.080913ms\nI0919 04:47:57.118279   74893 proxy.go:337] hybrid proxy: unidlingProxy.syncProxyRules complete\nI0919 04:47:59.027770   74893 roundrobin.go:310] LoadBalancerRR: Setting endpoints for openshift-apiserver/api:https to [10.128.0.86:8443 10.129.0.75:8443 10.130.0.58:8443]\nI0919 04:47:59.027913   74893 roundrobin.go:240] Delete endpoint 10.129.0.75:8443 for service "openshift-apiserver/api:https"\nI0919 04:47:59.028022   74893 proxy.go:331] hybrid proxy: syncProxyRules start\nI0919 04:47:59.235813   74893 proxy.go:334] hybrid proxy: mainProxy.syncProxyRules complete\nI0919 04:47:59.291344   74893 proxier.go:367] userspace proxy: processing 0 service events\nI0919 04:47:59.291361   74893 proxier.go:346] userspace syncProxyRules took 55.531459ms\nI0919 04:47:59.291369   74893 proxy.go:337] hybrid proxy: unidlingProxy.syncProxyRules complete\nI0919 04:48:29.291504   74893 proxy.go:331] hybrid proxy: syncProxyRules start\nI0919 04:48:29.428744   74893 proxy.go:334] hybrid proxy: mainProxy.syncProxyRules complete\nI0919 04:48:29.484397   74893 proxier.go:367] userspace proxy: processing 0 service events\nI0919 04:48:29.484414   74893 proxier.go:346] userspace syncProxyRules took 55.65218ms\nI0919 04:48:29.484422   74893 proxy.go:337] hybrid proxy: unidlingProxy.syncProxyRules complete\nF0919 04:48:44.030897   74893 healthcheck.go:82] SDN healthcheck detected OVS server change, restarting: timed out waiting for the condition\n
Sep 19 04:49:33.440 E ns/openshift-sdn pod/sdn-jqrqj node/ip-10-0-145-20.us-west-2.compute.internal container=sdn container exited with code 255 (Error): .0.129.244:9101 for service "openshift-sdn/sdn:metrics"\nI0919 04:48:45.063769   76970 proxy.go:331] hybrid proxy: syncProxyRules start\nI0919 04:48:45.249206   76970 proxy.go:334] hybrid proxy: mainProxy.syncProxyRules complete\nI0919 04:48:45.311816   76970 proxier.go:367] userspace proxy: processing 0 service events\nI0919 04:48:45.311834   76970 proxier.go:346] userspace syncProxyRules took 62.611424ms\nI0919 04:48:45.311843   76970 proxy.go:337] hybrid proxy: unidlingProxy.syncProxyRules complete\nI0919 04:48:52.768275   76970 roundrobin.go:310] LoadBalancerRR: Setting endpoints for openshift-sdn/sdn:metrics to [10.0.129.244:9101 10.0.129.73:9101 10.0.130.52:9101 10.0.138.182:9101 10.0.145.20:9101 10.0.158.47:9101]\nI0919 04:48:52.768307   76970 roundrobin.go:240] Delete endpoint 10.0.129.244:9101 for service "openshift-sdn/sdn:metrics"\nI0919 04:48:52.768349   76970 proxy.go:331] hybrid proxy: syncProxyRules start\nI0919 04:48:52.914285   76970 proxy.go:334] hybrid proxy: mainProxy.syncProxyRules complete\nI0919 04:48:52.971699   76970 proxier.go:367] userspace proxy: processing 0 service events\nI0919 04:48:52.971717   76970 proxier.go:346] userspace syncProxyRules took 57.411949ms\nI0919 04:48:52.971725   76970 proxy.go:337] hybrid proxy: unidlingProxy.syncProxyRules complete\nI0919 04:49:22.971856   76970 proxy.go:331] hybrid proxy: syncProxyRules start\nI0919 04:49:23.111179   76970 proxy.go:334] hybrid proxy: mainProxy.syncProxyRules complete\nI0919 04:49:23.168173   76970 proxier.go:367] userspace proxy: processing 0 service events\nI0919 04:49:23.168191   76970 proxier.go:346] userspace syncProxyRules took 56.99149ms\nI0919 04:49:23.168199   76970 proxy.go:337] hybrid proxy: unidlingProxy.syncProxyRules complete\nI0919 04:49:33.277220   76970 ovs.go:169] Error executing ovs-ofctl: ovs-ofctl: /var/run/openvswitch/br0.mgmt: failed to open socket (Connection refused)\nF0919 04:49:33.277251   76970 healthcheck.go:82] SDN healthcheck detected OVS server change, restarting: OVS reinitialization required: plugin is not setup\n
Sep 19 04:50:21.901 E ns/openshift-sdn pod/ovs-hvg92 node/ip-10-0-129-73.us-west-2.compute.internal container=openvswitch container exited with code 137 (Error): 9-19T04:46:24.276Z|00178|connmgr|INFO|br0<->unix#471: 1 flow_mods in the last 0 s (1 adds)\n2020-09-19T04:46:24.299Z|00179|connmgr|INFO|br0<->unix#474: 3 flow_mods in the last 0 s (3 adds)\n2020-09-19T04:46:24.327Z|00180|connmgr|INFO|br0<->unix#477: 1 flow_mods in the last 0 s (1 adds)\n2020-09-19T04:46:24.351Z|00181|connmgr|INFO|br0<->unix#480: 3 flow_mods in the last 0 s (3 adds)\n2020-09-19T04:46:24.381Z|00182|connmgr|INFO|br0<->unix#483: 1 flow_mods in the last 0 s (1 adds)\n2020-09-19T04:46:24.411Z|00183|connmgr|INFO|br0<->unix#486: 3 flow_mods in the last 0 s (3 adds)\n2020-09-19T04:46:24.433Z|00184|connmgr|INFO|br0<->unix#489: 1 flow_mods in the last 0 s (1 adds)\n2020-09-19T04:46:39.579Z|00185|bridge|WARN|could not open network device veth33bbc6fc (No such device)\n2020-09-19T04:46:39.587Z|00186|bridge|WARN|could not open network device veth33bbc6fc (No such device)\n2020-09-19T04:46:39.598Z|00187|bridge|WARN|could not open network device veth33bbc6fc (No such device)\n2020-09-19T04:46:39.602Z|00188|bridge|INFO|bridge br0: added interface veth98599570 on port 28\n2020-09-19T04:46:39.609Z|00189|bridge|WARN|could not open network device veth33bbc6fc (No such device)\n2020-09-19T04:46:39.632Z|00190|connmgr|INFO|br0<->unix#492: 5 flow_mods in the last 0 s (5 adds)\n2020-09-19T04:46:39.668Z|00191|connmgr|INFO|br0<->unix#495: 2 flow_mods in the last 0 s (2 deletes)\n2020-09-19T04:47:48.361Z|00192|bridge|WARN|could not open network device veth33bbc6fc (No such device)\n2020-09-19T04:47:48.375Z|00001|netdev_linux(revalidator3)|INFO|ioctl(SIOCGIFINDEX) on veth6ad774a7 device failed: No such device\n2020-09-19T04:47:48.375Z|00002|netdev_tc_offloads(revalidator3)|ERR|dump_create: failed to get ifindex for veth6ad774a7: No such device\n2020-09-19T04:47:48.377Z|00193|bridge|INFO|bridge br0: deleted interface veth6ad774a7 on port 25\n2020-09-19T04:47:48.380Z|00194|bridge|WARN|could not open network device veth6ad774a7 (No such device)\n2020-09-19T04:47:48.383Z|00195|bridge|WARN|could not open network device veth33bbc6fc (No such device)\n
Sep 19 04:50:23.931 E ns/openshift-sdn pod/sdn-vz24x node/ip-10-0-129-73.us-west-2.compute.internal container=sdn container exited with code 255 (Error): 0.0.145.20:9101 for service "openshift-sdn/sdn:metrics"\nI0919 04:49:33.406817   43267 proxy.go:331] hybrid proxy: syncProxyRules start\nI0919 04:49:33.572210   43267 proxy.go:334] hybrid proxy: mainProxy.syncProxyRules complete\nI0919 04:49:33.635106   43267 proxier.go:367] userspace proxy: processing 0 service events\nI0919 04:49:33.635131   43267 proxier.go:346] userspace syncProxyRules took 62.898478ms\nI0919 04:49:33.635141   43267 proxy.go:337] hybrid proxy: unidlingProxy.syncProxyRules complete\nI0919 04:49:40.301398   43267 roundrobin.go:310] LoadBalancerRR: Setting endpoints for openshift-sdn/sdn:metrics to [10.0.129.244:9101 10.0.129.73:9101 10.0.130.52:9101 10.0.138.182:9101 10.0.145.20:9101 10.0.158.47:9101]\nI0919 04:49:40.301505   43267 roundrobin.go:240] Delete endpoint 10.0.145.20:9101 for service "openshift-sdn/sdn:metrics"\nI0919 04:49:40.301611   43267 proxy.go:331] hybrid proxy: syncProxyRules start\nI0919 04:49:40.464223   43267 proxy.go:334] hybrid proxy: mainProxy.syncProxyRules complete\nI0919 04:49:40.526924   43267 proxier.go:367] userspace proxy: processing 0 service events\nI0919 04:49:40.526944   43267 proxier.go:346] userspace syncProxyRules took 62.698926ms\nI0919 04:49:40.526955   43267 proxy.go:337] hybrid proxy: unidlingProxy.syncProxyRules complete\nI0919 04:50:10.527176   43267 proxy.go:331] hybrid proxy: syncProxyRules start\nI0919 04:50:10.678300   43267 proxy.go:334] hybrid proxy: mainProxy.syncProxyRules complete\nI0919 04:50:10.740731   43267 proxier.go:367] userspace proxy: processing 0 service events\nI0919 04:50:10.740755   43267 proxier.go:346] userspace syncProxyRules took 62.431268ms\nI0919 04:50:10.740768   43267 proxy.go:337] hybrid proxy: unidlingProxy.syncProxyRules complete\nI0919 04:50:23.772142   43267 ovs.go:169] Error executing ovs-ofctl: ovs-ofctl: /var/run/openvswitch/br0.mgmt: failed to open socket (Connection refused)\nF0919 04:50:23.772183   43267 healthcheck.go:82] SDN healthcheck detected OVS server change, restarting: OVS reinitialization required: plugin is not setup\n
Sep 19 04:50:56.652 E ns/openshift-machine-config-operator pod/machine-config-operator-57567677cc-gtj8x node/ip-10-0-145-20.us-west-2.compute.internal container=machine-config-operator container exited with code 2 (Error): 
Sep 19 04:54:16.909 E ns/openshift-machine-config-operator pod/machine-config-controller-685b476f64-tcc2p node/ip-10-0-129-244.us-west-2.compute.internal container=machine-config-controller container exited with code 2 (Error): 
Sep 19 04:56:14.181 E ns/openshift-machine-config-operator pod/machine-config-server-f8p55 node/ip-10-0-129-244.us-west-2.compute.internal container=machine-config-server container exited with code 2 (Error): 
Sep 19 04:56:26.529 E ns/openshift-machine-config-operator pod/machine-config-operator-5d88c8ddbb-t94wt node/ip-10-0-130-52.us-west-2.compute.internal container=machine-config-operator container exited with code 2 (Error): 
Sep 19 04:56:29.111 E ns/openshift-service-ca pod/service-serving-cert-signer-5d594c8cbd-q26d9 node/ip-10-0-130-52.us-west-2.compute.internal container=service-serving-cert-signer-controller container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:56:30.809 E ns/openshift-operator-lifecycle-manager pod/packageserver-6f6c4b55cf-d925c node/ip-10-0-129-244.us-west-2.compute.internal container=packageserver container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 04:56:33.517 E ns/openshift-service-ca pod/configmap-cabundle-injector-7cddd89564-ksx49 node/ip-10-0-130-52.us-west-2.compute.internal container=configmap-cabundle-injector-controller container exited with code 255 (Error): 
Sep 19 04:56:37.515 E ns/openshift-machine-config-operator pod/machine-config-server-96htx node/ip-10-0-130-52.us-west-2.compute.internal container=machine-config-server container exited with code 2 (Error): 
Sep 19 04:56:40.718 E ns/openshift-cluster-node-tuning-operator pod/cluster-node-tuning-operator-844876d54-qclf6 node/ip-10-0-130-52.us-west-2.compute.internal container=cluster-node-tuning-operator container exited with code 255 (Error): t\nI0919 04:46:36.760363       1 status.go:25] syncOperatorStatus()\nI0919 04:46:36.769916       1 tuned_controller.go:187] syncServiceAccount()\nI0919 04:46:36.770045       1 tuned_controller.go:214] syncClusterRole()\nI0919 04:46:36.804152       1 tuned_controller.go:245] syncClusterRoleBinding()\nI0919 04:46:36.848911       1 tuned_controller.go:276] syncClusterConfigMap()\nI0919 04:46:36.854494       1 tuned_controller.go:276] syncClusterConfigMap()\nI0919 04:46:36.858969       1 tuned_controller.go:313] syncDaemonSet()\nI0919 04:46:39.022597       1 tuned_controller.go:432] Reconciling Tuned openshift-cluster-node-tuning-operator/default\nI0919 04:46:39.022617       1 status.go:25] syncOperatorStatus()\nI0919 04:46:39.031910       1 tuned_controller.go:187] syncServiceAccount()\nI0919 04:46:39.032075       1 tuned_controller.go:214] syncClusterRole()\nI0919 04:46:39.067980       1 tuned_controller.go:245] syncClusterRoleBinding()\nI0919 04:46:39.101576       1 tuned_controller.go:276] syncClusterConfigMap()\nI0919 04:46:39.106192       1 tuned_controller.go:276] syncClusterConfigMap()\nI0919 04:46:39.110269       1 tuned_controller.go:313] syncDaemonSet()\nI0919 04:46:41.400243       1 tuned_controller.go:432] Reconciling Tuned openshift-cluster-node-tuning-operator/default\nI0919 04:46:41.400266       1 status.go:25] syncOperatorStatus()\nI0919 04:46:41.409905       1 tuned_controller.go:187] syncServiceAccount()\nI0919 04:46:41.410030       1 tuned_controller.go:214] syncClusterRole()\nI0919 04:46:41.448869       1 tuned_controller.go:245] syncClusterRoleBinding()\nI0919 04:46:41.486040       1 tuned_controller.go:276] syncClusterConfigMap()\nI0919 04:46:41.490335       1 tuned_controller.go:276] syncClusterConfigMap()\nI0919 04:46:41.494861       1 tuned_controller.go:313] syncDaemonSet()\nW0919 04:54:50.180791       1 reflector.go:289] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:204: watch of *v1.ConfigMap ended with: too old resource version: 34038 (36982)\nF0919 04:56:28.275065       1 main.go:82] <nil>\n
Sep 19 04:56:41.314 E ns/openshift-console-operator pod/console-operator-7d5f9cbd6c-lss7t node/ip-10-0-130-52.us-west-2.compute.internal container=console-operator container exited with code 255 (Error): ld resource version: 32913 (35986)\nW0919 04:53:37.983193       1 reflector.go:289] k8s.io/client-go/informers/factory.go:133: watch of *v1.ConfigMap ended with: too old resource version: 32913 (36554)\nW0919 04:55:14.981502       1 reflector.go:289] k8s.io/client-go/informers/factory.go:133: watch of *v1.ConfigMap ended with: too old resource version: 32913 (37086)\nW0919 04:55:28.884778       1 reflector.go:289] github.com/openshift/client-go/oauth/informers/externalversions/factory.go:101: watch of *v1.OAuthClient ended with: The resourceVersion for the provided watch is too old.\nE0919 04:56:23.860954       1 status.go:71] DeploymentAvailable FailedUpdate 1 replicas ready at version 4.2.36\nI0919 04:56:23.876331       1 status_controller.go:165] clusteroperator/console diff {"status":{"conditions":[{"lastTransitionTime":"2020-09-19T04:43:09Z","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-09-19T04:46:32Z","reason":"AsExpected","status":"False","type":"Progressing"},{"lastTransitionTime":"2020-09-19T04:56:23Z","message":"DeploymentAvailable: 1 replicas ready at version 4.2.36","reason":"DeploymentAvailableFailedUpdate","status":"False","type":"Available"},{"lastTransitionTime":"2020-09-19T04:20:02Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}\nI0919 04:56:23.889122       1 event.go:209] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"5833d1c3-fa2f-11ea-8f7e-0603b42f4bc7", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/console changed: Available changed from True to False ("DeploymentAvailable: 1 replicas ready at version 4.2.36")\nE0919 04:56:24.370843       1 status.go:71] DeploymentAvailable FailedUpdate 1 replicas ready at version 4.2.36\nI0919 04:56:28.482965       1 cmd.go:79] Received SIGTERM or SIGINT signal, shutting down controller.\nF0919 04:56:28.483017       1 leaderelection.go:66] leaderelection lost\n
Sep 19 04:56:49.545 E kube-apiserver Kube API started failing: Get https://api.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com:6443/api/v1/namespaces/kube-system?timeout=5s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Sep 19 04:56:53.553 E ns/openshift-machine-config-operator pod/machine-config-server-7vpv7 node/ip-10-0-145-20.us-west-2.compute.internal container=machine-config-server container exited with code 2 (Error): 
Sep 19 04:57:56.552 - 14s   E openshift-apiserver OpenShift API is not responding to GET requests
Sep 19 04:58:16.455 E kube-apiserver failed contacting the API: Get https://api.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com:6443/apis/config.openshift.io/v1/clusterversions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dversion&resourceVersion=36607&timeout=5m23s&timeoutSeconds=323&watch=true: dial tcp 44.239.232.208:6443: connect: connection refused
Sep 19 04:58:45.879 E clusterversion/version changed Failing to True: ClusterOperatorNotAvailable: Cluster operator machine-config is still updating
Sep 19 04:59:45.750 E ns/openshift-image-registry pod/node-ca-5bshd node/ip-10-0-129-73.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 04:59:46.549 E ns/openshift-monitoring pod/node-exporter-vzzfx node/ip-10-0-129-73.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 04:59:46.950 E ns/openshift-dns pod/dns-default-flpzk node/ip-10-0-129-73.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 04:59:47.749 E ns/openshift-multus pod/multus-z6q2w node/ip-10-0-129-73.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 04:59:48.153 E ns/openshift-sdn pod/ovs-jg9pp node/ip-10-0-129-73.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 04:59:48.549 E ns/openshift-machine-config-operator pod/machine-config-daemon-888k4 node/ip-10-0-129-73.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 04:59:48.949 E ns/openshift-cluster-node-tuning-operator pod/tuned-lhl4k node/ip-10-0-129-73.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 04:59:56.308 E ns/openshift-monitoring pod/node-exporter-5tmg2 node/ip-10-0-130-52.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 04:59:56.708 E ns/openshift-controller-manager pod/controller-manager-8lpwf node/ip-10-0-130-52.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 04:59:57.106 E ns/openshift-multus pod/multus-admission-controller-l6kh2 node/ip-10-0-130-52.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 04:59:57.909 E ns/openshift-apiserver pod/apiserver-vh4l4 node/ip-10-0-130-52.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 04:59:58.307 E ns/openshift-machine-config-operator pod/machine-config-daemon-j4d8s node/ip-10-0-130-52.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 04:59:58.708 E ns/openshift-machine-config-operator pod/machine-config-server-2zbx9 node/ip-10-0-130-52.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 04:59:59.108 E ns/openshift-cluster-node-tuning-operator pod/tuned-99dvk node/ip-10-0-130-52.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:00:00.311 E ns/openshift-multus pod/multus-pv67t node/ip-10-0-130-52.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:00:00.707 E ns/openshift-sdn pod/sdn-controller-x5tgp node/ip-10-0-130-52.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:00:01.108 E ns/openshift-image-registry pod/node-ca-klr2n node/ip-10-0-130-52.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:00:01.509 E ns/openshift-dns pod/dns-default-pgfxf node/ip-10-0-130-52.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:00:01.908 E ns/openshift-sdn pod/ovs-9dbxh node/ip-10-0-130-52.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:00:20.222 E ns/openshift-monitoring pod/openshift-state-metrics-587758c87-rs9n2 node/ip-10-0-138-182.us-west-2.compute.internal container=openshift-state-metrics container exited with code 2 (Error): 
Sep 19 05:00:21.422 E ns/openshift-monitoring pod/prometheus-adapter-67cb549dcc-x8gc7 node/ip-10-0-138-182.us-west-2.compute.internal container=prometheus-adapter container exited with code 2 (Error): I0919 04:44:14.722061       1 adapter.go:93] successfully using in-cluster auth\nI0919 04:44:15.544881       1 secure_serving.go:116] Serving securely on [::]:6443\n
Sep 19 05:00:22.403 E clusteroperator/kube-apiserver changed Degraded to True: NodeControllerDegradedMasterNodesReady: NodeControllerDegraded: The master nodes not ready: node "ip-10-0-130-52.us-west-2.compute.internal" not ready since 2020-09-19 04:59:44 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: Missing CNI default network)
Sep 19 05:00:22.412 E clusteroperator/kube-scheduler changed Degraded to True: NodeControllerDegradedMasterNodesReady: NodeControllerDegraded: The master nodes not ready: node "ip-10-0-130-52.us-west-2.compute.internal" not ready since 2020-09-19 04:59:44 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: Missing CNI default network)
Sep 19 05:00:22.416 E clusteroperator/kube-controller-manager changed Degraded to True: NodeControllerDegradedMasterNodesReady: NodeControllerDegraded: The master nodes not ready: node "ip-10-0-130-52.us-west-2.compute.internal" not ready since 2020-09-19 04:59:44 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: Missing CNI default network)
Sep 19 05:00:23.422 E ns/openshift-ingress pod/router-default-79c6f9c6b6-dp5hv node/ip-10-0-138-182.us-west-2.compute.internal container=router container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 05:00:25.176 E clusteroperator/dns changed Degraded to True: NotAllDNSesAvailable: Not all desired DNS DaemonSets available
Sep 19 05:00:36.170 E ns/openshift-kube-apiserver-operator pod/kube-apiserver-operator-57547b4d45-px7l8 node/ip-10-0-129-244.us-west-2.compute.internal container=kube-apiserver-operator container exited with code 255 (Error): 020-09-19 04:59:44 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: Missing CNI default network)" to "StaticPodsDegraded: nodes/ip-10-0-130-52.us-west-2.compute.internal pods/kube-apiserver-ip-10-0-130-52.us-west-2.compute.internal container=\"kube-apiserver-8\" is not ready\nNodeControllerDegraded: The master nodes not ready: node \"ip-10-0-130-52.us-west-2.compute.internal\" not ready since 2020-09-19 04:59:44 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: Missing CNI default network)"\nI0919 05:00:22.371836       1 event.go:209] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"c5cf4d35-fa2e-11ea-8901-02f26ee80463", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-apiserver changed: Degraded changed from False to True ("NodeControllerDegraded: The master nodes not ready: node \"ip-10-0-130-52.us-west-2.compute.internal\" not ready since 2020-09-19 04:59:44 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: Missing CNI default network)")\nI0919 05:00:24.996063       1 event.go:209] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"c5cf4d35-fa2e-11ea-8901-02f26ee80463", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-apiserver changed: Degraded changed from True to False ("NodeControllerDegraded: All master nodes are ready")\nI0919 05:00:32.083166       1 cmd.go:79] Received SIGTERM or SIGINT signal, shutting down controller.\nF0919 05:00:32.083296       1 leaderelection.go:66] leaderelection lost\n
Sep 19 05:00:37.778 E ns/openshift-service-ca pod/service-serving-cert-signer-5d594c8cbd-flsj7 node/ip-10-0-129-244.us-west-2.compute.internal container=service-serving-cert-signer-controller container exited with code 255 (Error): 
Sep 19 05:00:44.571 E ns/openshift-operator-lifecycle-manager pod/packageserver-dffb947f-td65r node/ip-10-0-129-244.us-west-2.compute.internal container=packageserver container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 05:00:48.768 E ns/openshift-service-ca-operator pod/service-ca-operator-6c77dc59c8-lc5t7 node/ip-10-0-129-244.us-west-2.compute.internal container=operator container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 05:01:02.880 E ns/openshift-console pod/downloads-7f6cc7cd54-blqjp node/ip-10-0-129-244.us-west-2.compute.internal container=download-server container exited with code 137 (Error): 0 04:58:14] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 04:58:17] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 04:58:24] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 04:58:27] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 04:58:34] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 04:58:37] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 04:58:44] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 04:58:47] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 04:58:54] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 04:58:57] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 04:59:04] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 04:59:07] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 04:59:14] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 04:59:17] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 04:59:24] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 04:59:27] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 04:59:34] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 04:59:37] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 04:59:44] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 04:59:47] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 04:59:54] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 04:59:57] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 05:00:04] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 05:00:07] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 05:00:14] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 05:00:17] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 05:00:24] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 05:00:27] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 05:00:34] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 05:00:37] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 05:00:44] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 05:00:47] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 05:00:54] "GET / HTTP/1.1" 200 -\n10.129.0.1 - - [19/Sep/2020 05:00:57] "GET / HTTP/1.1" 200 -\n
Sep 19 05:01:06.318 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ip-10-0-129-73.us-west-2.compute.internal container=prometheus container exited with code 1 (Error): =info ts=2020-09-19T05:00:43.568Z caller=main.go:332 fd_limits="(soft=1048576, hard=1048576)"\nlevel=info ts=2020-09-19T05:00:43.568Z caller=main.go:333 vm_limits="(soft=unlimited, hard=unlimited)"\nlevel=info ts=2020-09-19T05:00:43.569Z caller=main.go:652 msg="Starting TSDB ..."\nlevel=info ts=2020-09-19T05:00:43.569Z caller=web.go:448 component=web msg="Start listening for connections" address=127.0.0.1:9090\nlevel=info ts=2020-09-19T05:00:43.579Z caller=main.go:667 fs_type=XFS_SUPER_MAGIC\nlevel=info ts=2020-09-19T05:00:43.579Z caller=main.go:668 msg="TSDB started"\nlevel=info ts=2020-09-19T05:00:43.579Z caller=main.go:738 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml\nlevel=info ts=2020-09-19T05:00:43.579Z caller=main.go:521 msg="Stopping scrape discovery manager..."\nlevel=info ts=2020-09-19T05:00:43.579Z caller=main.go:535 msg="Stopping notify discovery manager..."\nlevel=info ts=2020-09-19T05:00:43.579Z caller=main.go:517 msg="Scrape discovery manager stopped"\nlevel=info ts=2020-09-19T05:00:43.579Z caller=main.go:557 msg="Stopping scrape manager..."\nlevel=info ts=2020-09-19T05:00:43.579Z caller=manager.go:776 component="rule manager" msg="Stopping rule manager..."\nlevel=info ts=2020-09-19T05:00:43.579Z caller=manager.go:782 component="rule manager" msg="Rule manager stopped"\nlevel=info ts=2020-09-19T05:00:43.579Z caller=main.go:551 msg="Scrape manager stopped"\nlevel=info ts=2020-09-19T05:00:43.580Z caller=main.go:531 msg="Notify discovery manager stopped"\nlevel=info ts=2020-09-19T05:00:43.580Z caller=notifier.go:602 component=notifier msg="Stopping notification manager..."\nlevel=info ts=2020-09-19T05:00:43.580Z caller=main.go:722 msg="Notifier manager stopped"\nlevel=error ts=2020-09-19T05:00:43.580Z caller=main.go:731 err="error loading config from \"/etc/prometheus/config_out/prometheus.env.yaml\": couldn't load configuration (--config.file=\"/etc/prometheus/config_out/prometheus.env.yaml\"): open /etc/prometheus/config_out/prometheus.env.yaml: no such file or directory"\n
Sep 19 05:01:44.435 E ns/openshift-operator-lifecycle-manager pod/packageserver-7d9775ff78-5qjb4 node/ip-10-0-130-52.us-west-2.compute.internal container=packageserver container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 05:01:49.853 E ns/openshift-marketplace pod/redhat-operators-544dd95567-5w4vg node/ip-10-0-158-47.us-west-2.compute.internal container=redhat-operators container exited with code 2 (Error): 
Sep 19 05:02:11.552 - 60s   E openshift-apiserver OpenShift API is not responding to GET requests
Sep 19 05:03:29.076 E ns/openshift-marketplace pod/certified-operators-65db45bb5-nsz7f node/ip-10-0-158-47.us-west-2.compute.internal container=certified-operators container exited with code 2 (Error): 
Sep 19 05:03:29.110 E ns/openshift-marketplace pod/community-operators-6df6976898-k6gk5 node/ip-10-0-158-47.us-west-2.compute.internal container=community-operators container exited with code 2 (Error): 
Sep 19 05:03:51.115 E ns/openshift-monitoring pod/node-exporter-rrkxr node/ip-10-0-138-182.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:03:51.486 E ns/openshift-image-registry pod/node-ca-ppd4s node/ip-10-0-138-182.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:03:52.261 E ns/openshift-dns pod/dns-default-wbzc6 node/ip-10-0-138-182.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:03:52.661 E ns/openshift-sdn pod/ovs-vkk49 node/ip-10-0-138-182.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:03:53.060 E ns/openshift-multus pod/multus-dt84v node/ip-10-0-138-182.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:03:53.460 E ns/openshift-machine-config-operator pod/machine-config-daemon-4slfs node/ip-10-0-138-182.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:03:53.860 E ns/openshift-cluster-node-tuning-operator pod/tuned-gf6nw node/ip-10-0-138-182.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:04:14.128 E ns/openshift-sdn pod/sdn-controller-mm9n5 node/ip-10-0-129-244.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:04:14.511 E ns/openshift-multus pod/multus-82c2k node/ip-10-0-129-244.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:04:14.910 E ns/openshift-machine-config-operator pod/machine-config-daemon-hqg8n node/ip-10-0-129-244.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:04:15.312 E ns/openshift-machine-config-operator pod/machine-config-server-zvrct node/ip-10-0-129-244.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:04:15.710 E ns/openshift-cluster-node-tuning-operator pod/tuned-bjjd9 node/ip-10-0-129-244.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:04:16.909 E ns/openshift-multus pod/multus-admission-controller-26krh node/ip-10-0-129-244.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:04:17.312 E ns/openshift-apiserver pod/apiserver-b7wdg node/ip-10-0-129-244.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:04:17.711 E ns/openshift-monitoring pod/node-exporter-7ddbb node/ip-10-0-129-244.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:04:18.110 E ns/openshift-dns pod/dns-default-vgwrx node/ip-10-0-129-244.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:04:18.509 E ns/openshift-image-registry pod/node-ca-dnf7k node/ip-10-0-129-244.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:04:18.910 E ns/openshift-sdn pod/ovs-ctpmd node/ip-10-0-129-244.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:04:20.110 E ns/openshift-controller-manager pod/controller-manager-5c9nj node/ip-10-0-129-244.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:04:26.563 E ns/openshift-monitoring pod/grafana-6f9f975658-swfk5 node/ip-10-0-158-47.us-west-2.compute.internal container=grafana-proxy container exited with code 2 (Error): 
Sep 19 05:04:26.962 E ns/openshift-monitoring pod/kube-state-metrics-78fbf767b6-plgxn node/ip-10-0-158-47.us-west-2.compute.internal container=kube-state-metrics container exited with code 2 (Error): 
Sep 19 05:04:27.762 E ns/openshift-marketplace pod/redhat-operators-65795db6d4-4t7d2 node/ip-10-0-158-47.us-west-2.compute.internal container=redhat-operators container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 05:04:28.964 E ns/openshift-ingress pod/router-default-79c6f9c6b6-8czrn node/ip-10-0-158-47.us-west-2.compute.internal container=router container exited with code 2 (Error): d closed the connection; LastStreamID=145, ErrCode=NO_ERROR, debug=""\nE0919 05:03:06.797693       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=145, ErrCode=NO_ERROR, debug=""\nE0919 05:03:06.797947       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=145, ErrCode=NO_ERROR, debug=""\nI0919 05:03:27.894868       1 router.go:561] Router reloaded:\n - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\nI0919 05:03:32.867160       1 router.go:561] Router reloaded:\n - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\nI0919 05:03:40.550178       1 router.go:561] Router reloaded:\n - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\nI0919 05:03:45.538036       1 router.go:561] Router reloaded:\n - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\nI0919 05:03:51.118639       1 router.go:561] Router reloaded:\n - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\nI0919 05:03:56.117293       1 router.go:561] Router reloaded:\n - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\nI0919 05:04:01.116796       1 router.go:561] Router reloaded:\n - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\nI0919 05:04:11.132574       1 router.go:561] Router reloaded:\n - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\nI0919 05:04:16.127461       1 router.go:561] Router reloaded:\n - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\nI0919 05:04:21.128255       1 router.go:561] Router reloaded:\n - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n
Sep 19 05:04:43.092 E clusteroperator/dns changed Degraded to True: NotAllDNSesAvailable: Not all desired DNS DaemonSets available
Sep 19 05:04:49.868 E clusterversion/version changed Failing to True: ClusterOperatorNotAvailable: Cluster operator machine-config is still updating
Sep 19 05:04:53.237 E ns/openshift-console pod/downloads-7f6cc7cd54-hkdfs node/ip-10-0-158-47.us-west-2.compute.internal container=download-server container exited with code 137 (Error): 0 05:02:05] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:02:05] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:02:15] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:02:15] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:02:25] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:02:25] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:02:35] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:02:35] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:02:45] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:02:45] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:02:55] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:02:55] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:03:05] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:03:05] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:03:15] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:03:15] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:03:25] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:03:25] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:03:35] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:03:35] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:03:45] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:03:45] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:03:55] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:03:55] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:04:05] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:04:05] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:04:15] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:04:15] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:04:25] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:04:25] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:04:35] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:04:35] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:04:45] "GET / HTTP/1.1" 200 -\n10.131.0.1 - - [19/Sep/2020 05:04:45] "GET / HTTP/1.1" 200 -\n
Sep 19 05:04:56.805 E ns/openshift-monitoring pod/prometheus-k8s-0 node/ip-10-0-138-182.us-west-2.compute.internal container=prometheus container exited with code 1 (Error): =info ts=2020-09-19T05:04:47.452Z caller=main.go:332 fd_limits="(soft=1048576, hard=1048576)"\nlevel=info ts=2020-09-19T05:04:47.452Z caller=main.go:333 vm_limits="(soft=unlimited, hard=unlimited)"\nlevel=info ts=2020-09-19T05:04:47.462Z caller=main.go:652 msg="Starting TSDB ..."\nlevel=info ts=2020-09-19T05:04:47.465Z caller=web.go:448 component=web msg="Start listening for connections" address=127.0.0.1:9090\nlevel=info ts=2020-09-19T05:04:47.480Z caller=main.go:667 fs_type=XFS_SUPER_MAGIC\nlevel=info ts=2020-09-19T05:04:47.480Z caller=main.go:668 msg="TSDB started"\nlevel=info ts=2020-09-19T05:04:47.481Z caller=main.go:738 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml\nlevel=info ts=2020-09-19T05:04:47.483Z caller=main.go:521 msg="Stopping scrape discovery manager..."\nlevel=info ts=2020-09-19T05:04:47.483Z caller=main.go:535 msg="Stopping notify discovery manager..."\nlevel=info ts=2020-09-19T05:04:47.483Z caller=main.go:557 msg="Stopping scrape manager..."\nlevel=info ts=2020-09-19T05:04:47.483Z caller=main.go:531 msg="Notify discovery manager stopped"\nlevel=info ts=2020-09-19T05:04:47.484Z caller=main.go:517 msg="Scrape discovery manager stopped"\nlevel=info ts=2020-09-19T05:04:47.484Z caller=manager.go:776 component="rule manager" msg="Stopping rule manager..."\nlevel=info ts=2020-09-19T05:04:47.484Z caller=manager.go:782 component="rule manager" msg="Rule manager stopped"\nlevel=info ts=2020-09-19T05:04:47.485Z caller=notifier.go:602 component=notifier msg="Stopping notification manager..."\nlevel=info ts=2020-09-19T05:04:47.485Z caller=main.go:722 msg="Notifier manager stopped"\nlevel=info ts=2020-09-19T05:04:47.485Z caller=main.go:551 msg="Scrape manager stopped"\nlevel=error ts=2020-09-19T05:04:47.486Z caller=main.go:731 err="error loading config from \"/etc/prometheus/config_out/prometheus.env.yaml\": couldn't load configuration (--config.file=\"/etc/prometheus/config_out/prometheus.env.yaml\"): open /etc/prometheus/config_out/prometheus.env.yaml: no such file or directory"\n
Sep 19 05:04:59.364 E ns/openshift-service-catalog-controller-manager-operator pod/openshift-service-catalog-controller-manager-operator-7b44ld9k8 node/ip-10-0-145-20.us-west-2.compute.internal container=operator container exited with code 255 (Error): 1/servicecatalogcontrollermanagers/cluster\nI0919 04:41:35.038019       1 request.go:530] Throttling request took 193.75326ms, request: GET:https://172.30.0.1:443/apis/operator.openshift.io/v1/servicecatalogcontrollermanagers/cluster\nI0919 04:41:35.238021       1 request.go:530] Throttling request took 197.290914ms, request: GET:https://172.30.0.1:443/apis/operator.openshift.io/v1/servicecatalogcontrollermanagers/cluster\nI0919 04:42:02.875815       1 wrap.go:47] GET /metrics: (4.984221ms) 200 [Prometheus/2.7.2 10.129.2.10:37136]\nI0919 04:42:02.877442       1 wrap.go:47] GET /metrics: (1.993197ms) 200 [Prometheus/2.7.2 10.131.0.8:34512]\nI0919 04:42:32.876101       1 wrap.go:47] GET /metrics: (5.222292ms) 200 [Prometheus/2.7.2 10.129.2.10:37136]\nI0919 04:42:32.877637       1 wrap.go:47] GET /metrics: (2.287301ms) 200 [Prometheus/2.7.2 10.131.0.8:34512]\nI0919 04:43:02.922706       1 wrap.go:47] GET /metrics: (51.080085ms) 200 [Prometheus/2.7.2 10.129.2.10:37136]\nI0919 04:43:02.924355       1 wrap.go:47] GET /metrics: (38.743942ms) 200 [Prometheus/2.7.2 10.131.0.8:34512]\nI0919 04:43:32.882416       1 wrap.go:47] GET /metrics: (6.942988ms) 200 [Prometheus/2.7.2 10.131.0.8:34512]\nI0919 04:43:32.884091       1 wrap.go:47] GET /metrics: (13.209313ms) 200 [Prometheus/2.7.2 10.129.2.10:37136]\nI0919 04:43:53.214987       1 observer_polling.go:78] Observed change: file:/var/run/secrets/serving-cert/tls.crt (current: "02f5d958a415da2bb0ec1c76593463c6039825c9e479c00dcf75656eae2d396a", lastKnown: "6ce534c0906aaeeb1b24d7f8624dfc44ad2f35512c6d8af7ba7c229bbef8f3cb")\nW0919 04:43:53.215018       1 builder.go:108] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was modified\nI0919 04:43:53.215068       1 observer_polling.go:78] Observed change: file:/var/run/secrets/serving-cert/tls.key (current: "5e259f664efdaf3721c43cfe8757cb1f5eb73a9cf55a9ef0b41d9ff4850acf7f", lastKnown: "f7adf255779ecf0907a354a4fff126474b65e2a74174f702613b1eee445835d1")\nF0919 04:43:53.215095       1 leaderelection.go:66] leaderelection lost\n
Sep 19 05:05:03.554 E ns/openshift-console-operator pod/console-operator-7d5f9cbd6c-jzwxk node/ip-10-0-145-20.us-west-2.compute.internal container=console-operator container exited with code 255 (Error): -19T04:20:02Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}\nI0919 05:04:19.781198       1 event.go:209] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"5833d1c3-fa2f-11ea-8f7e-0603b42f4bc7", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/console changed: Degraded message changed from "RouteSyncDegraded: the server is currently unable to handle the request (get routes.route.openshift.io console)\nOAuthClientSyncDegraded: the server is currently unable to handle the request (get oauthclients.oauth.openshift.io console)" to ""\nE0919 05:04:50.182627       1 status.go:71] DeploymentAvailable FailedUpdate 1 replicas ready at version 4.2.36\nI0919 05:04:50.203825       1 status_controller.go:165] clusteroperator/console diff {"status":{"conditions":[{"lastTransitionTime":"2020-09-19T04:43:09Z","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2020-09-19T04:46:32Z","reason":"AsExpected","status":"False","type":"Progressing"},{"lastTransitionTime":"2020-09-19T05:04:50Z","message":"DeploymentAvailable: 1 replicas ready at version 4.2.36","reason":"DeploymentAvailableFailedUpdate","status":"False","type":"Available"},{"lastTransitionTime":"2020-09-19T04:20:02Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}\nI0919 05:04:50.223555       1 event.go:209] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"5833d1c3-fa2f-11ea-8f7e-0603b42f4bc7", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/console changed: Available changed from True to False ("DeploymentAvailable: 1 replicas ready at version 4.2.36")\nI0919 05:04:50.256001       1 cmd.go:79] Received SIGTERM or SIGINT signal, shutting down controller.\nF0919 05:04:50.256063       1 leaderelection.go:66] leaderelection lost\n
Sep 19 05:05:08.961 E ns/openshift-operator-lifecycle-manager pod/packageserver-7848c9bfd9-5sm87 node/ip-10-0-145-20.us-west-2.compute.internal container=packageserver container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 05:05:10.155 E ns/openshift-machine-config-operator pod/machine-config-controller-57665d5d8c-mrzkl node/ip-10-0-145-20.us-west-2.compute.internal container=machine-config-controller container exited with code 2 (Error): 
Sep 19 05:05:13.557 E ns/openshift-service-catalog-apiserver-operator pod/openshift-service-catalog-apiserver-operator-f8b76df5d-mtrrs node/ip-10-0-145-20.us-west-2.compute.internal container=operator container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 05:05:14.154 E ns/openshift-machine-api pod/machine-api-controllers-747746c457-xhrtt node/ip-10-0-145-20.us-west-2.compute.internal container=controller-manager container exited with code 1 (Error): 
Sep 19 05:05:15.872 E ns/openshift-operator-lifecycle-manager pod/packageserver-55cc45cfc6-wvtzr node/ip-10-0-129-244.us-west-2.compute.internal container=packageserver container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 05:05:23.545 E kube-apiserver Kube API started failing: Get https://api.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com:6443/api/v1/namespaces/kube-system?timeout=5s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Sep 19 05:06:20.436 E kube-apiserver failed contacting the API: Get https://api.ci-op-g5943rxq-92d62.origin-ci-int-aws.dev.rhcloud.com:6443/apis/config.openshift.io/v1/clusteroperators?allowWatchBookmarks=true&resourceVersion=46041&timeout=8m3s&timeoutSeconds=483&watch=true: dial tcp 44.239.232.208:6443: connect: connection refused
Sep 19 05:06:41.552 E openshift-apiserver OpenShift API is not responding to GET requests
Sep 19 05:07:11.552 E openshift-apiserver OpenShift API is not responding to GET requests
Sep 19 05:07:52.955 E ns/openshift-monitoring pod/node-exporter-r42qj node/ip-10-0-158-47.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:07:53.325 E ns/openshift-image-registry pod/node-ca-58rzq node/ip-10-0-158-47.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:07:53.694 E ns/openshift-multus pod/multus-xvjrw node/ip-10-0-158-47.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:07:54.063 E ns/openshift-sdn pod/ovs-rpgwz node/ip-10-0-158-47.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:07:54.432 E ns/openshift-sdn pod/sdn-q97c2 node/ip-10-0-158-47.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:07:54.802 E ns/openshift-dns pod/dns-default-lct9s node/ip-10-0-158-47.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:07:55.171 E ns/openshift-cluster-node-tuning-operator pod/tuned-mqpbr node/ip-10-0-158-47.us-west-2.compute.internal container=tuned container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 05:07:55.724 E ns/openshift-machine-config-operator pod/machine-config-daemon-gsjwb node/ip-10-0-158-47.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:07:56.093 E ns/openshift-cluster-node-tuning-operator pod/tuned-tgd9f node/ip-10-0-158-47.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:07:58.762 E clusteroperator/monitoring changed Degraded to True: UpdatingGrafanaFailed: Failed to rollout the stack. Error: running task Updating Grafana failed: reconciling Grafana Dashboard Definitions ConfigMaps failed: retrieving ConfigMap object failed: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/configmaps/grafana-dashboard-prometheus-remote-write: unexpected EOF
Sep 19 05:08:37.688 E ns/openshift-image-registry pod/node-ca-92xbg node/ip-10-0-145-20.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:08:38.087 E ns/openshift-sdn pod/ovs-pn9ld node/ip-10-0-145-20.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:08:38.487 E ns/openshift-machine-config-operator pod/machine-config-daemon-79j54 node/ip-10-0-145-20.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:08:38.886 E ns/openshift-cluster-version pod/cluster-version-operator-598b9d5599-gmg89 node/ip-10-0-145-20.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:08:39.687 E ns/openshift-monitoring pod/node-exporter-lf7z6 node/ip-10-0-145-20.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:08:40.087 E ns/openshift-multus pod/multus-admission-controller-hp7wg node/ip-10-0-145-20.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:08:40.487 E ns/openshift-apiserver pod/apiserver-bctsx node/ip-10-0-145-20.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:08:40.887 E ns/openshift-cluster-node-tuning-operator pod/tuned-p6vqg node/ip-10-0-145-20.us-west-2.compute.internal container=tuned container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Sep 19 05:08:41.487 E ns/openshift-machine-config-operator pod/machine-config-server-cnptk node/ip-10-0-145-20.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:08:42.286 E ns/openshift-controller-manager pod/controller-manager-k5tkr node/ip-10-0-145-20.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:08:42.686 E ns/openshift-sdn pod/sdn-controller-vgmhq node/ip-10-0-145-20.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:08:43.088 E ns/openshift-multus pod/multus-ndzvd node/ip-10-0-145-20.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:08:43.487 E ns/openshift-dns pod/dns-default-j6jkc node/ip-10-0-145-20.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:08:43.888 E ns/openshift-cluster-node-tuning-operator pod/tuned-b9jwr node/ip-10-0-145-20.us-west-2.compute.internal invariant violation: pod may not transition Running->Pending
Sep 19 05:12:06.772 E ns/openshift-machine-config-operator pod/etcd-quorum-guard-7f5b7cbf56-fx5z5 node/ip-10-0-145-20.us-west-2.compute.internal container=guard container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated