ResultSUCCESS
Tests 3 failed / 22 succeeded
Started2020-03-05 13:01
Elapsed1h27m
Work namespaceci-op-s76jkp2x
Refs openshift-4.5:d61ae9e1
38:59b0b8c6
pod577059f3-5ee1-11ea-8cff-0a58ac103daf
repoopenshift/etcd
revision1

Test Failures


Cluster upgrade Application behind service load balancer with PDB is not disrupted 35m49s

go run hack/e2e.go -v -test --test_args='--ginkgo.focus=Cluster\supgrade\sApplication\sbehind\sservice\sload\sbalancer\swith\sPDB\sis\snot\sdisrupted$'
Service was unreachable during disruption for at least 19s of 32m20s (1%):

Mar 05 13:38:05.028 E ns/e2e-k8s-service-lb-available-2612 svc/service-test Service stopped responding to GET requests over new connections
Mar 05 13:38:06.013 - 1s    E ns/e2e-k8s-service-lb-available-2612 svc/service-test Service is not responding to GET requests over new connections
Mar 05 13:38:08.061 I ns/e2e-k8s-service-lb-available-2612 svc/service-test Service started responding to GET requests over new connections
Mar 05 13:38:11.020 E ns/e2e-k8s-service-lb-available-2612 svc/service-test Service stopped responding to GET requests over new connections
Mar 05 13:38:12.013 E ns/e2e-k8s-service-lb-available-2612 svc/service-test Service is not responding to GET requests over new connections
Mar 05 13:38:12.047 I ns/e2e-k8s-service-lb-available-2612 svc/service-test Service started responding to GET requests over new connections
Mar 05 13:38:14.033 E ns/e2e-k8s-service-lb-available-2612 svc/service-test Service stopped responding to GET requests over new connections
Mar 05 13:38:15.013 E ns/e2e-k8s-service-lb-available-2612 svc/service-test Service is not responding to GET requests over new connections
Mar 05 13:38:15.056 I ns/e2e-k8s-service-lb-available-2612 svc/service-test Service started responding to GET requests over new connections
Mar 05 13:38:16.019 E ns/e2e-k8s-service-lb-available-2612 svc/service-test Service stopped responding to GET requests over new connections
Mar 05 13:38:17.013 - 3s    E ns/e2e-k8s-service-lb-available-2612 svc/service-test Service is not responding to GET requests over new connections
Mar 05 13:38:20.054 I ns/e2e-k8s-service-lb-available-2612 svc/service-test Service started responding to GET requests over new connections
Mar 05 13:38:22.020 E ns/e2e-k8s-service-lb-available-2612 svc/service-test Service stopped responding to GET requests over new connections
Mar 05 13:38:23.013 - 999ms E ns/e2e-k8s-service-lb-available-2612 svc/service-test Service is not responding to GET requests over new connections
Mar 05 13:38:24.050 I ns/e2e-k8s-service-lb-available-2612 svc/service-test Service started responding to GET requests over new connections
Mar 05 13:38:25.029 E ns/e2e-k8s-service-lb-available-2612 svc/service-test Service stopped responding to GET requests over new connections
Mar 05 13:38:26.013 E ns/e2e-k8s-service-lb-available-2612 svc/service-test Service is not responding to GET requests over new connections
Mar 05 13:38:26.047 I ns/e2e-k8s-service-lb-available-2612 svc/service-test Service started responding to GET requests over new connections
Mar 05 13:38:40.029 E ns/e2e-k8s-service-lb-available-2612 svc/service-test Service stopped responding to GET requests over new connections
Mar 05 13:38:41.013 E ns/e2e-k8s-service-lb-available-2612 svc/service-test Service is not responding to GET requests over new connections
Mar 05 13:38:41.069 I ns/e2e-k8s-service-lb-available-2612 svc/service-test Service started responding to GET requests over new connections
Mar 05 13:38:43.019 E ns/e2e-k8s-service-lb-available-2612 svc/service-test Service stopped responding to GET requests over new connections
Mar 05 13:38:44.013 E ns/e2e-k8s-service-lb-available-2612 svc/service-test Service is not responding to GET requests over new connections
Mar 05 13:38:44.047 I ns/e2e-k8s-service-lb-available-2612 svc/service-test Service started responding to GET requests over new connections
				from junit_upgrade_1583417483.xml

Filter through log files


Cluster upgrade Cluster frontend ingress remain available 34m49s

go run hack/e2e.go -v -test --test_args='--ginkgo.focus=Cluster\supgrade\sCluster\sfrontend\singress\sremain\savailable$'
Frontends were unreachable during disruption for at least 4s of 34m48s (0%):

Mar 05 13:48:53.416 E ns/openshift-authentication route/oauth-openshift Route stopped responding to GET requests over new connections
Mar 05 13:48:54.374 E ns/openshift-authentication route/oauth-openshift Route is not responding to GET requests over new connections
Mar 05 13:48:54.433 I ns/openshift-authentication route/oauth-openshift Route started responding to GET requests over new connections
Mar 05 13:49:38.416 E ns/openshift-authentication route/oauth-openshift Route stopped responding to GET requests over new connections
Mar 05 13:49:39.374 E ns/openshift-authentication route/oauth-openshift Route is not responding to GET requests over new connections
Mar 05 13:49:39.426 I ns/openshift-authentication route/oauth-openshift Route started responding to GET requests over new connections
				from junit_upgrade_1583417483.xml

Filter through log files


openshift-tests Monitor cluster while tests execute 35m52s

go run hack/e2e.go -v -test --test_args='--ginkgo.focus=openshift\-tests\sMonitor\scluster\swhile\stests\sexecute$'
246 error level events were detected during this test run:

Mar 05 13:36:26.803 E clusteroperator/monitoring changed Degraded to True: UpdatingprometheusAdapterFailed: Failed to rollout the stack. Error: running task Updating prometheus-adapter failed: reconciling PrometheusAdapter ClusterRole aggregating resource metrics read permissions failed: updating ClusterRole object failed: Put https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterroles/system:aggregated-metrics-reader: http2: server sent GOAWAY and closed the connection; LastStreamID=4385, ErrCode=NO_ERROR, debug=""
Mar 05 13:38:36.090 E ns/openshift-kube-apiserver-operator pod/kube-apiserver-operator-d47c6f659-8975s node/ip-10-0-135-26.ec2.internal container=kube-apiserver-operator container exited with code 255 (Error): kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"b40bd176-994a-4bcd-a479-ee14c7a48ac3", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-apiserver changed: Progressing changed from True to False ("NodeInstallerProgressing: 3 nodes are at revision 7"),Available message changed from "StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 3; 2 nodes are at revision 7" to "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 7"\nI0305 13:36:19.759877       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"b40bd176-994a-4bcd-a479-ee14c7a48ac3", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapUpdated' Updated ConfigMap/revision-status-7 -n openshift-kube-apiserver:\ncause by changes in data.status\nI0305 13:36:26.790766       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"b40bd176-994a-4bcd-a479-ee14c7a48ac3", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'PodCreated' Created Pod/revision-pruner-7-ip-10-0-135-26.ec2.internal -n openshift-kube-apiserver because it was missing\nI0305 13:38:35.059213       1 cmd.go:84] Received SIGTERM or SIGINT signal, shutting down controller.\nI0305 13:38:35.059901       1 client_cert_rotation_controller.go:180] Shutting down CertRotationController - "ExternalLoadBalancerServing"\nI0305 13:38:35.060016       1 targetconfigcontroller.go:440] Shutting down TargetConfigController\nI0305 13:38:35.060038       1 client_cert_rotation_controller.go:180] Shutting down CertRotationController - "KubeAPIServerToKubeletClientCert"\nI0305 13:38:35.060059       1 client_cert_rotation_controller.go:180] Shutting down CertRotationController - "AggregatorProxyClientCert"\nF0305 13:38:35.060068       1 builder.go:243] stopped\n
Mar 05 13:38:52.159 E ns/openshift-kube-controller-manager-operator pod/kube-controller-manager-operator-77fb74d94c-mpbw7 node/ip-10-0-135-26.ec2.internal container=kube-controller-manager-operator container exited with code 255 (Error):  data.status\nI0305 13:34:44.131430       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-controller-manager-operator", Name:"kube-controller-manager-operator", UID:"5d851ab4-dc0d-4f45-bc41-c4fa36057fba", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'PodCreated' Created Pod/revision-pruner-8-ip-10-0-150-76.ec2.internal -n openshift-kube-controller-manager because it was missing\nI0305 13:38:51.143343       1 cmd.go:84] Received SIGTERM or SIGINT signal, shutting down controller.\nI0305 13:38:51.143971       1 tlsconfig.go:234] Shutting down DynamicServingCertificateController\nI0305 13:38:51.144136       1 satokensigner_controller.go:332] Shutting down SATokenSignerController\nI0305 13:38:51.144183       1 base_controller.go:74] Shutting down InstallerStateController ...\nI0305 13:38:51.144218       1 base_controller.go:74] Shutting down StaticPodStateController ...\nI0305 13:38:51.144236       1 base_controller.go:74] Shutting down InstallerController ...\nI0305 13:38:51.144251       1 client_cert_rotation_controller.go:180] Shutting down CertRotationController - "CSRSigningCert"\nI0305 13:38:51.144296       1 base_controller.go:74] Shutting down  ...\nI0305 13:38:51.144318       1 base_controller.go:74] Shutting down NodeController ...\nI0305 13:38:51.144333       1 base_controller.go:74] Shutting down PruneController ...\nI0305 13:38:51.144349       1 base_controller.go:74] Shutting down UnsupportedConfigOverridesController ...\nI0305 13:38:51.144364       1 base_controller.go:74] Shutting down RevisionController ...\nI0305 13:38:51.144378       1 base_controller.go:74] Shutting down LoggingSyncer ...\nI0305 13:38:51.144392       1 status_controller.go:212] Shutting down StatusSyncer-kube-controller-manager\nI0305 13:38:51.144406       1 remove_stale_conditions.go:84] Shutting down RemoveStaleConditions\nI0305 13:38:51.144580       1 targetconfigcontroller.go:613] Shutting down TargetConfigController\nF0305 13:38:51.144619       1 builder.go:243] stopped\n
Mar 05 13:39:17.586 E ns/openshift-machine-api pod/machine-api-operator-77c576b788-n9795 node/ip-10-0-150-76.ec2.internal container=machine-api-operator container exited with code 2 (Error): 
Mar 05 13:39:33.095 E ns/openshift-apiserver-operator pod/openshift-apiserver-operator-7b4dc97d-64pdv node/ip-10-0-135-26.ec2.internal container=openshift-apiserver-operator container exited with code 255 (Error):        1 key_controller.go:363] Shutting down EncryptionKeyController\nI0305 13:39:10.370361       1 config_observer_controller.go:160] Shutting down ConfigObserver\nI0305 13:39:10.370379       1 base_controller.go:73] Shutting down  ...\nI0305 13:39:10.370397       1 base_controller.go:73] Shutting down UnsupportedConfigOverridesController ...\nI0305 13:39:10.370413       1 base_controller.go:73] Shutting down LoggingSyncer ...\nI0305 13:39:10.370425       1 status_controller.go:212] Shutting down StatusSyncer-openshift-apiserver\nI0305 13:39:10.370441       1 base_controller.go:73] Shutting down RevisionController ...\nI0305 13:39:10.370454       1 finalizer_controller.go:148] Shutting down NamespaceFinalizerController_openshift-apiserver\nI0305 13:39:10.370486       1 remove_stale_conditions.go:84] Shutting down RemoveStaleConditions\nI0305 13:39:10.370501       1 prune_controller.go:232] Shutting down PruneController\nI0305 13:39:10.370663       1 base_controller.go:48] Shutting down worker of  controller ...\nI0305 13:39:10.370676       1 base_controller.go:38] All  workers have been terminated\nI0305 13:39:10.370694       1 base_controller.go:48] Shutting down worker of UnsupportedConfigOverridesController controller ...\nI0305 13:39:10.370740       1 base_controller.go:38] All UnsupportedConfigOverridesController workers have been terminated\nI0305 13:39:10.370761       1 base_controller.go:48] Shutting down worker of LoggingSyncer controller ...\nI0305 13:39:10.370769       1 base_controller.go:38] All LoggingSyncer workers have been terminated\nI0305 13:39:10.370788       1 base_controller.go:48] Shutting down worker of RevisionController controller ...\nI0305 13:39:10.370797       1 base_controller.go:38] All RevisionController workers have been terminated\nI0305 13:39:10.370830       1 apiservice_controller.go:215] Shutting down APIServiceController_openshift-apiserver\nI0305 13:39:10.370944       1 workload_controller.go:204] Shutting down OpenShiftAPIServerOperator\nF0305 13:39:10.371162       1 builder.go:243] stopped\n
Mar 05 13:41:22.035 E ns/openshift-machine-api pod/machine-api-controllers-68b756c9b6-xxvt6 node/ip-10-0-130-18.ec2.internal container=nodelink-controller container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:41:22.035 E ns/openshift-machine-api pod/machine-api-controllers-68b756c9b6-xxvt6 node/ip-10-0-130-18.ec2.internal container=controller-manager container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:41:22.035 E ns/openshift-machine-api pod/machine-api-controllers-68b756c9b6-xxvt6 node/ip-10-0-130-18.ec2.internal container=machine-controller container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:41:22.035 E ns/openshift-machine-api pod/machine-api-controllers-68b756c9b6-xxvt6 node/ip-10-0-130-18.ec2.internal container=machine-healthcheck-controller container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:41:24.053 E ns/openshift-kube-storage-version-migrator-operator pod/kube-storage-version-migrator-operator-66fcf86665-wvq5v node/ip-10-0-135-26.ec2.internal container=kube-storage-version-migrator-operator container exited with code 255 (Error):  clusteroperator/kube-storage-version-migrator changed: {"conditions":[{"type":"Degraded","status":"False","lastTransitionTime":"2020-03-05T13:22:11Z","reason":"AsExpected"},{"type":"Progressing","status":"False","lastTransitionTime":"2020-03-05T13:22:12Z","reason":"AsExpected"},{"type":"Available","status":"False","lastTransitionTime":"2020-03-05T13:22:11Z","reason":"_NoMigratorPod","message":"Available: deployment/migrator.openshift-kube-storage-version-migrator: no replicas are available"},{"type":"Upgradeable","status":"Unknown","lastTransitionTime":"2020-03-05T13:22:11Z","reason":"NoData"}],"versions":[{"name":"operator","version":"0.0.1-2020-03-05-130227"}\n\nA: ],"relatedObjects":[{"group":"operator.openshift.io","resource":"kubestorageversionmigrators","name":"cluster"},{"group":"","resource":"namespaces","name":"openshift-kube-storage-version-migrator"},{"group":"","resource":"namespaces","name":"openshift-kube-storage-version-migrator-operator"}],"extension":null}\n\n\nB: ,{"name":"kube-storage-version-migrator","version":""}],"relatedObjects":[{"group":"operator.openshift.io","resource":"kubestorageversionmigrators","name":"cluster"},{"group":"","resource":"namespaces","name":"openshift-kube-storage-version-migrator"},{"group":"","resource":"namespaces","name":"openshift-kube-storage-version-migrator-operator"}],"extension":null}\n\n\nI0305 13:27:10.359750       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-storage-version-migrator-operator", Name:"kube-storage-version-migrator-operator", UID:"d08d3d14-3e2e-49bd-b1a0-2e517a9f28d1", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-storage-version-migrator changed: Available changed from False to True ("")\nI0305 13:41:22.947619       1 cmd.go:79] Received SIGTERM or SIGINT signal, shutting down controller.\nF0305 13:41:22.947673       1 leaderelection.go:66] leaderelection lost\nF0305 13:41:22.951066       1 builder.go:210] server exited\n
Mar 05 13:41:45.898 E kube-apiserver Kube API started failing: Get https://api.ci-op-s76jkp2x-f83f1.origin-ci-int-aws.dev.rhcloud.com:6443/api/v1/namespaces/kube-system?timeout=5s: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Mar 05 13:41:46.898 E kube-apiserver Kube API is not responding to GET requests
Mar 05 13:41:58.818 E clusteroperator/monitoring changed Degraded to True: UpdatingPrometheusK8SFailed: Failed to rollout the stack. Error: running task Updating Prometheus-k8s failed: reconciling Prometheus RoleBinding "prometheus-k8s" failed: retrieving RoleBinding object failed: etcdserver: leader changed
Mar 05 13:42:45.365 E ns/openshift-cluster-machine-approver pod/machine-approver-74749c6d79-mlrsr node/ip-10-0-135-26.ec2.internal container=machine-approver-controller container exited with code 2 (Error): ieve current serving cert: remote error: tls: internal error\nI0305 13:26:17.604199       1 csr_check.go:183] Falling back to machine-api authorization for ip-10-0-142-254.ec2.internal\nI0305 13:26:17.629758       1 main.go:196] CSR csr-s669d approved\nI0305 13:26:22.996806       1 main.go:146] CSR csr-fdz6b added\nI0305 13:26:23.038668       1 csr_check.go:418] retrieving serving cert from ip-10-0-130-45.ec2.internal (10.0.130.45:10250)\nW0305 13:26:23.042868       1 csr_check.go:178] Failed to retrieve current serving cert: remote error: tls: internal error\nI0305 13:26:23.042896       1 csr_check.go:183] Falling back to machine-api authorization for ip-10-0-130-45.ec2.internal\nI0305 13:26:23.064461       1 main.go:196] CSR csr-fdz6b approved\nI0305 13:26:25.698851       1 main.go:146] CSR csr-c6fwm added\nI0305 13:26:25.728175       1 csr_check.go:418] retrieving serving cert from ip-10-0-153-229.ec2.internal (10.0.153.229:10250)\nW0305 13:26:25.730688       1 csr_check.go:178] Failed to retrieve current serving cert: remote error: tls: internal error\nI0305 13:26:25.730822       1 csr_check.go:183] Falling back to machine-api authorization for ip-10-0-153-229.ec2.internal\nI0305 13:26:25.746013       1 main.go:196] CSR csr-c6fwm approved\nI0305 13:35:58.016528       1 streamwatcher.go:103] Unexpected EOF during watch stream event decoding: unexpected EOF\nE0305 13:35:58.019094       1 reflector.go:270] github.com/openshift/cluster-machine-approver/main.go:238: Failed to watch *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?resourceVersion=11421&timeoutSeconds=499&watch=true: dial tcp 127.0.0.1:6443: connect: connection refused\nE0305 13:35:59.019863       1 reflector.go:126] github.com/openshift/cluster-machine-approver/main.go:238: Failed to list *v1beta1.CertificateSigningRequest: Get https://127.0.0.1:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: dial tcp 127.0.0.1:6443: connect: connection refused\n
Mar 05 13:42:46.313 E ns/openshift-kube-storage-version-migrator pod/migrator-765f9bccd6-n48tv node/ip-10-0-142-254.ec2.internal container=migrator container exited with code 2 (Error): 
Mar 05 13:43:01.870 E ns/openshift-service-catalog-controller-manager-operator pod/openshift-service-catalog-controller-manager-operator-96f5kfxht node/ip-10-0-135-26.ec2.internal container=operator container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:43:04.381 E ns/openshift-monitoring pod/kube-state-metrics-7c4b6858b6-d4qwl node/ip-10-0-142-254.ec2.internal container=kube-state-metrics container exited with code 2 (Error): 
Mar 05 13:43:04.397 E ns/openshift-monitoring pod/openshift-state-metrics-5fbd6cd6f-r9jjr node/ip-10-0-142-254.ec2.internal container=openshift-state-metrics container exited with code 2 (Error): 
Mar 05 13:43:07.900 E ns/openshift-monitoring pod/alertmanager-main-2 node/ip-10-0-142-254.ec2.internal container=config-reloader container exited with code 2 (Error): 2020/03/05 13:31:17 Watching directory: "/etc/alertmanager/config"\n
Mar 05 13:43:07.900 E ns/openshift-monitoring pod/alertmanager-main-2 node/ip-10-0-142-254.ec2.internal container=alertmanager-proxy container exited with code 2 (Error): 2020/03/05 13:31:18 provider.go:118: Defaulting client-id to system:serviceaccount:openshift-monitoring:alertmanager-main\n2020/03/05 13:31:18 provider.go:123: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token\n2020/03/05 13:31:18 provider.go:311: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.\n2020/03/05 13:31:18 oauthproxy.go:200: mapping path "/" => upstream "http://localhost:9093/"\n2020/03/05 13:31:18 oauthproxy.go:221: compiled skip-auth-regex => "^/metrics"\n2020/03/05 13:31:18 oauthproxy.go:227: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:alertmanager-main\n2020/03/05 13:31:18 oauthproxy.go:237: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> refresh:disabled\n2020/03/05 13:31:18 http.go:107: HTTPS: listening on [::]:9095\nI0305 13:31:18.051048       1 dynamic_serving_content.go:129] Starting serving::/etc/tls/private/tls.crt::/etc/tls/private/tls.key\n
Mar 05 13:43:09.905 E ns/openshift-operator-lifecycle-manager pod/packageserver-886b98bf7-pd256 node/ip-10-0-135-26.ec2.internal container=packageserver container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:43:12.464 E ns/openshift-monitoring pod/alertmanager-main-2 node/ip-10-0-142-254.ec2.internal container=alertmanager container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:43:12.464 E ns/openshift-monitoring pod/alertmanager-main-2 node/ip-10-0-142-254.ec2.internal container=alertmanager-proxy container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:43:12.464 E ns/openshift-monitoring pod/alertmanager-main-2 node/ip-10-0-142-254.ec2.internal container=config-reloader container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:43:14.444 E ns/openshift-csi-snapshot-controller pod/csi-snapshot-controller-97469c4b4-v6kc5 node/ip-10-0-153-229.ec2.internal container=snapshot-controller container exited with code 2 (Error): 
Mar 05 13:43:15.038 E ns/openshift-authentication pod/oauth-openshift-c4dd74689-cb7kw node/ip-10-0-135-26.ec2.internal container=oauth-openshift container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:43:17.582 E ns/openshift-monitoring pod/prometheus-adapter-56fc7c8865-9ldvl node/ip-10-0-130-45.ec2.internal container=prometheus-adapter container exited with code 2 (Error): I0305 13:31:21.196047       1 adapter.go:93] successfully using in-cluster auth\nI0305 13:31:22.460024       1 secure_serving.go:116] Serving securely on [::]:6443\n
Mar 05 13:43:23.417 E ns/openshift-monitoring pod/prometheus-adapter-56fc7c8865-wvxxm node/ip-10-0-153-229.ec2.internal container=prometheus-adapter container exited with code 2 (Error): I0305 13:31:22.866734       1 adapter.go:93] successfully using in-cluster auth\nI0305 13:31:23.598756       1 secure_serving.go:116] Serving securely on [::]:6443\n
Mar 05 13:43:23.828 E ns/openshift-monitoring pod/node-exporter-pnkrr node/ip-10-0-150-76.ec2.internal container=node-exporter container exited with code 143 (Error): or gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:42:10Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:42:24Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:42:25Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:42:39Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:42:40Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:42:54Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:42:55Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\n
Mar 05 13:43:27.518 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ip-10-0-153-229.ec2.internal container=rules-configmap-reloader container exited with code 2 (Error): 2020/03/05 13:32:40 Watching directory: "/etc/prometheus/rules/prometheus-k8s-rulefiles-0"\n
Mar 05 13:43:27.518 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ip-10-0-153-229.ec2.internal container=prometheus-proxy container exited with code 2 (Error): 2020/03/05 13:32:41 provider.go:118: Defaulting client-id to system:serviceaccount:openshift-monitoring:prometheus-k8s\n2020/03/05 13:32:41 provider.go:123: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token\n2020/03/05 13:32:41 provider.go:311: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.\n2020/03/05 13:32:41 oauthproxy.go:200: mapping path "/" => upstream "http://localhost:9090/"\n2020/03/05 13:32:41 oauthproxy.go:221: compiled skip-auth-regex => "^/metrics"\n2020/03/05 13:32:41 oauthproxy.go:227: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:prometheus-k8s\n2020/03/05 13:32:41 oauthproxy.go:237: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> refresh:disabled\n2020/03/05 13:32:41 main.go:154: using htpasswd file /etc/proxy/htpasswd/auth\n2020/03/05 13:32:41 http.go:107: HTTPS: listening on [::]:9091\nI0305 13:32:41.069283       1 dynamic_serving_content.go:129] Starting serving::/etc/tls/private/tls.crt::/etc/tls/private/tls.key\n2020/03/05 13:43:12 oauthproxy.go:774: basicauth: 10.131.0.27:48678 Authorization header does not start with 'Basic', skipping basic authentication\n
Mar 05 13:43:27.518 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ip-10-0-153-229.ec2.internal container=prometheus-config-reloader container exited with code 2 (Error): ts=2020-03-05T13:32:40.235303026Z caller=main.go:85 msg="Starting prometheus-config-reloader version '1.12.16'."\nlevel=error ts=2020-03-05T13:32:40.238147756Z caller=runutil.go:95 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post http://localhost:9090/-/reload: dial tcp [::1]:9090: connect: connection refused"\nlevel=info ts=2020-03-05T13:32:45.388295736Z caller=reloader.go:286 msg="Prometheus reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml rule_dirs=\nlevel=info ts=2020-03-05T13:32:45.388424803Z caller=reloader.go:154 msg="started watching config file and non-recursively rule dirs for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=\n
Mar 05 13:43:28.609 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-130-45.ec2.internal container=config-reloader container exited with code 2 (Error): 2020/03/05 13:31:34 Watching directory: "/etc/alertmanager/config"\n
Mar 05 13:43:28.609 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-130-45.ec2.internal container=alertmanager-proxy container exited with code 2 (Error): 2020/03/05 13:31:34 provider.go:118: Defaulting client-id to system:serviceaccount:openshift-monitoring:alertmanager-main\n2020/03/05 13:31:34 provider.go:123: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token\n2020/03/05 13:31:34 provider.go:311: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.\n2020/03/05 13:31:34 oauthproxy.go:200: mapping path "/" => upstream "http://localhost:9093/"\n2020/03/05 13:31:34 oauthproxy.go:221: compiled skip-auth-regex => "^/metrics"\n2020/03/05 13:31:34 oauthproxy.go:227: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:alertmanager-main\n2020/03/05 13:31:34 oauthproxy.go:237: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> refresh:disabled\n2020/03/05 13:31:34 http.go:107: HTTPS: listening on [::]:9095\nI0305 13:31:34.692675       1 dynamic_serving_content.go:129] Starting serving::/etc/tls/private/tls.crt::/etc/tls/private/tls.key\n
Mar 05 13:43:28.774 E ns/openshift-ingress-operator pod/ingress-operator-579658c97-7hsrj node/ip-10-0-150-76.ec2.internal container=kube-rbac-proxy container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:43:28.774 E ns/openshift-ingress-operator pod/ingress-operator-579658c97-7hsrj node/ip-10-0-150-76.ec2.internal container=ingress-operator container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:43:29.647 E ns/openshift-monitoring pod/grafana-78fd989f69-qfgxl node/ip-10-0-130-45.ec2.internal container=grafana container exited with code 1 (Error): 
Mar 05 13:43:29.647 E ns/openshift-monitoring pod/grafana-78fd989f69-qfgxl node/ip-10-0-130-45.ec2.internal container=grafana-proxy container exited with code 2 (Error): 
Mar 05 13:43:30.689 E ns/openshift-cluster-node-tuning-operator pod/tuned-zhxkb node/ip-10-0-130-45.ec2.internal container=tuned container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:43:33.098 E ns/openshift-kube-scheduler pod/openshift-kube-scheduler-ip-10-0-150-76.ec2.internal node/ip-10-0-150-76.ec2.internal container=scheduler container exited with code 255 (Error): cp [::1]:6443: connect: connection refused\nE0305 13:43:26.242640       1 reflector.go:307] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:209: Failed to watch *v1.ConfigMap: Get https://localhost:6443/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=22134&timeout=8m30s&timeoutSeconds=510&watch=true: dial tcp [::1]:6443: connect: connection refused\nE0305 13:43:26.296349       1 reflector.go:307] k8s.io/client-go/informers/factory.go:135: Failed to watch *v1.CSINode: Get https://localhost:6443/apis/storage.k8s.io/v1/csinodes?allowWatchBookmarks=true&resourceVersion=19308&timeout=7m32s&timeoutSeconds=452&watch=true: dial tcp [::1]:6443: connect: connection refused\nE0305 13:43:26.296730       1 reflector.go:307] k8s.io/client-go/informers/factory.go:135: Failed to watch *v1.PersistentVolume: Get https://localhost:6443/api/v1/persistentvolumes?allowWatchBookmarks=true&resourceVersion=19305&timeout=7m16s&timeoutSeconds=436&watch=true: dial tcp [::1]:6443: connect: connection refused\nE0305 13:43:26.296902       1 reflector.go:307] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:209: Failed to watch *v1.ConfigMap: Get https://localhost:6443/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=22134&timeout=7m30s&timeoutSeconds=450&watch=true: dial tcp [::1]:6443: connect: connection refused\nI0305 13:43:32.381933       1 leaderelection.go:288] failed to renew lease openshift-kube-scheduler/kube-scheduler: failed to tryAcquireOrRenew context deadline exceeded\nF0305 13:43:32.384291       1 server.go:257] leaderelection lost\nE0305 13:43:32.382499       1 leaderelection.go:331] error retrieving resource lock openshift-kube-scheduler/kube-scheduler: Get https://localhost:6443/api/v1/namespaces/openshift-kube-scheduler/configmaps/kube-scheduler?timeout=10s: context deadline exceeded\n
Mar 05 13:43:45.627 E ns/openshift-image-registry pod/node-ca-5blwr node/ip-10-0-142-254.ec2.internal container=node-ca container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:43:51.036 E ns/openshift-marketplace pod/redhat-operators-5b6ff75798-g4mwv node/ip-10-0-153-229.ec2.internal container=redhat-operators container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:43:53.711 E ns/openshift-monitoring pod/node-exporter-sjj7k node/ip-10-0-130-45.ec2.internal container=node-exporter container exited with code 143 (Error): or gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:42:45Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:42:53Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:43:00Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:43:08Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:43:15Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:43:23Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:43:38Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\n
Mar 05 13:43:55.018 E ns/openshift-controller-manager pod/controller-manager-c99dt node/ip-10-0-135-26.ec2.internal container=controller-manager container exited with code 137 (Error): I0305 13:27:11.495150       1 controller_manager.go:39] Starting controllers on 0.0.0.0:8443 (unknown)\nI0305 13:27:11.496735       1 controller_manager.go:50] DeploymentConfig controller using images from "registry.svc.ci.openshift.org/ci-op-s76jkp2x/stable-initial@sha256:f12127cf48ffc713b25ad83e9d98b64c5e11b1f1af06306cf0a8a5a090a59b6a"\nI0305 13:27:11.496764       1 controller_manager.go:56] Build controller using images from "registry.svc.ci.openshift.org/ci-op-s76jkp2x/stable-initial@sha256:471891b26e981d2ed9c87cdd306bc028abe62b760a7af413bd9c05389c4ea5a4"\nI0305 13:27:11.496768       1 standalone_apiserver.go:98] Started health checks at 0.0.0.0:8443\nI0305 13:27:11.496880       1 leaderelection.go:242] attempting to acquire leader lease  openshift-controller-manager/openshift-master-controllers...\n
Mar 05 13:43:55.043 E ns/openshift-controller-manager pod/controller-manager-pkwq5 node/ip-10-0-130-18.ec2.internal container=controller-manager container exited with code 137 (Error): ch stream: stream error: stream ID 567; INTERNAL_ERROR") has prevented the request from succeeding\nW0305 13:40:40.922483       1 reflector.go:340] github.com/openshift/client-go/template/informers/externalversions/factory.go:101: watch of *v1.TemplateInstance ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 589; INTERNAL_ERROR") has prevented the request from succeeding\nW0305 13:40:40.922679       1 reflector.go:340] github.com/openshift/client-go/build/informers/externalversions/factory.go:101: watch of *v1.Build ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 541; INTERNAL_ERROR") has prevented the request from succeeding\nW0305 13:40:40.922818       1 reflector.go:340] github.com/openshift/client-go/image/informers/externalversions/factory.go:101: watch of *v1.Image ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 547; INTERNAL_ERROR") has prevented the request from succeeding\nW0305 13:40:40.922911       1 reflector.go:340] github.com/openshift/client-go/image/informers/externalversions/factory.go:101: watch of *v1.ImageStream ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 579; INTERNAL_ERROR") has prevented the request from succeeding\nW0305 13:41:30.043588       1 reflector.go:340] github.com/openshift/client-go/build/informers/externalversions/factory.go:101: watch of *v1.Build ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 49; INTERNAL_ERROR") has prevented the request from succeeding\nW0305 13:41:30.116271       1 reflector.go:340] github.com/openshift/client-go/image/informers/externalversions/factory.go:101: watch of *v1.ImageStream ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 7; INTERNAL_ERROR") has prevented the request from succeeding\n
Mar 05 13:43:55.398 E ns/openshift-controller-manager pod/controller-manager-j7ds8 node/ip-10-0-150-76.ec2.internal container=controller-manager container exited with code 137 (Error): I0305 13:27:21.337924       1 controller_manager.go:39] Starting controllers on 0.0.0.0:8443 (unknown)\nI0305 13:27:21.339257       1 controller_manager.go:50] DeploymentConfig controller using images from "registry.svc.ci.openshift.org/ci-op-s76jkp2x/stable-initial@sha256:f12127cf48ffc713b25ad83e9d98b64c5e11b1f1af06306cf0a8a5a090a59b6a"\nI0305 13:27:21.339278       1 controller_manager.go:56] Build controller using images from "registry.svc.ci.openshift.org/ci-op-s76jkp2x/stable-initial@sha256:471891b26e981d2ed9c87cdd306bc028abe62b760a7af413bd9c05389c4ea5a4"\nI0305 13:27:21.339347       1 standalone_apiserver.go:98] Started health checks at 0.0.0.0:8443\nI0305 13:27:21.339473       1 leaderelection.go:242] attempting to acquire leader lease  openshift-controller-manager/openshift-master-controllers...\n
Mar 05 13:44:00.886 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ip-10-0-153-229.ec2.internal container=prometheus container exited with code 1 (Error): caller=main.go:648 msg="Starting TSDB ..."\nlevel=info ts=2020-03-05T13:43:43.565Z caller=web.go:506 component=web msg="Start listening for connections" address=127.0.0.1:9090\nlevel=info ts=2020-03-05T13:43:43.571Z caller=head.go:584 component=tsdb msg="replaying WAL, this may take awhile"\nlevel=info ts=2020-03-05T13:43:43.571Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0\nlevel=info ts=2020-03-05T13:43:43.572Z caller=main.go:663 fs_type=XFS_SUPER_MAGIC\nlevel=info ts=2020-03-05T13:43:43.572Z caller=main.go:664 msg="TSDB started"\nlevel=info ts=2020-03-05T13:43:43.572Z caller=main.go:734 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml\nlevel=info ts=2020-03-05T13:43:43.572Z caller=main.go:517 msg="Stopping scrape discovery manager..."\nlevel=info ts=2020-03-05T13:43:43.572Z caller=main.go:531 msg="Stopping notify discovery manager..."\nlevel=info ts=2020-03-05T13:43:43.573Z caller=main.go:553 msg="Stopping scrape manager..."\nlevel=info ts=2020-03-05T13:43:43.573Z caller=main.go:527 msg="Notify discovery manager stopped"\nlevel=info ts=2020-03-05T13:43:43.573Z caller=main.go:513 msg="Scrape discovery manager stopped"\nlevel=info ts=2020-03-05T13:43:43.573Z caller=manager.go:814 component="rule manager" msg="Stopping rule manager..."\nlevel=info ts=2020-03-05T13:43:43.573Z caller=manager.go:820 component="rule manager" msg="Rule manager stopped"\nlevel=info ts=2020-03-05T13:43:43.573Z caller=main.go:547 msg="Scrape manager stopped"\nlevel=info ts=2020-03-05T13:43:43.574Z caller=notifier.go:598 component=notifier msg="Stopping notification manager..."\nlevel=info ts=2020-03-05T13:43:43.574Z caller=main.go:718 msg="Notifier manager stopped"\nlevel=error ts=2020-03-05
Mar 05 13:44:02.040 E clusterversion/version changed Failing to True: WorkloadNotAvailable: deployment openshift-console/downloads is progressing ReplicaSetUpdated: ReplicaSet "downloads-6b45d75b95" is progressing.
Mar 05 13:44:03.756 E ns/openshift-monitoring pod/prometheus-k8s-0 node/ip-10-0-130-45.ec2.internal container=rules-configmap-reloader container exited with code 2 (Error): 2020/03/05 13:33:02 Watching directory: "/etc/prometheus/rules/prometheus-k8s-rulefiles-0"\n
Mar 05 13:44:03.756 E ns/openshift-monitoring pod/prometheus-k8s-0 node/ip-10-0-130-45.ec2.internal container=prometheus-proxy container exited with code 2 (Error): 2020/03/05 13:33:03 provider.go:118: Defaulting client-id to system:serviceaccount:openshift-monitoring:prometheus-k8s\n2020/03/05 13:33:03 provider.go:123: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token\n2020/03/05 13:33:03 provider.go:311: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.\n2020/03/05 13:33:03 oauthproxy.go:200: mapping path "/" => upstream "http://localhost:9090/"\n2020/03/05 13:33:03 oauthproxy.go:221: compiled skip-auth-regex => "^/metrics"\n2020/03/05 13:33:03 oauthproxy.go:227: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:prometheus-k8s\n2020/03/05 13:33:03 oauthproxy.go:237: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> refresh:disabled\n2020/03/05 13:33:03 main.go:154: using htpasswd file /etc/proxy/htpasswd/auth\n2020/03/05 13:33:03 http.go:107: HTTPS: listening on [::]:9091\nI0305 13:33:03.250782       1 dynamic_serving_content.go:129] Starting serving::/etc/tls/private/tls.crt::/etc/tls/private/tls.key\n2020/03/05 13:36:50 oauthproxy.go:774: basicauth: 10.129.2.11:44496 Authorization header does not start with 'Basic', skipping basic authentication\n2020/03/05 13:41:20 oauthproxy.go:774: basicauth: 10.129.2.11:47318 Authorization header does not start with 'Basic', skipping basic authentication\n2020/03/05 13:41:54 oauthproxy.go:774: basicauth: 10.129.0.24:41980 Authorization header does not start with 'Basic', skipping basic authentication\n2020/03/05 13:43:49 oauthproxy.go:774: basicauth: 10.130.0.72:46416 Authorization header does not start with 'Basic', skipping basic authentication\n
Mar 05 13:44:03.756 E ns/openshift-monitoring pod/prometheus-k8s-0 node/ip-10-0-130-45.ec2.internal container=prometheus-config-reloader container exited with code 2 (Error): ts=2020-03-05T13:33:02.548404041Z caller=main.go:85 msg="Starting prometheus-config-reloader version '1.12.16'."\nlevel=error ts=2020-03-05T13:33:02.551359646Z caller=runutil.go:95 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post http://localhost:9090/-/reload: dial tcp [::1]:9090: connect: connection refused"\nlevel=info ts=2020-03-05T13:33:07.691549383Z caller=reloader.go:286 msg="Prometheus reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml rule_dirs=\nlevel=info ts=2020-03-05T13:33:07.691645734Z caller=reloader.go:154 msg="started watching config file and non-recursively rule dirs for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=\n
Mar 05 13:44:10.704 E ns/openshift-monitoring pod/node-exporter-qvfpw node/ip-10-0-142-254.ec2.internal container=node-exporter container exited with code 143 (Error): or gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:43:00Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:43:06Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:43:15Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:43:21Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:43:30Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:43:45Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:44:00Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\n
Mar 05 13:44:15.704 E ns/openshift-monitoring pod/prometheus-k8s-0 node/ip-10-0-142-254.ec2.internal container=prometheus container exited with code 1 (Error): caller=main.go:648 msg="Starting TSDB ..."\nlevel=info ts=2020-03-05T13:44:12.387Z caller=web.go:506 component=web msg="Start listening for connections" address=127.0.0.1:9090\nlevel=info ts=2020-03-05T13:44:12.393Z caller=head.go:584 component=tsdb msg="replaying WAL, this may take awhile"\nlevel=info ts=2020-03-05T13:44:12.393Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0\nlevel=info ts=2020-03-05T13:44:12.394Z caller=main.go:663 fs_type=XFS_SUPER_MAGIC\nlevel=info ts=2020-03-05T13:44:12.394Z caller=main.go:664 msg="TSDB started"\nlevel=info ts=2020-03-05T13:44:12.394Z caller=main.go:734 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml\nlevel=info ts=2020-03-05T13:44:12.394Z caller=main.go:517 msg="Stopping scrape discovery manager..."\nlevel=info ts=2020-03-05T13:44:12.394Z caller=main.go:531 msg="Stopping notify discovery manager..."\nlevel=info ts=2020-03-05T13:44:12.395Z caller=main.go:553 msg="Stopping scrape manager..."\nlevel=info ts=2020-03-05T13:44:12.395Z caller=main.go:527 msg="Notify discovery manager stopped"\nlevel=info ts=2020-03-05T13:44:12.395Z caller=manager.go:814 component="rule manager" msg="Stopping rule manager..."\nlevel=info ts=2020-03-05T13:44:12.395Z caller=manager.go:820 component="rule manager" msg="Rule manager stopped"\nlevel=info ts=2020-03-05T13:44:12.395Z caller=main.go:513 msg="Scrape discovery manager stopped"\nlevel=info ts=2020-03-05T13:44:12.395Z caller=main.go:547 msg="Scrape manager stopped"\nlevel=info ts=2020-03-05T13:44:12.396Z caller=notifier.go:598 component=notifier msg="Stopping notification manager..."\nlevel=info ts=2020-03-05T13:44:12.396Z caller=main.go:718 msg="Notifier manager stopped"\nlevel=error ts=2020-03-05
Mar 05 13:44:18.043 E ns/openshift-marketplace pod/redhat-marketplace-7456bfd654-r7jq5 node/ip-10-0-153-229.ec2.internal container=redhat-marketplace container exited with code 2 (Error): 
Mar 05 13:44:27.323 E ns/openshift-image-registry pod/node-ca-cq6tr node/ip-10-0-130-18.ec2.internal container=node-ca container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:44:33.131 E ns/openshift-marketplace pod/community-operators-7d979b4576-mctlg node/ip-10-0-153-229.ec2.internal container=community-operators container exited with code 2 (Error): 
Mar 05 13:45:01.441 E ns/openshift-console-operator pod/console-operator-68489b4b8c-4z842 node/ip-10-0-130-18.ec2.internal container=console-operator container exited with code 255 (Error): "AsExpected","status":"True","type":"Upgradeable"}]}}\nI0305 13:32:11.173390       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"adabe822-8c59-4e70-87d4-a85889dc9be0", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/console changed: Progressing changed from True to False (""),Available changed from False to True ("")\nW0305 13:40:40.923166       1 reflector.go:326] github.com/openshift/client-go/route/informers/externalversions/factory.go:101: watch of *v1.Route ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 775; INTERNAL_ERROR") has prevented the request from succeeding\nW0305 13:41:30.031664       1 reflector.go:326] github.com/openshift/client-go/route/informers/externalversions/factory.go:101: watch of *v1.Route ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 15; INTERNAL_ERROR") has prevented the request from succeeding\nI0305 13:45:00.837384       1 cmd.go:79] Received SIGTERM or SIGINT signal, shutting down controller.\nI0305 13:45:00.838198       1 status_controller.go:212] Shutting down StatusSyncer-console\nI0305 13:45:00.838225       1 controller.go:70] Shutting down Console\nI0305 13:45:00.838246       1 base_controller.go:74] Shutting down LoggingSyncer ...\nI0305 13:45:00.838261       1 controller.go:109] shutting down ConsoleResourceSyncDestinationController\nI0305 13:45:00.838299       1 base_controller.go:74] Shutting down UnsupportedConfigOverridesController ...\nI0305 13:45:00.838314       1 resourcesync_controller.go:228] Shutting down ResourceSyncController\nI0305 13:45:00.838329       1 controller.go:138] shutting down ConsoleServiceSyncController\nI0305 13:45:00.838346       1 management_state_controller.go:112] Shutting down management-state-controller-console\nF0305 13:45:00.838522       1 builder.go:243] stopped\n
Mar 05 13:46:41.045 E ns/openshift-console pod/console-78768c8b8b-ns9dg node/ip-10-0-150-76.ec2.internal container=console container exited with code 2 (Error): 2020-03-05T13:30:42Z cmd/main: cookies are secure!\n2020-03-05T13:30:42Z auth: error contacting auth provider (retrying in 10s): discovery through endpoint https://kubernetes.default.svc/.well-known/oauth-authorization-server failed: 404 Not Found\n2020-03-05T13:30:52Z auth: error contacting auth provider (retrying in 10s): discovery through endpoint https://kubernetes.default.svc/.well-known/oauth-authorization-server failed: 404 Not Found\n2020-03-05T13:31:02Z auth: error contacting auth provider (retrying in 10s): discovery through endpoint https://kubernetes.default.svc/.well-known/oauth-authorization-server failed: 404 Not Found\n2020-03-05T13:31:12Z auth: error contacting auth provider (retrying in 10s): discovery through endpoint https://kubernetes.default.svc/.well-known/oauth-authorization-server failed: 404 Not Found\n2020-03-05T13:31:22Z auth: error contacting auth provider (retrying in 10s): discovery through endpoint https://kubernetes.default.svc/.well-known/oauth-authorization-server failed: 404 Not Found\n2020-03-05T13:31:32Z auth: error contacting auth provider (retrying in 10s): discovery through endpoint https://kubernetes.default.svc/.well-known/oauth-authorization-server failed: 404 Not Found\n2020-03-05T13:31:42Z auth: error contacting auth provider (retrying in 10s): discovery through endpoint https://kubernetes.default.svc/.well-known/oauth-authorization-server failed: 404 Not Found\n2020-03-05T13:31:52Z auth: error contacting auth provider (retrying in 10s): discovery through endpoint https://kubernetes.default.svc/.well-known/oauth-authorization-server failed: 404 Not Found\n2020-03-05T13:32:02Z cmd/main: Binding to [::]:8443...\n2020-03-05T13:32:02Z cmd/main: using TLS\n
Mar 05 13:46:54.588 E ns/openshift-console pod/console-78768c8b8b-mdk6x node/ip-10-0-135-26.ec2.internal container=console container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:48:18.208 E ns/openshift-sdn pod/sdn-controller-g59kj node/ip-10-0-130-18.ec2.internal container=sdn-controller container exited with code 2 (Error): ated HostSubnet ip-10-0-142-254.ec2.internal (host: "ip-10-0-142-254.ec2.internal", ip: "10.0.142.254", subnet: "10.131.0.0/23")\nI0305 13:26:22.816482       1 subnets.go:149] Created HostSubnet ip-10-0-130-45.ec2.internal (host: "ip-10-0-130-45.ec2.internal", ip: "10.0.130.45", subnet: "10.128.2.0/23")\nI0305 13:26:25.398538       1 subnets.go:149] Created HostSubnet ip-10-0-153-229.ec2.internal (host: "ip-10-0-153-229.ec2.internal", ip: "10.0.153.229", subnet: "10.129.2.0/23")\nI0305 13:35:33.825839       1 vnids.go:115] Allocated netid 108391 for namespace "e2e-k8s-sig-apps-job-upgrade-9706"\nI0305 13:35:33.841315       1 vnids.go:115] Allocated netid 4034275 for namespace "e2e-k8s-service-lb-available-2612"\nI0305 13:35:33.866291       1 vnids.go:115] Allocated netid 9978401 for namespace "e2e-k8s-sig-apps-deployment-upgrade-1079"\nI0305 13:35:33.902829       1 vnids.go:115] Allocated netid 13439343 for namespace "e2e-control-plane-available-8462"\nI0305 13:35:33.941245       1 vnids.go:115] Allocated netid 12774597 for namespace "e2e-k8s-sig-storage-sig-api-machinery-secret-upgrade-4353"\nI0305 13:35:33.955039       1 vnids.go:115] Allocated netid 11180655 for namespace "e2e-k8s-sig-apps-daemonset-upgrade-2302"\nI0305 13:35:34.029744       1 vnids.go:115] Allocated netid 2148768 for namespace "e2e-frontend-ingress-available-8092"\nI0305 13:35:34.053194       1 vnids.go:115] Allocated netid 2718674 for namespace "e2e-k8s-sig-apps-replicaset-upgrade-6366"\nI0305 13:35:34.068168       1 vnids.go:115] Allocated netid 1562012 for namespace "e2e-k8s-sig-storage-sig-api-machinery-configmap-upgrade-2261"\nE0305 13:40:58.848542       1 reflector.go:307] github.com/openshift/client-go/network/informers/externalversions/factory.go:101: Failed to watch *v1.HostSubnet: Get https://api-int.ci-op-s76jkp2x-f83f1.origin-ci-int-aws.dev.rhcloud.com:6443/apis/network.openshift.io/v1/hostsubnets?allowWatchBookmarks=true&resourceVersion=19301&timeout=5m17s&timeoutSeconds=317&watch=true: dial tcp 10.0.149.20:6443: connect: connection refused\n
Mar 05 13:48:21.608 E ns/openshift-sdn pod/sdn-664v6 node/ip-10-0-153-229.ec2.internal container=sdn container exited with code 255 (Error): ole:https to [10.128.0.62:8443 10.129.0.66:8443 10.130.0.38:8443]\nI0305 13:46:27.943539    2940 roundrobin.go:267] LoadBalancerRR: Setting endpoints for openshift-console/console:https to [10.128.0.62:8443 10.129.0.66:8443]\nI0305 13:46:27.943583    2940 roundrobin.go:217] Delete endpoint 10.130.0.38:8443 for service "openshift-console/console:https"\nI0305 13:46:28.168682    2940 proxier.go:368] userspace proxy: processing 0 service events\nI0305 13:46:28.168707    2940 proxier.go:347] userspace syncProxyRules took 76.025152ms\nI0305 13:46:28.443297    2940 proxier.go:368] userspace proxy: processing 0 service events\nI0305 13:46:28.443326    2940 proxier.go:347] userspace syncProxyRules took 91.500986ms\nI0305 13:46:58.695660    2940 proxier.go:368] userspace proxy: processing 0 service events\nI0305 13:46:58.695686    2940 proxier.go:347] userspace syncProxyRules took 82.434422ms\nI0305 13:47:28.957754    2940 proxier.go:368] userspace proxy: processing 0 service events\nI0305 13:47:28.957781    2940 proxier.go:347] userspace syncProxyRules took 77.168427ms\nI0305 13:47:59.200192    2940 proxier.go:368] userspace proxy: processing 0 service events\nI0305 13:47:59.200218    2940 proxier.go:347] userspace syncProxyRules took 70.884905ms\nI0305 13:48:15.995177    2940 roundrobin.go:267] LoadBalancerRR: Setting endpoints for openshift-multus/multus-admission-controller: to [10.129.0.7:6443 10.130.0.16:6443]\nI0305 13:48:15.995226    2940 roundrobin.go:217] Delete endpoint 10.128.0.2:6443 for service "openshift-multus/multus-admission-controller:"\nI0305 13:48:16.235988    2940 proxier.go:368] userspace proxy: processing 0 service events\nI0305 13:48:16.236010    2940 proxier.go:347] userspace syncProxyRules took 70.778833ms\nI0305 13:48:21.516882    2940 ovs.go:169] Error executing ovs-ofctl: ovs-ofctl: /var/run/openvswitch/br0.mgmt: failed to open socket (Connection refused)\nF0305 13:48:21.516921    2940 healthcheck.go:82] SDN healthcheck detected OVS server change, restarting: OVS reinitialization required: plugin is not setup\n
Mar 05 13:48:29.399 E ns/openshift-sdn pod/sdn-controller-khmfj node/ip-10-0-150-76.ec2.internal container=sdn-controller container exited with code 2 (Error): I0305 13:18:10.514687       1 leaderelection.go:242] attempting to acquire leader lease  openshift-sdn/openshift-network-controller...\nE0305 13:23:46.967759       1 leaderelection.go:331] error retrieving resource lock openshift-sdn/openshift-network-controller: etcdserver: leader changed\nE0305 13:25:12.445351       1 leaderelection.go:331] error retrieving resource lock openshift-sdn/openshift-network-controller: Get https://api-int.ci-op-s76jkp2x-f83f1.origin-ci-int-aws.dev.rhcloud.com:6443/api/v1/namespaces/openshift-sdn/configmaps/openshift-network-controller: unexpected EOF\n
Mar 05 13:48:32.931 E ns/openshift-sdn pod/sdn-controller-qhlb7 node/ip-10-0-135-26.ec2.internal container=sdn-controller container exited with code 2 (Error): I0305 13:18:10.169463       1 leaderelection.go:242] attempting to acquire leader lease  openshift-sdn/openshift-network-controller...\nE0305 13:23:46.961701       1 leaderelection.go:331] error retrieving resource lock openshift-sdn/openshift-network-controller: etcdserver: leader changed\nE0305 13:25:12.451863       1 leaderelection.go:331] error retrieving resource lock openshift-sdn/openshift-network-controller: Get https://api-int.ci-op-s76jkp2x-f83f1.origin-ci-int-aws.dev.rhcloud.com:6443/api/v1/namespaces/openshift-sdn/configmaps/openshift-network-controller: unexpected EOF\n
Mar 05 13:48:44.261 E ns/openshift-sdn pod/sdn-25tcg node/ip-10-0-130-45.ec2.internal container=sdn container exited with code 255 (Error): ole:https to [10.128.0.62:8443 10.129.0.66:8443 10.130.0.38:8443]\nI0305 13:46:27.945984    2920 roundrobin.go:267] LoadBalancerRR: Setting endpoints for openshift-console/console:https to [10.128.0.62:8443 10.129.0.66:8443]\nI0305 13:46:27.946018    2920 roundrobin.go:217] Delete endpoint 10.130.0.38:8443 for service "openshift-console/console:https"\nI0305 13:46:28.167575    2920 proxier.go:368] userspace proxy: processing 0 service events\nI0305 13:46:28.167603    2920 proxier.go:347] userspace syncProxyRules took 76.367672ms\nI0305 13:46:28.405523    2920 proxier.go:368] userspace proxy: processing 0 service events\nI0305 13:46:28.405549    2920 proxier.go:347] userspace syncProxyRules took 70.303563ms\nI0305 13:46:58.649252    2920 proxier.go:368] userspace proxy: processing 0 service events\nI0305 13:46:58.649277    2920 proxier.go:347] userspace syncProxyRules took 71.037082ms\nI0305 13:47:28.892844    2920 proxier.go:368] userspace proxy: processing 0 service events\nI0305 13:47:28.892868    2920 proxier.go:347] userspace syncProxyRules took 70.618492ms\nI0305 13:47:59.139960    2920 proxier.go:368] userspace proxy: processing 0 service events\nI0305 13:47:59.139989    2920 proxier.go:347] userspace syncProxyRules took 70.754692ms\nI0305 13:48:15.996917    2920 roundrobin.go:267] LoadBalancerRR: Setting endpoints for openshift-multus/multus-admission-controller: to [10.129.0.7:6443 10.130.0.16:6443]\nI0305 13:48:15.996954    2920 roundrobin.go:217] Delete endpoint 10.128.0.2:6443 for service "openshift-multus/multus-admission-controller:"\nI0305 13:48:16.250680    2920 proxier.go:368] userspace proxy: processing 0 service events\nI0305 13:48:16.250708    2920 proxier.go:347] userspace syncProxyRules took 77.502929ms\nI0305 13:48:44.132203    2920 ovs.go:169] Error executing ovs-ofctl: ovs-ofctl: /var/run/openvswitch/br0.mgmt: failed to open socket (Connection refused)\nF0305 13:48:44.132242    2920 healthcheck.go:82] SDN healthcheck detected OVS server change, restarting: OVS reinitialization required: plugin is not setup\n
Mar 05 13:48:47.328 E ns/openshift-multus pod/multus-admission-controller-zwhgm node/ip-10-0-130-18.ec2.internal container=multus-admission-controller container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:49:05.417 E ns/openshift-sdn pod/sdn-4hzft node/ip-10-0-130-18.ec2.internal container=sdn container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:49:09.590 E ns/openshift-sdn pod/sdn-wfrqd node/ip-10-0-150-76.ec2.internal container=sdn container exited with code 255 (Error): 05 13:48:52.128339   10419 proxier.go:347] userspace syncProxyRules took 214.379527ms\nI0305 13:48:52.230869   10419 proxier.go:1609] Opened local port "nodePort for openshift-ingress/router-default:https" (:31605/tcp)\nI0305 13:48:52.231083   10419 proxier.go:1609] Opened local port "nodePort for e2e-k8s-service-lb-available-2612/service-test:" (:32333/tcp)\nI0305 13:48:52.231359   10419 proxier.go:1609] Opened local port "nodePort for openshift-ingress/router-default:http" (:30178/tcp)\nI0305 13:48:52.270113   10419 service_health.go:98] Opening healthcheck "openshift-ingress/router-default" on port 32354\nI0305 13:48:52.280051   10419 proxy.go:305] openshift-sdn proxy services and endpoints initialized\nI0305 13:48:52.280100   10419 cmd.go:173] openshift-sdn network plugin registering startup\nI0305 13:48:52.280187   10419 cmd.go:177] openshift-sdn network plugin ready\nI0305 13:48:59.358280   10419 roundrobin.go:267] LoadBalancerRR: Setting endpoints for openshift-multus/multus-admission-controller: to [10.128.0.63:6443 10.129.0.7:6443 10.130.0.16:6443]\nI0305 13:48:59.375085   10419 roundrobin.go:267] LoadBalancerRR: Setting endpoints for openshift-multus/multus-admission-controller: to [10.128.0.63:6443 10.130.0.16:6443]\nI0305 13:48:59.375129   10419 roundrobin.go:217] Delete endpoint 10.129.0.7:6443 for service "openshift-multus/multus-admission-controller:"\nI0305 13:48:59.636903   10419 proxier.go:368] userspace proxy: processing 0 service events\nI0305 13:48:59.636931   10419 proxier.go:347] userspace syncProxyRules took 73.650015ms\nI0305 13:48:59.949167   10419 proxier.go:368] userspace proxy: processing 0 service events\nI0305 13:48:59.949207   10419 proxier.go:347] userspace syncProxyRules took 116.695773ms\nI0305 13:49:09.177791   10419 ovs.go:169] Error executing ovs-ofctl: ovs-ofctl: /var/run/openvswitch/br0.mgmt: failed to open socket (Connection refused)\nF0305 13:49:09.177839   10419 healthcheck.go:82] SDN healthcheck detected OVS server change, restarting: OVS reinitialization required: plugin is not setup\n
Mar 05 13:49:23.625 E ns/openshift-multus pod/multus-ptbxq node/ip-10-0-150-76.ec2.internal container=kube-multus container exited with code 137 (Error): 
Mar 05 13:49:31.202 E ns/openshift-sdn pod/sdn-njfv7 node/ip-10-0-135-26.ec2.internal container=sdn container exited with code 255 (Error): ess/router-default:http" (:30178/tcp)\nI0305 13:48:34.586431   16150 proxier.go:1609] Opened local port "nodePort for openshift-ingress/router-default:https" (:31605/tcp)\nI0305 13:48:34.586801   16150 proxier.go:1609] Opened local port "nodePort for e2e-k8s-service-lb-available-2612/service-test:" (:32333/tcp)\nI0305 13:48:34.636595   16150 service_health.go:98] Opening healthcheck "openshift-ingress/router-default" on port 32354\nI0305 13:48:34.646488   16150 proxy.go:305] openshift-sdn proxy services and endpoints initialized\nI0305 13:48:34.646523   16150 cmd.go:173] openshift-sdn network plugin registering startup\nI0305 13:48:34.646998   16150 cmd.go:177] openshift-sdn network plugin ready\nI0305 13:48:59.360709   16150 roundrobin.go:267] LoadBalancerRR: Setting endpoints for openshift-multus/multus-admission-controller: to [10.128.0.63:6443 10.129.0.7:6443 10.130.0.16:6443]\nI0305 13:48:59.373013   16150 roundrobin.go:267] LoadBalancerRR: Setting endpoints for openshift-multus/multus-admission-controller: to [10.128.0.63:6443 10.130.0.16:6443]\nI0305 13:48:59.373063   16150 roundrobin.go:217] Delete endpoint 10.129.0.7:6443 for service "openshift-multus/multus-admission-controller:"\nI0305 13:48:59.631117   16150 proxier.go:368] userspace proxy: processing 0 service events\nI0305 13:48:59.631140   16150 proxier.go:347] userspace syncProxyRules took 74.448134ms\nI0305 13:48:59.923397   16150 proxier.go:368] userspace proxy: processing 0 service events\nI0305 13:48:59.923420   16150 proxier.go:347] userspace syncProxyRules took 90.850238ms\nI0305 13:49:30.245069   16150 proxier.go:368] userspace proxy: processing 0 service events\nI0305 13:49:30.245105   16150 proxier.go:347] userspace syncProxyRules took 100.350862ms\nI0305 13:49:30.359976   16150 ovs.go:169] Error executing ovs-ofctl: ovs-ofctl: /var/run/openvswitch/br0.mgmt: failed to open socket (Connection refused)\nF0305 13:49:30.360033   16150 healthcheck.go:82] SDN healthcheck detected OVS server change, restarting: OVS reinitialization required: plugin is not setup\n
Mar 05 13:49:57.626 E ns/openshift-sdn pod/sdn-zmh25 node/ip-10-0-130-18.ec2.internal container=sdn container exited with code 255 (Error): gress/router-default:https" (:31605/tcp)\nI0305 13:49:19.445019   12378 proxier.go:1609] Opened local port "nodePort for openshift-ingress/router-default:http" (:30178/tcp)\nI0305 13:49:19.445321   12378 proxier.go:1609] Opened local port "nodePort for e2e-k8s-service-lb-available-2612/service-test:" (:32333/tcp)\nI0305 13:49:19.491052   12378 service_health.go:98] Opening healthcheck "openshift-ingress/router-default" on port 32354\nI0305 13:49:19.675023   12378 proxy.go:305] openshift-sdn proxy services and endpoints initialized\nI0305 13:49:19.675058   12378 cmd.go:173] openshift-sdn network plugin registering startup\nI0305 13:49:19.675208   12378 cmd.go:177] openshift-sdn network plugin ready\nI0305 13:49:42.707973   12378 roundrobin.go:267] LoadBalancerRR: Setting endpoints for openshift-multus/multus-admission-controller: to [10.128.0.63:6443 10.129.0.67:6443 10.130.0.16:6443]\nI0305 13:49:42.720515   12378 roundrobin.go:267] LoadBalancerRR: Setting endpoints for openshift-multus/multus-admission-controller: to [10.128.0.63:6443 10.129.0.67:6443]\nI0305 13:49:42.720559   12378 roundrobin.go:217] Delete endpoint 10.130.0.16:6443 for service "openshift-multus/multus-admission-controller:"\nI0305 13:49:42.992862   12378 proxier.go:368] userspace proxy: processing 0 service events\nI0305 13:49:42.992914   12378 proxier.go:347] userspace syncProxyRules took 84.453523ms\nI0305 13:49:43.341613   12378 proxier.go:368] userspace proxy: processing 0 service events\nI0305 13:49:43.341744   12378 proxier.go:347] userspace syncProxyRules took 123.673984ms\nI0305 13:49:48.554905   12378 healthcheck.go:92] SDN healthcheck unable to reconnect to OVS server: dial unix /var/run/openvswitch/db.sock: connect: no such file or directory\nI0305 13:49:56.702519   12378 ovs.go:169] Error executing ovs-ofctl: ovs-ofctl: /var/run/openvswitch/br0.mgmt: failed to open socket (Connection refused)\nF0305 13:49:56.702570   12378 healthcheck.go:82] SDN healthcheck detected OVS server change, restarting: OVS reinitialization required: plugin is not setup\n
Mar 05 13:50:00.783 E ns/openshift-multus pod/multus-twc7w node/ip-10-0-153-229.ec2.internal container=kube-multus container exited with code 137 (Error): 
Mar 05 13:50:13.362 E ns/openshift-multus pod/multus-admission-controller-lk7lj node/ip-10-0-135-26.ec2.internal container=multus-admission-controller container exited with code 137 (Error): 
Mar 05 13:50:21.695 E ns/openshift-sdn pod/sdn-c768n node/ip-10-0-142-254.ec2.internal container=sdn container exited with code 255 (Error): .250871   21073 proxier.go:347] userspace syncProxyRules took 275.278339ms\nI0305 13:49:40.321991   21073 proxier.go:1609] Opened local port "nodePort for openshift-ingress/router-default:http" (:30178/tcp)\nI0305 13:49:40.322210   21073 proxier.go:1609] Opened local port "nodePort for e2e-k8s-service-lb-available-2612/service-test:" (:32333/tcp)\nI0305 13:49:40.322603   21073 proxier.go:1609] Opened local port "nodePort for openshift-ingress/router-default:https" (:31605/tcp)\nI0305 13:49:40.355208   21073 service_health.go:98] Opening healthcheck "openshift-ingress/router-default" on port 32354\nI0305 13:49:40.551158   21073 proxy.go:305] openshift-sdn proxy services and endpoints initialized\nI0305 13:49:40.551199   21073 cmd.go:173] openshift-sdn network plugin registering startup\nI0305 13:49:40.551325   21073 cmd.go:177] openshift-sdn network plugin ready\nI0305 13:49:42.701581   21073 roundrobin.go:267] LoadBalancerRR: Setting endpoints for openshift-multus/multus-admission-controller: to [10.128.0.63:6443 10.129.0.67:6443 10.130.0.16:6443]\nI0305 13:49:42.714978   21073 roundrobin.go:267] LoadBalancerRR: Setting endpoints for openshift-multus/multus-admission-controller: to [10.128.0.63:6443 10.129.0.67:6443]\nI0305 13:49:42.715037   21073 roundrobin.go:217] Delete endpoint 10.130.0.16:6443 for service "openshift-multus/multus-admission-controller:"\nI0305 13:49:42.958136   21073 proxier.go:368] userspace proxy: processing 0 service events\nI0305 13:49:42.958168   21073 proxier.go:347] userspace syncProxyRules took 71.961245ms\nI0305 13:49:43.217953   21073 proxier.go:368] userspace proxy: processing 0 service events\nI0305 13:49:43.217976   21073 proxier.go:347] userspace syncProxyRules took 72.825286ms\nI0305 13:50:13.468408   21073 proxier.go:368] userspace proxy: processing 0 service events\nI0305 13:50:13.468434   21073 proxier.go:347] userspace syncProxyRules took 72.136444ms\nF0305 13:50:21.463226   21073 healthcheck.go:82] SDN healthcheck detected OVS server change, restarting: timed out waiting for the condition\n
Mar 05 13:50:49.025 E ns/openshift-multus pod/multus-6ccw5 node/ip-10-0-130-45.ec2.internal container=kube-multus container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:51:25.848 E ns/openshift-multus pod/multus-5zn8d node/ip-10-0-142-254.ec2.internal container=kube-multus container exited with code 137 (OOMKilled): 
Mar 05 13:52:22.841 E ns/openshift-machine-config-operator pod/machine-config-operator-5c4c48f8ff-qfhrr node/ip-10-0-135-26.ec2.internal container=machine-config-operator container exited with code 2 (Error): e:"", Name:"machine-config", UID:"c28a5385-bee6-4a39-a5c6-ee09f50c082b", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorVersionChanged' clusteroperator/machine-config-operator is bootstrapping to [{operator 0.0.1-2020-03-05-130227}]\nE0305 13:22:06.253785       1 reflector.go:153] github.com/openshift/machine-config-operator/pkg/generated/informers/externalversions/factory.go:101: Failed to list *v1.ControllerConfig: the server could not find the requested resource (get controllerconfigs.machineconfiguration.openshift.io)\nE0305 13:22:06.281180       1 reflector.go:153] github.com/openshift/machine-config-operator/pkg/generated/informers/externalversions/factory.go:101: Failed to list *v1.MachineConfigPool: the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io)\nE0305 13:22:07.292952       1 reflector.go:153] github.com/openshift/machine-config-operator/pkg/generated/informers/externalversions/factory.go:101: Failed to list *v1.MachineConfigPool: the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io)\nI0305 13:22:11.377396       1 sync.go:61] [init mode] synced RenderConfig in 5.456758453s\nI0305 13:22:11.751354       1 sync.go:61] [init mode] synced MachineConfigPools in 373.433565ms\nI0305 13:22:40.153829       1 sync.go:61] [init mode] synced MachineConfigDaemon in 28.402437261s\nI0305 13:22:46.206597       1 sync.go:61] [init mode] synced MachineConfigController in 6.052721066s\nI0305 13:22:51.273763       1 sync.go:61] [init mode] synced MachineConfigServer in 5.067112985s\nI0305 13:23:05.280189       1 sync.go:61] [init mode] synced RequiredPools in 14.0063856s\nI0305 13:23:05.315178       1 sync.go:85] Initialization complete\nE0305 13:25:12.463849       1 leaderelection.go:331] error retrieving resource lock openshift-machine-config-operator/machine-config: Get https://172.30.0.1:443/api/v1/namespaces/openshift-machine-config-operator/configmaps/machine-config: unexpected EOF\n
Mar 05 13:53:17.054 E ns/openshift-dns pod/dns-default-ptgwp node/ip-10-0-135-26.ec2.internal container=dns container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:53:17.054 E ns/openshift-dns pod/dns-default-ptgwp node/ip-10-0-135-26.ec2.internal container=dns-node-resolver container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:54:18.598 E ns/openshift-machine-config-operator pod/machine-config-daemon-8b9hr node/ip-10-0-130-18.ec2.internal container=oauth-proxy container exited with code 143 (Error): 
Mar 05 13:54:30.469 E ns/openshift-machine-config-operator pod/machine-config-daemon-6rg76 node/ip-10-0-153-229.ec2.internal container=oauth-proxy container exited with code 143 (Error): 
Mar 05 13:54:36.308 E ns/openshift-machine-config-operator pod/machine-config-daemon-f5ksk node/ip-10-0-135-26.ec2.internal container=oauth-proxy container exited with code 143 (Error): 
Mar 05 13:54:47.331 E ns/openshift-machine-config-operator pod/machine-config-daemon-nwvtf node/ip-10-0-142-254.ec2.internal container=oauth-proxy container exited with code 143 (Error): 
Mar 05 13:54:59.474 E ns/openshift-machine-config-operator pod/machine-config-daemon-wlxtd node/ip-10-0-130-45.ec2.internal container=oauth-proxy container exited with code 143 (Error): 
Mar 05 13:56:47.165 E ns/openshift-machine-config-operator pod/machine-config-server-vcwbv node/ip-10-0-130-18.ec2.internal container=machine-config-server container exited with code 2 (Error): I0305 13:22:47.962995       1 start.go:38] Version: machine-config-daemon-4.5.0-202003050701-2-gb9fa5093-dirty (b9fa5093eb95cac163db9039e7d2d8bb38f10db6)\nI0305 13:22:47.963899       1 api.go:51] Launching server on :22624\nI0305 13:22:47.963975       1 api.go:51] Launching server on :22623\nI0305 13:24:02.610366       1 api.go:97] Pool worker requested by 10.0.149.20:62483\nI0305 13:24:04.599396       1 api.go:97] Pool worker requested by 10.0.149.20:3810\n
Mar 05 13:56:49.820 E ns/openshift-machine-config-operator pod/machine-config-server-qzbpq node/ip-10-0-135-26.ec2.internal container=machine-config-server container exited with code 2 (Error): I0305 13:22:50.747460       1 start.go:38] Version: machine-config-daemon-4.5.0-202003050701-2-gb9fa5093-dirty (b9fa5093eb95cac163db9039e7d2d8bb38f10db6)\nI0305 13:22:50.748219       1 api.go:51] Launching server on :22624\nI0305 13:22:50.748443       1 api.go:51] Launching server on :22623\n
Mar 05 13:56:57.779 E ns/openshift-monitoring pod/alertmanager-main-2 node/ip-10-0-142-254.ec2.internal container=config-reloader container exited with code 2 (Error): 2020/03/05 13:43:26 Watching directory: "/etc/alertmanager/config"\n
Mar 05 13:56:57.779 E ns/openshift-monitoring pod/alertmanager-main-2 node/ip-10-0-142-254.ec2.internal container=alertmanager-proxy container exited with code 2 (Error): 2020/03/05 13:43:26 provider.go:118: Defaulting client-id to system:serviceaccount:openshift-monitoring:alertmanager-main\n2020/03/05 13:43:26 provider.go:123: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token\n2020/03/05 13:43:26 provider.go:311: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.\n2020/03/05 13:43:27 oauthproxy.go:200: mapping path "/" => upstream "http://localhost:9093/"\n2020/03/05 13:43:27 oauthproxy.go:221: compiled skip-auth-regex => "^/metrics"\n2020/03/05 13:43:27 oauthproxy.go:227: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:alertmanager-main\n2020/03/05 13:43:27 oauthproxy.go:237: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> refresh:disabled\nI0305 13:43:27.137631       1 dynamic_serving_content.go:129] Starting serving::/etc/tls/private/tls.crt::/etc/tls/private/tls.key\n2020/03/05 13:43:27 http.go:107: HTTPS: listening on [::]:9095\n
Mar 05 13:56:57.797 E ns/openshift-monitoring pod/openshift-state-metrics-7cc769dfc4-8x2w6 node/ip-10-0-142-254.ec2.internal container=openshift-state-metrics container exited with code 2 (Error): 
Mar 05 13:56:58.847 E ns/openshift-csi-snapshot-controller-operator pod/csi-snapshot-controller-operator-6855d489cb-zvhck node/ip-10-0-142-254.ec2.internal container=operator container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:56:58.869 E ns/openshift-kube-storage-version-migrator pod/migrator-c89f75c5f-wvldb node/ip-10-0-142-254.ec2.internal container=migrator container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:56:58.940 E ns/openshift-monitoring pod/telemeter-client-68ffb5bdc9-4v2mc node/ip-10-0-142-254.ec2.internal container=reload container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:56:58.940 E ns/openshift-monitoring pod/telemeter-client-68ffb5bdc9-4v2mc node/ip-10-0-142-254.ec2.internal container=kube-rbac-proxy container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:56:58.940 E ns/openshift-monitoring pod/telemeter-client-68ffb5bdc9-4v2mc node/ip-10-0-142-254.ec2.internal container=telemeter-client container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:56:59.047 E ns/openshift-csi-snapshot-controller pod/csi-snapshot-controller-6547777c87-f4zb2 node/ip-10-0-142-254.ec2.internal container=snapshot-controller container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:56:59.843 E ns/openshift-insights pod/insights-operator-856b7c95cf-fslht node/ip-10-0-135-26.ec2.internal container=operator container exited with code 2 (Error): heus/2.15.2 10.129.2.25:46518]\nI0305 13:53:33.079569       1 httplog.go:90] GET /metrics: (8.252136ms) 200 [Prometheus/2.15.2 10.131.0.32:46554]\nI0305 13:53:38.465522       1 httplog.go:90] GET /metrics: (1.622241ms) 200 [Prometheus/2.15.2 10.129.2.25:46518]\nI0305 13:53:49.071473       1 configobserver.go:65] Refreshing configuration from cluster pull secret\nI0305 13:53:49.072400       1 status.go:298] The operator is healthy\nI0305 13:53:49.076617       1 configobserver.go:90] Found cloud.openshift.com token\nI0305 13:53:49.076641       1 configobserver.go:107] Refreshing configuration from cluster secret\nI0305 13:54:03.081304       1 httplog.go:90] GET /metrics: (9.837221ms) 200 [Prometheus/2.15.2 10.131.0.32:46554]\nI0305 13:54:08.465687       1 httplog.go:90] GET /metrics: (1.74788ms) 200 [Prometheus/2.15.2 10.129.2.25:46518]\nI0305 13:54:33.080428       1 httplog.go:90] GET /metrics: (8.996079ms) 200 [Prometheus/2.15.2 10.131.0.32:46554]\nI0305 13:54:38.467410       1 httplog.go:90] GET /metrics: (3.456011ms) 200 [Prometheus/2.15.2 10.129.2.25:46518]\nI0305 13:55:03.081258       1 httplog.go:90] GET /metrics: (9.858733ms) 200 [Prometheus/2.15.2 10.131.0.32:46554]\nI0305 13:55:08.465764       1 httplog.go:90] GET /metrics: (1.900429ms) 200 [Prometheus/2.15.2 10.129.2.25:46518]\nI0305 13:55:33.080033       1 httplog.go:90] GET /metrics: (8.141477ms) 200 [Prometheus/2.15.2 10.131.0.32:46554]\nI0305 13:55:38.465655       1 httplog.go:90] GET /metrics: (1.768716ms) 200 [Prometheus/2.15.2 10.129.2.25:46518]\nI0305 13:55:49.072368       1 status.go:298] The operator is healthy\nI0305 13:56:03.087976       1 httplog.go:90] GET /metrics: (16.523018ms) 200 [Prometheus/2.15.2 10.131.0.32:46554]\nI0305 13:56:08.465871       1 httplog.go:90] GET /metrics: (1.903751ms) 200 [Prometheus/2.15.2 10.129.2.25:46518]\nI0305 13:56:33.080558       1 httplog.go:90] GET /metrics: (9.22046ms) 200 [Prometheus/2.15.2 10.131.0.32:46554]\nI0305 13:56:38.465639       1 httplog.go:90] GET /metrics: (1.919765ms) 200 [Prometheus/2.15.2 10.129.2.25:46518]\n
Mar 05 13:57:00.955 E ns/openshift-authentication-operator pod/authentication-operator-5db6ffbbdc-6qs6l node/ip-10-0-135-26.ec2.internal container=operator container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:57:01.055 E ns/openshift-ingress-operator pod/ingress-operator-7db644686f-vk4gt node/ip-10-0-135-26.ec2.internal container=kube-rbac-proxy container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:57:01.055 E ns/openshift-ingress-operator pod/ingress-operator-7db644686f-vk4gt node/ip-10-0-135-26.ec2.internal container=ingress-operator container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:57:05.485 E ns/openshift-service-ca-operator pod/service-ca-operator-69fb9fcc8d-c84kp node/ip-10-0-135-26.ec2.internal container=operator container exited with code 255 (Error): 
Mar 05 13:57:05.536 E ns/openshift-monitoring pod/prometheus-operator-fb4bd7c6-2cql7 node/ip-10-0-135-26.ec2.internal container=prometheus-operator container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:57:05.664 E ns/openshift-machine-config-operator pod/machine-config-operator-6968b5c887-rrslk node/ip-10-0-135-26.ec2.internal container=machine-config-operator container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:57:05.826 E ns/openshift-operator-lifecycle-manager pod/packageserver-96f89f8b7-brx7c node/ip-10-0-130-18.ec2.internal container=packageserver container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:57:06.508 E ns/openshift-service-ca pod/service-ca-7b984fcc6-4zwxv node/ip-10-0-135-26.ec2.internal container=service-ca-controller container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:57:19.670 E ns/openshift-monitoring pod/prometheus-k8s-0 node/ip-10-0-130-45.ec2.internal container=prometheus container exited with code 1 (Error): caller=main.go:648 msg="Starting TSDB ..."\nlevel=info ts=2020-03-05T13:57:10.328Z caller=web.go:506 component=web msg="Start listening for connections" address=127.0.0.1:9090\nlevel=info ts=2020-03-05T13:57:10.338Z caller=head.go:584 component=tsdb msg="replaying WAL, this may take awhile"\nlevel=info ts=2020-03-05T13:57:10.338Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0\nlevel=info ts=2020-03-05T13:57:10.339Z caller=main.go:663 fs_type=XFS_SUPER_MAGIC\nlevel=info ts=2020-03-05T13:57:10.340Z caller=main.go:664 msg="TSDB started"\nlevel=info ts=2020-03-05T13:57:10.340Z caller=main.go:734 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml\nlevel=info ts=2020-03-05T13:57:10.340Z caller=main.go:517 msg="Stopping scrape discovery manager..."\nlevel=info ts=2020-03-05T13:57:10.340Z caller=main.go:531 msg="Stopping notify discovery manager..."\nlevel=info ts=2020-03-05T13:57:10.340Z caller=main.go:553 msg="Stopping scrape manager..."\nlevel=info ts=2020-03-05T13:57:10.340Z caller=main.go:527 msg="Notify discovery manager stopped"\nlevel=info ts=2020-03-05T13:57:10.340Z caller=main.go:513 msg="Scrape discovery manager stopped"\nlevel=info ts=2020-03-05T13:57:10.340Z caller=manager.go:814 component="rule manager" msg="Stopping rule manager..."\nlevel=info ts=2020-03-05T13:57:10.340Z caller=manager.go:820 component="rule manager" msg="Rule manager stopped"\nlevel=info ts=2020-03-05T13:57:10.340Z caller=main.go:547 msg="Scrape manager stopped"\nlevel=info ts=2020-03-05T13:57:10.341Z caller=notifier.go:598 component=notifier msg="Stopping notification manager..."\nlevel=info ts=2020-03-05T13:57:10.341Z caller=main.go:718 msg="Notifier manager stopped"\nlevel=error ts=2020-03-05
Mar 05 13:57:28.441 E ns/openshift-monitoring pod/prometheus-operator-fb4bd7c6-x9tdn node/ip-10-0-150-76.ec2.internal container=prometheus-operator container exited with code 1 (Error): ts=2020-03-05T13:57:21.81317014Z caller=main.go:208 msg="Starting Prometheus Operator version '0.37.0'."\nts=2020-03-05T13:57:21.862999859Z caller=main.go:98 msg="Staring insecure server on :8080"\nlevel=info ts=2020-03-05T13:57:22.037719674Z caller=operator.go:293 component=thanosoperator msg="connection established" cluster-version=v1.17.1\nlevel=info ts=2020-03-05T13:57:22.048481878Z caller=operator.go:452 component=prometheusoperator msg="connection established" cluster-version=v1.17.1\nlevel=info ts=2020-03-05T13:57:22.117768152Z caller=operator.go:214 component=alertmanageroperator msg="connection established" cluster-version=v1.17.1\nlevel=info ts=2020-03-05T13:57:24.068318864Z caller=operator.go:682 component=thanosoperator msg="CRD updated" crd=ThanosRuler\nlevel=info ts=2020-03-05T13:57:24.085757265Z caller=operator.go:655 component=alertmanageroperator msg="CRD updated" crd=Alertmanager\nlevel=info ts=2020-03-05T13:57:24.211181642Z caller=operator.go:1918 component=prometheusoperator msg="CRD updated" crd=Prometheus\nlevel=info ts=2020-03-05T13:57:24.235947082Z caller=operator.go:1918 component=prometheusoperator msg="CRD updated" crd=ServiceMonitor\nlevel=info ts=2020-03-05T13:57:24.262723951Z caller=operator.go:1918 component=prometheusoperator msg="CRD updated" crd=PodMonitor\nlevel=info ts=2020-03-05T13:57:24.290071931Z caller=operator.go:1918 component=prometheusoperator msg="CRD updated" crd=PrometheusRule\nlevel=info ts=2020-03-05T13:57:27.12131007Z caller=operator.go:230 component=alertmanageroperator msg="CRD API endpoints ready"\nlevel=error ts=2020-03-05T13:57:27.25646652Z caller=operator.go:176 component=alertmanageroperator msg="failed to sync StatefulSet cache"\nts=2020-03-05T13:57:27.256751162Z caller=main.go:304 msg="Unhandled error received. Exiting..." err="creating CRDs failed: waiting for ThanosRuler crd failed: timed out waiting for Custom Resource: failed to list CRD: etcdserver: leader changed"\n
Mar 05 13:57:32.898 E kube-apiserver Kube API started failing: Get https://api.ci-op-s76jkp2x-f83f1.origin-ci-int-aws.dev.rhcloud.com:6443/api/v1/namespaces/kube-system?timeout=5s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Mar 05 13:58:14.055 E ns/openshift-marketplace pod/redhat-operators-54b9fdf9cb-k4qhx node/ip-10-0-153-229.ec2.internal container=redhat-operators container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:58:18.673 E ns/openshift-csi-snapshot-controller pod/csi-snapshot-controller-7c6ddcf999-tgdj7 node/ip-10-0-153-229.ec2.internal container=snapshot-controller container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 13:58:21.078 E ns/openshift-marketplace pod/certified-operators-774cd6766-n5tqx node/ip-10-0-153-229.ec2.internal container=certified-operators container exited with code 2 (Error): 
Mar 05 13:59:40.898 E clusteroperator/openshift-apiserver changed Degraded to True: APIServerDeployment_UnavailablePod: APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable
Mar 05 13:59:41.844 E ns/openshift-cluster-node-tuning-operator pod/tuned-m788z node/ip-10-0-142-254.ec2.internal container=tuned container exited with code 143 (Error): 2-254.ec2.internal" added, tuned profile requested: openshift-node\nI0305 13:43:13.636321    1288 tuned.go:170] disabling system tuned...\nI0305 13:43:13.641541    1288 tuned.go:521] tuned "rendered" added\nI0305 13:43:13.641566    1288 tuned.go:219] extracting tuned profiles\nI0305 13:43:13.644022    1288 tuned.go:176] failed to disable system tuned: Failed to execute operation: Unit file tuned.service does not exist.\nI0305 13:43:14.409268    1288 tuned.go:393] getting recommended profile...\nI0305 13:43:14.621317    1288 tuned.go:421] active profile () != recommended profile (openshift-node)\nI0305 13:43:14.621481    1288 tuned.go:286] starting tuned...\n2020-03-05 13:43:14,795 INFO     tuned.daemon.application: dynamic tuning is globally disabled\n2020-03-05 13:43:14,804 INFO     tuned.daemon.daemon: using sleep interval of 1 second(s)\n2020-03-05 13:43:14,804 INFO     tuned.daemon.daemon: Running in automatic mode, checking what profile is recommended for your configuration.\n2020-03-05 13:43:14,805 INFO     tuned.daemon.daemon: Using 'openshift-node' profile\n2020-03-05 13:43:14,806 INFO     tuned.profiles.loader: loading profile: openshift-node\n2020-03-05 13:43:14,859 INFO     tuned.daemon.controller: starting controller\n2020-03-05 13:43:14,859 INFO     tuned.daemon.daemon: starting tuning\n2020-03-05 13:43:14,874 INFO     tuned.plugins.base: instance cpu: assigning devices cpu2, cpu3, cpu0, cpu1\n2020-03-05 13:43:14,875 INFO     tuned.plugins.plugin_cpu: We are running on an x86 GenuineIntel platform\n2020-03-05 13:43:14,879 WARNING  tuned.plugins.plugin_cpu: your CPU doesn't support MSR_IA32_ENERGY_PERF_BIAS, ignoring CPU energy performance bias\n2020-03-05 13:43:14,882 INFO     tuned.plugins.base: instance disk: assigning devices dm-0, xvda\n2020-03-05 13:43:14,884 INFO     tuned.plugins.base: instance net: assigning devices ens3\n2020-03-05 13:43:15,050 INFO     tuned.plugins.plugin_sysctl: reapplying system sysctl\n2020-03-05 13:43:15,059 INFO     tuned.daemon.daemon: static tuning from profile 'openshift-node' applied\n
Mar 05 13:59:41.884 E ns/openshift-monitoring pod/node-exporter-98gxc node/ip-10-0-142-254.ec2.internal container=node-exporter container exited with code 143 (Error): or gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:56:53Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:57:08Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:57:23Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:57:26Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:57:38Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:57:41Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:57:53Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\n
Mar 05 13:59:41.902 E ns/openshift-sdn pod/ovs-dmvm2 node/ip-10-0-142-254.ec2.internal container=openvswitch container exited with code 143 (Error): >unix#380: 4 flow_mods in the last 0 s (4 deletes)\n2020-03-05T13:56:58.152Z|00093|bridge|INFO|bridge br0: deleted interface veth66602af0 on port 26\n2020-03-05T13:56:58.204Z|00094|connmgr|INFO|br0<->unix#383: 2 flow_mods in the last 0 s (2 deletes)\n2020-03-05T13:56:58.248Z|00095|connmgr|INFO|br0<->unix#386: 4 flow_mods in the last 0 s (4 deletes)\n2020-03-05T13:56:58.281Z|00096|bridge|INFO|bridge br0: deleted interface vethc3d631bf on port 27\n2020-03-05T13:56:58.341Z|00097|connmgr|INFO|br0<->unix#389: 2 flow_mods in the last 0 s (2 deletes)\n2020-03-05T13:56:58.403Z|00098|connmgr|INFO|br0<->unix#392: 4 flow_mods in the last 0 s (4 deletes)\n2020-03-05T13:56:58.425Z|00099|bridge|INFO|bridge br0: deleted interface veth40b782be on port 30\n2020-03-05T13:57:26.477Z|00100|connmgr|INFO|br0<->unix#413: 2 flow_mods in the last 0 s (2 deletes)\n2020-03-05T13:57:26.515Z|00101|connmgr|INFO|br0<->unix#416: 4 flow_mods in the last 0 s (4 deletes)\n2020-03-05T13:57:26.543Z|00102|bridge|INFO|bridge br0: deleted interface veth43e82076 on port 20\n2020-03-05T13:57:26.602Z|00103|connmgr|INFO|br0<->unix#419: 2 flow_mods in the last 0 s (2 deletes)\n2020-03-05T13:57:26.633Z|00104|connmgr|INFO|br0<->unix#422: 4 flow_mods in the last 0 s (4 deletes)\n2020-03-05T13:57:26.662Z|00105|bridge|INFO|bridge br0: deleted interface veth7ab361a5 on port 18\n2020-03-05T13:57:26.609Z|00009|jsonrpc|WARN|unix#351: receive error: Connection reset by peer\n2020-03-05T13:57:26.609Z|00010|reconnect|WARN|unix#351: connection dropped (Connection reset by peer)\n2020-03-05T13:57:42.568Z|00106|connmgr|INFO|br0<->unix#437: 2 flow_mods in the last 0 s (2 deletes)\n2020-03-05T13:57:42.596Z|00107|connmgr|INFO|br0<->unix#440: 4 flow_mods in the last 0 s (4 deletes)\n2020-03-05T13:57:42.618Z|00108|bridge|INFO|bridge br0: deleted interface vethc3af86e6 on port 34\n2020-03-05T13:57:42.612Z|00011|jsonrpc|WARN|unix#370: receive error: Connection reset by peer\n2020-03-05T13:57:42.612Z|00012|reconnect|WARN|unix#370: connection dropped (Connection reset by peer)\ninfo: Saving flows ...\n
Mar 05 13:59:41.947 E ns/openshift-multus pod/multus-l2ltr node/ip-10-0-142-254.ec2.internal container=kube-multus container exited with code 143 (Error): 
Mar 05 13:59:41.991 E ns/openshift-machine-config-operator pod/machine-config-daemon-nb4fk node/ip-10-0-142-254.ec2.internal container=oauth-proxy container exited with code 143 (Error): 
Mar 05 13:59:44.788 E ns/openshift-multus pod/multus-l2ltr node/ip-10-0-142-254.ec2.internal invariant violation: pod may not transition Running->Pending
Mar 05 13:59:44.913 E ns/openshift-kube-controller-manager pod/kube-controller-manager-ip-10-0-135-26.ec2.internal node/ip-10-0-135-26.ec2.internal container=cluster-policy-controller container exited with code 1 (Error): I0305 13:40:06.978628       1 policy_controller.go:41] Starting controllers on 0.0.0.0:10357 (v0.0.0-unknown)\nI0305 13:40:06.981175       1 standalone_apiserver.go:103] Started health checks at 0.0.0.0:10357\nI0305 13:40:06.981298       1 leaderelection.go:242] attempting to acquire leader lease  openshift-kube-controller-manager/cluster-policy-controller...\n
Mar 05 13:59:44.913 E ns/openshift-kube-controller-manager pod/kube-controller-manager-ip-10-0-135-26.ec2.internal node/ip-10-0-135-26.ec2.internal container=kube-controller-manager-cert-syncer container exited with code 2 (Error):     1 certsync_controller.go:162] Syncing secrets: [{csr-signer false}]\nI0305 13:56:48.429024       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 13:56:48.429664       1 certsync_controller.go:162] Syncing secrets: [{csr-signer false}]\nI0305 13:56:53.367506       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 13:56:53.368082       1 certsync_controller.go:162] Syncing secrets: [{csr-signer false}]\nI0305 13:56:58.461399       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 13:56:58.461974       1 certsync_controller.go:162] Syncing secrets: [{csr-signer false}]\nI0305 13:57:03.390918       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 13:57:03.391399       1 certsync_controller.go:162] Syncing secrets: [{csr-signer false}]\nI0305 13:57:08.477951       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 13:57:08.478389       1 certsync_controller.go:162] Syncing secrets: [{csr-signer false}]\nI0305 13:57:13.397304       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 13:57:13.397791       1 certsync_controller.go:162] Syncing secrets: [{csr-signer false}]\nI0305 13:57:18.490081       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 13:57:18.490606       1 certsync_controller.go:162] Syncing secrets: [{csr-signer false}]\nI0305 13:57:23.414361       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 13:57:23.415611       1 certsync_controller.go:162] Syncing secrets: [{csr-signer false}]\n
Mar 05 13:59:44.913 E ns/openshift-kube-controller-manager pod/kube-controller-manager-ip-10-0-135-26.ec2.internal node/ip-10-0-135-26.ec2.internal container=kube-controller-manager container exited with code 2 (Error): roller_utils.go:603] Controller packageserver-7ff55c848c deleting pod openshift-operator-lifecycle-manager/packageserver-7ff55c848c-dp2rf\nI0305 13:57:25.132695       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-operator-lifecycle-manager", Name:"packageserver", UID:"2057d72c-059f-4f6c-987c-f64736087176", APIVersion:"apps/v1", ResourceVersion:"34139", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down replica set packageserver-7ff55c848c to 0\nI0305 13:57:25.145618       1 event.go:281] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"openshift-operator-lifecycle-manager", Name:"packageserver-7ff55c848c", UID:"54c9863c-5a5b-4a12-8492-40aa81fc008e", APIVersion:"apps/v1", ResourceVersion:"34561", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: packageserver-7ff55c848c-dp2rf\nI0305 13:57:25.154080       1 replica_set.go:561] Too few replicas for ReplicaSet openshift-operator-lifecycle-manager/packageserver-f74474b4d, need 2, creating 1\nI0305 13:57:25.154778       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-operator-lifecycle-manager", Name:"packageserver", UID:"2057d72c-059f-4f6c-987c-f64736087176", APIVersion:"apps/v1", ResourceVersion:"34139", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set packageserver-f74474b4d to 2\nI0305 13:57:25.161682       1 deployment_controller.go:484] Error syncing deployment openshift-operator-lifecycle-manager/packageserver: Operation cannot be fulfilled on deployments.apps "packageserver": the object has been modified; please apply your changes to the latest version and try again\nI0305 13:57:25.192660       1 event.go:281] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"openshift-operator-lifecycle-manager", Name:"packageserver-f74474b4d", UID:"4715e4cc-513f-492c-b1bc-5232cfb181ba", APIVersion:"apps/v1", ResourceVersion:"34566", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: packageserver-f74474b4d-pjlt7\n
Mar 05 13:59:44.913 E ns/openshift-kube-controller-manager pod/kube-controller-manager-ip-10-0-135-26.ec2.internal node/ip-10-0-135-26.ec2.internal container=kube-controller-manager-recovery-controller container exited with code 255 (Error): .authorization.k8s.io "system:basic-user" not found, clusterrole.rbac.authorization.k8s.io "basic-user" not found, clusterrole.rbac.authorization.k8s.io "system:public-info-viewer" not found, clusterrole.rbac.authorization.k8s.io "self-access-reviewer" not found, clusterrole.rbac.authorization.k8s.io "cluster-status" not found, clusterrole.rbac.authorization.k8s.io "cluster-admin" not found, clusterrole.rbac.authorization.k8s.io "system:build-strategy-jenkinspipeline" not found, clusterrole.rbac.authorization.k8s.io "system:build-strategy-source" not found, clusterrole.rbac.authorization.k8s.io "system:oauth-token-deleter" not found, clusterrole.rbac.authorization.k8s.io "system:openshift:discovery" not found, clusterrole.rbac.authorization.k8s.io "system:discovery" not found, clusterrole.rbac.authorization.k8s.io "system:webhook" not found, clusterrole.rbac.authorization.k8s.io "system:openshift:public-info-viewer" not found, clusterrole.rbac.authorization.k8s.io "system:build-strategy-docker" not found, clusterrole.rbac.authorization.k8s.io "system:image-puller" not found]\nE0305 13:45:32.707993       1 reflector.go:307] runtime/asm_amd64.s:1357: Failed to watch *v1.Secret: unknown (get secrets)\nE0305 13:45:32.708176       1 reflector.go:307] runtime/asm_amd64.s:1357: Failed to watch *v1.ConfigMap: unknown (get configmaps)\nE0305 13:45:32.708107       1 reflector.go:307] runtime/asm_amd64.s:1357: Failed to watch *v1.ConfigMap: unknown (get configmaps)\nE0305 13:45:32.708134       1 reflector.go:307] runtime/asm_amd64.s:1357: Failed to watch *v1.ConfigMap: unknown (get configmaps)\nE0305 13:45:32.708160       1 reflector.go:307] runtime/asm_amd64.s:1357: Failed to watch *v1.Secret: unknown (get secrets)\nI0305 13:57:26.747953       1 cmd.go:84] Received SIGTERM or SIGINT signal, shutting down controller.\nI0305 13:57:26.749333       1 csrcontroller.go:83] Shutting down CSR controller\nI0305 13:57:26.749416       1 csrcontroller.go:85] CSR controller shut down\nF0305 13:57:26.749601       1 builder.go:209] server exited\n
Mar 05 13:59:44.966 E ns/openshift-kube-scheduler pod/openshift-kube-scheduler-ip-10-0-135-26.ec2.internal node/ip-10-0-135-26.ec2.internal container=scheduler container exited with code 2 (Error): atch *v1.PersistentVolumeClaim: Get https://localhost:6443/api/v1/persistentvolumeclaims?allowWatchBookmarks=true&resourceVersion=20510&timeout=6m30s&timeoutSeconds=390&watch=true: dial tcp [::1]:6443: connect: connection refused\nE0305 13:45:25.942233       1 reflector.go:307] k8s.io/client-go/informers/factory.go:135: Failed to watch *v1.PersistentVolume: Get https://localhost:6443/api/v1/persistentvolumes?allowWatchBookmarks=true&resourceVersion=20510&timeout=8m24s&timeoutSeconds=504&watch=true: dial tcp [::1]:6443: connect: connection refused\nE0305 13:45:25.942993       1 reflector.go:307] k8s.io/client-go/informers/factory.go:135: Failed to watch *v1.StorageClass: Get https://localhost:6443/apis/storage.k8s.io/v1/storageclasses?allowWatchBookmarks=true&resourceVersion=20516&timeout=9m7s&timeoutSeconds=547&watch=true: dial tcp [::1]:6443: connect: connection refused\nE0305 13:45:25.945046       1 reflector.go:307] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:209: Failed to watch *v1.ConfigMap: Get https://localhost:6443/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=22908&timeout=8m16s&timeoutSeconds=496&watch=true: dial tcp [::1]:6443: connect: connection refused\nE0305 13:45:25.945130       1 reflector.go:307] k8s.io/client-go/informers/factory.go:135: Failed to watch *v1.Node: Get https://localhost:6443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=28274&timeout=8m54s&timeoutSeconds=534&watch=true: dial tcp [::1]:6443: connect: connection refused\nE0305 13:45:25.946552       1 reflector.go:307] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:209: Failed to watch *v1.ConfigMap: Get https://localhost:6443/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=22908&timeout=9m23s&timeoutSeconds=563&watch=true: dial tcp [::1]:6443: connect: connection refused\n
Mar 05 13:59:45.040 E ns/openshift-cluster-node-tuning-operator pod/tuned-6hl4s node/ip-10-0-135-26.ec2.internal container=tuned container exited with code 143 (Error): ested: openshift-control-plane\nI0305 13:43:28.352716    2038 tuned.go:170] disabling system tuned...\nI0305 13:43:28.354947    2038 tuned.go:521] tuned "rendered" added\nI0305 13:43:28.355014    2038 tuned.go:219] extracting tuned profiles\nI0305 13:43:28.405439    2038 tuned.go:176] failed to disable system tuned: Failed to execute operation: Unit file tuned.service does not exist.\nI0305 13:43:29.290115    2038 tuned.go:393] getting recommended profile...\nI0305 13:43:29.508619    2038 tuned.go:421] active profile () != recommended profile (openshift-control-plane)\nI0305 13:43:29.508713    2038 tuned.go:286] starting tuned...\n2020-03-05 13:43:29,708 INFO     tuned.daemon.application: dynamic tuning is globally disabled\n2020-03-05 13:43:29,746 INFO     tuned.daemon.daemon: using sleep interval of 1 second(s)\n2020-03-05 13:43:29,747 INFO     tuned.daemon.daemon: Running in automatic mode, checking what profile is recommended for your configuration.\n2020-03-05 13:43:29,748 INFO     tuned.daemon.daemon: Using 'openshift-control-plane' profile\n2020-03-05 13:43:29,749 INFO     tuned.profiles.loader: loading profile: openshift-control-plane\n2020-03-05 13:43:29,864 INFO     tuned.daemon.controller: starting controller\n2020-03-05 13:43:29,865 INFO     tuned.daemon.daemon: starting tuning\n2020-03-05 13:43:29,915 INFO     tuned.plugins.base: instance cpu: assigning devices cpu2, cpu3, cpu0, cpu1\n2020-03-05 13:43:29,924 INFO     tuned.plugins.plugin_cpu: We are running on an x86 GenuineIntel platform\n2020-03-05 13:43:29,931 WARNING  tuned.plugins.plugin_cpu: your CPU doesn't support MSR_IA32_ENERGY_PERF_BIAS, ignoring CPU energy performance bias\n2020-03-05 13:43:29,937 INFO     tuned.plugins.base: instance disk: assigning devices dm-0, xvda\n2020-03-05 13:43:29,947 INFO     tuned.plugins.base: instance net: assigning devices ens3\n2020-03-05 13:43:30,225 INFO     tuned.plugins.plugin_sysctl: reapplying system sysctl\n2020-03-05 13:43:30,242 INFO     tuned.daemon.daemon: static tuning from profile 'openshift-control-plane' applied\n
Mar 05 13:59:45.065 E ns/openshift-monitoring pod/node-exporter-p6lzl node/ip-10-0-135-26.ec2.internal container=node-exporter container exited with code 143 (Error): or gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:56:18Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:56:28Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:56:33Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:56:43Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:56:48Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:56:58Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T13:57:13Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\n
Mar 05 13:59:45.090 E ns/openshift-controller-manager pod/controller-manager-pl8xw node/ip-10-0-135-26.ec2.internal container=controller-manager container exited with code 1 (Error): I0305 13:44:00.255727       1 controller_manager.go:39] Starting controllers on 0.0.0.0:8443 (unknown)\nI0305 13:44:00.258615       1 controller_manager.go:50] DeploymentConfig controller using images from "registry.svc.ci.openshift.org/ci-op-s76jkp2x/stable@sha256:f12127cf48ffc713b25ad83e9d98b64c5e11b1f1af06306cf0a8a5a090a59b6a"\nI0305 13:44:00.258811       1 controller_manager.go:56] Build controller using images from "registry.svc.ci.openshift.org/ci-op-s76jkp2x/stable@sha256:471891b26e981d2ed9c87cdd306bc028abe62b760a7af413bd9c05389c4ea5a4"\nI0305 13:44:00.258754       1 standalone_apiserver.go:98] Started health checks at 0.0.0.0:8443\nI0305 13:44:00.259016       1 leaderelection.go:242] attempting to acquire leader lease  openshift-controller-manager/openshift-master-controllers...\n
Mar 05 13:59:45.157 E ns/openshift-sdn pod/sdn-controller-md2s7 node/ip-10-0-135-26.ec2.internal container=sdn-controller container exited with code 2 (Error): I0305 13:48:35.051234       1 leaderelection.go:242] attempting to acquire leader lease  openshift-sdn/openshift-network-controller...\n
Mar 05 13:59:45.175 E ns/openshift-multus pod/multus-c5k99 node/ip-10-0-135-26.ec2.internal container=kube-multus container exited with code 143 (Error): 
Mar 05 13:59:45.193 E ns/openshift-sdn pod/ovs-hhpq8 node/ip-10-0-135-26.ec2.internal container=openvswitch container exited with code 143 (Error): 3-05T13:57:03.862Z|00018|reconnect|WARN|unix#411: connection dropped (Broken pipe)\n2020-03-05T13:57:04.672Z|00139|connmgr|INFO|br0<->unix#504: 2 flow_mods in the last 0 s (2 deletes)\n2020-03-05T13:57:04.763Z|00140|connmgr|INFO|br0<->unix#507: 4 flow_mods in the last 0 s (4 deletes)\n2020-03-05T13:57:04.822Z|00141|bridge|INFO|bridge br0: deleted interface vethb2c8d19d on port 69\n2020-03-05T13:57:04.930Z|00142|connmgr|INFO|br0<->unix#510: 2 flow_mods in the last 0 s (2 deletes)\n2020-03-05T13:57:04.985Z|00143|connmgr|INFO|br0<->unix#513: 4 flow_mods in the last 0 s (4 deletes)\n2020-03-05T13:57:05.027Z|00144|bridge|INFO|bridge br0: deleted interface vethcd2f01e7 on port 67\n2020-03-05T13:57:05.074Z|00145|connmgr|INFO|br0<->unix#516: 2 flow_mods in the last 0 s (2 deletes)\n2020-03-05T13:57:05.135Z|00146|connmgr|INFO|br0<->unix#519: 4 flow_mods in the last 0 s (4 deletes)\n2020-03-05T13:57:05.162Z|00147|bridge|INFO|bridge br0: deleted interface veth9916ad3c on port 72\n2020-03-05T13:57:04.789Z|00019|jsonrpc|WARN|unix#427: send error: Broken pipe\n2020-03-05T13:57:04.789Z|00020|reconnect|WARN|unix#427: connection dropped (Broken pipe)\n2020-03-05T13:57:05.085Z|00021|jsonrpc|WARN|unix#435: send error: Broken pipe\n2020-03-05T13:57:05.085Z|00022|reconnect|WARN|unix#435: connection dropped (Broken pipe)\n2020-03-05T13:57:05.148Z|00023|jsonrpc|WARN|unix#437: receive error: Connection reset by peer\n2020-03-05T13:57:05.148Z|00024|reconnect|WARN|unix#437: connection dropped (Connection reset by peer)\n2020-03-05T13:57:23.363Z|00148|connmgr|INFO|br0<->unix#534: 2 flow_mods in the last 0 s (2 deletes)\n2020-03-05T13:57:23.427Z|00149|connmgr|INFO|br0<->unix#537: 4 flow_mods in the last 0 s (4 deletes)\n2020-03-05T13:57:23.461Z|00150|bridge|INFO|bridge br0: deleted interface veth9b2d330e on port 70\ninfo: Saving flows ...\n2020-03-05T13:57:26Z|00001|vconn|WARN|unix:/var/run/openvswitch/br0.mgmt: version negotiation failed (we support version 0x01, peer supports version 0x04)\novs-ofctl: br0: failed to connect to socket (Broken pipe)\nTerminated\n
Mar 05 13:59:45.216 E ns/openshift-multus pod/multus-admission-controller-6zx9g node/ip-10-0-135-26.ec2.internal container=multus-admission-controller container exited with code 255 (Error): 
Mar 05 13:59:45.258 E ns/openshift-machine-config-operator pod/machine-config-daemon-sq5rw node/ip-10-0-135-26.ec2.internal container=oauth-proxy container exited with code 143 (Error): 
Mar 05 13:59:45.270 E ns/openshift-machine-config-operator pod/machine-config-server-9qgft node/ip-10-0-135-26.ec2.internal container=machine-config-server container exited with code 2 (Error): I0305 13:57:01.516634       1 start.go:38] Version: machine-config-daemon-4.5.0-202003050701-2-gb9fa5093-dirty (b9fa5093eb95cac163db9039e7d2d8bb38f10db6)\nI0305 13:57:01.518058       1 api.go:51] Launching server on :22624\nI0305 13:57:01.519079       1 api.go:51] Launching server on :22623\n
Mar 05 13:59:48.007 E ns/openshift-kube-apiserver pod/kube-apiserver-ip-10-0-135-26.ec2.internal node/ip-10-0-135-26.ec2.internal container=kube-apiserver container exited with code 1 (Error): efused". Reconnecting...\nE0305 13:57:26.831767       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted\nE0305 13:57:26.847950       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted\nE0305 13:57:26.848120       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted\nE0305 13:57:26.848288       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted\nE0305 13:57:26.848539       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted\nE0305 13:57:26.848756       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted\nE0305 13:57:26.848801       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted\nE0305 13:57:26.849160       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted\nE0305 13:57:26.849378       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted\nE0305 13:57:26.849421       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted\nE0305 13:57:26.849648       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted\nE0305 13:57:26.849702       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted\nE0305 13:57:26.849932       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted\nE0305 13:57:26.850141       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted\nE0305 13:57:26.850175       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted\nE0305 13:57:26.850307       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted\nE0305 13:57:26.850451       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted\n
Mar 05 13:59:48.007 E ns/openshift-kube-apiserver pod/kube-apiserver-ip-10-0-135-26.ec2.internal node/ip-10-0-135-26.ec2.internal container=kube-apiserver-insecure-readyz container exited with code 2 (Error): I0305 13:45:27.909421       1 readyz.go:103] Listening on 0.0.0.0:6080\n
Mar 05 13:59:48.007 E ns/openshift-kube-apiserver pod/kube-apiserver-ip-10-0-135-26.ec2.internal node/ip-10-0-135-26.ec2.internal container=kube-apiserver-cert-syncer container exited with code 2 (Error): ce-network-serving-certkey false} {external-loadbalancer-serving-certkey false} {internal-loadbalancer-serving-certkey false} {bound-service-account-signing-key false} {user-serving-cert true} {user-serving-cert-000 true} {user-serving-cert-001 true} {user-serving-cert-002 true} {user-serving-cert-003 true} {user-serving-cert-004 true} {user-serving-cert-005 true} {user-serving-cert-006 true} {user-serving-cert-007 true} {user-serving-cert-008 true} {user-serving-cert-009 true}]\nI0305 13:57:13.741454       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 13:57:13.742288       1 certsync_controller.go:162] Syncing secrets: [{aggregator-client false} {localhost-serving-cert-certkey false} {service-network-serving-certkey false} {external-loadbalancer-serving-certkey false} {internal-loadbalancer-serving-certkey false} {bound-service-account-signing-key false} {user-serving-cert true} {user-serving-cert-000 true} {user-serving-cert-001 true} {user-serving-cert-002 true} {user-serving-cert-003 true} {user-serving-cert-004 true} {user-serving-cert-005 true} {user-serving-cert-006 true} {user-serving-cert-007 true} {user-serving-cert-008 true} {user-serving-cert-009 true}]\nI0305 13:57:23.756350       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 13:57:23.756719       1 certsync_controller.go:162] Syncing secrets: [{aggregator-client false} {localhost-serving-cert-certkey false} {service-network-serving-certkey false} {external-loadbalancer-serving-certkey false} {internal-loadbalancer-serving-certkey false} {bound-service-account-signing-key false} {user-serving-cert true} {user-serving-cert-000 true} {user-serving-cert-001 true} {user-serving-cert-002 true} {user-serving-cert-003 true} {user-serving-cert-004 true} {user-serving-cert-005 true} {user-serving-cert-006 true} {user-serving-cert-007 true} {user-serving-cert-008 true} {user-serving-cert-009 true}]\n
Mar 05 13:59:48.007 E ns/openshift-kube-apiserver pod/kube-apiserver-ip-10-0-135-26.ec2.internal node/ip-10-0-135-26.ec2.internal container=kube-apiserver-cert-regeneration-controller container exited with code 255 (Error): W0305 13:45:27.585469       1 cmd.go:200] Using insecure, self-signed certificates\nI0305 13:45:27.585989       1 crypto.go:580] Generating new CA for cert-regeneration-controller-signer@1583415927 cert, and key in /tmp/serving-cert-187214832/serving-signer.crt, /tmp/serving-cert-187214832/serving-signer.key\nI0305 13:45:28.325820       1 observer_polling.go:155] Starting file observer\nI0305 13:45:35.573912       1 leaderelection.go:242] attempting to acquire leader lease  openshift-kube-apiserver/cert-regeneration-controller-lock...\nI0305 13:57:26.564508       1 cmd.go:84] Received SIGTERM or SIGINT signal, shutting down controller.\nF0305 13:57:26.564615       1 leaderelection.go:67] leaderelection lost\n
Mar 05 13:59:48.748 E clusteroperator/etcd changed Degraded to True: EtcdMembers_UnhealthyMembers::NodeController_MasterNodesReady: NodeControllerDegraded: The master nodes not ready: node "ip-10-0-135-26.ec2.internal" not ready since 2020-03-05 13:59:44 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: Missing CNI default network)\nEtcdMembersDegraded: ip-10-0-135-26.ec2.internal members are unhealthy,  members are unknown
Mar 05 13:59:49.162 E ns/openshift-etcd pod/etcd-ip-10-0-135-26.ec2.internal node/ip-10-0-135-26.ec2.internal container=etcd-metrics container exited with code 2 (Error): 2020-03-05 13:39:34.020628 I | etcdmain: ServerTLS: cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-135-26.ec2.internal.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-135-26.ec2.internal.key, ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-metrics-proxy-serving-ca/ca-bundle.crt, trusted-ca = , client-cert-auth = false, crl-file = \n2020-03-05 13:39:34.022250 I | etcdmain: listening for grpc-proxy client requests on 127.0.0.1:9977\n2020-03-05 13:39:34.023051 I | etcdmain: ClientTLS: cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-peer/etcd-peer-ip-10-0-135-26.ec2.internal.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-peer/etcd-peer-ip-10-0-135-26.ec2.internal.key, ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-peer-client-ca/ca-bundle.crt, trusted-ca = , client-cert-auth = false, crl-file = \nWARNING: 2020/03/05 13:39:34 grpc: addrConn.createTransport failed to connect to {https://etcd-2.ci-op-s76jkp2x-f83f1.origin-ci-int-aws.dev.rhcloud.com:9978 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.135.26:9978: connect: connection refused". Reconnecting...\n2020-03-05 13:39:34.027768 I | etcdmain: grpc-proxy: listening for metrics on https://0.0.0.0:9979\nWARNING: 2020/03/05 13:39:35 grpc: addrConn.createTransport failed to connect to {https://etcd-2.ci-op-s76jkp2x-f83f1.origin-ci-int-aws.dev.rhcloud.com:9978 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.135.26:9978: connect: connection refused". Reconnecting...\n
Mar 05 13:59:50.292 E ns/openshift-multus pod/multus-c5k99 node/ip-10-0-135-26.ec2.internal invariant violation: pod may not transition Running->Pending
Mar 05 13:59:51.200 E ns/openshift-machine-config-operator pod/machine-config-daemon-nb4fk node/ip-10-0-142-254.ec2.internal container=oauth-proxy container exited with code 1 (Error): 
Mar 05 13:59:52.567 E ns/openshift-multus pod/multus-c5k99 node/ip-10-0-135-26.ec2.internal invariant violation: pod may not transition Running->Pending
Mar 05 13:59:55.194 E clusteroperator/dns changed Degraded to True: NotAllDNSesAvailable: Not all desired DNS DaemonSets available
Mar 05 13:59:56.665 E ns/openshift-machine-config-operator pod/machine-config-daemon-sq5rw node/ip-10-0-135-26.ec2.internal container=oauth-proxy container exited with code 1 (Error): 
Mar 05 14:00:29.685 E ns/openshift-monitoring pod/thanos-querier-657794b8f9-9s4kr node/ip-10-0-153-229.ec2.internal container=oauth-proxy container exited with code 2 (Error): 2020/03/05 13:43:43 provider.go:118: Defaulting client-id to system:serviceaccount:openshift-monitoring:thanos-querier\n2020/03/05 13:43:43 provider.go:123: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token\n2020/03/05 13:43:43 provider.go:311: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.\n2020/03/05 13:43:43 oauthproxy.go:200: mapping path "/" => upstream "http://localhost:9090/"\n2020/03/05 13:43:43 oauthproxy.go:221: compiled skip-auth-regex => "^/metrics"\n2020/03/05 13:43:43 oauthproxy.go:227: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:thanos-querier\n2020/03/05 13:43:43 oauthproxy.go:237: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> refresh:disabled\n2020/03/05 13:43:43 main.go:154: using htpasswd file /etc/proxy/htpasswd/auth\n2020/03/05 13:43:43 http.go:107: HTTPS: listening on [::]:9091\nI0305 13:43:43.187361       1 dynamic_serving_content.go:129] Starting serving::/etc/tls/private/tls.crt::/etc/tls/private/tls.key\n
Mar 05 14:00:30.683 E ns/openshift-monitoring pod/alertmanager-main-0 node/ip-10-0-153-229.ec2.internal container=config-reloader container exited with code 2 (Error): 2020/03/05 13:44:15 Watching directory: "/etc/alertmanager/config"\n
Mar 05 14:00:30.683 E ns/openshift-monitoring pod/alertmanager-main-0 node/ip-10-0-153-229.ec2.internal container=alertmanager-proxy container exited with code 2 (Error): 2020/03/05 13:44:16 provider.go:118: Defaulting client-id to system:serviceaccount:openshift-monitoring:alertmanager-main\n2020/03/05 13:44:16 provider.go:123: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token\n2020/03/05 13:44:16 provider.go:311: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.\n2020/03/05 13:44:16 oauthproxy.go:200: mapping path "/" => upstream "http://localhost:9093/"\n2020/03/05 13:44:16 oauthproxy.go:221: compiled skip-auth-regex => "^/metrics"\n2020/03/05 13:44:16 oauthproxy.go:227: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:alertmanager-main\n2020/03/05 13:44:16 oauthproxy.go:237: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> refresh:disabled\n2020/03/05 13:44:16 http.go:107: HTTPS: listening on [::]:9095\nI0305 13:44:16.434827       1 dynamic_serving_content.go:129] Starting serving::/etc/tls/private/tls.crt::/etc/tls/private/tls.key\n
Mar 05 14:00:30.804 E ns/openshift-marketplace pod/redhat-operators-6f8d4d44b6-vcg96 node/ip-10-0-153-229.ec2.internal container=redhat-operators container exited with code 2 (Error): 
Mar 05 14:00:30.852 E ns/openshift-csi-snapshot-controller pod/csi-snapshot-controller-db7cd8967-rdjs7 node/ip-10-0-153-229.ec2.internal container=snapshot-controller container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 14:00:30.864 E ns/openshift-marketplace pod/certified-operators-56b9fd45c4-4jzmf node/ip-10-0-153-229.ec2.internal container=certified-operators container exited with code 2 (Error): 
Mar 05 14:00:31.748 E ns/openshift-image-registry pod/image-registry-574cff6d9-nfhqp node/ip-10-0-153-229.ec2.internal container=registry container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 14:00:31.799 E ns/openshift-marketplace pod/community-operators-544c68887d-mj8w6 node/ip-10-0-153-229.ec2.internal container=community-operators container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 14:00:35.969 E ns/openshift-cluster-machine-approver pod/machine-approver-69bd5c88c-6vnp8 node/ip-10-0-150-76.ec2.internal container=machine-approver-controller container exited with code 2 (Error): 3:57:05.218776       1 config.go:23] machine approver config: {NodeClientCert:{Disabled:false}}\nI0305 13:57:05.219108       1 main.go:236] Starting Machine Approver\nI0305 13:57:05.319501       1 main.go:146] CSR csr-vpxw4 added\nI0305 13:57:05.319665       1 main.go:149] CSR csr-vpxw4 is already approved\nI0305 13:57:05.319726       1 main.go:146] CSR csr-4rj6f added\nI0305 13:57:05.319768       1 main.go:149] CSR csr-4rj6f is already approved\nI0305 13:57:05.319875       1 main.go:146] CSR csr-c6fwm added\nI0305 13:57:05.319920       1 main.go:149] CSR csr-c6fwm is already approved\nI0305 13:57:05.320044       1 main.go:146] CSR csr-nwr8q added\nI0305 13:57:05.320086       1 main.go:149] CSR csr-nwr8q is already approved\nI0305 13:57:05.320124       1 main.go:146] CSR csr-qqhnj added\nI0305 13:57:05.320221       1 main.go:149] CSR csr-qqhnj is already approved\nI0305 13:57:05.320269       1 main.go:146] CSR csr-rpxbt added\nI0305 13:57:05.320313       1 main.go:149] CSR csr-rpxbt is already approved\nI0305 13:57:05.320445       1 main.go:146] CSR csr-x84n7 added\nI0305 13:57:05.320563       1 main.go:149] CSR csr-x84n7 is already approved\nI0305 13:57:05.320637       1 main.go:146] CSR csr-b6ddx added\nI0305 13:57:05.320747       1 main.go:149] CSR csr-b6ddx is already approved\nI0305 13:57:05.320802       1 main.go:146] CSR csr-fdz6b added\nI0305 13:57:05.320905       1 main.go:149] CSR csr-fdz6b is already approved\nI0305 13:57:05.320960       1 main.go:146] CSR csr-nmstz added\nI0305 13:57:05.321000       1 main.go:149] CSR csr-nmstz is already approved\nI0305 13:57:05.321084       1 main.go:146] CSR csr-qb6pc added\nI0305 13:57:05.321135       1 main.go:149] CSR csr-qb6pc is already approved\nI0305 13:57:05.321195       1 main.go:146] CSR csr-s669d added\nI0305 13:57:05.321247       1 main.go:149] CSR csr-s669d is already approved\nW0305 13:57:34.222626       1 reflector.go:289] github.com/openshift/cluster-machine-approver/main.go:238: watch of *v1beta1.CertificateSigningRequest ended with: too old resource version: 25945 (34655)\n
Mar 05 14:00:35.996 E ns/openshift-insights pod/insights-operator-856b7c95cf-cn6s2 node/ip-10-0-150-76.ec2.internal container=operator container exited with code 2 (Error): 32]\nI0305 13:58:04.544806       1 httplog.go:90] GET /metrics: (7.109846ms) 200 [Prometheus/2.15.2 10.129.2.25:45990]\nI0305 13:58:16.596068       1 httplog.go:90] GET /metrics: (8.431457ms) 200 [Prometheus/2.15.2 10.128.2.33:53232]\nI0305 13:58:34.544503       1 httplog.go:90] GET /metrics: (6.746397ms) 200 [Prometheus/2.15.2 10.129.2.25:45990]\nI0305 13:58:37.758561       1 diskrecorder.go:303] Found files to send: [/var/lib/insights-operator/insights-2020-03-05-135723.tar.gz]\nI0305 13:58:37.758713       1 insightsuploader.go:126] Uploading latest report since 0001-01-01T00:00:00Z\nI0305 13:58:37.775346       1 insightsclient.go:163] Uploading application/vnd.redhat.openshift.periodic to https://cloud.redhat.com/api/ingress/v1/upload\nI0305 13:58:37.997911       1 insightsclient.go:213] Successfully reported id=2020-03-05T13:58:37Z x-rh-insights-request-id=e5a8a2f469764d4baa9a37eb34ba7ebf, wrote=15529\nI0305 13:58:37.997952       1 insightsuploader.go:150] Uploaded report successfully in 239.247853ms\nI0305 13:58:37.997976       1 status.go:89] Initializing last reported time to 2020-03-05T13:58:37Z\nI0305 13:58:38.004013       1 status.go:298] The operator is healthy\nI0305 13:58:46.595378       1 httplog.go:90] GET /metrics: (7.96413ms) 200 [Prometheus/2.15.2 10.128.2.33:53232]\nI0305 13:59:04.546010       1 httplog.go:90] GET /metrics: (8.011775ms) 200 [Prometheus/2.15.2 10.129.2.25:45990]\nI0305 13:59:16.594444       1 httplog.go:90] GET /metrics: (7.040659ms) 200 [Prometheus/2.15.2 10.128.2.33:53232]\nI0305 13:59:22.515477       1 status.go:298] The operator is healthy\nI0305 13:59:34.547495       1 httplog.go:90] GET /metrics: (9.776699ms) 200 [Prometheus/2.15.2 10.129.2.25:45990]\nI0305 13:59:46.597820       1 httplog.go:90] GET /metrics: (10.409521ms) 200 [Prometheus/2.15.2 10.128.2.33:53232]\nI0305 14:00:04.544136       1 httplog.go:90] GET /metrics: (6.541931ms) 200 [Prometheus/2.15.2 10.129.2.25:45990]\nI0305 14:00:16.595006       1 httplog.go:90] GET /metrics: (7.706539ms) 200 [Prometheus/2.15.2 10.128.2.33:53232]\n
Mar 05 14:00:36.080 E ns/openshift-apiserver-operator pod/openshift-apiserver-operator-7c7bb46bf5-tc4xg node/ip-10-0-150-76.ec2.internal container=openshift-apiserver-operator container exited with code 255 (Error): 6-3e4f-4758-b3f0-890a8014e58c", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ObserveStorageUpdated' Updated storage urls to https://10.0.130.18:2379,https://10.0.135.26:2379,https://10.0.150.76:2379\nI0305 14:00:33.243868       1 cmd.go:79] Received SIGTERM or SIGINT signal, shutting down controller.\nI0305 14:00:33.245479       1 prune_controller.go:204] Shutting down EncryptionPruneController\nI0305 14:00:33.245505       1 migration_controller.go:327] Shutting down EncryptionMigrationController\nI0305 14:00:33.245520       1 state_controller.go:171] Shutting down EncryptionStateController\nI0305 14:00:33.245544       1 key_controller.go:363] Shutting down EncryptionKeyController\nI0305 14:00:33.245559       1 condition_controller.go:202] Shutting down EncryptionConditionController\nI0305 14:00:33.245573       1 resourcesync_controller.go:228] Shutting down ResourceSyncController\nI0305 14:00:33.245593       1 base_controller.go:73] Shutting down LoggingSyncer ...\nI0305 14:00:33.245607       1 finalizer_controller.go:148] Shutting down NamespaceFinalizerController_openshift-apiserver\nI0305 14:00:33.245624       1 prune_controller.go:232] Shutting down PruneController\nI0305 14:00:33.245642       1 base_controller.go:73] Shutting down UnsupportedConfigOverridesController ...\nI0305 14:00:33.245655       1 status_controller.go:212] Shutting down StatusSyncer-openshift-apiserver\nI0305 14:00:33.245674       1 base_controller.go:73] Shutting down  ...\nI0305 14:00:33.245690       1 base_controller.go:73] Shutting down RevisionController ...\nI0305 14:00:33.245703       1 config_observer_controller.go:160] Shutting down ConfigObserver\nI0305 14:00:33.245719       1 remove_stale_conditions.go:84] Shutting down RemoveStaleConditions\nI0305 14:00:33.245862       1 apiservice_controller.go:215] Shutting down APIServiceController_openshift-apiserver\nI0305 14:00:33.245889       1 workload_controller.go:204] Shutting down OpenShiftAPIServerOperator\nF0305 14:00:33.246227       1 builder.go:243] stopped\n
Mar 05 14:00:36.169 E ns/openshift-kube-storage-version-migrator-operator pod/kube-storage-version-migrator-operator-644555b846-8hj4z node/ip-10-0-150-76.ec2.internal container=kube-storage-version-migrator-operator container exited with code 255 (Error): ourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-storage-version-migrator changed: Progressing changed from False to True ("Progressing: deployment/migrator.openshift-kube-storage-version-migrator:: observed generation is 1, desired generation is 2.")\nI0305 13:42:41.504864       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-storage-version-migrator-operator", Name:"kube-storage-version-migrator-operator", UID:"d08d3d14-3e2e-49bd-b1a0-2e517a9f28d1", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-storage-version-migrator changed: Progressing changed from True to False ("")\nI0305 13:56:56.346032       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-storage-version-migrator-operator", Name:"kube-storage-version-migrator-operator", UID:"d08d3d14-3e2e-49bd-b1a0-2e517a9f28d1", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-storage-version-migrator changed: Available changed from True to False ("Available: deployment/migrator.openshift-kube-storage-version-migrator: no replicas are available")\nI0305 13:57:10.662391       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-storage-version-migrator-operator", Name:"kube-storage-version-migrator-operator", UID:"d08d3d14-3e2e-49bd-b1a0-2e517a9f28d1", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-storage-version-migrator changed: Available changed from False to True ("")\nI0305 14:00:33.315576       1 cmd.go:79] Received SIGTERM or SIGINT signal, shutting down controller.\nF0305 14:00:33.315642       1 leaderelection.go:66] leaderelection lost\nI0305 14:00:33.322167       1 tlsconfig.go:234] Shutting down DynamicServingCertificateController\n
Mar 05 14:00:36.313 E ns/openshift-machine-api pod/machine-api-controllers-7678998c58-x4sbn node/ip-10-0-150-76.ec2.internal container=controller-manager container exited with code 1 (Error): 
Mar 05 14:00:38.613 E ns/openshift-machine-config-operator pod/machine-config-operator-6968b5c887-j4t2k node/ip-10-0-150-76.ec2.internal container=machine-config-operator container exited with code 2 (Error): I0305 13:57:19.405403       1 start.go:45] Version: machine-config-daemon-4.5.0-202003050701-2-gb9fa5093-dirty (b9fa5093eb95cac163db9039e7d2d8bb38f10db6)\nI0305 13:57:19.408931       1 leaderelection.go:242] attempting to acquire leader lease  openshift-machine-config-operator/machine-config...\nE0305 13:59:15.107130       1 event.go:319] Could not construct reference to: '&v1.ConfigMap{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"machine-config", GenerateName:"", Namespace:"openshift-machine-config-operator", SelfLink:"/api/v1/namespaces/openshift-machine-config-operator/configmaps/machine-config", UID:"5c61290d-b3e8-4673-bf61-60068a4d1889", ResourceVersion:"35738", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63719011325, loc:(*time.Location)(0x27fa020)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"machine-config-operator-6968b5c887-j4t2k_fa5de0da-0686-4938-9c6e-7b14ad593964\",\"leaseDurationSeconds\":90,\"acquireTime\":\"2020-03-05T13:59:15Z\",\"renewTime\":\"2020-03-05T13:59:15Z\",\"leaderTransitions\":2}"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Data:map[string]string(nil), BinaryData:map[string][]uint8(nil)}' due to: 'no kind is registered for the type v1.ConfigMap in scheme "github.com/openshift/machine-config-operator/cmd/common/helpers.go:30"'. Will not report event: 'Normal' 'LeaderElection' 'machine-config-operator-6968b5c887-j4t2k_fa5de0da-0686-4938-9c6e-7b14ad593964 became leader'\nI0305 13:59:15.107227       1 leaderelection.go:252] successfully acquired lease openshift-machine-config-operator/machine-config\nI0305 13:59:15.701378       1 operator.go:264] Starting MachineConfigOperator\n
Mar 05 14:00:41.733 E ns/openshift-service-ca pod/service-ca-7b984fcc6-zlx8s node/ip-10-0-150-76.ec2.internal container=service-ca-controller container exited with code 255 (Error): 
Mar 05 14:00:41.793 E ns/openshift-service-ca-operator pod/service-ca-operator-69fb9fcc8d-bsghb node/ip-10-0-150-76.ec2.internal container=operator container exited with code 255 (Error): 
Mar 05 14:00:42.228 E ns/openshift-operator-lifecycle-manager pod/packageserver-f74474b4d-k6tsn node/ip-10-0-135-26.ec2.internal container=packageserver container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 14:00:43.088 E ns/openshift-authentication-operator pod/authentication-operator-5db6ffbbdc-z2vjd node/ip-10-0-150-76.ec2.internal container=operator container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 14:00:48.922 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ip-10-0-142-254.ec2.internal container=prometheus container exited with code 1 (Error): caller=main.go:648 msg="Starting TSDB ..."\nlevel=info ts=2020-03-05T14:00:46.807Z caller=web.go:506 component=web msg="Start listening for connections" address=127.0.0.1:9090\nlevel=info ts=2020-03-05T14:00:46.814Z caller=head.go:584 component=tsdb msg="replaying WAL, this may take awhile"\nlevel=info ts=2020-03-05T14:00:46.814Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0\nlevel=info ts=2020-03-05T14:00:46.815Z caller=main.go:663 fs_type=XFS_SUPER_MAGIC\nlevel=info ts=2020-03-05T14:00:46.816Z caller=main.go:664 msg="TSDB started"\nlevel=info ts=2020-03-05T14:00:46.816Z caller=main.go:734 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml\nlevel=info ts=2020-03-05T14:00:46.816Z caller=main.go:517 msg="Stopping scrape discovery manager..."\nlevel=info ts=2020-03-05T14:00:46.816Z caller=main.go:531 msg="Stopping notify discovery manager..."\nlevel=info ts=2020-03-05T14:00:46.816Z caller=main.go:553 msg="Stopping scrape manager..."\nlevel=info ts=2020-03-05T14:00:46.816Z caller=main.go:547 msg="Scrape manager stopped"\nlevel=info ts=2020-03-05T14:00:46.816Z caller=manager.go:814 component="rule manager" msg="Stopping rule manager..."\nlevel=info ts=2020-03-05T14:00:46.816Z caller=manager.go:820 component="rule manager" msg="Rule manager stopped"\nlevel=info ts=2020-03-05T14:00:46.816Z caller=main.go:513 msg="Scrape discovery manager stopped"\nlevel=info ts=2020-03-05T14:00:46.816Z caller=main.go:527 msg="Notify discovery manager stopped"\nlevel=info ts=2020-03-05T14:00:46.817Z caller=notifier.go:598 component=notifier msg="Stopping notification manager..."\nlevel=info ts=2020-03-05T14:00:46.817Z caller=main.go:718 msg="Notifier manager stopped"\nlevel=error ts=2020-03-05
Mar 05 14:01:03.924 E ns/openshift-console pod/console-57f84759cf-4q54j node/ip-10-0-150-76.ec2.internal container=console container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 14:01:11.696 E ns/openshift-operator-lifecycle-manager pod/packageserver-f74474b4d-pjlt7 node/ip-10-0-130-18.ec2.internal container=packageserver container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 14:01:15.952 E ns/openshift-ingress pod/router-default-78878b467-c6fp7 node/ip-10-0-153-229.ec2.internal container=router container exited with code 137 (OOMKilled): aded"  "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"\nI0305 14:00:38.409229       1 router.go:551] template "level"=0 "msg"="router reloaded"  "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"\nI0305 14:00:43.417724       1 router.go:551] template "level"=0 "msg"="router reloaded"  "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"\nI0305 14:00:48.410264       1 router.go:551] template "level"=0 "msg"="router reloaded"  "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"\nI0305 14:00:53.420198       1 router.go:551] template "level"=0 "msg"="router reloaded"  "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"\nI0305 14:00:58.417544       1 router.go:551] template "level"=0 "msg"="router reloaded"  "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"\nI0305 14:01:03.410780       1 router.go:551] template "level"=0 "msg"="router reloaded"  "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"\nI0305 14:01:08.415297       1 router.go:551] template "level"=0 "msg"="router reloaded"  "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"\nI0305 14:01:13.412307       1 router.go:551] template "level"=0 "msg"="router reloaded"  "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"\nI0305 14:01:13.426145       1 template.go:551] router "level"=0 "msg"="Instructing the template router to terminate"  \nI0305 14:01:14.434430       1 router.go:551] template "level"=0 "msg"="router reloaded"  "output"=" - Shutting down\n"\nI0305 14:01:14.434479       1 template.go:555] router "level"=0 "msg"="Shutdown complete, exiting"  \n
Mar 05 14:01:18.336 E ns/openshift-machine-api pod/machine-api-controllers-7678998c58-vsmjd node/ip-10-0-135-26.ec2.internal container=machine-controller container exited with code 255 (Error): 
Mar 05 14:01:25.744 E ns/openshift-authentication pod/oauth-openshift-64d4dc994-gf7bm node/ip-10-0-130-18.ec2.internal container=oauth-openshift container exited with code 255 (Error): Copying system trust bundle\nW0305 14:01:25.038457       1 configmap_cafile_content.go:102] unable to load initial CA bundle for: "client-ca::kube-system::extension-apiserver-authentication::client-ca-file" due to: configmap "extension-apiserver-authentication" not found\nW0305 14:01:25.038689       1 configmap_cafile_content.go:102] unable to load initial CA bundle for: "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" due to: configmap "extension-apiserver-authentication" not found\nF0305 14:01:25.040842       1 cmd.go:49] unable to load configmap based request-header-client-ca-file: Get https://172.30.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 172.30.0.1:443: connect: connection refused\n
Mar 05 14:01:38.217 E ns/openshift-marketplace pod/certified-operators-56b9fd45c4-ldnpv node/ip-10-0-142-254.ec2.internal container=certified-operators container exited with code 2 (Error): 
Mar 05 14:01:39.221 E ns/openshift-marketplace pod/community-operators-544c68887d-lnchp node/ip-10-0-142-254.ec2.internal container=community-operators container exited with code 2 (Error): 
Mar 05 14:03:27.706 E ns/openshift-kube-controller-manager pod/kube-controller-manager-ip-10-0-150-76.ec2.internal node/ip-10-0-150-76.ec2.internal container=cluster-policy-controller container exited with code 1 (Error): I0305 13:43:29.708987       1 policy_controller.go:41] Starting controllers on 0.0.0.0:10357 (v0.0.0-unknown)\nI0305 13:43:29.721176       1 standalone_apiserver.go:103] Started health checks at 0.0.0.0:10357\nI0305 13:43:29.722194       1 leaderelection.go:242] attempting to acquire leader lease  openshift-kube-controller-manager/cluster-policy-controller...\n
Mar 05 14:03:27.706 E ns/openshift-kube-controller-manager pod/kube-controller-manager-ip-10-0-150-76.ec2.internal node/ip-10-0-150-76.ec2.internal container=kube-controller-manager-cert-syncer container exited with code 2 (Error):     1 certsync_controller.go:162] Syncing secrets: [{csr-signer false}]\nI0305 14:00:29.999841       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 14:00:30.000171       1 certsync_controller.go:162] Syncing secrets: [{csr-signer false}]\nI0305 14:00:38.870348       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 14:00:38.870779       1 certsync_controller.go:162] Syncing secrets: [{csr-signer false}]\nI0305 14:00:40.020278       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 14:00:40.020726       1 certsync_controller.go:162] Syncing secrets: [{csr-signer false}]\nI0305 14:00:48.886625       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 14:00:48.886976       1 certsync_controller.go:162] Syncing secrets: [{csr-signer false}]\nI0305 14:00:50.030699       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 14:00:50.031146       1 certsync_controller.go:162] Syncing secrets: [{csr-signer false}]\nI0305 14:00:58.907957       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 14:00:58.911712       1 certsync_controller.go:162] Syncing secrets: [{csr-signer false}]\nI0305 14:01:00.043960       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 14:01:00.044296       1 certsync_controller.go:162] Syncing secrets: [{csr-signer false}]\nI0305 14:01:08.916306       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 14:01:08.916623       1 certsync_controller.go:162] Syncing secrets: [{csr-signer false}]\n
Mar 05 14:03:27.706 E ns/openshift-kube-controller-manager pod/kube-controller-manager-ip-10-0-150-76.ec2.internal node/ip-10-0-150-76.ec2.internal container=kube-controller-manager container exited with code 2 (Error): strap-kubeconfig-signer" [] issuer="<self>" (2020-03-05 13:06:30 +0000 UTC to 2030-03-03 13:06:30 +0000 UTC (now=2020-03-05 13:43:28.948363849 +0000 UTC))\nI0305 13:43:28.948484       1 tlsconfig.go:179] loaded client CA [6/"client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt,request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt"]: "aggregator-signer" [] issuer="<self>" (2020-03-05 13:06:33 +0000 UTC to 2020-03-06 13:06:33 +0000 UTC (now=2020-03-05 13:43:28.948466821 +0000 UTC))\nI0305 13:43:28.953903       1 tlsconfig.go:201] loaded serving cert ["serving-cert::/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.crt::/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.key"]: "kube-controller-manager.openshift-kube-controller-manager.svc" [serving] validServingFor=[kube-controller-manager.openshift-kube-controller-manager.svc,kube-controller-manager.openshift-kube-controller-manager.svc.cluster.local] issuer="openshift-service-serving-signer@1583414535" (2020-03-05 13:22:30 +0000 UTC to 2022-03-05 13:22:31 +0000 UTC (now=2020-03-05 13:43:28.948846513 +0000 UTC))\nI0305 13:43:28.956062       1 named_certificates.go:53] loaded SNI cert [0/"self-signed loopback"]: "apiserver-loopback-client@1583415808" [serving] validServingFor=[apiserver-loopback-client] issuer="apiserver-loopback-client-ca@1583415808" (2020-03-05 12:43:26 +0000 UTC to 2021-03-05 12:43:26 +0000 UTC (now=2020-03-05 13:43:28.956036585 +0000 UTC))\nI0305 13:43:28.956128       1 secure_serving.go:178] Serving securely on [::]:10257\nI0305 13:43:28.956185       1 leaderelection.go:242] attempting to acquire leader lease  kube-system/kube-controller-manager...\nI0305 13:43:28.959463       1 tlsconfig.go:241] Starting DynamicServingCertificateController\nI0305 13:43:28.969546       1 dynamic_serving_content.go:130] Starting serving-cert::/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.crt::/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.key\n
Mar 05 14:03:27.706 E ns/openshift-kube-controller-manager pod/kube-controller-manager-ip-10-0-150-76.ec2.internal node/ip-10-0-150-76.ec2.internal container=kube-controller-manager-recovery-controller container exited with code 255 (Error): ] loaded client CA [4/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kubelet-bootstrap-kubeconfig-signer" [] issuer="<self>" (2020-03-05 13:06:30 +0000 UTC to 2030-03-03 13:06:30 +0000 UTC (now=2020-03-05 13:43:36.867524973 +0000 UTC))\nI0305 13:43:36.867634       1 tlsconfig.go:157] loaded client CA [5/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kube-csr-signer_@1583414535" [] issuer="kubelet-signer" (2020-03-05 13:22:14 +0000 UTC to 2020-03-06 13:06:35 +0000 UTC (now=2020-03-05 13:43:36.867613329 +0000 UTC))\nI0305 13:43:36.867701       1 tlsconfig.go:157] loaded client CA [6/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "aggregator-signer" [] issuer="<self>" (2020-03-05 13:06:33 +0000 UTC to 2020-03-06 13:06:33 +0000 UTC (now=2020-03-05 13:43:36.867685412 +0000 UTC))\nI0305 13:43:36.868074       1 tlsconfig.go:179] loaded serving cert ["serving-cert::/tmp/serving-cert-645643258/tls.crt::/tmp/serving-cert-645643258/tls.key"]: "localhost" [serving] validServingFor=[localhost] issuer="cert-recovery-controller-signer@1583415811" (2020-03-05 13:43:31 +0000 UTC to 2020-04-04 13:43:32 +0000 UTC (now=2020-03-05 13:43:36.868056065 +0000 UTC))\nI0305 13:43:36.868495       1 named_certificates.go:52] loaded SNI cert [0/"self-signed loopback"]: "apiserver-loopback-client@1583415816" [serving] validServingFor=[apiserver-loopback-client] issuer="apiserver-loopback-client-ca@1583415816" (2020-03-05 12:43:35 +0000 UTC to 2021-03-05 12:43:35 +0000 UTC (now=2020-03-05 13:43:36.868475983 +0000 UTC))\nI0305 14:01:09.693839       1 cmd.go:84] Received SIGTERM or SIGINT signal, shutting down controller.\nF0305 14:01:09.693882       1 leaderelection.go:67] leaderelection lost\n
Mar 05 14:03:27.752 E ns/openshift-etcd pod/etcd-ip-10-0-150-76.ec2.internal node/ip-10-0-150-76.ec2.internal container=etcd-metrics container exited with code 2 (Error): 2020-03-05 13:43:25.710544 I | etcdmain: ServerTLS: cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-150-76.ec2.internal.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-150-76.ec2.internal.key, ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-metrics-proxy-serving-ca/ca-bundle.crt, trusted-ca = , client-cert-auth = false, crl-file = \n2020-03-05 13:43:25.711203 I | etcdmain: listening for grpc-proxy client requests on 127.0.0.1:9977\n2020-03-05 13:43:25.711616 I | etcdmain: ClientTLS: cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-peer/etcd-peer-ip-10-0-150-76.ec2.internal.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-peer/etcd-peer-ip-10-0-150-76.ec2.internal.key, ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-peer-client-ca/ca-bundle.crt, trusted-ca = , client-cert-auth = false, crl-file = \n2020-03-05 13:43:25.714651 I | etcdmain: grpc-proxy: listening for metrics on https://0.0.0.0:9979\nWARNING: 2020/03/05 13:43:25 grpc: addrConn.createTransport failed to connect to {https://etcd-1.ci-op-s76jkp2x-f83f1.origin-ci-int-aws.dev.rhcloud.com:9978 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.150.76:9978: connect: connection refused". Reconnecting...\n
Mar 05 14:03:27.806 E ns/openshift-controller-manager pod/controller-manager-v8mhl node/ip-10-0-150-76.ec2.internal container=controller-manager container exited with code 1 (Error): cfg_secrets.go:224] caches synced\nI0305 13:45:04.271911       1 docker_registry_service.go:296] Updating registry URLs from map[172.30.158.40:5000:{} image-registry.openshift-image-registry.svc.cluster.local:5000:{} image-registry.openshift-image-registry.svc:5000:{}] to map[172.30.158.40:5000:{} image-registry.openshift-image-registry.svc.cluster.local:5000:{} image-registry.openshift-image-registry.svc:5000:{}]\nW0305 13:56:56.867959       1 reflector.go:340] github.com/openshift/client-go/image/informers/externalversions/factory.go:101: watch of *v1.ImageStream ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 481; INTERNAL_ERROR") has prevented the request from succeeding\nW0305 14:00:41.733795       1 reflector.go:340] github.com/openshift/client-go/template/informers/externalversions/factory.go:101: watch of *v1.TemplateInstance ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 719; INTERNAL_ERROR") has prevented the request from succeeding\nW0305 14:00:41.733989       1 reflector.go:340] github.com/openshift/client-go/build/informers/externalversions/factory.go:101: watch of *v1.Build ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 635; INTERNAL_ERROR") has prevented the request from succeeding\nW0305 14:00:41.734418       1 reflector.go:340] github.com/openshift/client-go/image/informers/externalversions/factory.go:101: watch of *v1.Image ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 791; INTERNAL_ERROR") has prevented the request from succeeding\nW0305 14:00:41.733687       1 reflector.go:340] github.com/openshift/client-go/image/informers/externalversions/factory.go:101: watch of *v1.ImageStream ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 727; INTERNAL_ERROR") has prevented the request from succeeding\n
Mar 05 14:03:27.824 E ns/openshift-cluster-node-tuning-operator pod/tuned-ksdkj node/ip-10-0-150-76.ec2.internal container=tuned container exited with code 143 (Error): ested: openshift-control-plane\nI0305 13:44:29.997465    1113 tuned.go:170] disabling system tuned...\nI0305 13:44:29.997179    1113 tuned.go:521] tuned "rendered" added\nI0305 13:44:29.998048    1113 tuned.go:219] extracting tuned profiles\nI0305 13:44:30.001568    1113 tuned.go:176] failed to disable system tuned: Failed to execute operation: Unit file tuned.service does not exist.\nI0305 13:44:30.984588    1113 tuned.go:393] getting recommended profile...\nI0305 13:44:31.111523    1113 tuned.go:421] active profile () != recommended profile (openshift-control-plane)\nI0305 13:44:31.111595    1113 tuned.go:286] starting tuned...\n2020-03-05 13:44:31,249 INFO     tuned.daemon.application: dynamic tuning is globally disabled\n2020-03-05 13:44:31,256 INFO     tuned.daemon.daemon: using sleep interval of 1 second(s)\n2020-03-05 13:44:31,256 INFO     tuned.daemon.daemon: Running in automatic mode, checking what profile is recommended for your configuration.\n2020-03-05 13:44:31,257 INFO     tuned.daemon.daemon: Using 'openshift-control-plane' profile\n2020-03-05 13:44:31,258 INFO     tuned.profiles.loader: loading profile: openshift-control-plane\n2020-03-05 13:44:31,301 INFO     tuned.daemon.controller: starting controller\n2020-03-05 13:44:31,301 INFO     tuned.daemon.daemon: starting tuning\n2020-03-05 13:44:31,312 INFO     tuned.plugins.base: instance cpu: assigning devices cpu2, cpu3, cpu0, cpu1\n2020-03-05 13:44:31,313 INFO     tuned.plugins.plugin_cpu: We are running on an x86 GenuineIntel platform\n2020-03-05 13:44:31,317 WARNING  tuned.plugins.plugin_cpu: your CPU doesn't support MSR_IA32_ENERGY_PERF_BIAS, ignoring CPU energy performance bias\n2020-03-05 13:44:31,319 INFO     tuned.plugins.base: instance disk: assigning devices dm-0, xvda\n2020-03-05 13:44:31,320 INFO     tuned.plugins.base: instance net: assigning devices ens3\n2020-03-05 13:44:31,444 INFO     tuned.plugins.plugin_sysctl: reapplying system sysctl\n2020-03-05 13:44:31,451 INFO     tuned.daemon.daemon: static tuning from profile 'openshift-control-plane' applied\n
Mar 05 14:03:27.849 E ns/openshift-sdn pod/sdn-controller-dmqdz node/ip-10-0-150-76.ec2.internal container=sdn-controller container exited with code 2 (Error): I0305 13:48:32.277966       1 leaderelection.go:242] attempting to acquire leader lease  openshift-sdn/openshift-network-controller...\n
Mar 05 14:03:27.869 E ns/openshift-monitoring pod/node-exporter-vtb2x node/ip-10-0-150-76.ec2.internal container=node-exporter container exited with code 143 (Error): or gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:00:07Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:00:20Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:00:22Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:00:35Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:00:50Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:01:05Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:01:07Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\n
Mar 05 14:03:27.913 E ns/openshift-sdn pod/ovs-94pxr node/ip-10-0-150-76.ec2.internal container=openvswitch container exited with code 1 (Error): T14:00:46.625Z|00229|bridge|INFO|bridge br0: added interface veth02300f85 on port 89\n2020-03-05T14:00:46.675Z|00230|connmgr|INFO|br0<->unix#881: 5 flow_mods in the last 0 s (5 adds)\n2020-03-05T14:00:46.731Z|00231|connmgr|INFO|br0<->unix#885: 2 flow_mods in the last 0 s (2 deletes)\n2020-03-05T14:00:46.749Z|00232|connmgr|INFO|br0<->unix#887: 2 flow_mods in the last 0 s (1 adds, 1 deletes)\n2020-03-05T14:00:46.681Z|00042|reconnect|WARN|unix#730: connection dropped (Connection reset by peer)\n2020-03-05T14:00:46.688Z|00043|reconnect|WARN|unix#731: connection dropped (Connection reset by peer)\n2020-03-05T14:00:46.694Z|00044|reconnect|WARN|unix#732: connection dropped (Connection reset by peer)\n2020-03-05T14:00:50.026Z|00233|connmgr|INFO|br0<->unix#890: 2 flow_mods in the last 0 s (2 deletes)\n2020-03-05T14:00:50.074Z|00234|connmgr|INFO|br0<->unix#893: 4 flow_mods in the last 0 s (4 deletes)\n2020-03-05T14:00:50.118Z|00235|bridge|INFO|bridge br0: deleted interface veth02300f85 on port 89\n2020-03-05T14:00:58.520Z|00236|connmgr|INFO|br0<->unix#902: 2 flow_mods in the last 0 s (2 deletes)\n2020-03-05T14:00:58.560Z|00237|connmgr|INFO|br0<->unix#905: 4 flow_mods in the last 0 s (4 deletes)\n2020-03-05T14:00:58.582Z|00238|bridge|INFO|bridge br0: deleted interface veth36d9b400 on port 87\n2020-03-05T14:00:58.567Z|00045|jsonrpc|WARN|Dropped 10 log messages in last 18 seconds (most recently, 12 seconds ago) due to excessive rate\n2020-03-05T14:00:58.567Z|00046|jsonrpc|WARN|unix#746: receive error: Connection reset by peer\n2020-03-05T14:00:58.567Z|00047|reconnect|WARN|unix#746: connection dropped (Connection reset by peer)\n2020-03-05T14:01:02.877Z|00239|connmgr|INFO|br0<->unix#911: 2 flow_mods in the last 0 s (2 deletes)\n2020-03-05T14:01:02.913Z|00240|connmgr|INFO|br0<->unix#914: 4 flow_mods in the last 0 s (4 deletes)\n2020-03-05T14:01:02.939Z|00241|bridge|INFO|bridge br0: deleted interface veth1d7305eb on port 67\ninfo: Saving flows ...\novs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)\n
Mar 05 14:03:27.945 E ns/openshift-multus pod/multus-2bwfs node/ip-10-0-150-76.ec2.internal container=kube-multus container exited with code 143 (Error): 
Mar 05 14:03:27.987 E ns/openshift-machine-config-operator pod/machine-config-daemon-h7dc2 node/ip-10-0-150-76.ec2.internal container=oauth-proxy container exited with code 143 (Error): 
Mar 05 14:03:28.002 E ns/openshift-machine-config-operator pod/machine-config-server-6n4zv node/ip-10-0-150-76.ec2.internal container=machine-config-server container exited with code 2 (Error): I0305 13:57:23.111852       1 start.go:38] Version: machine-config-daemon-4.5.0-202003050701-2-gb9fa5093-dirty (b9fa5093eb95cac163db9039e7d2d8bb38f10db6)\nI0305 13:57:23.113769       1 api.go:51] Launching server on :22624\nI0305 13:57:23.113920       1 api.go:51] Launching server on :22623\n
Mar 05 14:03:28.031 E ns/openshift-multus pod/multus-admission-controller-k5bnw node/ip-10-0-150-76.ec2.internal container=multus-admission-controller container exited with code 255 (Error): 
Mar 05 14:03:28.118 E ns/openshift-kube-apiserver pod/kube-apiserver-ip-10-0-150-76.ec2.internal node/ip-10-0-150-76.ec2.internal container=kube-apiserver container exited with code 1 (Error): teTransport failed to connect to {https://127.0.0.1:2379 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...\nI0305 14:01:09.458314       1 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"\nW0305 14:01:09.458635       1 clientconn.go:1120] grpc: addrConn.createTransport failed to connect to {https://10.0.150.76:2379 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.150.76:2379: connect: connection refused". Reconnecting...\nW0305 14:01:09.458729       1 clientconn.go:1120] grpc: addrConn.createTransport failed to connect to {https://127.0.0.1:2379 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...\nE0305 14:01:09.546531       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted\nE0305 14:01:09.546773       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted\nE0305 14:01:09.547794       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted\nE0305 14:01:09.547964       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted\nE0305 14:01:09.573294       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted\nE0305 14:01:09.573449       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted\nI0305 14:01:09.755476       1 genericapiserver.go:643] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-ip-10-0-150-76.ec2.internal", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TerminationStart' Received signal to terminate, becoming unready, but keeping serving\nI0305 14:01:09.755802       1 controller.go:180] Shutting down kubernetes service endpoint reconciler\n
Mar 05 14:03:28.118 E ns/openshift-kube-apiserver pod/kube-apiserver-ip-10-0-150-76.ec2.internal node/ip-10-0-150-76.ec2.internal container=kube-apiserver-insecure-readyz container exited with code 2 (Error): I0305 13:43:30.497906       1 readyz.go:103] Listening on 0.0.0.0:6080\n
Mar 05 14:03:28.118 E ns/openshift-kube-apiserver pod/kube-apiserver-ip-10-0-150-76.ec2.internal node/ip-10-0-150-76.ec2.internal container=kube-apiserver-cert-syncer container exited with code 2 (Error): ce-network-serving-certkey false} {external-loadbalancer-serving-certkey false} {internal-loadbalancer-serving-certkey false} {bound-service-account-signing-key false} {user-serving-cert true} {user-serving-cert-000 true} {user-serving-cert-001 true} {user-serving-cert-002 true} {user-serving-cert-003 true} {user-serving-cert-004 true} {user-serving-cert-005 true} {user-serving-cert-006 true} {user-serving-cert-007 true} {user-serving-cert-008 true} {user-serving-cert-009 true}]\nI0305 14:00:54.645792       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 14:00:54.646146       1 certsync_controller.go:162] Syncing secrets: [{aggregator-client false} {localhost-serving-cert-certkey false} {service-network-serving-certkey false} {external-loadbalancer-serving-certkey false} {internal-loadbalancer-serving-certkey false} {bound-service-account-signing-key false} {user-serving-cert true} {user-serving-cert-000 true} {user-serving-cert-001 true} {user-serving-cert-002 true} {user-serving-cert-003 true} {user-serving-cert-004 true} {user-serving-cert-005 true} {user-serving-cert-006 true} {user-serving-cert-007 true} {user-serving-cert-008 true} {user-serving-cert-009 true}]\nI0305 14:01:04.659598       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 14:01:04.659929       1 certsync_controller.go:162] Syncing secrets: [{aggregator-client false} {localhost-serving-cert-certkey false} {service-network-serving-certkey false} {external-loadbalancer-serving-certkey false} {internal-loadbalancer-serving-certkey false} {bound-service-account-signing-key false} {user-serving-cert true} {user-serving-cert-000 true} {user-serving-cert-001 true} {user-serving-cert-002 true} {user-serving-cert-003 true} {user-serving-cert-004 true} {user-serving-cert-005 true} {user-serving-cert-006 true} {user-serving-cert-007 true} {user-serving-cert-008 true} {user-serving-cert-009 true}]\n
Mar 05 14:03:28.118 E ns/openshift-kube-apiserver pod/kube-apiserver-ip-10-0-150-76.ec2.internal node/ip-10-0-150-76.ec2.internal container=kube-apiserver-cert-regeneration-controller container exited with code 255 (Error): W0305 13:43:29.610884       1 cmd.go:200] Using insecure, self-signed certificates\nI0305 13:43:29.617558       1 crypto.go:580] Generating new CA for cert-regeneration-controller-signer@1583415809 cert, and key in /tmp/serving-cert-684851679/serving-signer.crt, /tmp/serving-cert-684851679/serving-signer.key\nI0305 13:43:31.304845       1 observer_polling.go:155] Starting file observer\nI0305 13:43:35.842865       1 leaderelection.go:242] attempting to acquire leader lease  openshift-kube-apiserver/cert-regeneration-controller-lock...\nI0305 14:01:09.523580       1 cmd.go:84] Received SIGTERM or SIGINT signal, shutting down controller.\nF0305 14:01:09.523619       1 leaderelection.go:67] leaderelection lost\n
Mar 05 14:03:32.617 E ns/openshift-kube-scheduler pod/openshift-kube-scheduler-ip-10-0-150-76.ec2.internal node/ip-10-0-150-76.ec2.internal container=scheduler container exited with code 2 (Error): entication::requestheader-client-ca-file"]: "aggregator-signer" [] issuer="<self>" (2020-03-05 13:06:33 +0000 UTC to 2020-03-06 13:06:33 +0000 UTC (now=2020-03-05 13:43:36.991929201 +0000 UTC))\nI0305 13:43:36.992621       1 tlsconfig.go:201] loaded serving cert ["serving-cert::/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.crt::/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.key"]: "scheduler.openshift-kube-scheduler.svc" [serving] validServingFor=[scheduler.openshift-kube-scheduler.svc,scheduler.openshift-kube-scheduler.svc.cluster.local] issuer="openshift-service-serving-signer@1583414535" (2020-03-05 13:22:27 +0000 UTC to 2022-03-05 13:22:28 +0000 UTC (now=2020-03-05 13:43:36.992597673 +0000 UTC))\nI0305 13:43:36.993063       1 named_certificates.go:53] loaded SNI cert [0/"self-signed loopback"]: "apiserver-loopback-client@1583415816" [serving] validServingFor=[apiserver-loopback-client] issuer="apiserver-loopback-client-ca@1583415814" (2020-03-05 12:43:33 +0000 UTC to 2021-03-05 12:43:33 +0000 UTC (now=2020-03-05 13:43:36.993033898 +0000 UTC))\nI0305 13:43:36.994832       1 node_tree.go:86] Added node "ip-10-0-142-254.ec2.internal" in group "us-east-1:\x00:us-east-1b" to NodeTree\nI0305 13:43:36.995020       1 node_tree.go:86] Added node "ip-10-0-150-76.ec2.internal" in group "us-east-1:\x00:us-east-1c" to NodeTree\nI0305 13:43:36.995952       1 node_tree.go:86] Added node "ip-10-0-153-229.ec2.internal" in group "us-east-1:\x00:us-east-1c" to NodeTree\nI0305 13:43:36.996066       1 node_tree.go:86] Added node "ip-10-0-130-18.ec2.internal" in group "us-east-1:\x00:us-east-1b" to NodeTree\nI0305 13:43:36.996194       1 node_tree.go:86] Added node "ip-10-0-130-45.ec2.internal" in group "us-east-1:\x00:us-east-1b" to NodeTree\nI0305 13:43:36.996292       1 node_tree.go:86] Added node "ip-10-0-135-26.ec2.internal" in group "us-east-1:\x00:us-east-1b" to NodeTree\nI0305 13:43:37.066727       1 leaderelection.go:242] attempting to acquire leader lease  openshift-kube-scheduler/kube-scheduler...\n
Mar 05 14:03:34.057 E ns/openshift-multus pod/multus-2bwfs node/ip-10-0-150-76.ec2.internal invariant violation: pod may not transition Running->Pending
Mar 05 14:03:37.069 E ns/openshift-multus pod/multus-2bwfs node/ip-10-0-150-76.ec2.internal invariant violation: pod may not transition Running->Pending
Mar 05 14:03:42.109 E ns/openshift-machine-config-operator pod/machine-config-daemon-h7dc2 node/ip-10-0-150-76.ec2.internal container=oauth-proxy container exited with code 1 (Error): 
Mar 05 14:03:46.547 E clusteroperator/etcd changed Degraded to True: NodeController_MasterNodesReady: NodeControllerDegraded: The master nodes not ready: node "ip-10-0-150-76.ec2.internal" not ready since 2020-03-05 14:03:27 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: Missing CNI default network)
Mar 05 14:03:46.556 E clusteroperator/kube-controller-manager changed Degraded to True: NodeController_MasterNodesReady: NodeControllerDegraded: The master nodes not ready: node "ip-10-0-150-76.ec2.internal" not ready since 2020-03-05 14:03:27 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: Missing CNI default network)
Mar 05 14:03:46.559 E clusteroperator/kube-apiserver changed Degraded to True: NodeController_MasterNodesReady: NodeControllerDegraded: The master nodes not ready: node "ip-10-0-150-76.ec2.internal" not ready since 2020-03-05 14:03:27 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: Missing CNI default network)
Mar 05 14:03:46.561 E clusteroperator/kube-scheduler changed Degraded to True: NodeController_MasterNodesReady: NodeControllerDegraded: The master nodes not ready: node "ip-10-0-150-76.ec2.internal" not ready since 2020-03-05 14:03:27 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: Missing CNI default network)
Mar 05 14:03:58.017 E ns/openshift-service-catalog-apiserver-operator pod/openshift-service-catalog-apiserver-operator-849ccfd479-dcjgf node/ip-10-0-130-18.ec2.internal container=operator container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Mar 05 14:04:22.074 E ns/openshift-console pod/console-57f84759cf-tktfm node/ip-10-0-130-18.ec2.internal container=console container exited with code 2 (Error): 2020-03-05T13:46:08Z cmd/main: cookies are secure!\n2020-03-05T13:46:08Z cmd/main: Binding to [::]:8443...\n2020-03-05T13:46:08Z cmd/main: using TLS\n2020-03-05T13:57:32Z auth: failed to get latest auth source data: Get https://kubernetes.default.svc/.well-known/oauth-authorization-server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n2020-03-05T13:57:37Z auth: failed to get latest auth source data: Get https://kubernetes.default.svc/.well-known/oauth-authorization-server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n2020-03-05T13:57:42Z auth: failed to get latest auth source data: Get https://kubernetes.default.svc/.well-known/oauth-authorization-server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n2020-03-05T14:01:31Z auth: failed to get latest auth source data: request to OAuth issuer endpoint https://oauth-openshift.apps.ci-op-s76jkp2x-f83f1.origin-ci-int-aws.dev.rhcloud.com/oauth/token failed: Head https://oauth-openshift.apps.ci-op-s76jkp2x-f83f1.origin-ci-int-aws.dev.rhcloud.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n
Mar 05 14:04:26.285 E kube-apiserver failed contacting the API: Get https://api.ci-op-s76jkp2x-f83f1.origin-ci-int-aws.dev.rhcloud.com:6443/apis/config.openshift.io/v1/clusterversions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dversion&resourceVersion=38448&timeout=5m24s&timeoutSeconds=324&watch=true: dial tcp 3.233.2.8:6443: connect: connection refused
Mar 05 14:04:29.261 E ns/openshift-multus pod/multus-admission-controller-6zx9g node/ip-10-0-135-26.ec2.internal container=multus-admission-controller container exited with code 2 (Error): 
Mar 05 14:06:12.111 E ns/openshift-cluster-node-tuning-operator pod/tuned-bkns4 node/ip-10-0-153-229.ec2.internal container=tuned container exited with code 143 (Error): ecommended profile (openshift-node)\nI0305 13:44:04.360130    4058 tuned.go:286] starting tuned...\n2020-03-05 13:44:04,471 INFO     tuned.daemon.application: dynamic tuning is globally disabled\n2020-03-05 13:44:04,478 INFO     tuned.daemon.daemon: using sleep interval of 1 second(s)\n2020-03-05 13:44:04,478 INFO     tuned.daemon.daemon: Running in automatic mode, checking what profile is recommended for your configuration.\n2020-03-05 13:44:04,479 INFO     tuned.daemon.daemon: Using 'openshift-node' profile\n2020-03-05 13:44:04,480 INFO     tuned.profiles.loader: loading profile: openshift-node\n2020-03-05 13:44:04,513 INFO     tuned.daemon.controller: starting controller\n2020-03-05 13:44:04,513 INFO     tuned.daemon.daemon: starting tuning\n2020-03-05 13:44:04,524 INFO     tuned.plugins.base: instance cpu: assigning devices cpu2, cpu3, cpu0, cpu1\n2020-03-05 13:44:04,525 INFO     tuned.plugins.plugin_cpu: We are running on an x86 GenuineIntel platform\n2020-03-05 13:44:04,529 WARNING  tuned.plugins.plugin_cpu: your CPU doesn't support MSR_IA32_ENERGY_PERF_BIAS, ignoring CPU energy performance bias\n2020-03-05 13:44:04,531 INFO     tuned.plugins.base: instance disk: assigning devices dm-0, xvda\n2020-03-05 13:44:04,533 INFO     tuned.plugins.base: instance net: assigning devices ens3\n2020-03-05 13:44:04,672 INFO     tuned.plugins.plugin_sysctl: reapplying system sysctl\n2020-03-05 13:44:04,681 INFO     tuned.daemon.daemon: static tuning from profile 'openshift-node' applied\nI0305 14:01:09.828354    4058 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF\nI0305 14:01:09.828368    4058 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF\nW0305 14:01:09.833636    4058 reflector.go:340] github.com/openshift/cluster-node-tuning-operator/pkg/tuned/tuned.go:598: watch of *v1.Tuned ended with: very short watch: github.com/openshift/cluster-node-tuning-operator/pkg/tuned/tuned.go:598: Unexpected watch close - watch lasted less than a second and no items received\n
Mar 05 14:06:12.142 E ns/openshift-monitoring pod/node-exporter-x4bqf node/ip-10-0-153-229.ec2.internal container=node-exporter container exited with code 143 (Error): or gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:03:28Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:03:40Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:03:43Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:03:55Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:03:58Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:04:10Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:04:13Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\n
Mar 05 14:06:12.184 E ns/openshift-sdn pod/ovs-6xckj node/ip-10-0-153-229.ec2.internal container=openvswitch container exited with code 143 (Error): ridge br0: deleted interface veth9a4b2e57 on port 23\n2020-03-05T14:00:30.720Z|00145|connmgr|INFO|br0<->unix#723: 2 flow_mods in the last 0 s (2 deletes)\n2020-03-05T14:00:30.777Z|00146|connmgr|INFO|br0<->unix#726: 4 flow_mods in the last 0 s (4 deletes)\n2020-03-05T14:00:30.814Z|00147|bridge|INFO|bridge br0: deleted interface veth23c824eb on port 39\n2020-03-05T14:00:58.696Z|00148|connmgr|INFO|br0<->unix#750: 2 flow_mods in the last 0 s (2 deletes)\n2020-03-05T14:00:58.724Z|00149|connmgr|INFO|br0<->unix#753: 4 flow_mods in the last 0 s (4 deletes)\n2020-03-05T14:00:58.747Z|00150|bridge|INFO|bridge br0: deleted interface veth9cfea02f on port 36\n2020-03-05T14:01:14.038Z|00151|connmgr|INFO|br0<->unix#768: 2 flow_mods in the last 0 s (2 deletes)\n2020-03-05T14:01:14.075Z|00152|connmgr|INFO|br0<->unix#771: 4 flow_mods in the last 0 s (4 deletes)\n2020-03-05T14:01:14.102Z|00153|bridge|INFO|bridge br0: deleted interface veth42db7c63 on port 21\n2020-03-05T14:01:14.088Z|00009|jsonrpc|WARN|unix#661: receive error: Connection reset by peer\n2020-03-05T14:01:14.089Z|00010|reconnect|WARN|unix#661: connection dropped (Connection reset by peer)\n2020-03-05T14:01:15.555Z|00154|connmgr|INFO|br0<->unix#775: 2 flow_mods in the last 0 s (2 deletes)\n2020-03-05T14:01:15.583Z|00155|connmgr|INFO|br0<->unix#778: 4 flow_mods in the last 0 s (4 deletes)\n2020-03-05T14:01:15.605Z|00156|bridge|INFO|bridge br0: deleted interface veth348981a9 on port 35\n2020-03-05T14:01:15.594Z|00011|jsonrpc|WARN|unix#667: receive error: Connection reset by peer\n2020-03-05T14:01:15.594Z|00012|reconnect|WARN|unix#667: connection dropped (Connection reset by peer)\n2020-03-05T14:03:02.291Z|00013|jsonrpc|WARN|unix#742: receive error: Connection reset by peer\n2020-03-05T14:03:02.292Z|00014|reconnect|WARN|unix#742: connection dropped (Connection reset by peer)\n2020-03-05T14:04:02.350Z|00015|jsonrpc|WARN|unix#782: receive error: Connection reset by peer\n2020-03-05T14:04:02.350Z|00016|reconnect|WARN|unix#782: connection dropped (Connection reset by peer)\ninfo: Saving flows ...\n
Mar 05 14:06:12.225 E ns/openshift-multus pod/multus-69rqp node/ip-10-0-153-229.ec2.internal container=kube-multus container exited with code 143 (Error): 
Mar 05 14:06:12.267 E ns/openshift-machine-config-operator pod/machine-config-daemon-gcgkk node/ip-10-0-153-229.ec2.internal container=oauth-proxy container exited with code 143 (Error): 
Mar 05 14:06:15.021 E ns/openshift-sdn pod/sdn-6wntg node/ip-10-0-153-229.ec2.internal invariant violation: pod may not transition Running->Pending
Mar 05 14:06:16.087 E ns/openshift-monitoring pod/node-exporter-x4bqf node/ip-10-0-153-229.ec2.internal invariant violation: pod may not transition Running->Pending
Mar 05 14:06:23.114 E ns/openshift-machine-config-operator pod/machine-config-daemon-gcgkk node/ip-10-0-153-229.ec2.internal container=oauth-proxy container exited with code 1 (Error): 
Mar 05 14:06:34.980 E clusterversion/version changed Failing to True: ClusterOperatorDegraded: Cluster operator openshift-apiserver is reporting a failure: APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable
Mar 05 14:06:43.886 E ns/openshift-sdn pod/sdn-controller-lwww8 node/ip-10-0-130-18.ec2.internal container=sdn-controller container exited with code 2 (Error): -18\",\"leaseDurationSeconds\":60,\"acquireTime\":\"2020-03-05T13:18:09Z\",\"renewTime\":\"2020-03-05T13:48:28Z\",\"leaderTransitions\":0}"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Data:map[string]string(nil), BinaryData:map[string][]uint8(nil)}' due to: 'no kind is registered for the type v1.ConfigMap in scheme "k8s.io/kubernetes/pkg/api/legacyscheme/scheme.go:30"'. Will not report event: 'Normal' 'LeaderElection' 'ip-10-0-130-18 became leader'\nI0305 13:48:28.364536       1 leaderelection.go:252] successfully acquired lease openshift-sdn/openshift-network-controller\nI0305 13:48:28.375151       1 master.go:51] Initializing SDN master\nI0305 13:48:28.438683       1 network_controller.go:61] Started OpenShift Network Controller\nE0305 14:01:09.855504       1 reflector.go:307] github.com/openshift/client-go/network/informers/externalversions/factory.go:101: Failed to watch *v1.HostSubnet: Get https://api-int.ci-op-s76jkp2x-f83f1.origin-ci-int-aws.dev.rhcloud.com:6443/apis/network.openshift.io/v1/hostsubnets?allowWatchBookmarks=true&resourceVersion=34638&timeout=9m21s&timeoutSeconds=561&watch=true: dial tcp 10.0.132.169:6443: connect: connection refused\nE0305 14:01:09.863701       1 reflector.go:307] k8s.io/client-go/informers/factory.go:135: Failed to watch *v1.Node: Get https://api-int.ci-op-s76jkp2x-f83f1.origin-ci-int-aws.dev.rhcloud.com:6443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=38127&timeout=5m37s&timeoutSeconds=337&watch=true: dial tcp 10.0.132.169:6443: connect: connection refused\nE0305 14:01:09.866457       1 reflector.go:307] github.com/openshift/client-go/network/informers/externalversions/factory.go:101: Failed to watch *v1.NetNamespace: Get https://api-int.ci-op-s76jkp2x-f83f1.origin-ci-int-aws.dev.rhcloud.com:6443/apis/network.openshift.io/v1/netnamespaces?allowWatchBookmarks=true&resourceVersion=29535&timeout=7m5s&timeoutSeconds=425&watch=true: dial tcp 10.0.132.169:6443: connect: connection refused\n
Mar 05 14:06:43.984 E ns/openshift-sdn pod/ovs-f2q4q node/ip-10-0-130-18.ec2.internal container=openvswitch container exited with code 143 (Error): Z|00024|reconnect|WARN|unix#723: connection dropped (Broken pipe)\n2020-03-05T14:04:20.140Z|00025|jsonrpc|WARN|unix#726: send error: Broken pipe\n2020-03-05T14:04:20.140Z|00026|reconnect|WARN|unix#726: connection dropped (Broken pipe)\n2020-03-05T14:04:20.066Z|00152|bridge|INFO|bridge br0: added interface veth6f7428ef on port 74\n2020-03-05T14:04:20.109Z|00153|connmgr|INFO|br0<->unix#829: 5 flow_mods in the last 0 s (5 adds)\n2020-03-05T14:04:20.196Z|00154|connmgr|INFO|br0<->unix#833: 2 flow_mods in the last 0 s (2 deletes)\n2020-03-05T14:04:20.197Z|00155|connmgr|INFO|br0<->unix#835: 2 flow_mods in the last 0 s (1 adds, 1 deletes)\n2020-03-05T14:04:20.325Z|00156|connmgr|INFO|br0<->unix#838: 2 flow_mods in the last 0 s (2 deletes)\n2020-03-05T14:04:20.366Z|00157|connmgr|INFO|br0<->unix#841: 4 flow_mods in the last 0 s (4 deletes)\n2020-03-05T14:04:20.390Z|00158|bridge|INFO|bridge br0: deleted interface vetha689f960 on port 72\n2020-03-05T14:04:21.412Z|00159|connmgr|INFO|br0<->unix#845: 2 flow_mods in the last 0 s (2 deletes)\n2020-03-05T14:04:21.445Z|00160|connmgr|INFO|br0<->unix#848: 4 flow_mods in the last 0 s (4 deletes)\n2020-03-05T14:04:21.526Z|00161|bridge|INFO|bridge br0: deleted interface veth14544b1f on port 63\n2020-03-05T14:04:22.235Z|00162|bridge|INFO|bridge br0: added interface veth004472c9 on port 75\n2020-03-05T14:04:22.276Z|00163|connmgr|INFO|br0<->unix#851: 5 flow_mods in the last 0 s (5 adds)\n2020-03-05T14:04:22.333Z|00164|connmgr|INFO|br0<->unix#855: 2 flow_mods in the last 0 s (2 deletes)\n2020-03-05T14:04:22.335Z|00165|connmgr|INFO|br0<->unix#857: 2 flow_mods in the last 0 s (1 adds, 1 deletes)\n2020-03-05T14:04:23.177Z|00166|connmgr|INFO|br0<->unix#862: 2 flow_mods in the last 0 s (2 deletes)\n2020-03-05T14:04:23.226Z|00167|connmgr|INFO|br0<->unix#865: 4 flow_mods in the last 0 s (4 deletes)\n2020-03-05T14:04:23.271Z|00168|bridge|INFO|bridge br0: deleted interface veth6f7428ef on port 74\ninfo: Saving flows ...\novs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)\n
Mar 05 14:06:44.008 E ns/openshift-machine-config-operator pod/machine-config-daemon-vjdf8 node/ip-10-0-130-18.ec2.internal container=oauth-proxy container exited with code 143 (Error): 
Mar 05 14:06:44.038 E ns/openshift-kube-controller-manager pod/kube-controller-manager-ip-10-0-130-18.ec2.internal node/ip-10-0-130-18.ec2.internal container=kube-controller-manager-cert-syncer container exited with code 2 (Error):     1 certsync_controller.go:162] Syncing secrets: [{csr-signer false}]\nI0305 14:03:49.080028       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 14:03:49.080995       1 certsync_controller.go:162] Syncing secrets: [{csr-signer false}]\nI0305 14:03:50.260496       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 14:03:50.260859       1 certsync_controller.go:162] Syncing secrets: [{csr-signer false}]\nI0305 14:03:59.086902       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 14:03:59.087277       1 certsync_controller.go:162] Syncing secrets: [{csr-signer false}]\nI0305 14:04:00.269978       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 14:04:00.270407       1 certsync_controller.go:162] Syncing secrets: [{csr-signer false}]\nI0305 14:04:09.112959       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 14:04:09.113410       1 certsync_controller.go:162] Syncing secrets: [{csr-signer false}]\nI0305 14:04:10.278521       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 14:04:10.278886       1 certsync_controller.go:162] Syncing secrets: [{csr-signer false}]\nI0305 14:04:19.126066       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 14:04:19.126807       1 certsync_controller.go:162] Syncing secrets: [{csr-signer false}]\nI0305 14:04:20.289989       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 14:04:20.290565       1 certsync_controller.go:162] Syncing secrets: [{csr-signer false}]\n
Mar 05 14:06:44.038 E ns/openshift-kube-controller-manager pod/kube-controller-manager-ip-10-0-130-18.ec2.internal node/ip-10-0-130-18.ec2.internal container=kube-controller-manager container exited with code 2 (Error): controller_utils.go:603] Controller packageserver-5cddd6f4dc deleting pod openshift-operator-lifecycle-manager/packageserver-5cddd6f4dc-h6x6g\nI0305 14:04:20.678329       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-operator-lifecycle-manager", Name:"packageserver", UID:"2057d72c-059f-4f6c-987c-f64736087176", APIVersion:"apps/v1", ResourceVersion:"40764", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down replica set packageserver-5cddd6f4dc to 1\nI0305 14:04:20.684984       1 event.go:281] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"openshift-operator-lifecycle-manager", Name:"packageserver-5cddd6f4dc", UID:"d57d9fc6-d884-4611-9e8d-6d3f3aa48a73", APIVersion:"apps/v1", ResourceVersion:"40824", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: packageserver-5cddd6f4dc-h6x6g\nI0305 14:04:20.702875       1 replica_set.go:561] Too few replicas for ReplicaSet openshift-operator-lifecycle-manager/packageserver-9d9cd999, need 2, creating 1\nI0305 14:04:20.703823       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-operator-lifecycle-manager", Name:"packageserver", UID:"2057d72c-059f-4f6c-987c-f64736087176", APIVersion:"apps/v1", ResourceVersion:"40764", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set packageserver-9d9cd999 to 2\nI0305 14:04:20.707511       1 deployment_controller.go:484] Error syncing deployment openshift-operator-lifecycle-manager/packageserver: Operation cannot be fulfilled on deployments.apps "packageserver": the object has been modified; please apply your changes to the latest version and try again\nI0305 14:04:20.784337       1 event.go:281] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"openshift-operator-lifecycle-manager", Name:"packageserver-9d9cd999", UID:"bbd57898-b810-4021-8d3d-7205aa0b1ca9", APIVersion:"apps/v1", ResourceVersion:"40831", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: packageserver-9d9cd999-hxwpm\n
Mar 05 14:06:44.038 E ns/openshift-kube-controller-manager pod/kube-controller-manager-ip-10-0-130-18.ec2.internal node/ip-10-0-130-18.ec2.internal container=kube-controller-manager-recovery-controller container exited with code 255 (Error): 13:06:33 +0000 UTC (now=2020-03-05 13:41:26.130278651 +0000 UTC))\nI0305 13:41:26.130702       1 tlsconfig.go:179] loaded serving cert ["serving-cert::/tmp/serving-cert-103617731/tls.crt::/tmp/serving-cert-103617731/tls.key"]: "localhost" [serving] validServingFor=[localhost] issuer="cert-recovery-controller-signer@1583415683" (2020-03-05 13:41:23 +0000 UTC to 2020-04-04 13:41:24 +0000 UTC (now=2020-03-05 13:41:26.130684661 +0000 UTC))\nI0305 13:41:26.131067       1 named_certificates.go:52] loaded SNI cert [0/"self-signed loopback"]: "apiserver-loopback-client@1583415685" [serving] validServingFor=[apiserver-loopback-client] issuer="apiserver-loopback-client-ca@1583415684" (2020-03-05 12:41:24 +0000 UTC to 2021-03-05 12:41:24 +0000 UTC (now=2020-03-05 13:41:26.131051135 +0000 UTC))\nI0305 13:58:29.815327       1 leaderelection.go:252] successfully acquired lease openshift-kube-controller-manager/cert-recovery-controller-lock\nI0305 13:58:29.815389       1 event.go:281] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"openshift-kube-controller-manager", Name:"cert-recovery-controller-lock", UID:"8b91bf4c-33fc-40b4-ba13-5d645811c74d", APIVersion:"v1", ResourceVersion:"35469", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' daab28b4-5037-400e-a641-bc0679fcaffe became leader\nI0305 13:58:29.817513       1 csrcontroller.go:81] Starting CSR controller\nI0305 13:58:29.817535       1 shared_informer.go:197] Waiting for caches to sync for CSRController\nI0305 13:58:29.919392       1 shared_informer.go:204] Caches are synced for CSRController \nI0305 14:04:26.074422       1 cmd.go:84] Received SIGTERM or SIGINT signal, shutting down controller.\nI0305 14:04:26.074861       1 dynamic_serving_content.go:144] Shutting down serving-cert::/tmp/serving-cert-103617731/tls.crt::/tmp/serving-cert-103617731/tls.key\nI0305 14:04:26.077142       1 csrcontroller.go:83] Shutting down CSR controller\nI0305 14:04:26.077211       1 csrcontroller.go:85] CSR controller shut down\nF0305 14:04:26.077557       1 builder.go:209] server exited\n
Mar 05 14:06:44.038 E ns/openshift-kube-controller-manager pod/kube-controller-manager-ip-10-0-130-18.ec2.internal node/ip-10-0-130-18.ec2.internal container=cluster-policy-controller container exited with code 255 (Error): : dial tcp [::1]:6443: connect: connection refused\nE0305 14:05:03.496515       1 reflector.go:307] k8s.io/client-go/informers/factory.go:135: Failed to watch *v1.RoleBinding: Get https://localhost:6443/apis/rbac.authorization.k8s.io/v1/rolebindings?allowWatchBookmarks=true&resourceVersion=34638&timeout=9m51s&timeoutSeconds=591&watch=true: dial tcp [::1]:6443: connect: connection refused\nE0305 14:05:03.497616       1 reflector.go:307] k8s.io/client-go/informers/factory.go:135: Failed to watch *v1.PersistentVolumeClaim: Get https://localhost:6443/api/v1/persistentvolumeclaims?allowWatchBookmarks=true&resourceVersion=34655&timeout=7m14s&timeoutSeconds=434&watch=true: dial tcp [::1]:6443: connect: connection refused\nE0305 14:05:03.498786       1 reflector.go:307] github.com/openshift/client-go/build/informers/externalversions/factory.go:101: Failed to watch *v1.Build: Get https://localhost:6443/apis/build.openshift.io/v1/builds?allowWatchBookmarks=true&resourceVersion=40198&timeout=9m12s&timeoutSeconds=552&watch=true: dial tcp [::1]:6443: connect: connection refused\nE0305 14:05:03.509797       1 reflector.go:307] k8s.io/client-go/informers/factory.go:135: Failed to watch *v1beta1.PodDisruptionBudget: Get https://localhost:6443/apis/policy/v1beta1/poddisruptionbudgets?allowWatchBookmarks=true&resourceVersion=40345&timeout=5m2s&timeoutSeconds=302&watch=true: dial tcp [::1]:6443: connect: connection refused\nE0305 14:05:03.511163       1 reflector.go:307] github.com/openshift/client-go/apps/informers/externalversions/factory.go:101: Failed to watch *v1.DeploymentConfig: Get https://localhost:6443/apis/apps.openshift.io/v1/deploymentconfigs?allowWatchBookmarks=true&resourceVersion=40198&timeout=6m58s&timeoutSeconds=418&watch=true: dial tcp [::1]:6443: connect: connection refused\nI0305 14:05:04.127322       1 leaderelection.go:288] failed to renew lease openshift-kube-controller-manager/cluster-policy-controller: timed out waiting for the condition\nF0305 14:05:04.127374       1 policy_controller.go:94] leaderelection lost\n
Mar 05 14:06:44.054 E ns/openshift-multus pod/multus-admission-controller-pgwqs node/ip-10-0-130-18.ec2.internal container=multus-admission-controller container exited with code 255 (Error): 
Mar 05 14:06:44.071 E ns/openshift-kube-scheduler pod/openshift-kube-scheduler-ip-10-0-130-18.ec2.internal node/ip-10-0-130-18.ec2.internal container=scheduler container exited with code 2 (Error): -54c77c59d8-wx59p: no fit: 0/6 nodes are available: 2 node(s) didn't match node selector, 2 node(s) didn't match pod affinity/anti-affinity, 2 node(s) were unschedulable.; waiting\nI0305 14:04:06.327086       1 factory.go:453] Unable to schedule openshift-apiserver/apiserver-7df4757845-nhj49: no fit: 0/6 nodes are available: 2 node(s) didn't match node selector, 2 node(s) didn't match pod affinity/anti-affinity, 2 node(s) were unschedulable.; waiting\nI0305 14:04:08.006194       1 scheduler.go:751] pod openshift-operator-lifecycle-manager/packageserver-9d9cd999-tzlsq is bound successfully on node "ip-10-0-135-26.ec2.internal", 6 nodes evaluated, 2 nodes were found feasible.\nI0305 14:04:09.326222       1 factory.go:453] Unable to schedule openshift-machine-config-operator/etcd-quorum-guard-54c77c59d8-wx59p: no fit: 0/6 nodes are available: 2 node(s) didn't match node selector, 2 node(s) didn't match pod affinity/anti-affinity, 2 node(s) were unschedulable.; waiting\nI0305 14:04:14.349042       1 factory.go:453] Unable to schedule openshift-apiserver/apiserver-7df4757845-nhj49: no fit: 0/6 nodes are available: 2 node(s) didn't match node selector, 2 node(s) didn't match pod affinity/anti-affinity, 2 node(s) were unschedulable.; waiting\nI0305 14:04:17.347829       1 factory.go:453] Unable to schedule openshift-machine-config-operator/etcd-quorum-guard-54c77c59d8-wx59p: no fit: 0/6 nodes are available: 2 node(s) didn't match node selector, 2 node(s) didn't match pod affinity/anti-affinity, 2 node(s) were unschedulable.; waiting\nI0305 14:04:20.793283       1 scheduler.go:751] pod openshift-operator-lifecycle-manager/packageserver-9d9cd999-hxwpm is bound successfully on node "ip-10-0-150-76.ec2.internal", 6 nodes evaluated, 2 nodes were found feasible.\nI0305 14:04:25.351260       1 factory.go:453] Unable to schedule openshift-apiserver/apiserver-7df4757845-nhj49: no fit: 0/6 nodes are available: 2 node(s) didn't match node selector, 2 node(s) didn't match pod affinity/anti-affinity, 2 node(s) were unschedulable.; waiting\n
Mar 05 14:06:44.141 E ns/openshift-machine-config-operator pod/machine-config-server-hbb75 node/ip-10-0-130-18.ec2.internal container=machine-config-server container exited with code 2 (Error): I0305 13:56:49.137115       1 start.go:38] Version: machine-config-daemon-4.5.0-202003050701-2-gb9fa5093-dirty (b9fa5093eb95cac163db9039e7d2d8bb38f10db6)\nI0305 13:56:49.138562       1 api.go:51] Launching server on :22624\nI0305 13:56:49.138640       1 api.go:51] Launching server on :22623\n
Mar 05 14:06:44.160 E ns/openshift-cluster-node-tuning-operator pod/tuned-hgnrw node/ip-10-0-130-18.ec2.internal container=tuned container exited with code 143 (Error): ng tuned...\n2020-03-05 13:44:18,155 INFO     tuned.daemon.application: dynamic tuning is globally disabled\n2020-03-05 13:44:18,169 INFO     tuned.daemon.daemon: using sleep interval of 1 second(s)\n2020-03-05 13:44:18,170 INFO     tuned.daemon.daemon: Running in automatic mode, checking what profile is recommended for your configuration.\n2020-03-05 13:44:18,171 INFO     tuned.daemon.daemon: Using 'openshift-control-plane' profile\n2020-03-05 13:44:18,178 INFO     tuned.profiles.loader: loading profile: openshift-control-plane\n2020-03-05 13:44:18,272 INFO     tuned.daemon.controller: starting controller\n2020-03-05 13:44:18,272 INFO     tuned.daemon.daemon: starting tuning\n2020-03-05 13:44:18,290 INFO     tuned.plugins.base: instance cpu: assigning devices cpu2, cpu3, cpu0, cpu1\n2020-03-05 13:44:18,294 INFO     tuned.plugins.plugin_cpu: We are running on an x86 GenuineIntel platform\n2020-03-05 13:44:18,309 WARNING  tuned.plugins.plugin_cpu: your CPU doesn't support MSR_IA32_ENERGY_PERF_BIAS, ignoring CPU energy performance bias\n2020-03-05 13:44:18,311 INFO     tuned.plugins.base: instance disk: assigning devices dm-0, xvda\n2020-03-05 13:44:18,315 INFO     tuned.plugins.base: instance net: assigning devices ens3\n2020-03-05 13:44:18,541 INFO     tuned.plugins.plugin_sysctl: reapplying system sysctl\n2020-03-05 13:44:18,548 INFO     tuned.daemon.daemon: static tuning from profile 'openshift-control-plane' applied\nI0305 14:04:08.839196    1157 tuned.go:494] profile "ip-10-0-130-18.ec2.internal" changed, tuned profile requested: openshift-node\nI0305 14:04:09.222726    1157 tuned.go:494] profile "ip-10-0-130-18.ec2.internal" changed, tuned profile requested: openshift-control-plane\nI0305 14:04:09.747570    1157 tuned.go:393] getting recommended profile...\nI0305 14:04:09.876319    1157 tuned.go:430] active and recommended profile (openshift-control-plane) match; profile change will not trigger profile reload\nI0305 14:04:26.221913    1157 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF\n
Mar 05 14:06:44.184 E ns/openshift-etcd pod/etcd-ip-10-0-130-18.ec2.internal node/ip-10-0-130-18.ec2.internal container=etcd-metrics container exited with code 2 (Error): 2020-03-05 13:41:13.289203 I | etcdmain: ServerTLS: cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-130-18.ec2.internal.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-130-18.ec2.internal.key, ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-metrics-proxy-serving-ca/ca-bundle.crt, trusted-ca = , client-cert-auth = false, crl-file = \n2020-03-05 13:41:13.290048 I | etcdmain: listening for grpc-proxy client requests on 127.0.0.1:9977\n2020-03-05 13:41:13.290478 I | etcdmain: ClientTLS: cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-peer/etcd-peer-ip-10-0-130-18.ec2.internal.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-peer/etcd-peer-ip-10-0-130-18.ec2.internal.key, ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-peer-client-ca/ca-bundle.crt, trusted-ca = , client-cert-auth = false, crl-file = \n2020-03-05 13:41:13.292602 I | etcdmain: grpc-proxy: listening for metrics on https://0.0.0.0:9979\nWARNING: 2020/03/05 13:41:13 grpc: addrConn.createTransport failed to connect to {https://etcd-0.ci-op-s76jkp2x-f83f1.origin-ci-int-aws.dev.rhcloud.com:9978 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.130.18:9978: connect: connection refused". Reconnecting...\nWARNING: 2020/03/05 13:41:14 grpc: addrConn.createTransport failed to connect to {https://etcd-0.ci-op-s76jkp2x-f83f1.origin-ci-int-aws.dev.rhcloud.com:9978 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.130.18:9978: connect: connection refused". Reconnecting...\n
Mar 05 14:06:44.210 E ns/openshift-multus pod/multus-v5qwn node/ip-10-0-130-18.ec2.internal container=kube-multus container exited with code 143 (Error): 
Mar 05 14:06:44.235 E ns/openshift-kube-apiserver pod/kube-apiserver-ip-10-0-130-18.ec2.internal node/ip-10-0-130-18.ec2.internal container=kube-apiserver container exited with code 1 (Error): nnection error: desc = "transport: authentication handshake failed: x509: certificate is valid for 10.0.130.18, 127.0.0.1, not ::1". Reconnecting...\nW0305 14:04:22.458449       1 clientconn.go:1120] grpc: addrConn.createTransport failed to connect to {https://[::1]:2379 0  <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for 10.0.130.18, 127.0.0.1, not ::1". Reconnecting...\nW0305 14:04:23.384624       1 clientconn.go:1120] grpc: addrConn.createTransport failed to connect to {https://[::1]:2379 0  <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for 10.0.130.18, 127.0.0.1, not ::1". Reconnecting...\nW0305 14:04:24.502515       1 clientconn.go:1120] grpc: addrConn.createTransport failed to connect to {https://[::1]:2379 0  <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for 10.0.130.18, 127.0.0.1, not ::1". Reconnecting...\nW0305 14:04:25.332356       1 clientconn.go:1120] grpc: addrConn.createTransport failed to connect to {https://[::1]:2379 0  <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for 10.0.130.18, 127.0.0.1, not ::1". Reconnecting...\nW0305 14:04:25.534165       1 clientconn.go:1120] grpc: addrConn.createTransport failed to connect to {https://[::1]:2379 0  <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for 10.0.130.18, 127.0.0.1, not ::1". Reconnecting...\nI0305 14:04:26.107925       1 genericapiserver.go:643] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-ip-10-0-130-18.ec2.internal", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TerminationStart' Received signal to terminate, becoming unready, but keeping serving\nI0305 14:04:26.108415       1 controller.go:180] Shutting down kubernetes service endpoint reconciler\n
Mar 05 14:06:44.235 E ns/openshift-kube-apiserver pod/kube-apiserver-ip-10-0-130-18.ec2.internal node/ip-10-0-130-18.ec2.internal container=kube-apiserver-insecure-readyz container exited with code 2 (Error): I0305 13:41:14.656803       1 readyz.go:103] Listening on 0.0.0.0:6080\n
Mar 05 14:06:44.235 E ns/openshift-kube-apiserver pod/kube-apiserver-ip-10-0-130-18.ec2.internal node/ip-10-0-130-18.ec2.internal container=kube-apiserver-cert-syncer container exited with code 2 (Error): ing-cert-001 true} {user-serving-cert-002 true} {user-serving-cert-003 true} {user-serving-cert-004 true} {user-serving-cert-005 true} {user-serving-cert-006 true} {user-serving-cert-007 true} {user-serving-cert-008 true} {user-serving-cert-009 true}]\nI0305 14:04:24.903138       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 14:04:24.903627       1 certsync_controller.go:162] Syncing secrets: [{aggregator-client false} {localhost-serving-cert-certkey false} {service-network-serving-certkey false} {external-loadbalancer-serving-certkey false} {internal-loadbalancer-serving-certkey false} {bound-service-account-signing-key false} {user-serving-cert true} {user-serving-cert-000 true} {user-serving-cert-001 true} {user-serving-cert-002 true} {user-serving-cert-003 true} {user-serving-cert-004 true} {user-serving-cert-005 true} {user-serving-cert-006 true} {user-serving-cert-007 true} {user-serving-cert-008 true} {user-serving-cert-009 true}]\nI0305 14:04:26.043407       1 certsync_controller.go:65] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true}]\nI0305 14:04:26.044117       1 certsync_controller.go:162] Syncing secrets: [{aggregator-client false} {localhost-serving-cert-certkey false} {service-network-serving-certkey false} {external-loadbalancer-serving-certkey false} {internal-loadbalancer-serving-certkey false} {bound-service-account-signing-key false} {user-serving-cert true} {user-serving-cert-000 true} {user-serving-cert-001 true} {user-serving-cert-002 true} {user-serving-cert-003 true} {user-serving-cert-004 true} {user-serving-cert-005 true} {user-serving-cert-006 true} {user-serving-cert-007 true} {user-serving-cert-008 true} {user-serving-cert-009 true}]\nW0305 14:04:26.250829       1 reflector.go:326] runtime/asm_amd64.s:1357: watch of *v1.ConfigMap ended with: very short watch: runtime/asm_amd64.s:1357: Unexpected watch close - watch lasted less than a second and no items received\n
Mar 05 14:06:44.235 E ns/openshift-kube-apiserver pod/kube-apiserver-ip-10-0-130-18.ec2.internal node/ip-10-0-130-18.ec2.internal container=kube-apiserver-cert-regeneration-controller container exited with code 255 (Error): e.go:40] syncing servicenetwork hostnames: [172.30.0.1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local openshift openshift.default openshift.default.svc openshift.default.svc.cluster.local]\nI0305 14:02:22.516591       1 externalloadbalancer.go:26] syncing external loadbalancer hostnames: api.ci-op-s76jkp2x-f83f1.origin-ci-int-aws.dev.rhcloud.com\nI0305 14:04:26.034211       1 cmd.go:84] Received SIGTERM or SIGINT signal, shutting down controller.\nI0305 14:04:26.034841       1 client_cert_rotation_controller.go:180] Shutting down CertRotationController - "LocalhostServing"\nI0305 14:04:26.034918       1 client_cert_rotation_controller.go:180] Shutting down CertRotationController - "KubeControllerManagerClient"\nI0305 14:04:26.034952       1 client_cert_rotation_controller.go:180] Shutting down CertRotationController - "ExternalLoadBalancerServing"\nI0305 14:04:26.034973       1 client_cert_rotation_controller.go:180] Shutting down CertRotationController - "LocalhostRecoveryServing"\nI0305 14:04:26.035003       1 client_cert_rotation_controller.go:180] Shutting down CertRotationController - "ServiceNetworkServing"\nI0305 14:04:26.035021       1 client_cert_rotation_controller.go:180] Shutting down CertRotationController - "KubeSchedulerClient"\nI0305 14:04:26.035036       1 client_cert_rotation_controller.go:180] Shutting down CertRotationController - "KubeAPIServerToKubeletClientCert"\nI0305 14:04:26.035056       1 client_cert_rotation_controller.go:180] Shutting down CertRotationController - "InternalLoadBalancerServing"\nI0305 14:04:26.035076       1 client_cert_rotation_controller.go:180] Shutting down CertRotationController - "AggregatorProxyClientCert"\nI0305 14:04:26.035158       1 certrotationcontroller.go:560] Shutting down CertRotation\nI0305 14:04:26.035182       1 cabundlesyncer.go:84] Shutting down CA bundle controller\nI0305 14:04:26.035193       1 cabundlesyncer.go:86] CA bundle controller shut down\nF0305 14:04:26.044129       1 leaderelection.go:67] leaderelection lost\n
Mar 05 14:06:44.286 E ns/openshift-monitoring pod/node-exporter-ddbc8 node/ip-10-0-130-18.ec2.internal container=node-exporter container exited with code 143 (Error): or gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:03:29Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:03:44Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:03:44Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:03:59Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:03:59Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:04:14Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:04:14Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\n
Mar 05 14:06:44.302 E ns/openshift-controller-manager pod/controller-manager-skr84 node/ip-10-0-130-18.ec2.internal container=controller-manager container exited with code 1 (Error): 05 14:02:23.329384       1 factory.go:80] Deployer controller caches are synced. Starting workers.\nI0305 14:02:23.510745       1 build_controller.go:474] Starting build controller\nI0305 14:02:23.510770       1 build_controller.go:476] OpenShift image registry hostname: image-registry.openshift-image-registry.svc:5000\nI0305 14:02:23.547941       1 deleted_dockercfg_secrets.go:74] caches synced\nI0305 14:02:23.547941       1 docker_registry_service.go:154] caches synced\nI0305 14:02:23.547955       1 create_dockercfg_secrets.go:218] urls found\nI0305 14:02:23.548119       1 create_dockercfg_secrets.go:224] caches synced\nI0305 14:02:23.548304       1 docker_registry_service.go:296] Updating registry URLs from map[172.30.158.40:5000:{} image-registry.openshift-image-registry.svc.cluster.local:5000:{} image-registry.openshift-image-registry.svc:5000:{}] to map[172.30.158.40:5000:{} image-registry.openshift-image-registry.svc.cluster.local:5000:{} image-registry.openshift-image-registry.svc:5000:{}]\nI0305 14:02:23.547949       1 deleted_token_secrets.go:69] caches synced\nW0305 14:03:55.641364       1 reflector.go:340] github.com/openshift/client-go/build/informers/externalversions/factory.go:101: watch of *v1.BuildConfig ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 323; INTERNAL_ERROR") has prevented the request from succeeding\nW0305 14:03:55.641573       1 reflector.go:340] github.com/openshift/client-go/build/informers/externalversions/factory.go:101: watch of *v1.Build ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 317; INTERNAL_ERROR") has prevented the request from succeeding\nW0305 14:03:55.641746       1 reflector.go:340] github.com/openshift/client-go/route/informers/externalversions/factory.go:101: watch of *v1.Route ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 319; INTERNAL_ERROR") has prevented the request from succeeding\n
Mar 05 14:06:49.177 E ns/openshift-monitoring pod/node-exporter-ddbc8 node/ip-10-0-130-18.ec2.internal invariant violation: pod may not transition Running->Pending
Mar 05 14:06:57.826 E ns/openshift-machine-config-operator pod/machine-config-daemon-vjdf8 node/ip-10-0-130-18.ec2.internal container=oauth-proxy container exited with code 1 (Error): 
Mar 05 14:07:01.181 E ns/openshift-monitoring pod/prometheus-adapter-996df9bdb-4ztsn node/ip-10-0-130-45.ec2.internal container=prometheus-adapter container exited with code 2 (Error): I0305 13:43:15.867860       1 adapter.go:93] successfully using in-cluster auth\nI0305 13:43:17.028648       1 secure_serving.go:116] Serving securely on [::]:6443\n
Mar 05 14:07:01.214 E ns/openshift-monitoring pod/prometheus-k8s-0 node/ip-10-0-130-45.ec2.internal container=rules-configmap-reloader container exited with code 2 (Error): 2020/03/05 13:57:17 Watching directory: "/etc/prometheus/rules/prometheus-k8s-rulefiles-0"\n
Mar 05 14:07:01.214 E ns/openshift-monitoring pod/prometheus-k8s-0 node/ip-10-0-130-45.ec2.internal container=prometheus-proxy container exited with code 2 (Error): 2020/03/05 13:57:18 provider.go:118: Defaulting client-id to system:serviceaccount:openshift-monitoring:prometheus-k8s\n2020/03/05 13:57:18 provider.go:123: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token\n2020/03/05 13:57:18 provider.go:311: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.\n2020/03/05 13:57:18 oauthproxy.go:200: mapping path "/" => upstream "http://localhost:9090/"\n2020/03/05 13:57:18 oauthproxy.go:221: compiled skip-auth-regex => "^/metrics"\n2020/03/05 13:57:18 oauthproxy.go:227: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:prometheus-k8s\n2020/03/05 13:57:18 oauthproxy.go:237: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> refresh:disabled\n2020/03/05 13:57:18 main.go:154: using htpasswd file /etc/proxy/htpasswd/auth\n2020/03/05 13:57:18 http.go:107: HTTPS: listening on [::]:9091\nI0305 13:57:18.449375       1 dynamic_serving_content.go:129] Starting serving::/etc/tls/private/tls.crt::/etc/tls/private/tls.key\n2020/03/05 14:00:51 oauthproxy.go:774: basicauth: 10.130.0.20:34794 Authorization header does not start with 'Basic', skipping basic authentication\n
Mar 05 14:07:01.214 E ns/openshift-monitoring pod/prometheus-k8s-0 node/ip-10-0-130-45.ec2.internal container=prometheus-config-reloader container exited with code 2 (Error): ts=2020-03-05T13:57:17.455210335Z caller=main.go:85 msg="Starting prometheus-config-reloader version '1.12.16'."\nlevel=error ts=2020-03-05T13:57:17.457880623Z caller=runutil.go:95 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post http://localhost:9090/-/reload: dial tcp [::1]:9090: connect: connection refused"\nlevel=info ts=2020-03-05T13:57:22.638662476Z caller=reloader.go:286 msg="Prometheus reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml rule_dirs=\nlevel=info ts=2020-03-05T13:57:22.638750123Z caller=reloader.go:154 msg="started watching config file and non-recursively rule dirs for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=\n
Mar 05 14:07:01.346 E ns/openshift-monitoring pod/kube-state-metrics-6d4b44f449-zcskf node/ip-10-0-130-45.ec2.internal container=kube-state-metrics container exited with code 2 (Error): 
Mar 05 14:07:02.353 E ns/openshift-monitoring pod/alertmanager-main-2 node/ip-10-0-130-45.ec2.internal container=config-reloader container exited with code 2 (Error): 2020/03/05 13:57:02 Watching directory: "/etc/alertmanager/config"\n
Mar 05 14:07:02.353 E ns/openshift-monitoring pod/alertmanager-main-2 node/ip-10-0-130-45.ec2.internal container=alertmanager-proxy container exited with code 2 (Error): 2020/03/05 13:57:09 provider.go:118: Defaulting client-id to system:serviceaccount:openshift-monitoring:alertmanager-main\n2020/03/05 13:57:09 provider.go:123: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token\n2020/03/05 13:57:09 provider.go:311: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.\n2020/03/05 13:57:09 oauthproxy.go:200: mapping path "/" => upstream "http://localhost:9093/"\n2020/03/05 13:57:09 oauthproxy.go:221: compiled skip-auth-regex => "^/metrics"\n2020/03/05 13:57:09 oauthproxy.go:227: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:alertmanager-main\n2020/03/05 13:57:09 oauthproxy.go:237: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> refresh:disabled\n2020/03/05 13:57:09 http.go:107: HTTPS: listening on [::]:9095\nI0305 13:57:09.559253       1 dynamic_serving_content.go:129] Starting serving::/etc/tls/private/tls.crt::/etc/tls/private/tls.key\n
Mar 05 14:07:02.422 E ns/openshift-csi-snapshot-controller-operator pod/csi-snapshot-controller-operator-6855d489cb-7j8nc node/ip-10-0-130-45.ec2.internal container=operator container exited with code 255 (Error): .go:326] github.com/openshift/client-go/config/informers/externalversions/factory.go:101: watch of *v1.ClusterOperator ended with: very short watch: github.com/openshift/client-go/config/informers/externalversions/factory.go:101: Unexpected watch close - watch lasted less than a second and no items received\nW0305 14:01:09.836878       1 reflector.go:326] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: watch of *v1beta1.CustomResourceDefinition ended with: very short watch: k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: Unexpected watch close - watch lasted less than a second and no items received\nI0305 14:01:10.884452       1 operator.go:145] Starting syncing operator at 2020-03-05 14:01:10.884443781 +0000 UTC m=+241.024205372\nI0305 14:01:11.082750       1 operator.go:147] Finished syncing operator at 198.296237ms\nI0305 14:01:11.165247       1 operator.go:145] Starting syncing operator at 2020-03-05 14:01:11.165233312 +0000 UTC m=+241.304995022\nI0305 14:01:11.217897       1 operator.go:147] Finished syncing operator at 52.653917ms\nI0305 14:01:11.217957       1 operator.go:145] Starting syncing operator at 2020-03-05 14:01:11.217950597 +0000 UTC m=+241.357712316\nI0305 14:01:11.306207       1 operator.go:147] Finished syncing operator at 88.249107ms\nI0305 14:01:12.168917       1 operator.go:145] Starting syncing operator at 2020-03-05 14:01:12.168905844 +0000 UTC m=+242.308667547\nI0305 14:01:12.198473       1 operator.go:147] Finished syncing operator at 29.559555ms\nI0305 14:06:59.758074       1 cmd.go:79] Received SIGTERM or SIGINT signal, shutting down controller.\nI0305 14:06:59.758577       1 logging_controller.go:93] Shutting down LogLevelController\nI0305 14:06:59.758600       1 status_controller.go:212] Shutting down StatusSyncer-csi-snapshot-controller\nI0305 14:06:59.758619       1 management_state_controller.go:112] Shutting down management-state-controller-csi-snapshot-controller\nF0305 14:06:59.758707       1 builder.go:243] stopped\n
Mar 05 14:07:02.466 E ns/openshift-kube-storage-version-migrator pod/migrator-c89f75c5f-pb6z5 node/ip-10-0-130-45.ec2.internal container=migrator container exited with code 2 (Error): I0305 13:57:26.937527       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF\nI0305 14:01:09.837614       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF\nI0305 14:04:26.220838       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF\n
Mar 05 14:07:13.168 E ns/openshift-monitoring pod/prometheus-k8s-0 node/ip-10-0-153-229.ec2.internal container=prometheus container exited with code 1 (Error): caller=main.go:648 msg="Starting TSDB ..."\nlevel=info ts=2020-03-05T14:07:10.220Z caller=web.go:506 component=web msg="Start listening for connections" address=127.0.0.1:9090\nlevel=info ts=2020-03-05T14:07:10.234Z caller=head.go:584 component=tsdb msg="replaying WAL, this may take awhile"\nlevel=info ts=2020-03-05T14:07:10.235Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0\nlevel=info ts=2020-03-05T14:07:10.236Z caller=main.go:663 fs_type=XFS_SUPER_MAGIC\nlevel=info ts=2020-03-05T14:07:10.236Z caller=main.go:664 msg="TSDB started"\nlevel=info ts=2020-03-05T14:07:10.236Z caller=main.go:734 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml\nlevel=info ts=2020-03-05T14:07:10.236Z caller=main.go:517 msg="Stopping scrape discovery manager..."\nlevel=info ts=2020-03-05T14:07:10.236Z caller=main.go:531 msg="Stopping notify discovery manager..."\nlevel=info ts=2020-03-05T14:07:10.236Z caller=main.go:553 msg="Stopping scrape manager..."\nlevel=info ts=2020-03-05T14:07:10.236Z caller=main.go:527 msg="Notify discovery manager stopped"\nlevel=info ts=2020-03-05T14:07:10.236Z caller=main.go:513 msg="Scrape discovery manager stopped"\nlevel=info ts=2020-03-05T14:07:10.236Z caller=main.go:547 msg="Scrape manager stopped"\nlevel=info ts=2020-03-05T14:07:10.236Z caller=manager.go:814 component="rule manager" msg="Stopping rule manager..."\nlevel=info ts=2020-03-05T14:07:10.236Z caller=manager.go:820 component="rule manager" msg="Rule manager stopped"\nlevel=info ts=2020-03-05T14:07:10.240Z caller=notifier.go:598 component=notifier msg="Stopping notification manager..."\nlevel=info ts=2020-03-05T14:07:10.240Z caller=main.go:718 msg="Notifier manager stopped"\nlevel=error ts=2020-03-05
Mar 05 14:07:17.109 E ns/openshift-csi-snapshot-controller pod/csi-snapshot-controller-db7cd8967-mr7bq node/ip-10-0-142-254.ec2.internal container=snapshot-controller container exited with code 2 (Error): 
Mar 05 14:09:51.187 E ns/openshift-cluster-node-tuning-operator pod/tuned-ls8sd node/ip-10-0-130-45.ec2.internal container=tuned container exited with code 143 (Error): pected EOF\nW0305 14:01:09.836607    1413 reflector.go:340] github.com/openshift/cluster-node-tuning-operator/pkg/tuned/tuned.go:598: watch of *v1.Tuned ended with: very short watch: github.com/openshift/cluster-node-tuning-operator/pkg/tuned/tuned.go:598: Unexpected watch close - watch lasted less than a second and no items received\nI0305 14:04:26.233867    1413 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF\nI0305 14:04:26.234302    1413 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF\nE0305 14:04:26.239683    1413 reflector.go:320] github.com/openshift/cluster-node-tuning-operator/pkg/tuned/tuned.go:598: Failed to watch *v1.Tuned: Get https://172.30.0.1:443/apis/tuned.openshift.io/v1/namespaces/openshift-cluster-node-tuning-operator/tuneds?allowWatchBookmarks=true&fieldSelector=metadata.name%3Drendered&resourceVersion=38553&timeoutSeconds=544&watch=true: dial tcp 172.30.0.1:443: connect: connection refused\nE0305 14:04:26.239919    1413 reflector.go:320] github.com/openshift/cluster-node-tuning-operator/pkg/tuned/tuned.go:594: Failed to watch *v1.Profile: Get https://172.30.0.1:443/apis/tuned.openshift.io/v1/namespaces/openshift-cluster-node-tuning-operator/profiles?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dip-10-0-130-45.ec2.internal&resourceVersion=38554&timeoutSeconds=454&watch=true: dial tcp 172.30.0.1:443: connect: connection refused\nE0305 14:04:27.244786    1413 reflector.go:320] github.com/openshift/cluster-node-tuning-operator/pkg/tuned/tuned.go:594: Failed to watch *v1.Profile: Get https://172.30.0.1:443/apis/tuned.openshift.io/v1/namespaces/openshift-cluster-node-tuning-operator/profiles?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dip-10-0-130-45.ec2.internal&resourceVersion=38554&timeoutSeconds=414&watch=true: dial tcp 172.30.0.1:443: connect: connection refused\nI0305 14:08:01.299370    1413 tuned.go:115] received signal: terminated\nI0305 14:08:01.299782    1413 tuned.go:327] sending TERM to PID 1461\n
Mar 05 14:09:51.203 E ns/openshift-monitoring pod/node-exporter-hgs29 node/ip-10-0-130-45.ec2.internal container=node-exporter container exited with code 143 (Error): or gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:07:08Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:07:23Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:07:27Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:07:38Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:07:42Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:07:53Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\ntime="2020-03-05T14:07:57Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"virt_platform\" { label:<name:\"type\" value:\"aws\" > gauge:<value:1 > } was collected before with the same name and label values\n" source="log.go:172"\n
Mar 05 14:09:51.226 E ns/openshift-sdn pod/ovs-rbkbd node/ip-10-0-130-45.ec2.internal container=openvswitch container exited with code 143 (Error): r0: deleted interface vethb80058fb on port 26\n2020-03-05T14:07:02.036Z|00142|connmgr|INFO|br0<->unix#999: 2 flow_mods in the last 0 s (2 deletes)\n2020-03-05T14:07:02.072Z|00143|connmgr|INFO|br0<->unix#1002: 4 flow_mods in the last 0 s (4 deletes)\n2020-03-05T14:07:02.102Z|00144|bridge|INFO|bridge br0: deleted interface vethb82c92c5 on port 20\n2020-03-05T14:07:01.826Z|00011|jsonrpc|WARN|unix#862: receive error: Connection reset by peer\n2020-03-05T14:07:01.826Z|00012|reconnect|WARN|unix#862: connection dropped (Connection reset by peer)\n2020-03-05T14:07:29.729Z|00145|connmgr|INFO|br0<->unix#1023: 2 flow_mods in the last 0 s (2 deletes)\n2020-03-05T14:07:29.768Z|00146|connmgr|INFO|br0<->unix#1026: 4 flow_mods in the last 0 s (4 deletes)\n2020-03-05T14:07:29.795Z|00147|bridge|INFO|bridge br0: deleted interface veth6eddd0c4 on port 30\n2020-03-05T14:07:29.788Z|00013|jsonrpc|WARN|unix#894: receive error: Connection reset by peer\n2020-03-05T14:07:29.788Z|00014|reconnect|WARN|unix#894: connection dropped (Connection reset by peer)\n2020-03-05T14:07:45.079Z|00148|connmgr|INFO|br0<->unix#1039: 2 flow_mods in the last 0 s (2 deletes)\n2020-03-05T14:07:45.107Z|00149|connmgr|INFO|br0<->unix#1042: 4 flow_mods in the last 0 s (4 deletes)\n2020-03-05T14:07:45.128Z|00150|bridge|INFO|bridge br0: deleted interface veth6817a9b6 on port 15\n2020-03-05T14:07:45.118Z|00015|jsonrpc|WARN|unix#908: receive error: Connection reset by peer\n2020-03-05T14:07:45.118Z|00016|reconnect|WARN|unix#908: connection dropped (Connection reset by peer)\n2020-03-05T14:07:47.645Z|00151|connmgr|INFO|br0<->unix#1047: 2 flow_mods in the last 0 s (2 deletes)\n2020-03-05T14:07:47.673Z|00152|connmgr|INFO|br0<->unix#1050: 4 flow_mods in the last 0 s (4 deletes)\n2020-03-05T14:07:47.694Z|00153|bridge|INFO|bridge br0: deleted interface veth91cd22d6 on port 21\n2020-03-05T14:07:56.647Z|00017|jsonrpc|WARN|unix#925: receive error: Connection reset by peer\n2020-03-05T14:07:56.647Z|00018|reconnect|WARN|unix#925: connection dropped (Connection reset by peer)\ninfo: Saving flows ...\n
Mar 05 14:09:51.249 E ns/openshift-multus pod/multus-kc7t6 node/ip-10-0-130-45.ec2.internal container=kube-multus container exited with code 143 (Error): 
Mar 05 14:09:51.271 E ns/openshift-machine-config-operator pod/machine-config-daemon-d8skf node/ip-10-0-130-45.ec2.internal container=oauth-proxy container exited with code 143 (Error): 
Mar 05 14:09:55.059 E ns/openshift-multus pod/multus-kc7t6 node/ip-10-0-130-45.ec2.internal invariant violation: pod may not transition Running->Pending
Mar 05 14:10:01.716 E ns/openshift-machine-config-operator pod/machine-config-daemon-d8skf node/ip-10-0-130-45.ec2.internal container=oauth-proxy container exited with code 1 (Error): 
Mar 05 14:10:01.978 E clusteroperator/machine-config changed Degraded to True: MachineConfigDaemonFailed: Failed to resync 0.0.1-2020-03-05-130558 because: Operation cannot be fulfilled on daemonsets.apps "machine-config-daemon": the object has been modified; please apply your changes to the latest version and try again