release-openshift-ocp-installer-e2e-azure-serial-4.3 #333

• artifacts
• build log

Test Failures

openshift-tests Monitor cluster while tests execute 1h8m

go run hack/e2e.go -v -test --test_args='--ginkgo.focus=openshift\-tests\sMonitor\scluster\swhile\stests\sexecute$'

35 error level events were detected during this test run:

Nov 07 02:03:09.064 E ns/openshift-marketplace pod/redhat-operators-64566c9856-wzdq5 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 container=redhat-operators container exited with code 2 (Error): 
Nov 07 02:03:09.108 E ns/openshift-marketplace pod/certified-operators-69589995bf-lgmd5 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 container=certified-operators container exited with code 2 (Error): 
Nov 07 02:03:10.217 E ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-nn4m5 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 container=prometheus-adapter container exited with code 2 (Error): I1107 01:44:39.617894       1 adapter.go:93] successfully using in-cluster auth\nI1107 01:44:41.923893       1 secure_serving.go:116] Serving securely on [::]:6443\n
Nov 07 02:03:10.235 E ns/openshift-monitoring pod/kube-state-metrics-c6cdd9b44-ggpmk node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 container=kube-state-metrics container exited with code 2 (Error): 
Nov 07 02:03:10.275 E ns/openshift-marketplace pod/community-operators-9567c47bd-5jjp8 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 container=community-operators container exited with code 2 (Error): 
Nov 07 02:03:10.297 E ns/openshift-monitoring pod/alertmanager-main-2 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 container=config-reloader container exited with code 2 (Error): 2019/11/07 01:46:41 Watching directory: "/etc/alertmanager/config"\n
Nov 07 02:03:10.297 E ns/openshift-monitoring pod/alertmanager-main-2 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 container=alertmanager-proxy container exited with code 2 (Error): 2019/11/07 01:46:41 provider.go:117: Defaulting client-id to system:serviceaccount:openshift-monitoring:alertmanager-main\n2019/11/07 01:46:41 provider.go:122: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token\n2019/11/07 01:46:41 provider.go:310: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.\n2019/11/07 01:46:41 oauthproxy.go:200: mapping path "/" => upstream "http://localhost:9093/"\n2019/11/07 01:46:41 oauthproxy.go:221: compiled skip-auth-regex => "^/metrics"\n2019/11/07 01:46:41 oauthproxy.go:227: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:alertmanager-main\n2019/11/07 01:46:41 oauthproxy.go:237: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> refresh:disabled\n2019/11/07 01:46:41 http.go:96: HTTPS: listening on [::]:9095\n
Nov 07 02:03:10.339 E ns/openshift-machine-config-operator pod/machine-config-daemon-f2pd8 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 container=oauth-proxy container exited with code 143 (Error): 
Nov 07 02:03:10.359 E ns/openshift-ingress pod/router-default-68c6467955-x4lmx node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 container=router container exited with code 2 (Error): Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T01:50:57.386Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T01:51:02.371Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T01:51:11.041Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T01:51:16.068Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T01:52:19.090Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T01:52:28.458Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T01:52:52.486Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T01:53:31.151Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T01:59:05.297Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T01:59:10.269Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T02:03:04.005Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n
Nov 07 02:11:41.583 E clusteroperator/dns changed Degraded to True: NotAllDNSesAvailable: Not all desired DNS DaemonSets available
Nov 07 02:13:24.638 E ns/openshift-marketplace pod/certified-operators-69589995bf-9l67b node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 container=certified-operators container exited with code 2 (Error): 
Nov 07 02:13:24.660 E ns/openshift-monitoring pod/telemeter-client-55f55df45-lnv6w node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 container=reload container exited with code 2 (Error): 
Nov 07 02:13:24.660 E ns/openshift-monitoring pod/telemeter-client-55f55df45-lnv6w node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 container=telemeter-client container exited with code 2 (OOMKilled): 
Nov 07 02:13:24.707 E ns/openshift-ingress pod/router-default-68c6467955-pc5mb node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 container=router container exited with code 2 (Error): Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T02:11:14.967Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T02:11:19.961Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T02:11:29.303Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T02:11:34.933Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T02:11:47.860Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T02:12:09.282Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T02:12:14.004Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T02:12:22.759Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T02:12:28.202Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T02:12:35.837Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T02:13:20.237Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n
Nov 07 02:13:25.574 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 container=rules-configmap-reloader container exited with code 2 (Error): 2019/11/07 01:50:58 Watching directory: "/etc/prometheus/rules/prometheus-k8s-rulefiles-0"\n2019/11/07 01:51:08 config map updated\n2019/11/07 01:51:08 error: Post http://localhost:9090/-/reload: dial tcp [::1]:9090: connect: connection refused\n
Nov 07 02:13:25.574 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 container=prometheus-proxy container exited with code 2 (Error): 2019/11/07 01:51:04 provider.go:117: Defaulting client-id to system:serviceaccount:openshift-monitoring:prometheus-k8s\n2019/11/07 01:51:04 provider.go:122: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token\n2019/11/07 01:51:04 provider.go:310: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.\n2019/11/07 01:51:04 oauthproxy.go:200: mapping path "/" => upstream "http://localhost:9090/"\n2019/11/07 01:51:04 oauthproxy.go:221: compiled skip-auth-regex => "^/metrics"\n2019/11/07 01:51:04 oauthproxy.go:227: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:prometheus-k8s\n2019/11/07 01:51:04 oauthproxy.go:237: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> refresh:disabled\n2019/11/07 01:51:04 main.go:154: using htpasswd file /etc/proxy/htpasswd/auth\n2019/11/07 01:51:04 http.go:96: HTTPS: listening on [::]:9091\n
Nov 07 02:13:25.574 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 container=prometheus-config-reloader container exited with code 2 (Error): ts=2019-11-07T01:50:58.538241652Z caller=main.go:85 msg="Starting prometheus-config-reloader version '0.34.0'."\nlevel=info ts=2019-11-07T01:50:58.538389853Z caller=reloader.go:127 msg="started watching config file for changes" in=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml\nlevel=error ts=2019-11-07T01:50:58.541028065Z caller=runutil.go:87 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post http://localhost:9090/-/reload: dial tcp [::1]:9090: connect: connection refused"\nlevel=error ts=2019-11-07T01:51:03.5403237Z caller=runutil.go:87 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post http://localhost:9090/-/reload: dial tcp [::1]:9090: connect: connection refused"\nlevel=error ts=2019-11-07T01:51:08.540782441Z caller=runutil.go:87 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post http://localhost:9090/-/reload: dial tcp [::1]:9090: connect: connection refused"\nlevel=info ts=2019-11-07T01:51:13.860263968Z caller=reloader.go:258 msg="Prometheus reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml rule_dirs=\nlevel=info ts=2019-11-07T01:52:35.855807061Z caller=reloader.go:258 msg="Prometheus reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml rule_dirs=\n
Nov 07 02:13:25.583 E ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 container=config-reloader container exited with code 2 (Error): 2019/11/07 01:47:15 Watching directory: "/etc/alertmanager/config"\n
Nov 07 02:13:25.583 E ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 container=alertmanager-proxy container exited with code 2 (Error): 2019/11/07 01:47:15 provider.go:117: Defaulting client-id to system:serviceaccount:openshift-monitoring:alertmanager-main\n2019/11/07 01:47:15 provider.go:122: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token\n2019/11/07 01:47:15 provider.go:310: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.\n2019/11/07 01:47:15 oauthproxy.go:200: mapping path "/" => upstream "http://localhost:9093/"\n2019/11/07 01:47:15 oauthproxy.go:221: compiled skip-auth-regex => "^/metrics"\n2019/11/07 01:47:15 oauthproxy.go:227: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:alertmanager-main\n2019/11/07 01:47:15 oauthproxy.go:237: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> refresh:disabled\n2019/11/07 01:47:15 http.go:96: HTTPS: listening on [::]:9095\n
Nov 07 02:13:46.266 E ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g container=config-reloader container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Nov 07 02:13:46.266 E ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g container=alertmanager container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Nov 07 02:13:46.266 E ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g container=alertmanager-proxy container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Nov 07 02:14:03.495 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj container=prometheus container exited with code 1 (Error): caller=main.go:657 msg="Starting TSDB ..."\nlevel=info ts=2019-11-07T02:13:48.963Z caller=web.go:450 component=web msg="Start listening for connections" address=127.0.0.1:9090\nlevel=info ts=2019-11-07T02:13:48.971Z caller=head.go:514 component=tsdb msg="replaying WAL, this may take awhile"\nlevel=info ts=2019-11-07T02:13:48.971Z caller=head.go:562 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0\nlevel=info ts=2019-11-07T02:13:48.972Z caller=main.go:672 fs_type=XFS_SUPER_MAGIC\nlevel=info ts=2019-11-07T02:13:48.972Z caller=main.go:673 msg="TSDB started"\nlevel=info ts=2019-11-07T02:13:48.972Z caller=main.go:743 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml\nlevel=info ts=2019-11-07T02:13:48.973Z caller=main.go:526 msg="Stopping scrape discovery manager..."\nlevel=info ts=2019-11-07T02:13:48.973Z caller=main.go:540 msg="Stopping notify discovery manager..."\nlevel=info ts=2019-11-07T02:13:48.973Z caller=main.go:562 msg="Stopping scrape manager..."\nlevel=info ts=2019-11-07T02:13:48.973Z caller=main.go:536 msg="Notify discovery manager stopped"\nlevel=info ts=2019-11-07T02:13:48.973Z caller=main.go:522 msg="Scrape discovery manager stopped"\nlevel=info ts=2019-11-07T02:13:48.973Z caller=manager.go:814 component="rule manager" msg="Stopping rule manager..."\nlevel=info ts=2019-11-07T02:13:48.973Z caller=manager.go:820 component="rule manager" msg="Rule manager stopped"\nlevel=info ts=2019-11-07T02:13:48.973Z caller=main.go:556 msg="Scrape manager stopped"\nlevel=info ts=2019-11-07T02:13:48.977Z caller=notifier.go:602 component=notifier msg="Stopping notification manager..."\nlevel=info ts=2019-11-07T02:13:48.977Z caller=main.go:727 msg="Notifier manager stopped"\nlevel=error ts=2019-11-07
Nov 07 02:17:17.884 E ns/openshift-authentication pod/oauth-openshift-6564f675b6-njdhv node/ci-op-pg7mtgss-09f59-ng62w-master-2 container=oauth-openshift container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Nov 07 02:22:00.207 E ns/openshift-machine-config-operator pod/machine-config-daemon-l5rhl node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=oauth-proxy container exited with code 143 (Error): 
Nov 07 02:22:01.186 E ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=config-reloader container exited with code 2 (Error): 2019/11/07 02:15:51 Watching directory: "/etc/alertmanager/config"\n
Nov 07 02:22:01.186 E ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=alertmanager-proxy container exited with code 2 (Error): 2019/11/07 02:15:58 provider.go:117: Defaulting client-id to system:serviceaccount:openshift-monitoring:alertmanager-main\n2019/11/07 02:15:58 provider.go:122: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token\n2019/11/07 02:15:58 provider.go:310: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.\n2019/11/07 02:15:58 oauthproxy.go:200: mapping path "/" => upstream "http://localhost:9093/"\n2019/11/07 02:15:58 oauthproxy.go:221: compiled skip-auth-regex => "^/metrics"\n2019/11/07 02:15:58 oauthproxy.go:227: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:alertmanager-main\n2019/11/07 02:15:58 oauthproxy.go:237: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> refresh:disabled\n2019/11/07 02:15:58 http.go:96: HTTPS: listening on [::]:9095\n
Nov 07 02:22:01.271 E ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-8z6z2 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=openshift-state-metrics container exited with code 2 (Error): 
Nov 07 02:23:20.797 E ns/openshift-dns pod/dns-default-s9d7l node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=dns container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Nov 07 02:23:20.797 E ns/openshift-dns pod/dns-default-s9d7l node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=dns-node-resolver container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Nov 07 02:27:56.707 E ns/openshift-machine-config-operator pod/machine-config-daemon-6hnt5 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=oauth-proxy container exited with code 143 (Error): 
Nov 07 02:30:58.538 E ns/openshift-machine-config-operator pod/machine-config-daemon-hjjm4 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=machine-config-daemon container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Nov 07 02:30:58.538 E ns/openshift-machine-config-operator pod/machine-config-daemon-hjjm4 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=oauth-proxy container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Nov 07 02:42:44.825 E ns/openshift-marketplace pod/community-operators-9567c47bd-2thsq node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj container=community-operators container exited with code 2 (Error): 
Nov 07 02:54:02.547 E ns/openshift-marketplace pod/community-operators-7857bd4d58-rfnpn node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=community-operators container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated

				

Timeline:

Nov 07 01:52:32.934 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (3 times)
Nov 07 01:52:32.955 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-000" not found) (2 times)
Nov 07 01:52:32.959 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (2 times)
Nov 07 01:52:32.959 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-001" not found) (2 times)
Nov 07 01:52:32.966 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-0 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (2 times)
Nov 07 01:52:32.969 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-000" not found) (3 times)
Nov 07 01:52:32.979 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-0 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-000" not found) (2 times)
Nov 07 01:52:32.992 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-0 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-001" not found) (2 times)
Nov 07 01:52:33.000 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-001" not found) (4 times)
Nov 07 01:52:33.325 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-002" not found) (2 times)
Nov 07 01:52:33.345 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-0 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-002" not found) (3 times)
Nov 07 01:52:33.345 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-002" not found) (2 times)
Nov 07 01:52:33.728 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-003" not found) (2 times)
Nov 07 01:52:33.739 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-003" not found) (2 times)
Nov 07 01:52:33.740 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-0 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-003" not found) (2 times)
Nov 07 01:52:34.130 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-004" not found) (2 times)
Nov 07 01:52:34.136 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-004" not found) (2 times)
Nov 07 01:52:34.142 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-0 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-004" not found) (2 times)
Nov 07 01:52:34.526 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-005" not found) (4 times)
Nov 07 01:52:34.543 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-005" not found) (2 times)
Nov 07 01:52:34.552 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-0 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-005" not found) (2 times)
Nov 07 01:52:34.726 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-006" not found) (2 times)
Nov 07 01:52:34.744 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-006" not found) (2 times)
Nov 07 01:52:34.744 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-0 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-006" not found) (2 times)
Nov 07 01:52:34.926 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-007" not found) (5 times)
Nov 07 01:52:34.943 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-0 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-007" not found) (2 times)
Nov 07 01:52:34.943 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-007" not found) (5 times)
Nov 07 01:52:35.130 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-008" not found) (6 times)
Nov 07 01:52:35.142 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-0 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-008" not found) (4 times)
Nov 07 01:52:35.142 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-008" not found) (6 times)
Nov 07 01:52:35.326 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-009" not found)
Nov 07 01:52:35.330 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-009" not found)
Nov 07 01:52:35.335 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-0 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-009" not found)
Nov 07 01:52:40.126 I ns/openshift-kube-apiserver pod/revision-pruner-6-ci-op-pg7mtgss-09f59-ng62w-master-0 node/ci-op-pg7mtgss-09f59-ng62w-master-0 created
Nov 07 01:52:40.131 I ns/openshift-kube-apiserver-operator deployment/kube-apiserver-operator Created Pod/revision-pruner-6-ci-op-pg7mtgss-09f59-ng62w-master-0 -n openshift-kube-apiserver because it was missing
Nov 07 01:53:18.053 I ns/openshift-kube-apiserver pod/revision-pruner-6-ci-op-pg7mtgss-09f59-ng62w-master-0 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:19af1c1ce75f5ee43176c23dc17681e33651aa56d1aa0432f5abca9b19afb48d" already present on machine
Nov 07 01:53:18.287 I ns/openshift-kube-apiserver pod/revision-pruner-6-ci-op-pg7mtgss-09f59-ng62w-master-0 Created container pruner
Nov 07 01:53:18.324 I ns/openshift-kube-apiserver pod/revision-pruner-6-ci-op-pg7mtgss-09f59-ng62w-master-0 Started container pruner
Nov 07 01:53:49.902 I ns/openshift-ingress service/router-default Ensuring load balancer (7 times)
Nov 07 01:53:50.340 W ns/openshift-ingress service/router-default Error syncing load balancer: failed to ensure load balancer: ensure(openshift-ingress/router-default): backendPoolID(/subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-pg7mtgss-09f59-ng62w-rg/providers/Microsoft.Network/loadBalancers/ci-op-pg7mtgss-09f59-ng62w/backendAddressPools/ci-op-pg7mtgss-09f59-ng62w) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:NicGet" (4 times)
Nov 07 01:58:04.825 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (2 times)
Nov 07 01:58:04.833 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-000" not found) (3 times)
Nov 07 01:58:04.841 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-001" not found) (3 times)
Nov 07 01:58:05.213 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-002" not found) (7 times)
Nov 07 01:58:05.611 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-003" not found) (3 times)
Nov 07 01:58:50.365 I ns/openshift-ingress service/router-default Ensuring load balancer (8 times)
Nov 07 01:58:50.897 W ns/openshift-ingress service/router-default Error syncing load balancer: failed to ensure load balancer: ensure(openshift-ingress/router-default): backendPoolID(/subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-pg7mtgss-09f59-ng62w-rg/providers/Microsoft.Network/loadBalancers/ci-op-pg7mtgss-09f59-ng62w/backendAddressPools/ci-op-pg7mtgss-09f59-ng62w) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:NicGet" (5 times)
Nov 07 01:59:26.234 I ns/kube-system pod/pod0-system-node-critical node/ created
Nov 07 01:59:26.261 I ns/kube-system pod/pod0-system-node-critical Successfully assigned kube-system/pod0-system-node-critical to ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9
Nov 07 01:59:26.284 I ns/kube-system pod/pod1-system-cluster-critical node/ created
Nov 07 01:59:26.300 I ns/kube-system pod/pod1-system-cluster-critical Successfully assigned kube-system/pod1-system-cluster-critical to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8
Nov 07 01:59:26.335 W ns/kube-system pod/pod1-system-cluster-critical node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 graceful deletion within 0s
Nov 07 01:59:26.344 W ns/kube-system pod/pod1-system-cluster-critical node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 deleted
Nov 07 01:59:26.393 W ns/kube-system pod/pod0-system-node-critical node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 graceful deletion within 0s
Nov 07 01:59:26.402 W ns/kube-system pod/pod0-system-node-critical node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 deleted
Nov 07 02:00:04.625 W ns/openshift-sdn pod/ovs-mhlb5 Liveness probe failed: ovs-ofctl: /var/run/openvswitch/br0.mgmt: failed to open socket (Broken pipe)\n
Nov 07 02:00:26.211 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (7 times)
Nov 07 02:00:26.220 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-000" not found) (2 times)
Nov 07 02:00:26.230 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-001" not found) (2 times)
Nov 07 02:00:26.587 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-002" not found) (3 times)
Nov 07 02:00:26.988 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-003" not found) (8 times)
Nov 07 02:01:29.320 W ns/kube-system pod/pod0-system-node-critical Unable to attach or mount volumes: unmounted volumes=[default-token-j6v5c], unattached volumes=[default-token-j6v5c]: timed out waiting for the condition
Nov 07 02:01:29.396 W ns/kube-system pod/pod1-system-cluster-critical Unable to attach or mount volumes: unmounted volumes=[default-token-j6v5c], unattached volumes=[default-token-j6v5c]: timed out waiting for the condition
Nov 07 02:02:17.399 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-0 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (3 times)
Nov 07 02:02:17.411 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-0 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-000" not found) (3 times)
Nov 07 02:02:17.420 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-0 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-001" not found) (5 times)
Nov 07 02:02:17.791 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-0 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-002" not found) (6 times)
Nov 07 02:02:18.191 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-0 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-003" not found) (3 times)
Nov 07 02:03:03.758 I ns/openshift-monitoring pod/kube-state-metrics-c6cdd9b44-ggpmk Marking for deletion Pod openshift-monitoring/kube-state-metrics-c6cdd9b44-ggpmk
Nov 07 02:03:03.768 W ns/openshift-dns pod/dns-default-r4cfg node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 graceful deletion within 30s
Nov 07 02:03:03.783 I ns/openshift-monitoring pod/alertmanager-main-2 Marking for deletion Pod openshift-monitoring/alertmanager-main-2
Nov 07 02:03:03.783 W ns/openshift-marketplace pod/redhat-operators-64566c9856-wzdq5 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 graceful deletion within 30s
Nov 07 02:03:03.803 W ns/openshift-marketplace pod/certified-operators-69589995bf-lgmd5 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 graceful deletion within 30s
Nov 07 02:03:03.804 I ns/openshift-dns daemonset/dns-default Deleted pod: dns-default-r4cfg
Nov 07 02:03:03.804 I ns/openshift-ingress pod/router-default-68c6467955-x4lmx Marking for deletion Pod openshift-ingress/router-default-68c6467955-x4lmx
Nov 07 02:03:03.805 I ns/openshift-dns pod/dns-default-r4cfg Stopping container dns-node-resolver
Nov 07 02:03:03.806 W ns/openshift-monitoring pod/kube-state-metrics-c6cdd9b44-ggpmk node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 graceful deletion within 30s
Nov 07 02:03:03.806 W ns/openshift-machine-config-operator pod/machine-config-daemon-f2pd8 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 graceful deletion within 600s
Nov 07 02:03:03.815 W ns/openshift-marketplace pod/community-operators-9567c47bd-5jjp8 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 graceful deletion within 30s
Nov 07 02:03:03.821 W ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-pl992 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 graceful deletion within 30s
Nov 07 02:03:03.821 W ns/openshift-image-registry pod/node-ca-5dbw6 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 graceful deletion within 30s
Nov 07 02:03:03.841 W ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-nn4m5 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 graceful deletion within 30s
Nov 07 02:03:03.842 W ns/openshift-image-registry pod/image-registry-66fc6c486-kgz42 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 graceful deletion within 30s
Nov 07 02:03:03.843 I ns/openshift-image-registry daemonset/node-ca Deleted pod: node-ca-5dbw6
Nov 07 02:03:03.843 I ns/openshift-image-registry pod/image-registry-66fc6c486-kgz42 Marking for deletion Pod openshift-image-registry/image-registry-66fc6c486-kgz42
Nov 07 02:03:03.843 W ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-txgdb node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 graceful deletion within 30s
Nov 07 02:03:03.843 I ns/openshift-dns pod/dns-default-r4cfg Stopping container dns
Nov 07 02:03:03.844 W ns/openshift-ingress pod/router-default-68c6467955-x4lmx node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 graceful deletion within 30s
Nov 07 02:03:03.845 W ns/openshift-monitoring pod/alertmanager-main-2 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 graceful deletion within 120s
Nov 07 02:03:03.861 I ns/openshift-machine-config-operator daemonset/machine-config-daemon Deleted pod: machine-config-daemon-f2pd8
Nov 07 02:03:03.867 I ns/openshift-marketplace pod/redhat-operators-64566c9856-wzdq5 Marking for deletion Pod openshift-marketplace/redhat-operators-64566c9856-wzdq5
Nov 07 02:03:03.875 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-nn4m5 Marking for deletion Pod openshift-monitoring/prometheus-adapter-6fb7598968-nn4m5
Nov 07 02:03:03.889 I ns/openshift-marketplace pod/redhat-operators-64566c9856-wzdq5 Stopping container redhat-operators
Nov 07 02:03:03.894 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-txgdb Marking for deletion Pod openshift-monitoring/prometheus-adapter-6fb7598968-txgdb
Nov 07 02:03:03.897 I ns/openshift-marketplace pod/redhat-operators-64566c9856-gjdpn node/ created
Nov 07 02:03:03.915 I ns/openshift-image-registry pod/node-ca-5dbw6 Marking for deletion Pod openshift-image-registry/node-ca-5dbw6
Nov 07 02:03:03.926 I ns/openshift-marketplace replicaset/redhat-operators-64566c9856 Created pod: redhat-operators-64566c9856-gjdpn
Nov 07 02:03:03.927 I ns/openshift-marketplace pod/community-operators-9567c47bd-z66zz node/ created
Nov 07 02:03:03.935 I ns/openshift-dns pod/dns-default-r4cfg Marking for deletion Pod openshift-dns/dns-default-r4cfg
Nov 07 02:03:03.937 I ns/openshift-marketplace pod/certified-operators-69589995bf-9l67b node/ created
Nov 07 02:03:03.952 I ns/openshift-marketplace replicaset/community-operators-9567c47bd Created pod: community-operators-9567c47bd-z66zz
Nov 07 02:03:03.959 I ns/openshift-machine-config-operator pod/machine-config-daemon-f2pd8 Marking for deletion Pod openshift-machine-config-operator/machine-config-daemon-f2pd8
Nov 07 02:03:03.968 I ns/openshift-marketplace replicaset/certified-operators-69589995bf Created pod: certified-operators-69589995bf-9l67b
Nov 07 02:03:03.989 I ns/openshift-marketplace pod/community-operators-9567c47bd-z66zz Successfully assigned openshift-marketplace/community-operators-9567c47bd-z66zz to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8
Nov 07 02:03:03.989 I ns/openshift-marketplace pod/certified-operators-69589995bf-9l67b Successfully assigned openshift-marketplace/certified-operators-69589995bf-9l67b to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8
Nov 07 02:03:03.989 I ns/openshift-marketplace pod/redhat-operators-64566c9856-gjdpn Successfully assigned openshift-marketplace/redhat-operators-64566c9856-gjdpn to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8
Nov 07 02:03:04.001 I ns/openshift-monitoring pod/kube-state-metrics-c6cdd9b44-djcp6 node/ created
Nov 07 02:03:04.009 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-pl992 Marking for deletion Pod openshift-monitoring/openshift-state-metrics-6c78647cc7-pl992
Nov 07 02:03:04.026 I ns/openshift-marketplace pod/certified-operators-69589995bf-lgmd5 Stopping container certified-operators
Nov 07 02:03:04.026 I ns/openshift-monitoring replicaset/kube-state-metrics-c6cdd9b44 Created pod: kube-state-metrics-c6cdd9b44-djcp6
Nov 07 02:03:04.027 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-lwfzk node/ created
Nov 07 02:03:04.034 I ns/openshift-marketplace pod/community-operators-9567c47bd-5jjp8 Marking for deletion Pod openshift-marketplace/community-operators-9567c47bd-5jjp8
Nov 07 02:03:04.034 I ns/openshift-monitoring pod/kube-state-metrics-c6cdd9b44-ggpmk Stopping container kube-state-metrics
Nov 07 02:03:04.049 I ns/openshift-monitoring replicaset/openshift-state-metrics-6c78647cc7 Created pod: openshift-state-metrics-6c78647cc7-lwfzk
Nov 07 02:03:04.053 I ns/openshift-monitoring pod/kube-state-metrics-c6cdd9b44-djcp6 Successfully assigned openshift-monitoring/kube-state-metrics-c6cdd9b44-djcp6 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-z8fpv
Nov 07 02:03:04.054 I ns/openshift-marketplace pod/certified-operators-69589995bf-lgmd5 Marking for deletion Pod openshift-marketplace/certified-operators-69589995bf-lgmd5
Nov 07 02:03:04.088 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-txbbz node/ created
Nov 07 02:03:04.101 I ns/openshift-monitoring pod/kube-state-metrics-c6cdd9b44-ggpmk Stopping container kube-rbac-proxy-self
Nov 07 02:03:04.101 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-lwfzk Successfully assigned openshift-monitoring/openshift-state-metrics-6c78647cc7-lwfzk to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8
Nov 07 02:03:04.115 I ns/openshift-monitoring replicaset/prometheus-adapter-6fb7598968 Created pod: prometheus-adapter-6fb7598968-txbbz
Nov 07 02:03:04.139 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-txbbz Successfully assigned openshift-monitoring/prometheus-adapter-6fb7598968-txbbz to ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-z8fpv
Nov 07 02:03:04.147 I ns/openshift-image-registry pod/image-registry-66fc6c486-r59gt node/ created
Nov 07 02:03:04.160 I ns/openshift-monitoring pod/kube-state-metrics-c6cdd9b44-ggpmk Stopping container kube-rbac-proxy-main
Nov 07 02:03:04.168 I ns/openshift-image-registry replicaset/image-registry-66fc6c486 Created pod: image-registry-66fc6c486-r59gt
Nov 07 02:03:04.172 I ns/openshift-ingress pod/router-default-68c6467955-zg9m6 node/ created
Nov 07 02:03:04.194 I ns/openshift-ingress replicaset/router-default-68c6467955 Created pod: router-default-68c6467955-zg9m6
Nov 07 02:03:04.205 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-vsdf8 node/ created
Nov 07 02:03:04.224 I ns/openshift-monitoring replicaset/prometheus-adapter-6fb7598968 Created pod: prometheus-adapter-6fb7598968-vsdf8
Nov 07 02:03:04.252 I ns/openshift-image-registry pod/image-registry-66fc6c486-r59gt Successfully assigned openshift-image-registry/image-registry-66fc6c486-r59gt to ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-z8fpv
Nov 07 02:03:04.258 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-vsdf8 Successfully assigned openshift-monitoring/prometheus-adapter-6fb7598968-vsdf8 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8
Nov 07 02:03:04.262 I ns/openshift-machine-config-operator pod/machine-config-daemon-f2pd8 Stopping container machine-config-daemon
Nov 07 02:03:04.266 I ns/openshift-ingress pod/router-default-68c6467955-zg9m6 Successfully assigned openshift-ingress/router-default-68c6467955-zg9m6 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-z8fpv
Nov 07 02:03:04.301 I ns/openshift-machine-config-operator pod/machine-config-daemon-f2pd8 Stopping container oauth-proxy
Nov 07 02:03:04.312 I ns/openshift-marketplace pod/community-operators-9567c47bd-5jjp8 Stopping container community-operators
Nov 07 02:03:04.323 I ns/openshift-monitoring pod/alertmanager-main-2 Stopping container alertmanager-proxy
Nov 07 02:03:04.335 I ns/openshift-monitoring pod/alertmanager-main-2 Stopping container config-reloader
Nov 07 02:03:04.387 I ns/openshift-monitoring pod/alertmanager-main-2 Stopping container alertmanager
Nov 07 02:03:04.423 W clusteroperator/ingress changed Available to False: IngressUnavailable: Not all ingress controllers are available.
Nov 07 02:03:04.423 W clusteroperator/ingress changed Progressing to True: Reconciling: Not all ingress controllers are available.
Nov 07 02:03:04.592 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-txgdb Stopping container prometheus-adapter
Nov 07 02:03:04.787 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-nn4m5 Stopping container prometheus-adapter
Nov 07 02:03:04.985 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-pl992 Stopping container openshift-state-metrics
Nov 07 02:03:05.182 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-pl992 Stopping container kube-rbac-proxy-self
Nov 07 02:03:05.379 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-pl992 Stopping container kube-rbac-proxy-main
Nov 07 02:03:05.585 I ns/openshift-ingress pod/router-default-68c6467955-x4lmx Stopping container router
Nov 07 02:03:05.786 I ns/openshift-image-registry pod/image-registry-66fc6c486-kgz42 Stopping container registry
Nov 07 02:03:05.984 I ns/openshift-image-registry pod/node-ca-5dbw6 Stopping container node-ca
Nov 07 02:03:06.180 W ns/openshift-image-registry pod/image-registry-66fc6c486-kgz42 Liveness probe failed: Get https://10.131.0.9:5000/healthz: dial tcp 10.131.0.9:5000: connect: connection refused
Nov 07 02:03:06.380 W ns/openshift-marketplace pod/redhat-operators-64566c9856-wzdq5 Readiness probe errored: rpc error: code = Unknown desc = container is not created or running
Nov 07 02:03:06.585 W ns/openshift-marketplace pod/community-operators-9567c47bd-5jjp8 Readiness probe errored: rpc error: code = Unknown desc = container is not created or running
Nov 07 02:03:07.312 W ns/openshift-image-registry pod/image-registry-66fc6c486-kgz42 Readiness probe failed: Get https://10.131.0.9:5000/healthz: dial tcp 10.131.0.9:5000: connect: connection refused
Nov 07 02:03:09.064 E ns/openshift-marketplace pod/redhat-operators-64566c9856-wzdq5 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 container=redhat-operators container exited with code 2 (Error): 
Nov 07 02:03:09.090 W ns/openshift-dns pod/dns-default-r4cfg node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:03:09.090 W ns/openshift-dns pod/dns-default-r4cfg node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 container=dns-node-resolver container stopped being ready
Nov 07 02:03:09.090 W ns/openshift-dns pod/dns-default-r4cfg node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 container=dns container stopped being ready
Nov 07 02:03:09.108 E ns/openshift-marketplace pod/certified-operators-69589995bf-lgmd5 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 container=certified-operators container exited with code 2 (Error): 
Nov 07 02:03:10.203 W ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-pl992 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:03:10.203 W ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-pl992 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 container=openshift-state-metrics container stopped being ready
Nov 07 02:03:10.203 W ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-pl992 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 container=kube-rbac-proxy-self container stopped being ready
Nov 07 02:03:10.217 E ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-nn4m5 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 container=prometheus-adapter container exited with code 2 (Error): I1107 01:44:39.617894       1 adapter.go:93] successfully using in-cluster auth\nI1107 01:44:41.923893       1 secure_serving.go:116] Serving securely on [::]:6443\n
Nov 07 02:03:10.235 E ns/openshift-monitoring pod/kube-state-metrics-c6cdd9b44-ggpmk node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 container=kube-state-metrics container exited with code 2 (Error): 
Nov 07 02:03:10.257 W ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-txgdb node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:03:10.257 W ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-txgdb node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 container=prometheus-adapter container stopped being ready
Nov 07 02:03:10.275 E ns/openshift-marketplace pod/community-operators-9567c47bd-5jjp8 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 container=community-operators container exited with code 2 (Error): 
Nov 07 02:03:10.297 E ns/openshift-monitoring pod/alertmanager-main-2 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 container=config-reloader container exited with code 2 (Error): 2019/11/07 01:46:41 Watching directory: "/etc/alertmanager/config"\n
Nov 07 02:03:10.297 E ns/openshift-monitoring pod/alertmanager-main-2 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 container=alertmanager-proxy container exited with code 2 (Error): 2019/11/07 01:46:41 provider.go:117: Defaulting client-id to system:serviceaccount:openshift-monitoring:alertmanager-main\n2019/11/07 01:46:41 provider.go:122: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token\n2019/11/07 01:46:41 provider.go:310: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.\n2019/11/07 01:46:41 oauthproxy.go:200: mapping path "/" => upstream "http://localhost:9093/"\n2019/11/07 01:46:41 oauthproxy.go:221: compiled skip-auth-regex => "^/metrics"\n2019/11/07 01:46:41 oauthproxy.go:227: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:alertmanager-main\n2019/11/07 01:46:41 oauthproxy.go:237: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> refresh:disabled\n2019/11/07 01:46:41 http.go:96: HTTPS: listening on [::]:9095\n
Nov 07 02:03:10.339 E ns/openshift-machine-config-operator pod/machine-config-daemon-f2pd8 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 container=oauth-proxy container exited with code 143 (Error): 
Nov 07 02:03:10.359 E ns/openshift-ingress pod/router-default-68c6467955-x4lmx node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 container=router container exited with code 2 (Error): Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T01:50:57.386Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T01:51:02.371Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T01:51:11.041Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T01:51:16.068Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T01:52:19.090Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T01:52:28.458Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T01:52:52.486Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T01:53:31.151Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T01:59:05.297Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T01:59:10.269Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T02:03:04.005Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n
Nov 07 02:03:11.981 I ns/openshift-marketplace pod/redhat-operators-64566c9856-gjdpn Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:448858c7b1f70b8c0955965ef9970b59504b220bf03b760203305c81e5f606da"
Nov 07 02:03:12.165 W ns/openshift-dns pod/dns-default-r4cfg node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 deleted
Nov 07 02:03:12.201 W ns/openshift-marketplace pod/certified-operators-69589995bf-lgmd5 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 deleted
Nov 07 02:03:12.235 W ns/openshift-marketplace pod/redhat-operators-64566c9856-wzdq5 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 deleted
Nov 07 02:03:12.651 I ns/openshift-marketplace pod/community-operators-9567c47bd-z66zz Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:448858c7b1f70b8c0955965ef9970b59504b220bf03b760203305c81e5f606da"
Nov 07 02:03:12.703 I ns/openshift-image-registry pod/image-registry-66fc6c486-r59gt Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3191fc23d0d7e820100498c5085592219a747743f06fc059436945df6ab3e8ee" already present on machine
Nov 07 02:03:12.751 I ns/openshift-marketplace pod/certified-operators-69589995bf-9l67b Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:448858c7b1f70b8c0955965ef9970b59504b220bf03b760203305c81e5f606da"
Nov 07 02:03:12.835 I ns/openshift-monitoring pod/kube-state-metrics-c6cdd9b44-djcp6 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5366da9d7c15f912b609cdbb2851d01ea863fb22d452081914d923d8208c39c8" already present on machine
Nov 07 02:03:12.945 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-vsdf8 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4753bca36792206b5e9df9612f695fc2c3a80a54d56b01a10f774dc1b62899be"
Nov 07 02:03:12.965 I ns/openshift-image-registry pod/image-registry-66fc6c486-r59gt Created container registry
Nov 07 02:03:13.000 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-lwfzk Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5366da9d7c15f912b609cdbb2851d01ea863fb22d452081914d923d8208c39c8" already present on machine
Nov 07 02:03:13.026 I ns/openshift-image-registry pod/image-registry-66fc6c486-r59gt Started container registry
Nov 07 02:03:13.149 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-txbbz Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4753bca36792206b5e9df9612f695fc2c3a80a54d56b01a10f774dc1b62899be"
Nov 07 02:03:13.192 I ns/openshift-monitoring pod/kube-state-metrics-c6cdd9b44-djcp6 Created container kube-rbac-proxy-main
Nov 07 02:03:13.290 I ns/openshift-monitoring pod/kube-state-metrics-c6cdd9b44-djcp6 Started container kube-rbac-proxy-main
Nov 07 02:03:13.298 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-lwfzk Created container kube-rbac-proxy-main
Nov 07 02:03:13.299 I ns/openshift-monitoring pod/kube-state-metrics-c6cdd9b44-djcp6 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5366da9d7c15f912b609cdbb2851d01ea863fb22d452081914d923d8208c39c8" already present on machine
Nov 07 02:03:13.340 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-lwfzk Started container kube-rbac-proxy-main
Nov 07 02:03:13.350 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-lwfzk Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5366da9d7c15f912b609cdbb2851d01ea863fb22d452081914d923d8208c39c8" already present on machine
Nov 07 02:03:13.367 I ns/openshift-ingress pod/router-default-68c6467955-zg9m6 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ece12ae9ebfcce0c5b8d8a2eff009a1706a070e9021e96112dbbe6711795a35"
Nov 07 02:03:13.651 I ns/openshift-monitoring pod/kube-state-metrics-c6cdd9b44-djcp6 Created container kube-rbac-proxy-self
Nov 07 02:03:13.698 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-lwfzk Created container kube-rbac-proxy-self
Nov 07 02:03:13.737 I ns/openshift-monitoring pod/kube-state-metrics-c6cdd9b44-djcp6 Started container kube-rbac-proxy-self
Nov 07 02:03:13.751 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-lwfzk Started container kube-rbac-proxy-self
Nov 07 02:03:13.751 I ns/openshift-monitoring pod/kube-state-metrics-c6cdd9b44-djcp6 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4842a43d24fb9658a83ab8accad8cf1b8fb07141124517a0ec39a09637a5d186"
Nov 07 02:03:13.759 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-lwfzk Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9b0c04ed78fe07c8c7ba6efec2be18908955710281189051eaea6942d424a56f"
Nov 07 02:03:16.142 W ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-pl992 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 pod has been pending longer than a minute
Nov 07 02:03:16.142 W ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-txgdb node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 pod has been pending longer than a minute
Nov 07 02:03:20.607 I ns/openshift-monitoring pod/kube-state-metrics-c6cdd9b44-djcp6 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4842a43d24fb9658a83ab8accad8cf1b8fb07141124517a0ec39a09637a5d186"
Nov 07 02:03:20.724 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-txbbz Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4753bca36792206b5e9df9612f695fc2c3a80a54d56b01a10f774dc1b62899be"
Nov 07 02:03:20.770 I ns/openshift-ingress pod/router-default-68c6467955-zg9m6 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ece12ae9ebfcce0c5b8d8a2eff009a1706a070e9021e96112dbbe6711795a35"
Nov 07 02:03:21.014 I ns/openshift-monitoring pod/kube-state-metrics-c6cdd9b44-djcp6 Created container kube-state-metrics
Nov 07 02:03:21.158 I ns/openshift-monitoring pod/kube-state-metrics-c6cdd9b44-djcp6 Started container kube-state-metrics
Nov 07 02:03:21.309 I ns/openshift-ingress pod/router-default-68c6467955-zg9m6 Created container router
Nov 07 02:03:21.377 I ns/openshift-ingress pod/router-default-68c6467955-zg9m6 Started container router
Nov 07 02:03:21.459 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-txbbz Created container prometheus-adapter
Nov 07 02:03:21.519 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-txbbz Started container prometheus-adapter
Nov 07 02:03:22.107 W ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-pl992 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 deleted
Nov 07 02:03:22.117 W ns/openshift-monitoring pod/alertmanager-main-2 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 deleted
Nov 07 02:03:22.153 W ns/openshift-machine-config-operator pod/machine-config-daemon-f2pd8 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 deleted
Nov 07 02:03:22.204 I ns/openshift-monitoring pod/alertmanager-main-2 node/ created
Nov 07 02:03:22.228 W ns/openshift-image-registry pod/node-ca-5dbw6 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 deleted
Nov 07 02:03:22.233 I ns/openshift-monitoring statefulset/alertmanager-main create Pod alertmanager-main-2 in StatefulSet alertmanager-main successful
Nov 07 02:03:22.246 I ns/openshift-monitoring pod/alertmanager-main-2 Successfully assigned openshift-monitoring/alertmanager-main-2 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-z8fpv
Nov 07 02:03:22.246 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-vsdf8 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4753bca36792206b5e9df9612f695fc2c3a80a54d56b01a10f774dc1b62899be"
Nov 07 02:03:22.270 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-lwfzk Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9b0c04ed78fe07c8c7ba6efec2be18908955710281189051eaea6942d424a56f"
Nov 07 02:03:22.286 W ns/openshift-marketplace pod/community-operators-9567c47bd-5jjp8 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 deleted
Nov 07 02:03:22.324 W ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-txgdb node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 deleted
Nov 07 02:03:22.360 W ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-nn4m5 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 deleted
Nov 07 02:03:22.408 W ns/openshift-ingress pod/router-default-68c6467955-x4lmx node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 deleted
Nov 07 02:03:22.464 W ns/openshift-monitoring pod/kube-state-metrics-c6cdd9b44-ggpmk node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 deleted
Nov 07 02:03:22.519 W ns/openshift-image-registry pod/image-registry-66fc6c486-kgz42 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 deleted
Nov 07 02:03:23.801 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-vsdf8 Created container prometheus-adapter
Nov 07 02:03:24.282 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-vsdf8 Started container prometheus-adapter
Nov 07 02:03:24.501 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-lwfzk Created container openshift-state-metrics
Nov 07 02:03:24.723 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-lwfzk Started container openshift-state-metrics
Nov 07 02:03:30.679 I ns/openshift-monitoring pod/alertmanager-main-2 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78ba454f09557364dabd918265784493287ed242fb51ac2ffde695ae78823736" already present on machine
Nov 07 02:03:30.926 I ns/openshift-monitoring pod/alertmanager-main-2 Created container alertmanager
Nov 07 02:03:30.976 I ns/openshift-monitoring pod/alertmanager-main-2 Started container alertmanager
Nov 07 02:03:30.994 I ns/openshift-monitoring pod/alertmanager-main-2 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db10cd1f5efd9e9e6f2b0eea0a665f53399e25de374cf916ad7d8141f8ac643d" already present on machine
Nov 07 02:03:31.520 I ns/openshift-monitoring pod/alertmanager-main-2 Created container config-reloader
Nov 07 02:03:31.557 I ns/openshift-monitoring pod/alertmanager-main-2 Started container config-reloader
Nov 07 02:03:31.565 I ns/openshift-monitoring pod/alertmanager-main-2 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f711682524b64033a4ce9193bd2cd9621e08b4bae44cffffb4e397a43a8f945" already present on machine
Nov 07 02:03:31.804 I ns/openshift-monitoring pod/alertmanager-main-2 Created container alertmanager-proxy
Nov 07 02:03:31.862 I ns/openshift-monitoring pod/alertmanager-main-2 Started container alertmanager-proxy
Nov 07 02:03:37.996 I ns/openshift-marketplace pod/certified-operators-69589995bf-9l67b Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:448858c7b1f70b8c0955965ef9970b59504b220bf03b760203305c81e5f606da"
Nov 07 02:03:38.021 I ns/openshift-marketplace pod/redhat-operators-64566c9856-gjdpn Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:448858c7b1f70b8c0955965ef9970b59504b220bf03b760203305c81e5f606da"
Nov 07 02:03:38.032 I ns/openshift-marketplace pod/community-operators-9567c47bd-z66zz Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:448858c7b1f70b8c0955965ef9970b59504b220bf03b760203305c81e5f606da"
Nov 07 02:03:38.568 I ns/openshift-marketplace pod/certified-operators-69589995bf-9l67b Created container certified-operators
Nov 07 02:03:38.610 W clusteroperator/ingress changed Available to True: desired and current number of IngressControllers are equal
Nov 07 02:03:38.610 W clusteroperator/ingress changed Progressing to False: desired and current number of IngressControllers are equal
Nov 07 02:03:38.656 I ns/openshift-marketplace pod/certified-operators-69589995bf-9l67b Started container certified-operators
Nov 07 02:03:39.252 I ns/openshift-marketplace pod/community-operators-9567c47bd-z66zz Created container community-operators
Nov 07 02:03:39.270 I ns/openshift-marketplace pod/redhat-operators-64566c9856-gjdpn Created container redhat-operators
Nov 07 02:03:39.366 I ns/openshift-marketplace pod/community-operators-9567c47bd-z66zz Started container community-operators
Nov 07 02:03:39.379 I ns/openshift-marketplace pod/redhat-operators-64566c9856-gjdpn Started container redhat-operators
Nov 07 02:03:45.246 W ns/openshift-marketplace pod/redhat-operators-64566c9856-gjdpn Readiness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n
Nov 07 02:03:46.360 W ns/openshift-marketplace pod/redhat-operators-64566c9856-gjdpn Liveness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n
Nov 07 02:03:48.352 W ns/openshift-marketplace pod/certified-operators-69589995bf-9l67b Readiness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n
Nov 07 02:03:49.598 W ns/openshift-marketplace pod/certified-operators-69589995bf-9l67b Liveness probe errored: rpc error: code = Unknown desc = command error: command timed out, stdout: , stderr: , exit code -1
Nov 07 02:03:50.205 W ns/openshift-marketplace pod/community-operators-9567c47bd-z66zz Readiness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n
Nov 07 02:03:50.912 I ns/openshift-ingress service/router-default Ensuring load balancer (9 times)
Nov 07 02:03:54.256 W ns/openshift-marketplace pod/community-operators-9567c47bd-z66zz Liveness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n
Nov 07 02:03:57.894 W ns/openshift-marketplace pod/certified-operators-69589995bf-9l67b Liveness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n
Nov 07 02:03:58.441 W ns/openshift-marketplace pod/certified-operators-69589995bf-9l67b Readiness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n (2 times)
Nov 07 02:04:00.183 W ns/openshift-marketplace pod/community-operators-9567c47bd-z66zz Readiness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n (2 times)
Nov 07 02:04:04.138 W ns/openshift-marketplace pod/community-operators-9567c47bd-z66zz Liveness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n (2 times)
Nov 07 02:04:08.936 I ns/openshift-machine-config-operator pod/machine-config-daemon-lkzwf node/ created
Nov 07 02:04:08.936 I ns/openshift-machine-config-operator daemonset/machine-config-daemon Created pod: machine-config-daemon-lkzwf
Nov 07 02:04:08.964 I ns/openshift-machine-config-operator pod/machine-config-daemon-lkzwf Successfully assigned openshift-machine-config-operator/machine-config-daemon-lkzwf to ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9
Nov 07 02:04:08.975 I ns/openshift-dns pod/dns-default-c628x node/ created
Nov 07 02:04:08.996 I ns/openshift-dns daemonset/dns-default Created pod: dns-default-c628x
Nov 07 02:04:09.004 I ns/openshift-image-registry pod/node-ca-b6p4w node/ created
Nov 07 02:04:09.054 I ns/openshift-image-registry daemonset/node-ca Created pod: node-ca-b6p4w
Nov 07 02:04:09.075 I ns/openshift-dns pod/dns-default-c628x Successfully assigned openshift-dns/dns-default-c628x to ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9
Nov 07 02:04:09.093 I ns/openshift-image-registry pod/node-ca-b6p4w Successfully assigned openshift-image-registry/node-ca-b6p4w to ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9
Nov 07 02:04:09.171 W clusteroperator/dns changed Progressing to True: Reconciling: Not all DNS DaemonSets available.
Nov 07 02:04:10.437 I ns/openshift-machine-config-operator pod/machine-config-daemon-lkzwf Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:944d643d229b0a1028464df12f2a544163a6689947df774c6568b3f443a79cb8" already present on machine
Nov 07 02:04:10.627 I ns/openshift-machine-config-operator pod/machine-config-daemon-lkzwf Created container machine-config-daemon
Nov 07 02:04:10.663 I ns/openshift-machine-config-operator pod/machine-config-daemon-lkzwf Started container machine-config-daemon
Nov 07 02:04:10.685 I ns/openshift-machine-config-operator pod/machine-config-daemon-lkzwf Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f711682524b64033a4ce9193bd2cd9621e08b4bae44cffffb4e397a43a8f945" already present on machine
Nov 07 02:04:11.070 I ns/openshift-machine-config-operator pod/machine-config-daemon-lkzwf Created container oauth-proxy
Nov 07 02:04:11.140 I ns/openshift-machine-config-operator pod/machine-config-daemon-lkzwf Started container oauth-proxy
Nov 07 02:04:13.679 W ns/openshift-marketplace pod/community-operators-9567c47bd-z66zz Readiness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n (3 times)
Nov 07 02:04:14.121 W ns/openshift-marketplace pod/community-operators-9567c47bd-z66zz Liveness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n (3 times)
Nov 07 02:04:17.488 I ns/openshift-dns pod/dns-default-c628x Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:614764223d37c466a211f456a9d02c5881d947e1f05d541689dc34a3ec19acf2" already present on machine
Nov 07 02:04:17.698 I ns/openshift-dns pod/dns-default-c628x Created container dns
Nov 07 02:04:17.757 I ns/openshift-dns pod/dns-default-c628x Started container dns
Nov 07 02:04:17.764 I ns/openshift-dns pod/dns-default-c628x Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd150f3e4a29d160fae9610f1096180eaeb29ff29b63bf490b0022b4d3913e29" already present on machine
Nov 07 02:04:17.775 I ns/openshift-image-registry pod/node-ca-b6p4w Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3191fc23d0d7e820100498c5085592219a747743f06fc059436945df6ab3e8ee" already present on machine
Nov 07 02:04:18.012 I ns/openshift-dns pod/dns-default-c628x Created container dns-node-resolver
Nov 07 02:04:18.054 I ns/openshift-dns pod/dns-default-c628x Started container dns-node-resolver
Nov 07 02:04:18.080 I ns/openshift-image-registry pod/node-ca-b6p4w Created container node-ca
Nov 07 02:04:18.121 I ns/openshift-image-registry pod/node-ca-b6p4w Started container node-ca
Nov 07 02:04:21.215 W ns/openshift-ingress service/router-default Error syncing load balancer: failed to ensure load balancer: ensure(openshift-ingress/router-default): backendPoolID(/subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-pg7mtgss-09f59-ng62w-rg/providers/Microsoft.Network/loadBalancers/ci-op-pg7mtgss-09f59-ng62w/backendAddressPools/ci-op-pg7mtgss-09f59-ng62w) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:NicGet" (6 times)
Nov 07 02:04:28.998 W clusteroperator/dns changed Progressing to False: AsExpected: Desired and available number of DNS DaemonSets are equal
Nov 07 02:05:00.095 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj CreateError: failed to reconcile machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj"s: failed to create vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj : failed to create or get machine: compute.VirtualMachinesCreateOrUpdateFuture: asynchronous operation has not completed
Nov 07 02:05:01.055 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj CreateError: failed to reconcile machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj"s: failed to create vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj : vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj is still in provisioning state Creating, reconcile
Nov 07 02:05:03.102 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 CreateError: failed to reconcile machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9"s: failed to create vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 : failed to create or get machine: compute.VirtualMachinesCreateOrUpdateFuture: asynchronous operation has not completed
Nov 07 02:05:05.140 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g CreateError: failed to reconcile machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g"s: failed to create vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g : failed to create or get machine: compute.VirtualMachinesCreateOrUpdateFuture: asynchronous operation has not completed
Nov 07 02:05:06.454 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj CreateError: failed to reconcile machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj"s: failed to create vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj : vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj is still in provisioning state Creating, reconcile (2 times)
Nov 07 02:05:07.751 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 CreateError: failed to reconcile machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9"s: failed to create vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 : vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 is still in provisioning state Creating, reconcile
Nov 07 02:05:08.685 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g CreateError: failed to reconcile machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g"s: failed to create vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g : vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g is still in provisioning state Creating, reconcile
Nov 07 02:05:09.738 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj CreateError: failed to reconcile machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj"s: failed to create vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj : vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj is still in provisioning state Creating, reconcile (3 times)
Nov 07 02:05:10.472 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 CreateError: failed to reconcile machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9"s: failed to create vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 : vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 is still in provisioning state Creating, reconcile (2 times)
Nov 07 02:05:11.428 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g CreateError: failed to reconcile machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g"s: failed to create vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g : vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g is still in provisioning state Creating, reconcile (2 times)
Nov 07 02:05:12.314 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 CreateError: failed to reconcile machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9"s: failed to create vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 : vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 is still in provisioning state Creating, reconcile (3 times)
Nov 07 02:05:13.201 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g CreateError: failed to reconcile machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g"s: failed to create vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g : vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g is still in provisioning state Creating, reconcile (3 times)
Nov 07 02:05:20.986 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj CreateError: failed to reconcile machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj"s: failed to create vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj : vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj is still in provisioning state Creating, reconcile (4 times)
Nov 07 02:05:24.010 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 CreateError: failed to reconcile machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9"s: failed to create vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 : vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 is still in provisioning state Creating, reconcile (4 times)
Nov 07 02:05:26.074 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g CreateError: failed to reconcile machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g"s: failed to create vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g : vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g is still in provisioning state Creating, reconcile (4 times)
Nov 07 02:05:41.878 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj CreateError: failed to reconcile machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj"s: failed to create vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj : vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj is still in provisioning state Creating, reconcile (5 times)
Nov 07 02:05:44.920 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 CreateError: failed to reconcile machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9"s: failed to create vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 : vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 is still in provisioning state Creating, reconcile (5 times)
Nov 07 02:05:47.117 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g CreateError: failed to reconcile machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g"s: failed to create vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g : vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g is still in provisioning state Creating, reconcile (5 times)
Nov 07 02:06:03.050 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj CreateError: failed to reconcile machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj"s: failed to create vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj : vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj is still in provisioning state Creating, reconcile (6 times)
Nov 07 02:06:05.818 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 CreateError: failed to reconcile machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9"s: failed to create vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 : vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 is still in provisioning state Creating, reconcile (6 times)
Nov 07 02:06:07.970 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g CreateError: failed to reconcile machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g"s: failed to create vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g : vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g is still in provisioning state Creating, reconcile (6 times)
Nov 07 02:06:23.935 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj CreateError: failed to reconcile machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj"s: failed to create vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj : vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj is still in provisioning state Creating, reconcile (7 times)
Nov 07 02:06:26.930 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 CreateError: failed to reconcile machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9"s: failed to create vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 : vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 is still in provisioning state Creating, reconcile (7 times)
Nov 07 02:06:28.879 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g CreateError: failed to reconcile machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g"s: failed to create vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g : vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g is still in provisioning state Creating, reconcile (7 times)
Nov 07 02:06:44.688 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj Updated machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj"
Nov 07 02:06:45.065 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj Updated machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj" (2 times)
Nov 07 02:06:45.611 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj Updated machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj" (3 times)
Nov 07 02:06:47.683 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 Updated machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9"
Nov 07 02:06:48.068 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 Updated machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9" (2 times)
Nov 07 02:06:48.603 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 Updated machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9" (3 times)
Nov 07 02:06:49.809 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g CreateError: failed to reconcile machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g"s: failed to create vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g : vm ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g is still in provisioning state Creating, reconcile (8 times)
Nov 07 02:07:10.648 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g Updated machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g"
Nov 07 02:07:11.046 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g Updated machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g" (2 times)
Nov 07 02:07:13.375 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g Updated machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g" (3 times)
Nov 07 02:08:04.820 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (4 times)
Nov 07 02:08:05.416 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-008" not found)
Nov 07 02:09:21.241 I ns/openshift-ingress service/router-default Ensuring load balancer (10 times)
Nov 07 02:09:21.852 W ns/openshift-ingress service/router-default Error syncing load balancer: failed to ensure load balancer: ensure(openshift-ingress/router-default): backendPoolID(/subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-pg7mtgss-09f59-ng62w-rg/providers/Microsoft.Network/loadBalancers/ci-op-pg7mtgss-09f59-ng62w/backendAddressPools/ci-op-pg7mtgss-09f59-ng62w) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:NicGet" (7 times)
Nov 07 02:10:19.991 I node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj Node ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj status is now: NodeHasSufficientMemory (8 times)
Nov 07 02:10:20.007 I ns/openshift-sdn pod/ovs-v7mlm node/ created
Nov 07 02:10:20.028 I ns/openshift-sdn daemonset/ovs Created pod: ovs-v7mlm
Nov 07 02:10:20.031 I node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj Node ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj status is now: NodeHasNoDiskPressure (8 times)
Nov 07 02:10:20.056 I ns/openshift-sdn pod/sdn-hb5jx node/ created
Nov 07 02:10:20.059 W clusteroperator/network changed Progressing to True: Deploying: DaemonSet "openshift-sdn/ovs" is not available (awaiting 1 nodes)
Nov 07 02:10:20.064 I ns/openshift-sdn daemonset/sdn Created pod: sdn-hb5jx
Nov 07 02:10:20.104 I ns/openshift-multus pod/multus-jls67 node/ created
Nov 07 02:10:20.122 I ns/openshift-multus daemonset/multus Created pod: multus-jls67
Nov 07 02:10:20.141 I ns/openshift-sdn pod/sdn-hb5jx Successfully assigned openshift-sdn/sdn-hb5jx to ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj
Nov 07 02:10:20.141 I ns/openshift-sdn pod/ovs-v7mlm Successfully assigned openshift-sdn/ovs-v7mlm to ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj
Nov 07 02:10:20.162 I ns/openshift-monitoring pod/node-exporter-zjvx2 node/ created
Nov 07 02:10:20.168 I ns/openshift-multus pod/multus-jls67 Successfully assigned openshift-multus/multus-jls67 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj
Nov 07 02:10:20.178 I ns/openshift-monitoring daemonset/node-exporter Created pod: node-exporter-zjvx2
Nov 07 02:10:20.203 I ns/openshift-cluster-node-tuning-operator pod/tuned-kkv96 node/ created
Nov 07 02:10:20.208 I ns/openshift-monitoring pod/node-exporter-zjvx2 Successfully assigned openshift-monitoring/node-exporter-zjvx2 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj
Nov 07 02:10:20.221 I ns/openshift-cluster-node-tuning-operator daemonset/tuned Created pod: tuned-kkv96
Nov 07 02:10:20.234 I ns/openshift-cluster-node-tuning-operator pod/tuned-kkv96 Successfully assigned openshift-cluster-node-tuning-operator/tuned-kkv96 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj
Nov 07 02:10:20.607 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj Updated machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj" (4 times)
Nov 07 02:10:21.361 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj Updated machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj" (5 times)
Nov 07 02:10:21.894 I node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj Node ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj event: Registered Node ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj in Controller
Nov 07 02:10:26.204 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (3 times)
Nov 07 02:10:27.201 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-008" not found) (3 times)
Nov 07 02:10:31.142 - 44s   W node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj node is not ready
Nov 07 02:10:40.505 I node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g Node ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g status is now: NodeHasNoDiskPressure (8 times)
Nov 07 02:10:40.516 I ns/openshift-sdn pod/ovs-vw995 node/ created
Nov 07 02:10:40.528 I node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g Node ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g status is now: NodeHasSufficientPID (8 times)
Nov 07 02:10:40.538 I ns/openshift-sdn daemonset/ovs Created pod: ovs-vw995
Nov 07 02:10:40.567 I ns/openshift-sdn pod/ovs-vw995 Successfully assigned openshift-sdn/ovs-vw995 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g
Nov 07 02:10:40.584 I ns/openshift-sdn pod/sdn-stvl4 node/ created
Nov 07 02:10:40.600 I ns/openshift-sdn daemonset/sdn Created pod: sdn-stvl4
Nov 07 02:10:40.609 I ns/openshift-monitoring pod/node-exporter-p86zj node/ created
Nov 07 02:10:40.622 I ns/openshift-monitoring daemonset/node-exporter Created pod: node-exporter-p86zj
Nov 07 02:10:40.622 I ns/openshift-sdn pod/sdn-stvl4 Successfully assigned openshift-sdn/sdn-stvl4 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g
Nov 07 02:10:40.652 I ns/openshift-multus pod/multus-s2j8q node/ created
Nov 07 02:10:40.656 I ns/openshift-monitoring pod/node-exporter-p86zj Successfully assigned openshift-monitoring/node-exporter-p86zj to ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g
Nov 07 02:10:40.675 I ns/openshift-multus daemonset/multus Created pod: multus-s2j8q
Nov 07 02:10:40.678 I ns/openshift-cluster-node-tuning-operator pod/tuned-dvg48 node/ created
Nov 07 02:10:40.701 I ns/openshift-cluster-node-tuning-operator daemonset/tuned Created pod: tuned-dvg48
Nov 07 02:10:40.708 I ns/openshift-multus pod/multus-s2j8q Successfully assigned openshift-multus/multus-s2j8q to ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g
Nov 07 02:10:40.731 I ns/openshift-cluster-node-tuning-operator pod/tuned-dvg48 Successfully assigned openshift-cluster-node-tuning-operator/tuned-dvg48 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g
Nov 07 02:10:41.109 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g Updated machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g" (4 times)
Nov 07 02:10:41.673 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g Updated machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g" (5 times)
Nov 07 02:10:41.888 I node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g Node ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g event: Registered Node ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g in Controller
Nov 07 02:10:46.142 - 44s   W node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g node is not ready
Nov 07 02:10:50.361 I ns/openshift-sdn pod/sdn-hb5jx Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7583d6a8992295f82cd1817d55f847a545a8a8e3ca65a5472964b471490d6c1c"
Nov 07 02:10:50.511 I ns/openshift-sdn pod/ovs-v7mlm Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e239b1b6588f0a3bdfd921bb1e816f9d64e7a400fc334817521f3d9c67d2ce"
Nov 07 02:10:50.801 I ns/openshift-multus pod/multus-jls67 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7583d6a8992295f82cd1817d55f847a545a8a8e3ca65a5472964b471490d6c1c"
Nov 07 02:10:50.829 I ns/openshift-cluster-node-tuning-operator pod/tuned-kkv96 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3207589b466325f1ee9decd9f41416ca20597d409d6071c3614ce2f5e9626368"
Nov 07 02:10:50.872 I ns/openshift-monitoring pod/node-exporter-zjvx2 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7d5edb31f42f0b66d3f9848af5a65fa82c12014aecdc916b846d0b40a2d46e0a"
Nov 07 02:11:07.491 I ns/openshift-monitoring pod/node-exporter-zjvx2 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7d5edb31f42f0b66d3f9848af5a65fa82c12014aecdc916b846d0b40a2d46e0a"
Nov 07 02:11:07.538 I ns/openshift-cluster-node-tuning-operator pod/tuned-kkv96 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3207589b466325f1ee9decd9f41416ca20597d409d6071c3614ce2f5e9626368"
Nov 07 02:11:07.581 I ns/openshift-multus pod/multus-jls67 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7583d6a8992295f82cd1817d55f847a545a8a8e3ca65a5472964b471490d6c1c"
Nov 07 02:11:07.631 I ns/openshift-sdn pod/sdn-hb5jx Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7583d6a8992295f82cd1817d55f847a545a8a8e3ca65a5472964b471490d6c1c"
Nov 07 02:11:07.660 I ns/openshift-sdn pod/ovs-v7mlm Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e239b1b6588f0a3bdfd921bb1e816f9d64e7a400fc334817521f3d9c67d2ce"
Nov 07 02:11:07.975 I ns/openshift-monitoring pod/node-exporter-zjvx2 Created container node-exporter
Nov 07 02:11:08.137 I ns/openshift-cluster-node-tuning-operator pod/tuned-kkv96 Created container tuned
Nov 07 02:11:08.210 I ns/openshift-monitoring pod/node-exporter-zjvx2 Started container node-exporter
Nov 07 02:11:08.222 I ns/openshift-monitoring pod/node-exporter-zjvx2 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5366da9d7c15f912b609cdbb2851d01ea863fb22d452081914d923d8208c39c8"
Nov 07 02:11:08.282 I ns/openshift-multus pod/multus-jls67 Created container cni-plugins
Nov 07 02:11:08.354 I ns/openshift-cluster-node-tuning-operator pod/tuned-kkv96 Started container tuned
Nov 07 02:11:08.374 I ns/openshift-sdn pod/ovs-v7mlm Created container openvswitch
Nov 07 02:11:08.382 I ns/openshift-multus pod/multus-jls67 Started container cni-plugins
Nov 07 02:11:08.390 I ns/openshift-sdn pod/sdn-hb5jx Created container install-cni-plugins
Nov 07 02:11:08.598 I ns/openshift-sdn pod/ovs-v7mlm Started container openvswitch
Nov 07 02:11:08.690 I ns/openshift-sdn pod/sdn-hb5jx Started container install-cni-plugins
Nov 07 02:11:08.891 I ns/openshift-multus pod/multus-jls67 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7d96246be6464e56863cc2b85ad14586e5034d0ee39aced7c90adab6f46e260f"
Nov 07 02:11:09.624 I ns/openshift-sdn pod/sdn-hb5jx Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e239b1b6588f0a3bdfd921bb1e816f9d64e7a400fc334817521f3d9c67d2ce" already present on machine
Nov 07 02:11:09.673 I ns/openshift-sdn pod/ovs-vw995 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e239b1b6588f0a3bdfd921bb1e816f9d64e7a400fc334817521f3d9c67d2ce"
Nov 07 02:11:09.817 I ns/openshift-sdn pod/sdn-stvl4 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7583d6a8992295f82cd1817d55f847a545a8a8e3ca65a5472964b471490d6c1c"
Nov 07 02:11:09.896 I ns/openshift-sdn pod/sdn-hb5jx Created container sdn
Nov 07 02:11:10.016 I ns/openshift-sdn pod/sdn-hb5jx Started container sdn
Nov 07 02:11:10.106 I ns/openshift-monitoring pod/node-exporter-p86zj Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7d5edb31f42f0b66d3f9848af5a65fa82c12014aecdc916b846d0b40a2d46e0a"
Nov 07 02:11:10.260 I ns/openshift-cluster-node-tuning-operator pod/tuned-dvg48 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3207589b466325f1ee9decd9f41416ca20597d409d6071c3614ce2f5e9626368"
Nov 07 02:11:10.303 I ns/openshift-multus pod/multus-s2j8q Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7583d6a8992295f82cd1817d55f847a545a8a8e3ca65a5472964b471490d6c1c"
Nov 07 02:11:11.780 W ns/openshift-sdn pod/sdn-hb5jx Liveness probe failed: Get http://10.0.32.7:10256/healthz: dial tcp 10.0.32.7:10256: connect: connection refused
Nov 07 02:11:14.582 I node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj openshift-sdn done initializing node networking.
Nov 07 02:11:15.773 I ns/openshift-monitoring pod/node-exporter-zjvx2 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5366da9d7c15f912b609cdbb2851d01ea863fb22d452081914d923d8208c39c8"
Nov 07 02:11:15.795 I ns/openshift-multus pod/multus-jls67 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7d96246be6464e56863cc2b85ad14586e5034d0ee39aced7c90adab6f46e260f"
Nov 07 02:11:16.073 I ns/openshift-monitoring pod/node-exporter-zjvx2 Created container kube-rbac-proxy
Nov 07 02:11:16.081 I ns/openshift-multus pod/multus-jls67 Created container kube-multus
Nov 07 02:11:16.113 I ns/openshift-monitoring pod/node-exporter-zjvx2 Started container kube-rbac-proxy
Nov 07 02:11:16.142 I ns/openshift-multus pod/multus-jls67 Started container kube-multus
Nov 07 02:11:17.805 I ns/openshift-monitoring pod/node-exporter-p86zj Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7d5edb31f42f0b66d3f9848af5a65fa82c12014aecdc916b846d0b40a2d46e0a"
Nov 07 02:11:18.509 I ns/openshift-monitoring pod/node-exporter-p86zj Created container node-exporter
Nov 07 02:11:18.541 I ns/openshift-monitoring pod/node-exporter-p86zj Started container node-exporter
Nov 07 02:11:18.547 I ns/openshift-monitoring pod/node-exporter-p86zj Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5366da9d7c15f912b609cdbb2851d01ea863fb22d452081914d923d8208c39c8"
Nov 07 02:11:20.913 W node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj condition Ready changed
Nov 07 02:11:20.957 I ns/openshift-dns pod/dns-default-w768d node/ created
Nov 07 02:11:20.971 I ns/openshift-machine-config-operator pod/machine-config-daemon-z9w2v node/ created
Nov 07 02:11:20.986 I ns/openshift-dns daemonset/dns-default Created pod: dns-default-w768d
Nov 07 02:11:20.997 I ns/openshift-machine-config-operator daemonset/machine-config-daemon Created pod: machine-config-daemon-z9w2v
Nov 07 02:11:21.020 I ns/openshift-dns pod/dns-default-w768d Successfully assigned openshift-dns/dns-default-w768d to ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj
Nov 07 02:11:21.020 I ns/openshift-machine-config-operator pod/machine-config-daemon-z9w2v Successfully assigned openshift-machine-config-operator/machine-config-daemon-z9w2v to ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj
Nov 07 02:11:21.046 I ns/openshift-image-registry pod/node-ca-bnpwn node/ created
Nov 07 02:11:21.061 I ns/openshift-image-registry daemonset/node-ca Created pod: node-ca-bnpwn
Nov 07 02:11:21.099 I ns/openshift-image-registry pod/node-ca-bnpwn Successfully assigned openshift-image-registry/node-ca-bnpwn to ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj
Nov 07 02:11:21.142 W clusteroperator/dns changed Progressing to True: Reconciling: Not all DNS DaemonSets available.
Nov 07 02:11:21.600 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj Updated machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj" (6 times)
Nov 07 02:11:22.301 I ns/openshift-machine-config-operator pod/machine-config-daemon-z9w2v Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:944d643d229b0a1028464df12f2a544163a6689947df774c6568b3f443a79cb8"
Nov 07 02:11:26.950 I ns/openshift-sdn pod/sdn-stvl4 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7583d6a8992295f82cd1817d55f847a545a8a8e3ca65a5472964b471490d6c1c"
Nov 07 02:11:27.580 I ns/openshift-multus pod/multus-s2j8q Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7583d6a8992295f82cd1817d55f847a545a8a8e3ca65a5472964b471490d6c1c"
Nov 07 02:11:27.982 I ns/openshift-sdn pod/sdn-stvl4 Created container install-cni-plugins
Nov 07 02:11:27.997 I ns/openshift-monitoring pod/node-exporter-p86zj Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5366da9d7c15f912b609cdbb2851d01ea863fb22d452081914d923d8208c39c8"
Nov 07 02:11:28.006 I ns/openshift-cluster-node-tuning-operator pod/tuned-dvg48 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3207589b466325f1ee9decd9f41416ca20597d409d6071c3614ce2f5e9626368"
Nov 07 02:11:28.052 I ns/openshift-sdn pod/ovs-vw995 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e239b1b6588f0a3bdfd921bb1e816f9d64e7a400fc334817521f3d9c67d2ce"
Nov 07 02:11:28.059 I ns/openshift-sdn pod/sdn-stvl4 Started container install-cni-plugins
Nov 07 02:11:28.205 I ns/openshift-multus pod/multus-s2j8q Created container cni-plugins
Nov 07 02:11:28.245 I ns/openshift-sdn pod/sdn-stvl4 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e239b1b6588f0a3bdfd921bb1e816f9d64e7a400fc334817521f3d9c67d2ce" already present on machine
Nov 07 02:11:28.577 I ns/openshift-monitoring pod/node-exporter-p86zj Created container kube-rbac-proxy
Nov 07 02:11:28.687 I ns/openshift-sdn pod/ovs-vw995 Created container openvswitch
Nov 07 02:11:28.694 I ns/openshift-monitoring pod/node-exporter-p86zj Started container kube-rbac-proxy
Nov 07 02:11:28.703 I ns/openshift-cluster-node-tuning-operator pod/tuned-dvg48 Created container tuned
Nov 07 02:11:28.779 I ns/openshift-sdn pod/ovs-vw995 Started container openvswitch
Nov 07 02:11:28.787 I ns/openshift-sdn pod/sdn-stvl4 Created container sdn
Nov 07 02:11:28.805 I ns/openshift-cluster-node-tuning-operator pod/tuned-dvg48 Started container tuned
Nov 07 02:11:28.985 I ns/openshift-sdn pod/sdn-stvl4 Started container sdn
Nov 07 02:11:29.184 I ns/openshift-multus pod/multus-s2j8q Started container cni-plugins
Nov 07 02:11:29.392 I ns/openshift-multus pod/multus-s2j8q Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7d96246be6464e56863cc2b85ad14586e5034d0ee39aced7c90adab6f46e260f"
Nov 07 02:11:29.524 I ns/openshift-dns pod/dns-default-w768d Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:614764223d37c466a211f456a9d02c5881d947e1f05d541689dc34a3ec19acf2"
Nov 07 02:11:30.059 I ns/openshift-image-registry pod/node-ca-bnpwn Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3191fc23d0d7e820100498c5085592219a747743f06fc059436945df6ab3e8ee"
Nov 07 02:11:30.577 I node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g openshift-sdn done initializing node networking.
Nov 07 02:11:34.043 I ns/openshift-machine-config-operator pod/machine-config-daemon-z9w2v Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:944d643d229b0a1028464df12f2a544163a6689947df774c6568b3f443a79cb8"
Nov 07 02:11:34.930 I ns/openshift-machine-config-operator pod/machine-config-daemon-z9w2v Created container machine-config-daemon
Nov 07 02:11:35.512 I ns/openshift-machine-config-operator pod/machine-config-daemon-z9w2v Started container machine-config-daemon
Nov 07 02:11:35.603 I ns/openshift-machine-config-operator pod/machine-config-daemon-z9w2v Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f711682524b64033a4ce9193bd2cd9621e08b4bae44cffffb4e397a43a8f945"
Nov 07 02:11:35.691 I ns/openshift-multus pod/multus-s2j8q Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7d96246be6464e56863cc2b85ad14586e5034d0ee39aced7c90adab6f46e260f"
Nov 07 02:11:35.883 I ns/openshift-multus pod/multus-s2j8q Created container kube-multus
Nov 07 02:11:35.930 I ns/openshift-multus pod/multus-s2j8q Started container kube-multus
Nov 07 02:11:36.169 I node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 Node ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 status is now: NodeHasSufficientMemory (8 times)
Nov 07 02:11:36.193 I node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 Node ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 status is now: NodeHasNoDiskPressure (8 times)
Nov 07 02:11:36.195 I ns/openshift-sdn pod/ovs-l7jt8 node/ created
Nov 07 02:11:36.221 I ns/openshift-sdn pod/sdn-nm8f8 node/ created
Nov 07 02:11:36.234 I ns/openshift-sdn daemonset/ovs Created pod: ovs-l7jt8
Nov 07 02:11:36.246 I ns/openshift-sdn daemonset/sdn Created pod: sdn-nm8f8
Nov 07 02:11:36.260 I ns/openshift-sdn pod/sdn-nm8f8 Successfully assigned openshift-sdn/sdn-nm8f8 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:11:36.260 I ns/openshift-sdn pod/ovs-l7jt8 Successfully assigned openshift-sdn/ovs-l7jt8 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:11:36.270 I ns/openshift-multus pod/multus-4blp5 node/ created
Nov 07 02:11:36.291 I ns/openshift-multus daemonset/multus Created pod: multus-4blp5
Nov 07 02:11:36.327 I ns/openshift-cluster-node-tuning-operator pod/tuned-jbbvc node/ created
Nov 07 02:11:36.342 I ns/openshift-cluster-node-tuning-operator daemonset/tuned Created pod: tuned-jbbvc
Nov 07 02:11:36.354 I ns/openshift-multus pod/multus-4blp5 Successfully assigned openshift-multus/multus-4blp5 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:11:36.372 I ns/openshift-cluster-node-tuning-operator pod/tuned-jbbvc Successfully assigned openshift-cluster-node-tuning-operator/tuned-jbbvc to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:11:36.406 I ns/openshift-monitoring pod/node-exporter-9bxbh node/ created
Nov 07 02:11:36.420 I ns/openshift-monitoring daemonset/node-exporter Created pod: node-exporter-9bxbh
Nov 07 02:11:36.442 I ns/openshift-monitoring pod/node-exporter-9bxbh Successfully assigned openshift-monitoring/node-exporter-9bxbh to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:11:36.917 I node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 Node ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 event: Registered Node ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 in Controller
Nov 07 02:11:37.035 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 Updated machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9" (4 times)
Nov 07 02:11:37.583 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 Updated machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9" (5 times)
Nov 07 02:11:41.320 W node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g condition Ready changed
Nov 07 02:11:41.358 I ns/openshift-dns pod/dns-default-nzxdl node/ created
Nov 07 02:11:41.383 I ns/openshift-dns daemonset/dns-default Created pod: dns-default-nzxdl
Nov 07 02:11:41.402 I ns/openshift-image-registry pod/node-ca-7hhjn node/ created
Nov 07 02:11:41.423 I ns/openshift-machine-config-operator pod/machine-config-daemon-l7p2b node/ created
Nov 07 02:11:41.427 I ns/openshift-dns pod/dns-default-nzxdl Successfully assigned openshift-dns/dns-default-nzxdl to ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g
Nov 07 02:11:41.436 I ns/openshift-image-registry daemonset/node-ca Created pod: node-ca-7hhjn
Nov 07 02:11:41.450 I ns/openshift-machine-config-operator daemonset/machine-config-daemon Created pod: machine-config-daemon-l7p2b
Nov 07 02:11:41.450 I ns/openshift-image-registry pod/node-ca-7hhjn Successfully assigned openshift-image-registry/node-ca-7hhjn to ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g
Nov 07 02:11:41.465 I ns/openshift-machine-config-operator pod/machine-config-daemon-l7p2b Successfully assigned openshift-machine-config-operator/machine-config-daemon-l7p2b to ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g
Nov 07 02:11:41.583 E clusteroperator/dns changed Degraded to True: NotAllDNSesAvailable: Not all desired DNS DaemonSets available
Nov 07 02:11:41.979 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g Updated machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g" (6 times)
Nov 07 02:11:41.999 I node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj Setting node ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj, currentConfig rendered-worker-375794f00b8fe3b7bf8d2b6a16656633 to Done
Nov 07 02:11:42.650 I ns/openshift-machine-config-operator pod/machine-config-daemon-l7p2b Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:944d643d229b0a1028464df12f2a544163a6689947df774c6568b3f443a79cb8"
Nov 07 02:11:44.648 I ns/openshift-dns pod/dns-default-w768d Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:614764223d37c466a211f456a9d02c5881d947e1f05d541689dc34a3ec19acf2"
Nov 07 02:11:45.125 I ns/openshift-dns pod/dns-default-w768d Created container dns
Nov 07 02:11:45.218 I ns/openshift-dns pod/dns-default-w768d Started container dns
Nov 07 02:11:45.234 I ns/openshift-dns pod/dns-default-w768d Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd150f3e4a29d160fae9610f1096180eaeb29ff29b63bf490b0022b4d3913e29"
Nov 07 02:11:46.142 - 89s   W node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 node is not ready
Nov 07 02:11:46.969 I ns/openshift-machine-config-operator pod/machine-config-daemon-z9w2v Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f711682524b64033a4ce9193bd2cd9621e08b4bae44cffffb4e397a43a8f945"
Nov 07 02:11:47.243 I ns/openshift-machine-config-operator pod/machine-config-daemon-z9w2v Created container oauth-proxy
Nov 07 02:11:47.327 I ns/openshift-machine-config-operator pod/machine-config-daemon-z9w2v Started container oauth-proxy
Nov 07 02:11:49.848 I ns/openshift-dns pod/dns-default-nzxdl Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:614764223d37c466a211f456a9d02c5881d947e1f05d541689dc34a3ec19acf2"
Nov 07 02:11:50.598 I ns/openshift-image-registry pod/node-ca-7hhjn Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3191fc23d0d7e820100498c5085592219a747743f06fc059436945df6ab3e8ee"
Nov 07 02:11:52.142 W ns/openshift-ingress service/router-default Error updating load balancer with new hosts map[ci-op-pg7mtgss-09f59-ng62w-master-0:{} ci-op-pg7mtgss-09f59-ng62w-master-1:{} ci-op-pg7mtgss-09f59-ng62w-master-2:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-z8fpv:{}]: ensure(openshift-ingress/router-default): backendPoolID(/subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-pg7mtgss-09f59-ng62w-rg/providers/Microsoft.Network/loadBalancers/ci-op-pg7mtgss-09f59-ng62w/backendAddressPools/ci-op-pg7mtgss-09f59-ng62w) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:NicGet"
Nov 07 02:11:54.030 I ns/openshift-machine-config-operator pod/machine-config-daemon-l7p2b Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:944d643d229b0a1028464df12f2a544163a6689947df774c6568b3f443a79cb8"
Nov 07 02:11:54.235 I ns/openshift-machine-config-operator pod/machine-config-daemon-l7p2b Created container machine-config-daemon
Nov 07 02:11:54.385 I ns/openshift-machine-config-operator pod/machine-config-daemon-l7p2b Started container machine-config-daemon
Nov 07 02:11:54.474 I ns/openshift-machine-config-operator pod/machine-config-daemon-l7p2b Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f711682524b64033a4ce9193bd2cd9621e08b4bae44cffffb4e397a43a8f945"
Nov 07 02:12:00.881 I ns/openshift-dns pod/dns-default-w768d Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd150f3e4a29d160fae9610f1096180eaeb29ff29b63bf490b0022b4d3913e29"
Nov 07 02:12:01.188 I ns/openshift-dns pod/dns-default-w768d Created container dns-node-resolver
Nov 07 02:12:01.301 I ns/openshift-dns pod/dns-default-w768d Started container dns-node-resolver
Nov 07 02:12:01.541 I ns/openshift-dns pod/dns-default-nzxdl Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:614764223d37c466a211f456a9d02c5881d947e1f05d541689dc34a3ec19acf2"
Nov 07 02:12:01.800 I ns/openshift-dns pod/dns-default-nzxdl Created container dns
Nov 07 02:12:01.881 I ns/openshift-dns pod/dns-default-nzxdl Started container dns
Nov 07 02:12:01.890 I ns/openshift-dns pod/dns-default-nzxdl Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd150f3e4a29d160fae9610f1096180eaeb29ff29b63bf490b0022b4d3913e29"
Nov 07 02:12:06.983 I ns/openshift-machine-config-operator pod/machine-config-daemon-l7p2b Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f711682524b64033a4ce9193bd2cd9621e08b4bae44cffffb4e397a43a8f945"
Nov 07 02:12:07.233 I ns/openshift-machine-config-operator pod/machine-config-daemon-l7p2b Created container oauth-proxy
Nov 07 02:12:07.276 I ns/openshift-machine-config-operator pod/machine-config-daemon-l7p2b Started container oauth-proxy
Nov 07 02:12:09.064 I ns/openshift-sdn pod/ovs-l7jt8 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e239b1b6588f0a3bdfd921bb1e816f9d64e7a400fc334817521f3d9c67d2ce"
Nov 07 02:12:09.252 I ns/openshift-sdn pod/sdn-nm8f8 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7583d6a8992295f82cd1817d55f847a545a8a8e3ca65a5472964b471490d6c1c"
Nov 07 02:12:09.308 I ns/openshift-cluster-node-tuning-operator pod/tuned-jbbvc Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3207589b466325f1ee9decd9f41416ca20597d409d6071c3614ce2f5e9626368"
Nov 07 02:12:09.568 I ns/openshift-monitoring pod/node-exporter-9bxbh Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7d5edb31f42f0b66d3f9848af5a65fa82c12014aecdc916b846d0b40a2d46e0a"
Nov 07 02:12:09.648 I ns/openshift-multus pod/multus-4blp5 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7583d6a8992295f82cd1817d55f847a545a8a8e3ca65a5472964b471490d6c1c"
Nov 07 02:12:10.726 W clusteroperator/dns changed Degraded to False: AsExpected: All desired DNS DaemonSets available and operand Namespace exists
Nov 07 02:12:15.072 I ns/openshift-monitoring pod/node-exporter-9bxbh Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7d5edb31f42f0b66d3f9848af5a65fa82c12014aecdc916b846d0b40a2d46e0a"
Nov 07 02:12:15.219 I node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g Setting node ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g, currentConfig rendered-worker-375794f00b8fe3b7bf8d2b6a16656633 to Done
Nov 07 02:12:15.516 I ns/openshift-monitoring pod/node-exporter-9bxbh Created container node-exporter
Nov 07 02:12:15.667 I ns/openshift-monitoring pod/node-exporter-9bxbh Started container node-exporter
Nov 07 02:12:15.675 I ns/openshift-monitoring pod/node-exporter-9bxbh Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5366da9d7c15f912b609cdbb2851d01ea863fb22d452081914d923d8208c39c8"
Nov 07 02:12:17.432 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-0 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (10 times)
Nov 07 02:12:18.027 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-0 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-008" not found) (15 times)
Nov 07 02:12:21.198 I ns/openshift-dns pod/dns-default-nzxdl Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd150f3e4a29d160fae9610f1096180eaeb29ff29b63bf490b0022b4d3913e29"
Nov 07 02:12:21.390 I ns/openshift-dns pod/dns-default-nzxdl Created container dns-node-resolver
Nov 07 02:12:21.432 I ns/openshift-dns pod/dns-default-nzxdl Started container dns-node-resolver
Nov 07 02:12:22.768 W clusteroperator/dns changed Progressing to False: AsExpected: Desired and available number of DNS DaemonSets are equal
Nov 07 02:12:22.828 I ns/openshift-sdn pod/ovs-l7jt8 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e239b1b6588f0a3bdfd921bb1e816f9d64e7a400fc334817521f3d9c67d2ce"
Nov 07 02:12:23.039 I ns/openshift-sdn pod/ovs-l7jt8 Created container openvswitch
Nov 07 02:12:23.098 I ns/openshift-sdn pod/ovs-l7jt8 Started container openvswitch
Nov 07 02:12:26.817 I ns/openshift-multus pod/multus-4blp5 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7583d6a8992295f82cd1817d55f847a545a8a8e3ca65a5472964b471490d6c1c"
Nov 07 02:12:26.879 I ns/openshift-sdn pod/sdn-nm8f8 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7583d6a8992295f82cd1817d55f847a545a8a8e3ca65a5472964b471490d6c1c"
Nov 07 02:12:26.886 I ns/openshift-monitoring pod/node-exporter-9bxbh Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5366da9d7c15f912b609cdbb2851d01ea863fb22d452081914d923d8208c39c8"
Nov 07 02:12:27.309 I ns/openshift-multus pod/multus-4blp5 Created container cni-plugins
Nov 07 02:12:27.343 I ns/openshift-sdn pod/sdn-nm8f8 Created container install-cni-plugins
Nov 07 02:12:27.370 I ns/openshift-monitoring pod/node-exporter-9bxbh Created container kube-rbac-proxy
Nov 07 02:12:27.403 I ns/openshift-multus pod/multus-4blp5 Started container cni-plugins
Nov 07 02:12:27.424 I ns/openshift-sdn pod/sdn-nm8f8 Started container install-cni-plugins
Nov 07 02:12:27.627 I ns/openshift-monitoring pod/node-exporter-9bxbh Started container kube-rbac-proxy
Nov 07 02:12:28.149 I ns/openshift-multus pod/multus-4blp5 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7d96246be6464e56863cc2b85ad14586e5034d0ee39aced7c90adab6f46e260f"
Nov 07 02:12:28.160 I ns/openshift-sdn pod/sdn-nm8f8 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e239b1b6588f0a3bdfd921bb1e816f9d64e7a400fc334817521f3d9c67d2ce" already present on machine
Nov 07 02:12:28.383 I ns/openshift-sdn pod/sdn-nm8f8 Created container sdn
Nov 07 02:12:28.429 I ns/openshift-sdn pod/sdn-nm8f8 Started container sdn
Nov 07 02:12:30.342 I node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 openshift-sdn done initializing node networking.
Nov 07 02:12:31.142 W ns/openshift-image-registry pod/node-ca-bnpwn node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj pod has been pending longer than a minute
Nov 07 02:12:32.160 I ns/openshift-image-registry pod/node-ca-bnpwn Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3191fc23d0d7e820100498c5085592219a747743f06fc059436945df6ab3e8ee"
Nov 07 02:12:32.375 I ns/openshift-image-registry pod/node-ca-bnpwn Created container node-ca
Nov 07 02:12:32.448 I ns/openshift-image-registry pod/node-ca-bnpwn Started container node-ca
Nov 07 02:12:39.073 W ns/openshift-sdn pod/sdn-nm8f8 Liveness probe failed: Get http://10.0.32.8:10256/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Nov 07 02:12:44.691 I ns/openshift-image-registry pod/node-ca-7hhjn Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3191fc23d0d7e820100498c5085592219a747743f06fc059436945df6ab3e8ee"
Nov 07 02:12:44.884 I ns/openshift-image-registry pod/node-ca-7hhjn Created container node-ca
Nov 07 02:12:44.923 I ns/openshift-image-registry pod/node-ca-7hhjn Started container node-ca
Nov 07 02:12:46.142 - 14s   W ns/openshift-multus pod/multus-4blp5 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 pod has been pending longer than a minute
Nov 07 02:12:46.142 - 44s   W ns/openshift-cluster-node-tuning-operator pod/tuned-jbbvc node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 pod has been pending longer than a minute
Nov 07 02:13:09.061 I ns/openshift-multus pod/multus-4blp5 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7d96246be6464e56863cc2b85ad14586e5034d0ee39aced7c90adab6f46e260f"
Nov 07 02:13:09.270 I ns/openshift-multus pod/multus-4blp5 Created container kube-multus
Nov 07 02:13:09.334 I ns/openshift-multus pod/multus-4blp5 Started container kube-multus
Nov 07 02:13:10.262 W clusteroperator/network changed Progressing to False
Nov 07 02:13:17.555 W node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 condition Ready changed
Nov 07 02:13:17.598 I ns/openshift-dns pod/dns-default-9qpmp node/ created
Nov 07 02:13:17.628 I ns/openshift-dns daemonset/dns-default Created pod: dns-default-9qpmp
Nov 07 02:13:17.646 I ns/openshift-machine-config-operator pod/machine-config-daemon-l5rhl node/ created
Nov 07 02:13:17.646 I ns/openshift-image-registry pod/node-ca-6jr5f node/ created
Nov 07 02:13:17.663 I ns/openshift-machine-config-operator daemonset/machine-config-daemon Created pod: machine-config-daemon-l5rhl
Nov 07 02:13:17.679 I ns/openshift-dns pod/dns-default-9qpmp Successfully assigned openshift-dns/dns-default-9qpmp to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:13:17.679 I ns/openshift-image-registry daemonset/node-ca Created pod: node-ca-6jr5f
Nov 07 02:13:17.684 I ns/openshift-machine-config-operator pod/machine-config-daemon-l5rhl Successfully assigned openshift-machine-config-operator/machine-config-daemon-l5rhl to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:13:17.699 I ns/openshift-image-registry pod/node-ca-6jr5f Successfully assigned openshift-image-registry/node-ca-6jr5f to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:13:17.776 W clusteroperator/dns changed Progressing to True: Reconciling: Not all DNS DaemonSets available.
Nov 07 02:13:18.441 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 Updated machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9" (6 times)
Nov 07 02:13:18.954 I ns/openshift-machine-config-operator pod/machine-config-daemon-l5rhl Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:944d643d229b0a1028464df12f2a544163a6689947df774c6568b3f443a79cb8"
Nov 07 02:13:19.454 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 Node "ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9" drained
Nov 07 02:13:19.921 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 DeleteError: failed to delete machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9": failed to delete machine: compute.VirtualMachinesDeleteFuture: asynchronous operation has not completed
Nov 07 02:13:20.047 W ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-lwfzk node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 graceful deletion within 30s
Nov 07 02:13:20.061 W ns/openshift-marketplace pod/redhat-operators-64566c9856-gjdpn node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 graceful deletion within 30s
Nov 07 02:13:20.062 W ns/openshift-marketplace pod/community-operators-9567c47bd-z66zz node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 graceful deletion within 30s
Nov 07 02:13:20.063 W ns/openshift-monitoring pod/telemeter-client-55f55df45-lnv6w node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 graceful deletion within 30s
Nov 07 02:13:20.085 W ns/openshift-ingress pod/router-default-68c6467955-pc5mb node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 graceful deletion within 30s
Nov 07 02:13:20.100 W ns/openshift-monitoring pod/prometheus-k8s-1 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 graceful deletion within 600s
Nov 07 02:13:20.100 W ns/openshift-marketplace pod/certified-operators-69589995bf-9l67b node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 graceful deletion within 30s
Nov 07 02:13:20.101 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-lwfzk Stopping container kube-rbac-proxy-main
Nov 07 02:13:20.101 W ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 graceful deletion within 120s
Nov 07 02:13:20.101 W ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-vsdf8 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 graceful deletion within 30s
Nov 07 02:13:20.123 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-lwfzk Stopping container openshift-state-metrics
Nov 07 02:13:20.139 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-lwfzk Stopping container kube-rbac-proxy-self
Nov 07 02:13:20.173 I ns/openshift-ingress pod/router-default-68c6467955-474p6 node/ created
Nov 07 02:13:20.178 I ns/openshift-marketplace pod/redhat-operators-64566c9856-gjdpn Stopping container redhat-operators
Nov 07 02:13:20.191 I ns/openshift-marketplace pod/community-operators-9567c47bd-5s9sf node/ created
Nov 07 02:13:20.194 I ns/openshift-ingress replicaset/router-default-68c6467955 Created pod: router-default-68c6467955-474p6
Nov 07 02:13:20.195 I ns/openshift-marketplace pod/redhat-operators-64566c9856-zhk47 node/ created
Nov 07 02:13:20.216 I ns/openshift-marketplace replicaset/community-operators-9567c47bd Created pod: community-operators-9567c47bd-5s9sf
Nov 07 02:13:20.227 I ns/openshift-marketplace pod/community-operators-9567c47bd-z66zz Stopping container community-operators
Nov 07 02:13:20.228 I ns/openshift-monitoring pod/telemeter-client-55f55df45-r9w6d node/ created
Nov 07 02:13:20.229 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-8z6z2 node/ created
Nov 07 02:13:20.248 I ns/openshift-marketplace replicaset/redhat-operators-64566c9856 Created pod: redhat-operators-64566c9856-zhk47
Nov 07 02:13:20.265 I ns/openshift-monitoring pod/telemeter-client-55f55df45-lnv6w Stopping container telemeter-client
Nov 07 02:13:20.265 I ns/openshift-ingress pod/router-default-68c6467955-474p6 Successfully assigned openshift-ingress/router-default-68c6467955-474p6 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:13:20.265 I ns/openshift-marketplace pod/redhat-operators-64566c9856-zhk47 Successfully assigned openshift-marketplace/redhat-operators-64566c9856-zhk47 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj
Nov 07 02:13:20.265 I ns/openshift-marketplace pod/community-operators-9567c47bd-5s9sf Successfully assigned openshift-marketplace/community-operators-9567c47bd-5s9sf to ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g
Nov 07 02:13:20.281 I ns/openshift-monitoring replicaset/openshift-state-metrics-6c78647cc7 Created pod: openshift-state-metrics-6c78647cc7-8z6z2
Nov 07 02:13:20.302 I ns/openshift-monitoring pod/telemeter-client-55f55df45-lnv6w Stopping container kube-rbac-proxy
Nov 07 02:13:20.302 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-8z6z2 Successfully assigned openshift-monitoring/openshift-state-metrics-6c78647cc7-8z6z2 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:13:20.302 I ns/openshift-monitoring pod/telemeter-client-55f55df45-r9w6d Successfully assigned openshift-monitoring/telemeter-client-55f55df45-r9w6d to ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj
Nov 07 02:13:20.308 I ns/openshift-monitoring replicaset/telemeter-client-55f55df45 Created pod: telemeter-client-55f55df45-r9w6d
Nov 07 02:13:20.340 I ns/openshift-marketplace pod/certified-operators-69589995bf-bhhjn node/ created
Nov 07 02:13:20.343 I ns/openshift-monitoring pod/telemeter-client-55f55df45-lnv6w Stopping container reload
Nov 07 02:13:20.354 I ns/openshift-marketplace replicaset/certified-operators-69589995bf Created pod: certified-operators-69589995bf-bhhjn
Nov 07 02:13:20.374 I ns/openshift-ingress pod/router-default-68c6467955-pc5mb Stopping container router
Nov 07 02:13:20.380 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-k7t29 node/ created
Nov 07 02:13:20.391 I ns/openshift-marketplace pod/certified-operators-69589995bf-bhhjn Successfully assigned openshift-marketplace/certified-operators-69589995bf-bhhjn to ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g
Nov 07 02:13:20.402 I ns/openshift-monitoring replicaset/prometheus-adapter-6fb7598968 Created pod: prometheus-adapter-6fb7598968-k7t29
Nov 07 02:13:20.405 I ns/openshift-monitoring pod/prometheus-k8s-1 Stopping container prometheus
Nov 07 02:13:20.433 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-k7t29 Successfully assigned openshift-monitoring/prometheus-adapter-6fb7598968-k7t29 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:13:20.434 I ns/openshift-monitoring pod/prometheus-k8s-1 Stopping container prom-label-proxy
Nov 07 02:13:20.506 I ns/openshift-monitoring pod/prometheus-k8s-1 Stopping container kube-rbac-proxy
Nov 07 02:13:20.558 W clusteroperator/ingress changed Available to False: IngressUnavailable: Not all ingress controllers are available.
Nov 07 02:13:20.558 W clusteroperator/ingress changed Progressing to True: Reconciling: Not all ingress controllers are available.
Nov 07 02:13:20.661 I ns/openshift-monitoring pod/prometheus-k8s-1 Stopping container prometheus-proxy
Nov 07 02:13:20.878 I ns/openshift-monitoring pod/prometheus-k8s-1 Stopping container thanos-sidecar
Nov 07 02:13:21.065 I ns/openshift-marketplace pod/certified-operators-69589995bf-9l67b Stopping container certified-operators
Nov 07 02:13:21.262 I ns/openshift-monitoring pod/alertmanager-main-0 Stopping container alertmanager-proxy
Nov 07 02:13:21.463 I ns/openshift-monitoring pod/alertmanager-main-0 Stopping container config-reloader
Nov 07 02:13:21.673 I ns/openshift-monitoring pod/alertmanager-main-0 Stopping container alertmanager
Nov 07 02:13:21.870 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-vsdf8 Stopping container prometheus-adapter
Nov 07 02:13:21.958 I ns/openshift-ingress pod/router-default-68c6467955-474p6 Cancelling deletion of Pod openshift-ingress/router-default-68c6467955-474p6
Nov 07 02:13:21.965 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-8z6z2 Cancelling deletion of Pod openshift-monitoring/openshift-state-metrics-6c78647cc7-8z6z2
Nov 07 02:13:21.984 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-k7t29 Cancelling deletion of Pod openshift-monitoring/prometheus-adapter-6fb7598968-k7t29
Nov 07 02:13:22.072 I node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 Node ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 status is now: NodeNotSchedulable
Nov 07 02:13:24.580 W ns/openshift-marketplace pod/community-operators-9567c47bd-z66zz node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:13:24.580 W ns/openshift-marketplace pod/community-operators-9567c47bd-z66zz node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 container=community-operators container stopped being ready
Nov 07 02:13:24.620 W ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-lwfzk node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:13:24.620 W ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-lwfzk node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 container=openshift-state-metrics container stopped being ready
Nov 07 02:13:24.638 E ns/openshift-marketplace pod/certified-operators-69589995bf-9l67b node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 container=certified-operators container exited with code 2 (Error): 
Nov 07 02:13:24.660 E ns/openshift-monitoring pod/telemeter-client-55f55df45-lnv6w node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 container=reload container exited with code 2 (Error): 
Nov 07 02:13:24.660 E ns/openshift-monitoring pod/telemeter-client-55f55df45-lnv6w node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 container=telemeter-client container exited with code 2 (OOMKilled): 
Nov 07 02:13:24.683 W ns/openshift-marketplace pod/redhat-operators-64566c9856-gjdpn node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:13:24.683 W ns/openshift-marketplace pod/redhat-operators-64566c9856-gjdpn node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 container=redhat-operators container stopped being ready
Nov 07 02:13:24.707 E ns/openshift-ingress pod/router-default-68c6467955-pc5mb node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 container=router container exited with code 2 (Error): Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T02:11:14.967Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T02:11:19.961Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T02:11:29.303Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T02:11:34.933Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T02:11:47.860Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T02:12:09.282Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T02:12:14.004Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T02:12:22.759Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T02:12:28.202Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T02:12:35.837Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n2019-11-07T02:13:20.237Z	INFO	router.template	template/router.go:548	router reloaded	{"output": " - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"}\n
Nov 07 02:13:25.508 W ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-vsdf8 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:13:25.508 W ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-vsdf8 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 container=prometheus-adapter container stopped being ready
Nov 07 02:13:25.574 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 container=rules-configmap-reloader container exited with code 2 (Error): 2019/11/07 01:50:58 Watching directory: "/etc/prometheus/rules/prometheus-k8s-rulefiles-0"\n2019/11/07 01:51:08 config map updated\n2019/11/07 01:51:08 error: Post http://localhost:9090/-/reload: dial tcp [::1]:9090: connect: connection refused\n
Nov 07 02:13:25.574 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 container=prometheus-proxy container exited with code 2 (Error): 2019/11/07 01:51:04 provider.go:117: Defaulting client-id to system:serviceaccount:openshift-monitoring:prometheus-k8s\n2019/11/07 01:51:04 provider.go:122: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token\n2019/11/07 01:51:04 provider.go:310: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.\n2019/11/07 01:51:04 oauthproxy.go:200: mapping path "/" => upstream "http://localhost:9090/"\n2019/11/07 01:51:04 oauthproxy.go:221: compiled skip-auth-regex => "^/metrics"\n2019/11/07 01:51:04 oauthproxy.go:227: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:prometheus-k8s\n2019/11/07 01:51:04 oauthproxy.go:237: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> refresh:disabled\n2019/11/07 01:51:04 main.go:154: using htpasswd file /etc/proxy/htpasswd/auth\n2019/11/07 01:51:04 http.go:96: HTTPS: listening on [::]:9091\n
Nov 07 02:13:25.574 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 container=prometheus-config-reloader container exited with code 2 (Error): ts=2019-11-07T01:50:58.538241652Z caller=main.go:85 msg="Starting prometheus-config-reloader version '0.34.0'."\nlevel=info ts=2019-11-07T01:50:58.538389853Z caller=reloader.go:127 msg="started watching config file for changes" in=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml\nlevel=error ts=2019-11-07T01:50:58.541028065Z caller=runutil.go:87 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post http://localhost:9090/-/reload: dial tcp [::1]:9090: connect: connection refused"\nlevel=error ts=2019-11-07T01:51:03.5403237Z caller=runutil.go:87 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post http://localhost:9090/-/reload: dial tcp [::1]:9090: connect: connection refused"\nlevel=error ts=2019-11-07T01:51:08.540782441Z caller=runutil.go:87 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post http://localhost:9090/-/reload: dial tcp [::1]:9090: connect: connection refused"\nlevel=info ts=2019-11-07T01:51:13.860263968Z caller=reloader.go:258 msg="Prometheus reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml rule_dirs=\nlevel=info ts=2019-11-07T01:52:35.855807061Z caller=reloader.go:258 msg="Prometheus reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml rule_dirs=\n
Nov 07 02:13:25.583 E ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 container=config-reloader container exited with code 2 (Error): 2019/11/07 01:47:15 Watching directory: "/etc/alertmanager/config"\n
Nov 07 02:13:25.583 E ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 container=alertmanager-proxy container exited with code 2 (Error): 2019/11/07 01:47:15 provider.go:117: Defaulting client-id to system:serviceaccount:openshift-monitoring:alertmanager-main\n2019/11/07 01:47:15 provider.go:122: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token\n2019/11/07 01:47:15 provider.go:310: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.\n2019/11/07 01:47:15 oauthproxy.go:200: mapping path "/" => upstream "http://localhost:9093/"\n2019/11/07 01:47:15 oauthproxy.go:221: compiled skip-auth-regex => "^/metrics"\n2019/11/07 01:47:15 oauthproxy.go:227: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:alertmanager-main\n2019/11/07 01:47:15 oauthproxy.go:237: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> refresh:disabled\n2019/11/07 01:47:15 http.go:96: HTTPS: listening on [::]:9095\n
Nov 07 02:13:26.092 I ns/openshift-dns pod/dns-default-9qpmp Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:614764223d37c466a211f456a9d02c5881d947e1f05d541689dc34a3ec19acf2"
Nov 07 02:13:26.352 I ns/openshift-image-registry pod/node-ca-6jr5f Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3191fc23d0d7e820100498c5085592219a747743f06fc059436945df6ab3e8ee"
Nov 07 02:13:28.004 I ns/openshift-monitoring pod/telemeter-client-55f55df45-r9w6d Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24ef740f2376db3bc5e0f8ba14da8ee6ad52aa2dce357781b6fccc9081bf008f"
Nov 07 02:13:28.200 I ns/openshift-marketplace pod/redhat-operators-64566c9856-zhk47 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:448858c7b1f70b8c0955965ef9970b59504b220bf03b760203305c81e5f606da"
Nov 07 02:13:28.506 I ns/openshift-ingress pod/router-default-68c6467955-474p6 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ece12ae9ebfcce0c5b8d8a2eff009a1706a070e9021e96112dbbe6711795a35"
Nov 07 02:13:28.554 I ns/openshift-marketplace pod/community-operators-9567c47bd-5s9sf Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:448858c7b1f70b8c0955965ef9970b59504b220bf03b760203305c81e5f606da"
Nov 07 02:13:28.754 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-k7t29 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4753bca36792206b5e9df9612f695fc2c3a80a54d56b01a10f774dc1b62899be"
Nov 07 02:13:29.037 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-8z6z2 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5366da9d7c15f912b609cdbb2851d01ea863fb22d452081914d923d8208c39c8" already present on machine
Nov 07 02:13:29.390 I ns/openshift-marketplace pod/certified-operators-69589995bf-bhhjn Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:448858c7b1f70b8c0955965ef9970b59504b220bf03b760203305c81e5f606da"
Nov 07 02:13:29.840 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-8z6z2 Created container kube-rbac-proxy-main
Nov 07 02:13:29.973 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-8z6z2 Started container kube-rbac-proxy-main
Nov 07 02:13:29.987 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-8z6z2 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5366da9d7c15f912b609cdbb2851d01ea863fb22d452081914d923d8208c39c8" already present on machine
Nov 07 02:13:30.418 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-8z6z2 Created container kube-rbac-proxy-self
Nov 07 02:13:30.817 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-8z6z2 Started container kube-rbac-proxy-self
Nov 07 02:13:31.110 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-8z6z2 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9b0c04ed78fe07c8c7ba6efec2be18908955710281189051eaea6942d424a56f"
Nov 07 02:13:31.142 W ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-lwfzk node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 pod has been pending longer than a minute
Nov 07 02:13:31.142 W ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-vsdf8 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 pod has been pending longer than a minute
Nov 07 02:13:31.142 W ns/openshift-marketplace pod/redhat-operators-64566c9856-gjdpn node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 pod has been pending longer than a minute
Nov 07 02:13:31.142 W ns/openshift-marketplace pod/community-operators-9567c47bd-z66zz node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 pod has been pending longer than a minute
Nov 07 02:13:33.821 I ns/openshift-monitoring pod/telemeter-client-55f55df45-r9w6d Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24ef740f2376db3bc5e0f8ba14da8ee6ad52aa2dce357781b6fccc9081bf008f"
Nov 07 02:13:34.172 I ns/openshift-cluster-node-tuning-operator pod/tuned-jbbvc Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3207589b466325f1ee9decd9f41416ca20597d409d6071c3614ce2f5e9626368"
Nov 07 02:13:34.391 W ns/openshift-ingress pod/router-default-68c6467955-pc5mb node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 deleted
Nov 07 02:13:34.417 W ns/openshift-marketplace pod/redhat-operators-64566c9856-gjdpn node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 deleted
Nov 07 02:13:34.453 W ns/openshift-marketplace pod/community-operators-9567c47bd-z66zz node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 deleted
Nov 07 02:13:34.505 W ns/openshift-marketplace pod/certified-operators-69589995bf-9l67b node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 deleted
Nov 07 02:13:34.544 I ns/openshift-cluster-node-tuning-operator pod/tuned-jbbvc Created container tuned
Nov 07 02:13:34.555 W ns/openshift-monitoring pod/telemeter-client-55f55df45-lnv6w node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 deleted
Nov 07 02:13:34.590 W ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-vsdf8 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 deleted
Nov 07 02:13:34.640 W ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 deleted
Nov 07 02:13:34.689 W ns/openshift-monitoring pod/prometheus-k8s-1 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 deleted
Nov 07 02:13:34.718 I ns/openshift-monitoring pod/alertmanager-main-0 node/ created
Nov 07 02:13:34.732 I ns/openshift-monitoring statefulset/alertmanager-main create Pod alertmanager-main-0 in StatefulSet alertmanager-main successful
Nov 07 02:13:34.743 I ns/openshift-cluster-node-tuning-operator pod/tuned-jbbvc Started container tuned
Nov 07 02:13:34.753 I ns/openshift-monitoring pod/alertmanager-main-0 Successfully assigned openshift-monitoring/alertmanager-main-0 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g
Nov 07 02:13:34.754 W ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-lwfzk node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 deleted
Nov 07 02:13:34.826 I ns/openshift-monitoring pod/prometheus-k8s-1 node/ created
Nov 07 02:13:34.839 I ns/openshift-monitoring statefulset/prometheus-k8s create Pod prometheus-k8s-1 in StatefulSet prometheus-k8s successful
Nov 07 02:13:34.861 I ns/openshift-monitoring pod/prometheus-k8s-1 Successfully assigned openshift-monitoring/prometheus-k8s-1 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj
Nov 07 02:13:35.117 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 Node "ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8" drained
Nov 07 02:13:35.152 I ns/openshift-monitoring pod/telemeter-client-55f55df45-r9w6d Created container telemeter-client
Nov 07 02:13:35.228 I ns/openshift-monitoring pod/telemeter-client-55f55df45-r9w6d Started container telemeter-client
Nov 07 02:13:35.239 I ns/openshift-monitoring pod/telemeter-client-55f55df45-r9w6d Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db10cd1f5efd9e9e6f2b0eea0a665f53399e25de374cf916ad7d8141f8ac643d"
Nov 07 02:13:39.429 I ns/openshift-marketplace pod/redhat-operators-64566c9856-zhk47 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:448858c7b1f70b8c0955965ef9970b59504b220bf03b760203305c81e5f606da"
Nov 07 02:13:39.673 I ns/openshift-marketplace pod/redhat-operators-64566c9856-zhk47 Created container redhat-operators
Nov 07 02:13:39.772 I ns/openshift-marketplace pod/redhat-operators-64566c9856-zhk47 Started container redhat-operators
Nov 07 02:13:40.785 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 DeleteError: failed to delete machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8": failed to delete machine: compute.VirtualMachinesDeleteFuture: asynchronous operation has not completed
Nov 07 02:13:40.853 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 Node "ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9" drained (2 times)
Nov 07 02:13:40.863 I ns/openshift-monitoring pod/telemeter-client-55f55df45-r9w6d Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db10cd1f5efd9e9e6f2b0eea0a665f53399e25de374cf916ad7d8141f8ac643d"
Nov 07 02:13:41.080 I ns/openshift-monitoring pod/telemeter-client-55f55df45-r9w6d Created container reload
Nov 07 02:13:41.110 I ns/openshift-monitoring pod/telemeter-client-55f55df45-r9w6d Started container reload
Nov 07 02:13:41.117 I ns/openshift-monitoring pod/telemeter-client-55f55df45-r9w6d Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5366da9d7c15f912b609cdbb2851d01ea863fb22d452081914d923d8208c39c8" already present on machine
Nov 07 02:13:41.383 I ns/openshift-monitoring pod/telemeter-client-55f55df45-r9w6d Created container kube-rbac-proxy
Nov 07 02:13:41.424 I ns/openshift-monitoring pod/telemeter-client-55f55df45-r9w6d Started container kube-rbac-proxy
Nov 07 02:13:41.561 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-8z6z2 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9b0c04ed78fe07c8c7ba6efec2be18908955710281189051eaea6942d424a56f"
Nov 07 02:13:41.568 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-k7t29 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4753bca36792206b5e9df9612f695fc2c3a80a54d56b01a10f774dc1b62899be"
Nov 07 02:13:41.873 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-8z6z2 Created container openshift-state-metrics
Nov 07 02:13:41.931 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-8z6z2 Started container openshift-state-metrics
Nov 07 02:13:41.950 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-k7t29 Created container prometheus-adapter
Nov 07 02:13:41.979 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-k7t29 Started container prometheus-adapter
Nov 07 02:13:42.702 I ns/openshift-monitoring pod/prometheus-k8s-1 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce5cf60d260dd8be75a016c30c7724b254f607573b9388da266d78fb3ac49757"
Nov 07 02:13:42.778 I ns/openshift-marketplace pod/community-operators-9567c47bd-5s9sf Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:448858c7b1f70b8c0955965ef9970b59504b220bf03b760203305c81e5f606da"
Nov 07 02:13:42.792 I ns/openshift-marketplace pod/certified-operators-69589995bf-bhhjn Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:448858c7b1f70b8c0955965ef9970b59504b220bf03b760203305c81e5f606da"
Nov 07 02:13:43.093 I ns/openshift-marketplace pod/community-operators-9567c47bd-5s9sf Created container community-operators
Nov 07 02:13:43.134 I ns/openshift-marketplace pod/certified-operators-69589995bf-bhhjn Created container certified-operators
Nov 07 02:13:43.149 I ns/openshift-marketplace pod/community-operators-9567c47bd-5s9sf Started container community-operators
Nov 07 02:13:43.180 I ns/openshift-marketplace pod/certified-operators-69589995bf-bhhjn Started container certified-operators
Nov 07 02:13:43.285 I ns/openshift-monitoring pod/alertmanager-main-0 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78ba454f09557364dabd918265784493287ed242fb51ac2ffde695ae78823736"
Nov 07 02:13:46.120 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 DeleteError: failed to delete machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9": failed to delete machine: compute.VirtualMachinesDeleteFuture: asynchronous operation has not completed (2 times)
Nov 07 02:13:46.221 W ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g graceful deletion within 120s
Nov 07 02:13:46.227 W ns/openshift-marketplace pod/certified-operators-69589995bf-bhhjn node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g graceful deletion within 30s
Nov 07 02:13:46.232 W ns/openshift-marketplace pod/community-operators-9567c47bd-5s9sf node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g graceful deletion within 30s
Nov 07 02:13:46.247 I ns/openshift-marketplace pod/certified-operators-69589995bf-bhhjn Stopping container certified-operators
Nov 07 02:13:46.264 I ns/openshift-marketplace pod/community-operators-9567c47bd-5s9sf Stopping container community-operators
Nov 07 02:13:46.266 E ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g container=config-reloader container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Nov 07 02:13:46.266 E ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g container=alertmanager container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Nov 07 02:13:46.266 E ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g container=alertmanager-proxy container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Nov 07 02:13:46.291 I ns/openshift-marketplace pod/certified-operators-69589995bf-wtx68 node/ created
Nov 07 02:13:46.307 I ns/openshift-marketplace pod/community-operators-9567c47bd-6p48d node/ created
Nov 07 02:13:46.312 I ns/openshift-marketplace replicaset/certified-operators-69589995bf Created pod: certified-operators-69589995bf-wtx68
Nov 07 02:13:46.332 I ns/openshift-marketplace replicaset/community-operators-9567c47bd Created pod: community-operators-9567c47bd-6p48d
Nov 07 02:13:46.343 I ns/openshift-marketplace pod/certified-operators-69589995bf-wtx68 Successfully assigned openshift-marketplace/certified-operators-69589995bf-wtx68 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:13:46.355 I ns/openshift-marketplace pod/community-operators-9567c47bd-6p48d Successfully assigned openshift-marketplace/community-operators-9567c47bd-6p48d to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:13:48.290 W ns/openshift-marketplace pod/certified-operators-69589995bf-bhhjn Readiness probe errored: rpc error: code = Unknown desc = container is not created or running
Nov 07 02:13:48.675 I ns/openshift-monitoring pod/prometheus-k8s-1 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce5cf60d260dd8be75a016c30c7724b254f607573b9388da266d78fb3ac49757"
Nov 07 02:13:48.911 I ns/openshift-monitoring pod/prometheus-k8s-1 Created container prometheus
Nov 07 02:13:48.957 I ns/openshift-monitoring pod/prometheus-k8s-1 Started container prometheus
Nov 07 02:13:48.965 I ns/openshift-monitoring pod/prometheus-k8s-1 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9a3a81674e3aa3d54fc056ba4907a2f48b867c375d8fe86e05d9e1c99489eade"
Nov 07 02:13:49.715 W ns/openshift-marketplace pod/certified-operators-69589995bf-bhhjn Liveness probe errored: rpc error: code = Unknown desc = container is not created or running
Nov 07 02:13:51.555 W ns/openshift-marketplace pod/community-operators-9567c47bd-5s9sf Liveness probe errored: rpc error: code = Unknown desc = specified container not found: 4e93e75b08f703d4951f5a17cf7da7bfe70d1af7df5a4efd59119a93a39877db
Nov 07 02:13:51.599 I ns/openshift-monitoring pod/prometheus-k8s-1 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9a3a81674e3aa3d54fc056ba4907a2f48b867c375d8fe86e05d9e1c99489eade"
Nov 07 02:13:51.953 I ns/openshift-monitoring pod/prometheus-k8s-1 Created container prometheus-config-reloader
Nov 07 02:13:51.990 I ns/openshift-monitoring pod/prometheus-k8s-1 Started container prometheus-config-reloader
Nov 07 02:13:52.005 I ns/openshift-monitoring pod/prometheus-k8s-1 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db10cd1f5efd9e9e6f2b0eea0a665f53399e25de374cf916ad7d8141f8ac643d" already present on machine
Nov 07 02:13:52.360 I ns/openshift-monitoring pod/prometheus-k8s-1 Created container rules-configmap-reloader
Nov 07 02:13:52.401 I ns/openshift-monitoring pod/prometheus-k8s-1 Started container rules-configmap-reloader
Nov 07 02:13:52.427 I ns/openshift-monitoring pod/prometheus-k8s-1 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf187f44c7273f1328d614d1dcd9f701c5f66cf8849143e34c8d1f88d5e2ce55"
Nov 07 02:13:53.197 W ns/openshift-marketplace pod/redhat-operators-64566c9856-zhk47 Readiness probe errored: rpc error: code = Unknown desc = command error: command timed out, stdout: , stderr: , exit code -1
Nov 07 02:13:53.249 W ns/openshift-marketplace pod/community-operators-9567c47bd-5s9sf Readiness probe errored: rpc error: code = Unknown desc = specified container not found: 4e93e75b08f703d4951f5a17cf7da7bfe70d1af7df5a4efd59119a93a39877db
Nov 07 02:13:53.928 W ns/openshift-marketplace pod/redhat-operators-64566c9856-zhk47 Liveness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n
Nov 07 02:13:54.691 I ns/openshift-marketplace pod/community-operators-9567c47bd-6p48d Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:448858c7b1f70b8c0955965ef9970b59504b220bf03b760203305c81e5f606da"
Nov 07 02:13:54.747 I ns/openshift-marketplace pod/certified-operators-69589995bf-wtx68 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:448858c7b1f70b8c0955965ef9970b59504b220bf03b760203305c81e5f606da"
Nov 07 02:13:57.813 W ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g deleted
Nov 07 02:13:57.844 I ns/openshift-monitoring pod/alertmanager-main-0 node/ created
Nov 07 02:13:57.866 I ns/openshift-monitoring statefulset/alertmanager-main create Pod alertmanager-main-0 in StatefulSet alertmanager-main successful (2 times)
Nov 07 02:13:57.881 I ns/openshift-monitoring pod/alertmanager-main-0 Successfully assigned openshift-monitoring/alertmanager-main-0 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:13:58.278 W ns/openshift-marketplace pod/certified-operators-69589995bf-bhhjn Readiness probe errored: rpc error: code = Unknown desc = container is not created or running (2 times)
Nov 07 02:13:58.806 I ns/openshift-monitoring pod/prometheus-k8s-1 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf187f44c7273f1328d614d1dcd9f701c5f66cf8849143e34c8d1f88d5e2ce55"
Nov 07 02:13:59.003 I ns/openshift-monitoring pod/prometheus-k8s-1 Created container thanos-sidecar
Nov 07 02:13:59.043 I ns/openshift-monitoring pod/prometheus-k8s-1 Started container thanos-sidecar
Nov 07 02:13:59.053 I ns/openshift-monitoring pod/prometheus-k8s-1 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f711682524b64033a4ce9193bd2cd9621e08b4bae44cffffb4e397a43a8f945" already present on machine
Nov 07 02:13:59.280 I ns/openshift-monitoring pod/prometheus-k8s-1 Created container prometheus-proxy
Nov 07 02:13:59.323 I ns/openshift-monitoring pod/prometheus-k8s-1 Started container prometheus-proxy
Nov 07 02:13:59.336 I ns/openshift-monitoring pod/prometheus-k8s-1 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5366da9d7c15f912b609cdbb2851d01ea863fb22d452081914d923d8208c39c8" already present on machine
Nov 07 02:13:59.555 I ns/openshift-monitoring pod/prometheus-k8s-1 Created container kube-rbac-proxy
Nov 07 02:13:59.603 I ns/openshift-monitoring pod/prometheus-k8s-1 Started container kube-rbac-proxy
Nov 07 02:13:59.611 I ns/openshift-monitoring pod/prometheus-k8s-1 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8aefd089984c2e7d9dd7a162bd8e23bb9e492b494f6c285c2a8272f8230ff9e"
Nov 07 02:13:59.697 W ns/openshift-marketplace pod/certified-operators-69589995bf-bhhjn Liveness probe errored: rpc error: code = Unknown desc = container is not created or running (2 times)
Nov 07 02:14:02.403 I ns/openshift-monitoring pod/prometheus-k8s-1 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8aefd089984c2e7d9dd7a162bd8e23bb9e492b494f6c285c2a8272f8230ff9e"
Nov 07 02:14:02.622 I ns/openshift-monitoring pod/prometheus-k8s-1 Created container prom-label-proxy
Nov 07 02:14:02.657 I ns/openshift-monitoring pod/prometheus-k8s-1 Started container prom-label-proxy
Nov 07 02:14:03.203 W ns/openshift-marketplace pod/redhat-operators-64566c9856-zhk47 Readiness probe errored: rpc error: code = Unknown desc = command error: command timed out, stdout: , stderr: , exit code -1 (2 times)
Nov 07 02:14:03.495 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj container=prometheus container exited with code 1 (Error): caller=main.go:657 msg="Starting TSDB ..."\nlevel=info ts=2019-11-07T02:13:48.963Z caller=web.go:450 component=web msg="Start listening for connections" address=127.0.0.1:9090\nlevel=info ts=2019-11-07T02:13:48.971Z caller=head.go:514 component=tsdb msg="replaying WAL, this may take awhile"\nlevel=info ts=2019-11-07T02:13:48.971Z caller=head.go:562 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0\nlevel=info ts=2019-11-07T02:13:48.972Z caller=main.go:672 fs_type=XFS_SUPER_MAGIC\nlevel=info ts=2019-11-07T02:13:48.972Z caller=main.go:673 msg="TSDB started"\nlevel=info ts=2019-11-07T02:13:48.972Z caller=main.go:743 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml\nlevel=info ts=2019-11-07T02:13:48.973Z caller=main.go:526 msg="Stopping scrape discovery manager..."\nlevel=info ts=2019-11-07T02:13:48.973Z caller=main.go:540 msg="Stopping notify discovery manager..."\nlevel=info ts=2019-11-07T02:13:48.973Z caller=main.go:562 msg="Stopping scrape manager..."\nlevel=info ts=2019-11-07T02:13:48.973Z caller=main.go:536 msg="Notify discovery manager stopped"\nlevel=info ts=2019-11-07T02:13:48.973Z caller=main.go:522 msg="Scrape discovery manager stopped"\nlevel=info ts=2019-11-07T02:13:48.973Z caller=manager.go:814 component="rule manager" msg="Stopping rule manager..."\nlevel=info ts=2019-11-07T02:13:48.973Z caller=manager.go:820 component="rule manager" msg="Rule manager stopped"\nlevel=info ts=2019-11-07T02:13:48.973Z caller=main.go:556 msg="Scrape manager stopped"\nlevel=info ts=2019-11-07T02:13:48.977Z caller=notifier.go:602 component=notifier msg="Stopping notification manager..."\nlevel=info ts=2019-11-07T02:13:48.977Z caller=main.go:727 msg="Notifier manager stopped"\nlevel=error ts=2019-11-07
Nov 07 02:14:03.976 I ns/openshift-monitoring pod/alertmanager-main-0 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78ba454f09557364dabd918265784493287ed242fb51ac2ffde695ae78823736"
Nov 07 02:14:03.984 W ns/openshift-monitoring pod/alertmanager-main-0 Error: cannot find volume "config-volume" to mount into container "alertmanager"
Nov 07 02:14:03.998 I ns/openshift-monitoring pod/alertmanager-main-0 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db10cd1f5efd9e9e6f2b0eea0a665f53399e25de374cf916ad7d8141f8ac643d"
Nov 07 02:14:04.357 W ns/openshift-marketplace pod/community-operators-9567c47bd-5s9sf node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:14:04.372 W ns/openshift-marketplace pod/certified-operators-69589995bf-bhhjn node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:14:06.390 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 Node "ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8" drained (2 times)
Nov 07 02:14:06.744 I ns/openshift-monitoring pod/alertmanager-main-0 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db10cd1f5efd9e9e6f2b0eea0a665f53399e25de374cf916ad7d8141f8ac643d"
Nov 07 02:14:06.751 W ns/openshift-monitoring pod/alertmanager-main-0 Error: cannot find volume "config-volume" to mount into container "config-reloader"
Nov 07 02:14:06.762 I ns/openshift-monitoring pod/alertmanager-main-0 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f711682524b64033a4ce9193bd2cd9621e08b4bae44cffffb4e397a43a8f945" already present on machine
Nov 07 02:14:06.769 W ns/openshift-monitoring pod/alertmanager-main-0 Error: cannot find volume "secret-alertmanager-main-tls" to mount into container "alertmanager-proxy"
Nov 07 02:14:06.975 W node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 condition DiskPressure changed
Nov 07 02:14:06.975 W node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 condition MemoryPressure changed
Nov 07 02:14:06.975 W node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 condition PIDPressure changed
Nov 07 02:14:06.975 W node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 condition Ready changed
Nov 07 02:14:06.990 I node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 Node ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 status is now: NodeNotReady
Nov 07 02:14:07.156 W clusteroperator/network changed Progressing to True: Deploying: DaemonSet "openshift-multus/multus" is not available (awaiting 1 nodes)
Nov 07 02:14:07.783 W ns/openshift-marketplace pod/community-operators-9567c47bd-5s9sf node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g deleted
Nov 07 02:14:07.826 W ns/openshift-marketplace pod/certified-operators-69589995bf-bhhjn node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g deleted
Nov 07 02:14:08.049 W ns/openshift-marketplace pod/redhat-operators-64566c9856-zhk47 Liveness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n (2 times)
Nov 07 02:14:08.241 I ns/openshift-marketplace pod/community-operators-9567c47bd-6p48d Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:448858c7b1f70b8c0955965ef9970b59504b220bf03b760203305c81e5f606da"
Nov 07 02:14:08.250 I ns/openshift-marketplace pod/certified-operators-69589995bf-wtx68 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:448858c7b1f70b8c0955965ef9970b59504b220bf03b760203305c81e5f606da"
Nov 07 02:14:08.312 I ns/openshift-monitoring pod/alertmanager-main-0 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78ba454f09557364dabd918265784493287ed242fb51ac2ffde695ae78823736"
Nov 07 02:14:08.497 W ns/openshift-monitoring pod/prometheus-k8s-1 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj container=prometheus container restarted
Nov 07 02:14:08.622 I ns/openshift-marketplace pod/certified-operators-69589995bf-wtx68 Created container certified-operators
Nov 07 02:14:08.632 I ns/openshift-marketplace pod/community-operators-9567c47bd-6p48d Created container community-operators
Nov 07 02:14:08.693 I ns/openshift-marketplace pod/certified-operators-69589995bf-wtx68 Started container certified-operators
Nov 07 02:14:08.707 I ns/openshift-marketplace pod/community-operators-9567c47bd-6p48d Started container community-operators
Nov 07 02:14:11.514 I node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 Node ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 event: DeletingNode
Nov 07 02:14:11.522 W node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 deleted
Nov 07 02:14:12.153 I node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 Node ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 event: Removing Node ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 from Controller
Nov 07 02:14:12.299 W ns/openshift-machine-config-operator pod/machine-config-daemon-lkzwf node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 graceful deletion within 0s
Nov 07 02:14:12.316 W ns/openshift-machine-config-operator pod/machine-config-daemon-lkzwf node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 deleted
Nov 07 02:14:12.331 W ns/openshift-monitoring pod/node-exporter-pb7pf node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 graceful deletion within 0s
Nov 07 02:14:12.351 W ns/openshift-monitoring pod/node-exporter-pb7pf node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 deleted
Nov 07 02:14:12.366 W ns/openshift-sdn pod/sdn-jzr6w node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 graceful deletion within 0s
Nov 07 02:14:12.381 W ns/openshift-sdn pod/sdn-jzr6w node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 deleted
Nov 07 02:14:12.398 W ns/openshift-multus pod/multus-pdnvr node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 graceful deletion within 0s
Nov 07 02:14:12.419 W ns/openshift-multus pod/multus-pdnvr node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 deleted
Nov 07 02:14:12.432 W ns/openshift-sdn pod/ovs-rw7fq node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 graceful deletion within 0s
Nov 07 02:14:12.460 W ns/openshift-sdn pod/ovs-rw7fq node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 deleted
Nov 07 02:14:12.463 W clusteroperator/network changed Progressing to False
Nov 07 02:14:12.471 W ns/openshift-cluster-node-tuning-operator pod/tuned-6f2s7 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 graceful deletion within 0s
Nov 07 02:14:12.481 W ns/openshift-cluster-node-tuning-operator pod/tuned-6f2s7 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 deleted
Nov 07 02:14:12.498 W ns/openshift-image-registry pod/node-ca-b6p4w node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 graceful deletion within 0s
Nov 07 02:14:12.510 W ns/openshift-image-registry pod/node-ca-b6p4w node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 deleted
Nov 07 02:14:12.523 W ns/openshift-dns pod/dns-default-c628x node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 graceful deletion within 0s
Nov 07 02:14:12.541 W ns/openshift-dns pod/dns-default-c628x node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 deleted
Nov 07 02:14:13.381 W ns/openshift-ingress service/router-default Error updating load balancer with new hosts map[ci-op-pg7mtgss-09f59-ng62w-master-0:{} ci-op-pg7mtgss-09f59-ng62w-master-1:{} ci-op-pg7mtgss-09f59-ng62w-master-2:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-z8fpv:{}]: ensure(openshift-ingress/router-default): backendPoolID(/subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-pg7mtgss-09f59-ng62w-rg/providers/Microsoft.Network/loadBalancers/ci-op-pg7mtgss-09f59-ng62w/backendAddressPools/ci-op-pg7mtgss-09f59-ng62w) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:NicGet"
Nov 07 02:14:13.492 W ns/openshift-marketplace pod/redhat-operators-64566c9856-zhk47 Readiness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n
Nov 07 02:14:13.941 W ns/openshift-marketplace pod/redhat-operators-64566c9856-zhk47 Liveness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n (3 times)
Nov 07 02:14:15.675 W ns/openshift-marketplace pod/community-operators-9567c47bd-6p48d Liveness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n
Nov 07 02:14:20.726 W ns/openshift-marketplace pod/certified-operators-69589995bf-wtx68 Liveness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n
Nov 07 02:14:21.597 W ns/openshift-marketplace pod/community-operators-9567c47bd-6p48d Readiness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n
Nov 07 02:14:21.881 I ns/openshift-ingress service/router-default Ensuring load balancer (11 times)
Nov 07 02:14:23.218 W ns/openshift-marketplace pod/certified-operators-69589995bf-wtx68 Readiness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n
Nov 07 02:14:25.685 W ns/openshift-marketplace pod/community-operators-9567c47bd-6p48d Liveness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n (2 times)
Nov 07 02:14:27.161 W node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 condition DiskPressure changed
Nov 07 02:14:27.161 W node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 condition MemoryPressure changed
Nov 07 02:14:27.161 W node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 condition PIDPressure changed
Nov 07 02:14:27.161 W node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 condition Ready changed
Nov 07 02:14:27.170 I node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 Node ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 status is now: NodeNotReady
Nov 07 02:14:27.322 W clusteroperator/network changed Progressing to True: Deploying: DaemonSet "openshift-multus/multus" is not available (awaiting 1 nodes)
Nov 07 02:14:31.142 W node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 node is not ready
Nov 07 02:14:31.142 - 270s  W ns/openshift-dns pod/dns-default-9qpmp node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 pod has been pending longer than a minute
Nov 07 02:14:31.142 - 270s  W ns/openshift-machine-config-operator pod/machine-config-daemon-l5rhl node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 pod has been pending longer than a minute
Nov 07 02:14:31.142 - 284s  W ns/openshift-ingress pod/router-default-68c6467955-474p6 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 pod has been pending longer than a minute
Nov 07 02:14:31.142 - 419s  W ns/openshift-image-registry pod/node-ca-6jr5f node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 pod has been pending longer than a minute
Nov 07 02:14:31.573 I node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 Node ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 event: DeletingNode
Nov 07 02:14:31.582 W ns/openshift-marketplace pod/community-operators-9567c47bd-6p48d Readiness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n (2 times)
Nov 07 02:14:31.583 W node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 deleted
Nov 07 02:14:31.933 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 DeleteError: failed to delete machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8": failed to delete OS disk: result error: compute.DisksDeleteFuture: asynchronous operation has not completed
Nov 07 02:14:32.338 I node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 Node ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 event: Removing Node ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 from Controller
Nov 07 02:14:32.441 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 DeleteError: failed to delete machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9": failed to delete OS disk: result error: compute.DisksDeleteFuture: asynchronous operation has not completed
Nov 07 02:14:32.495 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g Node "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g" drained
Nov 07 02:14:32.562 W ns/openshift-multus pod/multus-w6p7n node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 graceful deletion within 0s
Nov 07 02:14:32.577 W ns/openshift-multus pod/multus-w6p7n node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 deleted
Nov 07 02:14:32.598 W ns/openshift-sdn pod/sdn-h8gfl node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 graceful deletion within 0s
Nov 07 02:14:32.616 W ns/openshift-sdn pod/sdn-h8gfl node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 deleted
Nov 07 02:14:32.630 W ns/openshift-monitoring pod/node-exporter-lcphw node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 graceful deletion within 0s
Nov 07 02:14:32.657 W ns/openshift-monitoring pod/node-exporter-lcphw node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 deleted
Nov 07 02:14:32.669 W ns/openshift-image-registry pod/node-ca-qxrtf node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 graceful deletion within 0s
Nov 07 02:14:32.683 W ns/openshift-image-registry pod/node-ca-qxrtf node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 deleted
Nov 07 02:14:32.702 W ns/openshift-cluster-node-tuning-operator pod/tuned-4skx9 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 graceful deletion within 0s
Nov 07 02:14:32.711 W ns/openshift-cluster-node-tuning-operator pod/tuned-4skx9 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 deleted
Nov 07 02:14:32.728 W ns/openshift-machine-config-operator pod/machine-config-daemon-9v2qn node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 graceful deletion within 0s
Nov 07 02:14:32.735 W ns/openshift-machine-config-operator pod/machine-config-daemon-9v2qn node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 deleted
Nov 07 02:14:32.755 W ns/openshift-sdn pod/ovs-mhlb5 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 graceful deletion within 0s
Nov 07 02:14:32.769 W ns/openshift-sdn pod/ovs-mhlb5 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 deleted
Nov 07 02:14:32.798 W ns/openshift-dns pod/dns-default-rjrlh node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 graceful deletion within 0s
Nov 07 02:14:32.809 W clusteroperator/network changed Progressing to False
Nov 07 02:14:32.812 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g DeleteError: failed to delete machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g": failed to delete machine: compute.VirtualMachinesDeleteFuture: asynchronous operation has not completed
Nov 07 02:14:32.822 W ns/openshift-dns pod/dns-default-rjrlh node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 deleted
Nov 07 02:14:41.730 W ns/openshift-marketplace pod/community-operators-9567c47bd-6p48d Liveness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n (3 times)
Nov 07 02:14:41.886 W ns/openshift-marketplace pod/community-operators-9567c47bd-6p48d Readiness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n (3 times)
Nov 07 02:14:43.602 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 Deleted machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8"
Nov 07 02:14:44.766 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g Node "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g" drained (2 times)
Nov 07 02:14:45.713 W ns/openshift-marketplace pod/community-operators-9567c47bd-6p48d Liveness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n (4 times)
Nov 07 02:14:50.420 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g DeleteError: failed to delete machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g": failed to delete machine: compute.VirtualMachinesDeleteFuture: asynchronous operation has not completed (2 times)
Nov 07 02:14:51.601 W ns/openshift-marketplace pod/community-operators-9567c47bd-6p48d Readiness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n (4 times)
Nov 07 02:15:00.866 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8 Deleted machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-hrgq8" (2 times)
Nov 07 02:15:00.953 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g Node "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g" drained (3 times)
Nov 07 02:15:01.142 - 45s   W ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 pod has been pending longer than a minute
Nov 07 02:15:12.342 W node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g condition DiskPressure changed
Nov 07 02:15:12.342 W node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g condition MemoryPressure changed
Nov 07 02:15:12.342 W node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g condition PIDPressure changed
Nov 07 02:15:12.342 W node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g condition Ready changed
Nov 07 02:15:12.360 I node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g Node ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g status is now: NodeNotReady
Nov 07 02:15:12.505 W clusteroperator/network changed Progressing to True: Deploying: DaemonSet "openshift-multus/multus" is not available (awaiting 1 nodes)
Nov 07 02:15:16.142 - 59s   W node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g node is not ready
Nov 07 02:15:16.396 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g DeleteError: failed to delete machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g": failed to delete machine: compute.VirtualMachinesDeleteFuture: asynchronous operation has not completed (3 times)
Nov 07 02:15:22.920 I ns/openshift-ingress service/router-default Ensured load balancer
Nov 07 02:15:26.792 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9 Deleted machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-2h5n9"
Nov 07 02:15:26.873 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g Node "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g" drained (4 times)
Nov 07 02:15:27.441 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g DeleteError: failed to delete machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g": failed to delete machine: compute.VirtualMachinesDeleteFuture: asynchronous operation has not completed (4 times)
Nov 07 02:15:36.470 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g Node "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g" drained (5 times)
Nov 07 02:15:37.091 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g DeleteError: failed to delete machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g": failed to delete machine: compute.VirtualMachinesDeleteFuture: asynchronous operation has not completed (5 times)
Nov 07 02:15:47.125 I ns/openshift-monitoring pod/alertmanager-main-0 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78ba454f09557364dabd918265784493287ed242fb51ac2ffde695ae78823736"
Nov 07 02:15:47.338 I ns/openshift-monitoring pod/alertmanager-main-0 Created container alertmanager
Nov 07 02:15:47.383 I ns/openshift-monitoring pod/alertmanager-main-0 Started container alertmanager
Nov 07 02:15:47.390 I ns/openshift-monitoring pod/alertmanager-main-0 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db10cd1f5efd9e9e6f2b0eea0a665f53399e25de374cf916ad7d8141f8ac643d"
Nov 07 02:15:51.500 I ns/openshift-monitoring pod/alertmanager-main-0 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db10cd1f5efd9e9e6f2b0eea0a665f53399e25de374cf916ad7d8141f8ac643d"
Nov 07 02:15:51.884 I ns/openshift-monitoring pod/alertmanager-main-0 Created container config-reloader
Nov 07 02:15:51.915 I ns/openshift-monitoring pod/alertmanager-main-0 Started container config-reloader
Nov 07 02:15:51.925 I ns/openshift-monitoring pod/alertmanager-main-0 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f711682524b64033a4ce9193bd2cd9621e08b4bae44cffffb4e397a43a8f945"
Nov 07 02:15:53.770 W ns/openshift-ingress service/router-default Error updating load balancer with new hosts map[ci-op-pg7mtgss-09f59-ng62w-master-0:{} ci-op-pg7mtgss-09f59-ng62w-master-1:{} ci-op-pg7mtgss-09f59-ng62w-master-2:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-z8fpv:{}]: ensure(openshift-ingress/router-default): backendPoolID(/subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-pg7mtgss-09f59-ng62w-rg/providers/Microsoft.Network/loadBalancers/ci-op-pg7mtgss-09f59-ng62w/backendAddressPools/ci-op-pg7mtgss-09f59-ng62w) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:NicGet"
Nov 07 02:15:57.226 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g Node "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g" drained (6 times)
Nov 07 02:15:57.675 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g DeleteError: failed to delete machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g": failed to delete machine: compute.VirtualMachinesDeleteFuture: asynchronous operation has not completed (6 times)
Nov 07 02:15:57.914 I ns/openshift-monitoring pod/alertmanager-main-0 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f711682524b64033a4ce9193bd2cd9621e08b4bae44cffffb4e397a43a8f945"
Nov 07 02:15:58.134 I ns/openshift-monitoring pod/alertmanager-main-0 Created container alertmanager-proxy
Nov 07 02:15:58.199 I ns/openshift-monitoring pod/alertmanager-main-0 Started container alertmanager-proxy
Nov 07 02:16:16.718 I node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g Node ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g event: DeletingNode
Nov 07 02:16:16.726 W node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g deleted
Nov 07 02:16:17.522 I node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g Node ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g event: Removing Node ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g from Controller
Nov 07 02:16:18.212 W ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g DeleteError: failed to delete machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g": failed to delete OS disk: result error: compute.DisksDeleteFuture: asynchronous operation has not completed
Nov 07 02:16:24.999 I ns/openshift-authentication-operator deployment/authentication-operator Status for clusteroperator/authentication changed: Degraded message changed from "" to "OperatorSyncDegraded: failed syncing configuration objects: config maps [v4-0-config-user-idp-0-ca] in openshift-authentication not synced"
Nov 07 02:16:25.160 I ns/openshift-authentication-operator deployment/authentication-operator Created ConfigMap/v4-0-config-user-idp-0-ca -n openshift-authentication because it was missing
Nov 07 02:16:27.954 I ns/openshift-authentication-operator deployment/authentication-operator Updated ConfigMap/v4-0-config-system-cliconfig -n openshift-authentication: cause by changes in data.v4-0-config-system-cliconfig
Nov 07 02:16:28.595 I ns/openshift-authentication-operator deployment/authentication-operator Updated Deployment.apps/oauth-openshift -n openshift-authentication because it changed
Nov 07 02:16:28.646 I ns/openshift-authentication deployment/oauth-openshift Scaled up replica set oauth-openshift-8d8b68f9 to 1
Nov 07 02:16:28.675 I ns/openshift-authentication pod/oauth-openshift-8d8b68f9-6hktd node/ created
Nov 07 02:16:28.690 I ns/openshift-authentication replicaset/oauth-openshift-8d8b68f9 Created pod: oauth-openshift-8d8b68f9-6hktd
Nov 07 02:16:28.744 I ns/openshift-authentication pod/oauth-openshift-8d8b68f9-6hktd Successfully assigned openshift-authentication/oauth-openshift-8d8b68f9-6hktd to ci-op-pg7mtgss-09f59-ng62w-master-1
Nov 07 02:16:29.003 W clusteroperator/authentication changed Progressing to True: ProgressingOAuthServerDeploymentNotReady: Progressing: deployment's observed generation did not reach the expected generation
Nov 07 02:16:29.011 I ns/openshift-authentication-operator deployment/authentication-operator Status for clusteroperator/authentication changed: Degraded message changed from "OperatorSyncDegraded: failed syncing configuration objects: config maps [v4-0-config-user-idp-0-ca] in openshift-authentication not synced" to "",Progressing changed from False to True ("Progressing: deployment's observed generation did not reach the expected generation")
Nov 07 02:16:32.405 I ns/openshift-authentication-operator deployment/authentication-operator Status for clusteroperator/authentication changed: Progressing message changed from "Progressing: deployment's observed generation did not reach the expected generation" to "Progressing: not all deployment replicas are ready"
Nov 07 02:16:32.879 W ns/openshift-multus pod/multus-s2j8q node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g graceful deletion within 0s
Nov 07 02:16:32.888 W ns/openshift-multus pod/multus-s2j8q node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g deleted
Nov 07 02:16:32.916 W ns/openshift-cluster-node-tuning-operator pod/tuned-dvg48 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g graceful deletion within 0s
Nov 07 02:16:32.925 W ns/openshift-cluster-node-tuning-operator pod/tuned-dvg48 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g deleted
Nov 07 02:16:32.939 W ns/openshift-image-registry pod/node-ca-7hhjn node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g graceful deletion within 0s
Nov 07 02:16:32.953 W ns/openshift-image-registry pod/node-ca-7hhjn node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g deleted
Nov 07 02:16:32.971 W ns/openshift-dns pod/dns-default-nzxdl node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g graceful deletion within 0s
Nov 07 02:16:32.985 W ns/openshift-dns pod/dns-default-nzxdl node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g deleted
Nov 07 02:16:33.003 W ns/openshift-machine-config-operator pod/machine-config-daemon-l7p2b node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g graceful deletion within 0s
Nov 07 02:16:33.019 W ns/openshift-machine-config-operator pod/machine-config-daemon-l7p2b node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g deleted
Nov 07 02:16:33.032 W ns/openshift-monitoring pod/node-exporter-p86zj node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g graceful deletion within 0s
Nov 07 02:16:33.039 W ns/openshift-monitoring pod/node-exporter-p86zj node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g deleted
Nov 07 02:16:33.057 W ns/openshift-sdn pod/ovs-vw995 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g graceful deletion within 0s
Nov 07 02:16:33.068 W ns/openshift-sdn pod/ovs-vw995 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g deleted
Nov 07 02:16:33.089 W ns/openshift-sdn pod/sdn-stvl4 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g graceful deletion within 0s
Nov 07 02:16:33.117 W ns/openshift-sdn pod/sdn-stvl4 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g deleted
Nov 07 02:16:33.138 W clusteroperator/network changed Progressing to False
Nov 07 02:16:38.883 I ns/openshift-machine-api machine/ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g Deleted machine "ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-gvl4g"
Nov 07 02:17:06.113 I ns/openshift-authentication pod/oauth-openshift-8d8b68f9-6hktd Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:748d8c1c889ad7aaa90dca2ede7435bfd94c5af44a9dc4cb741032bddf8fd3ed"
Nov 07 02:17:09.890 I ns/openshift-authentication pod/oauth-openshift-8d8b68f9-6hktd Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:748d8c1c889ad7aaa90dca2ede7435bfd94c5af44a9dc4cb741032bddf8fd3ed"
Nov 07 02:17:10.210 I ns/openshift-authentication pod/oauth-openshift-8d8b68f9-6hktd Created container oauth-openshift
Nov 07 02:17:10.245 I ns/openshift-authentication pod/oauth-openshift-8d8b68f9-6hktd Started container oauth-openshift
Nov 07 02:17:16.623 I ns/openshift-authentication deployment/oauth-openshift Scaled down replica set oauth-openshift-6564f675b6 to 1
Nov 07 02:17:16.624 W ns/openshift-authentication pod/oauth-openshift-6564f675b6-njdhv node/ci-op-pg7mtgss-09f59-ng62w-master-2 graceful deletion within 30s
Nov 07 02:17:16.643 I ns/openshift-authentication replicaset/oauth-openshift-6564f675b6 Deleted pod: oauth-openshift-6564f675b6-njdhv
Nov 07 02:17:16.648 I ns/openshift-authentication deployment/oauth-openshift Scaled up replica set oauth-openshift-8d8b68f9 to 2
Nov 07 02:17:16.662 I ns/openshift-authentication pod/oauth-openshift-6564f675b6-njdhv Stopping container oauth-openshift
Nov 07 02:17:16.711 I ns/openshift-authentication pod/oauth-openshift-8d8b68f9-2rqxr node/ created
Nov 07 02:17:16.725 I ns/openshift-authentication replicaset/oauth-openshift-8d8b68f9 Created pod: oauth-openshift-8d8b68f9-2rqxr
Nov 07 02:17:16.757 I ns/openshift-authentication pod/oauth-openshift-8d8b68f9-2rqxr Successfully assigned openshift-authentication/oauth-openshift-8d8b68f9-2rqxr to ci-op-pg7mtgss-09f59-ng62w-master-0
Nov 07 02:17:17.847 W ns/openshift-authentication pod/oauth-openshift-6564f675b6-njdhv node/ci-op-pg7mtgss-09f59-ng62w-master-2 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:17:17.847 W ns/openshift-authentication pod/oauth-openshift-6564f675b6-njdhv node/ci-op-pg7mtgss-09f59-ng62w-master-2 container=oauth-openshift container stopped being ready
Nov 07 02:17:17.884 E ns/openshift-authentication pod/oauth-openshift-6564f675b6-njdhv node/ci-op-pg7mtgss-09f59-ng62w-master-2 container=oauth-openshift container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Nov 07 02:17:21.671 W ns/openshift-authentication pod/oauth-openshift-6564f675b6-njdhv node/ci-op-pg7mtgss-09f59-ng62w-master-2 deleted
Nov 07 02:17:24.819 I ns/openshift-authentication pod/oauth-openshift-8d8b68f9-2rqxr Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:748d8c1c889ad7aaa90dca2ede7435bfd94c5af44a9dc4cb741032bddf8fd3ed" already present on machine
Nov 07 02:17:25.096 I ns/openshift-authentication pod/oauth-openshift-8d8b68f9-2rqxr Created container oauth-openshift
Nov 07 02:17:25.123 I ns/openshift-authentication pod/oauth-openshift-8d8b68f9-2rqxr Started container oauth-openshift
Nov 07 02:17:27.256 I ns/openshift-authentication deployment/oauth-openshift Scaled down replica set oauth-openshift-6564f675b6 to 0
Nov 07 02:17:27.266 W ns/openshift-authentication pod/oauth-openshift-6564f675b6-vq676 node/ci-op-pg7mtgss-09f59-ng62w-master-0 graceful deletion within 30s
Nov 07 02:17:27.286 I ns/openshift-authentication replicaset/oauth-openshift-6564f675b6 Deleted pod: oauth-openshift-6564f675b6-vq676
Nov 07 02:17:27.289 I ns/openshift-authentication pod/oauth-openshift-6564f675b6-vq676 Stopping container oauth-openshift
Nov 07 02:17:27.550 I ns/openshift-authentication-operator deployment/authentication-operator Status for clusteroperator/authentication changed: Degraded message changed from "" to "RouteHealthDegraded: failed to GET route: net/http: TLS handshake timeout"
Nov 07 02:17:28.990 W clusteroperator/authentication changed Progressing to False
Nov 07 02:17:28.998 I ns/openshift-authentication-operator deployment/authentication-operator Status for clusteroperator/authentication changed: Degraded message changed from "RouteHealthDegraded: failed to GET route: net/http: TLS handshake timeout" to "",Progressing changed from True to False ("")
Nov 07 02:17:29.050 W ns/openshift-authentication pod/oauth-openshift-6564f675b6-vq676 node/ci-op-pg7mtgss-09f59-ng62w-master-0 deleted
Nov 07 02:17:34.403 W ns/openshift-ingress service/router-default Error updating load balancer with new hosts map[ci-op-pg7mtgss-09f59-ng62w-master-0:{} ci-op-pg7mtgss-09f59-ng62w-master-1:{} ci-op-pg7mtgss-09f59-ng62w-master-2:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-z8fpv:{}]: ensure(openshift-ingress/router-default): backendPoolID(/subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-pg7mtgss-09f59-ng62w-rg/providers/Microsoft.Network/loadBalancers/ci-op-pg7mtgss-09f59-ng62w/backendAddressPools/ci-op-pg7mtgss-09f59-ng62w) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:NicGet" (2 times)
Nov 07 02:17:35.555 I ns/openshift-authentication-operator deployment/authentication-operator Updated ConfigMap/v4-0-config-system-cliconfig -n openshift-authentication: cause by changes in data.v4-0-config-system-cliconfig (2 times)
Nov 07 02:17:36.470 I ns/openshift-authentication-operator deployment/authentication-operator Updated Deployment.apps/oauth-openshift -n openshift-authentication because it changed (2 times)
Nov 07 02:17:36.501 I ns/openshift-authentication deployment/oauth-openshift Scaled up replica set oauth-openshift-96756d7f5 to 1
Nov 07 02:17:36.531 I ns/openshift-authentication pod/oauth-openshift-96756d7f5-dwrkg node/ created
Nov 07 02:17:36.560 I ns/openshift-authentication replicaset/oauth-openshift-96756d7f5 Created pod: oauth-openshift-96756d7f5-dwrkg
Nov 07 02:17:36.574 I ns/openshift-authentication pod/oauth-openshift-96756d7f5-dwrkg Successfully assigned openshift-authentication/oauth-openshift-96756d7f5-dwrkg to ci-op-pg7mtgss-09f59-ng62w-master-2
Nov 07 02:17:36.784 W clusteroperator/authentication changed Progressing to True: ProgressingOAuthServerDeploymentNotReady: Progressing: deployment's observed generation did not reach the expected generation
Nov 07 02:17:36.794 I ns/openshift-authentication-operator deployment/authentication-operator Status for clusteroperator/authentication changed: Progressing changed from False to True ("Progressing: deployment's observed generation did not reach the expected generation")
Nov 07 02:17:40.207 I ns/openshift-authentication-operator deployment/authentication-operator Status for clusteroperator/authentication changed: Progressing message changed from "Progressing: deployment's observed generation did not reach the expected generation" to "Progressing: not all deployment replicas are ready" (2 times)
Nov 07 02:18:04.828 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (6 times)
Nov 07 02:18:05.428 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-008" not found) (2 times)
Nov 07 02:18:14.368 I ns/openshift-authentication pod/oauth-openshift-96756d7f5-dwrkg Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:748d8c1c889ad7aaa90dca2ede7435bfd94c5af44a9dc4cb741032bddf8fd3ed" already present on machine
Nov 07 02:18:14.647 I ns/openshift-authentication pod/oauth-openshift-96756d7f5-dwrkg Created container oauth-openshift
Nov 07 02:18:14.685 I ns/openshift-authentication pod/oauth-openshift-96756d7f5-dwrkg Started container oauth-openshift
Nov 07 02:18:15.346 W ns/openshift-authentication pod/oauth-openshift-8d8b68f9-6hktd MountVolume.SetUp failed for volume "v4-0-config-user-idp-0-ca" : configmap "v4-0-config-user-idp-0-ca" not found
Nov 07 02:18:15.852 W ns/openshift-authentication pod/oauth-openshift-8d8b68f9-6hktd MountVolume.SetUp failed for volume "v4-0-config-user-idp-0-ca" : configmap "v4-0-config-user-idp-0-ca" not found (2 times)
Nov 07 02:18:16.854 W ns/openshift-authentication pod/oauth-openshift-8d8b68f9-6hktd MountVolume.SetUp failed for volume "v4-0-config-user-idp-0-ca" : configmap "v4-0-config-user-idp-0-ca" not found (3 times)
Nov 07 02:18:18.867 W ns/openshift-authentication pod/oauth-openshift-8d8b68f9-6hktd MountVolume.SetUp failed for volume "v4-0-config-user-idp-0-ca" : configmap "v4-0-config-user-idp-0-ca" not found (4 times)
Nov 07 02:18:22.197 I ns/openshift-authentication deployment/oauth-openshift Scaled down replica set oauth-openshift-8d8b68f9 to 1
Nov 07 02:18:22.212 W ns/openshift-authentication pod/oauth-openshift-8d8b68f9-2rqxr node/ci-op-pg7mtgss-09f59-ng62w-master-0 graceful deletion within 30s
Nov 07 02:18:22.234 I ns/openshift-authentication replicaset/oauth-openshift-8d8b68f9 Deleted pod: oauth-openshift-8d8b68f9-2rqxr
Nov 07 02:18:22.245 I ns/openshift-authentication deployment/oauth-openshift Scaled up replica set oauth-openshift-96756d7f5 to 2
Nov 07 02:18:22.246 I ns/openshift-authentication pod/oauth-openshift-8d8b68f9-2rqxr Stopping container oauth-openshift
Nov 07 02:18:22.302 I ns/openshift-authentication pod/oauth-openshift-96756d7f5-h8s52 node/ created
Nov 07 02:18:22.321 I ns/openshift-authentication replicaset/oauth-openshift-96756d7f5 Created pod: oauth-openshift-96756d7f5-h8s52
Nov 07 02:18:22.346 I ns/openshift-authentication pod/oauth-openshift-96756d7f5-h8s52 Successfully assigned openshift-authentication/oauth-openshift-96756d7f5-h8s52 to ci-op-pg7mtgss-09f59-ng62w-master-0
Nov 07 02:18:22.888 W ns/openshift-authentication pod/oauth-openshift-8d8b68f9-6hktd MountVolume.SetUp failed for volume "v4-0-config-user-idp-0-ca" : configmap "v4-0-config-user-idp-0-ca" not found (5 times)
Nov 07 02:18:26.222 W ns/openshift-authentication pod/oauth-openshift-8d8b68f9-2rqxr node/ci-op-pg7mtgss-09f59-ng62w-master-0 deleted
Nov 07 02:18:30.758 I ns/openshift-authentication pod/oauth-openshift-96756d7f5-h8s52 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:748d8c1c889ad7aaa90dca2ede7435bfd94c5af44a9dc4cb741032bddf8fd3ed" already present on machine
Nov 07 02:18:30.935 W ns/openshift-authentication pod/oauth-openshift-8d8b68f9-6hktd MountVolume.SetUp failed for volume "v4-0-config-user-idp-0-ca" : configmap "v4-0-config-user-idp-0-ca" not found (6 times)
Nov 07 02:18:31.032 I ns/openshift-authentication pod/oauth-openshift-96756d7f5-h8s52 Created container oauth-openshift
Nov 07 02:18:31.064 I ns/openshift-authentication pod/oauth-openshift-96756d7f5-h8s52 Started container oauth-openshift
Nov 07 02:18:32.887 I ns/openshift-authentication deployment/oauth-openshift Scaled down replica set oauth-openshift-8d8b68f9 to 0
Nov 07 02:18:32.898 W ns/openshift-authentication pod/oauth-openshift-8d8b68f9-6hktd node/ci-op-pg7mtgss-09f59-ng62w-master-1 graceful deletion within 30s
Nov 07 02:18:32.920 I ns/openshift-authentication replicaset/oauth-openshift-8d8b68f9 Deleted pod: oauth-openshift-8d8b68f9-6hktd
Nov 07 02:18:32.925 I ns/openshift-authentication pod/oauth-openshift-8d8b68f9-6hktd Stopping container oauth-openshift
Nov 07 02:18:34.012 W clusteroperator/authentication changed Progressing to False
Nov 07 02:18:34.022 I ns/openshift-authentication-operator deployment/authentication-operator Status for clusteroperator/authentication changed: Progressing changed from True to False ("")
Nov 07 02:18:39.324 W ns/openshift-ingress pod/router-default-68c6467955-474p6 Error: ErrImagePull
Nov 07 02:18:39.333 W ns/openshift-ingress pod/router-default-68c6467955-474p6 Failed to pull image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ece12ae9ebfcce0c5b8d8a2eff009a1706a070e9021e96112dbbe6711795a35": rpc error: code = Unknown desc = Error writing blob: error storing blob to file "/var/tmp/storage130349243/2": read tcp 10.0.32.8:58586->13.226.23.44:443: read: connection timed out
Nov 07 02:18:40.119 I ns/openshift-ingress pod/router-default-68c6467955-474p6 Back-off pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ece12ae9ebfcce0c5b8d8a2eff009a1706a070e9021e96112dbbe6711795a35"
Nov 07 02:18:40.123 W ns/openshift-ingress pod/router-default-68c6467955-474p6 Error: ImagePullBackOff
Nov 07 02:18:40.776 W ns/openshift-machine-config-operator pod/machine-config-daemon-l5rhl Failed to pull image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:944d643d229b0a1028464df12f2a544163a6689947df774c6568b3f443a79cb8": rpc error: code = Unknown desc = Error writing blob: error storing blob to file "/var/tmp/storage524359894/1": read tcp 10.0.32.8:60480->13.226.23.108:443: read: connection timed out
Nov 07 02:18:40.796 W ns/openshift-machine-config-operator pod/machine-config-daemon-l5rhl Error: ErrImagePull
Nov 07 02:18:40.804 I ns/openshift-machine-config-operator pod/machine-config-daemon-l5rhl Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f711682524b64033a4ce9193bd2cd9621e08b4bae44cffffb4e397a43a8f945" already present on machine
Nov 07 02:18:40.994 I ns/openshift-machine-config-operator pod/machine-config-daemon-l5rhl Created container oauth-proxy
Nov 07 02:18:41.032 I ns/openshift-machine-config-operator pod/machine-config-daemon-l5rhl Started container oauth-proxy
Nov 07 02:18:41.103 I ns/openshift-machine-config-operator pod/machine-config-daemon-l5rhl Back-off pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:944d643d229b0a1028464df12f2a544163a6689947df774c6568b3f443a79cb8"
Nov 07 02:18:41.118 W ns/openshift-machine-config-operator pod/machine-config-daemon-l5rhl Error: ImagePullBackOff
Nov 07 02:18:41.288 W ns/openshift-image-registry pod/node-ca-6jr5f Error: ErrImagePull
Nov 07 02:18:41.299 W ns/openshift-image-registry pod/node-ca-6jr5f Failed to pull image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3191fc23d0d7e820100498c5085592219a747743f06fc059436945df6ab3e8ee": rpc error: code = Unknown desc = Error writing blob: error storing blob to file "/var/tmp/storage063853463/1": read tcp 10.0.32.8:60556->13.226.23.108:443: read: connection timed out
Nov 07 02:18:41.374 W ns/openshift-dns pod/dns-default-9qpmp Failed to pull image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:614764223d37c466a211f456a9d02c5881d947e1f05d541689dc34a3ec19acf2": rpc error: code = Unknown desc = Error writing blob: error storing blob to file "/var/tmp/storage389373720/1": read tcp 10.0.32.8:58540->13.226.23.44:443: read: connection timed out
Nov 07 02:18:41.381 W ns/openshift-dns pod/dns-default-9qpmp Error: ErrImagePull
Nov 07 02:18:41.390 I ns/openshift-dns pod/dns-default-9qpmp Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd150f3e4a29d160fae9610f1096180eaeb29ff29b63bf490b0022b4d3913e29"
Nov 07 02:18:42.115 W ns/openshift-image-registry pod/node-ca-6jr5f Error: ImagePullBackOff
Nov 07 02:18:42.120 I ns/openshift-machine-config-operator pod/machine-config-daemon-l5rhl Back-off pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:944d643d229b0a1028464df12f2a544163a6689947df774c6568b3f443a79cb8" (2 times)
Nov 07 02:18:42.138 W ns/openshift-machine-config-operator pod/machine-config-daemon-l5rhl Error: ImagePullBackOff (2 times)
Nov 07 02:18:42.144 I ns/openshift-image-registry pod/node-ca-6jr5f Back-off pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3191fc23d0d7e820100498c5085592219a747743f06fc059436945df6ab3e8ee"
Nov 07 02:18:47.003 W ns/openshift-authentication pod/oauth-openshift-8d8b68f9-6hktd MountVolume.SetUp failed for volume "v4-0-config-user-idp-0-ca" : configmap "v4-0-config-user-idp-0-ca" not found (7 times)
Nov 07 02:18:53.450 I ns/openshift-image-registry pod/node-ca-6jr5f Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3191fc23d0d7e820100498c5085592219a747743f06fc059436945df6ab3e8ee" (2 times)
Nov 07 02:18:54.438 I ns/openshift-ingress pod/router-default-68c6467955-474p6 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ece12ae9ebfcce0c5b8d8a2eff009a1706a070e9021e96112dbbe6711795a35" (2 times)
Nov 07 02:18:56.439 I ns/openshift-machine-config-operator pod/machine-config-daemon-l5rhl Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:944d643d229b0a1028464df12f2a544163a6689947df774c6568b3f443a79cb8" (2 times)
Nov 07 02:18:56.569 W ns/openshift-authentication pod/oauth-openshift-8d8b68f9-6hktd node/ci-op-pg7mtgss-09f59-ng62w-master-1 container=oauth-openshift container stopped being ready
Nov 07 02:19:02.008 I ns/openshift-dns pod/dns-default-9qpmp Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd150f3e4a29d160fae9610f1096180eaeb29ff29b63bf490b0022b4d3913e29"
Nov 07 02:19:02.377 I ns/openshift-dns pod/dns-default-9qpmp Created container dns-node-resolver
Nov 07 02:19:02.446 I ns/openshift-dns pod/dns-default-9qpmp Started container dns-node-resolver
Nov 07 02:19:03.223 I ns/openshift-dns pod/dns-default-9qpmp Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:614764223d37c466a211f456a9d02c5881d947e1f05d541689dc34a3ec19acf2" (2 times)
Nov 07 02:19:04.476 W ns/openshift-authentication pod/oauth-openshift-8d8b68f9-6hktd node/ci-op-pg7mtgss-09f59-ng62w-master-1 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:19:14.874 I ns/openshift-dns pod/dns-default-9qpmp Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:614764223d37c466a211f456a9d02c5881d947e1f05d541689dc34a3ec19acf2"
Nov 07 02:19:14.887 I ns/openshift-machine-config-operator pod/machine-config-daemon-l5rhl Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:944d643d229b0a1028464df12f2a544163a6689947df774c6568b3f443a79cb8"
Nov 07 02:19:15.113 W ns/openshift-ingress service/router-default Error updating load balancer with new hosts map[ci-op-pg7mtgss-09f59-ng62w-master-0:{} ci-op-pg7mtgss-09f59-ng62w-master-1:{} ci-op-pg7mtgss-09f59-ng62w-master-2:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-z8fpv:{}]: ensure(openshift-ingress/router-default): backendPoolID(/subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-pg7mtgss-09f59-ng62w-rg/providers/Microsoft.Network/loadBalancers/ci-op-pg7mtgss-09f59-ng62w/backendAddressPools/ci-op-pg7mtgss-09f59-ng62w) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:NicGet" (3 times)
Nov 07 02:19:15.221 I ns/openshift-machine-config-operator pod/machine-config-daemon-l5rhl Created container machine-config-daemon
Nov 07 02:19:15.232 I ns/openshift-dns pod/dns-default-9qpmp Created container dns
Nov 07 02:19:15.302 W ns/openshift-authentication pod/oauth-openshift-8d8b68f9-6hktd node/ci-op-pg7mtgss-09f59-ng62w-master-1 deleted
Nov 07 02:19:15.309 I ns/openshift-machine-config-operator pod/machine-config-daemon-l5rhl Started container machine-config-daemon
Nov 07 02:19:15.316 I ns/openshift-dns pod/dns-default-9qpmp Started container dns
Nov 07 02:19:21.534 I ns/openshift-ingress pod/router-default-68c6467955-474p6 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ece12ae9ebfcce0c5b8d8a2eff009a1706a070e9021e96112dbbe6711795a35"
Nov 07 02:19:21.795 I ns/openshift-ingress pod/router-default-68c6467955-474p6 Created container router
Nov 07 02:19:21.866 I ns/openshift-ingress pod/router-default-68c6467955-474p6 Started container router
Nov 07 02:19:28.264 W clusteroperator/dns changed Progressing to False: AsExpected: Desired and available number of DNS DaemonSets are equal
Nov 07 02:19:29.051 I node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 Setting node ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9, currentConfig rendered-worker-375794f00b8fe3b7bf8d2b6a16656633 to Done
Nov 07 02:19:38.846 W clusteroperator/ingress changed Available to True: desired and current number of IngressControllers are equal
Nov 07 02:19:38.846 W clusteroperator/ingress changed Progressing to False: desired and current number of IngressControllers are equal
Nov 07 02:20:26.223 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (19 times)
Nov 07 02:20:26.807 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-008" not found) (4 times)
Nov 07 02:20:55.555 W ns/openshift-ingress service/router-default Error updating load balancer with new hosts map[ci-op-pg7mtgss-09f59-ng62w-master-0:{} ci-op-pg7mtgss-09f59-ng62w-master-1:{} ci-op-pg7mtgss-09f59-ng62w-master-2:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-z8fpv:{}]: ensure(openshift-ingress/router-default): backendPoolID(/subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-pg7mtgss-09f59-ng62w-rg/providers/Microsoft.Network/loadBalancers/ci-op-pg7mtgss-09f59-ng62w/backendAddressPools/ci-op-pg7mtgss-09f59-ng62w) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:NicGet" (4 times)
Nov 07 02:21:32.448 I ns/openshift-image-registry pod/node-ca-6jr5f Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3191fc23d0d7e820100498c5085592219a747743f06fc059436945df6ab3e8ee"
Nov 07 02:21:32.916 I ns/openshift-image-registry pod/node-ca-6jr5f Created container node-ca
Nov 07 02:21:32.967 I ns/openshift-image-registry pod/node-ca-6jr5f Started container node-ca
Nov 07 02:21:56.266 I ns/openshift-marketplace pod/certified-operators-69589995bf-wtx68 Marking for deletion Pod openshift-marketplace/certified-operators-69589995bf-wtx68
Nov 07 02:21:56.267 W ns/openshift-dns pod/dns-default-9qpmp node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 graceful deletion within 30s
Nov 07 02:21:56.267 W ns/openshift-image-registry pod/node-ca-6jr5f node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 graceful deletion within 30s
Nov 07 02:21:56.268 W ns/openshift-marketplace pod/certified-operators-69589995bf-wtx68 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 graceful deletion within 30s
Nov 07 02:21:56.286 W ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-8z6z2 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 graceful deletion within 30s
Nov 07 02:21:56.304 I ns/openshift-dns pod/dns-default-9qpmp Stopping container dns
Nov 07 02:21:56.304 I ns/openshift-dns daemonset/dns-default Deleted pod: dns-default-9qpmp
Nov 07 02:21:56.304 W ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-k7t29 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 graceful deletion within 30s
Nov 07 02:21:56.304 I ns/openshift-marketplace pod/community-operators-9567c47bd-6p48d Marking for deletion Pod openshift-marketplace/community-operators-9567c47bd-6p48d
Nov 07 02:21:56.305 W ns/openshift-marketplace pod/community-operators-9567c47bd-6p48d node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 graceful deletion within 30s
Nov 07 02:21:56.305 W ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 graceful deletion within 120s
Nov 07 02:21:56.306 W ns/openshift-machine-config-operator pod/machine-config-daemon-l5rhl node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 graceful deletion within 600s
Nov 07 02:21:56.306 W ns/openshift-ingress pod/router-default-68c6467955-474p6 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 graceful deletion within 30s
Nov 07 02:21:56.323 I ns/openshift-dns pod/dns-default-9qpmp Stopping container dns-node-resolver
Nov 07 02:21:56.323 I ns/openshift-image-registry pod/node-ca-6jr5f Marking for deletion Pod openshift-image-registry/node-ca-6jr5f
Nov 07 02:21:56.323 I ns/openshift-image-registry daemonset/node-ca Deleted pod: node-ca-6jr5f
Nov 07 02:21:56.323 I ns/openshift-image-registry pod/node-ca-6jr5f Stopping container node-ca
Nov 07 02:21:56.331 I ns/openshift-machine-config-operator pod/machine-config-daemon-l5rhl Marking for deletion Pod openshift-machine-config-operator/machine-config-daemon-l5rhl
Nov 07 02:21:56.331 I ns/openshift-marketplace pod/certified-operators-69589995bf-wtx68 Stopping container certified-operators
Nov 07 02:21:56.342 I ns/openshift-dns pod/dns-default-9qpmp Marking for deletion Pod openshift-dns/dns-default-9qpmp
Nov 07 02:21:56.350 I ns/openshift-monitoring pod/alertmanager-main-0 Marking for deletion Pod openshift-monitoring/alertmanager-main-0
Nov 07 02:21:56.358 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-k7t29 Marking for deletion Pod openshift-monitoring/prometheus-adapter-6fb7598968-k7t29
Nov 07 02:21:56.365 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-8z6z2 Marking for deletion Pod openshift-monitoring/openshift-state-metrics-6c78647cc7-8z6z2
Nov 07 02:21:56.390 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-8z6z2 Stopping container openshift-state-metrics
Nov 07 02:21:56.395 I ns/openshift-marketplace pod/community-operators-9567c47bd-2thsq node/ created
Nov 07 02:21:56.396 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-z5qnw node/ created
Nov 07 02:21:56.396 I ns/openshift-marketplace pod/certified-operators-69589995bf-p5xcd node/ created
Nov 07 02:21:56.416 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-8z6z2 Stopping container kube-rbac-proxy-self
Nov 07 02:21:56.428 I ns/openshift-ingress pod/router-default-68c6467955-474p6 Marking for deletion Pod openshift-ingress/router-default-68c6467955-474p6
Nov 07 02:21:56.428 I ns/openshift-monitoring replicaset/prometheus-adapter-6fb7598968 Created pod: prometheus-adapter-6fb7598968-z5qnw
Nov 07 02:21:56.429 I ns/openshift-ingress pod/router-default-68c6467955-slbl2 node/ created
Nov 07 02:21:56.429 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-992cc node/ created
Nov 07 02:21:56.445 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-8z6z2 Stopping container kube-rbac-proxy-main
Nov 07 02:21:56.445 I ns/openshift-marketplace replicaset/certified-operators-69589995bf Created pod: certified-operators-69589995bf-p5xcd
Nov 07 02:21:56.457 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-z5qnw Successfully assigned openshift-monitoring/prometheus-adapter-6fb7598968-z5qnw to ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj
Nov 07 02:21:56.469 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-k7t29 Stopping container prometheus-adapter
Nov 07 02:21:56.471 I ns/openshift-marketplace replicaset/community-operators-9567c47bd Created pod: community-operators-9567c47bd-2thsq
Nov 07 02:21:56.490 I ns/openshift-marketplace pod/community-operators-9567c47bd-6p48d Stopping container community-operators
Nov 07 02:21:56.502 I ns/openshift-monitoring replicaset/openshift-state-metrics-6c78647cc7 Created pod: openshift-state-metrics-6c78647cc7-992cc
Nov 07 02:21:56.504 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-992cc Successfully assigned openshift-monitoring/openshift-state-metrics-6c78647cc7-992cc to ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj
Nov 07 02:21:56.504 I ns/openshift-marketplace pod/certified-operators-69589995bf-p5xcd Successfully assigned openshift-marketplace/certified-operators-69589995bf-p5xcd to ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj
Nov 07 02:21:56.504 I ns/openshift-marketplace pod/community-operators-9567c47bd-2thsq Successfully assigned openshift-marketplace/community-operators-9567c47bd-2thsq to ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj
Nov 07 02:21:56.519 I ns/openshift-ingress replicaset/router-default-68c6467955 Created pod: router-default-68c6467955-slbl2
Nov 07 02:21:56.527 I ns/openshift-ingress pod/router-default-68c6467955-474p6 Stopping container router
Nov 07 02:21:56.532 I ns/openshift-ingress pod/router-default-68c6467955-slbl2 Successfully assigned openshift-ingress/router-default-68c6467955-slbl2 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj
Nov 07 02:21:56.544 I ns/openshift-monitoring pod/alertmanager-main-0 Stopping container alertmanager
Nov 07 02:21:56.678 I ns/openshift-machine-config-operator pod/machine-config-daemon-l5rhl Stopping container oauth-proxy
Nov 07 02:21:56.694 W clusteroperator/ingress changed Available to False: IngressUnavailable: Not all ingress controllers are available.
Nov 07 02:21:56.694 W clusteroperator/ingress changed Progressing to True: Reconciling: Not all ingress controllers are available.
Nov 07 02:21:56.874 I ns/openshift-monitoring pod/alertmanager-main-0 Stopping container alertmanager-proxy
Nov 07 02:21:57.065 I ns/openshift-monitoring pod/alertmanager-main-0 Stopping container config-reloader
Nov 07 02:21:57.269 I ns/openshift-machine-config-operator pod/machine-config-daemon-l5rhl Stopping container machine-config-daemon
Nov 07 02:21:58.682 W ns/openshift-ingress pod/router-default-68c6467955-474p6 Readiness probe failed: Get http://10.128.4.4:1936/healthz/ready: dial tcp 10.128.4.4:1936: connect: connection refused
Nov 07 02:22:00.170 W ns/openshift-dns pod/dns-default-9qpmp node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:22:00.170 W ns/openshift-dns pod/dns-default-9qpmp node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=dns container stopped being ready
Nov 07 02:22:00.170 W ns/openshift-dns pod/dns-default-9qpmp node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=dns-node-resolver container stopped being ready
Nov 07 02:22:00.207 E ns/openshift-machine-config-operator pod/machine-config-daemon-l5rhl node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=oauth-proxy container exited with code 143 (Error): 
Nov 07 02:22:01.142 W ns/openshift-dns pod/dns-default-9qpmp node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 pod has been pending longer than a minute
Nov 07 02:22:01.186 E ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=config-reloader container exited with code 2 (Error): 2019/11/07 02:15:51 Watching directory: "/etc/alertmanager/config"\n
Nov 07 02:22:01.186 E ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=alertmanager-proxy container exited with code 2 (Error): 2019/11/07 02:15:58 provider.go:117: Defaulting client-id to system:serviceaccount:openshift-monitoring:alertmanager-main\n2019/11/07 02:15:58 provider.go:122: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token\n2019/11/07 02:15:58 provider.go:310: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.\n2019/11/07 02:15:58 oauthproxy.go:200: mapping path "/" => upstream "http://localhost:9093/"\n2019/11/07 02:15:58 oauthproxy.go:221: compiled skip-auth-regex => "^/metrics"\n2019/11/07 02:15:58 oauthproxy.go:227: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:alertmanager-main\n2019/11/07 02:15:58 oauthproxy.go:237: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> refresh:disabled\n2019/11/07 02:15:58 http.go:96: HTTPS: listening on [::]:9095\n
Nov 07 02:22:01.223 W ns/openshift-ingress pod/router-default-68c6467955-474p6 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:22:01.223 W ns/openshift-ingress pod/router-default-68c6467955-474p6 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=router container stopped being ready
Nov 07 02:22:01.242 W ns/openshift-marketplace pod/certified-operators-69589995bf-wtx68 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:22:01.242 W ns/openshift-marketplace pod/certified-operators-69589995bf-wtx68 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=certified-operators container stopped being ready
Nov 07 02:22:01.254 W ns/openshift-marketplace pod/community-operators-9567c47bd-6p48d node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:22:01.254 W ns/openshift-marketplace pod/community-operators-9567c47bd-6p48d node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=community-operators container stopped being ready
Nov 07 02:22:01.271 E ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-8z6z2 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=openshift-state-metrics container exited with code 2 (Error): 
Nov 07 02:22:01.290 W ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-k7t29 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:22:01.290 W ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-k7t29 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=prometheus-adapter container stopped being ready
Nov 07 02:22:03.480 W ns/openshift-machine-config-operator pod/machine-config-daemon-l5rhl node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 deleted
Nov 07 02:22:03.520 W ns/openshift-dns pod/dns-default-9qpmp node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 deleted
Nov 07 02:22:03.585 W ns/openshift-image-registry pod/node-ca-6jr5f node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 deleted
Nov 07 02:22:03.668 I ns/openshift-dns pod/dns-default-s9d7l node/ created
Nov 07 02:22:03.695 I ns/openshift-dns daemonset/dns-default Created pod: dns-default-s9d7l
Nov 07 02:22:03.702 I ns/openshift-dns pod/dns-default-s9d7l Successfully assigned openshift-dns/dns-default-s9d7l to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:22:03.719 I ns/openshift-machine-config-operator pod/machine-config-daemon-pqhlq node/ created
Nov 07 02:22:03.721 I ns/openshift-image-registry pod/node-ca-2d8ll node/ created
Nov 07 02:22:03.737 I ns/openshift-machine-config-operator daemonset/machine-config-daemon Created pod: machine-config-daemon-pqhlq
Nov 07 02:22:03.752 I ns/openshift-image-registry daemonset/node-ca Created pod: node-ca-2d8ll
Nov 07 02:22:03.758 I ns/openshift-machine-config-operator pod/machine-config-daemon-pqhlq Successfully assigned openshift-machine-config-operator/machine-config-daemon-pqhlq to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:22:03.764 I ns/openshift-image-registry pod/node-ca-2d8ll Successfully assigned openshift-image-registry/node-ca-2d8ll to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:22:03.818 W clusteroperator/dns changed Progressing to True: Reconciling: Not all DNS DaemonSets available.
Nov 07 02:22:04.337 I ns/openshift-marketplace pod/certified-operators-69589995bf-p5xcd Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:448858c7b1f70b8c0955965ef9970b59504b220bf03b760203305c81e5f606da" already present on machine
Nov 07 02:22:04.547 I ns/openshift-marketplace pod/certified-operators-69589995bf-p5xcd Created container certified-operators
Nov 07 02:22:04.594 I ns/openshift-marketplace pod/certified-operators-69589995bf-p5xcd Started container certified-operators
Nov 07 02:22:05.040 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-992cc Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5366da9d7c15f912b609cdbb2851d01ea863fb22d452081914d923d8208c39c8" already present on machine
Nov 07 02:22:05.133 I ns/openshift-machine-config-operator pod/machine-config-daemon-pqhlq Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:944d643d229b0a1028464df12f2a544163a6689947df774c6568b3f443a79cb8" already present on machine
Nov 07 02:22:05.254 I ns/openshift-ingress pod/router-default-68c6467955-slbl2 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ece12ae9ebfcce0c5b8d8a2eff009a1706a070e9021e96112dbbe6711795a35"
Nov 07 02:22:05.307 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-992cc Created container kube-rbac-proxy-main
Nov 07 02:22:05.363 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-z5qnw Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4753bca36792206b5e9df9612f695fc2c3a80a54d56b01a10f774dc1b62899be"
Nov 07 02:22:05.382 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-992cc Started container kube-rbac-proxy-main
Nov 07 02:22:05.391 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-992cc Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5366da9d7c15f912b609cdbb2851d01ea863fb22d452081914d923d8208c39c8" already present on machine
Nov 07 02:22:05.402 I ns/openshift-machine-config-operator pod/machine-config-daemon-pqhlq Created container machine-config-daemon
Nov 07 02:22:05.467 I ns/openshift-machine-config-operator pod/machine-config-daemon-pqhlq Started container machine-config-daemon
Nov 07 02:22:05.479 I ns/openshift-machine-config-operator pod/machine-config-daemon-pqhlq Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f711682524b64033a4ce9193bd2cd9621e08b4bae44cffffb4e397a43a8f945" already present on machine
Nov 07 02:22:05.636 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-992cc Created container kube-rbac-proxy-self
Nov 07 02:22:05.705 I ns/openshift-marketplace pod/community-operators-9567c47bd-2thsq Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:448858c7b1f70b8c0955965ef9970b59504b220bf03b760203305c81e5f606da" already present on machine
Nov 07 02:22:05.715 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-992cc Started container kube-rbac-proxy-self
Nov 07 02:22:05.748 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-992cc Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9b0c04ed78fe07c8c7ba6efec2be18908955710281189051eaea6942d424a56f"
Nov 07 02:22:05.764 I ns/openshift-machine-config-operator pod/machine-config-daemon-pqhlq Created container oauth-proxy
Nov 07 02:22:05.808 I ns/openshift-machine-config-operator pod/machine-config-daemon-pqhlq Started container oauth-proxy
Nov 07 02:22:05.933 I ns/openshift-marketplace pod/community-operators-9567c47bd-2thsq Created container community-operators
Nov 07 02:22:05.977 I ns/openshift-marketplace pod/community-operators-9567c47bd-2thsq Started container community-operators
Nov 07 02:22:12.488 I ns/openshift-dns pod/dns-default-s9d7l Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:614764223d37c466a211f456a9d02c5881d947e1f05d541689dc34a3ec19acf2" already present on machine
Nov 07 02:22:12.712 I ns/openshift-dns pod/dns-default-s9d7l Created container dns
Nov 07 02:22:12.756 I ns/openshift-dns pod/dns-default-s9d7l Started container dns
Nov 07 02:22:12.767 I ns/openshift-dns pod/dns-default-s9d7l Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd150f3e4a29d160fae9610f1096180eaeb29ff29b63bf490b0022b4d3913e29" already present on machine
Nov 07 02:22:12.876 I ns/openshift-image-registry pod/node-ca-2d8ll Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3191fc23d0d7e820100498c5085592219a747743f06fc059436945df6ab3e8ee" already present on machine
Nov 07 02:22:13.076 I ns/openshift-dns pod/dns-default-s9d7l Created container dns-node-resolver
Nov 07 02:22:13.153 I ns/openshift-dns pod/dns-default-s9d7l Started container dns-node-resolver
Nov 07 02:22:13.239 I ns/openshift-image-registry pod/node-ca-2d8ll Created container node-ca
Nov 07 02:22:13.285 I ns/openshift-image-registry pod/node-ca-2d8ll Started container node-ca
Nov 07 02:22:13.488 W ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 deleted
Nov 07 02:22:13.502 W ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-8z6z2 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 deleted
Nov 07 02:22:13.518 I ns/openshift-monitoring pod/alertmanager-main-0 node/ created
Nov 07 02:22:13.547 I ns/openshift-monitoring statefulset/alertmanager-main create Pod alertmanager-main-0 in StatefulSet alertmanager-main successful (3 times)
Nov 07 02:22:13.553 W ns/openshift-marketplace pod/certified-operators-69589995bf-wtx68 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 deleted
Nov 07 02:22:13.558 I ns/openshift-monitoring pod/alertmanager-main-0 Successfully assigned openshift-monitoring/alertmanager-main-0 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:22:13.593 W ns/openshift-ingress pod/router-default-68c6467955-474p6 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 deleted
Nov 07 02:22:13.628 W ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-k7t29 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 deleted
Nov 07 02:22:13.669 W ns/openshift-marketplace pod/community-operators-9567c47bd-6p48d node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 deleted
Nov 07 02:22:14.996 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-z5qnw Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4753bca36792206b5e9df9612f695fc2c3a80a54d56b01a10f774dc1b62899be"
Nov 07 02:22:15.056 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-992cc Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9b0c04ed78fe07c8c7ba6efec2be18908955710281189051eaea6942d424a56f"
Nov 07 02:22:15.072 I ns/openshift-ingress pod/router-default-68c6467955-slbl2 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ece12ae9ebfcce0c5b8d8a2eff009a1706a070e9021e96112dbbe6711795a35"
Nov 07 02:22:15.486 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-z5qnw Created container prometheus-adapter
Nov 07 02:22:15.614 I ns/openshift-monitoring pod/prometheus-adapter-6fb7598968-z5qnw Started container prometheus-adapter
Nov 07 02:22:15.691 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-992cc Created container openshift-state-metrics
Nov 07 02:22:15.812 I ns/openshift-ingress pod/router-default-68c6467955-slbl2 Created container router
Nov 07 02:22:15.831 I ns/openshift-monitoring pod/openshift-state-metrics-6c78647cc7-992cc Started container openshift-state-metrics
Nov 07 02:22:15.878 I ns/openshift-ingress pod/router-default-68c6467955-slbl2 Started container router
Nov 07 02:22:17.469 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-0 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (5 times)
Nov 07 02:22:17.645 W ns/openshift-marketplace pod/certified-operators-69589995bf-p5xcd Readiness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n
Nov 07 02:22:18.463 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-0 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-008" not found) (3 times)
Nov 07 02:22:18.775 W ns/openshift-marketplace pod/community-operators-9567c47bd-2thsq Liveness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n
Nov 07 02:22:21.201 W ns/openshift-marketplace pod/certified-operators-69589995bf-p5xcd Liveness probe errored: rpc error: code = Unknown desc = command error: command timed out, stdout: , stderr: , exit code -1
Nov 07 02:22:21.217 W ns/openshift-marketplace pod/community-operators-9567c47bd-2thsq Readiness probe errored: rpc error: code = Unknown desc = command error: command timed out, stdout: , stderr: , exit code -1
Nov 07 02:22:21.566 I ns/openshift-monitoring pod/alertmanager-main-0 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78ba454f09557364dabd918265784493287ed242fb51ac2ffde695ae78823736" already present on machine
Nov 07 02:22:21.782 I ns/openshift-monitoring pod/alertmanager-main-0 Created container alertmanager
Nov 07 02:22:21.831 I ns/openshift-monitoring pod/alertmanager-main-0 Started container alertmanager
Nov 07 02:22:21.840 I ns/openshift-monitoring pod/alertmanager-main-0 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db10cd1f5efd9e9e6f2b0eea0a665f53399e25de374cf916ad7d8141f8ac643d" already present on machine
Nov 07 02:22:22.289 I ns/openshift-monitoring pod/alertmanager-main-0 Created container config-reloader
Nov 07 02:22:22.322 I ns/openshift-monitoring pod/alertmanager-main-0 Started container config-reloader
Nov 07 02:22:22.330 I ns/openshift-monitoring pod/alertmanager-main-0 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f711682524b64033a4ce9193bd2cd9621e08b4bae44cffffb4e397a43a8f945" already present on machine
Nov 07 02:22:22.541 I ns/openshift-monitoring pod/alertmanager-main-0 Created container alertmanager-proxy
Nov 07 02:22:22.577 I ns/openshift-monitoring pod/alertmanager-main-0 Started container alertmanager-proxy
Nov 07 02:22:25.622 W clusteroperator/dns changed Progressing to False: AsExpected: Desired and available number of DNS DaemonSets are equal
Nov 07 02:22:28.788 W ns/openshift-marketplace pod/community-operators-9567c47bd-2thsq Liveness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n (2 times)
Nov 07 02:22:31.200 W ns/openshift-marketplace pod/community-operators-9567c47bd-2thsq Readiness probe errored: rpc error: code = Unknown desc = command error: command timed out, stdout: , stderr: , exit code -1 (2 times)
Nov 07 02:22:32.035 W clusteroperator/ingress changed Available to True: desired and current number of IngressControllers are equal
Nov 07 02:22:32.035 W clusteroperator/ingress changed Progressing to False: desired and current number of IngressControllers are equal
Nov 07 02:22:36.058 W ns/openshift-ingress service/router-default Error updating load balancer with new hosts map[ci-op-pg7mtgss-09f59-ng62w-master-0:{} ci-op-pg7mtgss-09f59-ng62w-master-1:{} ci-op-pg7mtgss-09f59-ng62w-master-2:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-z8fpv:{}]: ensure(openshift-ingress/router-default): backendPoolID(/subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-pg7mtgss-09f59-ng62w-rg/providers/Microsoft.Network/loadBalancers/ci-op-pg7mtgss-09f59-ng62w/backendAddressPools/ci-op-pg7mtgss-09f59-ng62w) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:NicGet" (5 times)
Nov 07 02:22:38.787 W ns/openshift-marketplace pod/community-operators-9567c47bd-2thsq Liveness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n (3 times)
Nov 07 02:23:18.210 I ns/openshift-image-registry pod/node-ca-2d8ll Marking for deletion Pod openshift-image-registry/node-ca-2d8ll
Nov 07 02:23:18.224 W ns/openshift-image-registry pod/node-ca-2d8ll node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 graceful deletion within 30s
Nov 07 02:23:18.229 W ns/openshift-dns pod/dns-default-s9d7l node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 graceful deletion within 30s
Nov 07 02:23:18.229 I ns/openshift-machine-config-operator pod/machine-config-daemon-pqhlq Marking for deletion Pod openshift-machine-config-operator/machine-config-daemon-pqhlq
Nov 07 02:23:18.229 W ns/openshift-machine-config-operator pod/machine-config-daemon-pqhlq node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 graceful deletion within 600s
Nov 07 02:23:18.230 W ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 graceful deletion within 120s
Nov 07 02:23:18.254 I ns/openshift-dns daemonset/dns-default Deleted pod: dns-default-s9d7l
Nov 07 02:23:18.255 I ns/openshift-dns pod/dns-default-s9d7l Marking for deletion Pod openshift-dns/dns-default-s9d7l
Nov 07 02:23:18.280 I ns/openshift-image-registry daemonset/node-ca Deleted pod: node-ca-2d8ll
Nov 07 02:23:18.280 I ns/openshift-image-registry pod/node-ca-2d8ll Stopping container node-ca
Nov 07 02:23:18.285 I ns/openshift-monitoring pod/alertmanager-main-0 Marking for deletion Pod openshift-monitoring/alertmanager-main-0 (2 times)
Nov 07 02:23:18.294 I ns/openshift-dns pod/dns-default-s9d7l Stopping container dns
Nov 07 02:23:18.306 I ns/openshift-machine-config-operator pod/machine-config-daemon-pqhlq Stopping container machine-config-daemon
Nov 07 02:23:18.318 I ns/openshift-monitoring pod/alertmanager-main-0 Stopping container alertmanager-proxy
Nov 07 02:23:18.336 I ns/openshift-monitoring pod/alertmanager-main-0 Stopping container config-reloader
Nov 07 02:23:18.349 I ns/openshift-monitoring pod/alertmanager-main-0 Stopping container alertmanager
Nov 07 02:23:18.357 I ns/openshift-dns pod/dns-default-s9d7l Stopping container dns-node-resolver
Nov 07 02:23:18.364 I ns/openshift-machine-config-operator pod/machine-config-daemon-pqhlq Stopping container oauth-proxy
Nov 07 02:23:20.759 W ns/openshift-dns pod/dns-default-s9d7l node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:23:20.759 W ns/openshift-dns pod/dns-default-s9d7l node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=dns-node-resolver container stopped being ready
Nov 07 02:23:20.759 W ns/openshift-dns pod/dns-default-s9d7l node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=dns container stopped being ready
Nov 07 02:23:20.797 E ns/openshift-dns pod/dns-default-s9d7l node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=dns container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Nov 07 02:23:20.797 E ns/openshift-dns pod/dns-default-s9d7l node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=dns-node-resolver container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Nov 07 02:23:20.806 W ns/openshift-machine-config-operator pod/machine-config-daemon-pqhlq node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:23:20.806 W ns/openshift-machine-config-operator pod/machine-config-daemon-pqhlq node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=oauth-proxy container stopped being ready
Nov 07 02:23:20.846 W ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:23:20.846 W ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=alertmanager container stopped being ready
Nov 07 02:23:20.846 W ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=config-reloader container stopped being ready
Nov 07 02:23:20.846 W ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=alertmanager-proxy container stopped being ready
Nov 07 02:23:21.782 W ns/openshift-machine-config-operator pod/machine-config-daemon-pqhlq node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 deleted
Nov 07 02:23:21.820 W ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 deleted
Nov 07 02:23:21.851 W ns/openshift-image-registry pod/node-ca-2d8ll node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 deleted
Nov 07 02:23:21.883 I ns/openshift-monitoring pod/alertmanager-main-0 node/ created
Nov 07 02:23:21.906 I ns/openshift-monitoring statefulset/alertmanager-main create Pod alertmanager-main-0 in StatefulSet alertmanager-main successful (4 times)
Nov 07 02:23:21.914 I ns/openshift-monitoring pod/alertmanager-main-0 Successfully assigned openshift-monitoring/alertmanager-main-0 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj
Nov 07 02:23:23.482 W ns/openshift-dns pod/dns-default-s9d7l node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 deleted
Nov 07 02:23:30.109 I ns/openshift-monitoring pod/alertmanager-main-0 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78ba454f09557364dabd918265784493287ed242fb51ac2ffde695ae78823736"
Nov 07 02:23:34.460 I ns/openshift-monitoring pod/alertmanager-main-0 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78ba454f09557364dabd918265784493287ed242fb51ac2ffde695ae78823736"
Nov 07 02:23:34.723 I ns/openshift-monitoring pod/alertmanager-main-0 Created container alertmanager
Nov 07 02:23:34.775 I ns/openshift-monitoring pod/alertmanager-main-0 Started container alertmanager
Nov 07 02:23:34.787 I ns/openshift-monitoring pod/alertmanager-main-0 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db10cd1f5efd9e9e6f2b0eea0a665f53399e25de374cf916ad7d8141f8ac643d" already present on machine
Nov 07 02:23:35.169 I ns/openshift-monitoring pod/alertmanager-main-0 Created container config-reloader
Nov 07 02:23:35.217 I ns/openshift-monitoring pod/alertmanager-main-0 Started container config-reloader
Nov 07 02:23:35.225 I ns/openshift-monitoring pod/alertmanager-main-0 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f711682524b64033a4ce9193bd2cd9621e08b4bae44cffffb4e397a43a8f945" already present on machine
Nov 07 02:23:35.498 I ns/openshift-monitoring pod/alertmanager-main-0 Created container alertmanager-proxy
Nov 07 02:23:35.560 I ns/openshift-monitoring pod/alertmanager-main-0 Started container alertmanager-proxy
Nov 07 02:24:16.689 W ns/openshift-ingress service/router-default Error updating load balancer with new hosts map[ci-op-pg7mtgss-09f59-ng62w-master-0:{} ci-op-pg7mtgss-09f59-ng62w-master-1:{} ci-op-pg7mtgss-09f59-ng62w-master-2:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-z8fpv:{}]: ensure(openshift-ingress/router-default): backendPoolID(/subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-pg7mtgss-09f59-ng62w-rg/providers/Microsoft.Network/loadBalancers/ci-op-pg7mtgss-09f59-ng62w/backendAddressPools/ci-op-pg7mtgss-09f59-ng62w) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:NicGet" (6 times)
Nov 07 02:24:33.582 I ns/openshift-dns pod/dns-default-pgkfd node/ created
Nov 07 02:24:33.599 I ns/openshift-dns daemonset/dns-default Created pod: dns-default-pgkfd
Nov 07 02:24:33.609 I ns/openshift-image-registry pod/node-ca-tbz92 node/ created
Nov 07 02:24:33.628 I ns/openshift-image-registry daemonset/node-ca Created pod: node-ca-tbz92
Nov 07 02:24:33.631 I ns/openshift-dns pod/dns-default-pgkfd Successfully assigned openshift-dns/dns-default-pgkfd to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:24:33.639 I ns/openshift-machine-config-operator pod/machine-config-daemon-6hnt5 node/ created
Nov 07 02:24:33.655 I ns/openshift-machine-config-operator daemonset/machine-config-daemon Created pod: machine-config-daemon-6hnt5
Nov 07 02:24:33.662 I ns/openshift-image-registry pod/node-ca-tbz92 Successfully assigned openshift-image-registry/node-ca-tbz92 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:24:33.697 I ns/openshift-machine-config-operator pod/machine-config-daemon-6hnt5 Successfully assigned openshift-machine-config-operator/machine-config-daemon-6hnt5 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:24:33.762 W clusteroperator/dns changed Progressing to True: Reconciling: Not all DNS DaemonSets available.
Nov 07 02:24:35.112 I ns/openshift-machine-config-operator pod/machine-config-daemon-6hnt5 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:944d643d229b0a1028464df12f2a544163a6689947df774c6568b3f443a79cb8" already present on machine
Nov 07 02:24:35.358 I ns/openshift-machine-config-operator pod/machine-config-daemon-6hnt5 Created container machine-config-daemon
Nov 07 02:24:35.399 I ns/openshift-machine-config-operator pod/machine-config-daemon-6hnt5 Started container machine-config-daemon
Nov 07 02:24:35.406 I ns/openshift-machine-config-operator pod/machine-config-daemon-6hnt5 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f711682524b64033a4ce9193bd2cd9621e08b4bae44cffffb4e397a43a8f945" already present on machine
Nov 07 02:24:35.690 I ns/openshift-machine-config-operator pod/machine-config-daemon-6hnt5 Created container oauth-proxy
Nov 07 02:24:35.737 I ns/openshift-machine-config-operator pod/machine-config-daemon-6hnt5 Started container oauth-proxy
Nov 07 02:24:42.233 I ns/openshift-dns pod/dns-default-pgkfd Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:614764223d37c466a211f456a9d02c5881d947e1f05d541689dc34a3ec19acf2" already present on machine
Nov 07 02:24:42.440 I ns/openshift-dns pod/dns-default-pgkfd Created container dns
Nov 07 02:24:42.485 I ns/openshift-dns pod/dns-default-pgkfd Started container dns
Nov 07 02:24:42.496 I ns/openshift-dns pod/dns-default-pgkfd Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd150f3e4a29d160fae9610f1096180eaeb29ff29b63bf490b0022b4d3913e29" already present on machine
Nov 07 02:24:42.684 I ns/openshift-dns pod/dns-default-pgkfd Created container dns-node-resolver
Nov 07 02:24:42.724 I ns/openshift-dns pod/dns-default-pgkfd Started container dns-node-resolver
Nov 07 02:24:42.915 I ns/openshift-image-registry pod/node-ca-tbz92 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3191fc23d0d7e820100498c5085592219a747743f06fc059436945df6ab3e8ee" already present on machine
Nov 07 02:24:43.122 I ns/openshift-image-registry pod/node-ca-tbz92 Created container node-ca
Nov 07 02:24:43.160 I ns/openshift-image-registry pod/node-ca-tbz92 Started container node-ca
Nov 07 02:24:59.915 W clusteroperator/dns changed Progressing to False: AsExpected: Desired and available number of DNS DaemonSets are equal
Nov 07 02:25:57.079 W ns/openshift-ingress service/router-default Error updating load balancer with new hosts map[ci-op-pg7mtgss-09f59-ng62w-master-0:{} ci-op-pg7mtgss-09f59-ng62w-master-1:{} ci-op-pg7mtgss-09f59-ng62w-master-2:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-z8fpv:{}]: ensure(openshift-ingress/router-default): backendPoolID(/subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-pg7mtgss-09f59-ng62w-rg/providers/Microsoft.Network/loadBalancers/ci-op-pg7mtgss-09f59-ng62w/backendAddressPools/ci-op-pg7mtgss-09f59-ng62w) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:NicGet" (7 times)
Nov 07 02:27:37.536 W ns/openshift-ingress service/router-default Error updating load balancer with new hosts map[ci-op-pg7mtgss-09f59-ng62w-master-0:{} ci-op-pg7mtgss-09f59-ng62w-master-1:{} ci-op-pg7mtgss-09f59-ng62w-master-2:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-z8fpv:{}]: ensure(openshift-ingress/router-default): backendPoolID(/subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-pg7mtgss-09f59-ng62w-rg/providers/Microsoft.Network/loadBalancers/ci-op-pg7mtgss-09f59-ng62w/backendAddressPools/ci-op-pg7mtgss-09f59-ng62w) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:NicGet" (8 times)
Nov 07 02:27:55.734 I ns/openshift-machine-config-operator pod/machine-config-daemon-6hnt5 Marking for deletion Pod openshift-machine-config-operator/machine-config-daemon-6hnt5
Nov 07 02:27:55.760 W ns/openshift-image-registry pod/node-ca-tbz92 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 graceful deletion within 30s
Nov 07 02:27:55.760 I ns/openshift-dns pod/dns-default-pgkfd Marking for deletion Pod openshift-dns/dns-default-pgkfd
Nov 07 02:27:55.760 I ns/openshift-image-registry daemonset/node-ca Deleted pod: node-ca-tbz92
Nov 07 02:27:55.761 W ns/openshift-dns pod/dns-default-pgkfd node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 graceful deletion within 30s
Nov 07 02:27:55.779 I ns/openshift-image-registry pod/node-ca-tbz92 Marking for deletion Pod openshift-image-registry/node-ca-tbz92
Nov 07 02:27:55.779 I ns/openshift-dns daemonset/dns-default Deleted pod: dns-default-pgkfd
Nov 07 02:27:55.780 W ns/openshift-machine-config-operator pod/machine-config-daemon-6hnt5 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 graceful deletion within 600s
Nov 07 02:27:55.785 I ns/openshift-machine-config-operator pod/machine-config-daemon-6hnt5 Stopping container oauth-proxy
Nov 07 02:27:55.785 I ns/openshift-machine-config-operator daemonset/machine-config-daemon Deleted pod: machine-config-daemon-6hnt5
Nov 07 02:27:55.810 I ns/openshift-machine-config-operator pod/machine-config-daemon-6hnt5 Stopping container machine-config-daemon
Nov 07 02:27:55.840 I ns/openshift-image-registry pod/node-ca-tbz92 Stopping container node-ca
Nov 07 02:27:55.868 I ns/openshift-dns pod/dns-default-pgkfd Stopping container dns
Nov 07 02:27:55.875 I ns/openshift-dns pod/dns-default-pgkfd Stopping container dns-node-resolver
Nov 07 02:27:56.707 E ns/openshift-machine-config-operator pod/machine-config-daemon-6hnt5 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=oauth-proxy container exited with code 143 (Error): 
Nov 07 02:27:57.845 W ns/openshift-machine-config-operator pod/machine-config-daemon-6hnt5 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 deleted
Nov 07 02:27:57.871 W ns/openshift-image-registry pod/node-ca-tbz92 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:27:57.871 W ns/openshift-image-registry pod/node-ca-tbz92 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=node-ca container stopped being ready
Nov 07 02:27:57.871 I ns/openshift-machine-config-operator pod/machine-config-daemon-hjjm4 node/ created
Nov 07 02:27:57.883 W ns/openshift-dns pod/dns-default-pgkfd node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:27:57.883 W ns/openshift-dns pod/dns-default-pgkfd node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=dns container stopped being ready
Nov 07 02:27:57.883 W ns/openshift-dns pod/dns-default-pgkfd node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=dns-node-resolver container stopped being ready
Nov 07 02:27:57.892 I ns/openshift-machine-config-operator daemonset/machine-config-daemon Created pod: machine-config-daemon-hjjm4
Nov 07 02:27:57.914 I ns/openshift-machine-config-operator pod/machine-config-daemon-hjjm4 Successfully assigned openshift-machine-config-operator/machine-config-daemon-hjjm4 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:27:58.024 W clusteroperator/dns changed Progressing to True: Reconciling: Not all DNS DaemonSets available.
Nov 07 02:27:58.669 I ns/openshift-machine-config-operator pod/machine-config-daemon-hjjm4 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:944d643d229b0a1028464df12f2a544163a6689947df774c6568b3f443a79cb8" already present on machine
Nov 07 02:27:58.934 I ns/openshift-machine-config-operator pod/machine-config-daemon-hjjm4 Created container machine-config-daemon
Nov 07 02:27:58.999 I ns/openshift-machine-config-operator pod/machine-config-daemon-hjjm4 Started container machine-config-daemon
Nov 07 02:27:59.012 I ns/openshift-machine-config-operator pod/machine-config-daemon-hjjm4 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f711682524b64033a4ce9193bd2cd9621e08b4bae44cffffb4e397a43a8f945" already present on machine
Nov 07 02:27:59.545 I ns/openshift-machine-config-operator pod/machine-config-daemon-hjjm4 Created container oauth-proxy
Nov 07 02:27:59.583 I ns/openshift-machine-config-operator pod/machine-config-daemon-hjjm4 Started container oauth-proxy
Nov 07 02:28:01.142 W ns/openshift-image-registry pod/node-ca-tbz92 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 pod has been pending longer than a minute
Nov 07 02:28:01.142 W ns/openshift-dns pod/dns-default-pgkfd node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 pod has been pending longer than a minute
Nov 07 02:28:03.452 W ns/openshift-image-registry pod/node-ca-tbz92 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 deleted
Nov 07 02:28:03.490 W ns/openshift-dns pod/dns-default-pgkfd node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 deleted
Nov 07 02:28:03.515 I ns/openshift-dns pod/dns-default-v4gmv node/ created
Nov 07 02:28:03.515 I ns/openshift-image-registry pod/node-ca-6p69d node/ created
Nov 07 02:28:03.520 I ns/openshift-image-registry daemonset/node-ca Created pod: node-ca-6p69d
Nov 07 02:28:03.540 I ns/openshift-dns daemonset/dns-default Created pod: dns-default-v4gmv
Nov 07 02:28:03.554 I ns/openshift-image-registry pod/node-ca-6p69d Successfully assigned openshift-image-registry/node-ca-6p69d to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:28:03.558 I ns/openshift-dns pod/dns-default-v4gmv Successfully assigned openshift-dns/dns-default-v4gmv to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:28:04.846 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (26 times)
Nov 07 02:28:05.439 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-008" not found) (4 times)
Nov 07 02:28:12.261 I ns/openshift-image-registry pod/node-ca-6p69d Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3191fc23d0d7e820100498c5085592219a747743f06fc059436945df6ab3e8ee" already present on machine
Nov 07 02:28:12.421 I ns/openshift-dns pod/dns-default-v4gmv Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:614764223d37c466a211f456a9d02c5881d947e1f05d541689dc34a3ec19acf2" already present on machine
Nov 07 02:28:12.468 I ns/openshift-image-registry pod/node-ca-6p69d Created container node-ca
Nov 07 02:28:12.533 I ns/openshift-image-registry pod/node-ca-6p69d Started container node-ca
Nov 07 02:28:12.664 I ns/openshift-dns pod/dns-default-v4gmv Created container dns
Nov 07 02:28:12.705 I ns/openshift-dns pod/dns-default-v4gmv Started container dns
Nov 07 02:28:12.710 I ns/openshift-dns pod/dns-default-v4gmv Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd150f3e4a29d160fae9610f1096180eaeb29ff29b63bf490b0022b4d3913e29" already present on machine
Nov 07 02:28:12.910 I ns/openshift-dns pod/dns-default-v4gmv Created container dns-node-resolver
Nov 07 02:28:12.962 I ns/openshift-dns pod/dns-default-v4gmv Started container dns-node-resolver
Nov 07 02:28:23.936 W clusteroperator/dns changed Progressing to False: AsExpected: Desired and available number of DNS DaemonSets are equal
Nov 07 02:29:17.963 W ns/openshift-ingress service/router-default Error updating load balancer with new hosts map[ci-op-pg7mtgss-09f59-ng62w-master-0:{} ci-op-pg7mtgss-09f59-ng62w-master-1:{} ci-op-pg7mtgss-09f59-ng62w-master-2:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-z8fpv:{}]: ensure(openshift-ingress/router-default): backendPoolID(/subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-pg7mtgss-09f59-ng62w-rg/providers/Microsoft.Network/loadBalancers/ci-op-pg7mtgss-09f59-ng62w/backendAddressPools/ci-op-pg7mtgss-09f59-ng62w) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:NicGet" (9 times)
Nov 07 02:30:26.230 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (28 times)
Nov 07 02:30:26.817 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-008" not found) (5 times)
Nov 07 02:30:56.819 I ns/openshift-machine-config-operator pod/machine-config-daemon-hjjm4 Marking for deletion Pod openshift-machine-config-operator/machine-config-daemon-hjjm4
Nov 07 02:30:56.820 W ns/openshift-machine-config-operator pod/machine-config-daemon-hjjm4 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 graceful deletion within 600s
Nov 07 02:30:56.833 W ns/openshift-image-registry pod/node-ca-6p69d node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 graceful deletion within 30s
Nov 07 02:30:56.837 W ns/openshift-dns pod/dns-default-v4gmv node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 graceful deletion within 30s
Nov 07 02:30:56.844 I ns/openshift-image-registry pod/node-ca-6p69d Marking for deletion Pod openshift-image-registry/node-ca-6p69d
Nov 07 02:30:56.844 I ns/openshift-machine-config-operator pod/machine-config-daemon-hjjm4 Stopping container machine-config-daemon
Nov 07 02:30:56.857 I ns/openshift-image-registry daemonset/node-ca Deleted pod: node-ca-6p69d
Nov 07 02:30:56.858 I ns/openshift-machine-config-operator pod/machine-config-daemon-hjjm4 Stopping container oauth-proxy
Nov 07 02:30:56.862 I ns/openshift-dns pod/dns-default-v4gmv Marking for deletion Pod openshift-dns/dns-default-v4gmv
Nov 07 02:30:56.870 I ns/openshift-dns daemonset/dns-default Deleted pod: dns-default-v4gmv
Nov 07 02:30:56.879 I ns/openshift-image-registry pod/node-ca-6p69d Stopping container node-ca
Nov 07 02:30:56.893 I ns/openshift-dns pod/dns-default-v4gmv Stopping container dns-node-resolver
Nov 07 02:30:56.909 I ns/openshift-dns pod/dns-default-v4gmv Stopping container dns
Nov 07 02:30:58.518 W ns/openshift-machine-config-operator pod/machine-config-daemon-hjjm4 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:30:58.518 W ns/openshift-machine-config-operator pod/machine-config-daemon-hjjm4 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=machine-config-daemon container stopped being ready
Nov 07 02:30:58.518 W ns/openshift-machine-config-operator pod/machine-config-daemon-hjjm4 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=oauth-proxy container stopped being ready
Nov 07 02:30:58.538 E ns/openshift-machine-config-operator pod/machine-config-daemon-hjjm4 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=machine-config-daemon container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Nov 07 02:30:58.538 E ns/openshift-machine-config-operator pod/machine-config-daemon-hjjm4 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=oauth-proxy container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Nov 07 02:30:58.555 W ns/openshift-image-registry pod/node-ca-6p69d node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:30:58.555 W ns/openshift-image-registry pod/node-ca-6p69d node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=node-ca container stopped being ready
Nov 07 02:30:59.538 W ns/openshift-image-registry pod/node-ca-6p69d node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 deleted
Nov 07 02:30:59.566 W ns/openshift-dns pod/dns-default-v4gmv node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:30:59.566 W ns/openshift-dns pod/dns-default-v4gmv node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=dns-node-resolver container stopped being ready
Nov 07 02:30:59.566 W ns/openshift-dns pod/dns-default-v4gmv node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=dns container stopped being ready
Nov 07 02:30:59.595 I ns/openshift-image-registry pod/node-ca-pwldx node/ created
Nov 07 02:30:59.606 I ns/openshift-image-registry daemonset/node-ca Created pod: node-ca-pwldx
Nov 07 02:30:59.627 I ns/openshift-image-registry pod/node-ca-pwldx Successfully assigned openshift-image-registry/node-ca-pwldx to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:30:59.699 W clusteroperator/dns changed Progressing to True: Reconciling: Not all DNS DaemonSets available.
Nov 07 02:31:01.142 W ns/openshift-dns pod/dns-default-v4gmv node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 pod has been pending longer than a minute
Nov 07 02:31:01.142 W ns/openshift-machine-config-operator pod/machine-config-daemon-hjjm4 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 pod has been pending longer than a minute
Nov 07 02:31:03.452 W ns/openshift-machine-config-operator pod/machine-config-daemon-hjjm4 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 deleted
Nov 07 02:31:03.487 I ns/openshift-machine-config-operator pod/machine-config-daemon-pc6z7 node/ created
Nov 07 02:31:03.489 I ns/openshift-machine-config-operator daemonset/machine-config-daemon Created pod: machine-config-daemon-pc6z7
Nov 07 02:31:03.501 W ns/openshift-dns pod/dns-default-v4gmv node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 deleted
Nov 07 02:31:03.524 I ns/openshift-machine-config-operator pod/machine-config-daemon-pc6z7 Successfully assigned openshift-machine-config-operator/machine-config-daemon-pc6z7 to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:31:03.525 I ns/openshift-dns pod/dns-default-5sf8k node/ created
Nov 07 02:31:03.537 I ns/openshift-dns daemonset/dns-default Created pod: dns-default-5sf8k
Nov 07 02:31:03.564 I ns/openshift-dns pod/dns-default-5sf8k Successfully assigned openshift-dns/dns-default-5sf8k to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:31:04.792 I ns/openshift-machine-config-operator pod/machine-config-daemon-pc6z7 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:944d643d229b0a1028464df12f2a544163a6689947df774c6568b3f443a79cb8" already present on machine
Nov 07 02:31:05.037 I ns/openshift-machine-config-operator pod/machine-config-daemon-pc6z7 Created container machine-config-daemon
Nov 07 02:31:05.072 I ns/openshift-machine-config-operator pod/machine-config-daemon-pc6z7 Started container machine-config-daemon
Nov 07 02:31:05.083 I ns/openshift-machine-config-operator pod/machine-config-daemon-pc6z7 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f711682524b64033a4ce9193bd2cd9621e08b4bae44cffffb4e397a43a8f945" already present on machine
Nov 07 02:31:05.369 I ns/openshift-machine-config-operator pod/machine-config-daemon-pc6z7 Created container oauth-proxy
Nov 07 02:31:05.412 I ns/openshift-machine-config-operator pod/machine-config-daemon-pc6z7 Started container oauth-proxy
Nov 07 02:31:08.367 I ns/openshift-image-registry pod/node-ca-pwldx Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3191fc23d0d7e820100498c5085592219a747743f06fc059436945df6ab3e8ee" already present on machine
Nov 07 02:31:08.588 I ns/openshift-image-registry pod/node-ca-pwldx Created container node-ca
Nov 07 02:31:08.641 I ns/openshift-image-registry pod/node-ca-pwldx Started container node-ca
Nov 07 02:31:11.978 I ns/openshift-dns pod/dns-default-5sf8k Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:614764223d37c466a211f456a9d02c5881d947e1f05d541689dc34a3ec19acf2" already present on machine
Nov 07 02:31:12.210 I ns/openshift-dns pod/dns-default-5sf8k Created container dns
Nov 07 02:31:12.255 I ns/openshift-dns pod/dns-default-5sf8k Started container dns
Nov 07 02:31:12.269 I ns/openshift-dns pod/dns-default-5sf8k Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd150f3e4a29d160fae9610f1096180eaeb29ff29b63bf490b0022b4d3913e29" already present on machine
Nov 07 02:31:12.474 I ns/openshift-dns pod/dns-default-5sf8k Created container dns-node-resolver
Nov 07 02:31:12.508 I ns/openshift-dns pod/dns-default-5sf8k Started container dns-node-resolver
Nov 07 02:31:28.587 W clusteroperator/dns changed Progressing to False: AsExpected: Desired and available number of DNS DaemonSets are equal
Nov 07 02:32:17.519 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-0 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (7 times)
Nov 07 02:32:18.120 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-0 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-008" not found) (5 times)
Nov 07 02:34:19.239 W ns/openshift-ingress service/router-default Error updating load balancer with new hosts map[ci-op-pg7mtgss-09f59-ng62w-master-0:{} ci-op-pg7mtgss-09f59-ng62w-master-1:{} ci-op-pg7mtgss-09f59-ng62w-master-2:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-z8fpv:{}]: ensure(openshift-ingress/router-default): backendPoolID(/subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-pg7mtgss-09f59-ng62w-rg/providers/Microsoft.Network/loadBalancers/ci-op-pg7mtgss-09f59-ng62w/backendAddressPools/ci-op-pg7mtgss-09f59-ng62w) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:NicGet" (12 times)
Nov 07 02:38:04.855 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (9 times)
Nov 07 02:38:05.445 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-008" not found) (6 times)
Nov 07 02:39:20.549 W ns/openshift-ingress service/router-default Error updating load balancer with new hosts map[ci-op-pg7mtgss-09f59-ng62w-master-0:{} ci-op-pg7mtgss-09f59-ng62w-master-1:{} ci-op-pg7mtgss-09f59-ng62w-master-2:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-z8fpv:{}]: ensure(openshift-ingress/router-default): backendPoolID(/subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-pg7mtgss-09f59-ng62w-rg/providers/Microsoft.Network/loadBalancers/ci-op-pg7mtgss-09f59-ng62w/backendAddressPools/ci-op-pg7mtgss-09f59-ng62w) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:NicGet" (15 times)
Nov 07 02:40:26.232 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (5 times)
Nov 07 02:40:26.827 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-008" not found) (7 times)
Nov 07 02:41:34.311 I ns/openshift-dns pod/dns-default-5sf8k Marking for deletion Pod openshift-dns/dns-default-5sf8k
Nov 07 02:41:34.328 W ns/openshift-machine-config-operator pod/machine-config-daemon-pc6z7 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 graceful deletion within 600s
Nov 07 02:41:34.330 W ns/openshift-dns pod/dns-default-5sf8k node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 graceful deletion within 30s
Nov 07 02:41:34.331 I ns/openshift-machine-config-operator pod/machine-config-daemon-pc6z7 Stopping container oauth-proxy
Nov 07 02:41:34.349 I ns/openshift-image-registry pod/node-ca-pwldx Marking for deletion Pod openshift-image-registry/node-ca-pwldx
Nov 07 02:41:34.350 W ns/openshift-image-registry pod/node-ca-pwldx node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 graceful deletion within 30s
Nov 07 02:41:34.387 I ns/openshift-machine-config-operator pod/machine-config-daemon-pc6z7 Stopping container machine-config-daemon
Nov 07 02:41:34.387 I ns/openshift-machine-config-operator daemonset/machine-config-daemon Deleted pod: machine-config-daemon-pc6z7
Nov 07 02:41:34.387 I ns/openshift-machine-config-operator pod/machine-config-daemon-pc6z7 Marking for deletion Pod openshift-machine-config-operator/machine-config-daemon-pc6z7
Nov 07 02:41:34.403 I ns/openshift-dns daemonset/dns-default Deleted pod: dns-default-5sf8k
Nov 07 02:41:34.437 I ns/openshift-dns pod/dns-default-5sf8k Stopping container dns-node-resolver
Nov 07 02:41:34.452 I ns/openshift-dns pod/dns-default-5sf8k Stopping container dns
Nov 07 02:41:34.463 I ns/openshift-image-registry pod/node-ca-pwldx Stopping container node-ca
Nov 07 02:41:36.387 W ns/openshift-dns pod/dns-default-5sf8k node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:41:36.387 W ns/openshift-dns pod/dns-default-5sf8k node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=dns container stopped being ready
Nov 07 02:41:36.387 W ns/openshift-dns pod/dns-default-5sf8k node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=dns-node-resolver container stopped being ready
Nov 07 02:41:36.425 W ns/openshift-machine-config-operator pod/machine-config-daemon-pc6z7 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:41:36.425 W ns/openshift-machine-config-operator pod/machine-config-daemon-pc6z7 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=oauth-proxy container stopped being ready
Nov 07 02:41:36.425 W ns/openshift-machine-config-operator pod/machine-config-daemon-pc6z7 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=machine-config-daemon container stopped being ready
Nov 07 02:41:36.425 W ns/openshift-image-registry pod/node-ca-pwldx node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:41:36.425 W ns/openshift-image-registry pod/node-ca-pwldx node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=node-ca container stopped being ready
Nov 07 02:41:43.458 W ns/openshift-image-registry pod/node-ca-pwldx node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 deleted
Nov 07 02:41:43.490 W ns/openshift-machine-config-operator pod/machine-config-daemon-pc6z7 node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 deleted
Nov 07 02:41:43.525 W ns/openshift-dns pod/dns-default-5sf8k node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 deleted
Nov 07 02:42:03.613 I ns/openshift-dns pod/dns-default-mwtnp node/ created
Nov 07 02:42:03.620 I ns/openshift-dns daemonset/dns-default Created pod: dns-default-mwtnp
Nov 07 02:42:03.644 I ns/openshift-dns pod/dns-default-mwtnp Successfully assigned openshift-dns/dns-default-mwtnp to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:42:03.652 I ns/openshift-image-registry pod/node-ca-lsg4z node/ created
Nov 07 02:42:03.665 I ns/openshift-machine-config-operator pod/machine-config-daemon-w4s4p node/ created
Nov 07 02:42:03.676 I ns/openshift-image-registry daemonset/node-ca Created pod: node-ca-lsg4z
Nov 07 02:42:03.690 I ns/openshift-machine-config-operator daemonset/machine-config-daemon Created pod: machine-config-daemon-w4s4p
Nov 07 02:42:03.706 I ns/openshift-image-registry pod/node-ca-lsg4z Successfully assigned openshift-image-registry/node-ca-lsg4z to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:42:03.724 I ns/openshift-machine-config-operator pod/machine-config-daemon-w4s4p Successfully assigned openshift-machine-config-operator/machine-config-daemon-w4s4p to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:42:03.816 W clusteroperator/dns changed Progressing to True: Reconciling: Not all DNS DaemonSets available.
Nov 07 02:42:05.284 I ns/openshift-machine-config-operator pod/machine-config-daemon-w4s4p Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:944d643d229b0a1028464df12f2a544163a6689947df774c6568b3f443a79cb8" already present on machine
Nov 07 02:42:05.539 I ns/openshift-machine-config-operator pod/machine-config-daemon-w4s4p Created container machine-config-daemon
Nov 07 02:42:05.586 I ns/openshift-machine-config-operator pod/machine-config-daemon-w4s4p Started container machine-config-daemon
Nov 07 02:42:05.594 I ns/openshift-machine-config-operator pod/machine-config-daemon-w4s4p Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f711682524b64033a4ce9193bd2cd9621e08b4bae44cffffb4e397a43a8f945" already present on machine
Nov 07 02:42:05.865 I ns/openshift-machine-config-operator pod/machine-config-daemon-w4s4p Created container oauth-proxy
Nov 07 02:42:05.916 I ns/openshift-machine-config-operator pod/machine-config-daemon-w4s4p Started container oauth-proxy
Nov 07 02:42:06.676 I ns/openshift-marketplace deployment/community-operators Scaled up replica set community-operators-7857bd4d58 to 1
Nov 07 02:42:06.769 I ns/openshift-marketplace pod/community-operators-7857bd4d58-rfnpn node/ created
Nov 07 02:42:06.785 I ns/openshift-marketplace replicaset/community-operators-7857bd4d58 Created pod: community-operators-7857bd4d58-rfnpn
Nov 07 02:42:06.812 I ns/openshift-marketplace pod/community-operators-7857bd4d58-rfnpn Successfully assigned openshift-marketplace/community-operators-7857bd4d58-rfnpn to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:42:12.108 I ns/openshift-image-registry pod/node-ca-lsg4z Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3191fc23d0d7e820100498c5085592219a747743f06fc059436945df6ab3e8ee" already present on machine
Nov 07 02:42:12.156 I ns/openshift-dns pod/dns-default-mwtnp Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:614764223d37c466a211f456a9d02c5881d947e1f05d541689dc34a3ec19acf2" already present on machine
Nov 07 02:42:12.457 I ns/openshift-image-registry pod/node-ca-lsg4z Created container node-ca
Nov 07 02:42:12.467 I ns/openshift-dns pod/dns-default-mwtnp Created container dns
Nov 07 02:42:12.532 I ns/openshift-image-registry pod/node-ca-lsg4z Started container node-ca
Nov 07 02:42:12.539 I ns/openshift-dns pod/dns-default-mwtnp Started container dns
Nov 07 02:42:12.553 I ns/openshift-dns pod/dns-default-mwtnp Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd150f3e4a29d160fae9610f1096180eaeb29ff29b63bf490b0022b4d3913e29" already present on machine
Nov 07 02:42:12.740 I ns/openshift-dns pod/dns-default-mwtnp Created container dns-node-resolver
Nov 07 02:42:12.777 I ns/openshift-dns pod/dns-default-mwtnp Started container dns-node-resolver
Nov 07 02:42:14.615 I ns/openshift-marketplace pod/community-operators-7857bd4d58-rfnpn Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:448858c7b1f70b8c0955965ef9970b59504b220bf03b760203305c81e5f606da" already present on machine
Nov 07 02:42:14.848 I ns/openshift-marketplace pod/community-operators-7857bd4d58-rfnpn Created container community-operators
Nov 07 02:42:14.897 I ns/openshift-marketplace pod/community-operators-7857bd4d58-rfnpn Started container community-operators
Nov 07 02:42:17.545 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-0 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (9 times)
Nov 07 02:42:18.143 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-0 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-008" not found) (35 times)
Nov 07 02:42:23.226 W ns/openshift-marketplace pod/community-operators-7857bd4d58-rfnpn Liveness probe errored: rpc error: code = Unknown desc = command error: command timed out, stdout: , stderr: , exit code -1
Nov 07 02:42:24.674 W ns/openshift-marketplace pod/community-operators-7857bd4d58-rfnpn Readiness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n
Nov 07 02:42:27.135 W clusteroperator/dns changed Progressing to False: AsExpected: Desired and available number of DNS DaemonSets are equal
Nov 07 02:42:31.499 W ns/openshift-marketplace pod/community-operators-7857bd4d58-rfnpn Liveness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n
Nov 07 02:42:34.688 W ns/openshift-marketplace pod/community-operators-7857bd4d58-rfnpn Readiness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n (2 times)
Nov 07 02:42:41.939 W ns/openshift-marketplace pod/community-operators-7857bd4d58-rfnpn Liveness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n (2 times)
Nov 07 02:42:43.734 W ns/openshift-marketplace pod/community-operators-9567c47bd-2thsq node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj graceful deletion within 30s
Nov 07 02:42:43.741 I ns/openshift-marketplace deployment/community-operators Scaled down replica set community-operators-9567c47bd to 0
Nov 07 02:42:43.759 I ns/openshift-marketplace replicaset/community-operators-9567c47bd Deleted pod: community-operators-9567c47bd-2thsq
Nov 07 02:42:43.763 I ns/openshift-marketplace pod/community-operators-9567c47bd-2thsq Stopping container community-operators
Nov 07 02:42:44.825 E ns/openshift-marketplace pod/community-operators-9567c47bd-2thsq node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj container=community-operators container exited with code 2 (Error): 
Nov 07 02:42:45.837 W ns/openshift-marketplace pod/community-operators-9567c47bd-2thsq node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj deleted
Nov 07 02:44:52.330 W ns/openshift-ingress service/router-default Error updating load balancer with new hosts map[ci-op-pg7mtgss-09f59-ng62w-master-0:{} ci-op-pg7mtgss-09f59-ng62w-master-1:{} ci-op-pg7mtgss-09f59-ng62w-master-2:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-z8fpv:{}]: ensure(openshift-ingress/router-default): backendPoolID(/subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-pg7mtgss-09f59-ng62w-rg/providers/Microsoft.Network/loadBalancers/ci-op-pg7mtgss-09f59-ng62w/backendAddressPools/ci-op-pg7mtgss-09f59-ng62w) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:NicGet" (17 times)
Nov 07 02:48:04.862 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (39 times)
Nov 07 02:48:05.455 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-008" not found) (7 times)
Nov 07 02:49:53.784 W ns/openshift-ingress service/router-default Error updating load balancer with new hosts map[ci-op-pg7mtgss-09f59-ng62w-master-0:{} ci-op-pg7mtgss-09f59-ng62w-master-1:{} ci-op-pg7mtgss-09f59-ng62w-master-2:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-z8fpv:{}]: ensure(openshift-ingress/router-default): backendPoolID(/subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-pg7mtgss-09f59-ng62w-rg/providers/Microsoft.Network/loadBalancers/ci-op-pg7mtgss-09f59-ng62w/backendAddressPools/ci-op-pg7mtgss-09f59-ng62w) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:NicGet" (20 times)
Nov 07 02:50:26.259 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (6 times)
Nov 07 02:50:26.837 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-008" not found) (9 times)
Nov 07 02:52:17.586 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-0 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (10 times)
Nov 07 02:52:18.180 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-0 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-008" not found) (8 times)
Nov 07 02:53:59.948 W ns/openshift-image-registry pod/node-ca-lsg4z node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 graceful deletion within 30s
Nov 07 02:53:59.963 I ns/openshift-image-registry pod/node-ca-lsg4z Marking for deletion Pod openshift-image-registry/node-ca-lsg4z
Nov 07 02:53:59.976 I ns/openshift-machine-config-operator pod/machine-config-daemon-w4s4p Marking for deletion Pod openshift-machine-config-operator/machine-config-daemon-w4s4p
Nov 07 02:53:59.977 W ns/openshift-dns pod/dns-default-mwtnp node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 graceful deletion within 30s
Nov 07 02:53:59.984 I ns/openshift-image-registry pod/node-ca-lsg4z Stopping container node-ca
Nov 07 02:53:59.985 W ns/openshift-machine-config-operator pod/machine-config-daemon-w4s4p node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 graceful deletion within 600s
Nov 07 02:53:59.985 W ns/openshift-marketplace pod/community-operators-7857bd4d58-rfnpn node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 graceful deletion within 30s
Nov 07 02:53:59.991 I ns/openshift-dns pod/dns-default-mwtnp Marking for deletion Pod openshift-dns/dns-default-mwtnp
Nov 07 02:53:59.991 I ns/openshift-dns pod/dns-default-mwtnp Stopping container dns
Nov 07 02:54:00.004 I ns/openshift-machine-config-operator daemonset/machine-config-daemon Deleted pod: machine-config-daemon-w4s4p
Nov 07 02:54:00.011 I ns/openshift-machine-config-operator pod/machine-config-daemon-w4s4p Stopping container oauth-proxy
Nov 07 02:54:00.014 I ns/openshift-marketplace pod/community-operators-7857bd4d58-rfnpn Marking for deletion Pod openshift-marketplace/community-operators-7857bd4d58-rfnpn
Nov 07 02:54:00.021 I ns/openshift-image-registry daemonset/node-ca Deleted pod: node-ca-lsg4z
Nov 07 02:54:00.021 I ns/openshift-machine-config-operator pod/machine-config-daemon-w4s4p Stopping container machine-config-daemon
Nov 07 02:54:00.046 I ns/openshift-dns pod/dns-default-mwtnp Stopping container dns-node-resolver
Nov 07 02:54:00.062 I ns/openshift-marketplace pod/community-operators-7857bd4d58-bgtjg node/ created
Nov 07 02:54:00.081 I ns/openshift-marketplace replicaset/community-operators-7857bd4d58 Created pod: community-operators-7857bd4d58-bgtjg
Nov 07 02:54:00.095 I ns/openshift-marketplace pod/community-operators-7857bd4d58-rfnpn Stopping container community-operators
Nov 07 02:54:00.138 I ns/openshift-marketplace pod/community-operators-7857bd4d58-bgtjg Successfully assigned openshift-marketplace/community-operators-7857bd4d58-bgtjg to ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj
Nov 07 02:54:00.447 W ns/openshift-marketplace pod/community-operators-7857bd4d58-rfnpn Liveness probe errored: rpc error: code = Unknown desc = container is not created or running
Nov 07 02:54:02.530 W ns/openshift-marketplace pod/community-operators-7857bd4d58-rfnpn node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:54:02.530 W ns/openshift-marketplace pod/community-operators-7857bd4d58-rfnpn node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=community-operators container stopped being ready
Nov 07 02:54:02.547 E ns/openshift-marketplace pod/community-operators-7857bd4d58-rfnpn node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=community-operators container exited with code 137 (ContainerStatusUnknown): The container could not be located when the pod was terminated
Nov 07 02:54:02.572 W ns/openshift-dns pod/dns-default-mwtnp node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:54:02.572 W ns/openshift-dns pod/dns-default-mwtnp node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=dns container stopped being ready
Nov 07 02:54:02.572 W ns/openshift-dns pod/dns-default-mwtnp node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=dns-node-resolver container stopped being ready
Nov 07 02:54:02.593 W ns/openshift-image-registry pod/node-ca-lsg4z node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:54:02.593 W ns/openshift-image-registry pod/node-ca-lsg4z node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=node-ca container stopped being ready
Nov 07 02:54:02.618 W ns/openshift-machine-config-operator pod/machine-config-daemon-w4s4p node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 invariant violation (bug): pod should not transition Running->Pending even when terminated
Nov 07 02:54:02.618 W ns/openshift-machine-config-operator pod/machine-config-daemon-w4s4p node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=oauth-proxy container stopped being ready
Nov 07 02:54:02.618 W ns/openshift-machine-config-operator pod/machine-config-daemon-w4s4p node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 container=machine-config-daemon container stopped being ready
Nov 07 02:54:03.546 W ns/openshift-machine-config-operator pod/machine-config-daemon-w4s4p node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 deleted
Nov 07 02:54:03.584 W ns/openshift-dns pod/dns-default-mwtnp node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 deleted
Nov 07 02:54:08.384 I ns/openshift-marketplace pod/community-operators-7857bd4d58-bgtjg Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:448858c7b1f70b8c0955965ef9970b59504b220bf03b760203305c81e5f606da" already present on machine
Nov 07 02:54:09.463 I ns/openshift-marketplace pod/community-operators-7857bd4d58-bgtjg Created container community-operators
Nov 07 02:54:09.527 I ns/openshift-marketplace pod/community-operators-7857bd4d58-bgtjg Started container community-operators
Nov 07 02:54:13.453 W ns/openshift-image-registry pod/node-ca-lsg4z node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 deleted
Nov 07 02:54:13.493 W ns/openshift-marketplace pod/community-operators-7857bd4d58-rfnpn node/ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9 deleted
Nov 07 02:54:24.182 W ns/openshift-marketplace pod/community-operators-7857bd4d58-bgtjg Liveness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n
Nov 07 02:54:24.200 W ns/openshift-marketplace pod/community-operators-7857bd4d58-bgtjg Readiness probe errored: rpc error: code = Unknown desc = command error: command timed out, stdout: , stderr: , exit code -1
Nov 07 02:54:34.219 W ns/openshift-marketplace pod/community-operators-7857bd4d58-bgtjg Readiness probe errored: rpc error: code = Unknown desc = command error: command timed out, stdout: , stderr: , exit code -1 (2 times)
Nov 07 02:54:36.199 W ns/openshift-marketplace pod/community-operators-7857bd4d58-bgtjg Liveness probe errored: rpc error: code = Unknown desc = command error: command timed out, stdout: , stderr: , exit code -1
Nov 07 02:54:42.711 W ns/openshift-marketplace pod/community-operators-7857bd4d58-bgtjg Readiness probe failed: timeout: failed to connect service "localhost:50051" within 1s\n
Nov 07 02:54:55.043 W ns/openshift-ingress service/router-default Error updating load balancer with new hosts map[ci-op-pg7mtgss-09f59-ng62w-master-0:{} ci-op-pg7mtgss-09f59-ng62w-master-1:{} ci-op-pg7mtgss-09f59-ng62w-master-2:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-z8fpv:{}]: ensure(openshift-ingress/router-default): backendPoolID(/subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-pg7mtgss-09f59-ng62w-rg/providers/Microsoft.Network/loadBalancers/ci-op-pg7mtgss-09f59-ng62w/backendAddressPools/ci-op-pg7mtgss-09f59-ng62w) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:NicGet" (23 times)
Nov 07 02:55:05.116 I ns/openshift-dns pod/dns-default-gfqvb node/ created
Nov 07 02:55:05.127 I ns/openshift-dns daemonset/dns-default Created pod: dns-default-gfqvb
Nov 07 02:55:05.143 I ns/openshift-dns pod/dns-default-gfqvb Successfully assigned openshift-dns/dns-default-gfqvb to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:55:05.160 I ns/openshift-machine-config-operator pod/machine-config-daemon-hvx6z node/ created
Nov 07 02:55:05.160 I ns/openshift-image-registry pod/node-ca-vjk7b node/ created
Nov 07 02:55:05.179 I ns/openshift-image-registry daemonset/node-ca Created pod: node-ca-vjk7b
Nov 07 02:55:05.191 I ns/openshift-machine-config-operator daemonset/machine-config-daemon Created pod: machine-config-daemon-hvx6z
Nov 07 02:55:05.213 I ns/openshift-machine-config-operator pod/machine-config-daemon-hvx6z Successfully assigned openshift-machine-config-operator/machine-config-daemon-hvx6z to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:55:05.219 I ns/openshift-image-registry pod/node-ca-vjk7b Successfully assigned openshift-image-registry/node-ca-vjk7b to ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9
Nov 07 02:55:05.281 W clusteroperator/dns changed Progressing to True: Reconciling: Not all DNS DaemonSets available.
Nov 07 02:55:06.406 I ns/openshift-machine-config-operator pod/machine-config-daemon-hvx6z Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:944d643d229b0a1028464df12f2a544163a6689947df774c6568b3f443a79cb8" already present on machine
Nov 07 02:55:06.616 I ns/openshift-machine-config-operator pod/machine-config-daemon-hvx6z Created container machine-config-daemon
Nov 07 02:55:06.664 I ns/openshift-machine-config-operator pod/machine-config-daemon-hvx6z Started container machine-config-daemon
Nov 07 02:55:06.669 I ns/openshift-machine-config-operator pod/machine-config-daemon-hvx6z Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f711682524b64033a4ce9193bd2cd9621e08b4bae44cffffb4e397a43a8f945" already present on machine
Nov 07 02:55:06.939 I ns/openshift-machine-config-operator pod/machine-config-daemon-hvx6z Created container oauth-proxy
Nov 07 02:55:06.978 I ns/openshift-machine-config-operator pod/machine-config-daemon-hvx6z Started container oauth-proxy
Nov 07 02:55:13.964 I ns/openshift-image-registry pod/node-ca-vjk7b Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3191fc23d0d7e820100498c5085592219a747743f06fc059436945df6ab3e8ee" already present on machine
Nov 07 02:55:14.137 I ns/openshift-dns pod/dns-default-gfqvb Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:614764223d37c466a211f456a9d02c5881d947e1f05d541689dc34a3ec19acf2" already present on machine
Nov 07 02:55:14.196 I ns/openshift-image-registry pod/node-ca-vjk7b Created container node-ca
Nov 07 02:55:14.271 I ns/openshift-image-registry pod/node-ca-vjk7b Started container node-ca
Nov 07 02:55:14.429 I ns/openshift-dns pod/dns-default-gfqvb Created container dns
Nov 07 02:55:14.489 I ns/openshift-dns pod/dns-default-gfqvb Started container dns
Nov 07 02:55:14.499 I ns/openshift-dns pod/dns-default-gfqvb Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd150f3e4a29d160fae9610f1096180eaeb29ff29b63bf490b0022b4d3913e29" already present on machine
Nov 07 02:55:14.887 I ns/openshift-dns pod/dns-default-gfqvb Created container dns-node-resolver
Nov 07 02:55:14.922 I ns/openshift-dns pod/dns-default-gfqvb Started container dns-node-resolver
Nov 07 02:55:32.227 W clusteroperator/dns changed Progressing to False: AsExpected: Desired and available number of DNS DaemonSets are equal
Nov 07 02:55:41.723 W node/node1 deleted
Nov 07 02:58:04.871 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (10 times)
Nov 07 02:58:05.472 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-2 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-008" not found) (8 times)
Nov 07 02:59:56.570 W ns/openshift-ingress service/router-default Error updating load balancer with new hosts map[ci-op-pg7mtgss-09f59-ng62w-master-0:{} ci-op-pg7mtgss-09f59-ng62w-master-1:{} ci-op-pg7mtgss-09f59-ng62w-master-2:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus1-sfhqj:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus2-pvtx9:{} ci-op-pg7mtgss-09f59-ng62w-worker-centralus3-z8fpv:{}]: ensure(openshift-ingress/router-default): backendPoolID(/subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-pg7mtgss-09f59-ng62w-rg/providers/Microsoft.Network/loadBalancers/ci-op-pg7mtgss-09f59-ng62w/backendAddressPools/ci-op-pg7mtgss-09f59-ng62w) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:NicGet" (26 times)
Nov 07 03:00:26.248 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (7 times)
Nov 07 03:00:26.846 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-pg7mtgss-09f59-ng62w-master-1 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-008" not found) (47 times)


				Click to see stdout/stderrfrom junit_e2e_20191107-030030.xml

Find was mentions in log files