ResultFAILURE
Tests 3 failed / 22 succeeded
Started2020-04-05 04:02
Elapsed2h3m
Work namespaceci-op-cc8ypj4j
Refs openshift-4.5:fe90dcbe
43:6d37571e
pod2d198c42-76f2-11ea-a522-0a58ac101c32
repoopenshift/etcd
revision1

Test Failures


openshift-tests Monitor cluster while tests execute 1h19m

go run hack/e2e.go -v -test --test_args='--ginkgo.focus=openshift\-tests\sMonitor\scluster\swhile\stests\sexecute$'
31 error level events were detected during this test run:

Apr 05 04:38:54.915 E clusterversion/version changed Failing to True: WorkloadNotAvailable: deployment openshift-cluster-version/cluster-version-operator is progressing NewReplicaSetAvailable: ReplicaSet "cluster-version-operator-77f5f9f6bc" has successfully progressed.
Apr 05 04:40:30.435 E ns/openshift-etcd pod/etcd-ip-10-0-140-247.us-west-1.compute.internal node/ip-10-0-140-247.us-west-1.compute.internal container/etcd-metrics container exited with code 1 (Error): {"level":"info","ts":"2020-04-05T04:40:30.108Z","caller":"etcdmain/grpc_proxy.go:185","msg":"gRPC proxy server TLS","tls-info":"cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.key, trusted-ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-metrics-proxy-serving-ca/ca-bundle.crt, client-cert-auth = false, crl-file = "}\n{"level":"info","ts":"2020-04-05T04:40:30.109Z","caller":"etcdmain/grpc_proxy.go:320","msg":"listening for gRPC proxy client requests","address":"127.0.0.1:9977"}\ncert has non empty Common Name (system:etcd-peer:etcd-client)\n
Apr 05 04:40:48.599 E ns/openshift-etcd pod/etcd-ip-10-0-140-247.us-west-1.compute.internal node/ip-10-0-140-247.us-west-1.compute.internal container/etcd-metrics container exited with code 1 (Error): {"level":"info","ts":"2020-04-05T04:40:47.761Z","caller":"etcdmain/grpc_proxy.go:185","msg":"gRPC proxy server TLS","tls-info":"cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.key, trusted-ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-metrics-proxy-serving-ca/ca-bundle.crt, client-cert-auth = false, crl-file = "}\n{"level":"info","ts":"2020-04-05T04:40:47.763Z","caller":"etcdmain/grpc_proxy.go:320","msg":"listening for gRPC proxy client requests","address":"127.0.0.1:9977"}\ncert has non empty Common Name (system:etcd-peer:etcd-client)\n
Apr 05 04:41:17.713 E ns/openshift-etcd pod/etcd-ip-10-0-140-247.us-west-1.compute.internal node/ip-10-0-140-247.us-west-1.compute.internal container/etcd-metrics container exited with code 1 (Error): {"level":"info","ts":"2020-04-05T04:41:16.663Z","caller":"etcdmain/grpc_proxy.go:185","msg":"gRPC proxy server TLS","tls-info":"cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.key, trusted-ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-metrics-proxy-serving-ca/ca-bundle.crt, client-cert-auth = false, crl-file = "}\n{"level":"info","ts":"2020-04-05T04:41:16.663Z","caller":"etcdmain/grpc_proxy.go:320","msg":"listening for gRPC proxy client requests","address":"127.0.0.1:9977"}\ncert has non empty Common Name (system:etcd-peer:etcd-client)\n
Apr 05 04:42:13.302 E ns/openshift-etcd pod/etcd-ip-10-0-140-247.us-west-1.compute.internal node/ip-10-0-140-247.us-west-1.compute.internal container/etcd-metrics container exited with code 1 (Error): {"level":"info","ts":"2020-04-05T04:42:11.657Z","caller":"etcdmain/grpc_proxy.go:185","msg":"gRPC proxy server TLS","tls-info":"cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.key, trusted-ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-metrics-proxy-serving-ca/ca-bundle.crt, client-cert-auth = false, crl-file = "}\n{"level":"info","ts":"2020-04-05T04:42:11.658Z","caller":"etcdmain/grpc_proxy.go:320","msg":"listening for gRPC proxy client requests","address":"127.0.0.1:9977"}\ncert has non empty Common Name (system:etcd-peer:etcd-client)\n
Apr 05 04:42:32.168 E clusteroperator/etcd changed Degraded to True: StaticPods_Error: StaticPodsDegraded: nodes/ip-10-0-140-247.us-west-1.compute.internal pods/etcd-ip-10-0-140-247.us-west-1.compute.internal container="etcd-metrics" is not ready\nStaticPodsDegraded: nodes/ip-10-0-140-247.us-west-1.compute.internal pods/etcd-ip-10-0-140-247.us-west-1.compute.internal container="etcd-metrics" is waiting: "CrashLoopBackOff" - "back-off 1m20s restarting failed container=etcd-metrics pod=etcd-ip-10-0-140-247.us-west-1.compute.internal_openshift-etcd(0db818134821a4d9f141a9ed8d9bf03e)"
Apr 05 04:43:39.627 E ns/openshift-etcd pod/etcd-ip-10-0-140-247.us-west-1.compute.internal node/ip-10-0-140-247.us-west-1.compute.internal container/etcd-metrics container exited with code 1 (Error): {"level":"info","ts":"2020-04-05T04:43:38.773Z","caller":"etcdmain/grpc_proxy.go:185","msg":"gRPC proxy server TLS","tls-info":"cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.key, trusted-ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-metrics-proxy-serving-ca/ca-bundle.crt, client-cert-auth = false, crl-file = "}\n{"level":"info","ts":"2020-04-05T04:43:38.773Z","caller":"etcdmain/grpc_proxy.go:320","msg":"listening for gRPC proxy client requests","address":"127.0.0.1:9977"}\ncert has non empty Common Name (system:etcd-peer:etcd-client)\n
Apr 05 04:44:17.217 E clusteroperator/openshift-apiserver changed Degraded to True: APIServerDeployment_UnavailablePod: APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-apiserver
Apr 05 04:46:30.392 E ns/openshift-etcd pod/etcd-ip-10-0-140-247.us-west-1.compute.internal node/ip-10-0-140-247.us-west-1.compute.internal container/etcd-metrics container exited with code 1 (Error): {"level":"info","ts":"2020-04-05T04:46:29.657Z","caller":"etcdmain/grpc_proxy.go:185","msg":"gRPC proxy server TLS","tls-info":"cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.key, trusted-ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-metrics-proxy-serving-ca/ca-bundle.crt, client-cert-auth = false, crl-file = "}\n{"level":"info","ts":"2020-04-05T04:46:29.658Z","caller":"etcdmain/grpc_proxy.go:320","msg":"listening for gRPC proxy client requests","address":"127.0.0.1:9977"}\ncert has non empty Common Name (system:etcd-peer:etcd-client)\n
Apr 05 04:47:11.872 E ns/openshift-kube-apiserver pod/kube-apiserver-ip-10-0-139-195.us-west-1.compute.internal node/ip-10-0-139-195.us-west-1.compute.internal container/kube-apiserver container exited with code 1 (Error): t-signing-key-file string   Path to the file that contains the current private key of the service account token issuer. The issuer will sign issued ID tokens with this private key. (Requires the 'TokenRequest' feature gate.)\n      --service-cluster-ip-range string           A CIDR notation IP range from which to assign service cluster IPs. This must not overlap with any IP ranges assigned to nodes for pods.\n      --service-node-port-range portRange         A port range to reserve for services with NodePort visibility. Example: '30000-32767'. Inclusive at both ends of the range. (default 30000-32767)\n\nGlobal flags:\n\n      --add-dir-header                   If true, adds the file directory to the header\n      --alsologtostderr                  log to standard error as well as files\n  -h, --help                             help for kube-apiserver\n      --log-backtrace-at traceLocation   when logging hits line file:N, emit a stack trace (default :0)\n      --log-dir string                   If non-empty, write log files in this directory\n      --log-file string                  If non-empty, use this log file\n      --log-file-max-size uint           Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)\n      --log-flush-frequency duration     Maximum number of seconds between log flushes (default 5s)\n      --logtostderr                      log to standard error instead of files (default true)\n      --skip-headers                     If true, avoid header prefixes in the log messages\n      --skip-log-headers                 If true, avoid headers when opening log files\n      --stderrthreshold severity         logs at or above this threshold go to stderr (default 2)\n  -v, --v Level                          number for the log level verbosity (default 0)\n      --version version[=true]           Print version information and quit\n      --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging\n\n
Apr 05 04:51:07.444 E clusterversion/version changed Failing to True: ClusterOperatorDegraded: Cluster operator etcd is reporting a failure: StaticPodsDegraded: nodes/ip-10-0-140-247.us-west-1.compute.internal pods/etcd-ip-10-0-140-247.us-west-1.compute.internal container="etcd-metrics" is not ready\nStaticPodsDegraded: nodes/ip-10-0-140-247.us-west-1.compute.internal pods/etcd-ip-10-0-140-247.us-west-1.compute.internal container="etcd-metrics" is waiting: "CrashLoopBackOff" - "back-off 5m0s restarting failed container=etcd-metrics pod=etcd-ip-10-0-140-247.us-west-1.compute.internal_openshift-etcd(0db818134821a4d9f141a9ed8d9bf03e)"
Apr 05 04:51:39.434 E ns/openshift-etcd pod/etcd-ip-10-0-140-247.us-west-1.compute.internal node/ip-10-0-140-247.us-west-1.compute.internal container/etcd-metrics container exited with code 1 (Error): {"level":"info","ts":"2020-04-05T04:51:38.661Z","caller":"etcdmain/grpc_proxy.go:185","msg":"gRPC proxy server TLS","tls-info":"cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.key, trusted-ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-metrics-proxy-serving-ca/ca-bundle.crt, client-cert-auth = false, crl-file = "}\n{"level":"info","ts":"2020-04-05T04:51:38.662Z","caller":"etcdmain/grpc_proxy.go:320","msg":"listening for gRPC proxy client requests","address":"127.0.0.1:9977"}\ncert has non empty Common Name (system:etcd-peer:etcd-client)\n
Apr 05 04:56:47.484 E ns/openshift-etcd pod/etcd-ip-10-0-140-247.us-west-1.compute.internal node/ip-10-0-140-247.us-west-1.compute.internal container/etcd-metrics container exited with code 1 (Error): {"level":"info","ts":"2020-04-05T04:56:46.653Z","caller":"etcdmain/grpc_proxy.go:185","msg":"gRPC proxy server TLS","tls-info":"cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.key, trusted-ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-metrics-proxy-serving-ca/ca-bundle.crt, client-cert-auth = false, crl-file = "}\n{"level":"info","ts":"2020-04-05T04:56:46.654Z","caller":"etcdmain/grpc_proxy.go:320","msg":"listening for gRPC proxy client requests","address":"127.0.0.1:9977"}\ncert has non empty Common Name (system:etcd-peer:etcd-client)\n
Apr 05 04:57:37.454 E clusterversion/version changed Failing to True: ClusterOperatorDegraded: Cluster operator etcd is reporting a failure: StaticPodsDegraded: nodes/ip-10-0-140-247.us-west-1.compute.internal pods/etcd-ip-10-0-140-247.us-west-1.compute.internal container="etcd-metrics" is not ready\nStaticPodsDegraded: nodes/ip-10-0-140-247.us-west-1.compute.internal pods/etcd-ip-10-0-140-247.us-west-1.compute.internal container="etcd-metrics" is waiting: "CrashLoopBackOff" - "back-off 5m0s restarting failed container=etcd-metrics pod=etcd-ip-10-0-140-247.us-west-1.compute.internal_openshift-etcd(0db818134821a4d9f141a9ed8d9bf03e)"
Apr 05 05:01:48.467 E ns/openshift-etcd pod/etcd-ip-10-0-140-247.us-west-1.compute.internal node/ip-10-0-140-247.us-west-1.compute.internal container/etcd-metrics container exited with code 1 (Error): {"level":"info","ts":"2020-04-05T05:01:47.778Z","caller":"etcdmain/grpc_proxy.go:185","msg":"gRPC proxy server TLS","tls-info":"cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.key, trusted-ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-metrics-proxy-serving-ca/ca-bundle.crt, client-cert-auth = false, crl-file = "}\n{"level":"info","ts":"2020-04-05T05:01:47.779Z","caller":"etcdmain/grpc_proxy.go:320","msg":"listening for gRPC proxy client requests","address":"127.0.0.1:9977"}\ncert has non empty Common Name (system:etcd-peer:etcd-client)\n
Apr 05 05:04:52.440 E clusterversion/version changed Failing to True: ClusterOperatorDegraded: Cluster operator etcd is reporting a failure: StaticPodsDegraded: nodes/ip-10-0-140-247.us-west-1.compute.internal pods/etcd-ip-10-0-140-247.us-west-1.compute.internal container="etcd-metrics" is not ready\nStaticPodsDegraded: nodes/ip-10-0-140-247.us-west-1.compute.internal pods/etcd-ip-10-0-140-247.us-west-1.compute.internal container="etcd-metrics" is waiting: "CrashLoopBackOff" - "back-off 5m0s restarting failed container=etcd-metrics pod=etcd-ip-10-0-140-247.us-west-1.compute.internal_openshift-etcd(0db818134821a4d9f141a9ed8d9bf03e)"
Apr 05 05:06:52.520 E ns/openshift-etcd pod/etcd-ip-10-0-140-247.us-west-1.compute.internal node/ip-10-0-140-247.us-west-1.compute.internal container/etcd-metrics container exited with code 1 (Error): {"level":"info","ts":"2020-04-05T05:06:51.653Z","caller":"etcdmain/grpc_proxy.go:185","msg":"gRPC proxy server TLS","tls-info":"cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.key, trusted-ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-metrics-proxy-serving-ca/ca-bundle.crt, client-cert-auth = false, crl-file = "}\n{"level":"info","ts":"2020-04-05T05:06:51.654Z","caller":"etcdmain/grpc_proxy.go:320","msg":"listening for gRPC proxy client requests","address":"127.0.0.1:9977"}\ncert has non empty Common Name (system:etcd-peer:etcd-client)\n
Apr 05 05:11:55.540 E ns/openshift-etcd pod/etcd-ip-10-0-140-247.us-west-1.compute.internal node/ip-10-0-140-247.us-west-1.compute.internal container/etcd-metrics container exited with code 1 (Error): {"level":"info","ts":"2020-04-05T05:11:54.661Z","caller":"etcdmain/grpc_proxy.go:185","msg":"gRPC proxy server TLS","tls-info":"cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.key, trusted-ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-metrics-proxy-serving-ca/ca-bundle.crt, client-cert-auth = false, crl-file = "}\n{"level":"info","ts":"2020-04-05T05:11:54.661Z","caller":"etcdmain/grpc_proxy.go:320","msg":"listening for gRPC proxy client requests","address":"127.0.0.1:9977"}\ncert has non empty Common Name (system:etcd-peer:etcd-client)\n
Apr 05 05:13:37.439 E clusterversion/version changed Failing to True: ClusterOperatorDegraded: Cluster operator etcd is reporting a failure: StaticPodsDegraded: nodes/ip-10-0-140-247.us-west-1.compute.internal pods/etcd-ip-10-0-140-247.us-west-1.compute.internal container="etcd-metrics" is not ready\nStaticPodsDegraded: nodes/ip-10-0-140-247.us-west-1.compute.internal pods/etcd-ip-10-0-140-247.us-west-1.compute.internal container="etcd-metrics" is waiting: "CrashLoopBackOff" - "back-off 5m0s restarting failed container=etcd-metrics pod=etcd-ip-10-0-140-247.us-west-1.compute.internal_openshift-etcd(0db818134821a4d9f141a9ed8d9bf03e)"
Apr 05 05:17:00.577 E ns/openshift-etcd pod/etcd-ip-10-0-140-247.us-west-1.compute.internal node/ip-10-0-140-247.us-west-1.compute.internal container/etcd-metrics container exited with code 1 (Error): {"level":"info","ts":"2020-04-05T05:16:59.663Z","caller":"etcdmain/grpc_proxy.go:185","msg":"gRPC proxy server TLS","tls-info":"cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.key, trusted-ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-metrics-proxy-serving-ca/ca-bundle.crt, client-cert-auth = false, crl-file = "}\n{"level":"info","ts":"2020-04-05T05:16:59.664Z","caller":"etcdmain/grpc_proxy.go:320","msg":"listening for gRPC proxy client requests","address":"127.0.0.1:9977"}\ncert has non empty Common Name (system:etcd-peer:etcd-client)\n
Apr 05 05:22:13.654 E ns/openshift-etcd pod/etcd-ip-10-0-140-247.us-west-1.compute.internal node/ip-10-0-140-247.us-west-1.compute.internal container/etcd-metrics container exited with code 1 (Error): {"level":"info","ts":"2020-04-05T05:22:12.847Z","caller":"etcdmain/grpc_proxy.go:185","msg":"gRPC proxy server TLS","tls-info":"cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.key, trusted-ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-metrics-proxy-serving-ca/ca-bundle.crt, client-cert-auth = false, crl-file = "}\n{"level":"info","ts":"2020-04-05T05:22:12.848Z","caller":"etcdmain/grpc_proxy.go:320","msg":"listening for gRPC proxy client requests","address":"127.0.0.1:9977"}\ncert has non empty Common Name (system:etcd-peer:etcd-client)\n
Apr 05 05:22:17.182 E clusterversion/version changed Failing to True: ClusterOperatorDegraded: Cluster operator etcd is reporting a failure: StaticPodsDegraded: nodes/ip-10-0-140-247.us-west-1.compute.internal pods/etcd-ip-10-0-140-247.us-west-1.compute.internal container="etcd-metrics" is not ready\nStaticPodsDegraded: nodes/ip-10-0-140-247.us-west-1.compute.internal pods/etcd-ip-10-0-140-247.us-west-1.compute.internal container="etcd-metrics" is terminated: "Error" - "{\"level\":\"info\",\"ts\":\"2020-04-05T05:22:12.847Z\",\"caller\":\"etcdmain/grpc_proxy.go:185\",\"msg\":\"gRPC proxy server TLS\",\"tls-info\":\"cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.key, trusted-ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-metrics-proxy-serving-ca/ca-bundle.crt, client-cert-auth = false, crl-file = \"}\n{\"level\":\"info\",\"ts\":\"2020-04-05T05:22:12.848Z\",\"caller\":\"etcdmain/grpc_proxy.go:320\",\"msg\":\"listening for gRPC proxy client requests\",\"address\":\"127.0.0.1:9977\"}\ncert has non empty Common Name (system:etcd-peer:etcd-client)\n"\nEtcdMembersDegraded: ip-10-0-140-247.us-west-1.compute.internal members are unhealthy,  members are unknown
Apr 05 05:27:25.688 E ns/openshift-etcd pod/etcd-ip-10-0-140-247.us-west-1.compute.internal node/ip-10-0-140-247.us-west-1.compute.internal container/etcd-metrics container exited with code 1 (Error): {"level":"info","ts":"2020-04-05T05:27:24.666Z","caller":"etcdmain/grpc_proxy.go:185","msg":"gRPC proxy server TLS","tls-info":"cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.key, trusted-ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-metrics-proxy-serving-ca/ca-bundle.crt, client-cert-auth = false, crl-file = "}\n{"level":"info","ts":"2020-04-05T05:27:24.666Z","caller":"etcdmain/grpc_proxy.go:320","msg":"listening for gRPC proxy client requests","address":"127.0.0.1:9977"}\ncert has non empty Common Name (system:etcd-peer:etcd-client)\n
Apr 05 05:31:08.692 E clusterversion/version changed Failing to True: ClusterOperatorDegraded: Cluster operator etcd is reporting a failure: StaticPodsDegraded: nodes/ip-10-0-140-247.us-west-1.compute.internal pods/etcd-ip-10-0-140-247.us-west-1.compute.internal container="etcd-metrics" is not ready\nStaticPodsDegraded: nodes/ip-10-0-140-247.us-west-1.compute.internal pods/etcd-ip-10-0-140-247.us-west-1.compute.internal container="etcd-metrics" is waiting: "CrashLoopBackOff" - "back-off 5m0s restarting failed container=etcd-metrics pod=etcd-ip-10-0-140-247.us-west-1.compute.internal_openshift-etcd(0db818134821a4d9f141a9ed8d9bf03e)"
Apr 05 05:32:33.739 E ns/openshift-etcd pod/etcd-ip-10-0-140-247.us-west-1.compute.internal node/ip-10-0-140-247.us-west-1.compute.internal container/etcd-metrics container exited with code 1 (Error): {"level":"info","ts":"2020-04-05T05:32:32.729Z","caller":"etcdmain/grpc_proxy.go:185","msg":"gRPC proxy server TLS","tls-info":"cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.key, trusted-ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-metrics-proxy-serving-ca/ca-bundle.crt, client-cert-auth = false, crl-file = "}\n{"level":"info","ts":"2020-04-05T05:32:32.730Z","caller":"etcdmain/grpc_proxy.go:320","msg":"listening for gRPC proxy client requests","address":"127.0.0.1:9977"}\ncert has non empty Common Name (system:etcd-peer:etcd-client)\n
Apr 05 05:37:45.310 E ns/openshift-etcd pod/etcd-ip-10-0-140-247.us-west-1.compute.internal node/ip-10-0-140-247.us-west-1.compute.internal container/etcd-metrics container exited with code 1 (Error): {"level":"info","ts":"2020-04-05T05:37:43.663Z","caller":"etcdmain/grpc_proxy.go:185","msg":"gRPC proxy server TLS","tls-info":"cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.key, trusted-ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-metrics-proxy-serving-ca/ca-bundle.crt, client-cert-auth = false, crl-file = "}\n{"level":"info","ts":"2020-04-05T05:37:43.664Z","caller":"etcdmain/grpc_proxy.go:320","msg":"listening for gRPC proxy client requests","address":"127.0.0.1:9977"}\ncert has non empty Common Name (system:etcd-peer:etcd-client)\n
Apr 05 05:39:54.447 E clusterversion/version changed Failing to True: ClusterOperatorDegraded: Cluster operator etcd is reporting a failure: StaticPodsDegraded: nodes/ip-10-0-140-247.us-west-1.compute.internal pods/etcd-ip-10-0-140-247.us-west-1.compute.internal container="etcd-metrics" is not ready\nStaticPodsDegraded: nodes/ip-10-0-140-247.us-west-1.compute.internal pods/etcd-ip-10-0-140-247.us-west-1.compute.internal container="etcd-metrics" is waiting: "CrashLoopBackOff" - "back-off 5m0s restarting failed container=etcd-metrics pod=etcd-ip-10-0-140-247.us-west-1.compute.internal_openshift-etcd(0db818134821a4d9f141a9ed8d9bf03e)"\nEtcdMembersDegraded: ip-10-0-139-195.us-west-1.compute.internal,ip-10-0-140-247.us-west-1.compute.internal members are unhealthy,  members are unknown
Apr 05 05:42:52.333 E ns/openshift-etcd pod/etcd-ip-10-0-140-247.us-west-1.compute.internal node/ip-10-0-140-247.us-west-1.compute.internal container/etcd-metrics container exited with code 1 (Error): {"level":"info","ts":"2020-04-05T05:42:51.657Z","caller":"etcdmain/grpc_proxy.go:185","msg":"gRPC proxy server TLS","tls-info":"cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.key, trusted-ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-metrics-proxy-serving-ca/ca-bundle.crt, client-cert-auth = false, crl-file = "}\n{"level":"info","ts":"2020-04-05T05:42:51.658Z","caller":"etcdmain/grpc_proxy.go:320","msg":"listening for gRPC proxy client requests","address":"127.0.0.1:9977"}\ncert has non empty Common Name (system:etcd-peer:etcd-client)\n
Apr 05 05:48:04.442 E ns/openshift-etcd pod/etcd-ip-10-0-140-247.us-west-1.compute.internal node/ip-10-0-140-247.us-west-1.compute.internal container/etcd-metrics container exited with code 1 (Error): {"level":"info","ts":"2020-04-05T05:48:03.713Z","caller":"etcdmain/grpc_proxy.go:185","msg":"gRPC proxy server TLS","tls-info":"cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.key, trusted-ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-metrics-proxy-serving-ca/ca-bundle.crt, client-cert-auth = false, crl-file = "}\n{"level":"info","ts":"2020-04-05T05:48:03.714Z","caller":"etcdmain/grpc_proxy.go:320","msg":"listening for gRPC proxy client requests","address":"127.0.0.1:9977"}\ncert has non empty Common Name (system:etcd-peer:etcd-client)\n
Apr 05 05:49:00.682 E clusterversion/version changed Failing to True: ClusterOperatorDegraded: Cluster operator etcd is reporting a failure: StaticPodsDegraded: nodes/ip-10-0-140-247.us-west-1.compute.internal pods/etcd-ip-10-0-140-247.us-west-1.compute.internal container="etcd-metrics" is not ready\nStaticPodsDegraded: nodes/ip-10-0-140-247.us-west-1.compute.internal pods/etcd-ip-10-0-140-247.us-west-1.compute.internal container="etcd-metrics" is waiting: "CrashLoopBackOff" - "back-off 5m0s restarting failed container=etcd-metrics pod=etcd-ip-10-0-140-247.us-west-1.compute.internal_openshift-etcd(0db818134821a4d9f141a9ed8d9bf03e)"\nEtcdMembersDegraded: ip-10-0-140-247.us-west-1.compute.internal members are unhealthy,  members are unknown
Apr 05 05:53:11.457 E ns/openshift-etcd pod/etcd-ip-10-0-140-247.us-west-1.compute.internal node/ip-10-0-140-247.us-west-1.compute.internal container/etcd-metrics container exited with code 1 (Error): {"level":"info","ts":"2020-04-05T05:53:10.648Z","caller":"etcdmain/grpc_proxy.go:185","msg":"gRPC proxy server TLS","tls-info":"cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.key, trusted-ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-metrics-proxy-serving-ca/ca-bundle.crt, client-cert-auth = false, crl-file = "}\n{"level":"info","ts":"2020-04-05T05:53:10.649Z","caller":"etcdmain/grpc_proxy.go:320","msg":"listening for gRPC proxy client requests","address":"127.0.0.1:9977"}\ncert has non empty Common Name (system:etcd-peer:etcd-client)\n

				
				Click to see stdout/stderrfrom junit_e2e_20200405-055513.xml

Filter through log files


openshift-tests [sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial] [Suite:openshift] 1h19m

go run hack/e2e.go -v -test --test_args='--ginkgo.focus=openshift\-tests\s\[sig\-arch\]\[Feature\:ClusterUpgrade\]\sCluster\sshould\sremain\sfunctional\sduring\supgrade\s\[Disruptive\]\s\[Serial\]\s\[Suite\:openshift\]$'
fail [github.com/openshift/origin/test/e2e/upgrade/upgrade.go:135]: during upgrade to registry.svc.ci.openshift.org/ci-op-cc8ypj4j/release@sha256:a56fcb830579943aaa8fe7de8bdaafb4d3688f0416baa8fb15a22cb8465e152b
Unexpected error:
    <*errors.errorString | 0xc0026d98c0>: {
        s: "Cluster did not complete upgrade: timed out waiting for the condition: Working towards 0.0.1-2020-04-05-040510: 11% complete",
    }
    Cluster did not complete upgrade: timed out waiting for the condition: Working towards 0.0.1-2020-04-05-040510: 11% complete
occurred
				
				Click to see stdout/stderrfrom junit_e2e_20200405-055513.xml

Filter through log files


operator Run template e2e-aws-upgrade - e2e-aws-upgrade container test 1h19m

go run hack/e2e.go -v -test --test_args='--ginkgo.focus=operator\sRun\stemplate\se2e\-aws\-upgrade\s\-\se2e\-aws\-upgrade\scontainer\stest$'
se
Apr 05 05:53:11.457 E ns/openshift-etcd pod/etcd-ip-10-0-140-247.us-west-1.compute.internal node/ip-10-0-140-247.us-west-1.compute.internal container/etcd-metrics container exited with code 1 (Error): {"level":"info","ts":"2020-04-05T05:53:10.648Z","caller":"etcdmain/grpc_proxy.go:185","msg":"gRPC proxy server TLS","tls-info":"cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-ip-10-0-140-247.us-west-1.compute.internal.key, trusted-ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-metrics-proxy-serving-ca/ca-bundle.crt, client-cert-auth = false, crl-file = "}\n{"level":"info","ts":"2020-04-05T05:53:10.649Z","caller":"etcdmain/grpc_proxy.go:320","msg":"listening for gRPC proxy client requests","address":"127.0.0.1:9977"}\ncert has non empty Common Name (system:etcd-peer:etcd-client)\n
Apr 05 05:53:11.457 W ns/openshift-etcd pod/etcd-ip-10-0-140-247.us-west-1.compute.internal node/ip-10-0-140-247.us-west-1.compute.internal container/etcd-metrics reason/Restarted
Apr 05 05:54:38.189 I ns/openshift-etcd-operator deployment/etcd-operator reason/UnhealthyEtcdMember unhealthy members: ip-10-0-146-109.us-west-1.compute.internal,ip-10-0-139-195.us-west-1.compute.internal,ip-10-0-140-247.us-west-1.compute.internal (374 times)
Apr 05 05:55:00.516 W ns/openshift-etcd pod/etcd-ip-10-0-140-247.us-west-1.compute.internal node/ip-10-0-140-247.us-west-1.compute.internal reason/BackOff Back-off restarting failed container (339 times)
Apr 05 05:55:13.037 I test="[sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial] [Suite:openshift]" failed

Failing tests:

[sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial] [Suite:openshift]

Writing JUnit report to /tmp/artifacts/junit/junit_e2e_20200405-055513.xml

error: 1 fail, 0 pass, 0 skip (1h19m14s)

				from junit_operator.xml

Filter through log files


Show 22 Passed Tests

Show 1 Skipped Tests