Result | FAILURE |
Tests | 6 failed / 865 succeeded |
Started | |
Elapsed | 1h48m |
Work namespace | ci-op-yxjvhwj8 |
pod | 4.3.0-0.nightly-2019-11-08-060454-azure-ovn |
go run hack/e2e.go -v -test --test_args='--ginkgo.focus=openshift\-tests\sMonitor\scluster\swhile\stests\sexecute$'
1 error level events were detected during this test run: Nov 08 07:00:29.075 E ns/default pod/recycler-for-nfs-6qww9 node/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv pod failed (DeadlineExceeded): Pod was active on the node longer than the specified deadlinefrom junit_e2e_20191108-073136.xml
Timeline: Nov 08 06:46:20.588 I ns/openshift-kube-apiserver pod/revision-pruner-5-ci-op-yxjvhwj8-bfca5-ddv8x-master-2 node/ci-op-yxjvhwj8-bfca5-ddv8x-master-2 created Nov 08 06:46:35.362 - 2219s W ns/openshift-kube-scheduler pod/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 node/ci-op-yxjvhwj8-bfca5-ddv8x-master-0 pod has been pending longer than a minute Nov 08 06:47:05.568 I ns/openshift-ingress service/router-default Ensuring load balancer (7 times) Nov 08 06:47:05.891 W ns/openshift-ingress service/router-default Error syncing load balancer: failed to ensure load balancer: ensure(openshift-ingress/router-default): backendPoolID(/subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Network/loadBalancers/ci-op-yxjvhwj8-bfca5-ddv8x/backendAddressPools/ci-op-yxjvhwj8-bfca5-ddv8x) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:NicGet" (4 times) Nov 08 06:47:25.387 W ns/openshift-kube-scheduler pod/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(f71ade3707c664f9dca5a5b6ee83b2e79dd7d36fe795db7d1b1cd6d972be2721): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition\n' (3 times) Nov 08 06:47:39.466 - 52s I test="[sig-storage] Projected secret should be consumable from pods in volume with defaultMode set [LinuxOnly] [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]" running Nov 08 06:47:53.604 W ns/openshift-kube-scheduler pod/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(6fd90203672cbf3db0abe534b42e4f6da91ee126122adb8529c68745567f981e): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition\n' (4 times) Nov 08 06:48:31.479 I test="[sig-storage] Projected secret should be consumable from pods in volume with defaultMode set [LinuxOnly] [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]" failed Nov 08 06:48:53.113 W ns/openshift-kube-scheduler pod/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(fa0cbd94c6f31a3d95598400ae7195d97ac14d26786520bb23d0dd6b399f6d9a): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition\n' (5 times) Nov 08 06:49:22.360 W ns/openshift-kube-scheduler pod/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(66c2fa28a73fe4a2332c31c2cbb1bd23498905d4360f7d14c82a549876259687): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition\n' (6 times) Nov 08 06:50:12.419 W persistentvolume/pvc-43611f50-0d22-4806-a44d-2dc6720701aa compute.DisksClient#Delete: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Disk ci-op-yxjvhwj8-bfca5-ddv8x-dynamic-pvc-43611f50-0d22-4806-a44d-2dc6720701aa is attached to VM /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Compute/virtualMachines/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus1-zwphp." Nov 08 06:50:21.316 W ns/openshift-kube-scheduler pod/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(7f61f0dba25ccfadad3c4ad5b464b3074fd4fa701620490da5958c3a7d92dcd0): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition\n' (7 times) Nov 08 06:51:19.758 W ns/openshift-kube-scheduler pod/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(53a933530ec44db46d58f7814b8c7427a498e6a31d3e7e8f07776dfd050feecb): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition\n' (8 times) Nov 08 06:51:22.514 W persistentvolume/pvc-27929b61-c77e-411a-a4d3-e35e85856cda compute.DisksClient#Delete: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Disk ci-op-yxjvhwj8-bfca5-ddv8x-dynamic-pvc-27929b61-c77e-411a-a4d3-e35e85856cda is attached to VM /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Compute/virtualMachines/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus3-6zm9f." Nov 08 06:52:05.900 I ns/openshift-ingress service/router-default Ensuring load balancer (8 times) Nov 08 06:52:06.191 W ns/openshift-ingress service/router-default Error syncing load balancer: failed to ensure load balancer: azure - cloud provider rate limited(read) for operation:PublicIPGet Nov 08 06:52:09.589 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (3 times) Nov 08 06:52:09.612 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-000" not found) (3 times) Nov 08 06:52:09.623 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-001" not found) (3 times) Nov 08 06:52:09.647 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-002" not found) (5 times) Nov 08 06:52:10.175 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-005" not found) (3 times) Nov 08 06:52:36.822 W persistentvolume/pvc-d26aa7c2-b217-411b-afb7-2b0b5d71c603 compute.DisksClient#Delete: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Disk ci-op-yxjvhwj8-bfca5-ddv8x-dynamic-pvc-d26aa7c2-b217-411b-afb7-2b0b5d71c603 is attached to VM /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Compute/virtualMachines/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv." Nov 08 06:52:47.996 I ns/openshift-must-gather-v9xqt pod/must-gather-rr9rx node/ created Nov 08 06:52:48.026 I ns/openshift-must-gather-v9xqt pod/must-gather-rr9rx Successfully assigned openshift-must-gather-v9xqt/must-gather-rr9rx to ci-op-yxjvhwj8-bfca5-ddv8x-master-1 Nov 08 06:52:48.800 W ns/openshift-kube-scheduler pod/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(f06934fef590993d2c7edd1cbc403551ae2737d7c597c6ac9899a35eb0557a2c): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition\n' (9 times) Nov 08 06:53:47.252 W ns/openshift-kube-scheduler pod/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(ce3de8da424b72344dcd29446fcaad400d82b13eb658233fb7d2b5435882f157): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition\n' (10 times) Nov 08 06:53:50.361 - 74s W ns/openshift-must-gather-v9xqt pod/must-gather-rr9rx node/ci-op-yxjvhwj8-bfca5-ddv8x-master-1 pod has been pending longer than a minute Nov 08 06:53:58.507 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-1 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (5 times) Nov 08 06:53:58.522 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-1 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-000" not found) (6 times) Nov 08 06:53:58.531 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-1 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-001" not found) (3 times) Nov 08 06:53:58.663 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-1 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-002" not found) (2 times) Nov 08 06:53:59.262 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-1 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-006" not found) (2 times) Nov 08 06:54:15.812 W ns/openshift-kube-scheduler pod/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(42567481b7818ac01465de549e9a44ce8d3390a56d6cab66055f9704e08913c1): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition\n' (11 times) Nov 08 06:54:45.346 W ns/openshift-kube-scheduler pod/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(24e274b25a8b24f35faec056948d1e1ce44cf40f6856e87169e5b9af686f3857): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition\n' (12 times) Nov 08 06:54:58.400 I ns/default pod/recycler-for-nfs-6qww9 node/ created Nov 08 06:54:58.448 I ns/default pod/recycler-for-nfs-6qww9 Successfully assigned default/recycler-for-nfs-6qww9 to ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv Nov 08 06:54:58.468 I persistentvolume/nfs-6qww9 Recycler pod: Successfully assigned default/recycler-for-nfs-6qww9 to ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv Nov 08 06:55:03.219 I ns/openshift-must-gather-v9xqt pod/must-gather-rr9rx Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ddd62469a9bd0f38fd58b05e7e51065e4c4ac291eeb4650eb69c47ece5dfc8d" Nov 08 06:55:05.608 I ns/openshift-must-gather-v9xqt pod/must-gather-rr9rx Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ddd62469a9bd0f38fd58b05e7e51065e4c4ac291eeb4650eb69c47ece5dfc8d" Nov 08 06:55:05.929 I ns/openshift-must-gather-v9xqt pod/must-gather-rr9rx Created container gather Nov 08 06:55:05.989 I ns/openshift-must-gather-v9xqt pod/must-gather-rr9rx Started container gather Nov 08 06:55:06.421 I ns/openshift-must-gather-v9xqt pod/must-gather-rr9rx Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ddd62469a9bd0f38fd58b05e7e51065e4c4ac291eeb4650eb69c47ece5dfc8d" already present on machine Nov 08 06:55:06.690 I ns/openshift-must-gather-v9xqt pod/must-gather-rr9rx Created container copy Nov 08 06:55:06.724 I ns/openshift-must-gather-v9xqt pod/must-gather-rr9rx Started container copy Nov 08 06:55:09.116 I ns/default pod/recycler-for-nfs-6qww9 Pulling image "busybox:1.27" Nov 08 06:55:09.124 I persistentvolume/nfs-6qww9 Recycler pod: Pulling image "busybox:1.27" Nov 08 06:55:13.912 I ns/default pod/recycler-for-nfs-6qww9 Successfully pulled image "busybox:1.27" Nov 08 06:55:13.925 I persistentvolume/nfs-6qww9 Recycler pod: Successfully pulled image "busybox:1.27" Nov 08 06:55:14.047 W ns/openshift-kube-scheduler pod/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(cb227e00186b84653887b474237253f4a5dbce938d35762552ebf1bb7fa063bc): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition\n' (13 times) Nov 08 06:55:15.298 I ns/default pod/recycler-for-nfs-6qww9 Created container pv-recycler Nov 08 06:55:15.310 I persistentvolume/nfs-6qww9 Recycler pod: Created container pv-recycler Nov 08 06:55:15.427 I ns/default pod/recycler-for-nfs-6qww9 Started container pv-recycler Nov 08 06:55:15.449 I persistentvolume/nfs-6qww9 Recycler pod: Started container pv-recycler Nov 08 06:55:15.568 W ns/openshift-must-gather-v9xqt pod/must-gather-rr9rx node/ci-op-yxjvhwj8-bfca5-ddv8x-master-1 graceful deletion within 0s Nov 08 06:55:15.575 W ns/openshift-must-gather-v9xqt pod/must-gather-rr9rx node/ci-op-yxjvhwj8-bfca5-ddv8x-master-1 deleted Nov 08 06:55:16.077 W ns/default pod/recycler-for-nfs-6qww9 node/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv graceful deletion within 0s Nov 08 06:55:16.084 W ns/default pod/recycler-for-nfs-6qww9 node/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv deleted Nov 08 06:55:16.094 I persistentvolume/nfs-6qww9 Volume recycled Nov 08 06:55:17.328 W persistentvolume/pvc-5c2467cc-89d5-4744-98b8-27dcfe2c7b8a compute.DisksClient#Delete: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Disk ci-op-yxjvhwj8-bfca5-ddv8x-dynamic-pvc-5c2467cc-89d5-4744-98b8-27dcfe2c7b8a is attached to VM /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Compute/virtualMachines/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv." Nov 08 06:55:29.560 I ns/default pod/recycler-for-nfs-6qww9 node/ created Nov 08 06:55:29.600 I persistentvolume/nfs-6qww9 Recycler pod: Started container pv-recycler (2 times) Nov 08 06:55:29.605 I ns/default pod/recycler-for-nfs-6qww9 Successfully assigned default/recycler-for-nfs-6qww9 to ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv Nov 08 06:55:29.629 I persistentvolume/nfs-6qww9 Recycler pod: Successfully pulled image "busybox:1.27" (2 times) Nov 08 06:55:29.653 I persistentvolume/nfs-6qww9 Recycler pod: Created container pv-recycler (2 times) Nov 08 06:55:29.670 I persistentvolume/nfs-6qww9 Recycler pod: Successfully assigned default/recycler-for-nfs-6qww9 to ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv (2 times) Nov 08 06:55:29.684 I persistentvolume/nfs-6qww9 Recycler pod: Pulling image "busybox:1.27" (2 times) Nov 08 06:55:29.689 I persistentvolume/nfs-6qww9 Recycler pod: Successfully assigned default/recycler-for-nfs-6qww9 to ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv (3 times) Nov 08 06:55:40.049 I ns/default pod/recycler-for-nfs-6qww9 Container image "busybox:1.27" already present on machine Nov 08 06:55:40.061 I persistentvolume/nfs-6qww9 Recycler pod: Container image "busybox:1.27" already present on machine Nov 08 06:55:43.202 I ns/default pod/recycler-for-nfs-6qww9 Created container pv-recycler Nov 08 06:55:43.211 I persistentvolume/nfs-6qww9 Recycler pod: Created container pv-recycler (3 times) Nov 08 06:55:43.897 I ns/default pod/recycler-for-nfs-6qww9 Started container pv-recycler Nov 08 06:55:43.906 I persistentvolume/nfs-6qww9 Recycler pod: Started container pv-recycler (3 times) Nov 08 06:55:44.174 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-2 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (5 times) Nov 08 06:55:44.174 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-2 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-000" not found) (3 times) Nov 08 06:55:44.529 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-2 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-001" not found) (6 times) Nov 08 06:55:44.954 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-2 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-002" not found) (3 times) Nov 08 06:55:45.338 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-2 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-003" not found) (3 times) Nov 08 06:56:13.431 W ns/openshift-kube-scheduler pod/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(d177342d798027d6e7cad4d8a9402f2692df2f7284f83394c4a9974593a7b5ee): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition\n' (14 times) Nov 08 06:56:42.051 W ns/openshift-kube-scheduler pod/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(83d41813f82eb2846358125955701e7c93208636c2eb4cb409f108a9c8a2be3d): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition\n' (15 times) Nov 08 06:57:06.204 I ns/openshift-ingress service/router-default Ensuring load balancer (9 times) Nov 08 06:57:06.906 W ns/openshift-ingress service/router-default Error syncing load balancer: failed to ensure load balancer: azure - cloud provider rate limited(read) for operation:NSGGet Nov 08 06:57:10.754 W ns/openshift-kube-scheduler pod/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(dea4d8e8f506cca119455fe13e2652950c20ef9d4635b57c246a0a07bd510539): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition\n' (16 times) Nov 08 06:57:39.850 W ns/openshift-kube-scheduler pod/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(1e89a251765571f54f11400638306fc0b23e5ef0a20d26ab8af7b151b10e469d): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition\n' (17 times) Nov 08 06:58:38.674 W ns/openshift-kube-scheduler pod/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(4584c4bd42c5dbf098763722153a88f9cbbdb6830e84b00881583d24f7c485a0): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition\n' (18 times) Nov 08 06:59:07.925 W ns/openshift-kube-scheduler pod/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(53f99df31cc310a14ccf3a132958450d7eeddee5f4d80a368a9d2ec65e8474f2): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition\n' (19 times) Nov 08 07:00:29.056 I ns/default pod/recycler-for-nfs-6qww9 Pod was active on the node longer than the specified deadline Nov 08 07:00:29.075 E ns/default pod/recycler-for-nfs-6qww9 node/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv pod failed (DeadlineExceeded): Pod was active on the node longer than the specified deadline Nov 08 07:00:29.075 W ns/default pod/recycler-for-nfs-6qww9 node/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv container=pv-recycler container stopped being ready Nov 08 07:00:29.082 I persistentvolume/nfs-6qww9 Recycler pod: Pod was active on the node longer than the specified deadline Nov 08 07:00:29.085 I ns/default pod/recycler-for-nfs-6qww9 Stopping container pv-recycler Nov 08 07:00:29.113 W ns/default pod/recycler-for-nfs-6qww9 node/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv graceful deletion within 0s Nov 08 07:00:29.135 W ns/default pod/recycler-for-nfs-6qww9 node/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv deleted Nov 08 07:00:29.250 I ns/default pod/recycler-for-nfs-6qww9 Stopping container pv-recycler (2 times) Nov 08 07:00:31.681 W persistentvolume/pvc-d034ee5c-8bee-412a-8399-e1be86930f22 compute.DisksClient#Delete: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Disk ci-op-yxjvhwj8-bfca5-ddv8x-dynamic-pvc-d034ee5c-8bee-412a-8399-e1be86930f22 is attached to VM /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Compute/virtualMachines/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus1-zwphp." Nov 08 07:00:36.313 W ns/openshift-kube-scheduler pod/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(7b526cd127e7e40918a0b6d22b3611c0cb0ec481193f70cdd01ce0277deb4ddb): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition\n' (20 times) Nov 08 07:01:00.866 W persistentvolume/pvc-a87aa74c-7078-4df7-ae01-e5ff3aa2e0f0 compute.DisksClient#Delete: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Disk ci-op-yxjvhwj8-bfca5-ddv8x-dynamic-pvc-a87aa74c-7078-4df7-ae01-e5ff3aa2e0f0 is attached to VM /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Compute/virtualMachines/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus1-zwphp." Nov 08 07:02:00.028 - 125s I test="[Feature:Platform] Managed cluster should have no crashlooping pods in core namespaces over two minutes [Suite:openshift/conformance/parallel]" running Nov 08 07:02:05.574 W ns/openshift-kube-scheduler pod/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(18e56e802af238941057a9088a67aa949f0b03eb2c289a2e2ffad6ed82454025): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition\n' (21 times) Nov 08 07:02:06.911 I ns/openshift-ingress service/router-default Ensuring load balancer (10 times) Nov 08 07:02:07.317 W ns/openshift-ingress service/router-default Error syncing load balancer: failed to ensure load balancer: azure - cloud provider rate limited(read) for operation:PublicIPGet (2 times) Nov 08 07:02:09.589 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (11 times) Nov 08 07:02:10.186 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-008" not found) (4 times) Nov 08 07:02:17.864 W persistentvolume/pvc-381d1e9a-4e50-42de-be73-22da0c5651d4 compute.DisksClient#Delete: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Disk ci-op-yxjvhwj8-bfca5-ddv8x-dynamic-pvc-381d1e9a-4e50-42de-be73-22da0c5651d4 is attached to VM /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Compute/virtualMachines/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv." Nov 08 07:02:17.876 W persistentvolume/pvc-163e9db5-5dd6-49b9-a235-9d2a527bb6be compute.DisksClient#Delete: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Disk ci-op-yxjvhwj8-bfca5-ddv8x-dynamic-pvc-163e9db5-5dd6-49b9-a235-9d2a527bb6be is attached to VM /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Compute/virtualMachines/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus3-6zm9f." Nov 08 07:02:26.737 W persistentvolume/pvc-b998ea59-1088-4dee-82e6-5c7d5bb54ebb compute.DisksClient#Delete: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Disk ci-op-yxjvhwj8-bfca5-ddv8x-dynamic-pvc-b998ea59-1088-4dee-82e6-5c7d5bb54ebb is attached to VM /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Compute/virtualMachines/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus1-zwphp." Nov 08 07:03:58.268 I ns/openshift-must-gather-nk72k pod/must-gather-hqrjw node/ created Nov 08 07:03:58.301 I ns/openshift-must-gather-nk72k pod/must-gather-hqrjw Successfully assigned openshift-must-gather-nk72k/must-gather-hqrjw to ci-op-yxjvhwj8-bfca5-ddv8x-master-1 Nov 08 07:03:58.465 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-1 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (3 times) Nov 08 07:03:59.060 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-1 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-009" not found) (4 times) Nov 08 07:04:05.582 I test="[Feature:Platform] Managed cluster should have no crashlooping pods in core namespaces over two minutes [Suite:openshift/conformance/parallel]" failed Nov 08 07:04:21.948 W persistentvolume/pvc-f396ba32-c2d1-4a11-9b2e-16c8141069b2 compute.DisksClient#Delete: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Disk ci-op-yxjvhwj8-bfca5-ddv8x-dynamic-pvc-f396ba32-c2d1-4a11-9b2e-16c8141069b2 is attached to VM /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Compute/virtualMachines/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv." Nov 08 07:04:34.740 I ns/openshift-must-gather-nk72k pod/must-gather-hqrjw Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ddd62469a9bd0f38fd58b05e7e51065e4c4ac291eeb4650eb69c47ece5dfc8d" already present on machine Nov 08 07:04:35.009 I ns/openshift-must-gather-nk72k pod/must-gather-hqrjw Created container gather Nov 08 07:04:35.050 I ns/openshift-must-gather-nk72k pod/must-gather-hqrjw Started container gather Nov 08 07:05:05.360 - 374s W ns/openshift-must-gather-nk72k pod/must-gather-hqrjw node/ci-op-yxjvhwj8-bfca5-ddv8x-master-1 pod has been pending longer than a minute Nov 08 07:05:44.165 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-2 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (2 times) Nov 08 07:05:44.332 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-2 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-000" not found) (5 times) Nov 08 07:06:01.023 W ns/openshift-kube-scheduler pod/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(6895112a079e93c6b1071178789757c299f3bf90aba1a67a376c37b1e4a2fea1): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition\n' (26 times) Nov 08 07:06:25.457 W persistentvolume/pvc-b6433d5d-1e4f-49f8-8b63-56c505872b5b compute.DisksClient#Delete: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Disk ci-op-yxjvhwj8-bfca5-ddv8x-dynamic-pvc-b6433d5d-1e4f-49f8-8b63-56c505872b5b is attached to VM /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Compute/virtualMachines/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv." Nov 08 07:06:34.762 W persistentvolume/pvc-858da82c-cede-42c2-8256-81ddb650560c compute.DisksClient#Delete: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Disk ci-op-yxjvhwj8-bfca5-ddv8x-dynamic-pvc-858da82c-cede-42c2-8256-81ddb650560c is attached to VM /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Compute/virtualMachines/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv." Nov 08 07:07:04.566 W persistentvolume/pvc-6c7feeed-3f19-4f57-9aa1-695ab297b119 compute.DisksClient#Delete: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Disk ci-op-yxjvhwj8-bfca5-ddv8x-dynamic-pvc-6c7feeed-3f19-4f57-9aa1-695ab297b119 is attached to VM /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Compute/virtualMachines/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus3-6zm9f." Nov 08 07:07:04.646 W persistentvolume/pvc-0c299e40-da4d-4761-9462-d5f9939a9c8b compute.DisksClient#Delete: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Disk ci-op-yxjvhwj8-bfca5-ddv8x-dynamic-pvc-0c299e40-da4d-4761-9462-d5f9939a9c8b is attached to VM /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Compute/virtualMachines/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv." Nov 08 07:07:07.317 I ns/openshift-ingress service/router-default Ensuring load balancer (11 times) Nov 08 07:07:07.699 W ns/openshift-ingress service/router-default Error syncing load balancer: failed to ensure load balancer: Code="PreconditionFailed" Message="Precondition failed." Details=[{"code":"PreconditionFailedEtagMismatch","message":"Etag provided in if-match header W/\"1eedb274-1445-44a4-abef-996348424af7\" does not match etag W/\"d0794840-5902-415d-ae31-87efcf36c4b7\" of resource /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Network/loadBalancers/ci-op-yxjvhwj8-bfca5-ddv8x in NRP data store."}] Nov 08 07:07:31.031 W persistentvolume/pvc-acc7aaa6-ac53-4093-988f-078bd4a6b105 compute.DisksClient#Delete: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Disk ci-op-yxjvhwj8-bfca5-ddv8x-dynamic-pvc-acc7aaa6-ac53-4093-988f-078bd4a6b105 is attached to VM /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Compute/virtualMachines/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus3-6zm9f." Nov 08 07:07:50.914 - 900s I test="[Feature:Prometheus][Conformance] Prometheus when installed on the cluster shouldn't report any alerts in firing state apart from Watchdog [Suite:openshift/conformance/parallel/minimal]" running Nov 08 07:08:16.125 W persistentvolume/pvc-92e2d899-2d04-43e7-a298-538d096f6dc2 compute.DisksClient#Delete: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Disk ci-op-yxjvhwj8-bfca5-ddv8x-dynamic-pvc-92e2d899-2d04-43e7-a298-538d096f6dc2 is attached to VM /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Compute/virtualMachines/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv." Nov 08 07:08:49.057 W persistentvolume/pvc-f092bde3-ce83-4872-9f92-34005a1d0591 compute.DisksClient#Delete: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Disk ci-op-yxjvhwj8-bfca5-ddv8x-dynamic-pvc-f092bde3-ce83-4872-9f92-34005a1d0591 is attached to VM /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Compute/virtualMachines/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv." Nov 08 07:09:58.767 W persistentvolume/pvc-1a94a772-93d3-4948-bd76-cf5e1f6a8970 compute.DisksClient#Delete: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Disk ci-op-yxjvhwj8-bfca5-ddv8x-dynamic-pvc-1a94a772-93d3-4948-bd76-cf5e1f6a8970 is attached to VM /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Compute/virtualMachines/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv." Nov 08 07:10:20.612 W persistentvolume/pvc-8e751919-298a-47ed-b28b-09791675a6b8 compute.DisksClient#Delete: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Disk ci-op-yxjvhwj8-bfca5-ddv8x-dynamic-pvc-8e751919-298a-47ed-b28b-09791675a6b8 is attached to VM /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Compute/virtualMachines/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv." Nov 08 07:11:26.929 W ns/openshift-kube-scheduler pod/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(0c0e1ed461bcee32e76959561d12e1caeaffd0e4dfafb6db8e5b2d3b25fa9840): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition\n' (32 times) Nov 08 07:11:27.311 I ns/openshift-must-gather-nk72k pod/must-gather-hqrjw Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ddd62469a9bd0f38fd58b05e7e51065e4c4ac291eeb4650eb69c47ece5dfc8d" already present on machine Nov 08 07:11:27.695 I ns/openshift-must-gather-nk72k pod/must-gather-hqrjw Created container copy Nov 08 07:11:27.717 I ns/openshift-must-gather-nk72k pod/must-gather-hqrjw Started container copy Nov 08 07:11:29.019 W persistentvolume/pvc-520bcd85-6aea-4dd6-a2fe-f656385d5bdc compute.DisksClient#Delete: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Disk ci-op-yxjvhwj8-bfca5-ddv8x-dynamic-pvc-520bcd85-6aea-4dd6-a2fe-f656385d5bdc is attached to VM /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Compute/virtualMachines/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv." Nov 08 07:11:36.396 W ns/openshift-must-gather-nk72k pod/must-gather-hqrjw node/ci-op-yxjvhwj8-bfca5-ddv8x-master-1 graceful deletion within 0s Nov 08 07:11:36.403 W ns/openshift-must-gather-nk72k pod/must-gather-hqrjw node/ci-op-yxjvhwj8-bfca5-ddv8x-master-1 deleted Nov 08 07:12:04.513 - 69s I test="[Feature:ProjectAPI] TestProjectWatch should succeed [Suite:openshift/conformance/parallel]" running Nov 08 07:12:08.259 I ns/openshift-ingress service/router-default Ensuring load balancer (12 times) Nov 08 07:12:08.764 W ns/openshift-ingress service/router-default Error syncing load balancer: failed to ensure load balancer: ensure(openshift-ingress/router-default): backendPoolID(/subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Network/loadBalancers/ci-op-yxjvhwj8-bfca5-ddv8x/backendAddressPools/ci-op-yxjvhwj8-bfca5-ddv8x) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:NicGet" (5 times) Nov 08 07:12:09.921 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (17 times) Nov 08 07:12:10.231 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-000" not found) (5 times) Nov 08 07:13:13.691 I test="[Feature:ProjectAPI] TestProjectWatch should succeed [Suite:openshift/conformance/parallel]" failed Nov 08 07:13:58.472 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-1 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (5 times) Nov 08 07:13:59.057 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-1 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-009" not found) (23 times) Nov 08 07:14:06.247 W persistentvolume/pvc-f6aeb41e-22d0-4979-8769-817091173199 compute.DisksClient#Delete: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Disk ci-op-yxjvhwj8-bfca5-ddv8x-dynamic-pvc-f6aeb41e-22d0-4979-8769-817091173199 is attached to VM /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Compute/virtualMachines/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus1-zwphp." Nov 08 07:14:23.680 W persistentvolume/pvc-ba5272c5-da41-4fce-8017-1ee83e491335 compute.DisksClient#Delete: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Disk ci-op-yxjvhwj8-bfca5-ddv8x-dynamic-pvc-ba5272c5-da41-4fce-8017-1ee83e491335 is attached to VM /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Compute/virtualMachines/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus1-zwphp." Nov 08 07:15:44.136 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-2 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (21 times) Nov 08 07:15:44.730 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-2 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-008" not found) (4 times) Nov 08 07:16:22.931 W ns/openshift-kube-scheduler pod/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(832345ffff3db5c4787cca1ea6ed870af753485cb2e45e2b6183e17ff8ab827b): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition\n' (38 times) Nov 08 07:17:08.199 I ns/openshift-ingress service/router-default Ensuring load balancer (13 times) Nov 08 07:17:08.613 W ns/openshift-ingress service/router-default Error syncing load balancer: failed to ensure load balancer: Code="PreconditionFailed" Message="Precondition failed." Details=[{"code":"PreconditionFailedEtagMismatch","message":"Etag provided in if-match header W/\"4eb16af6-bc59-4d8f-a2f5-f1b3999ba159\" does not match etag W/\"df9ce28e-b4a0-4be6-9255-415eae29bf09\" of resource /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Network/loadBalancers/ci-op-yxjvhwj8-bfca5-ddv8x in NRP data store."}] Nov 08 07:17:11.745 W persistentvolume/pvc-59788b10-0e85-46f3-95e0-1681e17c226c compute.DisksClient#Delete: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Disk ci-op-yxjvhwj8-bfca5-ddv8x-dynamic-pvc-59788b10-0e85-46f3-95e0-1681e17c226c is attached to VM /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Compute/virtualMachines/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus1-zwphp." Nov 08 07:17:27.703 W persistentvolume/pvc-b4befe6d-b782-4ccb-95d4-585322e4b983 compute.DisksClient#Delete: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Disk ci-op-yxjvhwj8-bfca5-ddv8x-dynamic-pvc-b4befe6d-b782-4ccb-95d4-585322e4b983 is attached to VM /subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Compute/virtualMachines/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv." Nov 08 07:21:16.016 W ns/openshift-kube-scheduler pod/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(b291866d06c2ea4ad8adff96eeeaaef8cdc6453448ba3d966860e627f430f2ad): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition\n' (45 times) Nov 08 07:22:08.619 I ns/openshift-ingress service/router-default Ensuring load balancer (14 times) Nov 08 07:22:09.080 W ns/openshift-ingress service/router-default Error syncing load balancer: failed to ensure load balancer: ensure(openshift-ingress/router-default): backendPoolID(/subscriptions/d38f1e38-4bed-438e-b227-833f997adf6a/resourceGroups/ci-op-yxjvhwj8-bfca5-ddv8x-rg/providers/Microsoft.Network/loadBalancers/ci-op-yxjvhwj8-bfca5-ddv8x/backendAddressPools/ci-op-yxjvhwj8-bfca5-ddv8x) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:NicGet" (6 times) Nov 08 07:22:09.608 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 (combined from similar events): Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert" not found) (26 times) Nov 08 07:22:10.201 W ns/openshift-kube-apiserver pod/kube-apiserver-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 Removed file for secret: /%!(EXTRA *errors.StatusError=secrets "user-serving-cert-008" not found) (7 times) Nov 08 07:22:51.249 I test="[Feature:Prometheus][Conformance] Prometheus when installed on the cluster shouldn't report any alerts in firing state apart from Watchdog [Suite:openshift/conformance/parallel/minimal]" failed
Find failed mentions in log files
go run hack/e2e.go -v -test --test_args='--ginkgo.focus=openshift\-tests\s\[Feature\:Platform\]\sManaged\scluster\sshould\shave\sno\scrashlooping\spods\sin\score\snamespaces\sover\stwo\sminutes\s\[Suite\:openshift\/conformance\/parallel\]$'
fail [github.com/openshift/origin/test/extended/operators/cluster.go:122]: Expected <[]string | len:1, cap:1>: [ "Pod openshift-kube-scheduler/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 was pending entire time: unknown error", ] to be emptyfrom junit_e2e_20191108-073136.xml
Nov 8 07:02:02.519: INFO: >>> kubeConfig: /tmp/admin.kubeconfig Nov 8 07:02:02.521: INFO: >>> kubeConfig: /tmp/admin.kubeconfig Nov 8 07:02:02.782: INFO: >>> kubeConfig: /tmp/admin.kubeconfig Nov 8 07:02:04.534: INFO: >>> kubeConfig: /tmp/admin.kubeconfig Nov 8 07:02:04.536: INFO: Waiting up to 30m0s for all (but 100) nodes to be schedulable Nov 8 07:02:04.654: INFO: Waiting up to 10m0s for all pods (need at least 0) in namespace 'kube-system' to be running and ready Nov 8 07:02:04.816: INFO: 0 / 0 pods in namespace 'kube-system' are running and ready (0 seconds elapsed) Nov 8 07:02:04.816: INFO: expected 0 pod replicas in namespace 'kube-system', 0 are Running and Ready. Nov 8 07:02:04.816: INFO: Waiting up to 5m0s for all daemonsets in namespace 'kube-system' to start Nov 8 07:02:04.862: INFO: e2e test version: v1.16.2 Nov 8 07:02:04.901: INFO: kube-apiserver version: v1.16.2 Nov 8 07:02:04.901: INFO: >>> kubeConfig: /tmp/admin.kubeconfig Nov 8 07:02:04.943: INFO: Cluster IP family: ipv4 [BeforeEach] [Top Level] /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/util/test.go:60 [It] have no crashlooping pods in core namespaces over two minutes [Suite:openshift/conformance/parallel] /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/operators/cluster.go:21 Nov 8 07:02:04.949: INFO: >>> kubeConfig: /tmp/admin.kubeconfig STEP: Collecting events from namespace "openshift-kube-scheduler". STEP: Found 124 events. Nov 8 07:04:05.541: INFO: At 2019-11-08 06:30:55 +0000 UTC - event for installer-2-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Pulling: Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6491d5a928a102083a9f9ec66b3241cd2f22f88683b93850fef36f58717a43d" Nov 8 07:04:05.541: INFO: At 2019-11-08 06:31:00 +0000 UTC - event for installer-2-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Pulled: Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6491d5a928a102083a9f9ec66b3241cd2f22f88683b93850fef36f58717a43d" Nov 8 07:04:05.541: INFO: At 2019-11-08 06:31:01 +0000 UTC - event for installer-2-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Created: Created container installer Nov 8 07:04:05.541: INFO: At 2019-11-08 06:31:01 +0000 UTC - event for installer-2-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Started: Started container installer Nov 8 07:04:05.541: INFO: At 2019-11-08 06:31:01 +0000 UTC - event for installer-2-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {static-pod-installer } StaticPodInstallerCompleted: Successfully installed revision 2 Nov 8 07:04:05.541: INFO: At 2019-11-08 06:31:02 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Pulling: Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe4807d61614d0376992d9be7e1a61a44bb80edf9e8057f7d2357db67f0d665b" Nov 8 07:04:05.541: INFO: At 2019-11-08 06:31:11 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Pulled: Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe4807d61614d0376992d9be7e1a61a44bb80edf9e8057f7d2357db67f0d665b" Nov 8 07:04:05.541: INFO: At 2019-11-08 06:31:12 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Created: Created container wait-for-host-port Nov 8 07:04:05.541: INFO: At 2019-11-08 06:31:12 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Started: Started container wait-for-host-port Nov 8 07:04:05.541: INFO: At 2019-11-08 06:31:13 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Created: Created container scheduler Nov 8 07:04:05.541: INFO: At 2019-11-08 06:31:13 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Started: Started container scheduler Nov 8 07:04:05.541: INFO: At 2019-11-08 06:31:13 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Pulled: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe4807d61614d0376992d9be7e1a61a44bb80edf9e8057f7d2357db67f0d665b" already present on machine Nov 8 07:04:05.541: INFO: At 2019-11-08 06:31:13 +0000 UTC - event for revision-pruner-2-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Started: Started container pruner Nov 8 07:04:05.541: INFO: At 2019-11-08 06:31:13 +0000 UTC - event for revision-pruner-2-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Created: Created container pruner Nov 8 07:04:05.541: INFO: At 2019-11-08 06:31:13 +0000 UTC - event for revision-pruner-2-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Pulled: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6491d5a928a102083a9f9ec66b3241cd2f22f88683b93850fef36f58717a43d" already present on machine Nov 8 07:04:05.541: INFO: At 2019-11-08 06:31:17 +0000 UTC - event for installer-3-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Pulled: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6491d5a928a102083a9f9ec66b3241cd2f22f88683b93850fef36f58717a43d" already present on machine Nov 8 07:04:05.542: INFO: At 2019-11-08 06:31:17 +0000 UTC - event for installer-3-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {static-pod-installer } StaticPodInstallerCompleted: Successfully installed revision 3 Nov 8 07:04:05.542: INFO: At 2019-11-08 06:31:17 +0000 UTC - event for installer-3-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Created: Created container installer Nov 8 07:04:05.542: INFO: At 2019-11-08 06:31:17 +0000 UTC - event for installer-3-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Started: Started container installer Nov 8 07:04:05.542: INFO: At 2019-11-08 06:31:22 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Pulling: Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe4807d61614d0376992d9be7e1a61a44bb80edf9e8057f7d2357db67f0d665b" Nov 8 07:04:05.542: INFO: At 2019-11-08 06:31:30 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Started: Started container wait-for-host-port Nov 8 07:04:05.542: INFO: At 2019-11-08 06:31:30 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Created: Created container wait-for-host-port Nov 8 07:04:05.542: INFO: At 2019-11-08 06:31:30 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Pulled: Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe4807d61614d0376992d9be7e1a61a44bb80edf9e8057f7d2357db67f0d665b" Nov 8 07:04:05.542: INFO: At 2019-11-08 06:31:32 +0000 UTC - event for installer-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-1} Pulling: Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6491d5a928a102083a9f9ec66b3241cd2f22f88683b93850fef36f58717a43d" Nov 8 07:04:05.542: INFO: At 2019-11-08 06:31:32 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Pulled: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe4807d61614d0376992d9be7e1a61a44bb80edf9e8057f7d2357db67f0d665b" already present on machine Nov 8 07:04:05.542: INFO: At 2019-11-08 06:31:33 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Started: Started container scheduler Nov 8 07:04:05.542: INFO: At 2019-11-08 06:31:33 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Created: Created container scheduler Nov 8 07:04:05.542: INFO: At 2019-11-08 06:31:35 +0000 UTC - event for installer-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-1} Pulled: Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6491d5a928a102083a9f9ec66b3241cd2f22f88683b93850fef36f58717a43d" Nov 8 07:04:05.542: INFO: At 2019-11-08 06:31:36 +0000 UTC - event for installer-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-1} Created: Created container installer Nov 8 07:04:05.542: INFO: At 2019-11-08 06:31:36 +0000 UTC - event for installer-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-1} Started: Started container installer Nov 8 07:04:05.542: INFO: At 2019-11-08 06:31:36 +0000 UTC - event for installer-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {static-pod-installer } StaticPodInstallerCompleted: Successfully installed revision 4 Nov 8 07:04:05.542: INFO: At 2019-11-08 06:31:36 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-1} Pulling: Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe4807d61614d0376992d9be7e1a61a44bb80edf9e8057f7d2357db67f0d665b" Nov 8 07:04:05.542: INFO: At 2019-11-08 06:31:36 +0000 UTC - event for revision-pruner-3-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Pulled: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6491d5a928a102083a9f9ec66b3241cd2f22f88683b93850fef36f58717a43d" already present on machine Nov 8 07:04:05.542: INFO: At 2019-11-08 06:31:37 +0000 UTC - event for revision-pruner-3-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Created: Created container pruner Nov 8 07:04:05.542: INFO: At 2019-11-08 06:31:37 +0000 UTC - event for revision-pruner-3-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Started: Started container pruner Nov 8 07:04:05.542: INFO: At 2019-11-08 06:31:43 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-1} Created: Created container wait-for-host-port Nov 8 07:04:05.542: INFO: At 2019-11-08 06:31:43 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-1} Pulled: Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe4807d61614d0376992d9be7e1a61a44bb80edf9e8057f7d2357db67f0d665b" Nov 8 07:04:05.542: INFO: At 2019-11-08 06:31:43 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-1} Started: Started container wait-for-host-port Nov 8 07:04:05.542: INFO: At 2019-11-08 06:31:45 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-1} Pulled: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe4807d61614d0376992d9be7e1a61a44bb80edf9e8057f7d2357db67f0d665b" already present on machine Nov 8 07:04:05.542: INFO: At 2019-11-08 06:31:45 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-1} Started: Started container scheduler Nov 8 07:04:05.542: INFO: At 2019-11-08 06:31:45 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-1} Created: Created container scheduler Nov 8 07:04:05.542: INFO: At 2019-11-08 06:33:10 +0000 UTC - event for revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-1} Pulled: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6491d5a928a102083a9f9ec66b3241cd2f22f88683b93850fef36f58717a43d" already present on machine Nov 8 07:04:05.542: INFO: At 2019-11-08 06:33:10 +0000 UTC - event for revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-1} Started: Started container pruner Nov 8 07:04:05.542: INFO: At 2019-11-08 06:33:10 +0000 UTC - event for revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-1} Created: Created container pruner Nov 8 07:04:05.542: INFO: At 2019-11-08 06:33:28 +0000 UTC - event for installer-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Created: Created container installer Nov 8 07:04:05.542: INFO: At 2019-11-08 06:33:28 +0000 UTC - event for installer-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Pulled: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6491d5a928a102083a9f9ec66b3241cd2f22f88683b93850fef36f58717a43d" already present on machine Nov 8 07:04:05.542: INFO: At 2019-11-08 06:33:29 +0000 UTC - event for installer-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Started: Started container installer Nov 8 07:04:05.542: INFO: At 2019-11-08 06:33:29 +0000 UTC - event for installer-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {static-pod-installer } StaticPodInstallerCompleted: Successfully installed revision 4 Nov 8 07:04:05.542: INFO: At 2019-11-08 06:33:29 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Killing: Stopping container scheduler Nov 8 07:04:05.542: INFO: At 2019-11-08 06:33:29 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Pulled: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe4807d61614d0376992d9be7e1a61a44bb80edf9e8057f7d2357db67f0d665b" already present on machine Nov 8 07:04:05.542: INFO: At 2019-11-08 06:33:30 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Created: Created container wait-for-host-port Nov 8 07:04:05.542: INFO: At 2019-11-08 06:33:30 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Started: Started container wait-for-host-port Nov 8 07:04:05.542: INFO: At 2019-11-08 06:33:31 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Pulled: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe4807d61614d0376992d9be7e1a61a44bb80edf9e8057f7d2357db67f0d665b" already present on machine Nov 8 07:04:05.542: INFO: At 2019-11-08 06:33:31 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Created: Created container scheduler Nov 8 07:04:05.542: INFO: At 2019-11-08 06:33:31 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Started: Started container scheduler Nov 8 07:04:05.542: INFO: At 2019-11-08 06:33:32 +0000 UTC - event for kube-scheduler: {default-scheduler } LeaderElection: ci-op-yxjvhwj8-bfca5-ddv8x-master-0_2fc76086-d6a1-4205-a40f-b9b3fee3d170 became leader Nov 8 07:04:05.542: INFO: At 2019-11-08 06:34:59 +0000 UTC - event for installer-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Started: Started container installer Nov 8 07:04:05.542: INFO: At 2019-11-08 06:34:59 +0000 UTC - event for installer-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Created: Created container installer Nov 8 07:04:05.542: INFO: At 2019-11-08 06:34:59 +0000 UTC - event for installer-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Pulled: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6491d5a928a102083a9f9ec66b3241cd2f22f88683b93850fef36f58717a43d" already present on machine Nov 8 07:04:05.542: INFO: At 2019-11-08 06:34:59 +0000 UTC - event for installer-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {static-pod-installer } StaticPodInstallerCompleted: Successfully installed revision 4 Nov 8 07:04:05.542: INFO: At 2019-11-08 06:34:59 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Killing: Stopping container scheduler Nov 8 07:04:05.542: INFO: At 2019-11-08 06:35:01 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Started: Started container wait-for-host-port Nov 8 07:04:05.542: INFO: At 2019-11-08 06:35:01 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Pulled: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe4807d61614d0376992d9be7e1a61a44bb80edf9e8057f7d2357db67f0d665b" already present on machine Nov 8 07:04:05.542: INFO: At 2019-11-08 06:35:01 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Created: Created container wait-for-host-port Nov 8 07:04:05.542: INFO: At 2019-11-08 06:35:02 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Pulled: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe4807d61614d0376992d9be7e1a61a44bb80edf9e8057f7d2357db67f0d665b" already present on machine Nov 8 07:04:05.542: INFO: At 2019-11-08 06:35:03 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Started: Started container scheduler Nov 8 07:04:05.542: INFO: At 2019-11-08 06:35:03 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Created: Created container scheduler Nov 8 07:04:05.542: INFO: At 2019-11-08 06:35:44 +0000 UTC - event for revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Pulled: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6491d5a928a102083a9f9ec66b3241cd2f22f88683b93850fef36f58717a43d" already present on machine Nov 8 07:04:05.542: INFO: At 2019-11-08 06:35:45 +0000 UTC - event for revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Created: Created container pruner Nov 8 07:04:05.542: INFO: At 2019-11-08 06:35:45 +0000 UTC - event for revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Started: Started container pruner Nov 8 07:04:05.542: INFO: At 2019-11-08 06:35:58 +0000 UTC - event for revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} FailedCreatePodSandBox: Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(21127668c218ce51550499ef686bbe621754a45391b81c30ac7e21b3c630853f): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition ' Nov 8 07:04:05.542: INFO: At 2019-11-08 06:36:15 +0000 UTC - event for installer-5-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {static-pod-installer } StaticPodInstallerCompleted: Successfully installed revision 5 Nov 8 07:04:05.542: INFO: At 2019-11-08 06:36:15 +0000 UTC - event for installer-5-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Started: Started container installer Nov 8 07:04:05.542: INFO: At 2019-11-08 06:36:15 +0000 UTC - event for installer-5-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Pulled: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6491d5a928a102083a9f9ec66b3241cd2f22f88683b93850fef36f58717a43d" already present on machine Nov 8 07:04:05.542: INFO: At 2019-11-08 06:36:15 +0000 UTC - event for installer-5-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Created: Created container installer Nov 8 07:04:05.542: INFO: At 2019-11-08 06:36:15 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Killing: Stopping container scheduler Nov 8 07:04:05.542: INFO: At 2019-11-08 06:36:16 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Started: Started container wait-for-host-port Nov 8 07:04:05.542: INFO: At 2019-11-08 06:36:16 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Created: Created container wait-for-host-port Nov 8 07:04:05.542: INFO: At 2019-11-08 06:36:16 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Pulled: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe4807d61614d0376992d9be7e1a61a44bb80edf9e8057f7d2357db67f0d665b" already present on machine Nov 8 07:04:05.542: INFO: At 2019-11-08 06:36:34 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Created: Created container scheduler Nov 8 07:04:05.542: INFO: At 2019-11-08 06:36:34 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Started: Started container scheduler Nov 8 07:04:05.542: INFO: At 2019-11-08 06:36:34 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Pulled: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe4807d61614d0376992d9be7e1a61a44bb80edf9e8057f7d2357db67f0d665b" already present on machine Nov 8 07:04:05.542: INFO: At 2019-11-08 06:37:37 +0000 UTC - event for kube-scheduler: {default-scheduler } LeaderElection: ci-op-yxjvhwj8-bfca5-ddv8x-master-0_7fa766f3-f20b-4903-9265-1e5f4ca16b1f became leader Nov 8 07:04:05.542: INFO: At 2019-11-08 06:37:57 +0000 UTC - event for revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} FailedCreatePodSandBox: Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(ee294db28417d53a927043bfe32b7a8864ae64e249079f8369cf66128644781c): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition ' Nov 8 07:04:05.542: INFO: At 2019-11-08 06:38:12 +0000 UTC - event for installer-5-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Pulled: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6491d5a928a102083a9f9ec66b3241cd2f22f88683b93850fef36f58717a43d" already present on machine Nov 8 07:04:05.542: INFO: At 2019-11-08 06:38:12 +0000 UTC - event for installer-5-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Created: Created container installer Nov 8 07:04:05.542: INFO: At 2019-11-08 06:38:12 +0000 UTC - event for installer-5-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {static-pod-installer } StaticPodInstallerCompleted: Successfully installed revision 5 Nov 8 07:04:05.542: INFO: At 2019-11-08 06:38:12 +0000 UTC - event for installer-5-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Started: Started container installer Nov 8 07:04:05.542: INFO: At 2019-11-08 06:38:12 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Killing: Stopping container scheduler Nov 8 07:04:05.542: INFO: At 2019-11-08 06:38:12 +0000 UTC - event for revision-pruner-5-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Pulled: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6491d5a928a102083a9f9ec66b3241cd2f22f88683b93850fef36f58717a43d" already present on machine Nov 8 07:04:05.542: INFO: At 2019-11-08 06:38:12 +0000 UTC - event for revision-pruner-5-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Created: Created container pruner Nov 8 07:04:05.542: INFO: At 2019-11-08 06:38:12 +0000 UTC - event for revision-pruner-5-ci-op-yxjvhwj8-bfca5-ddv8x-master-2: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-2} Started: Started container pruner Nov 8 07:04:05.542: INFO: At 2019-11-08 06:38:13 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Created: Created container wait-for-host-port Nov 8 07:04:05.542: INFO: At 2019-11-08 06:38:13 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Pulled: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe4807d61614d0376992d9be7e1a61a44bb80edf9e8057f7d2357db67f0d665b" already present on machine Nov 8 07:04:05.542: INFO: At 2019-11-08 06:38:13 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Started: Started container wait-for-host-port Nov 8 07:04:05.542: INFO: At 2019-11-08 06:38:15 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Created: Created container scheduler Nov 8 07:04:05.542: INFO: At 2019-11-08 06:38:15 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Pulled: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe4807d61614d0376992d9be7e1a61a44bb80edf9e8057f7d2357db67f0d665b" already present on machine Nov 8 07:04:05.542: INFO: At 2019-11-08 06:38:15 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Started: Started container scheduler Nov 8 07:04:05.542: INFO: At 2019-11-08 06:38:29 +0000 UTC - event for kube-scheduler: {default-scheduler } LeaderElection: ci-op-yxjvhwj8-bfca5-ddv8x-master-1_e2f3385c-2c05-48a0-a02b-180ea6204977 became leader Nov 8 07:04:05.542: INFO: At 2019-11-08 06:39:02 +0000 UTC - event for revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} FailedCreatePodSandBox: Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(14ebf20f717e0bf40d6f573e42e4dc056ee7613f233ba90b4b7f0c13c4f00e00): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition ' Nov 8 07:04:05.542: INFO: At 2019-11-08 06:39:18 +0000 UTC - event for revision-pruner-5-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Pulled: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6491d5a928a102083a9f9ec66b3241cd2f22f88683b93850fef36f58717a43d" already present on machine Nov 8 07:04:05.542: INFO: At 2019-11-08 06:39:18 +0000 UTC - event for revision-pruner-5-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Started: Started container pruner Nov 8 07:04:05.542: INFO: At 2019-11-08 06:39:18 +0000 UTC - event for revision-pruner-5-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} Created: Created container pruner Nov 8 07:04:05.542: INFO: At 2019-11-08 06:40:01 +0000 UTC - event for revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} FailedCreatePodSandBox: Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(872d4290ce7ce8654a6f81d69c7f4820c7fc1d0d8d7c5220f6a089f93fca1de0): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition ' Nov 8 07:04:05.542: INFO: At 2019-11-08 06:40:24 +0000 UTC - event for installer-5-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-1} Pulled: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6491d5a928a102083a9f9ec66b3241cd2f22f88683b93850fef36f58717a43d" already present on machine Nov 8 07:04:05.542: INFO: At 2019-11-08 06:40:24 +0000 UTC - event for installer-5-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-1} Created: Created container installer Nov 8 07:04:05.542: INFO: At 2019-11-08 06:40:24 +0000 UTC - event for installer-5-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {static-pod-installer } StaticPodInstallerCompleted: Successfully installed revision 5 Nov 8 07:04:05.542: INFO: At 2019-11-08 06:40:24 +0000 UTC - event for installer-5-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-1} Started: Started container installer Nov 8 07:04:05.542: INFO: At 2019-11-08 06:40:24 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-1} Killing: Stopping container scheduler Nov 8 07:04:05.542: INFO: At 2019-11-08 06:40:25 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-1} Pulled: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe4807d61614d0376992d9be7e1a61a44bb80edf9e8057f7d2357db67f0d665b" already present on machine Nov 8 07:04:05.542: INFO: At 2019-11-08 06:40:26 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-1} Started: Started container wait-for-host-port Nov 8 07:04:05.542: INFO: At 2019-11-08 06:40:26 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-1} Created: Created container wait-for-host-port Nov 8 07:04:05.542: INFO: At 2019-11-08 06:40:43 +0000 UTC - event for kube-scheduler: {default-scheduler } LeaderElection: ci-op-yxjvhwj8-bfca5-ddv8x-master-2_e5fea4e3-a082-40b7-99c0-d9daab7cacf9 became leader Nov 8 07:04:05.542: INFO: At 2019-11-08 06:40:57 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-1} Pulled: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe4807d61614d0376992d9be7e1a61a44bb80edf9e8057f7d2357db67f0d665b" already present on machine Nov 8 07:04:05.542: INFO: At 2019-11-08 06:40:58 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-1} Started: Started container scheduler Nov 8 07:04:05.542: INFO: At 2019-11-08 06:40:58 +0000 UTC - event for openshift-kube-scheduler-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-1} Created: Created container scheduler Nov 8 07:04:05.542: INFO: At 2019-11-08 06:41:55 +0000 UTC - event for revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} FailedCreatePodSandBox: Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(1bb3d071a60ff214125af63c70188a092576959db418d1b0a67b80a3da021b46): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition ' Nov 8 07:04:05.542: INFO: At 2019-11-08 06:42:27 +0000 UTC - event for revision-pruner-5-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-1} Pulled: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6491d5a928a102083a9f9ec66b3241cd2f22f88683b93850fef36f58717a43d" already present on machine Nov 8 07:04:05.542: INFO: At 2019-11-08 06:42:27 +0000 UTC - event for revision-pruner-5-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-1} Created: Created container pruner Nov 8 07:04:05.542: INFO: At 2019-11-08 06:42:27 +0000 UTC - event for revision-pruner-5-ci-op-yxjvhwj8-bfca5-ddv8x-master-1: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-1} Started: Started container pruner Nov 8 07:04:05.542: INFO: At 2019-11-08 06:43:02 +0000 UTC - event for revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} FailedCreatePodSandBox: Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(bbf37b24e1179773e54d0009a797121e744a5f81fb1d8cea00340336ee85ecf0): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition ' Nov 8 07:04:05.542: INFO: At 2019-11-08 06:43:31 +0000 UTC - event for revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} FailedCreatePodSandBox: Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(2256e1ee25b04859c4ab20097593f39fe69ee9a37d1abf3d1c26ccbc3d9bad1b): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition ' Nov 8 07:04:05.542: INFO: At 2019-11-08 06:44:29 +0000 UTC - event for revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} FailedCreatePodSandBox: Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(bf30ab223efc31477a6d6ef6f82bdb965c02cabc5d28ea3175f11bbf9ae3483e): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition ' Nov 8 07:04:05.542: INFO: At 2019-11-08 06:44:58 +0000 UTC - event for revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-master-0} FailedCreatePodSandBox: (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0_openshift-kube-scheduler_a86a5983-d0a7-4168-be43-f80fc4db686b_0(18e56e802af238941057a9088a67aa949f0b03eb2c289a2e2ffad6ed82454025): Multus: error adding pod to network "ovn-kubernetes": delegateAdd: error invoking DelegateAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: 'failed to get pod annotation: timed out waiting for the condition ' Nov 8 07:04:05.543: INFO: Pod status openshift-kube-scheduler/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0: { "phase": "Pending", "conditions": [ { "type": "Initialized", "status": "True", "lastProbeTime": null, "lastTransitionTime": "2019-11-08T06:34:50Z" }, { "type": "Ready", "status": "False", "lastProbeTime": null, "lastTransitionTime": "2019-11-08T06:34:50Z", "reason": "ContainersNotReady", "message": "containers with unready status: [pruner]" }, { "type": "ContainersReady", "status": "False", "lastProbeTime": null, "lastTransitionTime": "2019-11-08T06:34:50Z", "reason": "ContainersNotReady", "message": "containers with unready status: [pruner]" }, { "type": "PodScheduled", "status": "True", "lastProbeTime": null, "lastTransitionTime": "2019-11-08T06:34:50Z" } ], "hostIP": "10.0.0.5", "startTime": "2019-11-08T06:34:50Z", "containerStatuses": [ { "name": "pruner", "state": { "waiting": { "reason": "ContainerCreating" } }, "lastState": {}, "ready": false, "restartCount": 0, "image": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6491d5a928a102083a9f9ec66b3241cd2f22f88683b93850fef36f58717a43d", "imageID": "", "started": false } ], "qosClass": "BestEffort" } Nov 8 07:04:05.556: INFO: Running AfterSuite actions on all nodes Nov 8 07:04:05.556: INFO: Running AfterSuite actions on node 1 fail [github.com/openshift/origin/test/extended/operators/cluster.go:122]: Expected <[]string | len:1, cap:1>: [ "Pod openshift-kube-scheduler/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 was pending entire time: unknown error", ] to be empty
Filter through log files
go run hack/e2e.go -v -test --test_args='--ginkgo.focus=openshift\-tests\s\[Feature\:ProjectAPI\]\s\sTestProjectWatch\sshould\ssucceed\s\[Suite\:openshift\/conformance\/parallel\]$'
fail [github.com/openshift/origin/test/extended/project/project.go:233]: timeout: e2e-test-project-api-4hbszfrom junit_e2e_20191108-073136.xml
Nov 8 07:12:06.419: INFO: >>> kubeConfig: /tmp/admin.kubeconfig Nov 8 07:12:06.422: INFO: >>> kubeConfig: /tmp/admin.kubeconfig Nov 8 07:12:06.923: INFO: >>> kubeConfig: /tmp/admin.kubeconfig Nov 8 07:12:08.457: INFO: >>> kubeConfig: /tmp/admin.kubeconfig Nov 8 07:12:08.459: INFO: Waiting up to 30m0s for all (but 100) nodes to be schedulable Nov 8 07:12:08.575: INFO: Waiting up to 10m0s for all pods (need at least 0) in namespace 'kube-system' to be running and ready Nov 8 07:12:09.055: INFO: 0 / 0 pods in namespace 'kube-system' are running and ready (0 seconds elapsed) Nov 8 07:12:09.055: INFO: expected 0 pod replicas in namespace 'kube-system', 0 are Running and Ready. Nov 8 07:12:09.055: INFO: Waiting up to 5m0s for all daemonsets in namespace 'kube-system' to start Nov 8 07:12:09.306: INFO: e2e test version: v1.16.2 Nov 8 07:12:09.344: INFO: kube-apiserver version: v1.16.2 Nov 8 07:12:09.344: INFO: >>> kubeConfig: /tmp/admin.kubeconfig Nov 8 07:12:09.692: INFO: Cluster IP family: ipv4 [BeforeEach] [Top Level] /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/util/test.go:60 [BeforeEach] [Feature:ProjectAPI] /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 STEP: Creating a kubernetes client Nov 8 07:12:09.697: INFO: >>> kubeConfig: /tmp/admin.kubeconfig [BeforeEach] [Feature:ProjectAPI] /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/util/client.go:109 Nov 8 07:12:11.416: INFO: configPath is now "/tmp/configfile172126516" Nov 8 07:12:11.416: INFO: The user is now "e2e-test-project-api-ptntf-user" Nov 8 07:12:11.416: INFO: Creating project "e2e-test-project-api-ptntf" Nov 8 07:12:12.075: INFO: Waiting on permissions in project "e2e-test-project-api-ptntf" ... Nov 8 07:12:12.118: INFO: Waiting for ServiceAccount "default" to be provisioned... Nov 8 07:12:14.444: INFO: Waiting for ServiceAccount "deployer" to be provisioned... Nov 8 07:12:14.692: INFO: Waiting for ServiceAccount "builder" to be provisioned... Nov 8 07:12:14.946: INFO: Waiting for RoleBinding "system:image-pullers" to be provisioned... Nov 8 07:12:15.235: INFO: Waiting for RoleBinding "system:image-builders" to be provisioned... Nov 8 07:12:18.143: INFO: Waiting for RoleBinding "system:deployers" to be provisioned... Nov 8 07:12:20.856: INFO: Project "e2e-test-project-api-ptntf" has been fully provisioned. [It] should succeed [Suite:openshift/conformance/parallel] /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/project/project.go:96 Nov 8 07:12:21.762: INFO: Creating project "e2e-test-project-api-m24xn" Nov 8 07:12:23.300: INFO: Waiting on permissions in project "e2e-test-project-api-m24xn" ... STEP: waitForAdd e2e-test-project-api-m24xn Nov 8 07:12:25.009: INFO: got watch.Event{Type:"ADDED", Object:(*v1.Project)(0xc002a04b00)} &v1.Project{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"e2e-test-project-api-m24xn", GenerateName:"", Namespace:"", SelfLink:"/apis/project.openshift.io/v1/projects/e2e-test-project-api-m24xn", UID:"60e2d850-931f-49b1-9d33-1c28a906dd06", ResourceVersion:"99397", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63708793941, loc:(*time.Location)(0xa359f80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"openshift.io/description":"", "openshift.io/display-name":"", "openshift.io/requester":"e2e-test-project-api-ptntf-user", "openshift.io/sa.scc.mcs":"s0:c101,c25", "openshift.io/sa.scc.supplemental-groups":"1010150000/10000", "openshift.io/sa.scc.uid-range":"1010150000/10000"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.ProjectSpec{Finalizers:[]v1.FinalizerName{"kubernetes"}}, Status:v1.ProjectStatus{Phase:"Active", Conditions:[]v1.NamespaceCondition(nil)}} Nov 8 07:12:25.169: INFO: Creating project "e2e-test-project-api-qjfsr" Nov 8 07:12:26.354: INFO: Waiting on permissions in project "e2e-test-project-api-qjfsr" ... STEP: waitForAdd e2e-test-project-api-qjfsr Nov 8 07:12:32.947: INFO: got watch.Event{Type:"ADDED", Object:(*v1.Project)(0xc002e5d080)} &v1.Project{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"e2e-test-project-api-qjfsr", GenerateName:"", Namespace:"", SelfLink:"/apis/project.openshift.io/v1/projects/e2e-test-project-api-qjfsr", UID:"67fc4f75-34f6-4a83-9929-35195874b9c9", ResourceVersion:"99638", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63708793945, loc:(*time.Location)(0xa359f80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"openshift.io/description":"", "openshift.io/display-name":"", "openshift.io/requester":"e2e-test-project-api-ptntf-user", "openshift.io/sa.scc.mcs":"s0:c101,c50", "openshift.io/sa.scc.supplemental-groups":"1010200000/10000", "openshift.io/sa.scc.uid-range":"1010200000/10000"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.ProjectSpec{Finalizers:[]v1.FinalizerName{"kubernetes"}}, Status:v1.ProjectStatus{Phase:"Active", Conditions:[]v1.NamespaceCondition(nil)}} STEP: waitForDelete e2e-test-project-api-qjfsr Nov 8 07:12:33.668: INFO: got watch.Event{Type:"DELETED", Object:(*v1.Project)(0xc002e5d760)} &v1.Project{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"e2e-test-project-api-qjfsr", GenerateName:"", Namespace:"", SelfLink:"/apis/project.openshift.io/v1/projects/e2e-test-project-api-qjfsr", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.ProjectSpec{Finalizers:[]v1.FinalizerName(nil)}, Status:v1.ProjectStatus{Phase:"", Conditions:[]v1.NamespaceCondition(nil)}} Nov 8 07:12:33.668: INFO: Creating project "e2e-test-project-api-4hbsz" Nov 8 07:12:35.476: INFO: Waiting on permissions in project "e2e-test-project-api-4hbsz" ... STEP: waitForAdd e2e-test-project-api-4hbsz Nov 8 07:12:38.358: INFO: got watch.Event{Type:"ADDED", Object:(*v1.Project)(0xc00345cf20)} &v1.Project{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"e2e-test-project-api-4hbsz", GenerateName:"", Namespace:"", SelfLink:"/apis/project.openshift.io/v1/projects/e2e-test-project-api-4hbsz", UID:"09212166-66b9-461b-8cec-dc5c6c1c2292", ResourceVersion:"100012", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63708793954, loc:(*time.Location)(0xa359f80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"openshift.io/description":"", "openshift.io/display-name":"", "openshift.io/requester":"e2e-test-project-api-ptntf-user"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.ProjectSpec{Finalizers:[]v1.FinalizerName{"kubernetes"}}, Status:v1.ProjectStatus{Phase:"Active", Conditions:[]v1.NamespaceCondition(nil)}} STEP: waitForDelete e2e-test-project-api-4hbsz Nov 8 07:12:39.005: INFO: got watch.Event{Type:"MODIFIED", Object:(*v1.Project)(0xc003652580)} &v1.Project{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"e2e-test-project-api-4hbsz", GenerateName:"", Namespace:"", SelfLink:"/apis/project.openshift.io/v1/projects/e2e-test-project-api-4hbsz", UID:"09212166-66b9-461b-8cec-dc5c6c1c2292", ResourceVersion:"100071", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63708793954, loc:(*time.Location)(0xa359f80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"openshift.io/description":"", "openshift.io/display-name":"", "openshift.io/requester":"e2e-test-project-api-ptntf-user", "openshift.io/sa.scc.mcs":"s0:c101,c80", "openshift.io/sa.scc.supplemental-groups":"1010260000/10000", "openshift.io/sa.scc.uid-range":"1010260000/10000"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.ProjectSpec{Finalizers:[]v1.FinalizerName{"kubernetes"}}, Status:v1.ProjectStatus{Phase:"Active", Conditions:[]v1.NamespaceCondition(nil)}} Nov 8 07:12:39.479: INFO: got watch.Event{Type:"MODIFIED", Object:(*v1.Project)(0xc003dca2c0)} &v1.Project{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"e2e-test-project-api-4hbsz", GenerateName:"", Namespace:"", SelfLink:"/apis/project.openshift.io/v1/projects/e2e-test-project-api-4hbsz", UID:"09212166-66b9-461b-8cec-dc5c6c1c2292", ResourceVersion:"100188", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63708793954, loc:(*time.Location)(0xa359f80)}}, DeletionTimestamp:(*v1.Time)(0xc003c25c40), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"openshift.io/description":"", "openshift.io/display-name":"", "openshift.io/requester":"e2e-test-project-api-ptntf-user", "openshift.io/sa.scc.mcs":"s0:c101,c80", "openshift.io/sa.scc.supplemental-groups":"1010260000/10000", "openshift.io/sa.scc.uid-range":"1010260000/10000"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.ProjectSpec{Finalizers:[]v1.FinalizerName{"kubernetes"}}, Status:v1.ProjectStatus{Phase:"Terminating", Conditions:[]v1.NamespaceCondition(nil)}} [AfterEach] [Feature:ProjectAPI] /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/util/client.go:101 STEP: Collecting events from namespace "e2e-test-project-api-ptntf". STEP: Found 0 events. Nov 8 07:13:09.700: INFO: POD NODE PHASE GRACE CONDITIONS Nov 8 07:13:09.700: INFO: Nov 8 07:13:09.847: INFO: skipping dumping cluster info - cluster too large Nov 8 07:13:09.972: INFO: Deleted {user.openshift.io/v1, Resource=users e2e-test-project-api-ptntf-user}, err: <nil> Nov 8 07:13:10.148: INFO: Deleted {oauth.openshift.io/v1, Resource=oauthclients e2e-client-e2e-test-project-api-ptntf}, err: <nil> Nov 8 07:13:10.498: INFO: Deleted {oauth.openshift.io/v1, Resource=oauthaccesstokens bdZqU_DsTT-C_fwku4MPHAAAAAAAAAAA}, err: <nil> Nov 8 07:13:10.865: INFO: Deleted {user.openshift.io/v1, Resource=users bob-e2e-test-project-api-ptntf}, err: <nil> Nov 8 07:13:11.132: INFO: Deleted {oauth.openshift.io/v1, Resource=oauthclients e2e-client-e2e-test-project-api-ptntf}, err: oauthclients.oauth.openshift.io "e2e-client-e2e-test-project-api-ptntf" not found Nov 8 07:13:11.425: INFO: Deleted {oauth.openshift.io/v1, Resource=oauthaccesstokens 0AT-tS91THicr8F9KKy_3QAAAAAAAAAA}, err: <nil> Nov 8 07:13:11.636: INFO: Deleted {oauth.openshift.io/v1, Resource=oauthclients e2e-client-e2e-test-project-api-ptntf}, err: oauthclients.oauth.openshift.io "e2e-client-e2e-test-project-api-ptntf" not found Nov 8 07:13:12.010: INFO: Deleted {oauth.openshift.io/v1, Resource=oauthaccesstokens Ccga583FR_e0DJwWN3zQzwAAAAAAAAAA}, err: <nil> Nov 8 07:13:12.155: INFO: Deleted {user.openshift.io/v1, Resource=users joe-e2e-test-project-api-ptntf}, err: <nil> Nov 8 07:13:12.336: INFO: Deleted {oauth.openshift.io/v1, Resource=oauthclients e2e-client-e2e-test-project-api-ptntf}, err: oauthclients.oauth.openshift.io "e2e-client-e2e-test-project-api-ptntf" not found Nov 8 07:13:12.442: INFO: Deleted {oauth.openshift.io/v1, Resource=oauthaccesstokens 4foTBUobSWWE_jfzt4-6ygAAAAAAAAAA}, err: <nil> Nov 8 07:13:12.492: INFO: Deleted {oauth.openshift.io/v1, Resource=oauthclients e2e-client-e2e-test-project-api-ptntf}, err: oauthclients.oauth.openshift.io "e2e-client-e2e-test-project-api-ptntf" not found Nov 8 07:13:12.587: INFO: Deleted {oauth.openshift.io/v1, Resource=oauthaccesstokens fzARmPPaQsKVRHOU41a5uAAAAAAAAAAA}, err: <nil> Nov 8 07:13:12.682: INFO: Deleted {oauth.openshift.io/v1, Resource=oauthclients e2e-client-e2e-test-project-api-ptntf}, err: oauthclients.oauth.openshift.io "e2e-client-e2e-test-project-api-ptntf" not found Nov 8 07:13:13.016: INFO: Deleted {oauth.openshift.io/v1, Resource=oauthaccesstokens Ti_OXzL1SveQBwnVLFjnFgAAAAAAAAAA}, err: <nil> [AfterEach] [Feature:ProjectAPI] /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:152 Nov 8 07:13:13.016: INFO: Waiting up to 3m0s for all (but 100) nodes to be ready STEP: Destroying namespace "e2e-test-project-api-ptntf" for this suite. STEP: Destroying namespace "e2e-test-project-api-m24xn" for this suite. STEP: Destroying namespace "e2e-test-project-api-qjfsr" for this suite. STEP: Destroying namespace "e2e-test-project-api-4hbsz" for this suite. Nov 8 07:13:13.666: INFO: Running AfterSuite actions on all nodes Nov 8 07:13:13.666: INFO: Running AfterSuite actions on node 1 fail [github.com/openshift/origin/test/extended/project/project.go:233]: timeout: e2e-test-project-api-4hbsz
Filter through log files
go run hack/e2e.go -v -test --test_args='--ginkgo.focus=openshift\-tests\s\[Feature\:Prometheus\]\[Conformance\]\sPrometheus\swhen\sinstalled\son\sthe\scluster\sshouldn\'t\sreport\sany\salerts\sin\sfiring\sstate\sapart\sfrom\sWatchdog\s\[Suite\:openshift\/conformance\/parallel\/minimal\]$'
Nov 8 07:21:26.494: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:21:28.260: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:21:28.260: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197688.23,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197688.23]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:21:29.261: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:21:33.363: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:21:33.363: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197693.322,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197693.322]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:21:34.364: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:21:37.636: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:21:37.636: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197697.593,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197697.593]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:21:38.636: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:21:48.651: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:21:48.651: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197708.609,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197708.609]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:21:49.651: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:21:58.903: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:21:58.904: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197718.862,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197718.862]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:21:59.904: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:01.675: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:01.675: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197721.633,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197721.633]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:02.675: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:04.140: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:04.140: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197724.097,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197724.097]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:05.141: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:08.349: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:08.349: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197728.309,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197728.309]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:09.349: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:13.520: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:13.520: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197733.481,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197733.481]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:14.521: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:16.064: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:16.064: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197736.021,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197736.021]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:17.065: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:18.189: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:18.189: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197738.154,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197738.154]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:19.190: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:20.282: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:20.282: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197740.25,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197740.25]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:21.282: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:22.613: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:22.614: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197742.582,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197742.582]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:23.614: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:24.761: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:24.761: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197744.726,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197744.726]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:25.761: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:27.221: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:27.221: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197747.18,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197747.18]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:28.223: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:29.398: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:29.398: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197749.364,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197749.364]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:30.398: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:31.456: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:31.456: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197751.425,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197751.425]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:32.457: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:33.642: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:33.642: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197753.611,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197753.611]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:34.642: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:35.870: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:35.870: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197755.839,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197755.839]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:36.871: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:38.129: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:38.129: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197758.097,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197758.097]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:39.129: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:44.710: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:44.710: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197764.574,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197764.574]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:45.711: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:47.240: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:47.240: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197767.176,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197767.176]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:48.240: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:49.488: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:49.488: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197769.457,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197769.457]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:50.489: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' --------------------------------------------------------- Received interrupt. Running AfterSuite... ^C again to terminate immediately Nov 8 07:22:50.914: INFO: Running AfterSuite actions on all nodes Nov 8 07:22:50.915: INFO: Waiting up to 3m0s for all (but 100) nodes to be ready STEP: Destroying namespace "e2e-test-prometheus-fgfz6" for this suite. Nov 8 07:22:51.229: INFO: Running AfterSuite actions on node 1from junit_e2e_20191108-073136.xml
Nov 8 07:07:52.957: INFO: >>> kubeConfig: /tmp/admin.kubeconfig Nov 8 07:07:52.961: INFO: >>> kubeConfig: /tmp/admin.kubeconfig Nov 8 07:07:53.209: INFO: >>> kubeConfig: /tmp/admin.kubeconfig Nov 8 07:07:54.274: INFO: >>> kubeConfig: /tmp/admin.kubeconfig Nov 8 07:07:54.275: INFO: Waiting up to 30m0s for all (but 100) nodes to be schedulable Nov 8 07:07:54.392: INFO: Waiting up to 10m0s for all pods (need at least 0) in namespace 'kube-system' to be running and ready Nov 8 07:07:54.545: INFO: 0 / 0 pods in namespace 'kube-system' are running and ready (0 seconds elapsed) Nov 8 07:07:54.545: INFO: expected 0 pod replicas in namespace 'kube-system', 0 are Running and Ready. Nov 8 07:07:54.546: INFO: Waiting up to 5m0s for all daemonsets in namespace 'kube-system' to start Nov 8 07:07:54.602: INFO: e2e test version: v1.16.2 Nov 8 07:07:54.640: INFO: kube-apiserver version: v1.16.2 Nov 8 07:07:54.640: INFO: >>> kubeConfig: /tmp/admin.kubeconfig Nov 8 07:07:54.682: INFO: Cluster IP family: ipv4 [BeforeEach] [Top Level] /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/util/test.go:60 [BeforeEach] [Feature:Prometheus][Conformance] Prometheus /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 STEP: Creating a kubernetes client Nov 8 07:07:54.684: INFO: >>> kubeConfig: /tmp/admin.kubeconfig [BeforeEach] [Feature:Prometheus][Conformance] Prometheus /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/prometheus/prometheus.go:46 [It] shouldn't report any alerts in firing state apart from Watchdog [Suite:openshift/conformance/parallel/minimal] /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/prometheus/prometheus.go:212 Nov 8 07:07:57.789: INFO: configPath is now "/tmp/configfile848494902" Nov 8 07:07:57.789: INFO: The user is now "e2e-test-prometheus-fgfz6-user" Nov 8 07:07:57.789: INFO: Creating project "e2e-test-prometheus-fgfz6" Nov 8 07:07:57.909: INFO: Waiting on permissions in project "e2e-test-prometheus-fgfz6" ... Nov 8 07:07:57.953: INFO: Waiting for ServiceAccount "default" to be provisioned... Nov 8 07:08:01.304: INFO: Waiting for ServiceAccount "deployer" to be provisioned... Nov 8 07:08:02.047: INFO: Waiting for ServiceAccount "builder" to be provisioned... Nov 8 07:08:02.190: INFO: Waiting for RoleBinding "system:image-pullers" to be provisioned... Nov 8 07:08:02.274: INFO: Waiting for RoleBinding "system:image-builders" to be provisioned... Nov 8 07:08:02.356: INFO: Waiting for RoleBinding "system:deployers" to be provisioned... Nov 8 07:08:02.435: INFO: Project "e2e-test-prometheus-fgfz6" has been fully provisioned. Nov 8 07:08:02.437: INFO: Creating new exec pod STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:08:36.620: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:08:47.978: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:08:47.978: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573196927.935,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573196927.935]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:08:48.980: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:08:53.380: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:08:53.380: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573196933.311,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573196933.311]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:08:54.382: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:08:58.045: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:08:58.045: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573196937.982,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573196937.982]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:08:59.045: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:09:01.209: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:09:01.209: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573196941.072,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573196941.072]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:09:02.209: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:09:04.044: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:09:04.044: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573196944.002,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573196944.002]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:09:05.045: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:09:16.344: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:09:16.344: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573196956.228,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573196956.228]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:09:17.345: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:09:20.717: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:09:20.717: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573196960.662,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573196960.662]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:09:21.718: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:09:23.718: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:09:23.718: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573196963.382,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573196963.382]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:09:24.721: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:09:26.525: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:09:26.525: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573196966.495,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573196966.495]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:09:27.528: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:09:30.435: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:09:30.435: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573196970.387,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573196970.387]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:09:31.436: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:09:35.879: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:09:35.880: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573196975.806,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573196975.806]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:09:36.884: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:09:47.043: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:09:47.043: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573196987.003,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573196987.003]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:09:48.044: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:09:57.397: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:09:57.397: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573196997.286,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573196997.286]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:09:58.397: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:10:00.223: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:10:00.223: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197000.155,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197000.155]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:10:01.223: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:10:04.361: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:10:04.361: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197004.281,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197004.281]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:10:05.361: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:10:18.248: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:10:18.248: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197018.15,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197018.15]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:10:19.248: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:10:28.213: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:10:28.213: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197028.179,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197028.179]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:10:29.213: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:10:30.668: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:10:30.668: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197030.62,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197030.62]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:10:31.668: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:10:33.638: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:10:33.638: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197033.589,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197033.589]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:10:34.638: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:10:44.179: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:10:44.179: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197044.088,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197044.088]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:10:45.179: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:10:47.143: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:10:47.143: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197047.078,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197047.078]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:10:48.143: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:10:49.953: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:10:49.953: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197049.901,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197049.901]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:10:50.954: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:10:53.998: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:10:53.998: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197053.945,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197053.945]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:10:54.999: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:10:59.044: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:10:59.044: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197058.996,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197058.996]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:11:00.045: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:11:03.483: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:11:03.483: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197063.427,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197063.427]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:11:04.483: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:11:15.130: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:11:15.130: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197075.047,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197075.047]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:11:16.130: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:11:19.397: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:11:19.397: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197079.327,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197079.327]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:11:20.397: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:11:23.585: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:11:23.585: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197083.5,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197083.5]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:11:24.586: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:11:28.740: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:11:28.740: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197088.685,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197088.685]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:11:29.740: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:11:31.387: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:11:31.387: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197091.35,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197091.35]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:11:32.388: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:11:33.663: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:11:33.663: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197093.627,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197093.627]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:11:34.663: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:11:36.387: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:11:36.387: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197096.344,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197096.344]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:11:37.388: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:11:38.312: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:11:38.313: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197098.283,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197098.283]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:11:39.313: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:11:44.775: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:11:44.775: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197104.61,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197104.61]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:11:45.776: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:11:48.544: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:11:48.544: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197108.463,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197108.463]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:11:49.544: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:11:50.745: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:11:50.745: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197110.693,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197110.693]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:11:51.745: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:11:53.273: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:11:53.273: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197113.209,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197113.209]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:11:54.273: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:11:58.695: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:11:58.695: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197118.658,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197118.658]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:11:59.696: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:12:03.587: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:12:03.587: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197123.546,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197123.546]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:12:04.598: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:12:06.967: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:12:06.967: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197126.926,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197126.926]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:12:07.967: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:12:12.043: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:12:12.043: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197132.007,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197132.007]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:12:13.043: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:12:16.618: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:12:16.618: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197136.57,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197136.57]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:12:17.619: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:12:19.976: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:12:19.976: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197139.897,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197139.897]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:12:20.976: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:12:23.860: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:12:23.860: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197143.822,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197143.822]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:12:24.861: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:12:27.317: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:12:27.317: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197147.273,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197147.273]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:12:28.318: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:12:32.063: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:12:32.063: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197152.03,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197152.03]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:12:33.063: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:12:34.985: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:12:34.986: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197154.946,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197154.946]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:12:35.986: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:12:38.970: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:12:38.970: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197158.923,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197158.923]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:12:39.970: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:12:42.824: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:12:42.824: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197162.774,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197162.774]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:12:43.824: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:12:47.343: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:12:47.343: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197167.298,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197167.298]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:12:48.344: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:12:52.816: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:12:52.816: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197172.779,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197172.779]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:12:53.817: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:12:56.222: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:12:56.222: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197176.192,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197176.192]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:12:57.222: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:13:00.420: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:13:00.420: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197180.349,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197180.349]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:13:01.420: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:13:02.792: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:13:02.792: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197182.762,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197182.762]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:13:03.792: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:13:05.049: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:13:05.049: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197185.017,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197185.017]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:13:06.049: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:13:14.336: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:13:14.336: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197194.283,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197194.283]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:13:15.339: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:13:18.726: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:13:18.726: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197198.689,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197198.689]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:13:19.726: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:13:23.526: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:13:23.526: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197203.429,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197203.429]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:13:24.527: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:13:26.848: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:13:26.848: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197206.819,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197206.819]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:13:27.849: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:13:30.579: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:13:30.579: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197210.442,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197210.442]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:13:31.579: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:13:35.012: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:13:35.012: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197214.863,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197214.863]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:13:36.013: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:13:38.576: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:13:38.576: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197218.451,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197218.451]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:13:39.576: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:13:48.152: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:13:48.152: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197227.794,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197227.794]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:13:49.153: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:13:53.573: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:13:53.573: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197233.509,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197233.509]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:13:54.573: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:13:57.309: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:13:57.309: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197237.031,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197237.031]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:13:58.309: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:14:00.072: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:14:00.072: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197240.025,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197240.025]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:14:01.073: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:14:03.668: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:14:03.668: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197243.599,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197243.599]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:14:04.668: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:14:22.192: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:14:22.192: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197262.062,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197262.062]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:14:23.192: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:14:26.303: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:14:26.304: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197266.185,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197266.185]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:14:27.304: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:14:30.603: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:14:30.603: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197270.541,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197270.541]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:14:31.604: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:14:33.546: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:14:33.546: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197273.496,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197273.496]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:14:34.547: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:14:37.780: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:14:37.780: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197277.712,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197277.712]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:14:38.781: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:14:41.523: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:14:41.523: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197281.401,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197281.401]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:14:42.524: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:14:46.485: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:14:46.485: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197286.428,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197286.428]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:14:47.488: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:14:50.075: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:14:50.075: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197290.033,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197290.033]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:14:51.075: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:14:52.962: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:14:52.962: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197292.921,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197292.921]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:14:53.965: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:14:55.419: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:14:55.419: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197295.363,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197295.363]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:14:56.419: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:14:58.019: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:14:58.019: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197297.964,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197297.964]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:14:59.026: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:15:01.158: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:15:01.158: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197301.083,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197301.083]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:15:02.159: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:15:15.055: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:15:15.055: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197314.993,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197314.993]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:15:16.056: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:15:17.789: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:15:17.789: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197317.691,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197317.691]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:15:18.793: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:15:21.158: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:15:21.158: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197320.906,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197320.906]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:15:22.158: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:15:27.357: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:15:27.357: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197327.3,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197327.3]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:15:28.357: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:15:31.872: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:15:31.872: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197331.823,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197331.823]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:15:32.874: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:15:41.355: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:15:41.355: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197341.083,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197341.083]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:15:42.361: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:15:47.143: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:15:47.143: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197347.076,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197347.076]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:15:48.151: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:15:50.742: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:15:50.742: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197350.67,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197350.67]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:15:51.742: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:15:54.457: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:15:54.457: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197354.405,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197354.405]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:15:55.459: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:15:58.503: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:15:58.503: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197358.43,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197358.43]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:15:59.507: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:16:02.579: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:16:02.579: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197362.51,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197362.51]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:16:03.580: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:16:15.611: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:16:15.611: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197375.56,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197375.56]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:16:16.612: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:16:20.724: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:16:20.724: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197380.667,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197380.667]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:16:21.724: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:16:23.112: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:16:23.112: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197383.078,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197383.078]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:16:24.112: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:16:25.334: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:16:25.334: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197385.287,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197385.287]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:16:26.334: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:16:36.096: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:16:36.096: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197396.06,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197396.06]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:16:37.097: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:16:38.149: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:16:38.149: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197398.117,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197398.117]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:16:39.150: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:16:47.732: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:16:47.732: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197407.59,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197407.59]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:16:48.732: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:16:50.172: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:16:50.172: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197410.092,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197410.092]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:16:51.173: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:16:53.235: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:16:53.235: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197413.167,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197413.167]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:16:54.236: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:16:55.144: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:16:55.144: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197415.113,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197415.113]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:16:56.144: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:16:58.423: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:16:58.423: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197418.387,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197418.387]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:16:59.423: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:17:02.138: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:17:02.139: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197421.995,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197421.995]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:17:03.139: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:17:06.273: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:17:06.273: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197426.198,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197426.198]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:17:07.273: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:17:08.405: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:17:08.405: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197428.374,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197428.374]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:17:09.406: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:17:20.785: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:17:20.785: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197440.677,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197440.677]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:17:21.786: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:17:23.498: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:17:23.498: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197443.411,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197443.411]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:17:24.498: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:17:26.391: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:17:26.391: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197446.351,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197446.351]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:17:27.394: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:17:29.326: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:17:29.326: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197449.281,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197449.281]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:17:30.327: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:17:34.112: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:17:34.112: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197454.048,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197454.048]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:17:35.112: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:17:36.877: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:17:36.877: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197456.793,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197456.793]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:17:37.877: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:17:40.951: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:17:40.951: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197460.666,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197460.666]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:17:41.952: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:17:44.476: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:17:44.476: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197464.414,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197464.414]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:17:45.477: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:17:47.296: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:17:47.296: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197467.258,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197467.258]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:17:48.297: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:17:49.837: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:17:49.837: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197469.799,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197469.799]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:17:50.838: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:18:00.343: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:18:00.343: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197480.222,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197480.222]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:18:01.344: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:18:06.592: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:18:06.593: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197486.413,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197486.413]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:18:07.593: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:18:13.008: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:18:13.008: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197492.911,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197492.911]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:18:14.009: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:18:18.456: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:18:18.456: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197498.411,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197498.411]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:18:19.456: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:18:20.798: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:18:20.798: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197500.734,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197500.734]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:18:21.798: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:18:23.306: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:18:23.307: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197503.046,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197503.046]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:18:24.307: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:18:26.903: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:18:26.903: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197506.616,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197506.616]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:18:27.903: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:18:30.499: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:18:30.499: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197510.445,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197510.445]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:18:31.499: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:18:33.969: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:18:33.970: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197513.929,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197513.929]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:18:34.970: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:18:43.424: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:18:43.424: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197519.974,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197519.974]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:18:44.425: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:18:47.815: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:18:47.815: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197527.728,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197527.728]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:18:48.815: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:18:55.177: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:18:55.177: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197535.117,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197535.117]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:18:56.178: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:18:57.451: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:18:57.451: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197537.389,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197537.389]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:18:58.452: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:18:59.926: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:18:59.926: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197539.893,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197539.893]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:19:00.930: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:19:03.891: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:19:03.891: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197543.856,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197543.856]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:19:04.892: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:19:07.617: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:19:07.617: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197547.56,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197547.56]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:19:08.618: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:19:13.516: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:19:13.516: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197553.398,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197553.398]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:19:14.516: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:19:23.464: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:19:23.464: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197563.37,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197563.37]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:19:24.465: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:19:26.917: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:19:26.917: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197566.883,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197566.883]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:19:27.917: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:19:29.707: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:19:29.707: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197569.155,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197569.155]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:19:30.708: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:19:32.036: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:19:32.036: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197571.994,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197571.994]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:19:33.039: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:19:37.569: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:19:37.569: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197577.485,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197577.485]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:19:38.569: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:19:43.327: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:19:43.327: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197583.098,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197583.098]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:19:44.327: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:19:47.533: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:19:47.533: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197587.428,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197587.428]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:19:48.534: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:19:49.597: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:19:49.598: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197589.527,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197589.527]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:19:50.598: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:19:59.764: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:19:59.764: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197599.665,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197599.665]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:20:00.764: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:20:03.472: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:20:03.472: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197603.409,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197603.409]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:20:04.473: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:20:06.424: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:20:06.424: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197606.388,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197606.388]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:20:07.425: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:20:16.989: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:20:16.989: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197616.95,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197616.95]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:20:17.989: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:20:20.238: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:20:20.238: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197620.204,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197620.204]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:20:21.239: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:20:23.581: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:20:23.582: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197623.548,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197623.548]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:20:24.584: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:20:25.639: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:20:25.639: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197625.605,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197625.605]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:20:26.639: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:20:30.555: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:20:30.555: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197629.535,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197629.535]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:20:31.557: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:20:33.814: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:20:33.814: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197633.76,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197633.76]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:20:34.815: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:20:38.135: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:20:38.135: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197638.064,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197638.064]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:20:39.135: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:20:48.736: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:20:48.736: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197648.672,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197648.672]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:20:49.737: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:20:57.322: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:20:57.322: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197657.265,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197657.265]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:20:58.323: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:20:59.719: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:20:59.719: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197659.685,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197659.685]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:21:00.723: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:21:02.576: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:21:02.576: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197662.489,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197662.489]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:21:03.577: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:21:17.871: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:21:17.871: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197677.45,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197677.45]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:21:18.871: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:21:25.493: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:21:25.493: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197685.458,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197685.458]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:21:26.494: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:21:28.260: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:21:28.260: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197688.23,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197688.23]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:21:29.261: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:21:33.363: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:21:33.363: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197693.322,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197693.322]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:21:34.364: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:21:37.636: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:21:37.636: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197697.593,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197697.593]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:21:38.636: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:21:48.651: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:21:48.651: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197708.609,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197708.609]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:21:49.651: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:21:58.903: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:21:58.904: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197718.862,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197718.862]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:21:59.904: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:01.675: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:01.675: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197721.633,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197721.633]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:02.675: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:04.140: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:04.140: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197724.097,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197724.097]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:05.141: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:08.349: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:08.349: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197728.309,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197728.309]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:09.349: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:13.520: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:13.520: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197733.481,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197733.481]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:14.521: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:16.064: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:16.064: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197736.021,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197736.021]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:17.065: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:18.189: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:18.189: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197738.154,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197738.154]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:19.190: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:20.282: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:20.282: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197740.25,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197740.25]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:21.282: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:22.613: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:22.614: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197742.582,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197742.582]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:23.614: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:24.761: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:24.761: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197744.726,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197744.726]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:25.761: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:27.221: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:27.221: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197747.18,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197747.18]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:28.223: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:29.398: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:29.398: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197749.364,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197749.364]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:30.398: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:31.456: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:31.456: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197751.425,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197751.425]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:32.457: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:33.642: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:33.642: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197753.611,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197753.611]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:34.642: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:35.870: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:35.870: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197755.839,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197755.839]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:36.871: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:38.129: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:38.129: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197758.097,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197758.097]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:39.129: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:44.710: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:44.710: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197764.574,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197764.574]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:45.711: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:47.240: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:47.240: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197767.176,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197767.176]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:48.240: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' Nov 8 07:22:49.488: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n" Nov 8 07:22:49.488: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197769.457,\"1\"]}]}}" promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197769.457]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 Nov 8 07:22:50.489: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"' --------------------------------------------------------- Received interrupt. Running AfterSuite... ^C again to terminate immediately Nov 8 07:22:50.914: INFO: Running AfterSuite actions on all nodes Nov 8 07:22:50.915: INFO: Waiting up to 3m0s for all (but 100) nodes to be ready STEP: Destroying namespace "e2e-test-prometheus-fgfz6" for this suite. Nov 8 07:22:51.229: INFO: Running AfterSuite actions on node 1
Filter through log files
go run hack/e2e.go -v -test --test_args='--ginkgo.focus=openshift\-tests\s\[sig\-storage\]\sProjected\ssecret\sshould\sbe\sconsumable\sfrom\spods\sin\svolume\swith\sdefaultMode\sset\s\[LinuxOnly\]\s\[NodeConformance\]\s\[Conformance\]\s\[Suite\:openshift\/conformance\/parallel\/minimal\]\s\[Suite\:k8s\]$'
fail [k8s.io/kubernetes/test/e2e/framework/util.go:1486]: Unexpected error: <*errors.errorString | 0xc003826240>: { s: "expected pod \"pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d\" success: pod \"pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d\" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-08 06:47:43 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-08 06:47:43 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [projected-secret-volume-test]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-08 06:47:43 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [projected-secret-volume-test]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-08 06:47:43 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.0.32.6 PodIP:10.128.2.23 PodIPs:[{IP:10.128.2.23}] StartTime:2019-11-08 06:47:43 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:projected-secret-volume-test State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:-1,Signal:0,Reason:OOMKilled,Message:,StartedAt:2019-11-08 06:48:20 +0000 UTC,FinishedAt:1754-08-30 22:43:41 +0000 UTC,ContainerID:cri-o://3f46cfd3d7b4bd3fa05aa78fa3c8c8dd619de04898f81c0052154d728dc2e7e2,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:gcr.io/kubernetes-e2e-test-images/mounttest:1.0 ImageID:gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 ContainerID:cri-o://3f46cfd3d7b4bd3fa05aa78fa3c8c8dd619de04898f81c0052154d728dc2e7e2 Started:<nil>}] QOSClass:BestEffort EphemeralContainerStatuses:[]}", } expected pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d" success: pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-08 06:47:43 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-08 06:47:43 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [projected-secret-volume-test]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-08 06:47:43 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [projected-secret-volume-test]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-08 06:47:43 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.0.32.6 PodIP:10.128.2.23 PodIPs:[{IP:10.128.2.23}] StartTime:2019-11-08 06:47:43 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:projected-secret-volume-test State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:-1,Signal:0,Reason:OOMKilled,Message:,StartedAt:2019-11-08 06:48:20 +0000 UTC,FinishedAt:1754-08-30 22:43:41 +0000 UTC,ContainerID:cri-o://3f46cfd3d7b4bd3fa05aa78fa3c8c8dd619de04898f81c0052154d728dc2e7e2,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:gcr.io/kubernetes-e2e-test-images/mounttest:1.0 ImageID:gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 ContainerID:cri-o://3f46cfd3d7b4bd3fa05aa78fa3c8c8dd619de04898f81c0052154d728dc2e7e2 Started:<nil>}] QOSClass:BestEffort EphemeralContainerStatuses:[]} occurredfrom junit_e2e_20191108-073136.xml
Nov 8 06:47:41.444: INFO: >>> kubeConfig: /tmp/admin.kubeconfig Nov 8 06:47:41.448: INFO: >>> kubeConfig: /tmp/admin.kubeconfig Nov 8 06:47:41.705: INFO: >>> kubeConfig: /tmp/admin.kubeconfig Nov 8 06:47:42.712: INFO: >>> kubeConfig: /tmp/admin.kubeconfig Nov 8 06:47:42.715: INFO: Waiting up to 30m0s for all (but 100) nodes to be schedulable Nov 8 06:47:42.831: INFO: Waiting up to 10m0s for all pods (need at least 0) in namespace 'kube-system' to be running and ready Nov 8 06:47:42.975: INFO: 0 / 0 pods in namespace 'kube-system' are running and ready (0 seconds elapsed) Nov 8 06:47:42.975: INFO: expected 0 pod replicas in namespace 'kube-system', 0 are Running and Ready. Nov 8 06:47:42.975: INFO: Waiting up to 5m0s for all daemonsets in namespace 'kube-system' to start Nov 8 06:47:43.023: INFO: e2e test version: v1.16.2 Nov 8 06:47:43.061: INFO: kube-apiserver version: v1.16.2 Nov 8 06:47:43.061: INFO: >>> kubeConfig: /tmp/admin.kubeconfig Nov 8 06:47:43.111: INFO: Cluster IP family: ipv4 [BeforeEach] [Top Level] /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/util/test.go:60 [BeforeEach] [sig-storage] Projected secret /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:151 STEP: Creating a kubernetes client Nov 8 06:47:43.124: INFO: >>> kubeConfig: /tmp/admin.kubeconfig STEP: Building a namespace api object, basename projected Nov 8 06:47:43.279: INFO: About to run a Kube e2e test, ensuring namespace is privileged Nov 8 06:47:43.762: INFO: No PodSecurityPolicies found; assuming PodSecurityPolicy is disabled. STEP: Waiting for a default service account to be provisioned in namespace [It] should be consumable from pods in volume with defaultMode set [LinuxOnly] [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s] /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:694 STEP: Creating projection with secret that has name projected-secret-test-31407d46-4946-42eb-9c04-10abb073e793 STEP: Creating a pod to test consume secrets Nov 8 06:47:43.905: INFO: Waiting up to 5m0s for pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d" in namespace "e2e-projected-875" to be "success or failure" Nov 8 06:47:43.948: INFO: Pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d": Phase="Pending", Reason="", readiness=false. Elapsed: 43.211242ms Nov 8 06:47:45.998: INFO: Pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d": Phase="Pending", Reason="", readiness=false. Elapsed: 2.092962964s Nov 8 06:47:48.049: INFO: Pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d": Phase="Pending", Reason="", readiness=false. Elapsed: 4.143699559s Nov 8 06:47:50.094: INFO: Pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d": Phase="Pending", Reason="", readiness=false. Elapsed: 6.188797517s Nov 8 06:47:52.135: INFO: Pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d": Phase="Pending", Reason="", readiness=false. Elapsed: 8.229830138s Nov 8 06:47:54.176: INFO: Pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d": Phase="Pending", Reason="", readiness=false. Elapsed: 10.271387573s Nov 8 06:47:56.217: INFO: Pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d": Phase="Pending", Reason="", readiness=false. Elapsed: 12.312224014s Nov 8 06:47:58.265: INFO: Pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d": Phase="Pending", Reason="", readiness=false. Elapsed: 14.360407564s Nov 8 06:48:00.307: INFO: Pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d": Phase="Pending", Reason="", readiness=false. Elapsed: 16.40163422s Nov 8 06:48:02.347: INFO: Pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d": Phase="Pending", Reason="", readiness=false. Elapsed: 18.442101898s Nov 8 06:48:04.388: INFO: Pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d": Phase="Pending", Reason="", readiness=false. Elapsed: 20.483093671s Nov 8 06:48:06.430: INFO: Pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d": Phase="Pending", Reason="", readiness=false. Elapsed: 22.524870752s Nov 8 06:48:08.471: INFO: Pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d": Phase="Pending", Reason="", readiness=false. Elapsed: 24.566093306s Nov 8 06:48:10.512: INFO: Pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d": Phase="Pending", Reason="", readiness=false. Elapsed: 26.607088479s Nov 8 06:48:12.555: INFO: Pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d": Phase="Pending", Reason="", readiness=false. Elapsed: 28.650130435s Nov 8 06:48:14.597: INFO: Pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d": Phase="Pending", Reason="", readiness=false. Elapsed: 30.691611466s Nov 8 06:48:16.641: INFO: Pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d": Phase="Pending", Reason="", readiness=false. Elapsed: 32.735897885s Nov 8 06:48:18.684: INFO: Pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d": Phase="Pending", Reason="", readiness=false. Elapsed: 34.779314458s Nov 8 06:48:20.728: INFO: Pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d": Phase="Pending", Reason="", readiness=false. Elapsed: 36.823351594s Nov 8 06:48:22.769: INFO: Pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d": Phase="Pending", Reason="", readiness=false. Elapsed: 38.864514688s Nov 8 06:48:24.811: INFO: Pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d": Phase="Pending", Reason="", readiness=false. Elapsed: 40.905571395s Nov 8 06:48:26.860: INFO: Pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d": Phase="Pending", Reason="", readiness=false. Elapsed: 42.954910333s Nov 8 06:48:28.902: INFO: Pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d": Phase="Pending", Reason="", readiness=false. Elapsed: 44.996603696s Nov 8 06:48:30.951: INFO: Pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d": Phase="Failed", Reason="", readiness=false. Elapsed: 47.045704442s Nov 8 06:48:31.051: INFO: Output of node "ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv" pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d" container "projected-secret-volume-test": mode of file "/etc/projected-secret-volume/data-1": -r-------- content of file "/etc/projected-secret-volume/data-1": value-1 STEP: delete the pod Nov 8 06:48:31.105: INFO: Waiting for pod pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d to disappear Nov 8 06:48:31.147: INFO: Pod pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d no longer exists [AfterEach] [sig-storage] Projected secret /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:152 STEP: Collecting events from namespace "e2e-projected-875". STEP: Found 5 events. Nov 8 06:48:31.193: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d: {default-scheduler } Scheduled: Successfully assigned e2e-projected-875/pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d to ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv Nov 8 06:48:31.193: INFO: At 2019-11-08 06:47:51 +0000 UTC - event for pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv} Pulling: Pulling image "gcr.io/kubernetes-e2e-test-images/mounttest:1.0" Nov 8 06:48:31.193: INFO: At 2019-11-08 06:48:13 +0000 UTC - event for pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv} Pulled: Successfully pulled image "gcr.io/kubernetes-e2e-test-images/mounttest:1.0" Nov 8 06:48:31.193: INFO: At 2019-11-08 06:48:20 +0000 UTC - event for pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv} Created: Created container projected-secret-volume-test Nov 8 06:48:31.193: INFO: At 2019-11-08 06:48:30 +0000 UTC - event for pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d: {kubelet ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv} Started: Started container projected-secret-volume-test Nov 8 06:48:31.235: INFO: POD NODE PHASE GRACE CONDITIONS Nov 8 06:48:31.235: INFO: Nov 8 06:48:31.357: INFO: skipping dumping cluster info - cluster too large Nov 8 06:48:31.357: INFO: Waiting up to 3m0s for all (but 100) nodes to be ready STEP: Destroying namespace "e2e-projected-875" for this suite. Nov 8 06:48:31.459: INFO: Running AfterSuite actions on all nodes Nov 8 06:48:31.459: INFO: Running AfterSuite actions on node 1 fail [k8s.io/kubernetes/test/e2e/framework/util.go:1486]: Unexpected error: <*errors.errorString | 0xc003826240>: { s: "expected pod \"pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d\" success: pod \"pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d\" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-08 06:47:43 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-08 06:47:43 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [projected-secret-volume-test]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-08 06:47:43 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [projected-secret-volume-test]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-08 06:47:43 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.0.32.6 PodIP:10.128.2.23 PodIPs:[{IP:10.128.2.23}] StartTime:2019-11-08 06:47:43 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:projected-secret-volume-test State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:-1,Signal:0,Reason:OOMKilled,Message:,StartedAt:2019-11-08 06:48:20 +0000 UTC,FinishedAt:1754-08-30 22:43:41 +0000 UTC,ContainerID:cri-o://3f46cfd3d7b4bd3fa05aa78fa3c8c8dd619de04898f81c0052154d728dc2e7e2,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:gcr.io/kubernetes-e2e-test-images/mounttest:1.0 ImageID:gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 ContainerID:cri-o://3f46cfd3d7b4bd3fa05aa78fa3c8c8dd619de04898f81c0052154d728dc2e7e2 Started:<nil>}] QOSClass:BestEffort EphemeralContainerStatuses:[]}", } expected pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d" success: pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-08 06:47:43 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-08 06:47:43 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [projected-secret-volume-test]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-08 06:47:43 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [projected-secret-volume-test]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-08 06:47:43 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.0.32.6 PodIP:10.128.2.23 PodIPs:[{IP:10.128.2.23}] StartTime:2019-11-08 06:47:43 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:projected-secret-volume-test State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:-1,Signal:0,Reason:OOMKilled,Message:,StartedAt:2019-11-08 06:48:20 +0000 UTC,FinishedAt:1754-08-30 22:43:41 +0000 UTC,ContainerID:cri-o://3f46cfd3d7b4bd3fa05aa78fa3c8c8dd619de04898f81c0052154d728dc2e7e2,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:gcr.io/kubernetes-e2e-test-images/mounttest:1.0 ImageID:gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 ContainerID:cri-o://3f46cfd3d7b4bd3fa05aa78fa3c8c8dd619de04898f81c0052154d728dc2e7e2 Started:<nil>}] QOSClass:BestEffort EphemeralContainerStatuses:[]} occurred
Find pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d mentions in log files
go run hack/e2e.go -v -test --test_args='--ginkgo.focus=operator\sRun\stemplate\se2e\-azure\s\-\se2e\-azure\-ovn\-kubernetes\scontainer\stest$'
when installed on the cluster shouldn't report any alerts in firing state apart from Watchdog [Suite:openshift/conformance/parallel/minimal]" failed
W1108 07:24:10.995684 186 reflector.go:299] github.com/openshift/origin/pkg/monitor/operator.go:126: watch of *v1.ClusterOperator ended with: too old resource version: 135007 (135191)
W1108 07:24:16.943253 186 reflector.go:299] github.com/openshift/origin/pkg/monitor/operator.go:126: watch of *v1.ClusterOperator ended with: too old resource version: 135191 (135274)
W1108 07:27:22.508587 186 reflector.go:299] github.com/openshift/origin/pkg/monitor/operator.go:126: watch of *v1.ClusterOperator ended with: too old resource version: 135849 (136109)
W1108 07:27:28.421147 186 reflector.go:299] github.com/openshift/origin/pkg/monitor/operator.go:126: watch of *v1.ClusterOperator ended with: too old resource version: 136112 (136125)
W1108 07:30:33.945488 186 reflector.go:299] github.com/openshift/origin/pkg/monitor/operator.go:126: watch of *v1.ClusterOperator ended with: too old resource version: 136873 (136979)
W1108 07:30:39.848272 186 reflector.go:299] github.com/openshift/origin/pkg/monitor/operator.go:126: watch of *v1.ClusterOperator ended with: too old resource version: 136979 (137003)
Flaky tests:
[Feature:ProjectAPI] TestProjectWatch should succeed [Suite:openshift/conformance/parallel]
[sig-storage] Projected secret should be consumable from pods in volume with defaultMode set [LinuxOnly] [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
Failing tests:
[Feature:Platform] Managed cluster should have no crashlooping pods in core namespaces over two minutes [Suite:openshift/conformance/parallel]
[Feature:Prometheus][Conformance] Prometheus when installed on the cluster shouldn't report any alerts in firing state apart from Watchdog [Suite:openshift/conformance/parallel/minimal]
Writing JUnit report to /tmp/artifacts/junit/junit_e2e_20191108-073136.xml
error: 4 fail, 859 pass, 1217 skip (37m21s)
from junit_operator.xml
Filter through log files
openshift-tests The bootstrap user should successfully login with password decoded from kubeadmin secret [Suite:openshift/conformance/parallel]
openshift-tests [Area:Networking] network isolation when using a plugin that does not isolate namespaces by default should allow communication between pods in different namespaces on different nodes [Suite:openshift/conformance/parallel]
openshift-tests [Area:Networking] network isolation when using a plugin that does not isolate namespaces by default should allow communication between pods in different namespaces on the same node [Suite:openshift/conformance/parallel]
openshift-tests [Area:Networking] services basic functionality should allow connections to another pod on a different node via a service IP [Suite:openshift/conformance/parallel]
openshift-tests [Area:Networking] services basic functionality should allow connections to another pod on the same node via a service IP [Suite:openshift/conformance/parallel]
openshift-tests [Area:Networking] services when using a plugin that does not isolate namespaces by default should allow connections to pods in different namespaces on different nodes via service IPs [Suite:openshift/conformance/parallel]
openshift-tests [Area:Networking] services when using a plugin that does not isolate namespaces by default should allow connections to pods in different namespaces on the same node via service IPs [Suite:openshift/conformance/parallel]
openshift-tests [Area:Networking][endpoints] admission TestEndpointAdmission [Suite:openshift/conformance/parallel]
openshift-tests [Conformance][Area:Networking][Feature:Router] The HAProxy router converges when multiple routers are writing conflicting status [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Conformance][Area:Networking][Feature:Router] The HAProxy router converges when multiple routers are writing status [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Conformance][Area:Networking][Feature:Router] The HAProxy router should enable openshift-monitoring to pull metrics [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Conformance][Area:Networking][Feature:Router] The HAProxy router should expose a health check on the metrics port [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Conformance][Area:Networking][Feature:Router] The HAProxy router should expose prometheus metrics for a route [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Conformance][Area:Networking][Feature:Router] The HAProxy router should expose the profiling endpoints [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Conformance][Area:Networking][Feature:Router] The HAProxy router should override the route host for overridden domains with a custom value [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Conformance][Area:Networking][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Conformance][Area:Networking][Feature:Router] The HAProxy router should respond with 503 to unrecognized hosts [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Conformance][Area:Networking][Feature:Router] The HAProxy router should run even if it has no access to update status [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Conformance][Area:Networking][Feature:Router] The HAProxy router should serve a route that points to two services and respect weights [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Conformance][Area:Networking][Feature:Router] The HAProxy router should serve routes that were created from an ingress [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Conformance][Area:Networking][Feature:Router] The HAProxy router should serve the correct routes when scoped to a single namespace and label set [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Conformance][Area:Networking][Feature:Router] The HAProxy router should support reencrypt to services backed by a serving certificate automatically [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature: RoleBinding Restrictions] RoleBindingRestrictions should be functional Create a RBAC rolebinding when subject is not already bound and is not permitted by any RBR should fail [Suite:openshift/conformance/parallel]
openshift-tests [Feature: RoleBinding Restrictions] RoleBindingRestrictions should be functional Create a rolebinding that also contains system:non-existing users should succeed [Suite:openshift/conformance/parallel]
openshift-tests [Feature: RoleBinding Restrictions] RoleBindingRestrictions should be functional Create a rolebinding when subject is already bound should succeed [Suite:openshift/conformance/parallel]
openshift-tests [Feature: RoleBinding Restrictions] RoleBindingRestrictions should be functional Create a rolebinding when subject is not already bound and is not permitted by any RBR should fail [Suite:openshift/conformance/parallel]
openshift-tests [Feature: RoleBinding Restrictions] RoleBindingRestrictions should be functional Create a rolebinding when subject is permitted by RBR should succeed [Suite:openshift/conformance/parallel]
openshift-tests [Feature: RoleBinding Restrictions] RoleBindingRestrictions should be functional Create a rolebinding when there are no restrictions should succeed [Suite:openshift/conformance/parallel]
openshift-tests [Feature: RoleBinding Restrictions] RoleBindingRestrictions should be functional Rolebinding restrictions tests single project should succeed [Suite:openshift/conformance/parallel]
openshift-tests [Feature:APIServer] anonymous browsers should get a 403 from / [Suite:openshift/conformance/parallel]
openshift-tests [Feature:APIServer] authenticated browser should get a 200 from / [Suite:openshift/conformance/parallel]
openshift-tests [Feature:AnnotationTrigger] Annotation trigger reconciles after the image is overwritten [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Authentication] TestFrontProxy should succeed [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Builds] Multi-stage image builds should succeed [Conformance] [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:Builds] Optimized image builds should succeed [Conformance] [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:Builds] build have source revision metadata started build should contain source revision information [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Builds] build with empty source started build should build even with an empty source in build config [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Builds] buildconfig secret injector should inject secrets to the appropriate buildconfigs [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Builds] result image should have proper labels set Docker build from a template should create a image from "test-docker-build.json" template with proper Docker labels [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Builds] result image should have proper labels set S2I build from a template should create a image from "test-s2i-build.json" template with proper Docker labels [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Builds][Conformance] build can reference a cluster service with a build being created from new-build should be able to run a build that references a cluster service [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:Builds][Conformance] build without output image building from templates should create an image from a S2i template without an output image reference defined [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:Builds][Conformance] build without output image building from templates should create an image from a docker template without an output image reference defined [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:Builds][Conformance] custom build with buildah being created from new-build should complete build with custom builder image [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:Builds][Conformance] imagechangetriggers imagechangetriggers should trigger builds of all types [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:Builds][Conformance] oc new-app should fail with a --name longer than 58 characters [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:Builds][Conformance] oc new-app should succeed with a --name of 58 characters [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:Builds][Conformance] remove all builds when build configuration is removed oc delete buildconfig should start builds and delete the buildconfig [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:Builds][Conformance] s2i build with a quota Building from a template should create an s2i build with a quota and run it [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:Builds][Conformance] s2i build with a root user image should create a root build and pass with a privileged SCC [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:Builds][Conformance][valueFrom] process valueFrom in build strategy environment variables should fail resolving unresolvable valueFrom in docker build environment variable references [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:Builds][Conformance][valueFrom] process valueFrom in build strategy environment variables should fail resolving unresolvable valueFrom in sti build environment variable references [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:Builds][Conformance][valueFrom] process valueFrom in build strategy environment variables should successfully resolve valueFrom in docker build environment variables [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:Builds][Conformance][valueFrom] process valueFrom in build strategy environment variables should successfully resolve valueFrom in s2i build environment variables [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:Builds][pruning] prune builds based on settings in the buildconfig [Conformance] buildconfigs should have a default history limit set when created via the group api [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:Builds][pruning] prune builds based on settings in the buildconfig should prune builds after a buildConfig change [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Builds][pruning] prune builds based on settings in the buildconfig should prune canceled builds based on the failedBuildsHistoryLimit setting [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Builds][pruning] prune builds based on settings in the buildconfig should prune completed builds based on the successfulBuildsHistoryLimit setting [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Builds][pruning] prune builds based on settings in the buildconfig should prune errored builds based on the failedBuildsHistoryLimit setting [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Builds][pruning] prune builds based on settings in the buildconfig should prune failed builds based on the failedBuildsHistoryLimit setting [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Builds][pullsecret][Conformance] docker build using a pull secret Building from a template should create a docker build that pulls using a secret run it [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:Builds][timing] capture build stages and durations should record build stages and durations for docker [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Builds][timing] capture build stages and durations should record build stages and durations for s2i [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Builds][webhook] TestWebhook [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Builds][webhook] TestWebhookGitHubPing [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Builds][webhook] TestWebhookGitHubPushWithImage [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Builds][webhook] TestWebhookGitHubPushWithImageStream [Suite:openshift/conformance/parallel]
openshift-tests [Feature:CLI] CLI can run inside of a busybox container [Suite:openshift/conformance/parallel]
openshift-tests [Feature:ClusterResourceQuota] TestClusterQuota should succeed [Suite:openshift/conformance/parallel]
openshift-tests [Feature:DeploymentConfig] deploymentconfigs should adhere to Three Laws of Controllers [Conformance] [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:DeploymentConfig] deploymentconfigs adoption [Conformance] will orphan all RCs and adopt them back when recreated [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:DeploymentConfig] deploymentconfigs generation [Conformance] should deploy based on a status version bump [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:DeploymentConfig] deploymentconfigs ignores deployer and lets the config with a NewReplicationControllerCreated reason [Conformance] should let the deployment config with a NewReplicationControllerCreated reason [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:DeploymentConfig] deploymentconfigs initially [Conformance] should not deploy if pods never transition to ready [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:DeploymentConfig] deploymentconfigs keep the deployer pod invariant valid [Conformance] should deal with cancellation after deployer pod succeeded [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:DeploymentConfig] deploymentconfigs keep the deployer pod invariant valid [Conformance] should deal with cancellation of running deployment [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:DeploymentConfig] deploymentconfigs keep the deployer pod invariant valid [Conformance] should deal with config change in case the deployment is still running [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:DeploymentConfig] deploymentconfigs paused [Conformance] should disable actions on deployments [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:DeploymentConfig] deploymentconfigs rolled back [Conformance] should rollback to an older deployment [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:DeploymentConfig] deploymentconfigs should respect image stream tag reference policy [Conformance] resolve the image pull spec [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:DeploymentConfig] deploymentconfigs viewing rollout history [Conformance] should print the rollout history [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:DeploymentConfig] deploymentconfigs when changing image change trigger [Conformance] should successfully trigger from an updated image [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:DeploymentConfig] deploymentconfigs when run iteratively [Conformance] should immediately start a new deployment [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:DeploymentConfig] deploymentconfigs when run iteratively [Conformance] should only deploy the last deployment [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:DeploymentConfig] deploymentconfigs when tagging images [Conformance] should successfully tag the deployed image [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:DeploymentConfig] deploymentconfigs with custom deployments [Conformance] should run the custom deployment steps [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:DeploymentConfig] deploymentconfigs with enhanced status [Conformance] should include various info in status [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:DeploymentConfig] deploymentconfigs with env in params referencing the configmap [Conformance] should expand the config map key to a value [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:DeploymentConfig] deploymentconfigs with failing hook [Conformance] should get all logs from retried hooks [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:DeploymentConfig] deploymentconfigs with minimum ready seconds set [Conformance] should not transition the deployment to Complete before satisfied [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:DeploymentConfig] deploymentconfigs with multiple image change triggers [Conformance] should run a successful deployment with a trigger used by different containers [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:DeploymentConfig] deploymentconfigs with multiple image change triggers [Conformance] should run a successful deployment with multiple triggers [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:DeploymentConfig] deploymentconfigs with revision history limits [Conformance] should never persist more old deployments than acceptable after being observed by the controller [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:DeploymentConfig] deploymentconfigs with test deployments [Conformance] should run a deployment to completion and then scale to zero [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:DeploymentConfig] deploymentconfigs won't deploy RC with unresolved images [Conformance] when patched with empty image [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:ImageAppend] Image append should create images by appending them [Suite:openshift/conformance/parallel]
openshift-tests [Feature:ImageExtract] Image extract should extract content from an image [Suite:openshift/conformance/parallel]
openshift-tests [Feature:ImageInfo] Image info should display information about images [Suite:openshift/conformance/parallel]
openshift-tests [Feature:ImageLayers][registry] Image layer subresource should identify a deleted image as missing [Suite:openshift/conformance/parallel]
openshift-tests [Feature:ImageLayers][registry] Image layer subresource should return layers from tagged images [Suite:openshift/conformance/parallel]
openshift-tests [Feature:ImageLookup][registry][Conformance] Image policy should perform lookup when the pod has the resolve-names annotation [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:ImageLookup][registry][Conformance] Image policy should update standard Kube object image fields when local names are on [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:Image] signature TestImageAddSignature [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Image] signature TestImageRemoveSignature [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Image][triggers] Image change build triggers TestMultipleImageChangeBuildTriggers [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Image][triggers] Image change build triggers TestSimpleImageChangeBuildTriggerFromImageStreamTagCustom [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Image][triggers] Image change build triggers TestSimpleImageChangeBuildTriggerFromImageStreamTagCustomWithConfigChange [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Image][triggers] Image change build triggers TestSimpleImageChangeBuildTriggerFromImageStreamTagDocker [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Image][triggers] Image change build triggers TestSimpleImageChangeBuildTriggerFromImageStreamTagDockerWithConfigChange [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Image][triggers] Image change build triggers TestSimpleImageChangeBuildTriggerFromImageStreamTagSTI [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Image][triggers] Image change build triggers TestSimpleImageChangeBuildTriggerFromImageStreamTagSTIWithConfigChange [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Machines] Managed cluster should have machine resources [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OAuthServer] OAuth server has the correct token and certificate fallback semantics [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OAuthServer] OAuth server should use http1.1 only to prevent http2 connection reuse [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OAuthServer] [Headers] expected headers returned from the authorize URL [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OAuthServer] [Headers] expected headers returned from the grant URL [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OAuthServer] [Headers] expected headers returned from the login URL for the allow all IDP [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OAuthServer] [Headers] expected headers returned from the login URL for the bootstrap IDP [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OAuthServer] [Headers] expected headers returned from the login URL for when there is only one IDP [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OAuthServer] [Headers] expected headers returned from the logout URL [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OAuthServer] [Headers] expected headers returned from the root URL [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OAuthServer] [Headers] expected headers returned from the token URL [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OAuthServer] [Headers] expected headers returned from the token request URL [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OAuthServer] [Token Expiration] Using a OAuth client with a non-default token max age to generate tokens that do not expire works as expected when using a code authorization flow [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OAuthServer] [Token Expiration] Using a OAuth client with a non-default token max age to generate tokens that do not expire works as expected when using a token authorization flow [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OAuthServer] [Token Expiration] Using a OAuth client with a non-default token max age to generate tokens that expire shortly works as expected when using a code authorization flow [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OAuthServer] [Token Expiration] Using a OAuth client with a non-default token max age to generate tokens that expire shortly works as expected when using a token authorization flow [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OAuthServer] well-known endpoint should be reachable [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OpenShiftAuthorization] RBAC proxy for openshift authz RunLegacyClusterRoleBindingEndpoint should succeed [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OpenShiftAuthorization] RBAC proxy for openshift authz RunLegacyClusterRoleEndpoint should succeed [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OpenShiftAuthorization] RBAC proxy for openshift authz RunLegacyEndpointConfirmNoEscalation should succeed [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OpenShiftAuthorization] RBAC proxy for openshift authz RunLegacyLocalRoleBindingEndpoint should succeed [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OpenShiftAuthorization] RBAC proxy for openshift authz RunLegacyLocalRoleEndpoint should succeed [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OpenShiftAuthorization] The default cluster RBAC policy should have correct RBAC rules [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OpenShiftAuthorization] authorization TestAuthorizationSubjectAccessReview should succeed [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OpenShiftAuthorization] authorization TestAuthorizationSubjectAccessReviewAPIGroup should succeed [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OpenShiftAuthorization] authorization TestBrowserSafeAuthorizer should succeed [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OpenShiftAuthorization] scopes TestScopeEscalations should succeed [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OpenShiftAuthorization] scopes TestScopedImpersonation should succeed [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OpenShiftAuthorization] scopes TestScopedTokens should succeed [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OpenShiftAuthorization] scopes TestTokensWithIllegalScopes should succeed [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OpenShiftAuthorization] scopes TestUnknownScopes should succeed [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OpenShiftAuthorization] self-SAR compatibility TestBootstrapPolicySelfSubjectAccessReviews should succeed [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OpenShiftAuthorization] self-SAR compatibility TestSelfSubjectAccessReviewsNonExistingNamespace should succeed [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OpenShiftControllerManager] TestAutomaticCreationOfPullSecrets [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OpenShiftControllerManager] TestDeployScale [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OpenShiftControllerManager] TestDeploymentConfigDefaults [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OpenShiftControllerManager] TestTriggers_MultipleICTs [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OpenShiftControllerManager] TestTriggers_configChange [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OpenShiftControllerManager] TestTriggers_imageChange [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OpenShiftControllerManager] TestTriggers_imageChange_nonAutomatic [Suite:openshift/conformance/parallel]
openshift-tests [Feature:OpenShiftControllerManager] TestTriggers_manual [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Platform] ClusterOperators should define at least one namespace in their lists of related objects [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Platform] ClusterOperators should define at least one related object that is not a namespace [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Platform] Managed cluster should ensure control plane operators do not make themselves unevictable [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Platform] Managed cluster should ensure control plane pods do not run in best-effort QoS [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Platform] Managed cluster should ensure pods use downstream images from our release image with proper ImagePullPolicy [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Platform] Managed cluster should have operators on the cluster version [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Platform] Managed cluster should should expose cluster services outside the cluster [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Platform] OLM should be installed with catalogsources at version v1alpha1 [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Platform] OLM should be installed with clusterserviceversions at version v1alpha1 [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Platform] OLM should be installed with installplans at version v1alpha1 [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Platform] OLM should be installed with operatorgroups at version v1 [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Platform] OLM should be installed with packagemanifests at version v1 [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Platform] OLM should be installed with subscriptions at version v1alpha1 [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Platform] OLM should have imagePullPolicy:IfNotPresent on thier deployments [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Platform][Smoke] Managed cluster should start all core operators [Suite:openshift/conformance/parallel]
openshift-tests [Feature:ProjectAPI] TestInvalidRoleRefs should succeed [Suite:openshift/conformance/parallel]
openshift-tests [Feature:ProjectAPI] TestProjectIsNamespace should succeed [Suite:openshift/conformance/parallel]
openshift-tests [Feature:ProjectAPI] TestProjectWatchWithSelectionPredicate should succeed [Suite:openshift/conformance/parallel]
openshift-tests [Feature:ProjectAPI] TestScopedProjectAccess should succeed [Suite:openshift/conformance/parallel]
openshift-tests [Feature:ProjectAPI] TestUnprivilegedNewProject [Suite:openshift/conformance/parallel]
openshift-tests [Feature:Prometheus][Conformance] Prometheus when installed on the cluster should have a Watchdog alert in firing state [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:Prometheus][Conformance] Prometheus when installed on the cluster should have non-Pod host cAdvisor metrics [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:Prometheus][Conformance] Prometheus when installed on the cluster should provide ingress metrics [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:Prometheus][Conformance] Prometheus when installed on the cluster should report telemetry if a cloud.openshift.com token is present [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:Prometheus][Conformance] Prometheus when installed on the cluster should start and expose a secured proxy and unsecured metrics [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:Prometheus][Conformance] Prometheus when installed on the cluster shouldn't have failing rules evaluation [Suite:openshift/conformance/parallel/minimal]
openshift-tests [Feature:Prometheus][Feature:Builds] Prometheus when installed on the cluster should start and expose a secured proxy and verify build metrics [Suite:openshift/conformance/parallel]
openshift-tests [Feature:SecurityContextConstraints] TestAllowedSCCViaRBAC [Suite:openshift/conformance/parallel]
openshift-tests [Feature:SecurityContextConstraints] TestPodUpdateSCCEnforcement [Suite:openshift/conformance/parallel]
openshift-tests [Feature:UserAPI] groups should work [Suite:openshift/conformance/parallel]
openshift-tests [Feature:UserAPI] users can manipulate groups [Suite:openshift/conformance/parallel]
openshift-tests [Suite:openshift/crdvalidation/apiserver] APIServer CR fields validation additionalCORSAllowedOrigins [Suite:openshift/conformance/parallel]
openshift-tests [Suite:openshift/oauth/htpasswd] HTPasswd IDP should successfully configure htpasswd and be responsive [Suite:openshift/conformance/parallel]
openshift-tests [Suite:openshift/oauth] LDAP IDP should authenticate against an ldap server [Suite:openshift/conformance/parallel]
openshift-tests [Suite:openshift/oauth] LDAP should start an OpenLDAP test server [Suite:openshift/conformance/parallel]
openshift-tests [cli] oc adm must-gather runs successfully [Suite:openshift/conformance/parallel]
openshift-tests [cli] oc adm must-gather runs successfully with options [Suite:openshift/conformance/parallel]
openshift-tests [cli] oc explain should contain proper fields description for special types [Suite:openshift/conformance/parallel]
openshift-tests [cli] oc explain should contain proper spec+status for CRDs [Suite:openshift/conformance/parallel]
openshift-tests [cli] oc explain should contain spec+status for builtinTypes [Suite:openshift/conformance/parallel]
openshift-tests [cli]oc rsh[Conformance] rsh specific flags should work well when access to a remote shell [Suite:openshift/conformance/parallel/minimal]
openshift-tests [image_ecosystem][mongodb] openshift mongodb image creating from a template should instantiate the template [Suite:openshift/conformance/parallel]
openshift-tests [job][Conformance] openshift can execute jobs controller should create and run a job in user project [Suite:openshift/conformance/parallel/minimal]
openshift-tests [k8s.io] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart exec hook properly [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart http hook properly [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestop exec hook properly [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestop http hook properly [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Container Runtime blackbox test on terminated container should report termination message [LinuxOnly] as empty when pod succeeds and TerminationMessagePolicy FallbackToLogsOnError is set [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Container Runtime blackbox test on terminated container should report termination message [LinuxOnly] from file when pod succeeds and TerminationMessagePolicy FallbackToLogsOnError is set [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Container Runtime blackbox test on terminated container should report termination message [LinuxOnly] from log output if TerminationMessagePolicy FallbackToLogsOnError is set [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Container Runtime blackbox test on terminated container should report termination message [LinuxOnly] if TerminationMessagePath is set [NodeConformance] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] Container Runtime blackbox test on terminated container should report termination message [LinuxOnly] if TerminationMessagePath is set as non-root user and at a non-default path [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Container Runtime blackbox test when running a container with a new image should be able to pull from private registry with secret [NodeConformance] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] Container Runtime blackbox test when running a container with a new image should be able to pull image from docker hub [NodeConformance] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] Container Runtime blackbox test when running a container with a new image should be able to pull image from gcr.io [NodeConformance] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] Container Runtime blackbox test when running a container with a new image should not be able to pull from private registry without secret [NodeConformance] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] Container Runtime blackbox test when running a container with a new image should not be able to pull image from invalid registry [NodeConformance] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] Container Runtime blackbox test when running a container with a new image should not be able to pull non-existing image from gcr.io [NodeConformance] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] Container Runtime blackbox test when starting a container that exits should run with the expected status [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Docker Containers should be able to override the image's default arguments (docker cmd) [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Docker Containers should be able to override the image's default command (docker entrypoint) [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Docker Containers should be able to override the image's default command and arguments [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Docker Containers should use the image defaults if command and args are blank [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] InitContainer [NodeConformance] should invoke init containers on a RestartAlways pod [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] InitContainer [NodeConformance] should invoke init containers on a RestartNever pod [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] InitContainer [NodeConformance] should not start app containers and fail the pod if init containers fail on a RestartNever pod [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] InitContainer [NodeConformance] should not start app containers if init containers fail on a RestartAlways pod [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Kubelet when scheduling a busybox Pod with hostAliases should write entries to /etc/hosts [LinuxOnly] [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Kubelet when scheduling a busybox command in a pod should print the output to logs [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Kubelet when scheduling a busybox command that always fails in a pod should be possible to delete [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Kubelet when scheduling a busybox command that always fails in a pod should have an terminated reason [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Kubelet when scheduling a read only busybox container should not write to root filesystem [LinuxOnly] [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] KubeletManagedEtcHosts should test kubelet managed /etc/hosts file [LinuxOnly] [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Lease API should be available [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] Pods should allow activeDeadlineSeconds to be updated [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Pods should be submitted and removed [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Pods should be updated [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Pods should contain environment variables for services [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Pods should get a host IP [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Pods should support pod readiness gates [NodeFeature:PodReadinessGate] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] Pods should support remote command execution over websockets [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Pods should support retrieving logs from the container over websockets [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] PrivilegedPod [NodeConformance] should enable privileged commands [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] Probing container should *not* be restarted with a /healthz http liveness probe [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Probing container should *not* be restarted with a exec "cat /tmp/health" liveness probe [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Probing container should *not* be restarted with a non-local redirect http liveness probe [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] Probing container should *not* be restarted with a tcp:8080 liveness probe [NodeConformance] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] Probing container should be restarted with a /healthz http liveness probe [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Probing container should be restarted with a exec "cat /tmp/health" liveness probe [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Probing container should be restarted with a local redirect http liveness probe [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] Probing container should have monotonically increasing restart count [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Probing container with readiness probe should not be ready before initial delay and never restart [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Probing container with readiness probe that fails should never be ready and never restart [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Security Context When creating a container with runAsNonRoot should not run with an explicit root user ID [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] Security Context When creating a container with runAsNonRoot should not run without a specified user ID [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] Security Context When creating a container with runAsNonRoot should run with an explicit non-root user ID [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] Security Context When creating a container with runAsNonRoot should run with an image specified user ID [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] Security Context When creating a container with runAsUser should run the container with uid 0 [LinuxOnly] [NodeConformance] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] Security Context When creating a container with runAsUser should run the container with uid 65534 [LinuxOnly] [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Security Context When creating a pod with privileged should run the container as unprivileged when false [LinuxOnly] [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Security Context When creating a pod with readOnlyRootFilesystem should run the container with readonly rootfs when readOnlyRootFilesystem=true [LinuxOnly] [NodeConformance] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] Security Context When creating a pod with readOnlyRootFilesystem should run the container with writable rootfs when readOnlyRootFilesystem=false [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Security Context when creating containers with AllowPrivilegeEscalation should allow privilege escalation when not explicitly set and uid != 0 [LinuxOnly] [NodeConformance] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] Security Context when creating containers with AllowPrivilegeEscalation should allow privilege escalation when true [LinuxOnly] [NodeConformance] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] Security Context when creating containers with AllowPrivilegeEscalation should not allow privilege escalation when false [LinuxOnly] [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Variable Expansion should allow composing env vars into new env vars [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Variable Expansion should allow substituting values in a container's args [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Variable Expansion should allow substituting values in a container's command [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] Variable Expansion should allow substituting values in a volume subpath [sig-storage][NodeFeature:VolumeSubpathEnvExpansion] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] [Feature:Example] [k8s.io] Downward API should create a pod that prints his name and namespace [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] [Feature:Example] [k8s.io] Liveness liveness pods should be automatically restarted [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] [Feature:Example] [k8s.io] Secret should create a pod that reads a secret [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] [sig-node] Events should be sent by kubelets and the scheduler about pods scheduling and running [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] [sig-node] Mount propagation should propagate mounts to the host [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] [sig-node] Pods Extended [k8s.io] Delete Grace Period should be submitted and removed [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] [sig-node] Pods Extended [k8s.io] Pods Set QOS Class should be set on Pods with matching resource requests and limits for memory and cpu [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] [sig-node] PreStop graceful pod terminated should wait until preStop hook completes the process [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] [sig-node] PreStop should call prestop when killing a pod [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [k8s.io] [sig-node] Security Context should support container.SecurityContext.RunAsUser And container.SecurityContext.RunAsGroup [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] [sig-node] Security Context should support container.SecurityContext.RunAsUser [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] [sig-node] Security Context should support pod.Spec.SecurityContext.RunAsUser And pod.Spec.SecurityContext.RunAsGroup [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] [sig-node] Security Context should support pod.Spec.SecurityContext.RunAsUser [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] [sig-node] Security Context should support pod.Spec.SecurityContext.SupplementalGroups [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] [sig-node] Security Context should support seccomp alpha runtime/default annotation [Feature:Seccomp] [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] [sig-node] Security Context should support seccomp alpha unconfined annotation on the container [Feature:Seccomp] [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] [sig-node] Security Context should support seccomp alpha unconfined annotation on the pod [Feature:Seccomp] [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [k8s.io] [sig-node] Security Context should support seccomp default which is unconfined [Feature:Seccomp] [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] listing mutating webhooks should work [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] listing validating webhooks should work [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] patching/updating a mutating webhook should work [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] patching/updating a validating webhook should work [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should be able to deny attaching pod [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should be able to deny custom resource creation, update and deletion [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should be able to deny pod and configmap creation [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should deny crd creation [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should honor timeout [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should include webhook resources in discovery documents [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should mutate configmap [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should mutate custom resource [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should mutate custom resource with different stored version [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should mutate custom resource with pruning [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should mutate pod and apply defaults after mutation [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should not be able to mutate or prevent deletion of webhook configuration objects [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should unconditionally reject operations on fail closed webhook [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] Aggregator Should be able to support the 1.10 Sample API Server using the current Aggregator [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] CustomResourceConversionWebhook [Privileged:ClusterAdmin] should be able to convert a non homogeneous list of CRs [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] CustomResourceConversionWebhook [Privileged:ClusterAdmin] should be able to convert from CR v1 to CR v2 [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] CustomResourceDefinition Watch [Privileged:ClusterAdmin] CustomResourceDefinition Watch watch on custom resource definition objects [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] CustomResourceDefinition resources [Privileged:ClusterAdmin] Simple CustomResourceDefinition creating/deleting custom resource definition objects works [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] CustomResourceDefinition resources [Privileged:ClusterAdmin] Simple CustomResourceDefinition getting/updating/patching custom resource definition status sub-resource works [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] CustomResourceDefinition resources [Privileged:ClusterAdmin] Simple CustomResourceDefinition listing custom resource definition objects works [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] CustomResourceDefinition resources [Privileged:ClusterAdmin] should include custom resource definition resources in discovery documents [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] removes definition from spec when one version gets changed to not be served [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] updates the published spec when one version gets renamed [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for CRD preserving unknown fields at the schema root [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for CRD preserving unknown fields in an embedded object [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for CRD with validation schema [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for CRD without validation schema [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for multiple CRDs of different groups [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for multiple CRDs of same group and version but different kinds [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for multiple CRDs of same group but different versions [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] Discovery Custom resource should have storage version hash [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-api-machinery] Garbage collector should delete RS created by deployment when not orphaning [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] Garbage collector should delete jobs and pods created by cronjob [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-api-machinery] Garbage collector should delete pods created by rc when not orphaning [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] Garbage collector should keep the rc around until all its pods are deleted if the deleteOptions says so [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] Garbage collector should not be blocked by dependency circle [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] Garbage collector should not delete dependents that have both valid owner and owner that's waiting for dependents to be deleted [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] Garbage collector should orphan RS created by deployment when deleteOptions.PropagationPolicy is Orphan [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] Garbage collector should orphan pods created by rc if delete options say so [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] Garbage collector should orphan pods created by rc if deleteOptions.OrphanDependents is nil [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-api-machinery] Garbage collector should support cascading deletion of custom resources [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-api-machinery] Garbage collector should support orphan deletion of custom resources [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-api-machinery] Generated clientset should create pods, set the deletionTimestamp and deletionGracePeriodSeconds of the pod [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-api-machinery] Generated clientset should create v1beta1 cronJobs, delete cronJobs, watch cronJobs [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-api-machinery] ResourceQuota [Feature:PodPriority] should verify ResourceQuota's multiple priority class scope (quota set to pod count: 2) against 2 pods with same priority classes. [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-api-machinery] ResourceQuota [Feature:PodPriority] should verify ResourceQuota's priority class scope (cpu, memory quota set) against a pod with same priority class. [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-api-machinery] ResourceQuota [Feature:PodPriority] should verify ResourceQuota's priority class scope (quota set to pod count: 1) against 2 pods with different priority class. [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-api-machinery] ResourceQuota [Feature:PodPriority] should verify ResourceQuota's priority class scope (quota set to pod count: 1) against 2 pods with same priority class. [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-api-machinery] ResourceQuota [Feature:PodPriority] should verify ResourceQuota's priority class scope (quota set to pod count: 1) against a pod with different priority class (ScopeSelectorOpExists). [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-api-machinery] ResourceQuota [Feature:PodPriority] should verify ResourceQuota's priority class scope (quota set to pod count: 1) against a pod with different priority class (ScopeSelectorOpNotIn). [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-api-machinery] ResourceQuota [Feature:PodPriority] should verify ResourceQuota's priority class scope (quota set to pod count: 1) against a pod with same priority class. [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-api-machinery] ResourceQuota [Feature:ScopeSelectors] should verify ResourceQuota with best effort scope using scope-selectors. [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-api-machinery] ResourceQuota [Feature:ScopeSelectors] should verify ResourceQuota with terminating scopes through scope selectors. [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-api-machinery] ResourceQuota should be able to update and delete ResourceQuota. [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] ResourceQuota should create a ResourceQuota and capture the life of a configMap. [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] ResourceQuota should create a ResourceQuota and capture the life of a custom resource. [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-api-machinery] ResourceQuota should create a ResourceQuota and capture the life of a persistent volume claim with a storage class. [sig-storage] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-api-machinery] ResourceQuota should create a ResourceQuota and capture the life of a persistent volume claim. [sig-storage] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-api-machinery] ResourceQuota should create a ResourceQuota and capture the life of a pod. [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] ResourceQuota should create a ResourceQuota and capture the life of a replica set. [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] ResourceQuota should create a ResourceQuota and capture the life of a replication controller. [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] ResourceQuota should create a ResourceQuota and capture the life of a secret. [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] ResourceQuota should create a ResourceQuota and capture the life of a service. [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] ResourceQuota should create a ResourceQuota and ensure its status is promptly calculated. [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] ResourceQuota should verify ResourceQuota with best effort scope. [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] ResourceQuota should verify ResourceQuota with terminating scopes. [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] Secrets should be consumable from pods in env vars [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] Secrets should be consumable via the environment [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] Secrets should fail to create secret due to empty secret key [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] Servers with support for API chunking should return chunks of results for list calls [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-api-machinery] Servers with support for Table transformation should return a 406 for a backend which does not implement metadata [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] Servers with support for Table transformation should return chunks of table results for list calls [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-api-machinery] Servers with support for Table transformation should return generic metadata details across all namespaces for nodes [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-api-machinery] Servers with support for Table transformation should return pod details [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-api-machinery] Watchers should be able to restart watching from the last resource version observed by the previous watch [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] Watchers should be able to start watching from a specific resource version [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] Watchers should observe add, update, and delete watch notifications on configmaps [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] Watchers should observe an object deletion if it stops meeting the requirements of the selector [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-api-machinery] Watchers should receive events on concurrent watches in same order [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-apps] CronJob should delete successful/failed finished jobs with limit of one job [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-apps] CronJob should not emit unexpected warnings [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-apps] CronJob should remove from active list jobs that have been deleted [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-apps] CronJob should replace jobs when ReplaceConcurrent [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-apps] CronJob should schedule multiple jobs concurrently [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-apps] Deployment RecreateDeployment should delete old pods and create new ones [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-apps] Deployment RollingUpdateDeployment should delete old pods and create new ones [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-apps] Deployment deployment reaping should cascade to its replica sets and pods [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-apps] Deployment deployment should delete old replica sets [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-apps] Deployment deployment should support proportional scaling [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-apps] Deployment deployment should support rollover [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-apps] Deployment iterative rollouts should eventually progress [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-apps] Deployment test Deployment ReplicaSet orphaning and adoption regarding controllerRef [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-apps] DisruptionController evictions: enough pods, absolute => should allow an eviction [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-apps] DisruptionController evictions: enough pods, replicaSet, percentage => should allow an eviction [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-apps] DisruptionController evictions: maxUnavailable allow single eviction, percentage => should allow an eviction [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-apps] DisruptionController evictions: maxUnavailable deny evictions, integer => should not allow an eviction [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-apps] DisruptionController evictions: no PDB => should allow an eviction [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-apps] DisruptionController evictions: too few pods, absolute => should not allow an eviction [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-apps] DisruptionController evictions: too few pods, replicaSet, percentage => should not allow an eviction [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-apps] DisruptionController should block an eviction until the PDB is updated to allow it [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-apps] DisruptionController should create a PodDisruptionBudget [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-apps] DisruptionController should update PodDisruptionBudget status [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-apps] Job should adopt matching orphans and release non-matching pods [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-apps] Job should delete a job [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-apps] Job should fail to exceed backoffLimit [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-apps] Job should fail when exceeds active deadline [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-apps] Job should remove pods when job is deleted [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-apps] Job should run a job to completion when tasks sometimes fail and are locally restarted [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-apps] Job should run a job to completion when tasks succeed [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-apps] ReplicaSet should adopt matching pods on creation and release no longer matching pods [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-apps] ReplicaSet should serve a basic image on each replica with a public image [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-apps] ReplicaSet should surface a failure condition on a common issue like exceeded quota [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-apps] ReplicationController should adopt matching pods on creation [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-apps] ReplicationController should release no longer matching pods [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-apps] ReplicationController should serve a basic image on each replica with a public image [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-apps] ReplicationController should surface a failure condition on a common issue like exceeded quota [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] Should recreate evicted statefulset [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should adopt matching orphans and release non-matching pods [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should have a working scale subresource [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should implement legacy replacement when the update strategy is OnDelete [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should not deadlock when a pod's predecessor fails [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should perform canary updates and phased rolling updates of template modifications [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications with PVCs [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should provide basic identity [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-auth] Certificates API should support building a client with a CSR [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-auth] ServiceAccounts should allow opting out of API token automount [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-auth] [Feature:NodeAuthenticator] The kubelet can delegate ServiceAccount tokens to the API server [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-auth] [Feature:NodeAuthenticator] The kubelet's main port 10250 should reject requests with no credentials [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-auth] [Feature:NodeAuthorizer] A node shouldn't be able to create another node [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-auth] [Feature:NodeAuthorizer] A node shouldn't be able to delete another node [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-auth] [Feature:NodeAuthorizer] Getting a non-existent configmap should exit with the Forbidden error, not a NotFound error [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-auth] [Feature:NodeAuthorizer] Getting a non-existent secret should exit with the Forbidden error, not a NotFound error [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-auth] [Feature:NodeAuthorizer] Getting a secret for a workload the node has access to should succeed [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-auth] [Feature:NodeAuthorizer] Getting an existing configmap should exit with the Forbidden error [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-auth] [Feature:NodeAuthorizer] Getting an existing secret should exit with the Forbidden error [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl Port forwarding With a server listening on 0.0.0.0 should support forwarding over websockets [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl Port forwarding With a server listening on 0.0.0.0 that expects NO client request should support a client that connects, sends DATA, and disconnects [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl Port forwarding With a server listening on 0.0.0.0 that expects a client request should support a client that connects, sends DATA, and disconnects [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl Port forwarding With a server listening on 0.0.0.0 that expects a client request should support a client that connects, sends NO DATA, and disconnects [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl Port forwarding With a server listening on localhost should support forwarding over websockets [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl Port forwarding With a server listening on localhost that expects NO client request should support a client that connects, sends DATA, and disconnects [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl Port forwarding With a server listening on localhost that expects a client request should support a client that connects, sends DATA, and disconnects [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl Port forwarding With a server listening on localhost that expects a client request should support a client that connects, sends NO DATA, and disconnects [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl api-versions should check if v1 is in available api versions [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl apply apply set/view last-applied [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl apply should apply a new configuration to an existing RC [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl apply should reuse port when apply to an existing SVC [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl client-side validation should create/apply a CR with unknown fields for CRD with no validation schema [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl client-side validation should create/apply a valid CR for CRD with validation schema [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl client-side validation should create/apply a valid CR with arbitrary-extra properties for CRD with partially-specified validation schema [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl cluster-info dump should check if cluster-info dump succeeds [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl copy should copy a file from a running Pod [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl create quota should create a quota with scopes [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl create quota should create a quota without scopes [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl create quota should reject quota with invalid scopes [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl describe should check if kubectl describe prints relevant information for rc and pods [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl expose should create services for rc [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl get componentstatuses should get componentstatuses [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl label should update the label on a resource [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl logs should be able to retrieve and filter logs [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl patch should add annotations for pods in rc [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl replace should update a single-container pod's image [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl rolling-update should support rolling-update to same image [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl run --rm job should create a job from an image, then delete the job [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl run CronJob should create a CronJob [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl run default should create an rc or deployment from an image [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl run deployment should create a deployment from an image [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl run job should create a job from an image when restart is OnFailure [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl run pod should create a pod from an image when restart is Never [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl run rc should create an rc from an image [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Kubectl version should check is all data is printed [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Proxy server should support --unix-socket=/path [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Proxy server should support proxy with --port 0 [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Simple pod should contain last line of the log [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Simple pod should return command exit codes [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Simple pod should support exec [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Simple pod should support exec through an HTTP proxy [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Simple pod should support exec through kubectl proxy [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Simple pod should support exec using resource/name [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Simple pod should support inline execution and attach [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Simple pod should support port-forward [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Update Demo should create and stop a replication controller [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Update Demo should do a rolling update of a replication controller [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-cli] Kubectl client Update Demo should scale a replication controller [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-instrumentation] Cadvisor should be healthy on every node. [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-instrumentation] MetricsGrabber should grab all metrics from API server. [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-instrumentation] MetricsGrabber should grab all metrics from a ControllerManager. [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-instrumentation] MetricsGrabber should grab all metrics from a Kubelet. [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-instrumentation] MetricsGrabber should grab all metrics from a Scheduler. [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-network] DNS should provide /etc/hosts entries for the cluster [LinuxOnly] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-network] DNS should provide DNS for ExternalName services [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-network] DNS should provide DNS for pods for Hostname [LinuxOnly] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-network] DNS should provide DNS for pods for Subdomain [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-network] DNS should provide DNS for services [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-network] DNS should provide DNS for the cluster [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-network] DNS should resolve DNS of partial qualified names for services [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-network] DNS should resolve DNS of partial qualified names for the cluster [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-network] DNS should support configurable pod DNS nameservers [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-network] DNS should support configurable pod resolv.conf [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should allow ingress access from updated namespace [Feature:NetworkPolicy] [Suite:openshift/conformance/parallel] [Suite:k8s] [Skipped:Network/OpenShiftSDN/Multitenant]
openshift-tests [sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should enforce multiple ingress policies with ingress allow-all policy taking precedence [Feature:NetworkPolicy] [Suite:openshift/conformance/parallel] [Suite:k8s] [Skipped:Network/OpenShiftSDN/Multitenant]
openshift-tests [sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should enforce multiple, stacked policies with overlapping podSelectors [Feature:NetworkPolicy] [Suite:openshift/conformance/parallel] [Suite:k8s] [Skipped:Network/OpenShiftSDN/Multitenant]
openshift-tests [sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should enforce policy based on NamespaceSelector with MatchExpressions[Feature:NetworkPolicy] [Suite:openshift/conformance/parallel] [Suite:k8s] [Skipped:Network/OpenShiftSDN/Multitenant]
openshift-tests [sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should enforce policy based on PodSelector and NamespaceSelector [Feature:NetworkPolicy] [Suite:openshift/conformance/parallel] [Suite:k8s] [Skipped:Network/OpenShiftSDN/Multitenant]
openshift-tests [sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should enforce policy based on PodSelector or NamespaceSelector [Feature:NetworkPolicy] [Suite:openshift/conformance/parallel] [Suite:k8s] [Skipped:Network/OpenShiftSDN/Multitenant]
openshift-tests [sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should enforce policy based on PodSelector with MatchExpressions[Feature:NetworkPolicy] [Suite:openshift/conformance/parallel] [Suite:k8s] [Skipped:Network/OpenShiftSDN/Multitenant]
openshift-tests [sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should enforce policy based on Ports [Feature:NetworkPolicy] [Suite:openshift/conformance/parallel] [Suite:k8s] [Skipped:Network/OpenShiftSDN/Multitenant]
openshift-tests [sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should enforce policy to allow traffic from pods within server namespace based on PodSelector [Feature:NetworkPolicy] [Suite:openshift/conformance/parallel] [Suite:k8s] [Skipped:Network/OpenShiftSDN/Multitenant]
openshift-tests [sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should enforce policy to allow traffic only from a different namespace, based on NamespaceSelector [Feature:NetworkPolicy] [Suite:openshift/conformance/parallel] [Suite:k8s] [Skipped:Network/OpenShiftSDN/Multitenant]
openshift-tests [sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should enforce policy to allow traffic only from a pod in a different namespace based on PodSelector and NamespaceSelector [Feature:NetworkPolicy] [Suite:openshift/conformance/parallel] [Suite:k8s] [Skipped:Network/OpenShiftSDN/Multitenant]
openshift-tests [sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should enforce updated policy [Feature:NetworkPolicy] [Suite:openshift/conformance/parallel] [Suite:k8s] [Skipped:Network/OpenShiftSDN/Multitenant]
openshift-tests [sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should stop enforcing policies after they are deleted [Feature:NetworkPolicy] [Suite:openshift/conformance/parallel] [Suite:k8s] [Skipped:Network/OpenShiftSDN/Multitenant]
openshift-tests [sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should support a 'default-deny' policy [Feature:NetworkPolicy] [Suite:openshift/conformance/parallel] [Suite:k8s] [Skipped:Network/OpenShiftSDN/Multitenant]
openshift-tests [sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should support allow-all policy [Feature:NetworkPolicy] [Suite:openshift/conformance/parallel] [Suite:k8s] [Skipped:Network/OpenShiftSDN/Multitenant]
openshift-tests [sig-network] Networking Granular Checks: Pods should function for intra-pod communication: http [LinuxOnly] [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-network] Networking Granular Checks: Pods should function for intra-pod communication: udp [LinuxOnly] [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-network] Networking Granular Checks: Pods should function for node-pod communication: http [LinuxOnly] [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-network] Networking Granular Checks: Pods should function for node-pod communication: udp [LinuxOnly] [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-network] Proxy version v1 should proxy logs on node using proxy subresource [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-network] Proxy version v1 should proxy logs on node with explicit kubelet port using proxy subresource [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-network] Proxy version v1 should proxy through a service and a pod [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-network] Services should be able to change the type from ClusterIP to ExternalName [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-network] Services should be able to change the type from ExternalName to ClusterIP [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-network] Services should be able to change the type from ExternalName to NodePort [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-network] Services should be able to update NodePorts with two same port numbers but different protocols [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-network] Services should check NodePort out-of-range [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-network] Services should create endpoints for unready pods [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-network] Services should preserve source pod IP for traffic thru service cluster IP [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s] [Skipped:Network/OpenShiftSDN/Multitenant]
openshift-tests [sig-network] Services should prevent NodePort collisions [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-network] Services should provide secure master service [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-network] Services should release NodePorts on delete [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-network] Services should serve a basic endpoint from pods [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-network] Services should serve multiport endpoints from pods [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-network] Services should use same NodePort with same port but different protocols [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-node] ConfigMap should be consumable via environment variable [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-node] ConfigMap should be consumable via the environment [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-node] ConfigMap should fail to create ConfigMap with empty key [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-node] ConfigMap should patch ConfigMap successfully [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-node] Downward API should provide container's limits.cpu/memory and requests.cpu/memory as env vars [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-node] Downward API should provide default limits.cpu/memory from node allocatable [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-node] Downward API should provide host IP and pod IP as an env var if pod uses host network [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-node] Downward API should provide host IP as an env var [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-node] Downward API should provide pod UID as env vars [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-node] Downward API should provide pod name, namespace and IP address as env vars [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
openshift-tests [sig-scheduling] LimitRange should create a LimitRange with defaults and ensure pod has those defaults applied. [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-storage] CSI mock volume CSI Volume expansion should expand volume by restarting pod if attach=off, nodeExpansion=on [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-storage] CSI mock volume CSI Volume expansion should expand volume by restarting pod if attach=on, nodeExpansion=on [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-storage] CSI mock volume CSI Volume expansion should expand volume without restarting pod if nodeExpansion=off [Suite:openshift/conformance/parallel] [Suite:k8s]
openshift-tests [sig-storage] CSI mock volume C