ResultFAILURE
Tests 6 failed / 865 succeeded
Started2019-11-08 06:07
Elapsed1h48m
Work namespaceci-op-yxjvhwj8
pod4.3.0-0.nightly-2019-11-08-060454-azure-ovn

Test Failures

openshift-tests Monitor cluster while tests execute 37m21s

go run hack/e2e.go -v -test --test_args='--ginkgo.focus=openshift\-tests\sMonitor\scluster\swhile\stests\sexecute$'
1 error level events were detected during this test run:

Nov 08 07:00:29.075 E ns/default pod/recycler-for-nfs-6qww9 node/ci-op-yxjvhwj8-bfca5-ddv8x-worker-centralus2-mc7fv pod failed (DeadlineExceeded): Pod was active on the node longer than the specified deadline

				
				Click to see stdout/stderrfrom junit_e2e_20191108-073136.xml

Find failed mentions in log files

openshift-tests [Feature:Platform] Managed cluster should have no crashlooping pods in core namespaces over two minutes [Suite:openshift/conformance/parallel] 2m6s

go run hack/e2e.go -v -test --test_args='--ginkgo.focus=openshift\-tests\s\[Feature\:Platform\]\sManaged\scluster\sshould\shave\sno\scrashlooping\spods\sin\score\snamespaces\sover\stwo\sminutes\s\[Suite\:openshift\/conformance\/parallel\]$'
fail [github.com/openshift/origin/test/extended/operators/cluster.go:122]: Expected
    <[]string | len:1, cap:1>: [
        "Pod openshift-kube-scheduler/revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0 was pending entire time: unknown error",
    ]
to be empty
				
				Click to see stdout/stderrfrom junit_e2e_20191108-073136.xml

Filter through log files

openshift-tests [Feature:ProjectAPI] TestProjectWatch should succeed [Suite:openshift/conformance/parallel] 1m9s

go run hack/e2e.go -v -test --test_args='--ginkgo.focus=openshift\-tests\s\[Feature\:ProjectAPI\]\s\sTestProjectWatch\sshould\ssucceed\s\[Suite\:openshift\/conformance\/parallel\]$'
fail [github.com/openshift/origin/test/extended/project/project.go:233]: timeout: e2e-test-project-api-4hbsz
				
				Click to see stdout/stderrfrom junit_e2e_20191108-073136.xml

Filter through log files

openshift-tests [Feature:Prometheus][Conformance] Prometheus when installed on the cluster shouldn't report any alerts in firing state apart from Watchdog [Suite:openshift/conformance/parallel/minimal] 15m0s

go run hack/e2e.go -v -test --test_args='--ginkgo.focus=openshift\-tests\s\[Feature\:Prometheus\]\[Conformance\]\sPrometheus\swhen\sinstalled\son\sthe\scluster\sshouldn\'t\sreport\sany\salerts\sin\sfiring\sstate\sapart\sfrom\sWatchdog\s\[Suite\:openshift\/conformance\/parallel\/minimal\]$'
Nov  8 07:21:26.494: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"'
Nov  8 07:21:28.260: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n"
Nov  8 07:21:28.260: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197688.23,\"1\"]}]}}"
promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197688.23]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1
Nov  8 07:21:29.261: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"'
Nov  8 07:21:33.363: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n"
Nov  8 07:21:33.363: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197693.322,\"1\"]}]}}"
promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197693.322]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1
Nov  8 07:21:34.364: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"'
Nov  8 07:21:37.636: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n"
Nov  8 07:21:37.636: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197697.593,\"1\"]}]}}"
promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197697.593]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1
Nov  8 07:21:38.636: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"'
Nov  8 07:21:48.651: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n"
Nov  8 07:21:48.651: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197708.609,\"1\"]}]}}"
promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197708.609]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1
Nov  8 07:21:49.651: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"'
Nov  8 07:21:58.903: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n"
Nov  8 07:21:58.904: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197718.862,\"1\"]}]}}"
promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197718.862]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1
Nov  8 07:21:59.904: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"'
Nov  8 07:22:01.675: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n"
Nov  8 07:22:01.675: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197721.633,\"1\"]}]}}"
promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197721.633]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1
Nov  8 07:22:02.675: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"'
Nov  8 07:22:04.140: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n"
Nov  8 07:22:04.140: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197724.097,\"1\"]}]}}"
promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197724.097]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1
Nov  8 07:22:05.141: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"'
Nov  8 07:22:08.349: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n"
Nov  8 07:22:08.349: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197728.309,\"1\"]}]}}"
promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197728.309]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1
Nov  8 07:22:09.349: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"'
Nov  8 07:22:13.520: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n"
Nov  8 07:22:13.520: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197733.481,\"1\"]}]}}"
promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197733.481]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1
Nov  8 07:22:14.521: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"'
Nov  8 07:22:16.064: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n"
Nov  8 07:22:16.064: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197736.021,\"1\"]}]}}"
promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197736.021]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1
Nov  8 07:22:17.065: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"'
Nov  8 07:22:18.189: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n"
Nov  8 07:22:18.189: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197738.154,\"1\"]}]}}"
promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197738.154]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1
Nov  8 07:22:19.190: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"'
Nov  8 07:22:20.282: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n"
Nov  8 07:22:20.282: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197740.25,\"1\"]}]}}"
promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197740.25]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1
Nov  8 07:22:21.282: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"'
Nov  8 07:22:22.613: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n"
Nov  8 07:22:22.614: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197742.582,\"1\"]}]}}"
promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197742.582]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1
Nov  8 07:22:23.614: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"'
Nov  8 07:22:24.761: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n"
Nov  8 07:22:24.761: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197744.726,\"1\"]}]}}"
promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197744.726]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1
Nov  8 07:22:25.761: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"'
Nov  8 07:22:27.221: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n"
Nov  8 07:22:27.221: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197747.18,\"1\"]}]}}"
promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197747.18]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1
Nov  8 07:22:28.223: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"'
Nov  8 07:22:29.398: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n"
Nov  8 07:22:29.398: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197749.364,\"1\"]}]}}"
promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197749.364]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1
Nov  8 07:22:30.398: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"'
Nov  8 07:22:31.456: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n"
Nov  8 07:22:31.456: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197751.425,\"1\"]}]}}"
promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197751.425]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1
Nov  8 07:22:32.457: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"'
Nov  8 07:22:33.642: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n"
Nov  8 07:22:33.642: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197753.611,\"1\"]}]}}"
promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197753.611]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1
Nov  8 07:22:34.642: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"'
Nov  8 07:22:35.870: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n"
Nov  8 07:22:35.870: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197755.839,\"1\"]}]}}"
promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197755.839]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1
Nov  8 07:22:36.871: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"'
Nov  8 07:22:38.129: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n"
Nov  8 07:22:38.129: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197758.097,\"1\"]}]}}"
promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197758.097]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1
Nov  8 07:22:39.129: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"'
Nov  8 07:22:44.710: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n"
Nov  8 07:22:44.710: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197764.574,\"1\"]}]}}"
promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197764.574]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1
Nov  8 07:22:45.711: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"'
Nov  8 07:22:47.240: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n"
Nov  8 07:22:47.240: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197767.176,\"1\"]}]}}"
promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197767.176]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1
Nov  8 07:22:48.240: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"'
Nov  8 07:22:49.488: INFO: stderr: "+ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1'\n"
Nov  8 07:22:49.488: INFO: stdout: "{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubePodNotReady\",\"alertstate\":\"firing\",\"namespace\":\"openshift-kube-scheduler\",\"pod\":\"revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0\",\"severity\":\"critical\"},\"value\":[1573197769.457,\"1\"]}]}}"
promQL query: ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1 had reported incorrect results: ALERTS{alertname="KubePodNotReady", alertstate="firing", namespace="openshift-kube-scheduler", pod="revision-pruner-4-ci-op-yxjvhwj8-bfca5-ddv8x-master-0", severity="critical"} => 1 @[1573197769.457]STEP: perform prometheus metric query ALERTS{alertname!~"Watchdog|ClusterMachineApproverDown|UsingDeprecatedAPIExtensionsV1Beta1",alertstate="firing"} >= 1
Nov  8 07:22:50.489: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-yxjvhwj8-bfca5.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-test-prometheus-fgfz6 execpoddkrl2 -- /bin/sh -x -c curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndHWUI4ZHhSM0d6Yk5IdU5kLW5YUEdlMXAyZUNtZlRHdEFXUFZhdFJoYUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWFkYXB0ZXItdG9rZW4tNWtwYjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cy1hZGFwdGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWFmMzMwOTgtZWQyZS00OGM4LTkwNjgtNzk5NWE1ZjJiZTBhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW5zaGlmdC1tb25pdG9yaW5nOnByb21ldGhldXMtYWRhcHRlciJ9.rtHFyaqpAUjW0cMBO2Z68l1DHsyo9wjiIKCtoqT9bNj7rTllm906-lalmRkVqTuqdkiFWOGTWDQ6cp2cURyDQF-7ujXcL3gSashOA7EolUJp-YroGFZTNCT_omdSw--aGcBBkwrkz0fjqKdZzAOfV6dWFZ3LP2FGpg6YVBpqXn5UkGQT6zE0C7gftpoWxweo7kbe9NFm2k-zWbPKAh8CXntoVE1oiVBioOiwLQ7ypAEEsl0At7G0R4fUrqu3xbGZRyHoQBQ91pky1MaDw1HZP3XlmWJnseCrBfAGvPlRFidfA36Lx1iVa_CTTkR5NkWj0uChUrmNOszLm7FuVSLsAA' "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=ALERTS%7Balertname%21~%22Watchdog%7CClusterMachineApproverDown%7CUsingDeprecatedAPIExtensionsV1Beta1%22%2Calertstate%3D%22firing%22%7D+%3E%3D+1"'

---------------------------------------------------------
Received interrupt.  Running AfterSuite...
^C again to terminate immediately
Nov  8 07:22:50.914: INFO: Running AfterSuite actions on all nodes
Nov  8 07:22:50.915: INFO: Waiting up to 3m0s for all (but 100) nodes to be ready
STEP: Destroying namespace "e2e-test-prometheus-fgfz6" for this suite.
Nov  8 07:22:51.229: INFO: Running AfterSuite actions on node 1
				
				Click to see stdout/stderrfrom junit_e2e_20191108-073136.xml

Filter through log files

openshift-tests [sig-storage] Projected secret should be consumable from pods in volume with defaultMode set [LinuxOnly] [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s] 52s

go run hack/e2e.go -v -test --test_args='--ginkgo.focus=openshift\-tests\s\[sig\-storage\]\sProjected\ssecret\sshould\sbe\sconsumable\sfrom\spods\sin\svolume\swith\sdefaultMode\sset\s\[LinuxOnly\]\s\[NodeConformance\]\s\[Conformance\]\s\[Suite\:openshift\/conformance\/parallel\/minimal\]\s\[Suite\:k8s\]$'
fail [k8s.io/kubernetes/test/e2e/framework/util.go:1486]: Unexpected error:
    <*errors.errorString | 0xc003826240>: {
        s: "expected pod \"pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d\" success: pod \"pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d\" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-08 06:47:43 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-08 06:47:43 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [projected-secret-volume-test]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-08 06:47:43 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [projected-secret-volume-test]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-08 06:47:43 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.0.32.6 PodIP:10.128.2.23 PodIPs:[{IP:10.128.2.23}] StartTime:2019-11-08 06:47:43 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:projected-secret-volume-test State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:-1,Signal:0,Reason:OOMKilled,Message:,StartedAt:2019-11-08 06:48:20 +0000 UTC,FinishedAt:1754-08-30 22:43:41 +0000 UTC,ContainerID:cri-o://3f46cfd3d7b4bd3fa05aa78fa3c8c8dd619de04898f81c0052154d728dc2e7e2,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:gcr.io/kubernetes-e2e-test-images/mounttest:1.0 ImageID:gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 ContainerID:cri-o://3f46cfd3d7b4bd3fa05aa78fa3c8c8dd619de04898f81c0052154d728dc2e7e2 Started:<nil>}] QOSClass:BestEffort EphemeralContainerStatuses:[]}",
    }
    expected pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d" success: pod "pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-08 06:47:43 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-08 06:47:43 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [projected-secret-volume-test]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-08 06:47:43 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [projected-secret-volume-test]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-08 06:47:43 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.0.32.6 PodIP:10.128.2.23 PodIPs:[{IP:10.128.2.23}] StartTime:2019-11-08 06:47:43 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:projected-secret-volume-test State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:-1,Signal:0,Reason:OOMKilled,Message:,StartedAt:2019-11-08 06:48:20 +0000 UTC,FinishedAt:1754-08-30 22:43:41 +0000 UTC,ContainerID:cri-o://3f46cfd3d7b4bd3fa05aa78fa3c8c8dd619de04898f81c0052154d728dc2e7e2,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:gcr.io/kubernetes-e2e-test-images/mounttest:1.0 ImageID:gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 ContainerID:cri-o://3f46cfd3d7b4bd3fa05aa78fa3c8c8dd619de04898f81c0052154d728dc2e7e2 Started:<nil>}] QOSClass:BestEffort EphemeralContainerStatuses:[]}
occurred
				
				Click to see stdout/stderrfrom junit_e2e_20191108-073136.xml

Find pod-projected-secrets-eee13741-6143-4e57-b9a2-fff1eedde88d mentions in log files

operator Run template e2e-azure - e2e-azure-ovn-kubernetes container test 45m22s

go run hack/e2e.go -v -test --test_args='--ginkgo.focus=operator\sRun\stemplate\se2e\-azure\s\-\se2e\-azure\-ovn\-kubernetes\scontainer\stest$'
when installed on the cluster shouldn't report any alerts in firing state apart from Watchdog [Suite:openshift/conformance/parallel/minimal]" failed

W1108 07:24:10.995684     186 reflector.go:299] github.com/openshift/origin/pkg/monitor/operator.go:126: watch of *v1.ClusterOperator ended with: too old resource version: 135007 (135191)
W1108 07:24:16.943253     186 reflector.go:299] github.com/openshift/origin/pkg/monitor/operator.go:126: watch of *v1.ClusterOperator ended with: too old resource version: 135191 (135274)
W1108 07:27:22.508587     186 reflector.go:299] github.com/openshift/origin/pkg/monitor/operator.go:126: watch of *v1.ClusterOperator ended with: too old resource version: 135849 (136109)
W1108 07:27:28.421147     186 reflector.go:299] github.com/openshift/origin/pkg/monitor/operator.go:126: watch of *v1.ClusterOperator ended with: too old resource version: 136112 (136125)
W1108 07:30:33.945488     186 reflector.go:299] github.com/openshift/origin/pkg/monitor/operator.go:126: watch of *v1.ClusterOperator ended with: too old resource version: 136873 (136979)
W1108 07:30:39.848272     186 reflector.go:299] github.com/openshift/origin/pkg/monitor/operator.go:126: watch of *v1.ClusterOperator ended with: too old resource version: 136979 (137003)
Flaky tests:

[Feature:ProjectAPI]  TestProjectWatch should succeed [Suite:openshift/conformance/parallel]
[sig-storage] Projected secret should be consumable from pods in volume with defaultMode set [LinuxOnly] [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]

Failing tests:

[Feature:Platform] Managed cluster should have no crashlooping pods in core namespaces over two minutes [Suite:openshift/conformance/parallel]
[Feature:Prometheus][Conformance] Prometheus when installed on the cluster shouldn't report any alerts in firing state apart from Watchdog [Suite:openshift/conformance/parallel/minimal]

Writing JUnit report to /tmp/artifacts/junit/junit_e2e_20191108-073136.xml

error: 4 fail, 859 pass, 1217 skip (37m21s)

				from junit_operator.xml

Filter through log files

Show 865 Passed Tests