[00:15:11] FIRING: MaxConntrack: Max conntrack at 80.13% on cloudvirt1050:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:16:11] FIRING: [2x] SystemdUnitDown: The service unit wikitech_run_jobs.service is in failed status on host cloudweb1003. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [00:25:10] RESOLVED: MaxConntrack: Max conntrack at 80.1% on cloudvirt1050:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:36:40] FIRING: MaxConntrack: Max conntrack at 80.68% on cloudvirt1050:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:41:11] RESOLVED: [2x] SystemdUnitDown: The service unit wikitech_run_jobs.service is in failed status on host cloudweb1003. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [00:56:40] RESOLVED: MaxConntrack: Max conntrack at 80.35% on cloudvirt1050:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [03:06:06] 10Tool-techcontribs: tech-contribs: Show names and not IDs of new Cloud VPS projects - https://phabricator.wikimedia.org/T373730#10108670 (10Chlod) 05Open→03Resolved a:03Chlod Fixed with the latest deploy. Thanks for reporting this! (and working on T373742 :D) [03:10:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-22 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [03:51:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-17 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [05:06:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-17 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [06:12:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-5:30000 has failed probes (http_admin_beta_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:20:56] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:25:56] FIRING: [2x] ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:30:56] RESOLVED: [2x] ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:32:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-5:30000 has failed probes (http_admin_beta_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:33:52] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/8 [06:33:58] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/7 [06:34:01] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/67 [06:34:10] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/8 [06:34:12] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/67 [06:34:17] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/13 [06:34:38] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/10 [06:34:45] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/87 [06:34:50] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/13 [06:34:53] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/87 [06:34:54] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/88 [06:34:54] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/10 [06:34:55] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/88 [06:35:02] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/53 [06:35:11] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/122 [06:35:18] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/122 [06:35:19] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/53 [06:35:21] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/60 [06:35:29] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/123 [06:35:34] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/61 [06:35:46] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/123 [06:35:47] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/60 [06:35:49] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/39 [06:35:49] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/28 [06:35:49] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/12 [06:35:50] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/17 [06:35:55] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/61 [06:35:58] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/39 [06:36:00] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/110 [06:36:04] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/12 [06:36:08] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/17 [06:36:12] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/40 [06:36:17] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/187 [06:36:21] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/28 [06:36:25] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/33 [06:36:29] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/110 [06:36:33] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/29 [06:36:37] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/40 [06:36:41] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/187 [06:36:45] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/33 [06:36:49] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/29 [07:00:56] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:25:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-6 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [07:30:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-6 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [08:20:56] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:29:00] (03CR) 10Jean-Frédéric: "> comparing this to the old update_monuments.sh the new harvester doesn't load any configuration changes into the source tables (line 16-3" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1069284 (https://phabricator.wikimedia.org/T319787) (owner: 10Jean-Frédéric) [08:30:56] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:39:29] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance cloudinfra-idp-1 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [10:42:07] (03CR) 10Lokal Profil: "found one comment that needs changing. other than that good to go. happy to leave the skipped bits of update_monuments.sh to a later patc" (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1069284 (https://phabricator.wikimedia.org/T319787) (owner: 10Jean-Frédéric) [10:43:31] (03close) 10lucaswerkmeister: Draft: Use printf to purge images in a single command [toolforge-repos/logos-purge] - 10https://gitlab.wikimedia.org/toolforge-repos/logos-purge/-/merge_requests/2 [13:20:56] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:35:56] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:50:56] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:00:56] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:19:32] 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T373767 (10Significa_liberdade) 03NEW [16:29:22] 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T373767#10108954 (10Significa_liberdade) Seems to be back up and running. Sorry! [16:45:42] 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T373767#10108958 (10Pppery) 05Open→03Invalid [18:28:07] (03CR) 10Lokal Profil: [C:03+2] Rewrite jobs shell scripts for Toolforge Jobs framework (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1069284 (https://phabricator.wikimedia.org/T319787) (owner: 10Jean-Frédéric) [18:30:01] (03Merged) 10jenkins-bot: Rewrite jobs shell scripts for Toolforge Jobs framework [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1069284 (https://phabricator.wikimedia.org/T319787) (owner: 10Jean-Frédéric) [19:34:10] (03PS1) 10Jean-Frédéric: Remove `cd $PWD` when spawning jobs in deploy script [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1069708 [20:20:56] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:30:56] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:51:30] (03CR) 10Lokal Profil: [C:03+2] Remove `cd $PWD` when spawning jobs in deploy script [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1069708 (owner: 10Jean-Frédéric) [22:53:11] (03Merged) 10jenkins-bot: Remove `cd $PWD` when spawning jobs in deploy script [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1069708 (owner: 10Jean-Frédéric) [23:20:56] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks