[00:02:34] 10PAWS: New upstream release for Wikimedia Commons Extension for OpenRefine - https://phabricator.wikimedia.org/T369326#9955376 (10LibUp-bot) [00:05:57] (03update) 10raymond-ndibe: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209) [00:07:14] (03update) 10raymond-ndibe: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209) [00:23:10] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [00:25:50] (03update) 10raymond-ndibe: [jobs-cli] move jobs load to backend [repos/cloud/toolforge/jobs-cli] (refactor_job_validation) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/44 (https://phabricator.wikimedia.org/T366209) [00:45:05] (03update) 10raymond-ndibe: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209) [00:45:42] (03update) 10raymond-ndibe: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209) [00:48:26] (03update) 10raymond-ndibe: [jobs-cli] move jobs load to backend [repos/cloud/toolforge/jobs-cli] (refactor_job_validation) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/44 (https://phabricator.wikimedia.org/T366209) [01:35:44] (03update) 10raymond-ndibe: [jobs-api] fix issues in openapi schema [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/96 [01:37:29] (03merge) 10raymond-ndibe: [jobs-api] fix issues in openapi schema [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/96 [01:39:20] (03update) 10raymond-ndibe: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209) [01:39:43] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: jobs-api: bump to 0.0.314-20240705013740-a02a80c3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/389 [01:41:14] !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [01:41:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [01:41:28] !log raymond@ubuntu toolsbeta END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-api [01:41:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [01:41:40] !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [01:41:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [01:42:30] !log raymond@ubuntu toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [01:42:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [01:46:45] !log raymond@ubuntu tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [01:46:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [01:47:39] !log raymond@ubuntu tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [01:47:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [01:47:51] (03approved) 10raymond-ndibe: jobs-api: bump to 0.0.314-20240705013740-a02a80c3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/389 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [01:47:54] (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.314-20240705013740-a02a80c3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/389 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [01:48:47] (03approved) 10raymond-ndibe: dev: update dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/7 (https://phabricator.wikimedia.org/T329671) (owner: 10sstefanova) [01:57:07] (03approved) 10raymond-ndibe: ci: add check-openapi-version-bump pre-commit hook [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/37 (https://phabricator.wikimedia.org/T356972) (owner: 10dcaro) [02:13:22] (03update) 10raymond-ndibe: [jobs-api] create seperate api.py and move flask things there [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/91 (https://phabricator.wikimedia.org/T359804) [02:17:20] (03update) 10raymond-ndibe: [jobs-api] create seperate api.py and move flask things there [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/91 (https://phabricator.wikimedia.org/T359804) [02:50:57] FIRING: CloudVPSDesignateLeaks: Detected 6 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:00:57] (03update) 10raymond-ndibe: [jobs-api] further group code into api, business and runtime logic [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/91 (https://phabricator.wikimedia.org/T359804) [03:04:27] (03update) 10raymond-ndibe: Draft: [jobs-api] further group code into api, business and runtime logic [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/91 (https://phabricator.wikimedia.org/T359804) [03:56:16] FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [04:23:25] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [05:51:55] (03update) 10sstefanova: dev: update dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/7 (https://phabricator.wikimedia.org/T329671) [05:56:44] 06cloud-services-team, 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.25 - https://phabricator.wikimedia.org/T329671#9955652 (10Slst2020) [06:06:28] (03open) 10sstefanova: build: update dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/12 (https://phabricator.wikimedia.org/T329671) [06:32:10] (03open) 10sstefanova: build: update dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/6 (https://phabricator.wikimedia.org/T329671) [06:36:39] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolsbeta-test-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:41:39] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolsbeta-test-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:50:57] FIRING: CloudVPSDesignateLeaks: Detected 6 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:53:24] (03update) 10sstefanova: build: update dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/12 (https://phabricator.wikimedia.org/T329671) [07:53:53] (03update) 10sstefanova: build: update dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/6 (https://phabricator.wikimedia.org/T329671) [07:56:16] FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [08:15:06] (03update) 10sstefanova: api: rename params for clarity [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/100 [08:23:25] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [08:32:10] 06cloud-services-team, 10Toolforge: toolforge: identify and cache in our container registry all kyverno images - https://phabricator.wikimedia.org/T364113#9955802 (10aborrero) 05Resolved→03Open reopen because we are missing this one: {F56234512} reported by @Slst2020 [08:38:49] (03update) 10sstefanova: dev: update dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/7 (https://phabricator.wikimedia.org/T329671) [08:39:00] (03merge) 10sstefanova: dev: update dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/7 (https://phabricator.wikimedia.org/T329671) [08:41:48] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: registry-admission: bump to 0.0.44-20240705083909-fbafef28 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/390 (https://phabricator.wikimedia.org/T329671) [08:48:42] 10Tools, 10Wikidata, 10wmde-wikidata-tech, 10Wikidata Dev Team (Wikidata.org Slice): [GENERAL] Deprecate connecting senses prototype - https://phabricator.wikimedia.org/T351829#9955913 (10ArthurTaylor) The tool is offline now: {F56234919} and the project is disabled: {F56234930} Should get deleted perm... [08:49:14] 10Tools, 10Wikidata, 10wmde-wikidata-tech, 10Wikidata Dev Team (Wikidata.org Slice): [GENERAL] Deprecate connecting senses prototype - https://phabricator.wikimedia.org/T351829#9955918 (10ArthurTaylor) a:05ArthurTaylor→03None [09:52:34] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 5.992% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [10:11:36] (03merge) 10aborrero: deployment: drop PSP reference [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/34 (https://phabricator.wikimedia.org/T368142) [10:12:56] FIRING: SystemdUnitDown: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [10:15:36] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: envvars-api: bump to 0.0.52-20240705101149-aa9da2fa [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/391 (https://phabricator.wikimedia.org/T368142) [10:22:56] RESOLVED: SystemdUnitDown: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [10:46:55] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [10:47:01] (03PS1) 10Btullis: Add dummy keytabs for analytics-wmde on stat servers. [labs/private] - 10https://gerrit.wikimedia.org/r/1052282 (https://phabricator.wikimedia.org/T340648) [10:47:24] (03CR) 10Btullis: [V:03+2 C:03+2] Add dummy keytabs for analytics-wmde on stat servers. [labs/private] - 10https://gerrit.wikimedia.org/r/1052282 (https://phabricator.wikimedia.org/T340648) (owner: 10Btullis) [10:50:57] FIRING: CloudVPSDesignateLeaks: Detected 6 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:51:48] (03open) 10sstefanova: wmcs-k8s-metrics: bump kube-state-metrics version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/392 (https://phabricator.wikimedia.org/T329671) [10:59:15] (03approved) 10aborrero: build: update dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/6 (https://phabricator.wikimedia.org/T329671) (owner: 10sstefanova) [11:00:10] (03approved) 10aborrero: build: update dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/12 (https://phabricator.wikimedia.org/T329671) (owner: 10sstefanova) [11:10:03] (03update) 10sstefanova: build: update dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/12 (https://phabricator.wikimedia.org/T329671) [11:10:12] (03merge) 10sstefanova: build: update dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/12 (https://phabricator.wikimedia.org/T329671) [11:13:07] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: volume-admission: bump to 0.0.50-20240705111023-80cfa300 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/393 (https://phabricator.wikimedia.org/T329671) [11:18:13] (03PS1) 10Arturo Borrero Gonzalez: toolforge: k8s: add generic image copy_to_registry cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1052286 [11:20:03] (03CR) 10Slavina Stefanova: [C:03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1052286 (owner: 10Arturo Borrero Gonzalez) [11:24:05] (03PS2) 10Arturo Borrero Gonzalez: toolforge: k8s: add generic image copy_to_registry cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1052286 [11:28:46] !log arturo@nostromo tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4 [11:28:46] !log arturo@nostromo tools Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4 [11:28:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:28:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:29:36] !log arturo@nostromo tools END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4 [11:29:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:30:37] (03PS3) 10Arturo Borrero Gonzalez: toolforge: k8s: add generic image copy_to_registry cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1052286 [11:33:41] (03open) 10sstefanova: kind: upgrade to k8s 1.25 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/161 (https://phabricator.wikimedia.org/T369165) [11:33:42] (03CR) 10Arturo Borrero Gonzalez: [C:03+2] toolforge: k8s: add generic image copy_to_registry cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1052286 (owner: 10Arturo Borrero Gonzalez) [11:46:23] (03update) 10aborrero: kyverno: use more images from our registry [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/394 (https://phabricator.wikimedia.org/T364113) [11:46:26] (03open) 10aborrero: kyverno: use more images from our registry [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/394 (https://phabricator.wikimedia.org/T364113) [11:50:38] (03update) 10aborrero: kyverno: use more images from our registry [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/394 (https://phabricator.wikimedia.org/T364113) [11:56:16] FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [12:00:08] (03update) 10aborrero: kyverno: use more images from our registry [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/394 (https://phabricator.wikimedia.org/T364113) [12:06:23] 10Cloud-VPS (Debian Buster Deprecation), 10Wikispeech: Cloud VPS "wikispeech" project Buster deprecation - https://phabricator.wikimedia.org/T367565#9956517 (10Sebastian_Berlin-WMSE) Instance has been removed. [12:07:46] (03update) 10aborrero: k8s: deploy registry-admission [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/162 [12:07:49] (03open) 10aborrero: k8s: deploy registry-admission [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/162 [12:10:03] (03approved) 10sstefanova: k8s: deploy registry-admission [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/162 (owner: 10aborrero) [12:15:42] (03update) 10raymond-ndibe: [lima-kilo] update bookworm arm64 image [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/155 [12:17:08] 10Toolforge: [lima-kilo] add the ingress-admission-controller - https://phabricator.wikimedia.org/T369355 (10Slst2020) 03NEW [12:17:11] (03merge) 10raymond-ndibe: [lima-kilo] update bookworm arm64 image [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/155 [12:23:28] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry [12:23:29] !log sstefanova@cloudcumin1001 tools Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.7.0 [12:23:38] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) [12:25:24] (03PS1) 10Arturo Borrero Gonzalez: kyverno: copy_images_to_registry: add bitnami/kubectl image [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1052296 (https://phabricator.wikimedia.org/T364113) [12:26:00] !log arturo@nostromo tools START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry [12:26:00] !log arturo@nostromo tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7 [12:26:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:26:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:26:16] !log arturo@nostromo tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7 [12:26:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:26:30] !log arturo@nostromo tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7 [12:26:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:26:44] !log arturo@nostromo tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7 [12:26:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:26:59] !log arturo@nostromo tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7 [12:27:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:27:14] !log arturo@nostromo tools END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) [12:27:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:27:59] !log arturo@nostromo tools START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry [12:27:59] !log arturo@nostromo tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7 [12:27:59] (03PS2) 10Arturo Borrero Gonzalez: kyverno: copy_images_to_registry: add bitnami/kubectl image [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1052296 (https://phabricator.wikimedia.org/T364113) [12:28:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:28:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:28:12] !log arturo@nostromo tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7 [12:28:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:28:32] !log arturo@nostromo tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7 [12:28:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:28:56] !log arturo@nostromo tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7 [12:28:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:29:10] !log arturo@nostromo tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7 [12:29:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:29:27] !log arturo@nostromo tools Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4 [12:29:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:29:46] !log arturo@nostromo tools END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) [12:29:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:30:23] (03CR) 10Arturo Borrero Gonzalez: [C:03+2] kyverno: copy_images_to_registry: add bitnami/kubectl image [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1052296 (https://phabricator.wikimedia.org/T364113) (owner: 10Arturo Borrero Gonzalez) [12:30:27] (03CR) 10Arturo Borrero Gonzalez: [V:03+2 C:03+2] kyverno: copy_images_to_registry: add bitnami/kubectl image [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1052296 (https://phabricator.wikimedia.org/T364113) (owner: 10Arturo Borrero Gonzalez) [12:30:35] (03CR) 10Arturo Borrero Gonzalez: [C:03+2] kyverno: copy_images_to_registry: add bitnami/kubectl image [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1052296 (https://phabricator.wikimedia.org/T364113) (owner: 10Arturo Borrero Gonzalez) [12:31:40] (03update) 10aborrero: k8s: deploy registry-admission [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/162 [12:32:35] (03open) 10raymond-ndibe: [lima-kilo] fix toolforge_deploy_mr restore [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/163 [12:33:24] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno [12:33:39] (03merge) 10aborrero: k8s: deploy registry-admission [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/162 [12:33:49] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno [12:34:20] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno [12:34:46] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno [12:35:09] 10Toolforge: [lima-kilo] add the ingress-admission-controller - https://phabricator.wikimedia.org/T369355#9956553 (10Slst2020) [12:35:30] (03merge) 10aborrero: kyverno: use more images from our registry [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/394 (https://phabricator.wikimedia.org/T364113) [12:40:25] (03open) 10aborrero: kyverno: settings: fix values for kubectl images [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/395 (https://phabricator.wikimedia.org/T364113) [12:41:27] (03update) 10sstefanova: kind: upgrade to k8s 1.25 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/161 (https://phabricator.wikimedia.org/T369165) [12:42:43] (03approved) 10raymond-ndibe: kyverno: settings: fix values for kubectl images [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/395 (https://phabricator.wikimedia.org/T364113) (owner: 10aborrero) [12:44:05] (03update) 10aborrero: kyverno: settings: fix values for kubectl images [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/395 (https://phabricator.wikimedia.org/T364113) [12:45:12] (03update) 10raymond-ndibe: kind: upgrade to k8s 1.25 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/161 (https://phabricator.wikimedia.org/T369165) (owner: 10sstefanova) [12:47:15] (03update) 10aborrero: kyverno: settings: fix values for kubectl images [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/395 (https://phabricator.wikimedia.org/T364113) [12:48:21] (03update) 10aborrero: kyverno: settings: fix values for kubectl images [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/395 (https://phabricator.wikimedia.org/T364113) [12:49:02] (03update) 10sstefanova: kind: upgrade to k8s 1.25 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/161 (https://phabricator.wikimedia.org/T369165) [12:50:11] (03update) 10sstefanova: kind: upgrade to k8s 1.25 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/161 (https://phabricator.wikimedia.org/T369165) [12:51:09] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno [12:51:37] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno [12:51:51] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno [12:52:19] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno [12:52:36] (03merge) 10aborrero: kyverno: settings: fix values for kubectl images [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/395 (https://phabricator.wikimedia.org/T364113) [13:11:07] (03update) 10sstefanova: wmcs-k8s-metrics: bump kube-state-metrics version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/392 (https://phabricator.wikimedia.org/T329671) [13:21:32] 06cloud-services-team, 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: toolforge: upgrade lima-kilo for kubernetes 1.25 - https://phabricator.wikimedia.org/T369165#9956662 (10Slst2020) 05Open→03In progress [13:22:38] 06cloud-services-team, 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.25 - https://phabricator.wikimedia.org/T329671#9956658 (10Slst2020) 05Open→03In progress [13:22:40] 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolforge: review k8s API usage by custom components for 1.25 upgrade - https://phabricator.wikimedia.org/T369164#9956677 (10Slst2020) [13:24:57] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#9956683 (10fgiunchedi) [13:25:35] 10Cloud-VPS (Debian Buster Deprecation), 10observability, 10Observability-Logging: Upgrade deployment-mwlog01.deployment-prep.eqiad1.wikimedia.cloud to bullseye or bookworm - https://phabricator.wikimedia.org/T369263#9956679 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi There is actually a mwlog0... [13:29:38] 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolforge: review k8s API usage by custom components for 1.25 upgrade - https://phabricator.wikimedia.org/T369164#9956688 (10Slst2020) builds-builder currently fails to deploy in lima-kilo with k8s 1.25. `deployment/chart/templates/tekton-pipelines.... [13:38:08] (03open) 10sstefanova: tekton: update apiVersion to autoscaling/v2 [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/50 (https://phabricator.wikimedia.org/T369164) [13:39:27] (03update) 10sstefanova: tekton: update apiVersion to autoscaling/v2 [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/50 (https://phabricator.wikimedia.org/T369164) [13:45:56] (03update) 10sstefanova: tekton: update apiVersion to autoscaling/v2 [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/50 (https://phabricator.wikimedia.org/T369164) [13:52:50] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 5.461% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [14:24:01] 10Data-Services: SQL function to recover the normal hostname, to install on Wiki Replica instances - https://phabricator.wikimedia.org/T344877#9956789 (10Ladsgroup) That's a nice trick. Feel free to add this to the existing views. [14:38:57] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#9956804 (10Andrew) [14:39:08] (03open) 10aborrero: deployment: remove PSP reference [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/8 (https://phabricator.wikimedia.org/T368142) [14:42:03] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: toolforge: identify and cache in our container registry all kyverno images - https://phabricator.wikimedia.org/T364113#9956817 (10aborrero) 05Open→03Resolved [14:43:33] 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolforge: get a working setup for ingress-nginx and webservices in lima-kilo - https://phabricator.wikimedia.org/T369363 (10aborrero) 03NEW [14:44:02] 06cloud-services-team, 10Cloud-VPS, 06Data-Persistence: Decommission clouddb2002-dev.codfw.wmnet - https://phabricator.wikimedia.org/T369308#9956821 (10Ladsgroup) Confirming what Manuel has said: This is needed as it powers https://labtestwikitech.wikimedia.org, it's quite a snowflake and I would appreciate... [14:45:29] 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolforge: integrate fourohfour as a custom component, rather than a normal tool - https://phabricator.wikimedia.org/T369364 (10aborrero) 03NEW [14:50:57] FIRING: CloudVPSDesignateLeaks: Detected 13 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:07:04] (03update) 10aborrero: kubernetes: add some basic kyverno alerts [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/17 (https://phabricator.wikimedia.org/T368515) [15:15:23] (03update) 10aborrero: kubernetes: add some basic kyverno alerts [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/17 (https://phabricator.wikimedia.org/T368515) [15:36:26] (03update) 10aborrero: kubernetes: add some basic kyverno alerts [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/17 (https://phabricator.wikimedia.org/T368515) [15:36:32] (03merge) 10aborrero: kubernetes: add some basic kyverno alerts [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/17 (https://phabricator.wikimedia.org/T368515) [15:37:17] 06cloud-services-team, 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: toolforge: kyverno: enable monitoring - https://phabricator.wikimedia.org/T368515#9956970 (10aborrero) [15:37:33] 06cloud-services-team, 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: toolforge: kyverno: enable monitoring - https://phabricator.wikimedia.org/T368515#9956971 (10aborrero) Expanded information at https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/Kyverno [15:37:39] 06cloud-services-team, 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: toolforge: kyverno: enable monitoring - https://phabricator.wikimedia.org/T368515#9956972 (10aborrero) 05In progress→03Resolved [15:49:45] FIRING: Toolforge Kyverno low policy resources: Toolforge Kyverno has low amount of policy resources - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/Toolforge_Kyverno_low_policy_resources - https://grafana-rw.wmcloud.org/d/kyverno/kyverno?orgId=1&var-DS_PROMETHEUS_KYVERNO=prometheus-tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforge+Kyverno+low+policy+resources [15:55:20] (03open) 10aborrero: kyverno: only deploy on tools for now [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/18 (https://phabricator.wikimedia.org/T368515) [15:55:27] (03update) 10aborrero: kyverno: only deploy on tools for now [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/18 (https://phabricator.wikimedia.org/T368515) [15:56:27] (03merge) 10aborrero: kyverno: only deploy on tools for now [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/18 (https://phabricator.wikimedia.org/T368515) [16:46:25] 10Tool-bridgebot, 10Tool-containers: Replace tool-bridgebot/znc container with tool-containers/bnc container - https://phabricator.wikimedia.org/T366970#9957130 (10bd808) 05Open→03In progress p:05Triage→03Medium a:03bd808 Added BNC envars: `lang=shell-session $ toolforge envvars create BNC_USER $(too... [16:52:42] 10Tool-dabfix, 06Indic-MediaWiki-Developers: Improve Footer Styling and Fix Position - https://phabricator.wikimedia.org/T369379 (10Gopavasanth) 03NEW [16:55:34] 10Tool-dabfix, 06Indic-MediaWiki-Developers: Fix Text Overflow - https://phabricator.wikimedia.org/T369380 (10Gopavasanth) 03NEW [16:59:52] FIRING: ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:01:57] 10Tool-bridgebot, 10Tool-containers: Replace tool-bridgebot/znc container with tool-containers/bnc container - https://phabricator.wikimedia.org/T366970#9957233 (10bd808) >>! In T366970#9957215, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-cloud), href=https://sal.toolforge.org/log/FJnW... [17:02:09] 10Toolforge-standards-committee (Maintainer needed), 10VideoCutTool: Co-maintainers needed for VideoCutTool. - https://phabricator.wikimedia.org/T237998#9957232 (10Reputation22) >>! In T237998#7327431, @Gopavasanth wrote: > Hi @Aklapper, I would say to let this ticket be open if any other folks wants to ta... [17:02:35] 10Toolforge-standards-committee (Maintainer needed), 10VideoCutTool: Co-maintainers needed for VideoCutTool. - https://phabricator.wikimedia.org/T237998#9957236 (10Reputation22) 05Open→03Resolved a:03Reputation22 [17:04:52] RESOLVED: ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:06:52] FIRING: ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:11:52] FIRING: [2x] ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:12:53] 10Tool-bridgebot, 10Tool-containers: Replace tool-bridgebot/znc container with tool-containers/bnc container - https://phabricator.wikimedia.org/T366970#9957271 (10bd808) 05In progress→03Resolved Docs updated: https://wikitech.wikimedia.org/w/index.php?title=Tool%3ABridgebot&diff=2203477&oldid=2196137 [17:16:25] 10Tool-bridgebot, 10Toolforge: bridgebot tool build service quota not going down - https://phabricator.wikimedia.org/T368317#9957311 (10bd808) 05Open→03In progress p:05Triage→03High a:03bd808 Let's start by deleting the unused build of https://gitlab.wikimedia.org/toolforge-repos/wikibugs2-znc: ` $ t... [17:16:52] RESOLVED: ProbeDown: Service tools-k8s-haproxy-6:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-6:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:40:32] 10Tool-bridgebot, 10Toolforge: bridgebot tool build service quota not going down - https://phabricator.wikimedia.org/T368317#9957359 (10bd808) Building to an alternate image name out of an abundance of caution found a weird bug: ` $ toolforge build start https://gitlab.wikimedia.org/toolforge-repos/bridgebot -... [17:52:49] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 5.141% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [17:54:52] FIRING: ProbeDown: Service tools-k8s-haproxy-6:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-6:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:59:52] RESOLVED: [2x] ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:50:57] FIRING: CloudVPSDesignateLeaks: Detected 10 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:24:52] FIRING: ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:29:52] RESOLVED: ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:38:52] FIRING: ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:43:52] RESOLVED: ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:44:04] 10Tool-bridgebot, 10Toolforge: bridgebot tool build service quota not going down - https://phabricator.wikimedia.org/T368317#9957634 (10bd808) ` $ toolforge build quota Registry =================== Storage ----------- Available 140.65Mi Capacity 86% Limit 1.00Gi Used 883.35Mi $ toolforge... [20:47:53] 10Tool-bridgebot, 10Toolforge: [builds-cli] No obvious way to delete individual `toolforge build` generated artifacts other than `toolforge clean` - https://phabricator.wikimedia.org/T368317#9957636 (10bd808) p:05High→03Medium a:05bd808→03None [20:50:58] 10Toolforge: [builds-cli] No obvious way to delete individual `toolforge build` generated artifacts other than `toolforge clean` - https://phabricator.wikimedia.org/T368317#9957641 (10bd808) [20:59:52] FIRING: ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:03:45] 10Tool-bridgebot: Files larger than 1MiB not downloaded and relayed to IRC - https://phabricator.wikimedia.org/T363777#9957657 (10bd808) 05Open→03Declined Things are working as per current config. [21:04:52] RESOLVED: ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:08:52] FIRING: ProbeDown: Service tools-k8s-haproxy-6:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-6:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:08:58] 10Tool-bridgebot, 07Upstream: Bridgebot freaks out and sends double messages from IRC to Telegram - https://phabricator.wikimedia.org/T305487#9957670 (10bd808) 05Open→03Stalled >>! In T305487#9830507, @bd808 wrote: > https://github.com/42wim/matterbridge/pull/2138 has been merged upstream. It looks like up... [21:13:52] RESOLVED: ProbeDown: Service tools-k8s-haproxy-6:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-6:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:23:57] 10Tool-masto-collab: HTTP status client error (422 Unprocessable Entity) on posting with SVG media - https://phabricator.wikimedia.org/T363314#9957676 (10Legoktm) 05Open→03Resolved a:03Legoktm >>! In T363314#9945859, @Legoktm wrote: > We could also download the rasterized PNG version via the thumbnails... [21:24:52] 10Striker: "Set source code repository in toolinfo to this repository" checkbox on repos/create screen is unexpectedly required - https://phabricator.wikimedia.org/T369395 (10bd808) 03NEW [21:25:14] 10Striker: "Set source code repository in toolinfo to this repository" checkbox on repos/create screen is unexpectedly required - https://phabricator.wikimedia.org/T369395#9957691 (10bd808) [21:30:28] 10Striker: "Set source code repository in toolinfo to this repository" checkbox on repos/create screen is unexpectedly required - https://phabricator.wikimedia.org/T369395#9957697 (10bd808) Django form fields default to `required=True`, so we need to add an explicit `required=False` to make the field optional. [21:31:52] FIRING: ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:36:52] RESOLVED: ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:40:56] (03PS1) 10BryanDavis: repo: Setting SCM URL is optional [labs/striker] - 10https://gerrit.wikimedia.org/r/1052355 (https://phabricator.wikimedia.org/T369395) [21:52:50] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 4.697% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [22:37:29] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 4 deleted instances on toolsbeta-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [22:50:57] FIRING: CloudVPSDesignateLeaks: Detected 10 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:09:52] FIRING: ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:14:52] RESOLVED: ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:26:52] FIRING: ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:31:52] RESOLVED: ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:48:52] FIRING: ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:53:52] RESOLVED: ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:54:52] FIRING: ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:59:07] RESOLVED: ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown