[00:18:22] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for tcywikisource - https://phabricator.wikimedia.org/T378469#10275261 (10Zabe) wiki has been created [01:21:31] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:09:18] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10275455 (10Platonides) >>! In T376267#10270792, @bd808 wrote: > > Wikitech no longer knows a password for anyone until they use https://wikitech.wikimedia.org/wiki/Special:PasswordReset to... [02:18:34] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 5.385% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [02:19:08] 10VPS-project-Wikistats: Add tcywikisource to wikistats - https://phabricator.wikimedia.org/T378475#10275468 (10Dzahn) 05Stalled→03Open a:03Dzahn [02:20:19] 10VPS-project-Wikistats: Add tcywiktionary to wikistats - https://phabricator.wikimedia.org/T378467#10275472 (10Dzahn) 05Stalled→03Open a:03Dzahn [03:38:06] (03PS1) 10AntiCompositeNumber: Handle temporary accounts as anons [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1084298 (https://phabricator.wikimedia.org/T378530) [03:42:27] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10275514 (10Robertsky) @bd808 thanks, it works.. now I have other unattached accounts to deal with. fun. haha. with stake out at T378289. [04:35:51] FIRING: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:40:51] RESOLVED: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:56:11] 06cloud-services-team, 10Cloud-VPS: Create mechanism to allow the use of vanity domains by projects behind the Cloud VPS shared HTTP proxy - https://phabricator.wikimedia.org/T342398#10275558 (10taavi) https://lists.wikimedia.org/hyperkitty/list/cloud-admin@lists.wikimedia.org/thread/XHUP2ERZ2W6LHILNXZOPHC37IT... [05:21:31] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:18:35] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 5.08% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [06:56:51] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for tcywiktionary - https://phabricator.wikimedia.org/T378462#10275613 (10ABran-WMF) cookbook sre.mysql.sanitize-pii --wiki tcywiktionary run, wiki is good to go [06:59:49] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for tcywikisource - https://phabricator.wikimedia.org/T378469#10275615 (10ABran-WMF) cookbook sre.mysql.sanitize-pii --wiki tcywikisource run, wiki is good to go [07:50:37] (03CR) 10Kosta Harlan: [C:03+1] Handle temporary accounts as anons (031 comment) [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1084298 (https://phabricator.wikimedia.org/T378530) (owner: 10AntiCompositeNumber) [09:21:31] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:32:52] (03update) 10dcaro: Draft: DONOTMERGE: test deployment for api-gateway+components-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/556 [09:33:25] (03update) 10dcaro: components-api: configure for local [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/556 [09:33:30] (03update) 10dcaro: components-api: configure for local [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/556 [09:45:38] (03update) 10dcaro: deploy: add support for deployment tokens [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/29 (https://phabricator.wikimedia.org/T362066) (owner: 10sstefanova) [09:45:39] (03approved) 10dcaro: deploy: add support for deployment tokens [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/29 (https://phabricator.wikimedia.org/T362066) (owner: 10sstefanova) [09:45:46] (03merge) 10dcaro: deploy: add support for deployment tokens [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/29 (https://phabricator.wikimedia.org/T362066) (owner: 10sstefanova) [09:45:47] (03update) 10sstefanova: tests: refactor [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/30 [09:48:19] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: DONOTMERGE components-api: bump to 0.0.29-20241002095441-cd2060f1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/544 (https://phabricator.wikimedia.org/T356261) [09:48:41] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [components-api] Develop the webhook mechanism to trigger a deployment - https://phabricator.wikimedia.org/T362066#10275999 (10dcaro) [10:18:35] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 4.724% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [10:34:12] (03update) 10dcaro: [maintain-harbor] Move to become a toolforge component [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/34 (https://phabricator.wikimedia.org/T358225) (owner: 10raymond-ndibe) [10:38:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-idp-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [11:05:37] 10Data-Services, 06Data-Platform-SRE, 06DBA: Prepare and check storage layer for annwiki - https://phabricator.wikimedia.org/T377118#10276219 (10Ladsgroup) [11:06:49] 10Data-Services, 06Data-Platform-SRE, 06DBA: Prepare and check storage layer for bclwikisource - https://phabricator.wikimedia.org/T377087#10276221 (10Ladsgroup) [11:06:51] (03open) 10dcaro: token: add created_at field to the token [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/31 [11:07:20] 10Data-Services, 06Data-Platform-SRE, 06DBA: Prepare and check storage layer for ibawiki - https://phabricator.wikimedia.org/T376571#10276214 (10Ladsgroup) [11:08:16] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for tcywikisource - https://phabricator.wikimedia.org/T378469#10276218 (10Ladsgroup) >>! In T378469#10275615, @ABran-WMF wrote: > cookbook sre.mysql.sanitize-pii --wiki tcywikisource run, wiki is good to go You mean our part is d... [11:09:27] 10Data-Services, 06Data-Platform-SRE, 06DBA: Prepare and check storage layer for rskwiki - https://phabricator.wikimedia.org/T375016#10276236 (10Ladsgroup) [11:30:27] (03update) 10dcaro: token: add created_at field to the token [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/31 [11:31:24] (03update) 10dcaro: token: add created_at field to the token [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/31 [11:37:33] (03update) 10dcaro: token: add created_at field to the token [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/31 [11:42:25] (03update) 10dcaro: token: add created_at field to the token [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/31 [12:26:35] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for tcywikisource - https://phabricator.wikimedia.org/T378469#10276480 (10ABran-WMF) >>! In T378469#10276218, @Ladsgroup wrote: >>>! In T378469#10275615, @ABran-WMF wrote: >> cookbook sre.mysql.sanitize-pii --wiki tcywikisource ru... [12:31:08] 10Data-Services, 06Data-Platform-SRE, 06DBA: Prepare and check storage layer for tcywikisource - https://phabricator.wikimedia.org/T378469#10276498 (10Ladsgroup) [12:32:35] 10Data-Services, 06Data-Platform-SRE, 06DBA: Prepare and check storage layer for tcywiktionary - https://phabricator.wikimedia.org/T378462#10276518 (10Ladsgroup) [13:16:15] !log root@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.27.16 to 1.28.14 (T362867) [13:16:19] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [13:16:35] !log root@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.27.16 to 1.28.14 (T362867) [13:30:24] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:41:00] (03update) 10dcaro: token: add created_at field to the token [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/31 [13:41:02] (03update) 10dcaro: token: add created_at field to the token [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/31 [13:46:42] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Remove stub certs for ms-fe [labs/private] - 10https://gerrit.wikimedia.org/r/1084150 (https://phabricator.wikimedia.org/T357750) (owner: 10Muehlenhoff) [13:56:45] (03update) 10dcaro: token: add created_at field to the token [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/31 [13:58:14] FIRING: ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: toolsbeta-test-k8s-control-10.toolsbeta.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [14:01:41] 06cloud-services-team, 06DC-Ops, 10ops-codfw, 06SRE: Test new hardware candidate for cloudbackup replacement - https://phabricator.wikimedia.org/T353746#10276940 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm [14:03:14] RESOLVED: ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: toolsbeta-test-k8s-control-10.toolsbeta.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [14:06:17] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: cloudcontrol2006-dev struggling with memory - https://phabricator.wikimedia.org/T370401#10276958 (10Jhancock.wm) I got the memory in. Is it safe to proceed with the upgrade at this time? I didn't see if it got depooled already. [14:14:33] !log root@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.27.16 to 1.28.14 (T362867) [14:14:37] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [14:18:14] FIRING: ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: toolsbeta-test-k8s-control-11.toolsbeta.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [14:18:56] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 4.23% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [14:20:12] !log root@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.27.16 to 1.28.14 (T362867) [14:20:17] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [14:23:14] RESOLVED: ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: toolsbeta-test-k8s-control-11.toolsbeta.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [14:28:15] !log root@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.27.16 to 1.28.14 (T362867) [14:28:19] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [14:31:34] 10Tool-video-answer-tool, 06Future-Audiences: Design: Add last modified and # of contributors metadata to video attribution - https://phabricator.wikimedia.org/T378383#10277101 (10Maryana) [14:32:14] FIRING: [2x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: toolsbeta-test-k8s-control-12.toolsbeta.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [14:33:50] !log root@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.27.16 to 1.28.14 (T362867) [14:33:54] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [14:37:14] RESOLVED: [3x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: toolsbeta-test-k8s-control-11.toolsbeta.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [14:38:47] FIRING: NodeDownForLong: The node cloudvirt1063 has been unreachable for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudvirt1063 - https://alerts.wikimedia.org/?q=alertname%3DNodeDownForLong [14:38:52] 06cloud-services-team: NodeDownForLong cloudvirt1063:9100 The node cloudvirt1063 has been unreachable for more than two hours. - https://phabricator.wikimedia.org/T378607 (10phaultfinder) 03NEW [14:48:35] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 (T362867) [14:48:40] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [14:49:38] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 (T362867) [14:49:39] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 (T362867) [14:50:41] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 (T362867) [14:50:42] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 (T362867) [14:51:41] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 (T362867) [14:51:42] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 (T362867) [14:52:43] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 (T362867) [14:52:44] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-12 from 1.27.16 to 1.28.14 (T362867) [14:53:41] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-12 from 1.27.16 to 1.28.14 (T362867) [14:53:42] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-13 from 1.27.16 to 1.28.14 (T362867) [14:53:46] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [14:54:39] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-13 from 1.27.16 to 1.28.14 (T362867) [14:55:44] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 13Patch-For-Review: Improve WMCS NodeDown alerts - https://phabricator.wikimedia.org/T375479#10277267 (10fnegri) I merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/1077038 and tested the new settings by shutting down cloudvirt1063 (which is a... [14:57:12] 10Data-Services, 06Data-Platform-SRE, 06DBA: Prepare and check storage layer for tcywikisource - https://phabricator.wikimedia.org/T378469#10277297 (10Gehel) p:05Triage→03High [14:57:13] 10Data-Services, 06Data-Platform-SRE, 06DBA: Prepare and check storage layer for tcywiktionary - https://phabricator.wikimedia.org/T378462#10277298 (10Gehel) p:05Triage→03High [14:57:18] 10Data-Services, 06Data-Platform-SRE, 06DBA: Prepare and check storage layer for annwiki - https://phabricator.wikimedia.org/T377118#10277299 (10Gehel) p:05Triage→03High [14:57:20] 10Data-Services, 06Data-Platform-SRE, 06DBA: Prepare and check storage layer for bclwikisource - https://phabricator.wikimedia.org/T377087#10277300 (10Gehel) p:05Triage→03High [14:57:28] 10Data-Services, 06Data-Platform-SRE, 06DBA: Prepare and check storage layer for rskwiki - https://phabricator.wikimedia.org/T375016#10277302 (10Gehel) p:05Triage→03High [14:57:36] 10Data-Services, 06Data-Platform-SRE, 06DBA: Prepare and check storage layer for ibawiki - https://phabricator.wikimedia.org/T376571#10277301 (10Gehel) p:05Triage→03High [14:57:56] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-10 from 1.27.16 to 1.28.14 (T362867) [14:58:52] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-10 from 1.27.16 to 1.28.14 (T362867) [14:58:53] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-11 from 1.27.16 to 1.28.14 (T362867) [14:58:56] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [14:59:48] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-11 from 1.27.16 to 1.28.14 (T362867) [14:59:49] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-9 from 1.27.16 to 1.28.14 (T362867) [15:00:47] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-9 from 1.27.16 to 1.28.14 (T362867) [15:03:08] 10Data-Services, 06DBA, 10Data-Platform-SRE (2024.10.19 - 2024.11.08): Prepare and check storage layer for annwiki - https://phabricator.wikimedia.org/T377118#10277339 (10Gehel) [15:03:19] 10Data-Services, 06DBA, 10Data-Platform-SRE (2024.10.19 - 2024.11.08): Prepare and check storage layer for bclwikisource - https://phabricator.wikimedia.org/T377087#10277337 (10Gehel) [15:04:23] 10Data-Services, 06DBA, 10Data-Platform-SRE (2024.10.19 - 2024.11.08): Prepare and check storage layer for tcywiktionary - https://phabricator.wikimedia.org/T378462#10277341 (10Gehel) [15:04:25] 10Data-Services, 06DBA, 10Data-Platform-SRE (2024.10.19 - 2024.11.08): Prepare and check storage layer for rskwiki - https://phabricator.wikimedia.org/T375016#10277333 (10Gehel) [15:05:27] 10Data-Services, 06DBA, 10Data-Platform-SRE (2024.10.19 - 2024.11.08): Prepare and check storage layer for ibawiki - https://phabricator.wikimedia.org/T376571#10277335 (10Gehel) [15:05:53] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Quarry, 10superset.wmcloud.org: Replace Quarry with an installation of Superset - https://phabricator.wikimedia.org/T169452#10277359 (10joanna_borun) 05Open→03Declined [15:06:37] 10Data-Services, 06DBA, 10Data-Platform-SRE (2024.10.19 - 2024.11.08): Prepare and check storage layer for tcywikisource - https://phabricator.wikimedia.org/T378469#10277344 (10Gehel) [15:07:49] 06cloud-services-team, 10Striker: Warn cloud users against re-using keys - https://phabricator.wikimedia.org/T208599#10277370 (10joanna_borun) 05Open→03Declined [15:09:52] 06cloud-services-team, 10Cloud-VPS, 10MediaWiki-Vagrant, 07Documentation: Improve content design of: https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Cloud_VPS - https://phabricator.wikimedia.org/T245083#10277377 (10joanna_borun) p:05Triage→03Low [15:10:39] 06cloud-services-team, 10Cloud-VPS, 10MediaWiki-Vagrant, 07Documentation: Improve content design of: https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Cloud_VPS - https://phabricator.wikimedia.org/T245083#10277379 (10joanna_borun) 05Open→03Resolved [15:10:48] 06cloud-services-team, 10Striker: Warn cloud users against re-using keys - https://phabricator.wikimedia.org/T208599#10277374 (10fnegri) Keys are now managed using https://idm.wikimedia.org/ [15:13:18] 06cloud-services-team, 10Toolforge, 07Documentation, 07good first task: Find and fix inaccuracies in Toolforge Django tutorial - https://phabricator.wikimedia.org/T245683#10277392 (10joanna_borun) p:05Triage→03Low [15:13:48] 06cloud-services-team, 10Toolforge, 07Documentation: Update and Improve Toolforge and Cloud VPS Technical Documentation - https://phabricator.wikimedia.org/T203131#10277381 (10joanna_borun) [15:14:27] 06cloud-services-team, 10Cloud-VPS, 10MediaWiki-Vagrant, 07Documentation: Improve content design of: https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Cloud_VPS - https://phabricator.wikimedia.org/T245083#10277384 (10joanna_borun) Closing the task as the documentation got some significant i... [15:16:22] 06cloud-services-team, 10wikitech.wikimedia.org, 10Wikimedia-Site-requests: Rename Wikitech Nova resource: namespace to something that is more commonly used - https://phabricator.wikimedia.org/T275796#10277408 (10joanna_borun) p:05Triage→03Low [15:18:11] 10wikitech.wikimedia.org, 10Wikidata: Enable interwiki links to/from Wikitech - https://phabricator.wikimedia.org/T290147#10277427 (10taavi) I don't think wikitech-static not displaying interwiki links is a big deal; if we're having to resort to wikitech-static then most of those links would be broken anyway. [15:18:27] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Privacy Engineering: Increased visibility in wiki-replicas for volunteers fighting vandals - https://phabricator.wikimedia.org/T284944#10277439 (10joanna_borun) p:05Triage→03High [15:19:09] 10wikitech.wikimedia.org, 10MediaWiki-libs-UUID, 07Wikimedia-production-error: Brief RuntimeException on wikitech: Could not open '/tmp/mw-GlobalIdGenerator33-UUID-128' - https://phabricator.wikimedia.org/T364684#10277442 (10taavi) Still a problem? [15:20:07] 10Striker, 10Bitu, 06Infrastructure-Foundations, 10Puppet-Core, and 2 others: Take some pointers from GitHub security updates - https://phabricator.wikimedia.org/T304231#10277449 (10taavi) [15:20:26] 10Striker, 10Bitu, 06Infrastructure-Foundations, 10Puppet-Core, and 2 others: Take some pointers from GitHub security updates - https://phabricator.wikimedia.org/T304231#10277451 (10taavi) [15:27:57] 06cloud-services-team, 10Cloud-VPS: Create mechanism to allow the use of vanity domains by projects behind the Cloud VPS shared HTTP proxy - https://phabricator.wikimedia.org/T342398#10277477 (10joanna_borun) p:05Triage→03Medium [15:28:06] 06cloud-services-team, 10Cloud-VPS: Create mechanism to allow the use of vanity domains by projects behind the Cloud VPS shared HTTP proxy - https://phabricator.wikimedia.org/T342398#10277478 (10fnegri) 05Open→03In progress [15:29:00] 06cloud-services-team, 10Toolforge, 10Elasticsearch: Deploy multi-tentant OpenSearch cluster as replacement for Elasticsearch - https://phabricator.wikimedia.org/T348943#10277483 (10joanna_borun) p:05Triage→03Medium [15:29:26] 06cloud-services-team, 10Toolforge, 10Elasticsearch, 07Epic: Deploy multi-tentant OpenSearch cluster as replacement for Elasticsearch - https://phabricator.wikimedia.org/T348943#10277485 (10fnegri) [15:29:28] 06cloud-services-team, 10Cloud-VPS: Add mwstake.org and wikiapiary.com to the domains handled by WMCS - https://phabricator.wikimedia.org/T342389#10277482 (10fnegri) [15:31:11] 06cloud-services-team, 10Cloud-VPS, 10Wikispore: vanity domain for Wikispore - https://phabricator.wikimedia.org/T368236#10277480 (10fnegri) [15:33:28] 06cloud-services-team, 10Cloud-VPS: [cloud-vps] creating a new project can override existing DNS entries - https://phabricator.wikimedia.org/T360294#10277503 (10joanna_borun) p:05Triage→03Medium [15:33:59] 10Striker, 10Bitu, 06Infrastructure-Foundations, 10Puppet-Core, and 2 others: Take some pointers from GitHub security updates - https://phabricator.wikimedia.org/T304231#10277498 (10SLyngshede-WMF) For Bitu / idm.wikimedia.org we've implemented rules for which type of new keys can be used. Current rules al... [15:34:28] 06cloud-services-team, 10Cloud-VPS: ProjectProxyMainProxyDown should have a runbook page - https://phabricator.wikimedia.org/T361873#10277508 (10joanna_borun) p:05Triage→03Medium [15:35:31] 06cloud-services-team, 10Toolforge, 06Language and Product Localization: https://cxdebugger.toolforge.org/ has become very slow - https://phabricator.wikimedia.org/T367022#10277511 (10joanna_borun) 05Open→03Resolved [15:37:22] 06cloud-services-team, 10Toolforge, 06Language and Product Localization: https://cxdebugger.toolforge.org/ has become very slow - https://phabricator.wikimedia.org/T367022#10277514 (10joanna_borun) Should be working fine now, please feel free to reopen the task if issue reoccurs. [15:39:05] 10Cloud-VPS, 10Wikispore: vanity domain for Wikispore - https://phabricator.wikimedia.org/T368236#10277522 (10joanna_borun) [15:39:49] 06cloud-services-team, 10Data-Services: [wikireplicas] Automatically check for missing tables - https://phabricator.wikimedia.org/T378470#10277540 (10fnegri) p:05Triage→03Medium [15:43:45] 06cloud-services-team: NodeDownForLong cloudvirt1063:9100 The node cloudvirt1063 has been unreachable for more than two hours. - https://phabricator.wikimedia.org/T378607#10277567 (10fnegri) →14Duplicate dup:03T375223 [15:43:48] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: 2024-09-21 NodeDown cloudvirt1063 - https://phabricator.wikimedia.org/T375223#10277569 (10fnegri) [15:44:04] 06cloud-services-team, 10Toolforge: I keep getting Toolforge messages about rustup - https://phabricator.wikimedia.org/T378437#10277571 (10fnegri) p:05Triage→03Medium [15:45:20] 06cloud-services-team: SystemdUnitDown Unit opentofu-infra-diff.service on node cloudcontrol1007 has been down for long. - https://phabricator.wikimedia.org/T377650#10277576 (10fnegri) 05Open→03Resolved a:03fnegri This is now working. [15:46:50] 06cloud-services-team, 10Horizon: Keystone auth endpoint should use a standard HTTPS port - https://phabricator.wikimedia.org/T377055#10277596 (10fnegri) p:05Triage→03Medium [15:49:13] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T376990#10277608 (10fnegri) 05Open→03Resolved a:03fnegri This was caused by {T376719}, now fixed. [15:49:22] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T376990#10277616 (10fnegri) [15:49:33] 06cloud-services-team: SystemdUnitDown Unit prometheus-node-kernel-panic.service on node cloudcephosd1020 has been down for long. - https://phabricator.wikimedia.org/T376989#10277614 (10fnegri) →14Duplicate dup:03T376990 [15:50:56] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 06DBA: Prepare and check storage layer for nrwiki - https://phabricator.wikimedia.org/T375101#10277623 (10fnegri) p:05Triage→03Medium [15:53:07] 06cloud-services-team, 10Data-Services, 10Projects-Cleanup: Archive the operations/debs/bdsync repository - https://phabricator.wikimedia.org/T377882#10277641 (10fnegri) p:05Triage→03Low [15:53:10] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 06DBA: Prepare and check storage layer for nrwiki - https://phabricator.wikimedia.org/T375101#10277625 (10fnegri) a:03fnegri [15:54:09] 06cloud-services-team, 10Cloud-VPS: tofuinfratest creates many pages in wikitech - https://phabricator.wikimedia.org/T376888#10277644 (10fnegri) p:05Triage→03Medium [15:55:36] 06cloud-services-team: SystemdUnitDown Unit rsync_enterprise_htmldumps.service on node clouddumps1001 has been down for long. - https://phabricator.wikimedia.org/T376271#10277651 (10fnegri) 05Open→03Resolved a:03fnegri This is now working again. [15:57:46] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 10AbuseFilter, 06Data Products, and 8 others: Public wiki replicas contain abuse filter logs for filters that are private or protected - https://phabricator.wikimedia.org/T375751#10277661 (10fnegri) p:05Medium→03High [16:02:06] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 10AbuseFilter, 06Data Products, and 8 others: Public wiki replicas contain abuse filter logs for filters that are private or protected - https://phabricator.wikimedia.org/T375751#10277683 (10fnegri) [16:06:31] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic: Set up a bitu instance for codfw1dev - https://phabricator.wikimedia.org/T360795#10277761 (10fnegri) [16:08:11] (03CR) 10Majavah: vps.create_project: add the checks mentioned in the wiki (034 comments) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1084042 (owner: 10David Caro) [16:08:19] 10Tools: s52421__commonsdelinquent_p.event needs index on done column? - https://phabricator.wikimedia.org/T178327#10277785 (10fnegri) [16:09:27] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: [openstack object storage] deleted files still occupying space - https://phabricator.wikimedia.org/T376673#10277791 (10fnegri) p:05Triage→03High [16:11:05] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge: [toolforge-prometheus] upgrade to bookworm - https://phabricator.wikimedia.org/T375523#10277813 (10fnegri) p:05Triage→03Medium [16:11:33] 06cloud-services-team, 10wikitech.wikimedia.org: Reimage eqiad cloudweb hosts to bookworm - https://phabricator.wikimedia.org/T376277#10277816 (10fnegri) p:05Triage→03Medium [16:12:24] 06cloud-services-team, 10wikitech.wikimedia.org: update labtestwiki user and password - https://phabricator.wikimedia.org/T328289#10277802 (10Ladsgroup) 05Open→03Declined Since {T378260} [16:16:55] (03CR) 10David Caro: vps.create_project: add the checks mentioned in the wiki (034 comments) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1084042 (owner: 10David Caro) [16:17:31] (03CR) 10Majavah: vps.create_project: add the checks mentioned in the wiki (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1084042 (owner: 10David Caro) [16:19:34] (03CR) 10David Caro: vps.create_project: add the checks mentioned in the wiki (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1084042 (owner: 10David Caro) [16:20:30] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: cloudcontrol2006-dev struggling with memory - https://phabricator.wikimedia.org/T370401#10277879 (10aborrero) >>! In T370401#10276958, @Jhancock.wm wrote: > I got the memory in. Is it safe to proceed with the upgrade at this time? I didn't see... [16:23:44] 10Tool-video-answer-tool, 06Future-Audiences: [Idea] Other video tool tweaks - https://phabricator.wikimedia.org/T378623 (10Maryana) 03NEW [16:27:29] (03PS4) 10David Caro: vps.create_project: add the checks mentioned in the wiki [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1084042 [16:27:29] (03PS5) 10David Caro: vps.create_project: add users and quotas [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1084044 [16:27:29] (03PS5) 10David Caro: vps.create_project: small refactor [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1084053 [16:27:30] (03PS7) 10David Caro: vps.create_project: create the tofu patch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1084054 [16:27:31] (03CR) 10David Caro: vps.create_project: create the tofu patch (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1084054 (owner: 10David Caro) [16:36:28] FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [16:37:10] 10Tool-video-answer-tool, 06Future-Audiences: [Idea] Other video tool tweaks - https://phabricator.wikimedia.org/T378623#10277985 (10Maryana) Maryana to set up meeting [16:43:53] 10Tool-video-answer-tool, 06Future-Audiences: [Idea] Other video tool tweaks - https://phabricator.wikimedia.org/T378623#10278027 (10etz) A thought here around more rapid content: I think if we end up testing a series of videos which are faster, it could be an opportunity to lean hard into a different tone, s... [16:57:28] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance maps-proxy-03 in project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [17:01:17] PROBLEM - Disk space on cloudbackup1004 is CRITICAL: DISK CRITICAL - free space: /srv 645213MiB (3% inode=99%): https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=cloudbackup1004&var-datasource=eqiad+prometheus/ops [17:07:28] FIRING: [2x] PuppetAgentFailure: Puppet agent failure detected on instance maps-proxy-03 in project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [17:11:41] (03update) 10dcaro: token: add created_at field to the token [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/31 [17:13:49] (03approved) 10dcaro: tests: refactor [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/30 (owner: 10sstefanova) [17:13:51] (03update) 10dcaro: tests: refactor [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/30 (owner: 10sstefanova) [17:13:54] (03merge) 10dcaro: tests: refactor [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/30 (owner: 10sstefanova) [17:16:15] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: DONOTMERGE components-api: bump to 0.0.29-20241002095441-cd2060f1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/544 (https://phabricator.wikimedia.org/T356261) [17:22:36] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:47:36] RESOLVED: DiskSpace: Disk space cloudbackup1004:9100:/srv 5.831% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [17:49:09] (03update) 10dcaro: token: add created_at field to the token [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/31 [17:49:19] (03update) 10dcaro: token: add created_at field to the token [repos/cloud/toolforge/components-api] (rename_deploy_token) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/31 [17:57:08] (03update) 10dcaro: token: add created_at field to the token [repos/cloud/toolforge/components-api] (rename_deploy_token) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/31 [17:57:15] (03update) 10dcaro: token: add created_at field to the token [repos/cloud/toolforge/components-api] (rename_deploy_token) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/31 [17:58:06] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 10AbuseFilter, 06Data Products, and 8 others: Public wiki replicas contain abuse filter logs for filters that are private or protected - https://phabricator.wikimedia.org/T375751#10278291 (10fnegri) 05In progress→03Resolved I did manually... [18:01:17] RECOVERY - Disk space on cloudbackup1004 is OK: DISK OK https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=cloudbackup1004&var-datasource=eqiad+prometheus/ops [18:01:32] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 06DBA: Prepare and check storage layer for nrwiki - https://phabricator.wikimedia.org/T375101#10278327 (10fnegri) 05Open→03Resolved I created the views by running `maintain-views` on clouddb* hosts, and I also created the [DNS records](https... [18:05:31] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Privacy Engineering: Increased visibility in wiki-replicas for volunteers fighting vandals - https://phabricator.wikimedia.org/T284944#10278336 (10fnegri) > excluding all data related to abusefilter (e.g. abusefilter, abuse_filter, abuse_filter_... [18:22:17] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 13Patch-For-Review: tofu-infra: replace wmcs-wikireplica-dns.py with tofu - https://phabricator.wikimedia.org/T374953#10278409 (10fnegri) This is not as easy as I hoped, because the python script also creates additional CNAMEs for each and every wiki, e... [18:29:26] 10Tool-video-answer-tool, 06Future-Audiences: [Spike] Other video tool tweaks: narration speedup - https://phabricator.wikimedia.org/T378623#10278440 (10Maryana) [18:31:51] 10Tool-video-answer-tool, 06Future-Audiences: [Spike] Experiment with nightcore video mode - https://phabricator.wikimedia.org/T378639 (10Maryana) 03NEW [18:33:49] 10Tool-video-answer-tool, 06Future-Audiences: [Spike] Experiment with carousel mode - https://phabricator.wikimedia.org/T378640#10278495 (10Maryana) [18:36:04] 10Tool-video-answer-tool, 06Future-Audiences: [Spike] Experiment with carousel mode - https://phabricator.wikimedia.org/T378640#10278499 (10Maryana) [18:36:05] 10Tool-video-answer-tool, 06Future-Audiences, 07Epic: [Epic] Video tool refinements - https://phabricator.wikimedia.org/T377392#10278498 (10Maryana) [18:36:08] 10Tool-video-answer-tool, 06Future-Audiences: [Spike] Experiment with nightcore video mode - https://phabricator.wikimedia.org/T378639#10278500 (10Maryana) [18:36:39] 10Tool-video-answer-tool, 06Future-Audiences, 07Epic: [Epic] Video tool refinements - https://phabricator.wikimedia.org/T377392#10278518 (10Maryana) [18:37:21] 10Tool-video-answer-tool, 06Future-Audiences, 07Spike: Investigate On This Day datasets available - https://phabricator.wikimedia.org/T375733#10278520 (10Maryana) 05Open→03Resolved [18:38:34] 10Tool-video-answer-tool, 06Future-Audiences: [Spike] Other video tool tweaks: narration speedup - https://phabricator.wikimedia.org/T378623#10278522 (10Maryana) [18:38:35] 10Tool-video-answer-tool, 06Future-Audiences, 07Epic: [Epic] Video tool refinements - https://phabricator.wikimedia.org/T377392#10278523 (10Maryana) [18:38:47] FIRING: NodeDownForLong: The node cloudvirt1063 has been unreachable for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudvirt1063 - https://alerts.wikimedia.org/?q=alertname%3DNodeDownForLong [18:39:14] 06cloud-services-team: NodeDownForLong cloudvirt1063:9100 The node cloudvirt1063 has been unreachable for more than two hours. - https://phabricator.wikimedia.org/T378642 (10phaultfinder) 03NEW [18:43:04] 10PAWS: paws nfs full - https://phabricator.wikimedia.org/T378643 (10rook) 03NEW [18:48:32] 10PAWS: paws nfs full - https://phabricator.wikimedia.org/T378643#10278563 (10rook) Confirmed on nfs host (paws-nfs-1.paws.eqiad1.wikimedia.cloud): ` rook@paws-nfs-1:~$ df -h Filesystem Size Used Avail Use% Mounted on udev 2.0G 0 2.0G 0% /dev tmpfs 394M 484K 393M 1% /run /de... [18:48:45] 10PAWS: paws nfs full - https://phabricator.wikimedia.org/T378643#10278566 (10rook) Large files have been removed. [18:48:52] 10PAWS: paws nfs full - https://phabricator.wikimedia.org/T378643#10278569 (10rook) 05Open→03Resolved [19:16:56] 10VPS-project-Wikistats: Add tcywikisource to wikistats - https://phabricator.wikimedia.org/T378475#10278690 (10Dzahn) 05Open→03Resolved ` MariaDB [wikistats]> insert into wikisources (prefix, lang, loclang, method) select prefix,lang,loclang,method from wikipedias where prefix="tcy"; ` ` dzahn@wikist... [19:18:19] 10VPS-project-Wikistats: Add tcywiktionary to wikistats - https://phabricator.wikimedia.org/T378467#10278694 (10Dzahn) 05Open→03Resolved ` MariaDB [wikistats]> insert into wiktionaries (prefix, lang, loclang, method) select prefix,lang,loclang,method from wikipedias where prefix="tcy"; ` ` dzahn@wikis... [20:39:20] 10Cloud-VPS (Project-requests), 06Data-Platform-SRE, 10Wikidata, 10Wikidata-Query-Service: Request creation of wikiqlever VPS project - https://phabricator.wikimedia.org/T377655#10279078 (10bking) @Seppl2013 I recommend [[ https://wiki.debian.org/sysstat | sysstat ]] (also known as sar) for tracking memory... [20:50:36] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 06Security-Team, 07SecTeam-Processed, 07Security: abuse_filter_log still exists on some replicas - https://phabricator.wikimedia.org/T378511#10279121 (10sbassett) [20:50:52] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 06Security-Team, 07SecTeam-Processed, 07Security: abuse_filter_log still exists on some replicas - https://phabricator.wikimedia.org/T378511#10279125 (10sbassett) [21:18:11] FIRING: PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-harbor-1 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:22:36] FIRING: CloudVPSDesignateLeaks: Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:02:04] 10wikitech.wikimedia.org, 10MediaWiki-libs-UUID, 07Wikimedia-production-error: Brief RuntimeException on wikitech: Could not open '/tmp/mw-GlobalIdGenerator33-UUID-128' - https://phabricator.wikimedia.org/T364684#10279316 (10Quiddity) 05Open→03Declined Closing, as no longer reproducible. The problema... [22:47:38] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate mwv-builder-03.mediawiki-vagrant.eqiad.wmflabs is about to expire in -5d 0h 1m 25s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [23:52:27] 10VPS-project-Wikistats: add ae.wikimedia.org to wikistats - https://phabricator.wikimedia.org/T369858#10279525 (10Dzahn) a:03Dzahn