[03:27:56] FIRING: ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:32:56] RESOLVED: ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:34:59] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (Arusha 30): Apply the 4x-iops Cinder volume type to the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T437127#12301434 (10komla) @fnegri, this is a fol... [08:49:29] 10Cloud-VPS (Debian Bullseye Deprecation), 10SRE-swift-storage: Migrate swift away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435783#12301877 (10MatthewVernon) @komla you'll want to ask @Eevans as I think the remaining bullseye systems are his; I note, though, that we're still... [09:22:40] 10Tool-commons-delinquent: Tool-commons-delinquent: HTTP Code 500 error - https://phabricator.wikimedia.org/T437416 (10SD_hehua) 03NEW [09:26:25] 10Toolforge, 06tools-platform-team, 07good first task: Missing space in "no deployments found" message - https://phabricator.wikimedia.org/T437212#12301983 (10aputhin) p:05Triage→03Low Thank you for tagging this task with #good_first_task for Wikimedia newcomers! Newcomers often may not be aware of thin... [09:48:35] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (Arusha 30): Apply the 4x-iops Cinder volume type to the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T437127#12302059 (10fnegri) > Does retyping a liv... [09:51:29] 10Cloud-VPS (Quota-requests), 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (Arusha 30): Provide more disk I/O headroom for the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T435571#12302096 (10fnegri) As noted in {T437127}, I pasted the wrong values in... [09:57:51] 10Toolforge, 06tools-platform-team: Establish design principles for parity between interfaces (CLI vs REST API) - https://phabricator.wikimedia.org/T434093#12302135 (10dcaro) Also related {T408782} [09:59:37] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: tofu-infra: add cinder volume types - https://phabricator.wikimedia.org/T410148#12302143 (10fnegri) > The volume-type and QoS inventory for codfw1dev-r. `lang=yaml # fnegri@cloudcontrol2005-dev:~$ sudo wmcs-openstack volu... [09:59:41] (03approved) 10tlepage: api: add envvars to source build [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/184 (https://phabricator.wikimedia.org/T434496) (owner: 10dcaro) [09:59:49] (03update) 10tlepage: api: add envvars to source build [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/184 (https://phabricator.wikimedia.org/T434496) (owner: 10dcaro) [10:09:16] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: tofu-infra: add cinder volume types - https://phabricator.wikimedia.org/T410148#12302168 (10fnegri) > The project names and domains corresponding to 488a233c5dc245799ba3b898d18d1b9c and c26d9d326bdf464fa1025939ded7e5a2 you... [10:14:04] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [10:14:14] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [10:14:24] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [10:16:21] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [10:16:46] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/80 [10:16:47] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/80 [10:17:22] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [10:17:25] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [10:18:17] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/42 [10:18:48] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [10:18:49] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [10:18:57] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/30 [10:19:06] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/33 [11:39:28] 10Toolforge, 06tools-platform-team, 07OKR-Work: Create template for maintainer alerts emails - https://phabricator.wikimedia.org/T437081#12302536 (10aputhin) p:05Triage→03High > We can use the [template provided by alertmanager](https://github.com/prometheus/alertmanager/blob/main/template/email.html) as... [11:58:43] !log fnegri@cloudcumin1001 deployment-prep START - Cookbook wmcs.vps.add_user_to_project for user 'denisse' in role 'member' [11:58:49] !log fnegri@cloudcumin1001 deployment-prep END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'denisse' in role 'member' [12:13:16] 10Tool-wikinewsie, 06translatewiki.net, 07Essential-Work, 10LPL Projects (Ongoing maintenance), 10LPL sprints (S3 2026 September): Add Wikinewsie to translatewiki.net - https://phabricator.wikimedia.org/T435739#12302773 (10GGalofre-WMF) [12:13:51] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [12:13:58] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [12:14:06] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [12:15:07] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [12:15:28] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/80 [12:15:50] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [12:16:25] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/42 [12:16:44] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [12:16:52] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/30 [12:16:57] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/33 [12:24:27] 06cloud-services-team, 10Toolforge, 06tools-platform-team: Policy for (external) contributions / code style - https://phabricator.wikimedia.org/T408331#12302799 (10dcaro) [12:28:30] 10Cloud-VPS, 06tools-infrastructure-team: cloud-vps haproxy port exhaustion - https://phabricator.wikimedia.org/T437350#12302807 (10taavi) [12:28:34] 10Cloud-VPS, 06tools-infrastructure-team: cloud-vps haproxy port exhaustion - https://phabricator.wikimedia.org/T437350#12302808 (10taavi) [12:28:40] 10Cloud-VPS, 06tools-infrastructure-team, 06All-and-every-Wikisource, 10WS Export: wsexport traffic issues - https://phabricator.wikimedia.org/T437349#12302809 (10taavi) [12:28:43] 10Cloud-VPS, 06tools-infrastructure-team: cloud-vps haproxy port exhaustion - https://phabricator.wikimedia.org/T437350#12302810 (10taavi) a:03taavi [12:39:20] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloud-vps haproxy port exhaustion - https://phabricator.wikimedia.org/T437350#12302875 (10taavi) [13:13:11] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloud-vps haproxy port exhaustion - https://phabricator.wikimedia.org/T437350#12302975 (10taavi) 05Open→03Resolved Optimistically resolving given the patch above. [13:46:44] (03approved) 10dcaro: refactor: Generalize job event models [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T432566) (owner: 10tlepage) [13:53:07] (03update) 10mahveotm: Draft: WIP: Add Cinder volume types and QoS management [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/350 (https://phabricator.wikimedia.org/T410148) [13:54:40] (03update) 10tlepage: refactor: Generalize job event models [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T432566) [13:54:58] (03update) 10tlepage: events: Add the GET /events endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [13:55:13] (03update) 10tlepage: refactor: Parse Loki result streams into events [repos/cloud/toolforge/logs-api] (generalize-loki-push-format) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) [13:55:25] (03update) 10tlepage: refactor: Generalize job event models [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T432566) [13:55:26] (03update) 10tlepage: refactor: Generalize Loki push format [repos/cloud/toolforge/logs-api] (write-refactor-for-read) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/41 (https://phabricator.wikimedia.org/T432566) [14:24:07] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12303470 (10elukey) I tried to reimage cloudcephosd1055 and it failed in d-i executing `/tmp/partman_early_command`. Tried to exec it... [14:27:54] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [jobs-cli, jobs-api] support --webservice option - https://phabricator.wikimedia.org/T428898#12303490 (10dcaro) 05Open→03In progress [14:48:48] FIRING: [3x] PuppetZeroResources: Puppet has failed generate resources on cloudcontrol1007:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [14:52:35] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12303703 (10Andrew) a:05Jclark-ctr→03fgiunchedi That's new to me -- looks like the most recent changes to that script are from @B... [14:53:48] FIRING: [4x] PuppetZeroResources: Puppet has failed generate resources on cloudcontrol1007:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [14:56:45] FIRING: WidespreadPuppetFailure: Puppet has failed on wmcs cluster - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=3&var-cluster=wmcs - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [14:58:48] FIRING: [5x] PuppetZeroResources: Puppet has failed generate resources on cloudcontrol1007:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [15:01:45] RESOLVED: WidespreadPuppetFailure: Puppet has failed on wmcs cluster - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=3&var-cluster=wmcs - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [15:03:48] FIRING: [5x] PuppetZeroResources: Puppet has failed generate resources on cloudcontrol1007:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [15:08:48] RESOLVED: [5x] PuppetZeroResources: Puppet has failed generate resources on cloudcontrol1007:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [15:09:22] !log komla@cloudcumin1001 imagor START - Cookbook wmcs.vps.create_project for project imagor in eqiad1 (T437088) [15:09:23] komla@cloudcumin1001: Unknown project "imagor" [15:09:24] T437088: Request creation of imagor VPS project - https://phabricator.wikimedia.org/T437088 [15:10:06] (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: added project imagor [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/351 (https://phabricator.wikimedia.org/T437088) [15:12:49] 10Cloud-VPS (Quota-requests): Quota increase request for project deployment-prep - https://phabricator.wikimedia.org/T437117#12303809 (10fnegri) +1 [15:13:23] komla@cloudcumin1001 create_project (PID 3291903) is awaiting input [15:14:18] FIRING: [5x] PuppetZeroResources: Puppet has failed generate resources on cloudcontrol1006:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [15:14:42] (03merge) 10komla: projects: added project imagor [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/351 (https://phabricator.wikimedia.org/T437088) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [15:16:34] !log komla@cloudcumin1001 imagor END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project imagor in eqiad1 (T437088) [15:16:35] komla@cloudcumin1001: Unknown project "imagor" [15:16:36] T437088: Request creation of imagor VPS project - https://phabricator.wikimedia.org/T437088 [15:18:19] 10Cloud-VPS (Project-requests), 06MediaWiki-Media-Platform-Team, 13Patch-For-Review: Request creation of imagor VPS project - https://phabricator.wikimedia.org/T437088#12303831 (10komla) 05Open→03Resolved a:03komla @Ladsgroup @Samwilson this has been done. [15:24:18] RESOLVED: [2x] PuppetZeroResources: Puppet has failed generate resources on cloudcontrol1006:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [15:30:05] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Write tutorial for migrating build service jobs to push-to-deploy - https://phabricator.wikimedia.org/T437477 (10komla) 03NEW [15:30:35] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Create a list of candidate tools for Push-to-Deploy adoption - https://phabricator.wikimedia.org/T437257#12303881 (10komla) 05Open→03Resolved [15:30:53] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 07Documentation: Write tutorial for migrating build service jobs to push-to-deploy - https://phabricator.wikimedia.org/T437477#12303884 (10taavi) [15:32:30] (03update) 10dcaro: emailer: include pod status message in failure emails [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/58 (https://phabricator.wikimedia.org/T384252) (owner: 10mahveotm) [15:40:01] !log komla@cloudcumin1001 deployment-prep START - Cookbook wmcs.openstack.quota_increase by 30 server-groups (T437117) [15:40:07] T437117: Quota increase request for project deployment-prep - https://phabricator.wikimedia.org/T437117 [15:40:09] !log komla@cloudcumin1001 deployment-prep END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 30 server-groups (T437117) [15:41:26] 10Cloud-VPS (Quota-requests): Quota increase request for project deployment-prep - https://phabricator.wikimedia.org/T437117#12303941 (10komla) 05Open→03Resolved a:03komla This is has been done. ` | server-group-members | 10 | | server-groups | 40 | ` [15:50:33] 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12303991 (10taavi) > Also for Toolforge to work, I think we need to get a build service working. The automatic build didn't work for me, so currently I'm manually deploying, f... [15:56:40] 10Cloud-VPS (Quota-requests): Quota increase request for project deployment-prep - https://phabricator.wikimedia.org/T437117#12304027 (10komla) [16:11:03] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12304100 (10Southparkfan) a:03Southparkfan [16:12:19] 10Toolforge, 06tools-platform-team, 07good first task: Missing space in "no deployments found" message - https://phabricator.wikimedia.org/T437212#12304105 (10CoderAnimeshPathak) a:03CoderAnimeshPathak Hii, I will love to work on this task... [16:51:22] 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, 10ops-eqiad, 06SRE: cloudvirt1067 missing hwraid BBU - https://phabricator.wikimedia.org/T437221#12304345 (10VRiley-WMF) 05Open→03In progress Since these were moved, I'm going to check the cable. If it's not that, I may have to see if I can search f... [16:53:44] PROBLEM - Host cloudvirt1067 is DOWN: PING CRITICAL - Packet loss = 100% [17:00:27] (03open) 10coderanimeshpathak04092007: Fix missing space in "No deployments found" output [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T437212) [17:03:56] RECOVERY - Host cloudvirt1067 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms [17:09:06] PROBLEM - Host cloudvirt1067 is DOWN: PING CRITICAL - Packet loss = 100% [17:13:06] 10Data-Services, 06tools-platform-team, 07User-notice: Update Wiki Replicas documentation for x4 rollout - https://phabricator.wikimedia.org/T437208#12304453 (10STei-WMF) @taavi This is how I have interpreted your instruction as an entry: "Developers who maintain a tool that queries the Wikimedia Commons l... [17:27:12] RECOVERY - Host cloudvirt1067 is UP: PING OK - Packet loss = 0%, RTA = 0.31 ms [17:29:06] 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, 10ops-eqiad, 06SRE: cloudvirt1067 missing hwraid BBU - https://phabricator.wikimedia.org/T437221#12304497 (10VRiley-WMF) 05In progress→03Resolved I unplugged the server, and removed the battery and did a flea power drain on the unit. Plugged the... [17:48:48] 10Cloud Services Proposals, 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: Decision Request - How openstack projects relate to tofu-infra - https://phabricator.wikimedia.org/T385604#12304543 (10Andrew) a:05Andrew→03None [17:52:23] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Move cloudvirt1075 from F4 to E4 - https://phabricator.wikimedia.org/T435923#12304559 (10VRiley-WMF) Starting this now... [17:52:31] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Move cloudvirt1075 from F4 to E4 - https://phabricator.wikimedia.org/T435923#12304560 (10VRiley-WMF) 05Open→03In progress [18:01:29] 06cloud-services-team, 10superset.wmcloud.org: Sunset superset.wmcloud.org - https://phabricator.wikimedia.org/T416373#12304615 (10Andrew) @Dev, is anything left to do on this task? [18:10:34] 10superset.wmcloud.org, 06tools-infrastructure-team: Update superset.wmcloud.org to use modern magnum templates - https://phabricator.wikimedia.org/T437492 (10Andrew) 03NEW [18:15:05] (03update) 10mahveotm: Add Cinder volume types and QoS management [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/350 (https://phabricator.wikimedia.org/T410148) [18:15:52] (03update) 10mahveotm: Add Cinder volume types and QoS management [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/350 (https://phabricator.wikimedia.org/T410148) [18:18:25] (03update) 10mahveotm: Add Cinder volume types and QoS management [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/350 (https://phabricator.wikimedia.org/T410148) [18:25:17] 10Openstack-Magnum, 06tools-infrastructure-team: Update procbot to use modern magnum templates - https://phabricator.wikimedia.org/T437494 (10Andrew) 03NEW [18:27:22] (03update) 10mahveotm: Add Cinder volume types and QoS management [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/350 (https://phabricator.wikimedia.org/T410148) [18:28:18] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 10Data-Engineering-Roadmap, 07Epic: Ability to create new views for tools and analytic purposes - https://phabricator.wikimedia.org/T424524#12304782 (10Ahoelzl) [18:29:19] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 10Data-Engineering-Roadmap, and 2 others: Ability to create new views for tools and analytic purposes - https://phabricator.wikimedia.org/T424524#12304786 (10Ottomata) [18:45:27] 10Openstack-Magnum, 06tools-infrastructure-team: Replace all magnum clusters using the old Heat driver - https://phabricator.wikimedia.org/T437496 (10Andrew) 03NEW [18:47:17] 10Openstack-Magnum, 06tools-infrastructure-team: Rebuild the cloud-runner-staging magnum cluster with the new magnum-cluster-api driver - https://phabricator.wikimedia.org/T437498 (10Andrew) 03NEW [18:47:52] 10Openstack-Magnum, 06tools-infrastructure-team: Replace all magnum clusters using the old Heat driver - https://phabricator.wikimedia.org/T437496#12304855 (10Andrew) [18:47:53] 10Openstack-Magnum, 06tools-infrastructure-team: Update procbot to use modern magnum templates - https://phabricator.wikimedia.org/T437494#12304854 (10Andrew) [18:48:13] 10superset.wmcloud.org, 06tools-infrastructure-team: Update superset.wmcloud.org to use modern magnum templates - https://phabricator.wikimedia.org/T437492#12304875 (10Andrew) [18:48:15] 10Openstack-Magnum, 06tools-infrastructure-team: Replace all magnum clusters using the old Heat driver - https://phabricator.wikimedia.org/T437496#12304876 (10Andrew) [18:50:04] 10Openstack-Magnum, 06tools-infrastructure-team: Rebuild the deployment-wikikube-v127 magnum cluster with the new magnum-cluster-api driver and k8s v1.34 - https://phabricator.wikimedia.org/T437500 (10Andrew) 03NEW [18:50:26] 10Openstack-Magnum, 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure: Rebuild the deployment-wikikube-v127 magnum cluster with the new magnum-cluster-api driver and k8s v1.34 - https://phabricator.wikimedia.org/T437500#12304892 (10Andrew) [18:51:51] 10Openstack-Magnum, 10Quarry, 06tools-infrastructure-team: Rebuild quarry with the new magnum-cluster-api driver - https://phabricator.wikimedia.org/T437501 (10Andrew) 03NEW [18:52:08] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Move cloudvirt1075 from F4 to E4 - https://phabricator.wikimedia.org/T435923#12304905 (10VRiley-WMF) E4 u38 cableID: 20220046 port 38 [18:53:29] 10Openstack-Magnum, 06tools-infrastructure-team: Rebuild PAWS with the magnum-cluster-api driver - https://phabricator.wikimedia.org/T437502 (10Andrew) 03NEW [18:53:42] 10Openstack-Magnum, 06tools-infrastructure-team: Rebuild PAWS with the magnum-cluster-api driver - https://phabricator.wikimedia.org/T437502#12304919 (10Andrew) [18:53:45] 06cloud-services-team, 10PAWS: replace ingress-nginx in PAWS - https://phabricator.wikimedia.org/T427848#12304920 (10Andrew) [18:58:33] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12304929 (10Andrew) Another attempt: I added 2GB more ram to each of the OSDs on cloudvirt1045. Does not seem to make a di... [19:20:05] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12304990 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudv... [19:31:12] (03open) 10mahveotm: flavors: Require aggregate settings and Ceph disk limits [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/352 (https://phabricator.wikimedia.org/T429336) [19:36:13] (03update) 10mahveotm: Add Cinder volume types and QoS management [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/350 (https://phabricator.wikimedia.org/T410148) [19:59:37] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: tofu-infra: add cinder volume types - https://phabricator.wikimedia.org/T410148#12305092 (10Mahveotm) Thanks! All done from my end. As I understand from [the docs](https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin... [20:35:38] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12305258 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudv... [21:15:01] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-docker-mathoid02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436465#12305393 (10Southparkfan) 05In progress→03Resolved [21:15:48] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12305397 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1... [21:16:56] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Move cloudvirt1075 from F4 to E4 - https://phabricator.wikimedia.org/T435923#12305407 (10VRiley-WMF) 05In progress→03Resolved @fgiunchedi this one is completed [22:08:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks