[00:27:38] 10VPS-project-Codesearch, 10Gerrit, 06Release-Engineering-Team: Gerrit Projects API listing does not include design/codex-php (Codesearch does not index codex-php) - https://phabricator.wikimedia.org/T380535#10347030 (10bd808) >>! In T380535#10346866, @thcipriani wrote: > Adding a description made it show up... [00:50:08] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: openstack: wmfkeystonehooks: project ids rather than names are being used in LDAP group creation - https://phabricator.wikimedia.org/T379030#10347063 (10bd808) >>! In T379030#10344421, @Andrew wrote: >> I am not coming up with reasons to keep mirroring... [00:55:24] 10Striker: [toolsadmin] Striker cannot create Developer accounts with names matching existing SUL accounts - https://phabricator.wikimedia.org/T380384#10347070 (10RoyZuo) @Andrew err can i wait for the fix? i dont urgently need toolsadmin account. was wanting to try starting a bot, but my coding is rudimentary. :) [03:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:08:31] (03update) 10sstefanova: cli: add config subcommands [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T379091) [08:38:14] (03update) 10sstefanova: cli: add config subcommands [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T379091) [09:35:28] (03CR) 10Arturo Borrero Gonzalez: Get openstack project list from keystone (031 comment) [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/1093997 (https://phabricator.wikimedia.org/T379030) (owner: 10Andrew Bogott) [09:36:52] 10VPS-project-Codesearch, 10Gerrit, 06Release-Engineering-Team: Gerrit Projects API listing does not include design/codex-php (Codesearch does not index codex-php) - https://phabricator.wikimedia.org/T380535#10347394 (10hashar) 05Open→03Resolved a:03bd808 `design/codex-php` was created with a large... [09:40:43] 10VPS-project-Codesearch, 10Gerrit, 06Release-Engineering-Team: Gerrit Projects API listing does not include design/codex-php (Codesearch does not index codex-php) - https://phabricator.wikimedia.org/T380535#10347404 (10hashar) The repository shows up in the {nav Select repo} selection list of https://co... [09:52:30] (03open) 10sstefanova: cli: add deploy-token subcommands [repos/cloud/toolforge/components-cli] (slavina/add-config-subcommands) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/3 (https://phabricator.wikimedia.org/T379091) [09:57:30] 10Cloud-Services, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Replace optics in cloudsw1-d5-eqiad et-0/0/52 and cloudsw1-e4-eqiad et-0/0/54 - https://phabricator.wikimedia.org/T380503#10347431 (10cmooney) This port bounced again overnight: ` cmooney@cloudsw1-d5-eqiad> show log messages.... [10:07:20] (03open) 10sstefanova: cli: add deployment subcommands [repos/cloud/toolforge/components-cli] (slavina/add-deploy-token-commands) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/4 (https://phabricator.wikimedia.org/T379091) [10:45:38] (03update) 10sstefanova: cli: add deployment subcommands [repos/cloud/toolforge/components-cli] (slavina/add-deploy-token-commands) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/4 (https://phabricator.wikimedia.org/T379091) [11:27:55] (03update) 10fnegri: Draft: Import DNS records for db.svc.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/135 (https://phabricator.wikimedia.org/T374953) [11:40:46] (03update) 10sstefanova: deployment: cleanup old deployment on deployment creation [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T380283) (owner: 10dcaro) [11:41:59] (03open) 10sstefanova: deployment list response: update format [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/42 [11:44:07] (03update) 10fnegri: Draft: Import DNS records for db.svc.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/135 (https://phabricator.wikimedia.org/T374953) [11:45:40] (03update) 10fnegri: Draft: Import DNS records for db.svc.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/135 (https://phabricator.wikimedia.org/T374953) [11:52:03] (03update) 10sstefanova: cli: add deployment subcommands [repos/cloud/toolforge/components-cli] (slavina/add-deploy-token-commands) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/4 (https://phabricator.wikimedia.org/T379091) [11:58:49] (03update) 10fnegri: Draft: Import DNS records for db.svc.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/135 (https://phabricator.wikimedia.org/T374953) [12:02:16] (03update) 10fnegri: Draft: Import DNS records for db.svc.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/135 (https://phabricator.wikimedia.org/T374953) [12:07:09] (03update) 10fnegri: Draft: Import DNS records for db.svc.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/135 (https://phabricator.wikimedia.org/T374953) [12:13:34] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Migrate "db.svc.eqiad.wmflabs." DNS zone to cloudinfra project - https://phabricator.wikimedia.org/T380491#10347862 (10taavi) >` >| 7dc63e1d-6eda-4c3d-96fb-e755f3f61235 | osm.db.svc.eqiad.wmflabs. | A | 172.16.6.105... [12:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:43:52] (03update) 10fnegri: Draft: Import DNS records for db.svc.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/135 (https://phabricator.wikimedia.org/T374953) [12:44:29] (03update) 10fnegri: Draft: Import DNS records for db.svc.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/135 (https://phabricator.wikimedia.org/T374953) [12:45:23] (03approved) 10sstefanova: deployment: cleanup old deployment on deployment creation [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T380283) (owner: 10dcaro) [12:45:25] (03update) 10sstefanova: deployment: cleanup old deployment on deployment creation [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T380283) (owner: 10dcaro) [13:08:50] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16): [components-api] Add functional tests for the components api - https://phabricator.wikimedia.org/T379092#10347996 (10Slst2020) we might want to add an endpoint for bulk deletion of deployments so that we can clean them up easily [13:11:41] (03update) 10fnegri: Draft: Import DNS records for db.svc.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/135 (https://phabricator.wikimedia.org/T374953) [13:15:10] (03update) 10fnegri: Draft: Import DNS records for db.svc.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/135 (https://phabricator.wikimedia.org/T374953) [13:16:09] (03update) 10fnegri: Draft: Import DNS records for db.svc.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/135 (https://phabricator.wikimedia.org/T374953) [13:20:39] (03update) 10fnegri: Draft: Import DNS records for db.svc.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/135 (https://phabricator.wikimedia.org/T374953) [13:21:17] (03update) 10fnegri: Import DNS records for db.svc.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/135 (https://phabricator.wikimedia.org/T374953) [13:25:02] (03update) 10fnegri: Import DNS records for db.svc.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/135 (https://phabricator.wikimedia.org/T374953) [13:26:17] (03approved) 10aborrero: Import DNS records for db.svc.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/135 (https://phabricator.wikimedia.org/T374953) (owner: 10fnegri) [13:27:07] (03update) 10fnegri: Import DNS records for db.svc.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/135 (https://phabricator.wikimedia.org/T374953) [13:35:38] (03update) 10fnegri: Import DNS records for db.svc.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/135 (https://phabricator.wikimedia.org/T374953) [13:36:54] (03update) 10fnegri: Import DNS records for db.svc.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/135 (https://phabricator.wikimedia.org/T374953) [13:39:43] (03update) 10fnegri: Import DNS records for db.svc.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/135 (https://phabricator.wikimedia.org/T374953) [13:44:39] 06cloud-services-team, 10Cloud-VPS, 07Documentation, 07IPv6: Cloud VPS: prepare documentation on VXLAN/IPV6 migration - https://phabricator.wikimedia.org/T380054#10348077 (10aborrero) expanded even more https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_VXLAN_IPv6_migration [13:44:50] 06cloud-services-team, 10Cloud-VPS, 07Documentation, 07IPv6: Cloud VPS: prepare documentation on VXLAN/IPV6 migration - https://phabricator.wikimedia.org/T380054#10348078 (10aborrero) [13:47:15] (03merge) 10fnegri: Import DNS records for db.svc.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/135 (https://phabricator.wikimedia.org/T374953) [13:48:36] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [13:48:57] !log fnegri@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [13:48:57] 06Toolforge-standards-committee: Adoption request for bullseye - https://phabricator.wikimedia.org/T380537#10348104 (10SD0001) >>! In T380537#10346360, @LucasWerkmeister wrote: > - the database credentials in `replica.my.cnf` – the policy asks us to delete these (and I believe there is a process for regenerating... [13:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:00:32] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16), 05Goal: [toolsdb] Upgrade to MariaDB 10.6 - https://phabricator.wikimedia.org/T352206#10348157 (10Marostegui) >>! In T352206#10309719, @fnegri wrote: >> Edit: the only other available LTS version is 10.11, so we could consider u... [14:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:01:07] 06cloud-services-team, 10Cloud-VPS, 07Documentation, 07IPv6: Cloud VPS: prepare documentation on VXLAN/IPV6 migration - https://phabricator.wikimedia.org/T380054#10348161 (10aborrero) reviewed and updated https://wikitech.wikimedia.org/wiki/Help:Security_groups [14:01:18] 06cloud-services-team, 10Cloud-VPS, 07Documentation, 07IPv6: Cloud VPS: prepare documentation on VXLAN/IPV6 migration - https://phabricator.wikimedia.org/T380054#10348162 (10aborrero) [14:31:33] (03open) 10fnegri: Remove NS and SOA records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/136 (https://phabricator.wikimedia.org/T374953) [14:35:45] (03merge) 10fnegri: Remove NS and SOA records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/136 (https://phabricator.wikimedia.org/T374953) [14:35:50] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [14:36:15] !log fnegri@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [14:57:59] (03update) 10sstefanova: cli: add config subcommands [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T379091) [15:02:18] (03update) 10sstefanova: cli: add deploy-token subcommands [repos/cloud/toolforge/components-cli] (slavina/add-config-subcommands) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/3 (https://phabricator.wikimedia.org/T379091) [15:12:11] (03update) 10sstefanova: cli: add deployment subcommands [repos/cloud/toolforge/components-cli] (slavina/add-deploy-token-commands) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/4 (https://phabricator.wikimedia.org/T379091) [15:14:10] (03update) 10sstefanova: cli: add deployment subcommands [repos/cloud/toolforge/components-cli] (slavina/add-deploy-token-commands) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/4 (https://phabricator.wikimedia.org/T379091) [15:28:14] FIRING: Kernel error: Server cloudcontrol1005 may have kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Kernel_panic - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-panic-detector?orgId=1&var-instance=cloudcontrol1005 - https://alerts.wikimedia.org/?q=alertname%3DKernel+error [15:28:23] 06cloud-services-team: Kernel error Server cloudcontrol1005 may have kernel errors - https://phabricator.wikimedia.org/T380607 (10phaultfinder) 03NEW [15:39:04] (03PS4) 10Krinkle: frontend: Upgrade to PHP 8.4 and enable JIT [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1072826 [15:40:37] (03CR) 10Krinkle: [C:03+2] frontend: Upgrade to PHP 8.4 and enable JIT [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1072826 (owner: 10Krinkle) [15:41:28] (03Merged) 10jenkins-bot: frontend: Upgrade to PHP 8.4 and enable JIT [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1072826 (owner: 10Krinkle) [15:46:22] FIRING: HAProxyBackendUnavailable: HAProxy service glance-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [15:50:34] FIRING: DiskSpace: Disk space cloudcontrol1005:9100:/ 1.48% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudcontrol1005 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [15:51:22] RESOLVED: HAProxyBackendUnavailable: HAProxy service glance-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [15:53:41] 06cloud-services-team, 10Cloud-VPS, 07Documentation, 07IPv6: Cloud VPS: prepare documentation on VXLAN/IPV6 migration - https://phabricator.wikimedia.org/T380054#10348839 (10aborrero) created https://wikitech.wikimedia.org/wiki/Help:Exposing_IPv6_services [15:58:10] PROBLEM - Disk space on cloudcontrol1005 is CRITICAL: DISK CRITICAL - free space: / 1073MiB (1% inode=96%): /tmp 1073MiB (1% inode=96%): /var/tmp 1073MiB (1% inode=96%): https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=cloudcontrol1005&var-datasource=eqiad+prometheus/ops [16:02:34] 06cloud-services-team, 10Cloud-VPS, 07Documentation, 07IPv6: Cloud VPS: prepare documentation on VXLAN/IPV6 migration - https://phabricator.wikimedia.org/T380054#10348939 (10aborrero) [16:03:28] 06cloud-services-team, 10Cloud-VPS, 07Documentation, 07IPv6: Cloud VPS: prepare documentation on VXLAN/IPV6 migration - https://phabricator.wikimedia.org/T380054#10348940 (10aborrero) [16:03:33] (03update) 10fnegri: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) [16:04:18] 06cloud-services-team, 10Cloud-VPS, 07Documentation, 07IPv6: Cloud VPS: prepare documentation on VXLAN/IPV6 migration - https://phabricator.wikimedia.org/T380054#10348944 (10aborrero) [16:05:21] 06cloud-services-team, 10Cloud-VPS, 07Documentation, 07IPv6: Cloud VPS: prepare documentation on VXLAN/IPV6 migration - https://phabricator.wikimedia.org/T380054#10348945 (10aborrero) 05Open→03Stalled To update https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_Instances we will need to finish {T38008... [16:12:17] (03update) 10aborrero: eqiad1: add IPv6 support [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/132 (https://phabricator.wikimedia.org/T380174) [16:17:04] RESOLVED: DiskSpace: Disk space cloudcontrol1005:9100:/ 1.693% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudcontrol1005 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [16:18:10] RECOVERY - Disk space on cloudcontrol1005 is OK: DISK OK https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=cloudcontrol1005&var-datasource=eqiad+prometheus/ops [16:23:00] (03update) 10fnegri: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) [16:24:31] (03update) 10fnegri: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) [16:25:50] (03update) 10fnegri: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) [16:26:29] (03update) 10fnegri: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) [16:27:52] 10VPS-project-Codesearch, 10Gerrit, 06Release-Engineering-Team: Gerrit Projects API listing does not include design/codex-php (Codesearch does not index codex-php) - https://phabricator.wikimedia.org/T380535#10349040 (10bd808) a:05bd808→03thcipriani >>! In T380535#10347394, @hashar wrote: > My guess... [16:29:02] 06cloud-services-team, 10Cloud-VPS, 07Epic, 07IPv6: Enable IPv6 on CloudVPS - https://phabricator.wikimedia.org/T37947#10349042 (10aborrero) We are targeting to announce/start the user-facing migration on 2025-01-06, see also https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_VXLAN_IPv6_migration [16:36:48] (03update) 10fnegri: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) [16:36:56] FIRING: SystemdUnitDown: The service unit labs-ip-alias-dump.service is in failed status on host cloudservices1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudservices1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [17:23:23] 10Cloud-Services, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Replace optics in cloudsw1-d5-eqiad et-0/0/52 and cloudsw1-e4-eqiad et-0/0/54 - https://phabricator.wikimedia.org/T380503#10349255 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=6b283bec-74b8-4f8c-9a46-f9... [17:31:56] RESOLVED: SystemdUnitDown: The service unit labs-ip-alias-dump.service is in failed status on host cloudservices1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudservices1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [17:33:54] 10Cloud-Services, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Replace optics in cloudsw1-d5-eqiad et-0/0/52 and cloudsw1-e4-eqiad et-0/0/54 - https://phabricator.wikimedia.org/T380503#10349290 (10VRiley-WMF) Replaced the transciever in cloudsw1-e4-eqiad et-0/0/54. Will test to see if th... [17:47:06] (03update) 10fnegri: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) [17:52:16] (03update) 10fnegri: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) [18:00:01] 10Cloud-Services, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Replace optics in cloudsw1-d5-eqiad et-0/0/52 and cloudsw1-e4-eqiad et-0/0/54 - https://phabricator.wikimedia.org/T380503#10349375 (10cmooney) Thanks @VRiley-WMF! Seems ok so far but we can make a call Monday based on if we... [18:02:52] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: tofu-infra: refactor repo structure - https://phabricator.wikimedia.org/T375283#10349376 (10fnegri) @aborrero I spent some time today on this refactor. The MR is now producing a working "tofu plan", but it's still a work in progress. I introduced a few... [18:03:24] (03update) 10fnegri: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) [18:03:31] 10PAWS: New upstream release for OpenRefine - https://phabricator.wikimedia.org/T380436#10349384 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/462 [18:04:22] vivian-rook closed https://github.com/toolforge/paws/pull/462 [18:05:20] 10PAWS: New upstream release for OpenRefine - https://phabricator.wikimedia.org/T380436#10349385 (10rook) 05Open→03Resolved a:03rook [18:09:45] (03update) 10fnegri: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) [18:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:00:56] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:16:17] 06cloud-services-team, 10Cloud-VPS: Audit WMCS compute capacity - https://phabricator.wikimedia.org/T380099#10349806 (10Andrew) >>! In T380099#10333092, @rook wrote: > When running queries in https://thanos.wikimedia.org > ` > avg by (instance)(sum by (cpu,instance)(rate(node_cpu_seconds_total{instance=~"cloud... [20:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:26:28] FIRING: InstanceDown: Project cvn instance cvn-app10 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [20:29:07] 06cloud-services-team, 10Cloud-VPS: Audit WMCS compute capacity - https://phabricator.wikimedia.org/T380099#10349833 (10Andrew) I've designated cloudvirt1035 to be the stress test victim, and I've created two flavors (visible only in the 'testlabs' project) that schedule only on that VM: g4.cores8.ram16.disk20... [20:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:35:37] 06cloud-services-team, 10Cloud-VPS: Audit WMCS compute capacity - https://phabricator.wikimedia.org/T380099#10349842 (10Andrew) Throttling performance drop-off seems to be close to linear. For instance if I run ` sudo cumin --force O{name:"perftest*"} 'sudo stress-ng --cpu 1 --timeout 60 --metrics-brief' ` U... [20:36:28] RESOLVED: InstanceDown: Project cvn instance cvn-app10 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [20:40:17] (03CR) 10Krinkle: ""The pattern" may be unlikely to change, but I do note that what's in this patch isn't the configured pattern. The configured pattern is `" [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1084298 (https://phabricator.wikimedia.org/T378530) (owner: 10AntiCompositeNumber) [20:41:28] (03PS3) 10Krinkle: Handle temporary accounts as anons [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1084298 (https://phabricator.wikimedia.org/T378530) (owner: 10AntiCompositeNumber) [20:44:13] (03PS4) 10Krinkle: Handle temporary accounts as anons [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1084298 (https://phabricator.wikimedia.org/T378530) (owner: 10AntiCompositeNumber) [20:44:56] 06cloud-services-team, 10Cloud-VPS: Audit WMCS compute capacity - https://phabricator.wikimedia.org/T380099#10349853 (10Andrew) [20:53:01] 06cloud-services-team, 10Cloud-VPS: Audit WMCS compute capacity - https://phabricator.wikimedia.org/T380099#10349890 (10Andrew) [21:49:21] (03CR) 10Krinkle: "Trying to compile locally and test this on test.wikipedia.org to see the before-and-after, but will need some time to figure out how to ge" [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1084298 (https://phabricator.wikimedia.org/T378530) (owner: 10AntiCompositeNumber) [22:05:17] (03PS5) 10Krinkle: ListManager: Detect temporary accounts and treat as IP/anons [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1084298 (https://phabricator.wikimedia.org/T378530) (owner: 10AntiCompositeNumber) [22:05:24] (03CR) 10Krinkle: [C:03+2] "Done" [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1084298 (https://phabricator.wikimedia.org/T378530) (owner: 10AntiCompositeNumber) [22:06:02] (03Merged) 10jenkins-bot: ListManager: Detect temporary accounts and treat as IP/anons [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1084298 (https://phabricator.wikimedia.org/T378530) (owner: 10AntiCompositeNumber) [22:24:43] (03PS1) 10AntiCompositeNumber: Tag v4.0.4 [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1094538 [22:25:35] (03CR) 10AntiCompositeNumber: [C:03+2] Tag v4.0.4 [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1094538 (owner: 10AntiCompositeNumber) [22:26:06] (03Merged) 10jenkins-bot: Tag v4.0.4 [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1094538 (owner: 10AntiCompositeNumber) [22:51:59] 10Tool-wikiqanda, 06Future-Audiences: Review data logging practices for past experiments - https://phabricator.wikimedia.org/T380655 (10Maryana) 03NEW