[00:00:50] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [01:35:15] (03PS1) 10Andrew Bogott: Add __init__ to enable backup strategies [openstack/horizon/trove-dashboard] (2024.1) - 10https://gerrit.wikimedia.org/r/1024987 [01:36:01] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Add __init__ to enable backup strategies [openstack/horizon/trove-dashboard] (2024.1) - 10https://gerrit.wikimedia.org/r/1024987 (owner: 10Andrew Bogott) [01:37:48] (03PS1) 10Andrew Bogott: Add __init__ to enable backup strategies [openstack/horizon/trove-dashboard] - 10https://gerrit.wikimedia.org/r/1024988 [01:38:20] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Add __init__ to enable backup strategies [openstack/horizon/trove-dashboard] - 10https://gerrit.wikimedia.org/r/1024988 (owner: 10Andrew Bogott) [01:42:26] 06cloud-services-team, 13Patch-For-Review: Migrate cloudweb, cloudbackup, cloudmetrics physical servers off buster - https://phabricator.wikimedia.org/T332400#9751543 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudbackup1003.eqiad.wmnet with OS bookworm [03:11:47] 06cloud-services-team, 13Patch-For-Review: Migrate cloudweb, cloudbackup, cloudmetrics physical servers off buster - https://phabricator.wikimedia.org/T332400#9751595 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1002 for host cloudbackup1003.eqiad.wmnet with OS bookworm co... [03:12:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:15:55] PROBLEM - Host cloudbackup1003 is DOWN: PING CRITICAL - Packet loss = 100% [03:16:44] ACKNOWLEDGEMENT - SSH on cloudbackup1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds Andrew Bogott rebooting/reimaging https://wikitech.wikimedia.org/wiki/SSH/monitoring [03:16:44] ACKNOWLEDGEMENT - Host cloudbackup1003 is DOWN: PING CRITICAL - Packet loss = 100% Andrew Bogott rebooting/reimaging [03:18:23] RECOVERY - Host cloudbackup1003 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [03:22:41] (CloudVPSDesignateLeaks) resolved: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:11:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:16:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:21:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:26:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:14:15] 10Data-Services, 06DBA: Prepare and check storage layer for sysop_plwiki - https://phabricator.wikimedia.org/T363276#9751755 (10Marostegui) This requires restarting all sanitarium hosts. [06:35:32] 10Data-Services, 06DBA: Prepare and check storage layer for sysop_plwiki - https://phabricator.wikimedia.org/T363276#9751769 (10Marostegui) I have restarted all sanitarium hosts so this wiki creation can happen anytime. There shouldn't be anything pending from DBA side. [06:42:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:52:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:57:41] (CloudVPSDesignateLeaks) resolved: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:43:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:53:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:59:00] 06cloud-services-team, 10Data-Services, 10Toolforge, 07Performance Issue: NFS storage slow on Toolforge? - https://phabricator.wikimedia.org/T363621#9752021 (10taavi) [08:02:09] 10cloud-services-team (FY2023/2024-Q3-Q4): Test using phabricator-maintenance-bot to sync wmcs-related boards - https://phabricator.wikimedia.org/T358251#9752025 (10Aklapper) > moving tasks to the right column in the team board ("in progress", "blocked", "done") when the task status changes There is [no API for... [09:12:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:22:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:13:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:23:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:29:03] (ToolforgeKubernetesWorkerTooManyDProcesses) firing: Kubernetes worker tools-k8s-worker-nfs-42 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [10:34:27] 14Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9752487 (10dcaro) > Another reason is that I really don't like to push any minor changes to repository, even testing and intentionally broking, it creates very dirty repos... [10:40:28] 10Cloud Services Proposals: Decision request template - kubernetes upgrade workgroup - https://phabricator.wikimedia.org/T363683 (10dcaro) 03NEW [10:45:00] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent enforcement model - https://phabricator.wikimedia.org/T362872#9752551 (10dcaro) The decision about commiting to drop the extra component on the upgrade to k8s 1.26 might become way more relevant with {T3... [10:57:44] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 10Temporary accounts: Surface Temporary user information to Cloud Wiki Replicas - https://phabricator.wikimedia.org/T346679#9752609 (10kostajh) [11:32:04] (03PS1) 10EoghanGaffney: apt-staging: Add dummy token for gitlab package puller [labs/private] - 10https://gerrit.wikimedia.org/r/1025327 [12:07:02] (03CR) 10EoghanGaffney: [V:03+2 C:03+2] apt-staging: Add dummy token for gitlab package puller [labs/private] - 10https://gerrit.wikimedia.org/r/1025327 (owner: 10EoghanGaffney) [12:42:41] (CloudVPSDesignateLeaks) firing: (2) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:47:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:52:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:57:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:25:49] 10cloud-services-team (FY2023/2024-Q3-Q4): [tf-infra-test] Authentication failed - https://phabricator.wikimedia.org/T363696 (10fnegri) 03NEW [13:26:59] 10cloud-services-team (FY2023/2024-Q3-Q4): [tf-infra-test] Authentication failed - https://phabricator.wikimedia.org/T363696#9753047 (10fnegri) cc @Andrew maybe the token was invalidated during the recent OpenStack upgrade? [13:43:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:53:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:56:29] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: [tf-infra-test] Authentication failed - https://phabricator.wikimedia.org/T363696#9753247 (10fnegri) p:05Triage→03Medium [14:08:01] 06cloud-services-team: CephSlowOps Ceph cluster in eqiad has slow ops, which might be blocking some writes - https://phabricator.wikimedia.org/T358907#9753328 (10fnegri) 05Open→03Resolved a:03fnegri [14:10:10] 06cloud-services-team: SystemdUnitDown Unit ceph-osd@132.service on node cloudcephosd1017 has been down for long. - https://phabricator.wikimedia.org/T358925#9753344 (10fnegri) 05Open→03Resolved a:03fnegri The alert is not firing anymore, I will resolve this task but {T358945} is still open until the... [14:11:49] 06cloud-services-team: PuppetFailure Puppet failure on cloudbackup1004:9100 - https://phabricator.wikimedia.org/T360280#9753353 (10fnegri) 05Open→03Resolved a:03fnegri [14:11:51] (03PS1) 10Andrew Bogott: WMFHACK: don't compress template_cache_preloads [openstack/horizon/horizon] (2024.1) - 10https://gerrit.wikimedia.org/r/1025371 [14:11:55] 06cloud-services-team: SystemdUnitDown Unit backup_vms.service on node cloudbackup1004 has been down for long. - https://phabricator.wikimedia.org/T360278#9753351 (10fnegri) 05Open→03Resolved a:03fnegri [14:12:11] 06cloud-services-team: PuppetFailure Puppet failure on cloudbackup1001-dev:9100 - https://phabricator.wikimedia.org/T361731#9753355 (10fnegri) 05Open→03Resolved a:03fnegri [14:12:14] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] WMFHACK: don't compress template_cache_preloads [openstack/horizon/horizon] (2024.1) - 10https://gerrit.wikimedia.org/r/1025371 (owner: 10Andrew Bogott) [14:13:05] 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T361732#9753357 (10fnegri) 05Open→03Resolved a:03fnegri [14:13:13] 06cloud-services-team: SystemdUnitDown Unit postgresql@15-main.service on node cloudbackup1001-dev has been down for long. - https://phabricator.wikimedia.org/T361733#9753359 (10fnegri) 05Open→03Resolved a:03fnegri [14:13:19] 06cloud-services-team: SystemdUnitDown Unit backup_cinder_volumes.service on node cloudbackup1001-dev has been down for long. - https://phabricator.wikimedia.org/T361751#9753364 (10fnegri) 05Open→03Resolved a:03fnegri [14:13:41] 06cloud-services-team: NovafullstackSustainedFailures The automated tests were unable to create, provision and decommission a VM in the last 5h - https://phabricator.wikimedia.org/T361773#9753366 (10fnegri) 05Open→03Resolved a:03fnegri [14:13:49] 06cloud-services-team: SystemdUnitDown Unit postgresql@15-main.service on node cloudbackup1002-dev has been down for long. - https://phabricator.wikimedia.org/T361882#9753371 (10fnegri) 05Open→03Resolved a:03fnegri [14:13:52] 06cloud-services-team: PuppetFailure Puppet failure on cloudbackup1002-dev:9100 - https://phabricator.wikimedia.org/T361880#9753368 (10fnegri) 05Open→03Resolved a:03fnegri [14:14:22] 06cloud-services-team: SystemdUnitDown Unit remove_dangling_cinder_snapshots.service on node cloudbackup1001-dev has been down for long. - https://phabricator.wikimedia.org/T362845#9753376 (10fnegri) 05Open→03Resolved a:03fnegri [14:14:25] 06cloud-services-team: InterfaceSpeedError brq05a5494a-18 on cloudvirt2001-dev:9100 has the wrong speed: 1.25e+06. - https://phabricator.wikimedia.org/T363164#9753374 (10fnegri) 05Open→03Resolved a:03fnegri [14:14:37] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T360279#9753378 (10fnegri) 05Open→03Resolved a:03fnegri [14:14:39] 06cloud-services-team: HAProxyServiceUnavailable - https://phabricator.wikimedia.org/T358607#9753388 (10fnegri) 05Open→03Resolved a:03fnegri [14:14:49] 06cloud-services-team: SystemdUnitDown Unit wmf_auto_restart_virtlogd.service on node cloudvirt1036 has been down for long. - https://phabricator.wikimedia.org/T361662#9753384 (10fnegri) 05Open→03Resolved a:03fnegri [14:19:25] (03PS1) 10Andrew Bogott: Merge remote-tracking branch 'remotes/origin/2024.1' [openstack/horizon/horizon] - 10https://gerrit.wikimedia.org/r/1025374 [14:28:22] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge: [toolforge] Redis refusing connections - https://phabricator.wikimedia.org/T363709 (10fnegri) 03NEW [14:28:59] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge: [toolforge] Redis refusing connections - https://phabricator.wikimedia.org/T363709#9753447 (10taavi) [14:29:03] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge: [toolforge] Redis refusing connections - https://phabricator.wikimedia.org/T363709#9753448 (10fnegri) p:05Triage→03High [15:02:40] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge: [toolforge] Redis refusing connections - https://phabricator.wikimedia.org/T363709#9753593 (10bd808) Redis 7, which is available in bookworm, includes the ability to configure client eviction when the server hits a memory pool limit. https://redis.io/docs... [15:04:31] 10PAWS: rpy2 preventing singleuser from building - https://phabricator.wikimedia.org/T363715 (10rook) 03NEW [15:06:18] 10PAWS: rpy2 preventing singleuser from building - https://phabricator.wikimedia.org/T363715#9753610 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/403 [15:06:24] vivian-rook opened https://github.com/toolforge/paws/pull/403 [15:30:41] 10PAWS: PAWS down - https://phabricator.wikimedia.org/T363719 (10rook) 03NEW [15:33:55] (PawsJupyterHubDown) firing: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [15:51:59] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: [tf-infra-test] Authentication failed - https://phabricator.wikimedia.org/T363696#9753898 (10rook) This might be keeping the hub container on paws down (Thus all of paws down) T363719 Looks like paws can't detach/attach the pvc for the hub container. `{"... [16:00:53] (03CR) 10Dzahn: [C:03+1] "I still see it in private repo, fwiw. But deleting this can at worse break something in cloud VPS, so easy +1 regardless" [labs/private] - 10https://gerrit.wikimedia.org/r/1024824 (https://phabricator.wikimedia.org/T360414) (owner: 10Andrea Denisse) [16:04:16] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: [tf-infra-test] Authentication failed - https://phabricator.wikimedia.org/T363696#9753977 (10rook) A new application credential does seem to get tofu working [16:05:31] 10PAWS: Test PAWS on k8s 1.25 - https://phabricator.wikimedia.org/T326985#9753979 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/404 [16:05:35] vivian-rook opened https://github.com/toolforge/paws/pull/404 [16:22:45] 10PAWS: PAWS down - https://phabricator.wikimedia.org/T363719#9754082 (10rook) Application certificate had become invalid. T363696 [16:22:52] 10PAWS: PAWS down - https://phabricator.wikimedia.org/T363719#9754087 (10rook) 05Open→03Resolved [16:23:55] (PawsJupyterHubDown) resolved: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [16:36:16] 10PAWS: update openrefine - https://phabricator.wikimedia.org/T363732 (10rook) 03NEW [16:42:26] 10Openstack-Magnum: Deploy k8s greater than 1.23 - https://phabricator.wikimedia.org/T363504#9754224 (10rook) 05Open→03Resolved [16:42:41] 10Openstack-Magnum: Deploy k8s greater than 1.23 - https://phabricator.wikimedia.org/T363504#9754222 (10rook) Done in T326985 [16:43:04] 10PAWS: Test PAWS on k8s 1.25 - https://phabricator.wikimedia.org/T326985#9754226 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/404 [16:43:17] vivian-rook closed https://github.com/toolforge/paws/pull/404 [16:50:30] 10PAWS: Test PAWS on k8s 1.25 - https://phabricator.wikimedia.org/T326985#9754273 (10rook) 05In progress→03Resolved [16:51:54] 10PAWS: Pod Security Policies - https://phabricator.wikimedia.org/T317787#9754291 (10rook) [16:51:56] 10PAWS: Test PAWS on k8s 1.25 - https://phabricator.wikimedia.org/T326985#9754292 (10rook) [16:52:01] 10PAWS: Pod Security Policies - https://phabricator.wikimedia.org/T317787#9754295 (10rook) 05Open→03Resolved [16:55:34] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge: [toolforge] Redis refusing connections - https://phabricator.wikimedia.org/T363709#9754305 (10fnegri) a:03fnegri [16:56:19] 10Tools, 10Gerrit, 03Wikimedia-Hackathon-2024: Gerrit reviewer bot should add reviewers as CC instead of actual reviewers - https://phabricator.wikimedia.org/T363290#9754314 (10Jdlrobson) If this is done, could you provide an option for retaining the status quo? I'm using that page specifically for the purpo... [16:59:53] 10Cloud Services Proposals, 10cloud-services-team (FY2023/2024-Q3-Q4): Decision Request - Incident Response Process - https://phabricator.wikimedia.org/T348887#9754329 (10fnegri) ### 2024-04-28 [WMCS] Toolforge Redis refusing connections By chance, we had a real outage yesterday, which was a good chance to te... [17:02:44] 10PAWS: jupyterlab to 4.1.8 - https://phabricator.wikimedia.org/T363596#9754348 (10rook) [17:03:00] 10PAWS: jupyterlab to 4.1.8 - https://phabricator.wikimedia.org/T363596#9754350 (10rook) [17:03:58] 10PAWS: rpy2 preventing singleuser from building - https://phabricator.wikimedia.org/T363715#9754359 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/403 [17:04:03] vivian-rook closed https://github.com/toolforge/paws/pull/403 [17:04:13] 10PAWS: rpy2 preventing singleuser from building - https://phabricator.wikimedia.org/T363715#9754363 (10rook) 05Open→03Resolved [17:21:05] 10Tools, 10Gerrit, 03Wikimedia-Hackathon-2024: Gerrit reviewer bot should add reviewers as CC instead of actual reviewers - https://phabricator.wikimedia.org/T363290#9754448 (10Dzahn) I would focus on editing the actual page content first. Like trying to determine who is not active anymore and remove them wh... [17:29:52] 10PAWS: jupyterlab to 4.1.8 - https://phabricator.wikimedia.org/T363596#9754512 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/402 [17:30:02] vivian-rook closed https://github.com/toolforge/paws/pull/402 [17:30:36] 10Cloud Services Proposals: Decision request - kubernetes upgrade workgroup - https://phabricator.wikimedia.org/T363683#9754514 (10fnegri) [17:33:21] 10PAWS, 10Pywikibot: add species to fam list - https://phabricator.wikimedia.org/T363738 (10rook) 03NEW [17:35:03] 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T363131#9754540 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/405 [17:35:13] vivian-rook opened https://github.com/toolforge/paws/pull/405 [17:35:29] 10PAWS, 10Pywikibot: add species to fam list - https://phabricator.wikimedia.org/T363738#9754538 (10rook) @Xqt could you review the pr above and comment if it is fine? [17:44:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:53:53] 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T363131#9754647 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/405 [17:54:07] vivian-rook closed https://github.com/toolforge/paws/pull/405 [17:54:11] 10PAWS: jupyterlab to 4.1.8 - https://phabricator.wikimedia.org/T363596#9754651 (10rook) 05Open→03Resolved [17:54:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:54:43] 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T363131#9754649 (10rook) 05Open→03Resolved a:03rook [17:56:27] 10PAWS: update openrefine - https://phabricator.wikimedia.org/T363732#9754668 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/406 [17:56:36] vivian-rook opened https://github.com/toolforge/paws/pull/406 [18:00:53] (03PS1) 10Andrew Bogott: requirements: add pymemcache>=4.00 [openstack/horizon/horizon] - 10https://gerrit.wikimedia.org/r/1025433 [18:01:26] (03PS1) 10Andrew Bogott: requirements: add pymemcache>=4.00 [openstack/horizon/horizon] (2024.1) - 10https://gerrit.wikimedia.org/r/1025434 [18:02:09] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] requirements: add pymemcache>=4.00 [openstack/horizon/horizon] - 10https://gerrit.wikimedia.org/r/1025433 (owner: 10Andrew Bogott) [18:02:22] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Merge remote-tracking branch 'remotes/origin/2024.1' [openstack/horizon/horizon] - 10https://gerrit.wikimedia.org/r/1025374 (owner: 10Andrew Bogott) [18:02:38] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] requirements: add pymemcache>=4.00 [openstack/horizon/horizon] (2024.1) - 10https://gerrit.wikimedia.org/r/1025434 (owner: 10Andrew Bogott) [18:22:45] (03CR) 10Andrea Denisse: [V:03+2] ssl: Remove unnecessary dummy key from thanos-query hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1024824 (https://phabricator.wikimedia.org/T360414) (owner: 10Andrea Denisse) [18:22:47] (03CR) 10Andrea Denisse: [V:03+2 C:03+2] ssl: Remove unnecessary dummy key from thanos-query hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1024824 (https://phabricator.wikimedia.org/T360414) (owner: 10Andrea Denisse) [18:25:07] 10PAWS: update openrefine - https://phabricator.wikimedia.org/T363732#9754843 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/406 [18:25:35] vivian-rook closed https://github.com/toolforge/paws/pull/406 [18:25:42] 10PAWS: update openrefine - https://phabricator.wikimedia.org/T363732#9754845 (10rook) 05Open→03Resolved a:03rook [19:12:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:22:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:22:56] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:23:11] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:54:51] 10Quarry: [bug] Internal server error & backed up queue - https://phabricator.wikimedia.org/T363644#9755193 (10rook) →14Duplicate dup:03T362213 [19:54:58] 10Quarry: Error 500 when clicking "stop query" - https://phabricator.wikimedia.org/T362213#9755195 (10rook) [21:44:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:54:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:11:21] 10Tool-bridgebot: Bot seemingly only links 1 file if > 1 uploaded in the same message - https://phabricator.wikimedia.org/T363777 (10Reedy) 03NEW [23:12:43] 10Tool-bridgebot: Bot seemingly only links 1 file if > 1 uploaded in the same message - https://phabricator.wikimedia.org/T363777#9755898 (10Reedy) p:05Triage→03Low [23:22:07] 10Tool-bridgebot, 07Upstream: Bot seemingly only links 1 file if > 1 uploaded in the same message - https://phabricator.wikimedia.org/T363777#9755942 (10bd808) `lang=irc [23:13] < wm-bb> strange, I can’t find an existing upstream issue for it [23:13] < wm-bb> (I wo... [23:30:19] 14Grid-Engine-to-K8s-Migration: Migrate parliamentdiagram from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319955#9755962 (10Gouvernathor) I did figure it out by telling .lighttpd.conf to use python3 to handle the .py requests. [23:38:00] 10Tool-bridgebot, 07Upstream: Bot seemingly only links 1 file if > 1 uploaded in the same message - https://phabricator.wikimedia.org/T363777#9755981 (10LucasWerkmeister) I can’t find an upstream bug report about this, and it’s not immediately obvious why it shouldn’t work – [Telegram’s handleUploadFile](https... [23:38:21] 10Tool-bridgebot, 07Upstream: Bot seemingly only links 1 file if > 1 uploaded in the same message - https://phabricator.wikimedia.org/T363777#9755982 (10bd808) `lang=go time="2024-04-29T23:05:40Z" level=debug msg="Trying to download "Welcome.pdf" with size 5679824" func=HandleDownloadSize file="bridgebot-matte... [23:39:42] 10Tool-bridgebot, 07Upstream: Files larger than 1MiB not downloaded and relayed to IRC - https://phabricator.wikimedia.org/T363777#9755983 (10bd808) [23:40:13] 10Tool-bridgebot: Files larger than 1MiB not downloaded and relayed to IRC - https://phabricator.wikimedia.org/T363777#9755984 (10bd808) [23:41:23] (03PS1) 10Andrew Bogott: Added MANIFEST.in [openstack/horizon/designate-dashboard] (2024.1) - 10https://gerrit.wikimedia.org/r/1025479 [23:41:24] (03PS1) 10Andrew Bogott: Add default policy files [openstack/horizon/designate-dashboard] (2024.1) - 10https://gerrit.wikimedia.org/r/1025480 [23:41:24] (03PS1) 10Andrew Bogott: remove READMEFIRST, accidentally copied from the master branch [openstack/horizon/designate-dashboard] (2024.1) - 10https://gerrit.wikimedia.org/r/1025481 [23:41:43] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Added MANIFEST.in [openstack/horizon/designate-dashboard] (2024.1) - 10https://gerrit.wikimedia.org/r/1025479 (owner: 10Andrew Bogott) [23:41:55] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] remove READMEFIRST, accidentally copied from the master branch [openstack/horizon/designate-dashboard] (2024.1) - 10https://gerrit.wikimedia.org/r/1025481 (owner: 10Andrew Bogott) [23:42:00] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Add default policy files [openstack/horizon/designate-dashboard] (2024.1) - 10https://gerrit.wikimedia.org/r/1025480 (owner: 10Andrew Bogott) [23:58:39] (03PS1) 10Andrew Bogott: Added MANIFEST.in [openstack/horizon/designate-dashboard] - 10https://gerrit.wikimedia.org/r/1025483 [23:58:39] (03PS1) 10Andrew Bogott: Remove ref to designatedashboard/designatedashboard.scss [openstack/horizon/designate-dashboard] - 10https://gerrit.wikimedia.org/r/1025484 [23:59:19] (03PS1) 10Andrew Bogott: Remove ref to designatedashboard/designatedashboard.scss [openstack/horizon/designate-dashboard] (2024.1) - 10https://gerrit.wikimedia.org/r/1025485 [23:59:35] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Added MANIFEST.in [openstack/horizon/designate-dashboard] - 10https://gerrit.wikimedia.org/r/1025483 (owner: 10Andrew Bogott) [23:59:40] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Remove ref to designatedashboard/designatedashboard.scss [openstack/horizon/designate-dashboard] - 10https://gerrit.wikimedia.org/r/1025484 (owner: 10Andrew Bogott) [23:59:51] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Remove ref to designatedashboard/designatedashboard.scss [openstack/horizon/designate-dashboard] (2024.1) - 10https://gerrit.wikimedia.org/r/1025485 (owner: 10Andrew Bogott)