[00:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:34:15] 06cloud-services-team, 10Data-Services, 06Data-Persistence: pt-heartbeat updates table even if read_only=1 - https://phabricator.wikimedia.org/T381690#10392631 (10Marostegui) I will remove the tag but I will keep subscribed. [08:34:34] 06cloud-services-team, 10Data-Services: pt-heartbeat updates table even if read_only=1 - https://phabricator.wikimedia.org/T381690#10392634 (10Marostegui) [09:54:48] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Alert for "TooManyCloud*Down" are firing when they should not - https://phabricator.wikimedia.org/T381807#10392854 (10fnegri) 05In progress→03Resolved [10:10:55] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Trove DB full - https://phabricator.wikimedia.org/T381745#10392898 (10Magnus) @rook Thanks, the project is called `baglama2`. At the moment the volume size shows 60GB, which I suggest to increast to 100GB. The two large tables in the DB are 7.4GB and 16.3GB... [11:38:12] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission cloudcephmon100[1-3].eqiad.wmnet - https://phabricator.wikimedia.org/T380893#10393186 (10VRiley-WMF) [11:41:08] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission cloudcephmon100[1-3].eqiad.wmnet - https://phabricator.wikimedia.org/T380893#10393191 (10VRiley-WMF) Hey @andrew I was able to do the first unit, however, when I was running the script on the other 2 devices, it s... [12:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:05:43] 06cloud-services-team, 10Cloud-VPS, 06collaboration-services, 10Continuous-Integration-Infrastructure, and 2 others: Future testing-infra growth on cloud-vps - https://phabricator.wikimedia.org/T381419#10393737 (10LSobanski) Zuul workers post-migration are also in scope. [14:22:50] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Trove DB full - https://phabricator.wikimedia.org/T381745#10393790 (10Andrew) +1 Resizing may or may not be graceful but has worked for me recently [14:32:01] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Trove DB full - https://phabricator.wikimedia.org/T381745#10393822 (10rook) C'est fait ` root@cloudcontrol1007:~# openstack database quota show baglama2 +-----------+--------+----------+-------+ | Resource | In Use | Reserved | Limit | +-----------+----... [14:32:04] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Trove DB full - https://phabricator.wikimedia.org/T381745#10393823 (10rook) 05Open→03Resolved a:03rook [15:34:04] 06cloud-services-team, 06DC-Ops, 10ops-codfw, 06SRE: PowerSupplyFailure Power Supply - Status - issue on cloudbackup2003:9290 - https://phabricator.wikimedia.org/T380479#10394077 (10Andrew) Timing is flexible although I'd like to do a graceful shutdown and check after the fact. Can this wait until next wee... [15:48:50] 06cloud-services-team, 06DC-Ops, 10ops-codfw, 06SRE: PowerSupplyFailure Power Supply - Status - issue on cloudbackup2003:9290 - https://phabricator.wikimedia.org/T380479#10394157 (10Jhancock.wm) It can wait until you're back from offsite. Enjoy =) [16:08:04] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Trove DB full - https://phabricator.wikimedia.org/T381745#10394214 (10Magnus) 05Resolved→03Open Sorry, but I don't think it's done yet. Horizon still tells me it's 60GB. ALso, the DB server has stopped responding: ` Used command: /usr/bin/ssh -v -N -S... [16:30:29] 10VPS-project-Codesearch, 06collaboration-services: Graduate codesearch to production - https://phabricator.wikimedia.org/T268199#10394274 (10Dzahn) Just dumping some data here for a comparison. The current codesearch VM, codesearch9 is a **g4.cores4.ram8.disk20** running Debian bookworm. [16:59:13] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Trove DB full - https://phabricator.wikimedia.org/T381745#10394366 (10rook) I see the volume as listed as 60GB in horizon. To verify did you try to resize the db in horizon by going to databases > instances > select the dropdown menu on the database > resize... [17:16:29] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10394512 (10Andrew) I was just looking over @Sarai-WMF's shoulder and wikitech is for some reason not editable for her. I gave her the comfirmed user right and that didn't change anything. @... [17:33:01] 06cloud-services-team, 10Tool-python-toolforge, 10Toolforge: Add support for Python 3.13 - https://phabricator.wikimedia.org/T381899#10394620 (10Ranjithraj) [17:41:49] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10394651 (10taavi) > I was just looking over @Sarai-WMF's shoulder and wikitech is for some reason not editable for her. What exactly does this mean? Is there some permission error on the edi... [17:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:06:16] 06cloud-services-team, 10Tool-python-toolforge, 10Toolforge: Add support for Python 3.13 - https://phabricator.wikimedia.org/T381899#10394795 (10Aklapper) Please do not use LLMs to create any tasks here. Thanks a lot. [18:12:09] 06cloud-services-team, 10Toolforge: Add support for Python 3.13 - https://phabricator.wikimedia.org/T381899#10394800 (10LucasWerkmeister) “Classic” webservice images – the `python3.9`, `python3.11`, etc. types – are backed by Debian, so historically they’ve become available with each new Debian release. The re... [18:27:19] 10PAWS: upgrade jupyterlab - https://phabricator.wikimedia.org/T381907 (10rook) 03NEW [18:28:18] vivian-rook opened https://github.com/toolforge/paws/pull/469 [18:44:41] 10PAWS: upgrade jupyterlab - https://phabricator.wikimedia.org/T381907#10394879 (10rook) https://github.com/toolforge/paws/pull/469 [19:03:22] 10PAWS: upgrade jupyterlab - https://phabricator.wikimedia.org/T381907#10394919 (10rook) 05Open→03Resolved [19:03:23] vivian-rook closed https://github.com/toolforge/paws/pull/469 [19:05:31] 10Cloud-VPS (Quota-requests): cloud VPS project devtools - quota increase request for storage volumes - https://phabricator.wikimedia.org/T381911 (10Dzahn) 03NEW [19:13:09] 10Cloud-VPS (Quota-requests): cloud VPS project devtools - quota increase request for storage volumes - https://phabricator.wikimedia.org/T381911#10394936 (10Dzahn) Thinking about it, this could also be granted in the project called "codesearch". It would basically make no difference for us in which project we h... [19:16:40] 10Cloud-VPS (Quota-requests): cloud VPS project codesearch or devtools - quota increase request for storage volumes - https://phabricator.wikimedia.org/T381911#10394949 (10Dzahn) [19:16:55] 10Cloud-VPS (Quota-requests): cloud VPS project codesearch or devtools - quota increase request for storage volumes - https://phabricator.wikimedia.org/T381911#10394952 (10Dzahn) [19:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:23:10] 10Cloud-VPS (Quota-requests): cloud VPS project codesearch or devtools - quota increase request for storage volumes - https://phabricator.wikimedia.org/T381911#10394979 (10Dzahn) justification: demo/testing/staging for T268199 [19:23:32] 10Cloud-VPS (Quota-requests), 06collaboration-services: cloud VPS project codesearch or devtools - quota increase request for storage volumes - https://phabricator.wikimedia.org/T381911#10394994 (10Dzahn) [19:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:32:10] 10Cloud-VPS (Quota-requests), 06collaboration-services: cloud VPS project codesearch or devtools - quota increase request for storage volumes - https://phabricator.wikimedia.org/T381911#10395031 (10Dzahn) regarding the actual size: The data points I have right now are: - gerrit has about 2900 repos - gitlab a... [20:41:02] 06cloud-services-team, 10Toolforge: Toolforge Build Service does not support .python-version - https://phabricator.wikimedia.org/T381923 (10LucasWerkmeister) 03NEW [21:11:53] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission cloudcephmon100[1-3].eqiad.wmnet - https://phabricator.wikimedia.org/T380893#10395298 (10Jhancock.wm) An exception occurred: KeyError: 'device_name' Traceback (most recent call last): File "/srv/netbox/customscrip... [22:46:43] 06cloud-services-team, 10Toolforge: Add support for Python 3.13 - https://phabricator.wikimedia.org/T381899#10395552 (10Ranjithraj) > In particular, what is "the platform"? I mean Toolforge PAAS here [22:47:13] 06cloud-services-team, 10Toolforge: Add support for Python 3.13 - https://phabricator.wikimedia.org/T381899#10395553 (10Ranjithraj) >>! In T381899#10394795, @Aklapper wrote: > Please do not use LLMs to create any tasks here. Thanks a lot. New here. Thank you very much, noted. [23:16:09] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10395581 (10GTrang) It appears that Phase 2 did not actually happen on November 30 as planned. In particular, the Wikitech accounts with mismatching S...