[01:13:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [01:21:30] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:13:49] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [05:21:30] FIRING: CloudVPSDesignateLeaks: Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:12:32] 06cloud-services-team, 10Data-Services, 10VPS-Projects, 06Data-Engineering, 10WMDE-References-FocusArea: Requesting Cloud VPS access to NFS mount /public/dumps - https://phabricator.wikimedia.org/T333549#10252851 (10awight) [08:12:51] (03update) 10sstefanova: [lima-kilo] cache disk for caching container images [repos/cloud/toolforge/lima-kilo] (refactor_in_preparation_for_cache) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/201 (owner: 10raymond-ndibe) [08:36:58] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-idp-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [08:41:51] (03update) 10sstefanova: [lima-kilo] cache disk for caching container images [repos/cloud/toolforge/lima-kilo] (refactor_in_preparation_for_cache) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/201 (owner: 10raymond-ndibe) [09:13:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [09:21:30] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:23:48] RESOLVED: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [09:43:12] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Cloud VPS: 2024-10-22 cloud-wide puppet problem related to java update - https://phabricator.wikimedia.org/T377803#10253322 (10aborrero) 05In progress→03Resolved a:03aborrero [10:03:19] 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Decision Request - How to do the Cloud VPS VXLAN/IPv6 migration - https://phabricator.wikimedia.org/T377467#10253404 (10aborrero) >>! In T377467#10251408, @bd808 wrote: > > Thanks for digging those scripts up @aborrero. If... [10:15:27] 10cloud-services-team (Hardware), 10Cloud-VPS: wmcs codfw hardware changes proposal - https://phabricator.wikimedia.org/T377568#10253415 (10aborrero) >>! In T377568#10247882, @RobH wrote: > I just checked with Valerie @ eqiad and she is sending some spare 3200 speed 32GB dimms for some other upgrades in codfw,... [10:19:08] 10cloud-services-team (Hardware), 10Cloud-VPS, 06DC-Ops, 10ops-codfw: cloudcontrol2006-dev struggling with memory - https://phabricator.wikimedia.org/T370401#10253422 (10aborrero) a:03Papaul hey @Papaul and/or @Jhancock.wm per https://phabricator.wikimedia.org/T377568#10247882 you should be receiving mem... [10:19:25] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw: cloudcontrol2006-dev struggling with memory - https://phabricator.wikimedia.org/T370401#10253430 (10aborrero) [10:22:52] 10cloud-services-team (Hardware), 10Cloud-VPS: wmcs codfw hardware changes proposal - https://phabricator.wikimedia.org/T377568#10253439 (10aborrero) >>! In T377568#10247706, @RobH wrote: >> cloudcontrol2008-dev: give back to spares > This was just purchased on 2023-07-24, does WMC have no use? We no long... [10:28:07] 14cloud-services-team (Kanban): openstack: mirror cloudrabbit setup from eqiad1 to codfw1dev - https://phabricator.wikimedia.org/T377934 (10aborrero) 03NEW [10:29:29] 14cloud-services-team (Kanban): openstack: mirror cloudrabbit setup from eqiad1 to codfw1dev - https://phabricator.wikimedia.org/T377934#10253467 (10aborrero) p:05Triage→03Low [10:32:10] 10cloud-services-team (Hardware), 10Cloud-VPS: wmcs codfw hardware changes proposal - https://phabricator.wikimedia.org/T377568#10253471 (10aborrero) [10:32:23] 14cloud-services-team (Kanban): openstack: mirror cloudrabbit setup from eqiad1 to codfw1dev - https://phabricator.wikimedia.org/T377934#10253486 (10aborrero) per {T377568} we may want to use spare hardware for this: * cloudcontrol2008-dev https://netbox.wikimedia.org/dcim/devices/4771/ -- procured in T341239 --... [11:34:10] (03PS1) 10Awight: Follow cldr change renaming data files "CldrMain*" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1082452 (https://phabricator.wikimedia.org/T377939) [11:34:58] (03CR) 10Awight: "(Note that the dependency patch is still unmerged.)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1082452 (https://phabricator.wikimedia.org/T377939) (owner: 10Awight) [11:54:17] 10Quarry, 10Toolforge, 10ChangeProp, 06collaboration-services, and 10 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#10253657 (10jijiki) [12:10:09] 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops: Cleanup: Wikitech code leftovers - https://phabricator.wikimedia.org/T371378#10253730 (10jijiki) p:05Triage→03Low [12:11:33] 06cloud-services-team, 10Quarry, 10Toolforge, 10ChangeProp, and 11 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#10253695 (10jijiki) [12:29:40] 06cloud-services-team, 10Cloud-VPS: neutron: clarify why DNS extension is not enabled - https://phabricator.wikimedia.org/T377740#10253753 (10aborrero) We are using our own code, see: * https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/b1501fd1f6caded20b99eaf9fadb80b627c43804/modules/openstac... [12:34:29] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, 06SRE: openstack: work out IPv6 and designate integration - https://phabricator.wikimedia.org/T374715#10253766 (10aborrero) 05In progress→03Stalled Turns out, to enable PTR creation support, per {T377740} we would need to eit... [13:21:30] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:35:01] 06cloud-services-team, 10Quarry, 10Toolforge, 10ChangeProp, and 11 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#10254048 (10bking) Forgive the drive-by comment, but at the 6-month anniversary of this ticket, it might be worth che... [13:43:29] 10Cloud-VPS (Project-requests), 06Data-Platform-SRE, 10Wikidata, 10Wikidata-Query-Service: Request creation of wikiqlever VPS project - https://phabricator.wikimedia.org/T377655#10254081 (10Physikerwelt) >>! In T377655#10250552, @bking wrote: > Memory: At least 64 GB RAM qlever seems to use quite a bit mo... [15:02:06] 06Toolforge-standards-committee: Reset members and owners for toolforge-standards-committee@lists.wikimedia.org - https://phabricator.wikimedia.org/T375134#10254494 (10bd808) [15:04:09] 06Toolforge-standards-committee: Reset members and owners for toolforge-standards-committee@lists.wikimedia.org - https://phabricator.wikimedia.org/T375134#10254510 (10bd808) > [x] Subscribe new committee members to the list I sent invitations to join the list to the Developer account email addresses for each c... [15:44:00] (03update) 10raymond-ndibe: [lima-kilo] minor project refactor [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/198 [15:51:19] (03update) 10raymond-ndibe: [lima-kilo] minor project refactor [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/198 [15:52:01] (03update) 10raymond-ndibe: [lima-kilo] minor project refactor [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/198 [16:00:12] (03update) 10raymond-ndibe: [lima-kilo] cache disk for caching container images [repos/cloud/toolforge/lima-kilo] (refactor_in_preparation_for_cache) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/201 [16:01:51] (03update) 10raymond-ndibe: [lima-kilo] configure high-availability [repos/cloud/toolforge/lima-kilo] (add_cache_disk) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/189 (https://phabricator.wikimedia.org/T374585) [16:03:06] (03update) 10raymond-ndibe: [lima-kilo] test k8s 1.28 upgrade [repos/cloud/toolforge/lima-kilo] (configure_high_availability) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) [16:22:39] (03approved) 10fnegri: [lima-kilo] test k8s 1.28 upgrade [repos/cloud/toolforge/lima-kilo] (configure_high_availability) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) (owner: 10raymond-ndibe) [16:31:04] 10Tool-video-answer-tool, 06Future-Audiences: [Bug] Article title attribution for Prawo Jazdy - https://phabricator.wikimedia.org/T377735#10255166 (10derenrich) 05Open→03Resolved a:03derenrich [16:34:03] (03update) 10raymond-ndibe: [lima-kilo] minor project refactor [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/198 [16:34:20] 10VPS-Projects, 10fundraising-tech-ops, 10Puppet (Puppet 7.0): Update puppet civicrm-prototype puppetmaster - https://phabricator.wikimedia.org/T361595#10255182 (10Dwisehaupt) Hi, sorry about this. I did make progress and then promptly got placed on a jury where I have been for the last month. I'll plan to l... [16:42:57] (03update) 10raymond-ndibe: [lima-kilo] cache disk for caching container images [repos/cloud/toolforge/lima-kilo] (refactor_in_preparation_for_cache) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/201 [16:44:31] (03update) 10raymond-ndibe: [lima-kilo] configure high-availability [repos/cloud/toolforge/lima-kilo] (add_cache_disk) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/189 (https://phabricator.wikimedia.org/T374585) [16:45:52] (03update) 10raymond-ndibe: [lima-kilo] test k8s 1.28 upgrade [repos/cloud/toolforge/lima-kilo] (configure_high_availability) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) [17:01:24] (03update) 10raymond-ndibe: [lima-kilo] configure high-availability [repos/cloud/toolforge/lima-kilo] (add_cache_disk) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/189 (https://phabricator.wikimedia.org/T374585) [17:02:08] (03update) 10raymond-ndibe: [lima-kilo] test k8s 1.28 upgrade [repos/cloud/toolforge/lima-kilo] (configure_high_availability) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) [17:03:07] 10Tool-ldap, 10Phabricator: https://ldap.toolforge.org/ integration assumes that `cn` and `uid` are equivalent - https://phabricator.wikimedia.org/T376769#10255342 (10Legoktm) 05Open→03Resolved The Phabricator part of this was deployed, so we should be all set here! [17:03:42] 10Tool-ldap, 10Phabricator (2024-10-22): https://ldap.toolforge.org/ integration assumes that `cn` and `uid` are equivalent - https://phabricator.wikimedia.org/T376769#10255345 (10Pppery) [17:05:14] (03update) 10raymond-ndibe: [lima-kilo] configure high-availability [repos/cloud/toolforge/lima-kilo] (add_cache_disk) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/189 (https://phabricator.wikimedia.org/T374585) [17:05:47] (03update) 10raymond-ndibe: [lima-kilo] test k8s 1.28 upgrade [repos/cloud/toolforge/lima-kilo] (configure_high_availability) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) [17:07:48] (03update) 10raymond-ndibe: [lima-kilo] test k8s 1.28 upgrade [repos/cloud/toolforge/lima-kilo] (configure_high_availability) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) [17:08:55] 10Cloud-VPS (Project-requests), 06Data-Platform-SRE, 10Wikidata, 10Wikidata-Query-Service: Request creation of wikiqlever VPS project - https://phabricator.wikimedia.org/T377655#10255379 (10Seppl2013) it's 4 TB of SSD and 128 GB RAM for starters which will give you a single QLever instance to be indexable.... [17:09:47] (03update) 10raymond-ndibe: [lima-kilo] configure high-availability [repos/cloud/toolforge/lima-kilo] (add_cache_disk) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/189 (https://phabricator.wikimedia.org/T374585) [17:12:08] (03open) 10gjg: Add BDC-Implementation to -fundraising announces [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/48 [17:15:34] 10cloud-services-team (Hardware), 10Cloud-VPS: wmcs codfw hardware changes proposal - https://phabricator.wikimedia.org/T377568#10255408 (10RobH) I think re-purposing these for rabbitmq is better than 'spares' which tend to age out and never get used. @wiki_willy: Would you agree with this? Basically WMCS... [17:21:30] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:27:23] (03update) 10gjg: Add BDC-Implementation to -fundraising announces [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/48 [17:37:31] (/topic should add Gitlab to list :) ) [18:17:23] 10cloud-services-team (Hardware), 10Cloud-VPS: wmcs codfw hardware changes proposal - https://phabricator.wikimedia.org/T377568#10255711 (10wiki_willy) Yup, agreed. If the servers can be reallocated for something else that is currently needed, I think it makes more sense to just repurpose them vs keeping them... [20:03:11] greg-g: {{done}} :) [20:03:30] bd808: <3 [20:16:47] 06cloud-services-team, 10Data-Services, 06Data Products, 06Data-Engineering, and 2 others: Hide rows in the globalblocks table when the associated globaluser row has gu_hidden_level as not 0 - https://phabricator.wikimedia.org/T371488#10256132 (10Ottomata) @Milimetric @BTullis [[ https://docs.google.com/sp... [20:25:20] 06cloud-services-team, 10Data-Services, 06Data Products, 06Data-Engineering, and 2 others: Hide rows in the globalblocks table when the associated globaluser row has gu_hidden_level as not 0 - https://phabricator.wikimedia.org/T371488#10256180 (10Dreamy_Jazz) 05Open→03Resolved a:03Dreamy_Jazz Thi... [20:31:22] (03merge) 10bd808: Add BDC-Implementation to -fundraising announces [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/48 (owner: 10gjg) [21:09:15] 10Tool-ldap, 10Phabricator (2024-10-22): https://ldap.toolforge.org/ integration assumes that `cn` and `uid` are equivalent - https://phabricator.wikimedia.org/T376769#10256481 (10bd808) >>! In T376769#10255342, @Legoktm wrote: > The Phabricator part of this was deployed, so we should be all set here! Tha... [21:21:30] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:47:36] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate mwv-builder-03.mediawiki-vagrant.eqiad.wmflabs is about to expire in 1d 23h 58m 34s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [22:56:10] (03update) 10raymond-ndibe: [lima-kilo] minor project refactor [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/198 [23:17:16] (03update) 10raymond-ndibe: [lima-kilo] minor project refactor [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/198 [23:17:37] (03update) 10raymond-ndibe: [lima-kilo] cache disk for caching container images [repos/cloud/toolforge/lima-kilo] (refactor_in_preparation_for_cache) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/201 [23:18:05] (03update) 10raymond-ndibe: [lima-kilo] configure high-availability [repos/cloud/toolforge/lima-kilo] (add_cache_disk) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/189 (https://phabricator.wikimedia.org/T374585) [23:18:28] (03update) 10raymond-ndibe: [lima-kilo] test k8s 1.28 upgrade [repos/cloud/toolforge/lima-kilo] (configure_high_availability) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) [23:28:07] 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Decision Request - How to do the Cloud VPS VXLAN/IPv6 migration - https://phabricator.wikimedia.org/T377467#10256868 (10Dzahn) bd808 wrote: > The main difference I can see this time is that the migration will be introducing...