[02:14:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:24:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:00:56] FIRING: SystemdUnitDown: The service unit wikitech_run_jobs.service is in failed status on host cloudweb1003. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudweb1003 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [05:05:56] RESOLVED: [2x] SystemdUnitDown: The service unit wikitech_run_jobs.service is in failed status on host cloudweb1003. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [05:32:01] 14Grid-Engine-to-K8s-Migration, 10Tools, 06All-and-every-Wikisource: Migrate phetools from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319965#9788793 (10Soda) >>! In T319965#9788348, @Epigeneticist wrote: >>>! In T319965#9787909, @Soda wrote: >>>>! In T319965#9787724, @E... [06:12:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:17:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:22:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:27:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:54:24] 14Grid-Engine-to-K8s-Migration, 10Tools: Migrate multichill from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319912#9788857 (10Aklapper) [07:40:11] 06cloud-services-team, 10Toolforge: [horizon,swift] When accessing a private file without authenticating first you get a 500 error - https://phabricator.wikimedia.org/T364706 (10dcaro) 03NEW [07:42:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:47:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:50:08] 06cloud-services-team, 10Toolforge: [horizon,swift] When accessing any file (public/private) without authenticating first you get a 500 error - https://phabricator.wikimedia.org/T364706#9788983 (10dcaro) [07:52:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:56:08] 06cloud-services-team, 10Toolforge: [horizon,swift] When accessing any file (public/private) without authenticating first you get a 500 error - https://phabricator.wikimedia.org/T364706#9788990 (10taavi) ` [Mon May 13 07:48:46.507418 2024] [wsgi:error] [pid 10:tid 139706677188352] [remote 208.80.154.150:34242]... [07:57:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:32:22] 10Toolforge (Toolforge iteration 09), 07Upstream: [builds-builder] golang based images get infinite nested loops for procfile entries - https://phabricator.wikimedia.org/T363417#9789114 (10dcaro) a:03dcaro [08:32:42] 10Toolforge (Toolforge iteration 09), 07Upstream: [builds-builder] golang based images get infinite nested loops for procfile entries - https://phabricator.wikimedia.org/T363417#9789112 (10dcaro) Flagging as upstream to check in on it eventually [08:32:54] 10Toolforge (Toolforge iteration 09), 07Upstream: [builds-builder] golang based images get infinite nested loops for procfile entries - https://phabricator.wikimedia.org/T363417#9789116 (10dcaro) 05In progress→03Stalled [08:44:49] (03approved) 10aborrero: [oapi-spec] add oapi-server to gateway [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/17 [08:46:57] 10Tool-spacemedia, 10Toolforge: Toolforge jobs logs -f almost always ends in error - https://phabricator.wikimedia.org/T364468#9789194 (10dcaro) This should have reduced the error rate somewhat, if the job returns any logs, it will not timeout (it will still timeout if the job does not return anything for a wh... [08:47:04] 10Wikibugs: Link to Phabricator tasks attached to GitLab MRs - https://phabricator.wikimedia.org/T364719 (10taavi) 03NEW [08:47:13] 10Toolforge: `toolforge jobs logs -f` crashes after a while with internal k8s api errors - https://phabricator.wikimedia.org/T359953#9789212 (10dcaro) This should have reduced the error rate somewhat, if the job returns any logs, it will not timeout (it will still timeout if the job does not return anything for... [09:46:59] 14cloud-services-team (FY2022/2023-Q3), 10Cloud-VPS: upgrade cloud-vps openstack to Openstack version 'Zed' - https://phabricator.wikimedia.org/T323086#9789526 (10taavi) [09:47:01] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: Migrate Cloud VPS to Neutron Open vSwitch agent - https://phabricator.wikimedia.org/T326373#9789525 (10taavi) [10:20:24] 06cloud-services-team, 10Cloud-VPS: Migrate Cloud VPS instances to VXLAN based networks - https://phabricator.wikimedia.org/T364725 (10taavi) 03NEW [10:20:26] 06cloud-services-team, 10Cloud-VPS: Migrate Cloud VPS instances to VXLAN based networks - https://phabricator.wikimedia.org/T364725#9789623 (10taavi) [10:20:29] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: Migrate Cloud VPS to Neutron Open vSwitch agent - https://phabricator.wikimedia.org/T326373#9789624 (10taavi) [10:22:55] 06cloud-services-team, 10Cloud-VPS: Migrate Cloud VPS instances to VXLAN based networks - https://phabricator.wikimedia.org/T364725#9789656 (10taavi) [10:22:56] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: Use BGP to announce VM ranges from cloudnet to cloudgw - https://phabricator.wikimedia.org/T358868#9789655 (10taavi) [10:23:01] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: Deploy OVS test setup in codfw1dev - https://phabricator.wikimedia.org/T358761#9789657 (10taavi) [10:25:11] 06cloud-services-team, 06Infrastructure-Foundations, 10netops, 06SRE: CloudVPS: enable BGP in the neutron transport network - https://phabricator.wikimedia.org/T245606#9789660 (10taavi) 05Stalled→03Declined Closing this in favour of the slightly different approach in {T358868} that's likely going t... [10:26:48] 06cloud-services-team, 10Cloud-VPS: Migrate Cloud VPS instances to VXLAN based networks - https://phabricator.wikimedia.org/T364725#9789672 (10taavi) [10:26:49] 06cloud-services-team, 10Cloud-VPS: CloudVPS: research VXLAN implementation for neutron - https://phabricator.wikimedia.org/T248881#9789671 (10taavi) [10:26:51] 06cloud-services-team, 10Cloud-VPS, 07Epic: CloudVPS: networking improvements - https://phabricator.wikimedia.org/T244727#9789673 (10taavi) [10:28:35] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, 07Epic: CloudVPS: network architecture - https://phabricator.wikimedia.org/T209460#9789669 (10taavi) 05Open→03Resolved Closing this task since I don't see a clear end goal here. Current ongoing and planned work is already... [10:29:48] (03update) 10dcaro: [oapi-spec] add oapi-server to gateway [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/17 [10:31:11] (03update) 10dcaro: [oapi-spec] add oapi-server to gateway [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/17 [10:31:39] 06cloud-services-team, 10Cloud-VPS, 07Epic: CloudVPS: networking improvements - https://phabricator.wikimedia.org/T244727#9789676 (10taavi) 05Open→03Resolved Closing as I don't see any actionable end goal here. Individual projects are already tracked in their own tasks. [10:43:54] 10Tool-spacemedia, 10Toolforge: Toolforge jobs logs -f almost always ends in error - https://phabricator.wikimedia.org/T364468#9789719 (10Don-vip) Thank you! I'll test the change this afternoon. [11:18:47] 10cloud-services-team (FY2023/2024-Q3-Q4), 10MediaWiki-extensions-CentralAuth, 10MediaWiki-Platform-Team (Radar), 10MW-1.43-notes (1.43.0-wmf.5; 2024-05-14), 13Patch-For-Review: Drop gu_salt from globaluser - https://phabricator.wikimedia.org/T364435#9789790 (10larissagaulia) [11:43:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:45:30] 14Grid-Engine-to-K8s-Migration, 10Tools, 06All-and-every-Wikisource: Migrate phetools from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319965#9789848 (10Epigeneticist) >>! In T319965#9788793, @Soda wrote: >>>! In T319965#9788348, @Epigeneticist wrote: >>>>! In T319965#97... [11:53:05] 10Quarry, 10Toolforge, 10ChangeProp, 06collaboration-services, and 10 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9789858 (10MoritzMuehlenhoff) Redict is now packaged in Debian: https://tracker.debian.org/pkg/redict [11:53:46] 10Quarry, 10Toolforge, 10ChangeProp, 06collaboration-services, and 10 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9789859 (10MoritzMuehlenhoff) [11:58:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:03:51] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 [12:04:23] 10Cloud-VPS (Quota-requests): Request temporary storage quota increase for project iiab for migration to bookworm image - https://phabricator.wikimedia.org/T361946#9789869 (10taavi) [12:05:11] 10Cloud-VPS: Drop 68.10.in-addr.arpa. from Designate - https://phabricator.wikimedia.org/T361220#9789872 (10taavi) @aborrero can you think of any reason not to delete that zone? [12:08:27] 10Cloud-VPS: Drop 68.10.in-addr.arpa. from Designate - https://phabricator.wikimedia.org/T361220#9789877 (10aborrero) that `cloudinstances2b-gw-compat.svc.eqiad.wmflabs` entry suggests to me that this is a leftover from a previous migration. Maybe go ahead and delete it. I doubt anything will break (famous last... [12:08:59] 06cloud-services-team, 10Cloud-VPS: Add a nicer interface in Spicerack/wmcs-cookbooks to downtime Cloud VPS instances - https://phabricator.wikimedia.org/T364733 (10taavi) 03NEW [12:10:04] 10Cloud-VPS: Support downtiming metricsinfra alerts in wmcs-cookbooks - https://phabricator.wikimedia.org/T360932#9789894 (10taavi) 05Open→03Resolved I created {T364733} to improve the API somewhat, but silencing things works now so closing this one. [12:22:23] (03update) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/4 [12:22:59] 10Wikibugs: Link to Phabricator tasks attached to GitLab MRs - https://phabricator.wikimedia.org/T364719#9789939 (10Esanders) Duplicate of T337570? [12:23:40] 10Wikibugs: Link to Phabricator tasks attached to GitLab MRs - https://phabricator.wikimedia.org/T364719#9789945 (10taavi) No, that is for the GitLab interface and this is for the Wikibugs IRC bot. [12:35:12] (03update) 10dcaro: [oapi-spec] add oapi-server to gateway [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/17 [12:43:22] 10Tool-spacemedia, 10Toolforge: Toolforge jobs logs -f almost always ends in error - https://phabricator.wikimedia.org/T364468#9790004 (10Don-vip) No visible difference :( Is the change already live? ` 2024-05-13T12:40:54+00:00 [build-nsgzq] [INFO] --- compiler:3.13.0:compile (default-compile) @ spacemedia -... [12:51:39] 10Tool-spacemedia, 10Toolforge: Toolforge jobs logs -f almost always ends in error - https://phabricator.wikimedia.org/T364468#9790056 (10dcaro) >>! In T364468#9790004, @Don-vip wrote: > No visible difference :( Is the change already live? > Yep, it's out there, does the job stay silent for ~1min before the... [13:04:39] 06cloud-services-team, 10Toolforge: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#9790100 (10dcaro) Thanks for all the replies! >> If I'm a user, do I need to go gather the tool credentials to be able to access it's buckets? > They'd be... [13:07:31] 06cloud-services-team, 10Toolforge: [horizon,swift] When accessing any file (public/private) without authenticating first you get a 500 error - https://phabricator.wikimedia.org/T364706#9790105 (10dcaro) I had a quick look at the code, it seems that the anonymous user is not extended and it's falling back to t... [13:10:15] 06cloud-services-team, 10Toolforge: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#9790111 (10taavi) May I ask why are we are talking about tools interacting with Horizon? I would hope everything happens either via the APIs directly or via St... [13:12:15] 06cloud-services-team, 10Toolforge: [horizon,swift] When accessing any file (public/private) without authenticating first you get a 500 error - https://phabricator.wikimedia.org/T364706#9790121 (10taavi) How does Horizon expose these /api/swift URLs? I would assume they'd only be embedded in pages that do have... [13:12:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:13:11] 10Tool-spacemedia, 10Toolforge: Toolforge jobs logs -f almost always ends in error - https://phabricator.wikimedia.org/T364468#9790135 (10Don-vip) The compile step is the slowest, and the longest without progress logs. It spends about ~35-50s without log (37s in this example below: ` 2024-05-13T13:09:10+00:00... [13:17:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:21:32] 06cloud-services-team, 10Toolforge: [horizon,swift] When accessing any file (public/private) without authenticating first you get a 500 error - https://phabricator.wikimedia.org/T364706#9790216 (10dcaro) >>! In T364706#9790121, @taavi wrote: > How does Horizon expose these /api/swift URLs? Just found that the... [13:22:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:23:09] 06cloud-services-team, 10Toolforge: [horizon,swift] When accessing any file (public/private) without authenticating first you get a 500 error - https://phabricator.wikimedia.org/T364706#9790223 (10dcaro) The private bucket has no such link though :/ {F52905661} Public bucket: {F52905703} [13:27:25] 06cloud-services-team, 10Toolforge: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#9790252 (10dcaro) >>! In T358496#9790111, @taavi wrote: > May I ask why are we are talking about tools interacting with Horizon? I would hope everything happen... [13:27:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:32:28] 06cloud-services-team, 10Toolforge: [horizon,swift] When accessing any file (public/private) without authenticating first you get a 500 error - https://phabricator.wikimedia.org/T364706#9790279 (10dcaro) The private one works also, but only if you use the url https://object.eqiad1.wikimediacloud.org/swift/v1/A... [13:34:51] 06cloud-services-team, 10Toolforge: [horizon,swift] When accessing any file (public/private) without authenticating first you get a 500 error - https://phabricator.wikimedia.org/T364706#9790285 (10dcaro) 05Open→03Invalid [13:36:29] 06cloud-services-team, 10Toolforge: [horizon,swift] When accessing any file (public/private) without authenticating first you get a 500 error - https://phabricator.wikimedia.org/T364706#9790293 (10dcaro) [13:38:46] 10Tool-spacemedia, 10Toolforge: Toolforge jobs logs -f almost always ends in error - https://phabricator.wikimedia.org/T364468#9790298 (10dcaro) Got it :), I was able to reproduce with ~30s of inactivity: ` tools.wm-lol@tools-bastion-13:~$ time toolforge jobs logs -f test 2024-05-13T13:25:18+00:00 [test-64dc8b... [13:55:40] 06cloud-services-team, 06Infrastructure-Foundations, 10Puppet-Infrastructure: Ownership confusion on cloud-local puppet servers - https://phabricator.wikimedia.org/T364492#9790341 (10Andrew) I'm now learned that new prod puppservers also use the 'gitpuppet' user. So eliminating that user will increase the di... [14:23:32] 06cloud-services-team, 10Toolforge: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#9790462 (10Andrew) >>! In T358496#9790100, @dcaro wrote: > > I'm still thinking on use the cases > > > This task is specifically tackling this one:... [14:42:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:43:42] 06cloud-services-team, 06Infrastructure-Foundations, 10Puppet-Infrastructure, 13Patch-For-Review: Ownership confusion on cloud-local puppet servers - https://phabricator.wikimedia.org/T364492#9790564 (10jbond) > Puppet 7 has some new ownership constraints which means that we can no longer investigate these... [14:47:19] 06cloud-services-team, 10Puppet-Infrastructure, 13Patch-For-Review: Ownership confusion on cloud-local puppet servers - https://phabricator.wikimedia.org/T364492#9790579 (10joanna_borun) [14:48:31] 06cloud-services-team, 06Infrastructure-Foundations, 10netops, 10ops-codfw, 06SRE: Create (or teach Andrew how to create) private connections+dns entries for new cloudcontrols - https://phabricator.wikimedia.org/T364559#9790593 (10cmooney) 05Open→03Resolved p:05Triage→03Medium >>! In T364559#... [14:52:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:04:04] (03reopen) 10bd808: This is a test of the gitlab irc reporter [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/33 [15:04:08] (03close) 10bd808: This is a test of the gitlab irc reporter [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/33 [15:07:43] 10Cloud Services Proposals: Decision request - kubernetes upgrade workgroup - https://phabricator.wikimedia.org/T363683#9790658 (10Raymond_Ndibe) Can we link any resources we already have (automations, cookbooks, instructions, etc) on how we handle k8s upgrade here too? k8s upgrade is easy on paper but I assume... [15:09:45] 10Cloud Services Proposals: Decision request - kubernetes upgrade workgroup - https://phabricator.wikimedia.org/T363683#9790665 (10Raymond_Ndibe) one of the resources we need to be aware of here https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Upgrading_Kubernetes [15:20:07] (03update) 10dcaro: [oapi-spec] add oapi-server to gateway [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/17 [15:35:35] 10Quarry: Shutdown quarry VMs - https://phabricator.wikimedia.org/T361470#9790810 (10rook) 05Open→03Resolved [15:36:44] 10Quarry: remove buster systems - https://phabricator.wikimedia.org/T364753 (10rook) 03NEW [15:36:50] 10Quarry: remove buster systems - https://phabricator.wikimedia.org/T364753#9790830 (10rook) [15:36:52] 10Quarry: Shutdown quarry VMs - https://phabricator.wikimedia.org/T361470#9790831 (10rook) [15:40:30] 10Tools: Tool:Panoviewer - Grid Engine web service cannot be reached. - https://phabricator.wikimedia.org/T354949#9790843 (10Ligliotoi) Hallo, https://panoviewer.toolforge.org/ does not work. Error 403 "Forbidden". Can somebody fixed it maybe? [15:46:01] 06cloud-services-team, 10Toolforge: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#9790877 (10dcaro) > This all seems correct, although I reiterate that the interesting part is the scope creation or management. We don't currently have a good... [16:00:14] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 [16:00:55] (03update) 10dcaro: [oapi-spec] add oapi-server to gateway [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/17 [16:15:38] 10Cloud Services Proposals: Decision request - kubernetes upgrade workgroup - https://phabricator.wikimedia.org/T363683#9791065 (10dcaro) [16:17:24] 10Cloud Services Proposals: Decision request - kubernetes upgrade workgroup - https://phabricator.wikimedia.org/T363683#9791073 (10dcaro) >>! In T363683#9790665, @Raymond_Ndibe wrote: > one of the resources we need to be aware of here https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Upgradin... [16:27:01] 10Toolforge (Toolforge iteration 09): [toolforge] Investigate authentication - https://phabricator.wikimedia.org/T363983#9791114 (10dcaro) [16:31:05] 10Cloud Services Proposals: Decision request - kubernetes upgrade workgroup - https://phabricator.wikimedia.org/T363683#9791133 (10Slst2020) Option 2 seems to me like the obviously good choice :) On a first read, I was under the impression that the working group would exist only until we catch up, but from opti... [16:36:05] 10Cloud Services Proposals: Decision request - kubernetes upgrade workgroup - https://phabricator.wikimedia.org/T363683#9791169 (10dcaro) >>! In T363683#9791133, @Slst2020 wrote: > Option 2 seems to me like the obviously good choice :) > > On a first read, I was under the impression that the working group would... [16:37:18] 06cloud-services-team, 10Toolforge: Request for access for user dr0ptp4kt for 'admin' tool - https://phabricator.wikimedia.org/T364761 (10dr0ptp4kt) 03NEW [16:39:17] 06cloud-services-team, 10Toolforge: Request for access for user dr0ptp4kt for 'admin' tool - https://phabricator.wikimedia.org/T364761#9791199 (10taavi) [16:47:29] 06cloud-services-team, 10Toolforge: Request for access for user dr0ptp4kt for 'admin' tool - https://phabricator.wikimedia.org/T364761#9791226 (10bd808) [16:51:08] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: Use BGP to announce VM ranges from cloudnet to cloudgw - https://phabricator.wikimedia.org/T358868#9791232 (10cmooney) Happy to discuss. I think if we are doing this it makes sense to do the cloudgw <-> cloudsw BGP at the same time (we will need to creat... [16:52:41] (03update) 10dcaro: [oapi-spec] add oapi-server to gateway [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/17 [16:53:48] 06cloud-services-team, 10Toolforge: Request for access for user dr0ptp4kt for 'admin' tool - https://phabricator.wikimedia.org/T364761#9791238 (10bd808) > The use is for correlation of request patterns in the data lake and those originating from tools. I don't have any objections to @dr0ptp4kt becoming a Tool... [17:06:20] 06cloud-services-team, 10Toolforge: Request for access for user dr0ptp4kt for 'admin' tool - https://phabricator.wikimedia.org/T364761#9791307 (10dr0ptp4kt) The immediate term thing I'm checking is query density for WDQS, in particular for scholarly article oriented queries as part of the WDQS graph split. Fo... [18:11:03] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/4 [18:11:09] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/4 [18:26:34] 10Wikibugs: Wikibugs' gitlab connector stops working without a strong sign of why - https://phabricator.wikimedia.org/T364490#9791672 (10bd808) >>! In T364490#9786658, @bd808 wrote: > Let's see what happens with the health-check band-aid in place for a while. `lang=shell-session $ kubectl get po | grep -E 'NAME... [18:43:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:48:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:53:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:03:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-cloudvps-puppetserver-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:04:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-meta-monitor-1 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:06:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance extdist-06 on project extdist - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:07:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance tf-bastion on project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:07:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance cvn-apache10 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:08:28] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-cloudvps-puppetserver-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:08:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance runner-1026 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:09:28] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-controller-2 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:10:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance bastion on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:12:28] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache10 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:13:28] FIRING: [5x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-cloudvps-puppetserver-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:13:28] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance runner-1021 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:14:28] FIRING: [6x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:14:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance project-proxy-puppetserver-1 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:16:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance clouddb-services-puppetserver-1 on project clouddb-services - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:18:28] FIRING: [12x] PuppetAgentNoResources: No Puppet resources found on instance cloud-cumin-03 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:18:28] FIRING: [6x] PuppetAgentNoResources: No Puppet resources found on instance runner-1021 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:19:28] FIRING: [8x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:20:28] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance bastion on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:23:28] FIRING: [17x] PuppetAgentNoResources: No Puppet resources found on instance cloud-cumin-03 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:23:28] FIRING: [7x] PuppetAgentNoResources: No Puppet resources found on instance runner-1021 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:24:28] FIRING: [9x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:27:28] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache10 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:27:43] (03update) 10raymond-ndibe: [jobs-api] support services in jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/71 [19:33:28] FIRING: [8x] PuppetAgentNoResources: No Puppet resources found on instance runner-1021 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:34:28] FIRING: [10x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:37:28] FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache10 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:38:28] FIRING: [11x] PuppetAgentNoResources: No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:54:18] 10Toolforge (Toolforge iteration 09): increase quota for services - https://phabricator.wikimedia.org/T364780 (10Raymond_Ndibe) 03NEW [19:57:29] (03PS1) 10Jforrester: Move Wikifunctions services into Services [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1031054 [20:02:48] (03reopen) 10pywikibugs: This is a test of the gitlab irc reporter [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/33 [20:08:31] (03open) 10bd808: gitlab: Show MR owner if different than user triggering event [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/40 [20:11:45] (03close) 10pywikibugs: This is a test of the gitlab irc reporter [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/33 [20:15:53] (03approved) 10bd808: gitlab: Show MR owner if different than user triggering event [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/40 [20:15:56] (03merge) 10bd808: gitlab: Show MR owner if different than user triggering event [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/40 [20:25:58] (03reopen) 10pywikibugs: This is a test of the gitlab irc reporter [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/33 (owner: 10bd808) [20:26:58] (03update) 10bd808: This is a test of the gitlab irc reporter [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/33 [20:27:03] 10Wikibugs: Wikibugs testing task - https://phabricator.wikimedia.org/T90594#9792143 (10CodeReviewBot) bd808 updated https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/33 This is a test of the gitlab irc reporter [20:40:40] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 [20:40:55] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 [20:46:27] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 [20:51:33] (03open) 10bd808: utils: Avoid read race between touch_healthz_file and check_healthz [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/41 [20:52:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance tf-bastion on project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:52:28] FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache10 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:53:28] FIRING: [17x] PuppetAgentNoResources: No Puppet resources found on instance cloud-cumin-03 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:53:28] FIRING: [11x] PuppetAgentNoResources: No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:54:28] FIRING: [10x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:55:28] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance bastion on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:56:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance extdist-06 on project extdist - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:57:28] FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache10 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:58:28] FIRING: [17x] PuppetAgentNoResources: No Puppet resources found on instance cloud-cumin-03 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:58:28] FIRING: [11x] PuppetAgentNoResources: No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:58:53] (03merge) 10bd808: utils: Avoid read race between touch_healthz_file and check_healthz [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/41 [20:59:28] FIRING: [10x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:03:28] FIRING: [17x] PuppetAgentNoResources: No Puppet resources found on instance cloud-cumin-03 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:03:28] FIRING: [11x] PuppetAgentNoResources: No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:04:28] FIRING: [10x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:04:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance project-proxy-puppetserver-1 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:05:28] RESOLVED: [2x] PuppetAgentNoResources: No Puppet resources found on instance bastion on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:06:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance clouddb-services-puppetserver-1 on project clouddb-services - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:08:28] FIRING: [17x] PuppetAgentNoResources: No Puppet resources found on instance cloud-cumin-03 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:09:28] FIRING: [10x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:12:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance cvn-nfs-1 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:12:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:13:28] FIRING: [10x] PuppetAgentNoResources: No Puppet resources found on instance runner-1021 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:18:28] RESOLVED: [13x] PuppetAgentNoResources: No Puppet resources found on instance cloud-cumin-03 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:18:28] FIRING: [8x] PuppetAgentNoResources: No Puppet resources found on instance runner-1022 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:19:28] RESOLVED: [5x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-alertmanager-3 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:20:55] 10Wikibugs: Wikibugs' gitlab connector stops working without a strong sign of why - https://phabricator.wikimedia.org/T364490#9792291 (10bd808) The [[https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/41|utils: Avoid read race between touch_healthz_file and check_healthz MR]] should have bee... [21:20:57] 10Wikibugs: Wikibugs' gitlab connector stops working without a strong sign of why - https://phabricator.wikimedia.org/T364490#9792291 (10bd808) The [[https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/41|utils: Avoid read race between touch_healthz_file and check_healthz MR]] should have bee... [21:22:56] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:23:28] RESOLVED: [5x] PuppetAgentNoResources: No Puppet resources found on instance runner-1025 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:24:22] 10Wikibugs: Link to Phabricator tasks attached to GitLab MRs - https://phabricator.wikimedia.org/T364719#9792298 (10bd808) 05Open→03In progress p:05Triage→03Medium a:03bd808 [21:38:34] (03update) 10raymond-ndibe: [jobs-cli] support services in jobs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/18 [21:41:35] (03update) 10raymond-ndibe: [jobs-cli] support services in jobs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/18 [21:52:54] (03update) 10raymond-ndibe: [toolforge-weld] move _display_message into toolforge weld [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/46 [21:53:14] (03update) 10raymond-ndibe: [toolforge-weld] move _display_message into toolforge weld [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/46 [21:56:24] (03update) 10raymond-ndibe: [toolforge-weld] move _display_message into toolforge weld [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/46 [21:56:55] (03update) 10raymond-ndibe: [toolforge-weld] move _display_message into toolforge weld [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/46 [23:12:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:13:27] (03open) 10bd808: gitlab: report linked tasks [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/42 [23:15:01] 10Wikibugs, 13Patch-For-Review: Link to Phabricator tasks attached to GitLab MRs - https://phabricator.wikimedia.org/T364719#9792673 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/42 gitlab: report linked tasks [23:16:50] (03close) 10pywikibugs: This is a test of the gitlab irc reporter [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/33 (owner: 10bd808) [23:16:56] 10Wikibugs: Wikibugs testing task - https://phabricator.wikimedia.org/T90594#9792677 (10CodeReviewBot) pywikibugs closed https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/33 This is a test of the gitlab irc reporter [23:18:10] (03merge) 10bd808: gitlab: report linked tasks [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/42 [23:18:25] 10Wikibugs, 13Patch-For-Review: Link to Phabricator tasks attached to GitLab MRs - https://phabricator.wikimedia.org/T364719#9792680 (10CodeReviewBot) bd808 merged https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/42 gitlab: report linked tasks [23:22:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:36:51] (03reopen) 10pywikibugs: This is a test of the gitlab irc reporter [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/33 (https://phabricator.wikimedia.org/T90594) (owner: 10bd808) [23:37:02] (03close) 10pywikibugs: This is a test of the gitlab irc reporter [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/33 (https://phabricator.wikimedia.org/T90594) (owner: 10bd808) [23:37:11] 10Wikibugs, 13Patch-For-Review: Wikibugs testing task - https://phabricator.wikimedia.org/T90594#9792736 (10CodeReviewBot) pywikibugs closed https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/33 This is a test of the gitlab irc reporter [23:37:43] 10Wikibugs, 13Patch-For-Review: Wikibugs testing task - https://phabricator.wikimedia.org/T90594#9792734 (10CodeReviewBot) pywikibugs reopened https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/33 This is a test of the gitlab irc reporter [23:38:10] 10Wikibugs: Link to Phabricator tasks attached to GitLab MRs - https://phabricator.wikimedia.org/T364719#9792737 (10bd808) 05In progress→03Resolved [23:59:40] 10Toolforge: Query appears to run for a longer time when invoked via toolforge jobs framework - https://phabricator.wikimedia.org/T363286#9792772 (10Huji) @taavi any ideas?