[00:09:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [00:16:41] 10Wikibugs, 13Patch-For-Review, 15User-bd808: Add support for alternate channels files to make testing/debugging easier - https://phabricator.wikimedia.org/T359202#9609712 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/12 Testing enhancements: settings... [00:17:36] 10Wikibugs, 13Patch-For-Review, 15User-bd808: Allow configuration of update announce channel - https://phabricator.wikimedia.org/T359228#9609714 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/12 Testing enhancements: settings for SAL announce and chann... [00:46:11] (03CR) 10BryanDavis: [C: 04-2] "testing channelfilter switch in wikibugs-testing deploy" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/1008016 (https://phabricator.wikimedia.org/T90594) (owner: 10BryanDavis) [00:51:56] (03CR) 10BryanDavis: [C: 04-2] "> testing channelfilter switch in wikibugs-testing deploy" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/1008016 (https://phabricator.wikimedia.org/T90594) (owner: 10BryanDavis) [01:02:12] (03CR) 10BryanDavis: [C: 04-2] "> > testing channelfilter switch in wikibugs-testing deploy" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/1008016 (https://phabricator.wikimedia.org/T90594) (owner: 10BryanDavis) [01:02:50] 10Wikibugs: wikibugs test bug part II - https://phabricator.wikimedia.org/T90594#9609754 (10bd808) test [01:24:48] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [03:09:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [03:47:39] 10Wikibugs, 15User-bd808: Filter debug log messages out of logs sent to disk - https://phabricator.wikimedia.org/T359230#9609846 (10bd808) 05Open→03In progress a:03bd808 [04:04:13] 10Wikibugs, 13Patch-For-Review, 15User-bd808: Filter debug log messages out of logs sent to disk - https://phabricator.wikimedia.org/T359230#9609853 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/13 Filter debug log messages out of logs sent to disk [05:25:04] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [05:43:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:48:41] (CloudVPSDesignateLeaks) firing: (4) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:53:41] (CloudVPSDesignateLeaks) firing: (4) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:53:56] (CloudVPSDesignateLeaks) firing: (4) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:58:42] (CloudVPSDesignateLeaks) resolved: (4) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:09:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [08:57:35] 10Toolforge (Toolforge iteration 07), 06cloud-services-team, 13Patch-For-Review, 15User-aborrero: Upgrade Toolforge Kubernetes to version 1.24 - https://phabricator.wikimedia.org/T307651#9610117 (10CodeReviewBot) aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/... [08:58:11] 05Grid-Engine-to-K8s-Migration: Migrate yapperbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320195#9610126 (10DavidTornheim) In researching the above jobs, I looked up the "uncurrenter", originally approved as: https://en.wikipedia.org/wiki/Wikipedia:Bots/Requests_f... [09:01:20] 05Grid-Engine-to-K8s-Migration: Migrate ganfilter from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357554#9610165 (10dcaro) >>! In T357554#9609612, @coldchrist wrote: > I'll plan on doing the switchover this weekend. Can I make the code changes first and the job changes sec... [09:04:31] (03PS4) 10Majavah: Use normal login flow for admin site [labs/striker] - 10https://gerrit.wikimedia.org/r/1009258 (https://phabricator.wikimedia.org/T284400) [09:04:33] (03PS1) 10Majavah: labsauth: Make groups readonly in admin [labs/striker] - 10https://gerrit.wikimedia.org/r/1009455 [09:05:32] (03CR) 10Majavah: [C: 03+2] Link to idm.wm.o for password resets [labs/striker] - 10https://gerrit.wikimedia.org/r/1009295 (https://phabricator.wikimedia.org/T174469) (owner: 10Majavah) [09:06:52] (03CR) 10Majavah: [C: 03+2] Consistently use 'user' icon for membership requests [labs/striker] - 10https://gerrit.wikimedia.org/r/1009242 (owner: 10Majavah) [09:07:38] (03CR) 10Majavah: [C: 03+2] Display membership requests in navbar for admins [labs/striker] - 10https://gerrit.wikimedia.org/r/1009243 (https://phabricator.wikimedia.org/T316832) (owner: 10Majavah) [09:07:49] (03CR) 10Majavah: [C: 03+2] Display number of open access requests in navbar [labs/striker] - 10https://gerrit.wikimedia.org/r/1009244 (https://phabricator.wikimedia.org/T316832) (owner: 10Majavah) [09:08:06] (03CR) 10Majavah: [C: 03+2] Stop sending notifications about access requests [labs/striker] - 10https://gerrit.wikimedia.org/r/1009245 (https://phabricator.wikimedia.org/T316832) (owner: 10Majavah) [09:08:19] (03Merged) 10jenkins-bot: Link to idm.wm.o for password resets [labs/striker] - 10https://gerrit.wikimedia.org/r/1009295 (https://phabricator.wikimedia.org/T174469) (owner: 10Majavah) [09:08:22] (03Merged) 10jenkins-bot: Consistently use 'user' icon for membership requests [labs/striker] - 10https://gerrit.wikimedia.org/r/1009242 (owner: 10Majavah) [09:08:51] (03Merged) 10jenkins-bot: Display membership requests in navbar for admins [labs/striker] - 10https://gerrit.wikimedia.org/r/1009243 (https://phabricator.wikimedia.org/T316832) (owner: 10Majavah) [09:09:02] (03Merged) 10jenkins-bot: Display number of open access requests in navbar [labs/striker] - 10https://gerrit.wikimedia.org/r/1009244 (https://phabricator.wikimedia.org/T316832) (owner: 10Majavah) [09:09:20] (03Merged) 10jenkins-bot: Stop sending notifications about access requests [labs/striker] - 10https://gerrit.wikimedia.org/r/1009245 (https://phabricator.wikimedia.org/T316832) (owner: 10Majavah) [09:09:26] (03CR) 10Majavah: [C: 03+2] Use normal login flow for admin site [labs/striker] - 10https://gerrit.wikimedia.org/r/1009258 (https://phabricator.wikimedia.org/T284400) (owner: 10Majavah) [09:09:33] (03CR) 10Majavah: [C: 03+2] labsauth: Make groups readonly in admin [labs/striker] - 10https://gerrit.wikimedia.org/r/1009455 (owner: 10Majavah) [09:09:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [09:10:31] (03Merged) 10jenkins-bot: Use normal login flow for admin site [labs/striker] - 10https://gerrit.wikimedia.org/r/1009258 (https://phabricator.wikimedia.org/T284400) (owner: 10Majavah) [09:11:44] (03CR) 10Majavah: [C: 03+2] kubernetes: Cleanup namespace handling [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1007955 (owner: 10Majavah) [09:12:03] (03Merged) 10jenkins-bot: labsauth: Make groups readonly in admin [labs/striker] - 10https://gerrit.wikimedia.org/r/1009455 (owner: 10Majavah) [09:16:39] (03Merged) 10jenkins-bot: kubernetes: Cleanup namespace handling [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1007955 (owner: 10Majavah) [09:25:04] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [10:09:00] (03PS1) 10Majavah: build: Add ldap.conf with TLS settings [labs/striker] - 10https://gerrit.wikimedia.org/r/1009467 [10:09:43] (03CR) 10Majavah: [C: 03+2] build: Add ldap.conf with TLS settings [labs/striker] - 10https://gerrit.wikimedia.org/r/1009467 (owner: 10Majavah) [10:12:18] (03Merged) 10jenkins-bot: build: Add ldap.conf with TLS settings [labs/striker] - 10https://gerrit.wikimedia.org/r/1009467 (owner: 10Majavah) [10:25:44] 10Toolforge, 15User-aborrero: [toolforge.infra] create fullstack tests - https://phabricator.wikimedia.org/T357977#9610395 (10aborrero) [10:27:27] 05Grid-Engine-to-K8s-Migration: Migrate wd-shex-infer from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320140#9610410 (10dcaro) Hi @LucasWerkmeister, I don't see any more jobs running on the grid for this tool, is there anything left? Can we close this task if not? Cheers! [10:31:10] 05Grid-Engine-to-K8s-Migration: Migrate wikihistory from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320157#9610433 (10dcaro) Hi @Wurgl, I see there's still some processes running on the grid, were you able to have a look into migrating? If not, can you make your code availa... [10:33:11] 06cloud-services-team, 13Patch-For-Review, 15User-dcaro, 15User-fgiunchedi: [wmcs][alerting] Allow volunteer admins silencing alerts from cloudvps/toolforge/paws/quarry - https://phabricator.wikimedia.org/T320973#9610449 (10dcaro) >>! In T320973#9609361, @andrea.denisse wrote: > Hi @dcaro , T333615 is comp... [10:33:26] 06cloud-services-team, 13Patch-For-Review, 15User-dcaro, 15User-fgiunchedi: [wmcs][alerting] Allow volunteer admins silencing alerts from cloudvps/toolforge/paws/quarry - https://phabricator.wikimedia.org/T320973#9610451 (10dcaro) 05Stalled→03Open [10:33:28] 06cloud-services-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 07Epic, and 2 others: Streamline WMCS Alerting and Paging - https://phabricator.wikimedia.org/T313444#9610452 (10dcaro) [11:09:15] 05Grid-Engine-to-K8s-Migration: Migrate convert from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319646#9610575 (10dcaro) Hi @Rillke, I see there's still some code running on the grid for this tool, have you had a chance to try to migrate it? Let us know if you want some hel... [11:10:25] 05Grid-Engine-to-K8s-Migration: Migrate enwikt-translations from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319724#9610589 (10dcaro) Hi @Erutuon! I don't see any more processes running on the grid for this tool, were you able to migrate? If so, you can close this task :) If... [11:13:52] 05Grid-Engine-to-K8s-Migration: Migrate persondata from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319962#9610603 (10dcaro) Hi @Wurgl, I see there's still one cron running on the grid, do you need any more help or guidance on getting it migrated? We introduced the health c... [11:15:40] 05Grid-Engine-to-K8s-Migration, 10WMCZ-General: Make it possible to run pandoc in Toolforge's jobs framework - https://phabricator.wikimedia.org/T345029#9610605 (10dcaro) Hi @Urbanecm! Were you able to get this working? Do you need more help on getting started with the build service? (I see your tool migrated... [11:19:34] 05Grid-Engine-to-K8s-Migration, 06Growth-Team: Migrate ERANBOT project off of Grid Engine - https://phabricator.wikimedia.org/T306888#9610610 (10dcaro) @MusikAnimal Do you need any help getting this working? Would providing the python2 image work for you wile you rewrite the tool? Note that the grid is being... [11:22:09] 05Grid-Engine-to-K8s-Migration: Migrate dawikibot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319661#9610625 (10dcaro) Hi @Steenth, were you able to migrate your tool? (I don't see any processes running on the grid anymore) Do you need more help/assistance? If so, can y... [11:23:42] 05Grid-Engine-to-K8s-Migration: Migrate dawikitool from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319662#9610633 (10dcaro) @Steenth you can find the crontab in the tool home (see {https://phabricator.wikimedia.org/T319661#9424006}). And same as that task too :), have you... [11:25:13] 06cloud-services-team, 06Infrastructure-Foundations, 06SRE: Track source of packages in reprepro - https://phabricator.wikimedia.org/T105385#9610637 (10MoritzMuehlenhoff) 05Open→03Resolved a:03MoritzMuehlenhoff This task is quite old and these days we've established the scheme of using separate compone... [11:29:57] 05Grid-Engine-to-K8s-Migration: Migrate xiplus from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357567#9610656 (10Xiplus) 05Open→03Resolved a:03Xiplus Thanks! I have migrated it. [11:41:07] 10Striker: Mark notifs about approved/declined membership requests as read for all admins - https://phabricator.wikimedia.org/T316832#9610718 (10taavi) 05Open→03Resolved Pending membership requests now show on its own "membership" tab on the navbar instead of as notifications. [11:41:35] 10Striker, 06cloud-services-team, 13Patch-For-Review, 07SecTeam-Processed, and 2 others: Wikitech 2FA can be bypassed in Striker via Django admin console login - https://phabricator.wikimedia.org/T284400#9610715 (10taavi) 05Open→03Resolved [11:42:30] 10Striker: Mark all alerts as read - https://phabricator.wikimedia.org/T332579#9610726 (10taavi) > Although I would have a hunch that if T316832 were implemented most admins would not feel the need for a "ignore this backlog because it is too deep" button. :) Yes, but I have 721 unread notifications from before... [11:43:52] 06cloud-services-team, 13Patch-For-Review, 15User-dcaro, 15User-fgiunchedi: [wmcs][alerting] Allow volunteer admins silencing alerts from cloudvps/toolforge/paws/quarry - https://phabricator.wikimedia.org/T320973#9610730 (10dcaro) @andrea.denisse I see though that alertmanager is still at 0.99-1, the block... [11:44:44] 10Striker, 10wikitech.wikimedia.org, 10Bitu, 06Infrastructure-Foundations: LDAP account that is not attached on wikitech has no means for password reset - https://phabricator.wikimedia.org/T174469#9610734 (10taavi) 05Open→03Resolved a:03taavi [12:09:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [12:16:01] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.set_maintenance [12:16:24] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) [12:16:32] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2001-dev.codfw.wmnet' [12:23:30] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2001-dev.codfw.wmnet' [12:27:40] PROBLEM - Check nf_conntrack usage in neutron netns on cloudnet2007-dev is CRITICAL: CRITICAL: no netns defined? https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting [12:27:57] ^ me [12:46:54] 10Cloud-VPS: Better support for Postgres on Trove - https://phabricator.wikimedia.org/T337396#9610992 (10fnegri) [12:46:58] 10Cloud-VPS, 06cloud-services-team, 07PostgreSQL: Consider removing Postgres support from Trove - https://phabricator.wikimedia.org/T353018#9610993 (10fnegri) [12:58:50] 10Cloud-VPS: [trove] move docker images from quay.io to self-hosted registry - https://phabricator.wikimedia.org/T359531 (10fnegri) 03NEW [13:10:21] 10tool-wdlocator, 06translatewiki.net, 10Language-Team (Language-2024-January-March), 03Localization Infrastructure FY2023-24, 07Unplanned-Sprint-Work: Add wdlocator to translatewiki.net - https://phabricator.wikimedia.org/T357495#9611061 (10Nikerabbit) 05In progress→03Resolved First commit https://g... [13:11:53] 05Grid-Engine-to-K8s-Migration: Migrate addletterboxdfilmidbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357549#9611071 (10dcaro) I created a first merge request to get you started: https://gitlab.com/carlinmack/addletterboxdfilmid/-/merge_requests/2, building works a... [13:14:49] (TfInfraTestApplyFailed) resolved: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [13:18:28] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:19:42] 10tool-wdlocator, 06translatewiki.net, 10Language-Team (Language-2024-January-March), 03Localization Infrastructure FY2023-24, 07Unplanned-Sprint-Work: Add wdlocator to translatewiki.net - https://phabricator.wikimedia.org/T357495#9611143 (10Samwilson) Looks good: https://wdlocator.toolforge.org/?uselang... [13:20:18] 10Cloud-VPS: [trove] define process for updating docker images - https://phabricator.wikimedia.org/T359534 (10fnegri) 03NEW [13:21:10] 10Cloud-VPS: [trove] move docker images from quay.io to self-hosted registry - https://phabricator.wikimedia.org/T359531#9611161 (10fnegri) [13:23:28] (InstanceDown) resolved: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:25:04] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [13:32:55] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review, 15User-aborrero: toolforge: introduce OpenAPI to jobs framework - https://phabricator.wikimedia.org/T356523#9611315 (10CodeReviewBot) aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/65 job: adjust max job... [13:35:18] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review, 15User-aborrero: toolforge: introduce OpenAPI to jobs framework - https://phabricator.wikimedia.org/T356523#9611322 (10CodeReviewBot) project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge... [13:41:27] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review, 15User-aborrero: toolforge: introduce OpenAPI to jobs framework - https://phabricator.wikimedia.org/T356523#9611361 (10CodeReviewBot) aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/216 jobs-api: b... [13:41:32] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [13:41:37] 06cloud-services-team, 10wikitech.wikimedia.org: Disable SSH key management on Wikitech - https://phabricator.wikimedia.org/T359544 (10taavi) 03NEW [13:41:42] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [13:41:55] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [13:42:05] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [13:44:16] 06cloud-services-team, 10wikitech.wikimedia.org: Developer account creation without OpenStackManager - https://phabricator.wikimedia.org/T196171#9611374 (10taavi) 05In progress→03Resolved a:03taavi Account creation on Wikitech is currently disabled. I've filed {T359544} for the SSH key management functio... [13:44:23] 10MediaWiki-extensions-OpenStackManager, 06cloud-services-team, 10wikitech.wikimedia.org: Remove OpenStackManager from Wikitech - https://phabricator.wikimedia.org/T161553#9611378 (10taavi) [13:45:02] 06cloud-services-team, 10wikitech.wikimedia.org: Disable SSH key management on Wikitech - https://phabricator.wikimedia.org/T359544#9611383 (10taavi) [13:46:03] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9611398 (10MBH) > I might not have time to help you tweak the build script or might take me a while to get to it, specially if it's not blocking anything. I understand it. I r... [13:48:01] 10Toolforge, 10observability, 15User-aborrero: Set up monitoring for community cronjobs - https://phabricator.wikimedia.org/T306790#9611404 (10aborrero) [13:48:31] 10Toolforge, 10observability, 15User-aborrero: Set up monitoring for community cronjobs - https://phabricator.wikimedia.org/T306790#9611403 (10aborrero) I think kubernetes should be logging somewhere all cronjob failures. What if we extend the jobs framework emailer logic a bit to monitor for such cronjob f... [14:05:05] 14Toolforge (Toolforge iteration 05): [jobs] Enable filelog for buildservice-based images - https://phabricator.wikimedia.org/T357897#9611485 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/61 [command] wrap buildservice with a shell [14:05:11] 10Toolforge (Toolforge iteration 07): [jobs-cli,jobs-api] Allow using file logs with build service images - https://phabricator.wikimedia.org/T353537#9611486 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/61 [command] wrap buildservice with a shell [14:07:02] 10Striker: Strikerbot adds a welcome message to a user's talk page every time that an approved membership request is edited - https://phabricator.wikimedia.org/T323447#9611488 (10taavi) a:03taavi [14:08:00] (03PS2) 10Majavah: Add status filter to membership request list [labs/striker] - 10https://gerrit.wikimedia.org/r/1009251 (https://phabricator.wikimedia.org/T359338) [14:08:01] 14Toolforge (Toolforge iteration 05), 13Patch-For-Review: [jobs] Enable filelog for buildservice-based images - https://phabricator.wikimedia.org/T357897#9611506 (10CodeReviewBot) project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merg... [14:08:04] (03PS2) 10Majavah: Add username search filter to membership request list [labs/striker] - 10https://gerrit.wikimedia.org/r/1009252 (https://phabricator.wikimedia.org/T282704) [14:08:12] (03PS1) 10Majavah: Do not allow double approving users [labs/striker] - 10https://gerrit.wikimedia.org/r/1009537 [14:08:48] (03PS2) 10Majavah: Fix double-approving users [labs/striker] - 10https://gerrit.wikimedia.org/r/1009537 [14:10:43] (03CR) 10Majavah: [C: 03+2] Add status filter to membership request list [labs/striker] - 10https://gerrit.wikimedia.org/r/1009251 (https://phabricator.wikimedia.org/T359338) (owner: 10Majavah) [14:10:59] 05Grid-Engine-to-K8s-Migration: Migrate addletterboxdfilmidbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357549#9611516 (10dcaro) How are you running the other bots? On the same tool account? Essentially, you'll have to create a new tool account for each of the porje... [14:11:15] (03CR) 10Majavah: [C: 03+2] Add username search filter to membership request list [labs/striker] - 10https://gerrit.wikimedia.org/r/1009252 (https://phabricator.wikimedia.org/T282704) (owner: 10Majavah) [14:11:49] (03Merged) 10jenkins-bot: Add status filter to membership request list [labs/striker] - 10https://gerrit.wikimedia.org/r/1009251 (https://phabricator.wikimedia.org/T359338) (owner: 10Majavah) [14:12:56] (03Merged) 10jenkins-bot: Add username search filter to membership request list [labs/striker] - 10https://gerrit.wikimedia.org/r/1009252 (https://phabricator.wikimedia.org/T282704) (owner: 10Majavah) [14:13:16] (03PS3) 10Majavah: Fix double-approving users [labs/striker] - 10https://gerrit.wikimedia.org/r/1009537 (https://phabricator.wikimedia.org/T323447) [14:16:55] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [14:16:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:17:26] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [14:17:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:23:06] (03PS1) 10Majavah: Format with black [labs/striker] - 10https://gerrit.wikimedia.org/r/1009540 [14:25:26] 06cloud-services-team, 10wikitech.wikimedia.org, 07LDAP: Replace wikitech as source of two-factor auth protection for developer accounts - https://phabricator.wikimedia.org/T359551 (10taavi) 03NEW [14:26:07] (03CR) 10CI reject: [V: 04-1] Format with black [labs/striker] - 10https://gerrit.wikimedia.org/r/1009540 (owner: 10Majavah) [14:32:45] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [14:32:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:33:37] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [14:33:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:35:44] 10Striker: Use IDP for authentication in Striker - https://phabricator.wikimedia.org/T359554 (10taavi) 03NEW [14:35:54] 14Toolforge (Toolforge iteration 05), 13Patch-For-Review: [jobs] Enable filelog for buildservice-based images - https://phabricator.wikimedia.org/T357897#9611676 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/217 jobs-api: bump to 0.0.268-20... [14:44:04] 10Toolforge (Toolforge iteration 07): [jobs-cli,jobs-api] Allow using file logs with build service images - https://phabricator.wikimedia.org/T353537#9611715 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/8 Allow enabling filelog with buildpack images [14:45:22] (03PS2) 10Majavah: Format with black [labs/striker] - 10https://gerrit.wikimedia.org/r/1009540 [14:54:30] (03PS3) 10Majavah: Format with black [labs/striker] - 10https://gerrit.wikimedia.org/r/1009540 [14:55:37] (03CR) 10Majavah: [C: 03+2] Fix double-approving users [labs/striker] - 10https://gerrit.wikimedia.org/r/1009537 (https://phabricator.wikimedia.org/T323447) (owner: 10Majavah) [14:57:07] (03Merged) 10jenkins-bot: Fix double-approving users [labs/striker] - 10https://gerrit.wikimedia.org/r/1009537 (https://phabricator.wikimedia.org/T323447) (owner: 10Majavah) [15:00:36] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [jobs-cli,jobs-api] Allow using file logs with build service images - https://phabricator.wikimedia.org/T353537#9611789 (10CodeReviewBot) dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/15 d/changelog: bump to... [15:11:58] 06cloud-services-team, 10wikitech.wikimedia.org: Disable SSH key management on Wikitech - https://phabricator.wikimedia.org/T359544#9611806 (10bd808) Counter proposal: Bitu should expose web APIs for operations like this and Striker should be a client. [15:16:11] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [jobs-cli,jobs-api] Allow using file logs with build service images - https://phabricator.wikimedia.org/T353537#9611811 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/15 d/changelog: bump to... [15:17:59] 10Toolforge: [envvars-cli] Either hide or show envvars values, but not both - https://phabricator.wikimedia.org/T359558 (10Slst2020) 03NEW [15:24:36] 10Striker: django-ratelimit-backend is not compatible with Django 3.x - https://phabricator.wikimedia.org/T359559 (10taavi) 03NEW [15:30:00] 06cloud-services-team: PuppetFailure Puppet failure on cloudcumin1001:9100 - https://phabricator.wikimedia.org/T358702#9611892 (10fnegri) 05Open→03Resolved a:03fnegri [15:30:25] 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T358705#9611894 (10fnegri) 05Open→03Resolved a:03fnegri [15:30:49] 10Striker: django-ratelimit-backend is not compatible with Django 3.x - https://phabricator.wikimedia.org/T359559#9611902 (10bd808) The actually algorithm bits of this package are pretty small if I remember correctly. They can probably just be recreated inside Striker with credit to the original project. [15:37:18] (PuppetConstantChange) resolved: Puppet performing a change on every puppet run on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [15:51:05] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [builds-api,jobs-api,envvars-api,api-gateway] FIgure out and document how to do non-backwards compatible changes - https://phabricator.wikimedia.org/T356974#9611963 (10dcaro) a:05dcaro→03Raymond_Ndibe Handing over to @Raymond_Ndibe :) [15:51:22] (03CR) 10Majavah: [C: 03+2] Format with black [labs/striker] - 10https://gerrit.wikimedia.org/r/1009540 (owner: 10Majavah) [15:52:42] (03Merged) 10jenkins-bot: Format with black [labs/striker] - 10https://gerrit.wikimedia.org/r/1009540 (owner: 10Majavah) [16:02:24] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9611983 (10dcaro) >>! In T319883#9611398, @MBH wrote: >> I might not have time to help you tweak the build script or might take me a while to get to it, specially if it's not... [16:05:59] (03PS3) 10Majavah: Require SUL/Phab links before applying for access [labs/striker] - 10https://gerrit.wikimedia.org/r/1008960 (https://phabricator.wikimedia.org/T172899) [16:15:59] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9612019 (10MBH) I did it ` tools.mbh@tools-sgebastion-10:~$ toolforge webservice restart Restarting.............................. Your webservice is taking quite while to rest... [16:25:05] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9612053 (10MBH) > it will not use anything from the NFS anymore And only one working way to put something into my `public_html` (I mean not NFS folder, but a space, visible fr... [16:32:00] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [16:40:21] 10Cloud-Services: Advice needed: creating a row for every article across every language Wikipedia in ToolsDB - https://phabricator.wikimedia.org/T359564 (10Audiodude) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/prof... [16:42:16] 05Grid-Engine-to-K8s-Migration: Migrate persondata from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319962#9612146 (10Wurgl) Just jlocal parts. The watchdog and I think a cleanup of log files. Nothing important. [16:42:29] 10Data-Services: Advice needed: creating a row for every article across every language Wikipedia in ToolsDB - https://phabricator.wikimedia.org/T359564#9612147 (10JJMC89) [17:06:05] 10Cloud-VPS, 10cloud-services-team (FY2023/2024-Q3-Q4): [trove] wrong quota_usages values in project tf-infra-test - https://phabricator.wikimedia.org/T359412#9612308 (10Andrew) If you aren't interested in diving into the trove code then yeah, zero-ing out the db values is what I'd do. It turns out that mainta... [17:07:14] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9612336 (10dcaro) For static files, you can still use your public_html folder, but you'll have to access it through tools-static (https://wikitech.wikimedia.org/wiki/Help:Tool... [17:07:54] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9612337 (10dcaro) >>! In T319883#9612019, @MBH wrote: > I did it > ` > tools.mbh@tools-sgebastion-10:~$ toolforge webservice restart > Restarting................................. [17:11:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:12:26] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9612373 (10dcaro) >>! In T319883#9612337, @dcaro wrote: > Looking.... It's running the non-buildservice webservice, starting it with the buildservice image fails with crashlo... [17:18:40] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9612415 (10dcaro) >>! In T319883#9612373, @dcaro wrote: >>>! In T319883#9612337, @dcaro wrote: >> Looking.... > > It's running the non-buildservice webservice, starting it wi... [17:20:14] 05Grid-Engine-to-K8s-Migration, 06Growth-Team, 10Community-Tech (CommTech-Kanban): Migrate ERANBOT project off of Grid Engine - https://phabricator.wikimedia.org/T306888#9612416 (10TheresNoTime) FYI @JWheeler-WMF — this may need new prioritisation. Our project #copypatrol ~depends on this. [17:21:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:25:23] 10Toolforge: [envvars-cli] Either hide or show envvars values, but not both - https://phabricator.wikimedia.org/T359558#9612462 (10dcaro) p:05Triage→03Low [17:26:28] 10Toolforge: [envvars-cli] Either hide or show envvars values, but not both - https://phabricator.wikimedia.org/T359558#9612460 (10dcaro) Agree, maybe we don't need to hide the prompt at all, the main issue was avoiding having it in the shell history, showing it when prompting should be ok imo [17:29:31] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9612477 (10dcaro) There you go: https://github.com/Saisengen/wikibots/pull/4 Tested that with the envvar set too (so it goes over the exec statements). [17:29:49] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9612478 (10MBH) > For static files, you can still use your public_html folder, but you'll have to access it through tools-static And will my tools be able to read this file fr... [17:30:09] 05Grid-Engine-to-K8s-Migration, 06Growth-Team, 10Community-Tech (CommTech-Kanban): Migrate ERANBOT project off of Grid Engine - https://phabricator.wikimedia.org/T306888#9612480 (10MusikAnimal) > Has the team given you any timeline on this? We haven't been given a timeline. We did receive the first draft of... [17:31:38] 10Cloud-VPS, 10cloud-services-team (FY2023/2024-Q3-Q4): [trove] wrong quota_usages values in project tf-infra-test - https://phabricator.wikimedia.org/T359412#9612490 (10fnegri) 05In progress→03Resolved I did reset `in_use` and `reserved` values to zero, but I did not truncate the `reservations` table as i... [17:34:59] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9612511 (10dcaro) >>! In T319883#9612478, @MBH wrote: >> For static files, you can still use your public_html folder, but you'll have to access it through tools-static > And w... [17:39:14] 05Grid-Engine-to-K8s-Migration, 06Growth-Team, 10Community-Tech (CommTech-Kanban): Migrate ERANBOT project off of Grid Engine - https://phabricator.wikimedia.org/T306888#9612533 (10dcaro) Fyi. using `toolforge jobs run --image python2 ...` should work (just tested a silly script), even if it does not show in... [17:49:40] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9612645 (10MBH) Hmmm, now both links https://mbh.toolforge.org/cgi-bin/cpf and https://mbh.toolforge.org/cgi-bin/category-pathfinder responds with {F42446625} Toolforge acts... [17:49:52] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9612659 (10MBH) Works now, may be a random failure. [18:13:44] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9612763 (10MBH) No, the problem persists: now three tools you initially added to `.sln` file works, and ~seven new, added by you to sln later - responds with `No webservice`.... [18:16:34] 06cloud-services-team, 13Patch-For-Review, 15User-dcaro, 15User-fgiunchedi: [wmcs][alerting] Allow volunteer admins silencing alerts from cloudvps/toolforge/paws/quarry - https://phabricator.wikimedia.org/T320973#9612797 (10andrea.denisse) @dcaro My apologies for the confusion, I'm working on the packages... [18:51:09] (03PS3) 10Majavah: Refresh user group membership after membership is granted [labs/striker] - 10https://gerrit.wikimedia.org/r/1009232 (https://phabricator.wikimedia.org/T144943) [18:59:58] (03PS2) 10Majavah: labsauth: Add field for SUL account ID [labs/striker] - 10https://gerrit.wikimedia.org/r/1009310 (https://phabricator.wikimedia.org/T359428) [19:00:00] (03PS3) 10Majavah: labsauth: Store SUL user ID like username [labs/striker] - 10https://gerrit.wikimedia.org/r/1009311 (https://phabricator.wikimedia.org/T359428) [19:29:42] 05Grid-Engine-to-K8s-Migration: Migrate erex-yomi from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319727#9613172 (10Asaf) 05Open→03Resolved a:03Asaf Migration complete, with many thanks to @dcaro . [19:33:51] (03PS1) 10Majavah: admin: Do not use ratelimitbackend [labs/striker] - 10https://gerrit.wikimedia.org/r/1009590 [19:33:53] (03PS1) 10Majavah: labsauth: Stop using django.utils.six [labs/striker] - 10https://gerrit.wikimedia.org/r/1009591 [19:34:00] (03PS1) 10Majavah: Migrate dependency management to Poetry [labs/striker] - 10https://gerrit.wikimedia.org/r/1009592 [19:34:08] (03PS1) 10Majavah: Add test to make sure app boots up properly. [labs/striker] - 10https://gerrit.wikimedia.org/r/1009593 [19:34:16] (03PS1) 10Majavah: WIP: Upgrade to Django 3.2 LTS [labs/striker] - 10https://gerrit.wikimedia.org/r/1009594 (https://phabricator.wikimedia.org/T359217) [19:34:28] 10Striker, 13Patch-For-Review: Update Django version used in Striker - https://phabricator.wikimedia.org/T359217#9613191 (10taavi) a:03taavi [19:35:46] (03CR) 10CI reject: [V: 04-1] WIP: Upgrade to Django 3.2 LTS [labs/striker] - 10https://gerrit.wikimedia.org/r/1009594 (https://phabricator.wikimedia.org/T359217) (owner: 10Majavah) [19:37:00] (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [19:50:26] 10Toolforge (Toolforge iteration 07): [apt-buildpak] Some APT packages are not installed during the image build - https://phabricator.wikimedia.org/T355252#9613240 (10Dapete) 05Open→03Resolved Sorry, I thought I had closed this - turns out I never hit the submit button. The problem disappeared, the build has... [20:05:37] (03CR) 10Majavah: [C: 03+2] admin: Do not use ratelimitbackend [labs/striker] - 10https://gerrit.wikimedia.org/r/1009590 (owner: 10Majavah) [20:05:48] (03CR) 10Majavah: [C: 03+2] labsauth: Stop using django.utils.six [labs/striker] - 10https://gerrit.wikimedia.org/r/1009591 (owner: 10Majavah) [20:07:02] (03Merged) 10jenkins-bot: admin: Do not use ratelimitbackend [labs/striker] - 10https://gerrit.wikimedia.org/r/1009590 (owner: 10Majavah) [20:08:39] (03Merged) 10jenkins-bot: labsauth: Stop using django.utils.six [labs/striker] - 10https://gerrit.wikimedia.org/r/1009591 (owner: 10Majavah) [20:15:45] 10PAWS: Remove puppet code related to paws kubeadmin cluster - https://phabricator.wikimedia.org/T327674#9613295 (10rook) I believe this was removed with https://gerrit.wikimedia.org/r/c/operations/puppet/+/1006852 [20:15:47] 10PAWS: Remove puppet code related to paws kubeadmin cluster - https://phabricator.wikimedia.org/T327674#9613296 (10rook) 05Open→03Resolved [20:17:22] 10PAWS: jupyterlab to 4.1.4 - https://phabricator.wikimedia.org/T359588#9613307 (10rook) a:03rook [20:18:05] 10PAWS: jupyterlab to 4.1.4 - https://phabricator.wikimedia.org/T359588#9613314 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/385 [20:18:21] vivian-rook opened https://github.com/toolforge/paws/pull/385 [20:23:20] (03PS2) 10Majavah: Migrate dependency management to Poetry [labs/striker] - 10https://gerrit.wikimedia.org/r/1009592 [20:23:22] (03PS2) 10Majavah: Add test to make sure app boots up properly [labs/striker] - 10https://gerrit.wikimedia.org/r/1009593 [20:23:26] (03PS2) 10Majavah: WIP: Upgrade to Django 3.2 LTS [labs/striker] - 10https://gerrit.wikimedia.org/r/1009594 (https://phabricator.wikimedia.org/T359217) [20:23:34] (03PS1) 10Majavah: tools: Don't query OpenStack on every page view [labs/striker] - 10https://gerrit.wikimedia.org/r/1009599 [20:24:57] (03CR) 10CI reject: [V: 04-1] WIP: Upgrade to Django 3.2 LTS [labs/striker] - 10https://gerrit.wikimedia.org/r/1009594 (https://phabricator.wikimedia.org/T359217) (owner: 10Majavah) [20:34:02] 10Horizon: Use IDP for authentication in Horizon - https://phabricator.wikimedia.org/T359590 (10taavi) 03NEW [20:34:12] 10Horizon: Use IDP for authentication in Horizon - https://phabricator.wikimedia.org/T359590#9613353 (10taavi) [20:34:16] 10Horizon: Use IDP for authentication in Horizon - https://phabricator.wikimedia.org/T359590#9613355 (10taavi) 05Open→03Stalled [20:34:20] 06cloud-services-team, 10wikitech.wikimedia.org, 07LDAP: Replace wikitech as source of two-factor auth protection for developer accounts - https://phabricator.wikimedia.org/T359551#9613356 (10taavi) [20:39:01] 10PAWS: Update labpawspublic extension to jupyterlab 4 system - https://phabricator.wikimedia.org/T358604#9613364 (10rook) a:05rook→03None [20:41:12] 05Grid-Engine-to-K8s-Migration: Migrate wd-shex-infer from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320140#9613369 (10LucasWerkmeister) I’d still like to be able to increase the `requests` (T357209 / T357881), but otherwise this is done. (Well, eventually I should remove... [20:46:54] 10PAWS: add worker to paws - https://phabricator.wikimedia.org/T359591 (10rook) 03NEW [20:48:04] 10PAWS: add worker to paws - https://phabricator.wikimedia.org/T359591#9613394 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/386 [20:48:27] vivian-rook opened https://github.com/toolforge/paws/pull/386 [20:52:14] 10Wikibugs, 15User-bd808: Add support for alternate channels files to make testing/debugging easier - https://phabricator.wikimedia.org/T359202#9613413 (10CodeReviewBot) bd808 merged https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/12 Testing enhancements: settings for SAL announce and... [20:52:16] 10Wikibugs, 15User-bd808: Allow configuration of update announce channel - https://phabricator.wikimedia.org/T359228#9613414 (10CodeReviewBot) bd808 merged https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/12 Testing enhancements: settings for SAL announce and channel files [21:00:50] 10PAWS: add worker to paws - https://phabricator.wikimedia.org/T359591#9613428 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/386 [21:01:00] vivian-rook closed https://github.com/toolforge/paws/pull/386 [21:04:53] 10Wikibugs, 15User-bd808: Filter debug log messages out of logs sent to disk - https://phabricator.wikimedia.org/T359230#9613438 (10CodeReviewBot) bd808 merged https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/13 Filter debug log messages out of logs sent to disk [21:11:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:14:22] 10VPS-Projects, 06cloud-services-team, 10Puppet (Puppet 7.0): Migrate Puppet servers in Cloud Services team managed projects to Puppet 7 - https://phabricator.wikimedia.org/T351453#9613481 (10Andrew) @taavi, project-proxy seems to have been partially migrated, with some hosts using project-proxy-puppetmaster... [21:15:33] 10VPS-Projects, 06cloud-services-team, 10Puppet (Puppet 7.0): Migrate Puppet servers in Cloud Services team managed projects to Puppet 7 - https://phabricator.wikimedia.org/T351453#9613496 (10Andrew) [21:17:36] 10VPS-Projects, 06cloud-services-team, 10Puppet (Puppet 7.0): Migrate Puppet servers in Cloud Services team managed projects to Puppet 7 - https://phabricator.wikimedia.org/T351453#9613497 (10taavi) >>! In T351453#9613481, @Andrew wrote: > @taavi, project-proxy seems to have been partially migrated, with som... [21:19:53] 10Wikibugs: wikibugs test bug part II - https://phabricator.wikimedia.org/T90594#9613510 (10bd808) test [21:20:32] (03CR) 10BryanDavis: [C: 04-2] "Now testing the same change in the main deploy" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/1008016 (https://phabricator.wikimedia.org/T90594) (owner: 10BryanDavis) [21:21:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:22:15] (03PS2) 10Majavah: tools: Don't query OpenStack on every page view [labs/striker] - 10https://gerrit.wikimedia.org/r/1009599 [21:22:17] (03PS3) 10Majavah: Migrate dependency management to Poetry [labs/striker] - 10https://gerrit.wikimedia.org/r/1009592 [21:22:19] (03PS3) 10Majavah: Add test to make sure app boots up properly [labs/striker] - 10https://gerrit.wikimedia.org/r/1009593 [21:22:25] (03PS3) 10Majavah: WIP: Upgrade to Django 3.2 LTS [labs/striker] - 10https://gerrit.wikimedia.org/r/1009594 (https://phabricator.wikimedia.org/T359217) [21:22:45] 10Wikibugs, 15User-bd808: Allow configuration of update announce channel - https://phabricator.wikimedia.org/T359228#9613528 (10bd808) 05In progress→03Resolved [21:22:53] 10Wikibugs, 15User-bd808: Add support for alternate channels files to make testing/debugging easier - https://phabricator.wikimedia.org/T359202#9613529 (10bd808) 05In progress→03Resolved [21:23:01] 10Wikibugs, 15User-bd808: Filter debug log messages out of logs sent to disk - https://phabricator.wikimedia.org/T359230#9613530 (10bd808) 05In progress→03Resolved [21:23:50] (03CR) 10CI reject: [V: 04-1] WIP: Upgrade to Django 3.2 LTS [labs/striker] - 10https://gerrit.wikimedia.org/r/1009594 (https://phabricator.wikimedia.org/T359217) (owner: 10Majavah) [21:28:28] (InstanceDown) firing: Project metricsinfra instance metricsinfra-puppet-2 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [21:30:28] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 1 deleted instances on metricsinfra-puppetmaster-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [21:33:28] (InstanceDown) resolved: Project metricsinfra instance metricsinfra-puppet-2 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [23:37:16] (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [23:41:28] (PuppetAgentFailure) firing: Puppet agent failure detected on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure