[00:00:40] 10Wikibugs, 13Patch-For-Review, 15User-bd808: Set a timeout for phorge requests - https://phabricator.wikimedia.org/T359145#9603749 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/10 phorge: Replace `fab` library with local client [00:09:35] 10Wikibugs: Allow configuration of update announce channel - https://phabricator.wikimedia.org/T359228 (10bd808) [00:09:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [00:31:33] 10Wikibugs, 15User-bd808: wikibugs no longer says new tasks are "NEW" - https://phabricator.wikimedia.org/T127506#9603792 (10bd808) 05Open→03In progress a:03bd808 [00:34:39] 10Wikibugs, 13Patch-For-Review, 15User-bd808: wikibugs no longer says new tasks are "NEW" - https://phabricator.wikimedia.org/T127506#9603814 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/11 phorge: Check for "core:create" transaction type to flag new... [00:40:51] 10Wikibugs: Filter debug log messages out of logs sent to disk - https://phabricator.wikimedia.org/T359230 (10bd808) [02:57:28] (PuppetCertificateAboutToExpire) firing: Puppet CA certificate Puppet CA: cloudinfra-internal-puppetmaster01.cloudinfra.eqiad.wmflabs is about to expire in 25d 17h 38m 11s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [03:09:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [03:37:22] 10Wikibugs, 15User-bd808: Allow configuration of update announce channel - https://phabricator.wikimedia.org/T359228#9604108 (10bd808) 05Open→03In progress a:03bd808 [03:41:04] 10Wikibugs, 15User-bd808: Add support for alternate channels files to make testing/debugging easier - https://phabricator.wikimedia.org/T359202#9604136 (10bd808) 05Open→03In progress a:03bd808 [05:57:28] (PuppetCertificateAboutToExpire) firing: Puppet CA certificate Puppet CA: cloudinfra-internal-puppetmaster01.cloudinfra.eqiad.wmflabs is about to expire in 25d 14h 38m 11s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [06:09:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [07:10:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:20:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:57:28] (PuppetCertificateAboutToExpire) firing: Puppet CA certificate Puppet CA: cloudinfra-internal-puppetmaster01.cloudinfra.eqiad.wmflabs is about to expire in 25d 11h 38m 11s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [09:09:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [09:43:13] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors [09:43:15] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) [09:47:28] (InstanceDown) firing: Project tools instance tools-sgeexec-10-21 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:52:28] (InstanceDown) resolved: Project tools instance tools-sgeexec-10-21 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [10:01:24] 05Grid-Engine-to-K8s-Migration: Migrate articles-by-lat-lon-without-images from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319573#9604944 (10dcaro) Hi @Abbe98, any luck migrating the tool? Need any more help/guidance? Next week (14th March) the grid will stop working. [10:01:52] 05Grid-Engine-to-K8s-Migration: Migrate derivative from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319671#9604949 (10dcaro) I have not seen anything on the grid for a while here, @Tgr did the migration go well? Can you close this task if so? Cheers! [10:03:41] 05Grid-Engine-to-K8s-Migration: Migrate templatehoard from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320083#9604976 (10dcaro) Hi @Erutuon, any updates on this? Were you able to make your code public? Remember that the grid will be shutting down next week (14th March) and... [10:04:13] 05Grid-Engine-to-K8s-Migration: Migrate updatewikiprojectmovies from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357563#9604996 (10dcaro) 05Open→03Resolved a:03dcaro I see no owner and no tool running on the grid anymore, I'll close this task :) [10:05:45] 05Grid-Engine-to-K8s-Migration, 10urbanecmbot, 15User-Urbanecm: Migrate urbanecmbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320108#9605017 (10dcaro) Reminder that next week (14th March) the grid will go down for good, any tools that have not migrated will stop w... [10:06:28] (WidespreadPuppetAgentFailure) firing: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [10:08:28] (WidespreadPuppetAgentFailure) firing: Widespread puppet agent failures in project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [10:08:28] (WidespreadPuppetAgentFailure) firing: Widespread puppet agent failures in project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [10:09:38] 05Grid-Engine-to-K8s-Migration: Migrate xiplus from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357567#9605064 (10dcaro) @Xiplus Hi! Have you had a look already to migrating this tool? Do you need any guidance? Remember that we have a hard stop next week (14th of March). [10:09:54] 05Grid-Engine-to-K8s-Migration: Migrate erex-yomi from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319727#9605068 (10dcaro) I don't see any more grid jobs here, @Asaf feel free to close the task if it has meen already migrated :) [10:11:07] 05Grid-Engine-to-K8s-Migration: Migrate ganfilter from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357554#9605089 (10dcaro) Hi @coldchrist! Have you been able to give a look at migrating your tool? Is there any issues you have found I can help with? If so, can you make your... [10:11:39] 05Grid-Engine-to-K8s-Migration: Migrate gergesbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357555#9605092 (10dcaro) Hi @GergesSH! Have you had time to take a look into migrating this tool? Is there anything I can assist with? Remember that the grid is going to be sh... [10:13:27] 05Grid-Engine-to-K8s-Migration: Migrate himowd from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357556#9605097 (10dcaro) Hi @Mr.Ibrahem! Have you had the time to look into migrating this tool? Do you need any assistance/guidance? If so, making your source code public (ex. o... [10:14:09] 05Grid-Engine-to-K8s-Migration: Migrate hnatsumi-bot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357558#9605107 (10dcaro) Hi @DWYoungDLS! Have you had time to look into migrating this tool? Do you need any help or guidance? If so, can you make your code public? (ex. on... [10:15:22] 05Grid-Engine-to-K8s-Migration: Migrate pagecounts from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357559#9605117 (10dcaro) Hi @Amire80, @Jdrewniak, @MaxSem, have you been able to look into migrating this tool? Do you need any assistance or guidance? If so, can you make yo... [10:15:30] 05Grid-Engine-to-K8s-Migration: Migrate recoin from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357560#9605118 (10dcaro) 05Open→03Resolved a:03dcaro I see no more grid instances for this tool, will close. [10:16:28] (WidespreadPuppetAgentFailure) firing: Widespread puppet agent failures in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [10:16:32] 05Grid-Engine-to-K8s-Migration: Migrate spacemedia from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357561#9605127 (10dcaro) Hi @Don-vip! I see that there [10:17:28] (WidespreadPuppetAgentFailure) firing: Widespread puppet agent failures in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [10:17:42] 05Grid-Engine-to-K8s-Migration: Migrate addletterboxdfilmidbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357549#9605137 (10dcaro) Hi @Carlinmack! Have you had time to look into migrating this tool? Let me know if you need some help or guidance. Note that making your... [10:17:58] 05Grid-Engine-to-K8s-Migration: Migrate aka from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357550#9605138 (10dcaro) 05Open→03Resolved a:03dcaro There has not been any grid activity for this tool in a while, closing \o/ [10:18:22] 05Grid-Engine-to-K8s-Migration: Migrate anno from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319556#9605146 (10dcaro) Hi @AleXXw! I see no grid activity in a while for this tool, if you have finished migrating feel free to close the task, thanks! [10:19:51] (03CR) 10Majavah: [C: 03+2] Convert remaining images to shell webservice-runner [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1005952 (https://phabricator.wikimedia.org/T293552) (owner: 10Majavah) [10:20:05] (03Merged) 10jenkins-bot: Convert remaining images to shell webservice-runner [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1005952 (https://phabricator.wikimedia.org/T293552) (owner: 10Majavah) [10:20:13] 05Grid-Engine-to-K8s-Migration: Migrate apersonbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319561#9605162 (10dcaro) Friendly reminder that next week (14th of March) the grid is being turned off. Is there anything specific you need help or assistance? If so, can you... [10:21:01] 05Grid-Engine-to-K8s-Migration: Migrate arbclerkbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357551#9605170 (10dcaro) Hi @Bradv, have you had time to look into migrating this tool? Is there anything I can help with or guide you through? If so, can you make your code... [10:21:44] 05Grid-Engine-to-K8s-Migration: Migrate backup-bot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357553#9605174 (10dcaro) Hi @Nokib_Sarkar! Have you had a try to migrate this tool? Is there anything I can help or guide you through? If so, can you make the code for your t... [10:22:32] 05Grid-Engine-to-K8s-Migration: Migrate unblock-zh-status from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357562#9605188 (10dcaro) @Xiplus Hi! Have you had a look already to migrating this tool? Do you need any guidance? If so, can you make your code public? (ex. using git... [10:22:48] 05Grid-Engine-to-K8s-Migration: Migrate wikiflix from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357566#9605189 (10dcaro) 05Open→03Resolved a:03dcaro I don't see any more grid jobs for this tool, closing 🎉 [10:23:28] (WidespreadPuppetAgentFailure) resolved: Widespread puppet agent failures in project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [10:23:54] 05Grid-Engine-to-K8s-Migration, 07Chinese-Sites: Migrate zhwiki-perm-qualicheck from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357568#9605209 (10dcaro) Hi @DWYoungDLS! Have you been able to look into migrating this tool? Is there any help or assistance you need? If so,... [10:24:28] (WidespreadPuppetAgentFailure) firing: Widespread puppet agent failures in project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [10:24:33] 05Grid-Engine-to-K8s-Migration, 10Wiki-Loves-Monuments-Database: Migrate heritage from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319787#9605216 (10dcaro) Hi! Have you had a chance to look into migrating already? I still see some jobs running on the grid. If you want any... [10:25:05] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9605225 (10dcaro) @MBH let me know if/when you have a patch moving all the `.sln` files, and I'll try to help you with the `compile.sh` script. [10:25:13] 10Cloud-VPS (Project-requests): Request creation of ipoid-opensearch VPS project - https://phabricator.wikimedia.org/T359298 (10kostajh) [10:26:11] 05Grid-Engine-to-K8s-Migration: Migrate phetools from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319965#9605247 (10dcaro) Hi! The phetools repository seems empty (https://gitlab.wikimedia.org/toolforge-repos/phetools.git), I can try to help a bit, but I would need to the co... [10:26:28] (WidespreadPuppetAgentFailure) resolved: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [10:26:59] 05Grid-Engine-to-K8s-Migration, 07Chinese-Sites: Migrate zhwiki-perm-qualicheck from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357568#9605250 (10DWYoungDLS) 05Open→03Resolved a:03DWYoungDLS >>! In T357568#9605208, @dcaro wrote: > Hi @DWYoungDLS! Have you been able... [10:27:23] 05Grid-Engine-to-K8s-Migration: Migrate spacemedia from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357561#9605258 (10Don-vip) Hi @dcaro! I moved away from gridengine a long time ago, what is the cron you are still seeing? [10:27:46] (03CR) 10Eugene233: "recheck" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1007683 (owner: 10AgnesAbah) [10:27:54] (03CR) 10Eugene233: Bug:T357376 Remove unused/not-required imports from _init_.py (031 comment) [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1007683 (owner: 10AgnesAbah) [10:29:02] 05Grid-Engine-to-K8s-Migration: Migrate spacemedia from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357561#9605266 (10dcaro) From https://grid-deprecation.toolforge.org/t/spacemedia, there's a `cron-29` that was last seen running two days ago, maybe it's already migrated? [10:31:08] 05Grid-Engine-to-K8s-Migration: Migrate hnatsumi-bot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357558#9605284 (10DWYoungDLS) 05Open→03Resolved a:03DWYoungDLS >>! In T357558#9605106, @dcaro wrote: > Hi @DWYoungDLS! Have you had time to look into migrating this to... [10:31:24] 05Grid-Engine-to-K8s-Migration: Migrate spacemedia from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357561#9605289 (10Don-vip) Ah yes ok, indeed I still have one, it's the task that checks the duplicates every 30 minutes, I forgot about it: ` # m h dom mon dow command */30... [10:31:28] (WidespreadPuppetAgentFailure) resolved: Widespread puppet agent failures in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [10:31:48] 05Grid-Engine-to-K8s-Migration, 07Chinese-Sites: Migrate zhwiki-perm-qualicheck from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357568#9605290 (10DWYoungDLS) Now migrated. [10:34:42] 10Toolforge, 06cloud-services-team: Remove Python/webservice-runner from toolforge web containers - https://phabricator.wikimedia.org/T293552#9605316 (10taavi) [10:34:52] 10Toolforge: [toolforge-webservice] Remove old webservice-runner code - https://phabricator.wikimedia.org/T358320#9605330 (10taavi) [10:34:59] 10Toolforge, 06cloud-services-team: Remove Python/webservice-runner from toolforge web containers - https://phabricator.wikimedia.org/T293552#9605329 (10taavi) 05Open→03Resolved [10:35:07] 10Toolforge (Software install/update): Build Bookworm based Toolforge Kubernetes images - https://phabricator.wikimedia.org/T335507#9605331 (10taavi) [10:37:28] (WidespreadPuppetAgentFailure) resolved: Widespread puppet agent failures in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [10:39:28] (WidespreadPuppetAgentFailure) resolved: Widespread puppet agent failures in project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [10:47:52] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-32 [10:47:57] !log taavi@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.remove_grid_node (exit_code=1) for tools-sgeweblight-10-17, tools-sgeweblight-10-32 [10:48:18] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-32 [10:48:28] (WidespreadPuppetAgentFailure) resolved: Widespread puppet agent failures in project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [10:51:42] 10Toolforge (Toolforge iteration 07): [tbs] Unable to get pywikibot + wget on a python build service image - https://phabricator.wikimedia.org/T354157#9605471 (10YFdyh000) Sorry for the delay. Thank you for your detailed answer, which is very useful. It's already running with some workaround code, and you can c... [10:53:27] 10Tool-phab-ban, 10Phabricator, 06collaboration-services: Phabricator admins should have access to phab-ban tool - https://phabricator.wikimedia.org/T359212#9605475 (10Bugreporter) [10:55:28] (InstanceDown) firing: Project tools instance tools-sgeweblight-10-32 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [10:58:57] (03CR) 10Eugene233: Bug: T225798. Swipe left right to change participant image. (031 comment) [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1005839 (https://phabricator.wikimedia.org/T225798) (owner: 10Ketulucas) [10:59:11] (03CR) 10Eugene233: "recheck" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1005839 (https://phabricator.wikimedia.org/T225798) (owner: 10Ketulucas) [11:00:28] (InstanceDown) resolved: Project tools instance tools-sgeweblight-10-32 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [11:07:19] 05Grid-Engine-to-K8s-Migration: Migrate ganfilter from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357554#9605531 (10coldchrist) @dcaro, thanks for the ping -- ganfilter doesn't use a webservice, so as far as I can tell the only thing that needs to happen is to change the cr... [11:24:30] 05Grid-Engine-to-K8s-Migration, 10Tool-spacemedia: Migrate spacemedia from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357561#9605596 (10Don-vip) 05Open→03In progress [11:42:41] (CloudVPSDesignateLeaks) firing: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:44:12] 10Striker, 10wikitech.wikimedia.org, 10Bitu, 06Infrastructure-Foundations: LDAP account that is not attached on wikitech has no means for password reset - https://phabricator.wikimedia.org/T174469#9605736 (10taavi) Is this still an issue as #Bitu is now used for password resets? [11:45:39] (03PS1) 10Majavah: Refresh user group membership after membership is granted [labs/striker] - 10https://gerrit.wikimedia.org/r/1009232 (https://phabricator.wikimedia.org/T144943) [11:46:15] (03CR) 10Majavah: "This is unfortunately untested as the Keystone container in my local dev env seems to be having some problems." [labs/striker] - 10https://gerrit.wikimedia.org/r/1009232 (https://phabricator.wikimedia.org/T144943) (owner: 10Majavah) [12:04:49] (TfInfraTestApplyFailed) resolved: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [12:07:05] 10PAWS: Increase paws nodes for outreachy demand - https://phabricator.wikimedia.org/T359316#9605823 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/384 [12:07:15] vivian-rook closed https://github.com/toolforge/paws/pull/384 [12:07:36] 10PAWS: Increase paws nodes for outreachy demand - https://phabricator.wikimedia.org/T359316#9605827 (10rook) 05Open→03Resolved [12:10:51] 10PAWS: PAWS shell - lack of i18n submodule or files or an outdated submodule - https://phabricator.wikimedia.org/T343676#9605855 (10rook) Closing, though please reopen if this is still an issue. [12:11:15] 10PAWS: PAWS shell - lack of i18n submodule or files or an outdated submodule - https://phabricator.wikimedia.org/T343676#9605857 (10rook) 05Open→03Declined [12:26:57] (03PS1) 10Majavah: Consistently use 'user' icon for membership requests [labs/striker] - 10https://gerrit.wikimedia.org/r/1009242 [12:27:00] (03PS1) 10Majavah: Display membership requests in navbar for admins [labs/striker] - 10https://gerrit.wikimedia.org/r/1009243 (https://phabricator.wikimedia.org/T316832) [12:27:08] (03PS1) 10Majavah: Display number of open access requests in navbar [labs/striker] - 10https://gerrit.wikimedia.org/r/1009244 (https://phabricator.wikimedia.org/T316832) [12:27:16] (03PS1) 10Majavah: Stop sending notifications about access requests [labs/striker] - 10https://gerrit.wikimedia.org/r/1009245 (https://phabricator.wikimedia.org/T316832) [12:40:56] 05Grid-Engine-to-K8s-Migration, 10Tool-spacemedia: Migrate spacemedia from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357561#9606052 (10Don-vip) 05In progress→03Resolved a:03Don-vip done; thanks for the notice! [12:43:20] 10Cloud Services Proposals, 10cloud-services-team (FY2023/2024-Q3-Q4): Decision Request - Incident Response Process - https://phabricator.wikimedia.org/T348887#9606074 (10fnegri) In the absence of further comments, I'm proposing the following plan: * Resolving this Decision Request choosing option 2.1 * I wil... [12:49:49] (03PS1) 10Josefanthony: Bug:T334584 Adjusted footer to the bottom [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1008917 [12:52:42] (03CR) 10Josefanthony: "Adjusted footer which was initially at the center of the page, to the bottom of the page." [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1008917 (owner: 10Josefanthony) [12:56:34] 10Cloud Services Proposals, 10cloud-services-team (FY2023/2024-Q3-Q4): Decision Request - Incident Response Process - https://phabricator.wikimedia.org/T348887#9606132 (10dcaro) yay \o/ [13:00:15] 05Grid-Engine-to-K8s-Migration: Migrate derivative from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319671#9606146 (10Tgr) 05Open→03Resolved The tool is still generating uploads. [13:05:06] (03CR) 10David Caro: [C: 03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1007955 (owner: 10Majavah) [13:09:45] 10Striker: Add option to only show open access requests - https://phabricator.wikimedia.org/T359338 (10taavi) [13:10:36] 10Striker: Add option to only show open access requests - https://phabricator.wikimedia.org/T359338#9606229 (10taavi) p:05Triage→03Medium a:03taavi [13:12:51] 05Grid-Engine-to-K8s-Migration: Migrate xiplus from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357567#9606234 (10Xiplus) I use cron to execute a logrotate job. The cron command is ` 13 5 * * * jsub /usr/sbin/logrotate -s /data/project/xiplus/logrotate-status /data/project/x... [13:13:56] 10Striker: Striker: Allow searching for access requests for a given user - https://phabricator.wikimedia.org/T282704#9606256 (10taavi) a:03taavi [13:14:04] 05Grid-Engine-to-K8s-Migration: Migrate unblock-zh-status from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357562#9606258 (10Xiplus) 05Open→03Resolved a:03Xiplus I have migrated it today. [13:14:25] (03PS2) 10Majavah: Consistently use 'user' icon for membership requests [labs/striker] - 10https://gerrit.wikimedia.org/r/1009242 [13:14:33] (03PS2) 10Majavah: Display membership requests in navbar for admins [labs/striker] - 10https://gerrit.wikimedia.org/r/1009243 (https://phabricator.wikimedia.org/T316832) [13:14:41] (03PS2) 10Majavah: Display number of open access requests in navbar [labs/striker] - 10https://gerrit.wikimedia.org/r/1009244 (https://phabricator.wikimedia.org/T316832) [13:14:49] (03PS2) 10Majavah: Stop sending notifications about access requests [labs/striker] - 10https://gerrit.wikimedia.org/r/1009245 (https://phabricator.wikimedia.org/T316832) [13:14:57] (03PS1) 10Majavah: Add status filter to membership request list [labs/striker] - 10https://gerrit.wikimedia.org/r/1009251 (https://phabricator.wikimedia.org/T359338) [13:15:05] (03PS1) 10Majavah: Add username search filter to membership request list [labs/striker] - 10https://gerrit.wikimedia.org/r/1009252 (https://phabricator.wikimedia.org/T282704) [13:18:56] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review, 15User-aborrero: toolforge: introduce OpenAPI to jobs framework - https://phabricator.wikimedia.org/T356523#9606284 (10CodeReviewBot) aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/66 jobs-api: add /api/v... [13:23:33] 10Striker, 13Patch-For-Review: Striker: Allow searching for access requests for a given user - https://phabricator.wikimedia.org/T282704#9606339 (10taavi) p:05Triage→03Medium [13:25:17] 05Grid-Engine-to-K8s-Migration: Migrate gergesbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357555#9606344 (10GergesSH) Hi @dcaro, I have disabled webservice, I don't need it currently. [13:26:13] 05Grid-Engine-to-K8s-Migration: Migrate ganfilter from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357554#9606359 (10dcaro) I think that might not work out of the box, you might need something like https://wikitech.wikimedia.org/wiki/Help:Toolforge/Running_Pywikibot_scripts:... [13:33:14] 10Cloud-VPS, 06cloud-services-team, 13Patch-For-Review, 15User-aborrero: Improve cloudgw filter between VM instances and cloud-private - https://phabricator.wikimedia.org/T356986#9606423 (10aborrero) the latest iteration on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1007007 brings increased granu... [13:33:59] 05Grid-Engine-to-K8s-Migration: Migrate gergesbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357555#9606424 (10dcaro) Okok, feel free to close this task if you want, or keep it for yourself will want to migrate it in the future, thanks! [13:36:47] 05Grid-Engine-to-K8s-Migration: Migrate gergesbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357555#9606440 (10GergesSH) 05Open→03Resolved a:03GergesSH [13:59:14] (03PS1) 10Majavah: Use normal login flow for admin site [labs/striker] - 10https://gerrit.wikimedia.org/r/1009258 (https://phabricator.wikimedia.org/T284400) [14:00:58] (03CR) 10CI reject: [V: 04-1] Use normal login flow for admin site [labs/striker] - 10https://gerrit.wikimedia.org/r/1009258 (https://phabricator.wikimedia.org/T284400) (owner: 10Majavah) [14:02:25] (03PS2) 10Majavah: Use normal login flow for admin site [labs/striker] - 10https://gerrit.wikimedia.org/r/1009258 (https://phabricator.wikimedia.org/T284400) [14:04:02] (03CR) 10CI reject: [V: 04-1] Use normal login flow for admin site [labs/striker] - 10https://gerrit.wikimedia.org/r/1009258 (https://phabricator.wikimedia.org/T284400) (owner: 10Majavah) [14:04:54] (03PS3) 10Majavah: Use normal login flow for admin site [labs/striker] - 10https://gerrit.wikimedia.org/r/1009258 (https://phabricator.wikimedia.org/T284400) [14:11:57] 10Striker, 13Patch-For-Review: Set description to tool URL when creating project tags - https://phabricator.wikimedia.org/T320916#9606702 (10taavi) 05Open→03Resolved a:05Aklapper→03taavi Let's say this was done via https://gerrit.wikimedia.org/r/plugins/gitiles/labs/striker/+/bd8f423ca23d856f02a4e4dad5... [14:15:26] 10Toolforge (Toolforge iteration 07): [tbs] Unable to get pywikibot + wget on a python build service image - https://phabricator.wikimedia.org/T354157#9606719 (10dcaro) 05Open→03Resolved a:03dcaro Really glad to hear that :) I'll take a look (I'm curious ;) ), I'll close this task then 👍 [14:15:34] 10Striker, 13Patch-Needs-Improvement: Set code repository URI when creating project tags - https://phabricator.wikimedia.org/T320915#9606725 (10taavi) 05Open→03Resolved a:05Aklapper→03taavi Done in https://gerrit.wikimedia.org/r/plugins/gitiles/labs/striker/+/40f174ee00e72f393282266859b9d118441a9767. [14:15:50] 10Striker: Toolsadmin Tools ID checker should link to Tool Labs Tool DB usage tool - https://phabricator.wikimedia.org/T179221#9606751 (10taavi) [14:19:34] 10Striker: Tool display view should link to other tools showing information about that tool - https://phabricator.wikimedia.org/T179221#9606808 (10taavi) [14:20:51] 10Striker: Striker dev env fails to start with `manage.py runserver: error: unrecognized arguments: --nostatic` - https://phabricator.wikimedia.org/T355522#9606863 (10taavi) 05Open→03Declined This seems to have been an issue with loading the .env file indeed. [14:22:09] (03PS1) 10David Caro: bullseye-standalone: add logrotate [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1009264 (https://phabricator.wikimedia.org/T357567) [14:22:44] 05Grid-Engine-to-K8s-Migration, 13Patch-For-Review: Migrate xiplus from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357567#9606910 (10dcaro) >>! In T357567#9606234, @Xiplus wrote: > I use cron to execute a logrotate job. The cron command is > ` > 13 5 * * * jsub /usr/sbin/... [14:30:32] (03Abandoned) 10David Caro: bullseye-standalone: add logrotate [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1009264 (https://phabricator.wikimedia.org/T357567) (owner: 10David Caro) [14:30:56] 05Grid-Engine-to-K8s-Migration, 13Patch-For-Review: Migrate xiplus from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357567#9607037 (10dcaro) Actually (@taavi reminded me) we already have logrotate in one of our images, `mariadb`, so you cat try doing something like: ` tool... [14:35:26] 05Grid-Engine-to-K8s-Migration: Migrate xiplus from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357567#9607083 (10dcaro) For future reference, the docs on how to prune things https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework#Pruning_log_files :) [14:48:36] 10Striker: Better error handling needed when OAuth grant is broken - https://phabricator.wikimedia.org/T218968#9607261 (10taavi) p:05Triage→03Low [14:49:50] 10Striker, 10Toolhub: Create a feature that allows a tool to be surfaced as "co-maintainer wanted" - https://phabricator.wikimedia.org/T231499#9607265 (10taavi) [14:53:50] (ProbeDown) firing: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:57:28] (PuppetCertificateAboutToExpire) firing: (2) Puppet CA certificate Puppet CA: puppet is about to expire in 27d 23h 58m 6s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [14:58:50] (ProbeDown) resolved: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:11:55] 10PAWS: Add a simple script to connect to a replica database - https://phabricator.wikimedia.org/T358984#9607409 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/383 [15:12:21] vivian-rook closed https://github.com/toolforge/paws/pull/383 [15:12:28] (PuppetCertificateAboutToExpire) firing: (3) Puppet CA certificate Puppet CA: puppet is about to expire in 27d 23h 42m 6s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [15:12:56] 10PAWS: Add a simple script to connect to a replica database - https://phabricator.wikimedia.org/T358984#9607413 (10rook) 05Open→03Resolved a:03rook [15:32:28] (PuppetCertificateAboutToExpire) firing: (3) Puppet CA certificate Puppet CA: puppet is about to expire in 27d 23h 22m 6s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [15:46:41] 10Toolforge (Toolforge iteration 07), 06cloud-services-team, 13Patch-For-Review, 15User-aborrero: Upgrade Toolforge Kubernetes to version 1.24 - https://phabricator.wikimedia.org/T307651#9607585 (10CodeReviewBot) aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/... [15:47:28] (PuppetCertificateAboutToExpire) resolved: (3) Puppet CA certificate Puppet CA: puppet is about to expire in 27d 23h 8m 6s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [15:47:41] (CloudVPSDesignateLeaks) firing: (5) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:57:11] 10Cloud-VPS, 10cloud-services-team (FY2023/2024-Q3-Q4): [trove] wrong quota_usages values in project tf-infra-test - https://phabricator.wikimedia.org/T359412 (10fnegri) [15:58:51] (ProbeDown) firing: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:59:58] 10Cloud-VPS, 10cloud-services-team (FY2023/2024-Q3-Q4): [trove] wrong quota_usages values in project tf-infra-test - https://phabricator.wikimedia.org/T359412#9607667 (10fnegri) [16:02:43] 10Cloud-VPS, 10cloud-services-team (FY2023/2024-Q3-Q4): [trove] wrong quota_usages values in project tf-infra-test - https://phabricator.wikimedia.org/T359412#9607688 (10fnegri) 05Open→03In progress p:05Triage→03Medium a:03fnegri [16:03:50] (ProbeDown) resolved: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:14:53] 10Striker, 10wikitech.wikimedia.org, 10Bitu, 06Infrastructure-Foundations: LDAP account that is not attached on wikitech has no means for password reset - https://phabricator.wikimedia.org/T174469#9607713 (10bd808) >>! In T174469#9605736, @taavi wrote: > Is this still an issue as #Bitu is now used for pass... [16:24:03] (03PS1) 10Majavah: Link to idm.wm.o for password resets [labs/striker] - 10https://gerrit.wikimedia.org/r/1009295 (https://phabricator.wikimedia.org/T174469) [16:24:11] (03PS2) 10Majavah: Link to idm.wm.o for password resets [labs/striker] - 10https://gerrit.wikimedia.org/r/1009295 (https://phabricator.wikimedia.org/T174469) [16:30:50] 10Cloud-VPS (Project-requests): Request creation of ipoid-opensearch VPS project - https://phabricator.wikimedia.org/T359298#9607773 (10dcaro) +1 [16:51:17] 10Cloud-VPS (Project-requests): Request creation of ipoid-opensearch VPS project - https://phabricator.wikimedia.org/T359298#9607904 (10fnegri) a:03fnegri [16:59:40] !log fnegri@cloudcumin1001 ipoidopensearch START - Cookbook wmcs.vps.create_project for project ipoidopensearch in eqiad1 (T359298) [16:59:42] fnegri@cloudcumin1001: Unknown project "ipoidopensearch" [16:59:42] T359298: Request creation of ipoid-opensearch VPS project - https://phabricator.wikimedia.org/T359298 [17:01:30] 10Cloud Services Proposals, 10cloud-services-team (FY2023/2024-Q3-Q4): Decision Request - Incident Response Process - https://phabricator.wikimedia.org/T348887#9607960 (10fnegri) 05Open→03In progress [17:02:47] !log fnegri@cloudcumin1001 ipoidopensearch END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project ipoidopensearch in eqiad1 (T359298) [17:02:50] fnegri@cloudcumin1001: Unknown project "ipoidopensearch" [17:05:50] !log fnegri@cloudcumin1001 ipoidopensearch START - Cookbook wmcs.vps.add_user_to_project for user 'kharlan' in role 'member' (T359298) [17:05:53] T359298: Request creation of ipoid-opensearch VPS project - https://phabricator.wikimedia.org/T359298 [17:05:56] !log fnegri@cloudcumin1001 ipoidopensearch END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'kharlan' in role 'member' (T359298) [17:08:57] 10Wikibugs: Store PhorgeFeedReader.poll_last_seen_chrono_key in Redis - https://phabricator.wikimedia.org/T359009#9607988 (10valhallasw) The original design choice was to start from "now" to prevent the situation where the bot is down for one reason or another, and then starts to catch up on several days of back... [17:10:28] 10Cloud-VPS (Project-requests): Request creation of ipoid-opensearch VPS project - https://phabricator.wikimedia.org/T359298#9607989 (10fnegri) 05Open→03Resolved @kostajh project created, I've added you as a member. The default quotas are 8 cores, 16 GB or ram and 80 GB of disk space. Let us know if you need... [17:15:37] 10Wikibugs: Store PhorgeFeedReader.poll_last_seen_chrono_key in Redis - https://phabricator.wikimedia.org/T359009#9608031 (10bd808) The potential of multi-day replay is a good thing to consider. I agree that probably wouldn't be wanted. Having a "max_lag" setting of some sort seems possible. Or we could leave th... [17:29:18] 10Striker: Striker should use ID instead of username to identify SUL accounts - https://phabricator.wikimedia.org/T359428 (10taavi) [17:29:48] 10Striker: Striker should use ID instead of username to identify SUL accounts - https://phabricator.wikimedia.org/T359428#9608131 (10taavi) p:05Triage→03Medium [17:35:25] (03PS1) 10Majavah: labsauth: Add field for SUL account ID [labs/striker] - 10https://gerrit.wikimedia.org/r/1009310 (https://phabricator.wikimedia.org/T359428) [17:35:32] (03PS1) 10Majavah: labsauth: Store SUL user ID like username [labs/striker] - 10https://gerrit.wikimedia.org/r/1009311 (https://phabricator.wikimedia.org/T359428) [17:35:36] 10Cloud-VPS (Project-requests): Request creation of ipoid-opensearch VPS project - https://phabricator.wikimedia.org/T359298#9608158 (10kostajh) @fnegri thanks! [17:35:57] 10Striker, 06Infrastructure-Foundations, 07LDAP, 15User-bd808: Store Wikimedia unified account name (SUL) in LDAP directory - https://phabricator.wikimedia.org/T148048#9608162 (10bd808) `lang=irc [15:58] < taavi> bd808: is the striker work for T148048 still something you plan to do soon or can I take o... [17:37:30] (03CR) 10CI reject: [V: 04-1] labsauth: Store SUL user ID like username [labs/striker] - 10https://gerrit.wikimedia.org/r/1009311 (https://phabricator.wikimedia.org/T359428) (owner: 10Majavah) [17:43:20] 10Toolforge (Toolforge iteration 07), 06cloud-services-team: Harbor uploads sometimes fail due to tmpfs space on project-proxy - https://phabricator.wikimedia.org/T354116#9608209 (10dcaro) The only idea I have is to try to upload several things in parallel. Another option to try the same, would be to setup an... [17:47:06] 10Wikibugs, 15User-bd808: Add support for alternate channels files to make testing/debugging easier - https://phabricator.wikimedia.org/T359202#9608220 (10bd808) This has expanded in my working copy to also include addressing the `# TODO use the channelfilter module` comment in wikibugs2.gerrit.channel_filter.... [17:47:22] (03PS2) 10Majavah: labsauth: Store SUL user ID like username [labs/striker] - 10https://gerrit.wikimedia.org/r/1009311 (https://phabricator.wikimedia.org/T359428) [17:51:49] (03PS1) 10Eugene233: Editing a claim does not define the inital claim for the same session [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1008921 (https://phabricator.wikimedia.org/T359430) [17:52:12] (CloudVPSDesignateLeaks) firing: (5) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:53:05] (03CR) 10Eugene233: [C: 03+2] Editing a claim does not define the inital claim for the same session [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1008921 (https://phabricator.wikimedia.org/T359430) (owner: 10Eugene233) [17:53:29] (03Merged) 10jenkins-bot: Editing a claim does not define the inital claim for the same session [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1008921 (https://phabricator.wikimedia.org/T359430) (owner: 10Eugene233) [17:54:14] 10Cloud-VPS (Project-requests): Request creation of ipoid-opensearch VPS project - https://phabricator.wikimedia.org/T359298#9608247 (10kostajh) >>! In T359298#9607989, @fnegri wrote: > @kostajh project created, I've added you as a member. The default quotas are 8 cores, 16 GB or ram and 80 GB of disk space. Let... [17:57:11] (CloudVPSDesignateLeaks) resolved: (5) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:57:45] 10superset.wmcloud.org: Lack or limitations of concurrency - https://phabricator.wikimedia.org/T359431 (10Lofhi) [18:01:37] 10Wikibugs, 15User-bd808: Set a timeout for phorge requests - https://phabricator.wikimedia.org/T359145#9608280 (10CodeReviewBot) bd808 merged https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/10 phorge: Replace `fab` library with local client [18:05:53] 10Wikibugs, 15User-bd808: wikibugs no longer says new tasks are "NEW" - https://phabricator.wikimedia.org/T127506#9608292 (10CodeReviewBot) bd808 merged https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/11 phorge: Check for "core:create" transaction type to flag new tasks [18:36:38] 05Grid-Engine-to-K8s-Migration: Migrate addletterboxdfilmidbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357549#9608350 (10Carlinmack) Hi, Yes I operate three regularly running bots https://www.wikidata.org/wiki/User:Carlinmack (the latter three). The code for them... [18:38:51] 10Wikibugs, 15User-bd808: Set a timeout for phorge requests - https://phabricator.wikimedia.org/T359145#9608357 (10bd808) 05In progress→03Resolved [18:40:06] 10Wikibugs, 15User-bd808: Post deploy verification for T127506 fix - https://phabricator.wikimedia.org/T359433 (10bd808) 03NEW [18:40:41] 10Wikibugs, 15User-bd808: wikibugs no longer says new tasks are "NEW" - https://phabricator.wikimedia.org/T127506#9608379 (10bd808) [18:40:53] 10Wikibugs, 15User-bd808: Post deploy verification for T127506 fix - https://phabricator.wikimedia.org/T359433#9608377 (10bd808) 05Open→03Resolved `lang=irc [18:40] < wikibugs> Wikibugs, User-bd808: Post deploy verification for T127506 fix - https://phabricator.wikimedia.org/T359433 (bd808) NEW ` [18:41:09] 10Wikibugs, 15User-bd808: wikibugs no longer says new tasks are "NEW" - https://phabricator.wikimedia.org/T127506#9608380 (10bd808) 05In progress→03Resolved See T359433#9608377 [18:58:27] (03CR) 10BryanDavis: [C: 04-2] "poke" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/1008016 (https://phabricator.wikimedia.org/T90594) (owner: 10BryanDavis) [19:04:38] 10Wikibugs, 15User-bd808: wikibugs having a hard time staying connected to libera.chat IRC network - https://phabricator.wikimedia.org/T357729#9608455 (10bd808) The "connection lost" log event metric is looking much better, but with znc as a proxy I would have expected this no matter what. `lang=shell-session... [19:11:10] 10Wikibugs: Filter debug log messages out of logs sent to disk - https://phabricator.wikimedia.org/T359230#9608470 (10bd808) One way to expose control of this that might be cute would be to make the `-v, --verbose` flag's additive effect follow this progression: * no flag: log to disk and console at WARNING leve... [19:24:23] (03CR) 10Majavah: [C: 03+2] Fix MediaWiki dependency installation issues [labs/striker] - 10https://gerrit.wikimedia.org/r/1008948 (https://phabricator.wikimedia.org/T355523) (owner: 10Majavah) [19:24:26] (03CR) 10Majavah: [C: 03+2] Drop Nose testing setup [labs/striker] - 10https://gerrit.wikimedia.org/r/1008957 (owner: 10Majavah) [19:26:07] (03Merged) 10jenkins-bot: Fix MediaWiki dependency installation issues [labs/striker] - 10https://gerrit.wikimedia.org/r/1008948 (https://phabricator.wikimedia.org/T355523) (owner: 10Majavah) [19:27:10] 10Cloud-VPS (Project-requests): Request creation of ipoid-opensearch VPS project - https://phabricator.wikimedia.org/T359298#9608580 (10bd808) >>! In T359298#9608247, @kostajh wrote: > Should there be an option like `g3.cores8.ram16.disk80` for 80GB? I see `g3.cores8.ram16.disk20` for 20GB of disk space. See htt... [19:27:19] (03Merged) 10jenkins-bot: Drop Nose testing setup [labs/striker] - 10https://gerrit.wikimedia.org/r/1008957 (owner: 10Majavah) [19:27:21] (03Merged) 10jenkins-bot: Upgrade base container to Debian Bookworm [labs/striker] - 10https://gerrit.wikimedia.org/r/1008958 (owner: 10Majavah) [19:32:47] 10Striker: Striker dev env fails to link sulwiki account to phabricator - https://phabricator.wikimedia.org/T355523#9608616 (10taavi) 05Open→03Resolved [19:38:21] (03PS1) 10BryanDavis: [WIP] Store SUL information in LDAP [labs/striker] - 10https://gerrit.wikimedia.org/r/1009321 (https://phabricator.wikimedia.org/T148048) [19:40:20] (03CR) 10CI reject: [V: 04-1] [WIP] Store SUL information in LDAP [labs/striker] - 10https://gerrit.wikimedia.org/r/1009321 (https://phabricator.wikimedia.org/T148048) (owner: 10BryanDavis) [19:40:42] (03CR) 10BryanDavis: "Here's what I had in my local tree Taavi. Hopefully some parts of it will be useful for you." [labs/striker] - 10https://gerrit.wikimedia.org/r/1009321 (https://phabricator.wikimedia.org/T148048) (owner: 10BryanDavis) [19:43:29] (03PS2) 10BryanDavis: [WIP] Store SUL information in LDAP [labs/striker] - 10https://gerrit.wikimedia.org/r/1009321 (https://phabricator.wikimedia.org/T148048) [19:44:41] (03CR) 10CI reject: [V: 04-1] [WIP] Store SUL information in LDAP [labs/striker] - 10https://gerrit.wikimedia.org/r/1009321 (https://phabricator.wikimedia.org/T148048) (owner: 10BryanDavis) [19:47:52] (03PS3) 10BryanDavis: [WIP] Store SUL information in LDAP [labs/striker] - 10https://gerrit.wikimedia.org/r/1009321 (https://phabricator.wikimedia.org/T148048) [20:07:55] 10Cloud-VPS (Project-requests): Request creation of ipoid-opensearch VPS project - https://phabricator.wikimedia.org/T359298#9608669 (10kostajh) >>! In T359298#9608580, @bd808 wrote: >>>! In T359298#9608247, @kostajh wrote: >> Should there be an option like `g3.cores8.ram16.disk80` for 80GB? I see `g3.cores8.ram... [20:26:41] (03CR) 10Majavah: "Thanks. This looks very similar to what I had in mind." [labs/striker] - 10https://gerrit.wikimedia.org/r/1009321 (https://phabricator.wikimedia.org/T148048) (owner: 10BryanDavis) [20:57:50] (ProbeDown) firing: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:02:50] (ProbeDown) resolved: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:22:32] (03CR) 10BryanDavis: "Agreed. The LabsUser.update_sul_info method I added here was the start of my idea about how to backfill that information for existing acco" [labs/striker] - 10https://gerrit.wikimedia.org/r/1009321 (https://phabricator.wikimedia.org/T148048) (owner: 10BryanDavis) [21:30:28] (03PS2) 10Majavah: Require SUL/Phab links before applying for access [labs/striker] - 10https://gerrit.wikimedia.org/r/1008960 (https://phabricator.wikimedia.org/T172899) [21:37:18] 06cloud-services-team, 13Patch-For-Review, 15User-dcaro, 15User-fgiunchedi: [wmcs][alerting] Allow volunteer admins silencing alerts from cloudvps/toolforge/paws/quarry - https://phabricator.wikimedia.org/T320973#9609361 (10andrea.denisse) Hi @dcaro , T333615 is completed. This is unblocked. [22:21:53] 10Wikibugs, 10GitLab (Integrations), 10Release-Engineering-Team (Priority Backlog 📥): Connect WikiBugs IRC bot to Wikimedia GitLab - https://phabricator.wikimedia.org/T288381#9609449 (10bd808) a:05Legoktm→03None Unlicking this cookie for @Legoktm. I have some hope that this will be the fun treat I get to... [23:08:07] 05Grid-Engine-to-K8s-Migration: Migrate ganfilter from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357554#9609612 (10coldchrist) I'll plan on doing the switchover this weekend. Can I make the code changes first and the job changes second, or do they have to be coordinated?...