[00:10:03] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [00:44:17] 10Tool-gitlab-account-approval, 10User-bd808: Consider adding Gerrit Trusted-Users group as source of trust - https://phabricator.wikimedia.org/T353914 (10CodeReviewBot) bd808 merged https://gitlab.wikimedia.org/toolforge-repos/gitlab-account-approval/-/merge_requests/8 Trust users trusted in Gerrit [01:15:19] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [01:20:19] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [01:35:49] 10Striker, 10GitLab (Auth & Access), 10Release-Engineering-Team (Radar): Automatically approve GitLab accounts created by Striker integration - https://phabricator.wikimedia.org/T344667 (10bd808) The new #tool-gitlab-account-approval tool is now patrolling for unapproved accounts every 3 minutes and marking... [03:10:03] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [04:27:44] (03PS1) 10Legoktm: List the source pages on the gallery pages [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/987213 [04:28:05] (03CR) 10CI reject: [V: 04-1] List the source pages on the gallery pages [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/987213 (owner: 10Legoktm) [04:29:10] (03PS2) 10Legoktm: List the source pages on the gallery pages [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/987213 [04:59:57] 10Grid-Engine-to-K8s-Migration: Migrate potd from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319974 (10Legoktm) I think we're probably ready to go on this. I flipped the list configuration to allow HTML mails to go through untouched (i.e. not converted to plaintext). FYI @P... [05:11:38] 10Grid-Engine-to-K8s-Migration: Migrate potd from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319974 (10CodeReviewBot) legoktm merged https://gitlab.wikimedia.org/toolforge-repos/potd/-/merge_requests/1 Send an HTML email over SMTP [05:16:43] 10Grid-Engine-to-K8s-Migration: Migrate potd from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319974 (10Legoktm) I've commented out the crontab and: ` tools.potd@tools-sgebastion-10:~/potd$ toolforge jobs list Job name: Job type: Status: ----------- ---------... [05:40:49] 10Toolforge (Toolforge iteration 02), 10Toolforge Build Service, 10Patch-For-Review: Add Rust buildpack to Toolforge build service - https://phabricator.wikimedia.org/T337066 (10Legoktm) Thanks!!! I skimmed through https://gitlab.wikimedia.org/repos/cloud/toolforge/rust-buildpack/-/blob/move_to_api_0.10/targ... [05:57:22] 10Tool-toolwatch: Improve the functionality of Toolwatch - https://phabricator.wikimedia.org/T353573 (10Aklapper) >>! In T353573#9411419, @Reputation22 wrote: > We will close the github issue @Aklapper, thanks :) Yay. When? :) [06:10:03] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [06:21:24] 10Grid-Engine-to-K8s-Migration, 10User-Jayprakash12345: Migrate indic-wsstats from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319811 (10Jayprakash12345) a:05Jayprakash12345→03ParasharSarthak [06:27:10] 10cloud-services-team: Cloud Services: introduce some kind of email list as feedback collector - https://phabricator.wikimedia.org/T332907 (10Aklapper) 05Open→03Declined Boldly declining after a potential feel-good "let the team brainstorm a bit without a followup concept" team offsite activity. The proposa... [06:33:55] 10Tool-refill, 10Internet-Archive: Properly handle WebCite links - https://phabricator.wikimedia.org/T352409 (10Green_Cardamom) We still have over 1.2 million instances of webcitation.org .. half are on two sites ruwiki (441k) and ukwiki (216k). Other large sites include Chechnya (64k) and Armenia (50k). It's... [07:59:19] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [08:04:19] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [09:10:03] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [09:45:00] 10Tool-toolwatch: Improve the functionality of Toolwatch - https://phabricator.wikimedia.org/T353573 (10Gopavasanth) Hi @Aklapper, it seems like there are no issues listed on https://github.com/gopavasanth/ToolWatch/issues. To ensure centralized issue tracking, let's use the Phabricator tool under https://phabri... [09:46:00] 10Tool-toolwatch: Improve the functionality of Toolwatch - https://phabricator.wikimedia.org/T353573 (10Gopavasanth) [09:56:01] 10Tool-toolwatch: Improve the functionality of Toolwatch - https://phabricator.wikimedia.org/T353573 (10Aklapper) @Gopavasanth Anyone ending up first on https://github.com/gopavasanth/ToolWatch/issues will likely not find your comment that you posted at https://phabricator.wikimedia.org/T353573#9431896 . :) Plea... [10:01:56] (ToolsNfsAlmostFull) firing: Toolforge NFS is 0.852064614041399/1 full - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNfsAlmostFull - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsNfsAlmostFull [10:26:40] 10Toolforge (Toolforge iteration 02), 10cloud-services-team (FY2023/2024-Q1-Q2), 10User-dcaro: [harbor] Redis using all available memory - https://phabricator.wikimedia.org/T354176 (10dcaro) 05In progress→03Stalled [10:28:39] 10Toolforge (Toolforge iteration 02): [builds-builder] Clojure support was dropped from the builder image - https://phabricator.wikimedia.org/T354252 (10dcaro) [10:33:16] 10Toolforge (Toolforge iteration 02), 10cloud-services-team (FY2023/2024-Q1-Q2), 10User-dcaro: [harbor] Redis using all available memory - https://phabricator.wikimedia.org/T354176 (10fnegri) There's still a downward trend that is likely to result in OOM after a few days :/ {F41649531} [10:46:44] 10Toolforge (Toolforge iteration 02), 10Toolforge Build Service, 10Patch-For-Review: Add Rust buildpack to Toolforge build service - https://phabricator.wikimedia.org/T337066 (10dcaro) The idea is not to fork it no, I just tried to add the minimum changes to make it work with the upstream builder (that uses... [10:54:41] 10Toolforge (Toolforge iteration 02), 10cloud-services-team (FY2023/2024-Q1-Q2), 10User-dcaro: [harbor] Redis using all available memory - https://phabricator.wikimedia.org/T354176 (10dcaro) Looking 90 days far, the rate at which the memory was being used was ~0.089 GB/day, currently is around 0.065 GB/day,... [11:04:01] 10Toolforge (Toolforge iteration 02), 10cloud-services-team (FY2023/2024-Q1-Q2), 10User-dcaro: [harbor] Redis using all available memory - https://phabricator.wikimedia.org/T354176 (10dcaro) >>! In T354176#9432056, @dcaro wrote: > Looking 90 days far, the rate at which the memory was being used was ~0.089 GB... [11:08:23] 10Toolforge (Toolforge iteration 02), 10cloud-services-team (FY2023/2024-Q1-Q2), 10User-dcaro: [harbor] Redis using all available memory - https://phabricator.wikimedia.org/T354176 (10dcaro) There has been a substantial increase in the number of pulls on harbor side https://grafana.wmcloud.org/d/m9V1RQs4k/ha... [11:28:12] 10Cloud Services Proposals, 10Toolforge Build Service (Iteration 11): Decision request - What buildpacks to allow and include for toolforge build service beta - https://phabricator.wikimedia.org/T330102 (10dcaro) >>! In T330102#9412537, @taavi wrote: > Yes, but that seems to come with the cost of increased com... [12:10:03] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [12:17:06] 10Tool-inteGraality, 10Wikibase (3rd party installations), 10Wikibase Suite Team, 10Product-Feature: integraality for Wikibases? - https://phabricator.wikimedia.org/T294892 (10darthmon_wmde) [13:01:56] (ToolsNfsAlmostFull) firing: Toolforge NFS is 0.8522071394373583/1 full - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNfsAlmostFull - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsNfsAlmostFull [13:10:33] 10Toolforge (Toolforge iteration 02), 10Documentation: [tbs] Improve Harbor quota handling and docs - https://phabricator.wikimedia.org/T351092 (10Slst2020) [13:10:35] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [maintain-harbor] Manage project quotas via maintain-harbor - https://phabricator.wikimedia.org/T352417 (10Slst2020) 05In progress→03Stalled [13:11:06] 10cloud-services-team, 10WMDE-Analytics-Engineering, 10Wikibase - Federated Properties, 10Wikidata, 10Cloud-Services-Origin-Alert: Wikidata-related Cloud VPS alerts about puppet - https://phabricator.wikimedia.org/T354268 (10Michael) [13:15:04] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [maintain-harbor] Manage project quotas via maintain-harbor - https://phabricator.wikimedia.org/T352417 (10Slst2020) Turns out accessing the GET `/quotas` endpoint also requires the user to have system admin rights. Pausing this task while investigating... [13:19:19] 10Toolforge (Toolforge iteration 02): [harbor] Investigate new robot account permissions in Harbor 2.10 - https://phabricator.wikimedia.org/T354270 (10Slst2020) [13:40:41] 10cloud-services-team, 10WMDE-Analytics-Engineering, 10Wikibase - Federated Properties, 10Wikidata, 10Cloud-Services-Origin-Alert: Wikidata-related Cloud VPS alerts about puppet - https://phabricator.wikimedia.org/T354268 (10dcaro) For the wikidata-federated-properties and wb-reconcile, the issue is that... [13:51:07] 10cloud-services-team, 10WMDE-Analytics-Engineering, 10Wikibase - Federated Properties, 10Wikidata, 10Cloud-Services-Origin-Alert: Wikidata-related Cloud VPS alerts about puppet - https://phabricator.wikimedia.org/T354268 (10dcaro) For `wikidata-analytics-1`, it seems that there's a docker installation... [13:53:09] 10cloud-services-team, 10WMDE-Analytics-Engineering, 10Wikibase - Federated Properties, 10Wikidata, 10Cloud-Services-Origin-Alert: Wikidata-related Cloud VPS alerts about puppet - https://phabricator.wikimedia.org/T354268 (10dcaro) The last alert, is because your puppetmaster `wdqspuppet` was not cleaned... [13:53:47] 10cloud-services-team, 10WMDE-Analytics-Engineering, 10Wikibase - Federated Properties, 10Wikidata, 10Cloud-Services-Origin-Alert: Wikidata-related Cloud VPS alerts about puppet - https://phabricator.wikimedia.org/T354268 (10Michael) >>! In T354268#9432600, @dcaro wrote: > For `wikidata-analytics-1`, it... [13:58:23] 10cloud-services-team, 10WMDE-Analytics-Engineering, 10Wikibase - Federated Properties, 10Wikidata, 10Cloud-Services-Origin-Alert: Wikidata-related Cloud VPS alerts about puppet - https://phabricator.wikimedia.org/T354268 (10Michael) >>! In T354268#9432605, @dcaro wrote: > The last alert, is because your... [14:06:04] 10cloud-services-team, 10WMDE-Analytics-Engineering, 10Wikibase - Federated Properties, 10Wikidata, 10Cloud-Services-Origin-Alert: Wikidata-related Cloud VPS alerts about puppet - https://phabricator.wikimedia.org/T354268 (10AndrewTavis_WMDE) Hey @dcaro 👋 Really appreciate you checking all of this for us... [14:10:29] 10cloud-services-team, 10WMDE-Analytics-Engineering, 10Wikibase - Federated Properties, 10Wikidata, 10Cloud-Services-Origin-Alert: Wikidata-related Cloud VPS alerts about puppet - https://phabricator.wikimedia.org/T354268 (10dcaro) >>! In T354268#9432641, @AndrewTavis_WMDE wrote: > Hey @dcaro 👋 Really ap... [14:14:16] 10cloud-services-team, 10WMDE-Analytics-Engineering, 10Wikibase - Federated Properties, 10Wikidata, 10Cloud-Services-Origin-Alert: Wikidata-related Cloud VPS alerts about puppet - https://phabricator.wikimedia.org/T354268 (10dcaro) >>! In T354268#9432608, @Michael wrote: >>>! In T354268#9432605, @dcaro w... [14:15:23] 10Toolforge (Toolforge iteration 02): [harbor] Investigate new robot account permissions in Harbor 2.10 - https://phabricator.wikimedia.org/T354270 (10Slst2020) 05Open→03In progress Tried this locally. While robot accounts now have extended and much more granular permissions, dealing with quotas is sadly sti... [14:15:31] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [maintain-harbor] Manage project quotas via maintain-harbor - https://phabricator.wikimedia.org/T352417 (10Slst2020) [14:15:54] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [maintain-harbor] Manage project quotas via maintain-harbor - https://phabricator.wikimedia.org/T352417 (10Slst2020) [14:16:23] 10Toolforge (Toolforge iteration 02): [harbor] Investigate new robot account permissions in Harbor 2.10 - https://phabricator.wikimedia.org/T354270 (10Slst2020) 05In progress→03Resolved [14:23:35] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [maintain-harbor] Manage project quotas via maintain-harbor - https://phabricator.wikimedia.org/T352417 (10Slst2020) >>! In T352417#9432507, @Slst2020 wrote: > Turns out accessing the GET `/quotas` endpoint also requires the user to have system admin ri... [14:26:33] 10cloud-services-team, 10WMDE-Analytics-Engineering, 10Wikibase - Federated Properties, 10Wikidata, 10Cloud-Services-Origin-Alert: Wikidata-related Cloud VPS alerts about puppet - https://phabricator.wikimedia.org/T354268 (10Michael) >>! In T354268#9432687, @dcaro wrote: >>>! In T354268#9432608, @Michael... [14:30:47] 10cloud-services-team, 10WMDE-Analytics-Engineering, 10Wikibase - Federated Properties, 10Wikidata, 10Cloud-Services-Origin-Alert: Wikidata-related Cloud VPS alerts about puppet - https://phabricator.wikimedia.org/T354268 (10AndrewTavis_WMDE) Thanks from me as well, @dcaro! Following up on my point: > Y... [14:37:17] 10cloud-services-team, 10WMDE-Analytics-Engineering, 10Wikibase - Federated Properties, 10Wikidata, 10Cloud-Services-Origin-Alert: Wikidata-related Cloud VPS alerts about puppet - https://phabricator.wikimedia.org/T354268 (10Michael) >>! In T354268#9432766, @AndrewTavis_WMDE wrote: > Can I ask what the... [14:40:07] 10cloud-services-team, 10WMDE-Analytics-Engineering, 10Wikibase - Federated Properties, 10Wikidata, 10Cloud-Services-Origin-Alert: Wikidata-related Cloud VPS alerts about puppet - https://phabricator.wikimedia.org/T354268 (10AndrewTavis_WMDE) Thanks, @Michael 🙏 For further understanding, what's the frequ... [14:43:28] 10Toolforge (Toolforge iteration 02): [buildservice] Feature request - Indicate when long envvars are cutoff when listing - https://phabricator.wikimedia.org/T353287 (10Slst2020) 05Open→03In progress [14:48:39] 10cloud-services-team, 10WMDE-Analytics-Engineering, 10Wikibase - Federated Properties, 10Wikidata, 10Cloud-Services-Origin-Alert: Wikidata-related Cloud VPS alerts about puppet - https://phabricator.wikimedia.org/T354268 (10AndrewTavis_WMDE) Quoting myself: > For this we'll need to go into the deployme... [14:54:05] 10cloud-services-team, 10WMDE-Analytics-Engineering, 10Wikibase - Federated Properties, 10Wikidata, 10Cloud-Services-Origin-Alert: Wikidata-related Cloud VPS alerts about puppet - https://phabricator.wikimedia.org/T354268 (10dcaro) >>! In T354268#9432766, @AndrewTavis_WMDE wrote: > Thanks from me as well... [14:59:24] 10cloud-services-team, 10WMDE-Analytics-Engineering, 10Wikibase - Federated Properties, 10Wikidata, 10Cloud-Services-Origin-Alert: Wikidata-related Cloud VPS alerts about puppet - https://phabricator.wikimedia.org/T354268 (10dcaro) You can probably change the default address pools in docker: ` # /etc/doc... [15:04:42] 10cloud-services-team, 10WMDE-Analytics-Engineering, 10Wikibase - Federated Properties, 10Wikidata, 10Cloud-Services-Origin-Alert: Wikidata-related Cloud VPS alerts about puppet - https://phabricator.wikimedia.org/T354268 (10Michael) >>! In T354268#9432817, @AndrewTavis_WMDE wrote: > Thanks, @Michael 🙏 F... [15:05:15] 10ToolforgeBundle, 10AhoCorasick, 10CommonPasswords, 10Equivset, and 26 others: Make new releases of all Wikimedia-authored PHP libraries, and bump their usages (mid-2021) - https://phabricator.wikimedia.org/T287972 (10Krinkle) [15:08:32] 10cloud-services-team, 10WMDE-Analytics-Engineering, 10Wikibase - Federated Properties, 10Wikidata, 10Cloud-Services-Origin-Alert: Wikidata-related Cloud VPS alerts about puppet - https://phabricator.wikimedia.org/T354268 (10AndrewTavis_WMDE) Thank you for the further information here, @dcaro! Sorry that... [15:10:03] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [15:34:31] 10Tool-Pageviews: Add support Wikifunctions - https://phabricator.wikimedia.org/T354285 (10Wargo) [15:35:27] 10PAWS, 10Pywikibot, 10Documentation, 10Pywikibot-Documentation, 10good first task: Move Pywikibot PAWS tutorial out of Jupyter notebook onto wiki - https://phabricator.wikimedia.org/T342397 (10rook) 05Open→03Resolved [16:01:56] (ToolsNfsAlmostFull) firing: Toolforge NFS is 0.8526418645770052/1 full - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNfsAlmostFull - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsNfsAlmostFull [17:19:30] 10Tool-Pageviews: Add support for Wikifunctions.org - https://phabricator.wikimedia.org/T354285 (10Reedy) [17:27:06] 10cloud-services-team (FY2023/2024-Q1-Q2), 10Cloud-Services-Origin-User, 10Cloud-Services-Worktype-Unplanned, 10User-dcaro: [puppet] Remove expired and unused certs from modules/profile/files/ssl/ and modules/base/files/ca - https://phabricator.wikimedia.org/T354294 (10dcaro) p:05Triage→03High [17:27:26] 10cloud-services-team (FY2023/2024-Q1-Q2), 10Cloud-Services-Origin-User, 10Cloud-Services-Worktype-Unplanned, 10User-dcaro: [puppet] Remove expired and unused certs from modules/profile/files/ssl/ and modules/base/files/ca - https://phabricator.wikimedia.org/T354295 (10dcaro) p:05Triage→03High [17:31:29] 10Data-Services: Missing linktarget rows on wiki replicas - https://phabricator.wikimedia.org/T354089 (10Legoktm) 05Open→03Invalid I didn't realize the view only surfaces linktarget rows if there's a corresponding *links row, that feels weird to me but makes sense. I'm not sure why it's fluctuating either, b... [17:31:56] 10cloud-services-team (FY2023/2024-Q1-Q2), 10Cloud-Services-Origin-User, 10Cloud-Services-Worktype-Unplanned, 10User-dcaro: [puppet] Remove expired and unused certs from modules/profile/files/ssl/ and modules/base/files/ca - https://phabricator.wikimedia.org/T354295 (10dcaro) [17:33:44] 10cloud-services-team, 10WMDE-Analytics-Engineering, 10Wikibase - Federated Properties, 10Wikidata, 10Cloud-Services-Origin-Alert: Wikidata-related Cloud VPS alerts about puppet - https://phabricator.wikimedia.org/T354268 (10dcaro) No problem, I'm here to help :}, my time is limited though, so might take... [17:58:19] 10cloud-services-team (FY2023/2024-Q1-Q2), 10Cloud-Services-Origin-User, 10Cloud-Services-Worktype-Unplanned, 10User-dcaro: [puppet] Remove expired and unused certs from modules/profile/files/ssl/ and modules/base/files/ca - https://phabricator.wikimedia.org/T354295 (10fnegri) @BBlack you mentioned in IRC... [18:07:37] 10VPS-project-Wikistats, 10collaboration-services, 10User-RhinosF1: Add 'wikitide' to wikistats - https://phabricator.wikimedia.org/T349660 (10Dzahn) a:03Dzahn [18:10:03] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [18:13:53] 10VPS-project-Wikistats, 10collaboration-services, 10User-RhinosF1: Add 'wikitide' to wikistats - https://phabricator.wikimedia.org/T349660 (10Dzahn) 05Open→03Stalled Talked to RhinosF1. Wikitide will merge into Miraheze. We are hoping that all Wikitide wikis will be returned by MH Wikidiscover and th... [18:23:26] 10PAWS: update fcos version in PAWS in eqiad1 - https://phabricator.wikimedia.org/T353077 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/361 [18:23:33] vivian-rook opened https://github.com/toolforge/paws/pull/361 [19:01:56] (ToolsNfsAlmostFull) firing: Toolforge NFS is 0.8529272221763826/1 full - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNfsAlmostFull - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsNfsAlmostFull [19:40:07] 10PAWS: update fcos version in PAWS in eqiad1 - https://phabricator.wikimedia.org/T353077 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/361 [19:40:09] 10PAWS: update fcos version in PAWS in eqiad1 - https://phabricator.wikimedia.org/T353077 (10rook) 05Open→03Resolved [19:40:13] vivian-rook closed https://github.com/toolforge/paws/pull/361 [19:40:35] 10PAWS: update fcos version in PAWS in eqiad1 - https://phabricator.wikimedia.org/T353077 (10rook) [20:01:39] 10Tool-bub2: Google Books, Trove and PDL integration to Commons - https://phabricator.wikimedia.org/T352578 (10Okerekechinweotito) [20:03:28] 10Tool-bub2: Google Books and Trove integration to Commons - https://phabricator.wikimedia.org/T352578 (10Okerekechinweotito) [20:04:11] 10Tool-bub2: Integrate PDL to Commons - https://phabricator.wikimedia.org/T354304 (10Okerekechinweotito) [20:06:16] 10Grid-Engine-to-K8s-Migration: Migrate redirtalkdeleter from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319998 (10Andriy.v) >>! In T319998#9431013, @Qwerfjkl wrote: >>>! In T319998#9428949, @Andriy.v wrote: >>>>! In T319998#9423736, @Lejonel wrote: >>> https://wikitech... [20:48:46] 10wikitech.wikimedia.org: Requesting access for - https://phabricator.wikimedia.org/T354307 (10FabioCarrieri) [20:49:19] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [20:54:19] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [20:56:56] (ToolsNfsAlmostFull) resolved: Toolforge NFS is 0.8516741784548701/1 full - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNfsAlmostFull - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsNfsAlmostFull [21:10:03] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [22:49:42] 10Grid-Engine-to-K8s-Migration: Migrate resizeimagesbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320004 (10Andriy.v) 05Open→03Resolved a:03Andriy.v Ok, I've migrated the tool to k8s.