[00:09:03] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:19:03] (InstanceDown) resolved: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:47:28] (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [01:02:28] (OpenstackAPIResponse) resolved: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [02:14:50] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [03:40:57] 10Tool-bub2, 10Outreach-Programs-Projects, 10Outreachy (Round 27): Use SMTP to send Emails to the users - https://phabricator.wikimedia.org/T338267 (10wassan.anmol117) [04:49:05] 10PAWS, 10Pywikibot, 10Documentation, 10Pywikibot-Documentation, 10good first task: Move Pywikibot PAWS tutorial out of Jupyter notebook onto wiki - https://phabricator.wikimedia.org/T342397 (10Enag2000) Link to current draft. https://www.mediawiki.org/wiki/Manual:Pywikibot/Draft:Pywikibot_and_PAWS [05:14:50] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [07:14:33] (03PS1) 10Hashar: Send Fresnel patches to releng [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/967130 [07:23:24] (03PS2) 10Hashar: Send Fresnel notifications to Release Engineering [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/967130 [07:27:15] (03PS1) 10Hashar: Remove #wikimedia-quibble (already in #wikimedia-releng) [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/967133 (https://phabricator.wikimedia.org/T346901) [08:14:50] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [10:27:03] (SystemdUnitDown) resolved: The service unit wmf_auto_restart_virtlogd.service is in failed status on host cloudvirt1051. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudvirt1051 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [10:27:03] (SystemdUnitDownForLong) resolved: The systemd unit wmf_auto_restart_virtlogd.service on node cloudvirt1051 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDownForLong - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudvirt1051 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDownForLong [10:53:15] 10Striker, 10Patch-For-Review, 10Release-Engineering-Team (Social Piranhas 🐟): Striker-created Diffusion mirrors of GitLab repos are empty (due to master vs main branch name mismatch) - https://phabricator.wikimedia.org/T348131 (10Aklapper) [10:56:37] 10Striker, 10Patch-For-Review, 10Release-Engineering-Team (Social Piranhas 🐟): Striker-created Diffusion mirrors of GitLab repos are empty (due to master vs main branch name mismatch) - https://phabricator.wikimedia.org/T348131 (10taavi) > This is due to using the default branch name `main` in Wikimedia GitL... [10:57:43] (03CR) 10Majavah: [C: 04-1] "I don't think this is a feature we want. Let's instead add a filter to the toolinfo editing interface to ensure URLs use a proper format a" [labs/striker] - 10https://gerrit.wikimedia.org/r/962144 (https://phabricator.wikimedia.org/T345776) (owner: 10Sohom Datta) [11:05:37] 10Striker, 10Patch-For-Review, 10Release-Engineering-Team (Social Piranhas 🐟): Striker-created Diffusion mirrors of GitLab repos are empty (due to master vs main branch name mismatch) - https://phabricator.wikimedia.org/T348131 (10Aklapper) @taavi: I think the answer to your question is "default" but let me... [11:06:33] 10Tool-Pageviews, 10Data-Engineering, 10Data Products (Sprint 02), 10Patch-For-Review: Mediarequests returning "file not found" for filenames with specific characters - https://phabricator.wikimedia.org/T347899 (10Sfaci) a:05Sfaciβ†’03SGupta-WMF [11:14:50] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [11:30:01] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [labs/tools/massmailer] - 10https://gerrit.wikimedia.org/r/967183 (owner: 10L10n-bot) [11:30:03] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [labs/tools/watch-translations] - 10https://gerrit.wikimedia.org/r/967184 (owner: 10L10n-bot) [12:32:54] (03CR) 10Majavah: [C: 03+2] Send Fresnel notifications to Release Engineering [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/967130 (owner: 10Hashar) [12:33:26] (03CR) 10Majavah: [C: 03+2] Remove #wikimedia-quibble (already in #wikimedia-releng) [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/967133 (https://phabricator.wikimedia.org/T346901) (owner: 10Hashar) [12:33:29] (03Merged) 10jenkins-bot: Send Fresnel notifications to Release Engineering [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/967130 (owner: 10Hashar) [12:34:10] (03Merged) 10jenkins-bot: Remove #wikimedia-quibble (already in #wikimedia-releng) [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/967133 (https://phabricator.wikimedia.org/T346901) (owner: 10Hashar) [12:47:32] 10PAWS, 10Pywikibot, 10Documentation, 10Pywikibot-Documentation, 10good first task: Move Pywikibot PAWS tutorial out of Jupyter notebook onto wiki - https://phabricator.wikimedia.org/T342397 (10rook) >>! In T342397#9263941, @Enag2000 wrote: > Link to current draft. https://www.mediawiki.org/wiki/Manual:P... [12:52:57] vivian-rook closed https://github.com/toolforge/paws/pull/336 [13:00:00] vivian-rook closed https://github.com/toolforge/paws/pull/336 [13:00:39] dependabot[bot] opened https://github.com/toolforge/paws/pull/340 [13:14:56] 10Cloud-VPS, 10cloud-services-team, 10SRE-OnFire, 10Sustainability (Incident Followup), 10User-dcaro: openstack: create a cookbook to inject commands to VMs via console at scale - https://phabricator.wikimedia.org/T347683 (10taavi) a:03taavi [13:42:35] (HarborDown) firing: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown [14:07:35] (HarborDown) resolved: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown [14:14:50] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [14:15:20] 10Tool-iw, 10cloud-services-team (Kanban), 10User-bd808: https://iw.toolforge.org breaks when not given any values - https://phabricator.wikimedia.org/T257104 (10bd808) [14:16:02] 10Tool-iw, 10Toolforge, 10cloud-services-team (Kanban), 10User-bd808: iw.toolforge.org strips query parameters when redirecting - https://phabricator.wikimedia.org/T258444 (10bd808) [14:29:29] 10Tool-iw, 10Toolforge: iw.toolforge.org does not support URL-encoded query parameters ([[toolforge:foo?bar]]) - https://phabricator.wikimedia.org/T345783 (10bd808) As far as I understand it, implementing this behavior basically make iw.toolforge the only interwiki target capable of accepting query strings. By... [14:53:17] 10cloud-services-team (Hardware), 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cloudcontrol100[8-10]-dev cloudnet100[7-8]-dev - https://phabricator.wikimedia.org/T342455 (10Jclark-ctr) [14:54:02] vivian-rook closed https://github.com/toolforge/paws/pull/340 [14:55:11] 10PAWS: jupyterlab to 4.0.7 - https://phabricator.wikimedia.org/T349203 (10rook) 05Openβ†’03In progress a:03rook [15:00:16] 10PAWS: update build-and-push action - https://phabricator.wikimedia.org/T348874 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/339 [15:00:19] vivian-rook closed https://github.com/toolforge/paws/pull/339 [15:01:15] 10PAWS: jupyterlab to 4.0.7 - https://phabricator.wikimedia.org/T349203 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/341 [15:01:30] vivian-rook opened https://github.com/toolforge/paws/pull/341 [15:10:22] 10cloud-services-team (Hardware), 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cloudcontrol100[8-10]-dev cloudnet100[7-8]-dev - https://phabricator.wikimedia.org/T342455 (10Jclark-ctr) [15:35:01] 10Toolforge (Toolforge iteration 01): [tbs][harbor] Improve Harbor admin docs - https://phabricator.wikimedia.org/T349313 (10Slst2020) [15:36:57] 10PAWS: jupyterlab to 4.0.7 - https://phabricator.wikimedia.org/T349203 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/341 [15:37:10] vivian-rook closed https://github.com/toolforge/paws/pull/341 [15:37:28] 10PAWS: jupyterlab to 4.0.7 - https://phabricator.wikimedia.org/T349203 (10rook) 05In progressβ†’03Resolved [15:59:09] (03CR) 10BryanDavis: [C: 03+2] Set default branch to "main" for GitLab repos mirrored to Diffusion [labs/striker] - 10https://gerrit.wikimedia.org/r/963292 (https://phabricator.wikimedia.org/T348131) (owner: 10Aklapper) [16:02:12] (03Merged) 10jenkins-bot: Set default branch to "main" for GitLab repos mirrored to Diffusion [labs/striker] - 10https://gerrit.wikimedia.org/r/963292 (https://phabricator.wikimedia.org/T348131) (owner: 10Aklapper) [16:09:34] (03PS1) 10David Caro: general: move to spicerack>8 [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/967244 [16:12:45] (03CR) 10CI reject: [V: 04-1] general: move to spicerack>8 [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/967244 (owner: 10David Caro) [16:16:06] (03CR) 10BryanDavis: [C: 04-1] Use full url if provided in the suburl field (031 comment) [labs/striker] - 10https://gerrit.wikimedia.org/r/962144 (https://phabricator.wikimedia.org/T345776) (owner: 10Sohom Datta) [16:32:03] (InstanceDown) firing: Project toolsbeta instance toolsbeta-puppetdb-02 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [16:37:03] (InstanceDown) resolved: Project toolsbeta instance toolsbeta-puppetdb-02 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [16:37:39] (03PS2) 10David Caro: general: move to spicerack>8 [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/967244 [16:52:45] 10Toolforge (Quota-requests): Request increased quota for Montage Toolforge tool - https://phabricator.wikimedia.org/T348894 (10mahmoud) Yeah, after a few days of monitoring, there's definite improvement. User complaints have gone down, but I'm still getting perf-related error notifications from our monitoring... [17:14:50] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [17:36:13] (03PS3) 10FNegri: general: move to spicerack>8 [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/967244 (https://phabricator.wikimedia.org/T348726) (owner: 10David Caro) [17:54:50] (PuppetAgentDisabled) firing: Puppet agent disabled on instance quarry-web-02 in project quarry - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentDisabled [18:05:07] 10Toolforge, 10MediaWiki-Core-AuthManager, 10MediaWiki-Platform-Team: BotPasswords + Toolforge combination causing daily "login from new device" warning emails - https://phabricator.wikimedia.org/T349336 (10Novem_Linguae) [18:06:34] 10Toolforge, 10MediaWiki-Core-AuthManager, 10MediaWiki-Platform-Team: BotPasswords + Toolforge combination causing daily "login from new device" warning emails - https://phabricator.wikimedia.org/T349336 (10Novem_Linguae) [18:07:14] (03CR) 10FNegri: "Instead of pinning elasticsearch-curator, you can pin" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/967244 (https://phabricator.wikimedia.org/T348726) (owner: 10David Caro) [18:08:52] 10Toolforge, 10MediaWiki-Core-AuthManager, 10MediaWiki-Platform-Team: BotPasswords + Toolforge combination causing daily "login from new device" warning emails - https://phabricator.wikimedia.org/T349336 (10Novem_Linguae) [18:13:53] 10Toolforge, 10MediaWiki-Core-AuthManager, 10MediaWiki-Platform-Team: BotPasswords + Toolforge combination causing daily "login from new device" warning emails - https://phabricator.wikimedia.org/T349336 (10Novem_Linguae) [18:21:35] 10Cloud-VPS, 10cloud-services-team (FY2023/2024-Q1), 10Patch-For-Review, 10User-dcaro: [wmcs-cookbooks] tox is failing - https://phabricator.wikimedia.org/T348726 (10fnegri) I'll cross-post here for future reference what I wrote in the patch comments: Version 2.31.0.7 of `types-requests` [introduced a dep... [18:29:19] 10cloud-services-team: Need baremetal system(s) with internet access - https://phabricator.wikimedia.org/T349003 (10rook) [19:05:09] 10Toolforge, 10Community-Tech, 10MediaWiki-Core-AuthManager, 10MediaWiki-Platform-Team, 10MediaWiki-extensions-LoginNotify: BotPasswords + Toolforge combination causing daily "login from new device" warning emails - https://phabricator.wikimedia.org/T349336 (10Ammarpad) [19:11:39] 10Toolforge: Can't pip install mysqlclient on Toolforge - https://phabricator.wikimedia.org/T349341 (10Ederporto) [19:15:12] 10Toolforge: Can't pip install mysqlclient on Toolforge - https://phabricator.wikimedia.org/T349341 (10LucasWerkmeister) > but pkg-config isn’t (Apparently the correct Debian package these days is [pkgconf](https://packages.debian.org/bookworm/pkgconf), a replacement for the now-transitional [pkg-config](https:... [19:42:53] 10Toolforge, 10Community-Tech, 10MediaWiki-Core-AuthManager, 10MediaWiki-Platform-Team, 10MediaWiki-extensions-LoginNotify: BotPasswords + Toolforge combination causing daily "login from new device" warning emails - https://phabricator.wikimedia.org/T349336 (10MusikAnimal) > Possible change to loginnotif... [19:43:42] 10Toolforge, 10MediaWiki-Email, 10MediaWiki-User-login-and-signup, 10MediaWiki-extensions-LoginNotify, 10User-notice: BotPasswords + Toolforge combination causing daily "login from new device" warning emails - https://phabricator.wikimedia.org/T182867 (10MusikAnimal) [19:43:52] 10Toolforge, 10Community-Tech, 10MediaWiki-Core-AuthManager, 10MediaWiki-Platform-Team, 10MediaWiki-extensions-LoginNotify: BotPasswords + Toolforge combination causing daily "login from new device" warning emails - https://phabricator.wikimedia.org/T349336 (10MusikAnimal) [19:44:30] 10Toolforge, 10MediaWiki-Email, 10MediaWiki-User-login-and-signup, 10MediaWiki-extensions-LoginNotify, 10User-notice: BotPasswords + Toolforge combination causing daily "login from new device" warning emails - https://phabricator.wikimedia.org/T182867 (10MusikAnimal) [19:47:55] 10Toolforge, 10MediaWiki-Email, 10MediaWiki-User-login-and-signup, 10MediaWiki-extensions-LoginNotify, 10User-notice: BotPasswords + Toolforge combination causing daily "login from new device" warning emails - https://phabricator.wikimedia.org/T182867 (10MusikAnimal) [19:47:57] 10Toolforge, 10MediaWiki-Email, 10MediaWiki-User-login-and-signup, 10MediaWiki-extensions-LoginNotify, 10User-notice: BotPasswords + Toolforge combination causing daily "login from new device" warning emails - https://phabricator.wikimedia.org/T182867 (10MusikAnimal) Syncing description from the dup repo... [19:48:39] 10Toolforge, 10MediaWiki-Email, 10MediaWiki-User-login-and-signup, 10MediaWiki-extensions-LoginNotify, 10User-notice: BotPasswords + Toolforge combination causing daily "login from new device" warning emails - https://phabricator.wikimedia.org/T182867 (10MusikAnimal) [19:51:00] 10Toolforge, 10MediaWiki-Email, 10MediaWiki-User-login-and-signup, 10MediaWiki-extensions-LoginNotify, 10User-notice: BotPasswords + Toolforge combination causing daily "login from new device" warning emails - https://phabricator.wikimedia.org/T182867 (10Novem_Linguae) [19:51:05] 10Toolforge, 10MediaWiki-Email, 10MediaWiki-User-login-and-signup, 10MediaWiki-extensions-LoginNotify, 10User-notice: BotPasswords + Toolforge combination causing daily "login from new device" warning emails - https://phabricator.wikimedia.org/T182867 (10Novem_Linguae) [20:14:50] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [20:54:50] (PuppetAgentDisabled) firing: Puppet agent disabled on instance quarry-web-02 in project quarry - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentDisabled [21:58:15] 10Toolforge, 10MediaWiki-Email, 10MediaWiki-User-login-and-signup, 10MediaWiki-extensions-LoginNotify, 10User-notice: BotPasswords + Toolforge combination causing daily "login from new device" warning emails - https://phabricator.wikimedia.org/T182867 (10Tgr) Once {T345052} gets deployed and the performa... [22:37:39] 10Toolforge, 10MediaWiki-Email, 10MediaWiki-User-login-and-signup, 10MediaWiki-extensions-LoginNotify, 10User-notice: BotPasswords + Toolforge combination causing daily "login from new device" warning emails - https://phabricator.wikimedia.org/T182867 (10Wbm1058) I got a boatload of these "login from a d... [23:19:50] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [23:54:50] (PuppetAgentDisabled) firing: Puppet agent disabled on instance quarry-web-02 in project quarry - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentDisabled