[00:08:28] (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance tf-infra-test in project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [00:13:28] (PuppetAgentStaleLastRun) resolved: Last Puppet run was over 24 hours ago on instance tf-infra-test in project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [00:20:50] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [00:34:05] 10Wikibugs: Ignore "changed the status of subtask" Phabricator transactions - https://phabricator.wikimedia.org/T357875#9656114 (10bd808) 05Open→03In progress a:03bd808 [01:49:31] 10Wikibugs: Wikibugs should ignore changes to the security field - https://phabricator.wikimedia.org/T105625#9656146 (10bd808) 05Open→03In progress a:03bd808 [02:00:31] 10Wikibugs, 13Patch-For-Review: Wikibugs should ignore changes to the security field - https://phabricator.wikimedia.org/T105625#9656150 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/21 Ignore more phorge transaction types [02:01:09] 10Wikibugs, 13Patch-For-Review: Ignore "changed the status of subtask" Phabricator transactions - https://phabricator.wikimedia.org/T357875#9656152 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/21 Ignore more phorge transaction types [02:38:28] (PuppetStaleCertificates) firing: (2) Found non-revoked Puppet certificates for 1 deleted instances on project-proxy-puppetmaster-01 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [03:20:50] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [03:41:09] 10Wikibugs, 07Software-Licensing: Relicense Wikibugs from MIT to GPL-3.0-or-later after approval by all substantive contributors - https://phabricator.wikimedia.org/T360718#9656178 (10Legoktm) +1 from me. [03:42:38] 10Wikibugs, 07Software-Licensing: Relicense Wikibugs from MIT to GPL-3.0-or-later after approval by all substantive contributors - https://phabricator.wikimedia.org/T360718#9656179 (10bd808) Commits by user to files other than `*.yaml` or `test_*`: `lang=shell-session $ git shortlog --no-merges -sn -- ':!*.yam... [04:02:32] 10Wikibugs, 07Software-Licensing: Relicense Wikibugs from MIT to GPL-3.0-or-later after approval by all substantive contributors - https://phabricator.wikimedia.org/T360718#9656181 (10bd808) >>! In T360718#9656179, @bd808 wrote: > More detail on folks with 4 or fewer matching commits: Upon inspection, these c... [04:07:25] 10Wikibugs, 07Software-Licensing: Relicense Wikibugs from MIT to GPL-3.0-or-later after approval by all substantive contributors - https://phabricator.wikimedia.org/T360718#9656183 (10bd808) [04:14:01] 10Wikibugs, 07Software-Licensing: Relicense Wikibugs from MIT to GPL-3.0-or-later after approval by all substantive contributors - https://phabricator.wikimedia.org/T360718#9656184 (10bd808) [05:38:28] (PuppetStaleCertificates) firing: (2) Found non-revoked Puppet certificates for 1 deleted instances on project-proxy-puppetmaster-01 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [06:20:50] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [08:38:28] (PuppetStaleCertificates) firing: (2) Found non-revoked Puppet certificates for 1 deleted instances on project-proxy-puppetmaster-01 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [09:20:50] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [11:38:28] (PuppetStaleCertificates) firing: (2) Found non-revoked Puppet certificates for 1 deleted instances on project-proxy-puppetmaster-01 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [12:20:50] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [14:38:28] (PuppetStaleCertificates) firing: (2) Found non-revoked Puppet certificates for 1 deleted instances on project-proxy-puppetmaster-01 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [15:20:50] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [17:38:28] (PuppetStaleCertificates) firing: (2) Found non-revoked Puppet certificates for 1 deleted instances on project-proxy-puppetmaster-01 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [18:20:50] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [19:01:38] (ProbeDown) firing: Service toolsbeta-test-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolsbeta-test-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:06:38] (ProbeDown) resolved: Service toolsbeta-test-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolsbeta-test-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:23:10] 10Wikibugs: Wikibugs testing task - https://phabricator.wikimedia.org/T90594#9656389 (10bd808) test [19:58:42] 10Wikibugs, 10QuarryBot-enwiki, 13Patch-For-Review: 14wikibugs config for repo messages - 14https://phabricator.wikimedia.org/T158355#9656396 (10bd808) [20:38:28] (PuppetStaleCertificates) firing: (2) Found non-revoked Puppet certificates for 1 deleted instances on project-proxy-puppetmaster-01 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [20:50:24] 10Wikibugs: Convert `wikibugs2 phorge` to asyncio - https://phabricator.wikimedia.org/T359883#9656426 (10CodeReviewBot) bd808 merged https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/20 Convert `wikibugs2 phorge` to asyncio [20:51:55] 10Wikibugs: Wikibugs testing task - https://phabricator.wikimedia.org/T90594#9656427 (10bd808) test [20:54:47] 10Wikibugs: Wikibugs testing task - https://phabricator.wikimedia.org/T90594#9656431 (10bd808) test [20:55:13] (03CR) 10BryanDavis: [C:04-2] "Done" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/1008016 (https://phabricator.wikimedia.org/T90594) (owner: 10BryanDavis) [20:56:45] 10Wikibugs: 14Convert `wikibugs2 phorge` to asyncio - 14https://phabricator.wikimedia.org/T359883#9656432 (10bd808) 05In progress→03Resolved [21:09:28] 10Wikibugs, 13Patch-For-Review: Wikibugs should ignore changes to the security field - https://phabricator.wikimedia.org/T105625#9656433 (10CodeReviewBot) bd808 merged https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/21 Ignore more phorge transaction types [21:09:29] 10Wikibugs: Ignore "changed the status of subtask" Phabricator transactions - https://phabricator.wikimedia.org/T357875#9656434 (10CodeReviewBot) bd808 merged https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/21 Ignore more phorge transaction types [21:12:16] 10Wikibugs, 13Patch-For-Review: 14Wikibugs should ignore changes to the security field - 14https://phabricator.wikimedia.org/T105625#9656441 (10bd808) 05In progress→03Resolved [21:12:29] 10Wikibugs: 14Ignore "changed the status of subtask" Phabricator transactions - 14https://phabricator.wikimedia.org/T357875#9656442 (10bd808) 05In progress→03Resolved [21:20:50] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [21:24:15] 06cloud-services-team, 10Cloud-VPS (Quota-requests): 14Temporarily increase quota for dwl Buster migration - 14https://phabricator.wikimedia.org/T360788#9656446 (10Andrew) 05Open→03Invalid 14Thanks for offering to have your quotas lowered! At the moment you are using 81 CPUs so I'm leaving your quotas... [21:26:52] !log andrew@cloudcumin1001 pm20database START - Cookbook wmcs.vps.create_project for trove-only project pm20database in eqiad1 (T359785) [21:26:53] andrew@cloudcumin1001: Unknown project "pm20database" [21:26:53] T359785: Request increased quota for pm20-* Toolforge tool - https://phabricator.wikimedia.org/T359785 [21:27:29] !log andrew@cloudcumin1001 pm20database END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for trove-only project pm20database in eqiad1 (T359785) [21:27:29] andrew@cloudcumin1001: Unknown project "pm20database" [21:28:45] 10Toolforge (Quota-requests): 14Request increased quota for pm20-* Toolforge tool - 14https://phabricator.wikimedia.org/T359785#9656451 (10Andrew) 05Open→03Resolved a:03Andrew 14@Jneubert, I've created a project with trove quotas for you to try. If Trove doesn't suit you re-open this task and we can g... [21:33:49] 10Wikibugs, 10wikimedia-irc-libera: Autovoice wikibugs in all channels it speaks in - https://phabricator.wikimedia.org/T283983#9656457 (10bd808) [21:48:18] 10Wikibugs, 10wikimedia-irc-libera: Autovoice wikibugs in all channels it speaks in - https://phabricator.wikimedia.org/T283983#9656462 (10bd808) [22:15:28] (WidespreadPuppetAgentFailure) firing: Widespread puppet agent failures in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [22:20:20] 10Wikibugs: Reimagine channel configuration (re)loading to avoid need for git pull - https://phabricator.wikimedia.org/T360860 (10bd808) 03NEW [22:37:28] (PuppetAgentFailure) firing: Puppet agent failure detected on instance mx-out06 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [22:40:28] (WidespreadPuppetAgentFailure) resolved: Widespread puppet agent failures in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [22:52:28] (PuppetAgentFailure) resolved: Puppet agent failure detected on instance mx-out06 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [23:18:56] 10Wikibugs: Reimagine channel configuration (re)loading to avoid need for git pull - https://phabricator.wikimedia.org/T360860#9656484 (10bd808) p:05Triage→03High a:03bd808 This is the last remaining blocker to switching to a #toolforge_build_service container for the tool. We want to preserve the ability... [23:21:01] 10Wikibugs: 14Update wikibugs's Gerrit ssh host keys - 14https://phabricator.wikimedia.org/T257383#9656491 (10bd808) 05Stalled→03Declined 14Unneeded with current ssh solution which ignores host key validation. [23:38:28] (PuppetStaleCertificates) firing: (2) Found non-revoked Puppet certificates for 1 deleted instances on project-proxy-puppetmaster-01 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [23:38:37] 10Tools, 06Diffusion-Repository-Administrators, 10Projects-Cleanup: Archive Gerrit repository "labs/tools/pywikibugs" (20141017) - https://phabricator.wikimedia.org/T351601#9656494 (10bd808) [23:39:18] 10Wikibugs, 06Diffusion-Repository-Administrators, 10Projects-Cleanup: Archive Gerrit repository "labs/tools/pywikibugs" (20141017) - https://phabricator.wikimedia.org/T351601#9656498 (10bd808)