[00:03:26] (03PS1) 10Operator873: Fix log_action [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1013653 (https://phabricator.wikimedia.org/T360790) [00:03:54] (03CR) 10Urbanecm: [C:03+2] Fix log_action [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1013653 (https://phabricator.wikimedia.org/T360790) (owner: 10Operator873) [00:04:29] (03Merged) 10jenkins-bot: Fix log_action [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1013653 (https://phabricator.wikimedia.org/T360790) (owner: 10Operator873) [00:09:28] (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance tf-infra-test in project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [00:14:28] (PuppetAgentStaleLastRun) resolved: Last Puppet run was over 24 hours ago on instance tf-infra-test in project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [00:15:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [02:33:28] (PuppetStaleCertificates) firing: (2) Found non-revoked Puppet certificates for 1 deleted instances on project-proxy-puppetmaster-01 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [02:45:57] (CloudVPSDesignateLeaks) firing: (5) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:15:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [05:18:43] 10Toolforge: I can't connect to Toolforge DB replicas from my PC using MySQL Workbench - https://phabricator.wikimedia.org/T360839 (10MBH) 03NEW [05:33:28] (PuppetStaleCertificates) firing: (2) Found non-revoked Puppet certificates for 1 deleted instances on project-proxy-puppetmaster-01 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [06:15:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [06:45:57] (CloudVPSDesignateLeaks) firing: (5) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:33:28] (PuppetStaleCertificates) firing: (2) Found non-revoked Puppet certificates for 1 deleted instances on project-proxy-puppetmaster-01 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [09:20:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [10:45:57] (CloudVPSDesignateLeaks) firing: (5) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:38:28] (PuppetStaleCertificates) firing: (2) Found non-revoked Puppet certificates for 1 deleted instances on project-proxy-puppetmaster-01 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [12:20:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [13:31:50] (ProbeDown) firing: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:36:50] (ProbeDown) resolved: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:38:28] (PuppetStaleCertificates) firing: (2) Found non-revoked Puppet certificates for 1 deleted instances on project-proxy-puppetmaster-01 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [14:45:57] (CloudVPSDesignateLeaks) firing: (5) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:20:50] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [17:25:42] (CloudVPSDesignateLeaks) firing: (5) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:38:28] (PuppetStaleCertificates) firing: (2) Found non-revoked Puppet certificates for 1 deleted instances on project-proxy-puppetmaster-01 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [17:50:42] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:20:50] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [18:40:25] (03PS1) 10Operator873: Handle expiry better [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1013683 [18:40:52] (03CR) 10CI reject: [V:04-1] Handle expiry better [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1013683 (owner: 10Operator873) [18:45:16] (03PS2) 10Operator873: Handle expiry better [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1013683 [19:50:48] (03PS1) 10Krinkle: frontend: Implement `action=repos`, add "Complete repository list" link [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1013685 (https://phabricator.wikimedia.org/T346074) [19:50:50] (03PS1) 10Krinkle: build: Fix phpcs/phan warnings [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1013686 [20:07:08] (03PS1) 10Ketulucas: Bug:T225798 Swipe left/right to change Participate image [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1013688 [20:38:28] (PuppetStaleCertificates) firing: (2) Found non-revoked Puppet certificates for 1 deleted instances on project-proxy-puppetmaster-01 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [21:06:53] 10Wikibugs: Wikibugs testing task - https://phabricator.wikimedia.org/T90594#9656043 (10bd808) test [21:15:14] (03CR) 10Krinkle: [C:03+2] frontend: Implement `action=repos`, add "Complete repository list" link [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1013685 (https://phabricator.wikimedia.org/T346074) (owner: 10Krinkle) [21:16:04] (03Merged) 10jenkins-bot: frontend: Implement `action=repos`, add "Complete repository list" link [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1013685 (https://phabricator.wikimedia.org/T346074) (owner: 10Krinkle) [21:17:46] (03PS2) 10Krinkle: build: Fix phpcs/phan warnings [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1013686 [21:18:06] (03CR) 10Krinkle: [C:03+2] build: Fix phpcs/phan warnings [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1013686 (owner: 10Krinkle) [21:18:53] (03Merged) 10jenkins-bot: build: Fix phpcs/phan warnings [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1013686 (owner: 10Krinkle) [21:20:50] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [21:24:11] 10VPS-project-Codesearch, 13Patch-For-Review, 07patch-welcome: Allow browsing of full list of indexed repositories - https://phabricator.wikimedia.org/T346074#9656047 (10Krinkle) {F43163513 height=150, layout=inline} {F43163511 height=150, layout=inline} {F43163514 height=200} [21:40:48] 10Wikibugs: Wikibugs testing task - https://phabricator.wikimedia.org/T90594#9656054 (10bd808) test [22:34:11] 10Wikibugs: Wikibugs testing task - https://phabricator.wikimedia.org/T90594#9656070 (10bd808) test [23:38:28] (PuppetStaleCertificates) firing: (2) Found non-revoked Puppet certificates for 1 deleted instances on project-proxy-puppetmaster-01 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [23:40:25] 10Wikibugs, 13Patch-For-Review: Convert `wikibugs2 phorge` to asyncio - https://phabricator.wikimedia.org/T359883#9656084 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/20 Convert `wikibugs2 phorge` to asyncio [23:55:05] 10Wikibugs: Wikibugs testing task - https://phabricator.wikimedia.org/T90594#9656086 (10bd808) test