[00:01:48] (CodesearchConfigWriteFailed) firing: codesearch-write-config.service failed on codesearch8 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchConfigWriteFailed [00:02:18] (CodesearchBackendDown) firing: (2) Codesearch backend design is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchBackendDown [00:08:03] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:18:03] (InstanceDown) resolved: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [01:54:55] 10Toolforge: Access denied for user 's55564'@'IP.ADRESS' (using password: YES) - https://phabricator.wikimedia.org/T348293 (10Ederporto) [02:21:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [02:43:04] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 17 deleted instances on integration-puppetmaster-02 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [03:01:48] (CodesearchConfigWriteFailed) firing: codesearch-write-config.service failed on codesearch8 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchConfigWriteFailed [03:02:18] (CodesearchBackendDown) firing: (2) Codesearch backend design is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchBackendDown [03:18:20] 10Tool-bub2, 10Internet-Archive, 10Outreach-Programs-Projects, 10Outreachy (Round 27): Allow multi-lingual books to be uploaded to Internet Archive - https://phabricator.wikimedia.org/T346388 (10Okerekechinweotito) I have opened a PR to fix this issue. Opened here - [[ https://github.com/coderwassananmol/B... [04:23:28] 10Toolforge: regenerate sql credentials for tool wosretbot - https://phabricator.wikimedia.org/T348259 (10Ederporto) @Janui I had a similar problem. I just opened a ticket to it: T348293 Access denied for user 's55564'@'IP.ADRESS' (using password: YES) [05:21:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [05:43:04] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 17 deleted instances on integration-puppetmaster-02 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [05:51:22] 10Toolforge: Access denied for user 's55564'@'IP.ADRESS' (using password: YES) - https://phabricator.wikimedia.org/T348293 (10JJMC89) [05:51:24] 10Toolforge: regenerate sql credentials for tool wosretbot - https://phabricator.wikimedia.org/T348259 (10JJMC89) [05:53:56] 10Toolforge: regenerate sql credentials for tool wosretbot - https://phabricator.wikimedia.org/T348259 (10JJMC89) [05:56:46] 10Toolforge: Bad credentials for tools - https://phabricator.wikimedia.org/T348259 (10JJMC89) [06:01:48] (CodesearchConfigWriteFailed) firing: codesearch-write-config.service failed on codesearch8 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchConfigWriteFailed [06:02:18] (CodesearchBackendDown) firing: (2) Codesearch backend design is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchBackendDown [06:13:31] PROBLEM - Disk space on cloudbackup2001 is CRITICAL: DISK CRITICAL - free space: /srv/cinder-backups 3144289 MB (3% inode=98%): https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=cloudbackup2001&var-datasource=codfw+prometheus/ops [07:11:27] 10Data-Services, 10cloud-services-team: Bad credentials for tools - https://phabricator.wikimedia.org/T348259 (10taavi) a:03taavi [07:20:00] 10Tool-bub2, 10Internet-Archive, 10Outreach-Programs-Projects, 10Outreachy (Round 27): For PDL, download and stream the PDF if available - https://phabricator.wikimedia.org/T348188 (10Shreyashidabral) Github PR [[ https://github.com/coderwassananmol/BUB2/pull/186 | here ]] [07:28:20] 10Data-Services, 10cloud-services-team: Bad credentials for tools - https://phabricator.wikimedia.org/T348259 (10taavi) 05Open→03Resolved [07:34:08] (03PS1) 10Majavah: cli: Use new more structured URL routes [cloud/toolforge/jobs-framework-cli] - 10https://gerrit.wikimedia.org/r/963941 [07:34:10] (03PS1) 10Majavah: cli: run: Use JSON in POST body [cloud/toolforge/jobs-framework-cli] - 10https://gerrit.wikimedia.org/r/963942 [08:14:24] 10Data-Services, 10cloud-services-team: Bad credentials for tools - https://phabricator.wikimedia.org/T348259 (10dcaro) @taavi what was the issue? [08:16:25] 10Tool-bub2, 10Internet-Archive, 10Outreach-Programs-Projects, 10Outreachy (Round 27): Add max character limit while creating identifier in Internet Archive and remove some special characters - https://phabricator.wikimedia.org/T348192 (10Okerekechinweotito) @wassan.anmol117 I have opened a PR that fixes... [08:21:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [08:22:12] 10Data-Services, 10cloud-services-team: Bad credentials for tools - https://phabricator.wikimedia.org/T348259 (10taavi) The password in the replica.my.cnf file was wrong, and the modification date was far enough in the past so we no longer have maintain-dbusers logs from that time. I re-generated the config fi... [08:22:59] 10cloud-services-team: HAProxyServiceUnavailable cloudlb1002:9900 HAProxy service radosgw-api_backend has no available backends on cloudlb1002:9900 - https://phabricator.wikimedia.org/T347921 (10taavi) 05Open→03Resolved a:03taavi [08:23:02] 10cloud-services-team: HAProxyServiceUnavailable cloudlb1001:9900 HAProxy service radosgw-api_backend has no available backends on cloudlb1001:9900 - https://phabricator.wikimedia.org/T347920 (10taavi) 05Open→03Resolved a:03taavi [08:31:06] 10Data-Services, 10cloud-services-team: Bad credentials for tools - https://phabricator.wikimedia.org/T348259 (10Janui) Thank you very much :) [08:43:04] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 17 deleted instances on integration-puppetmaster-02 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [08:49:38] 10cloud-services-team (FY2023/2024-Q1), 10Cloud-Services-Origin-User, 10Cloud-Services-Worktype-Maintenance, 10User-dcaro: [webservice] Error shown when restarting buildpack-based tool - https://phabricator.wikimedia.org/T348312 (10dcaro) p:05Triage→03Medium [08:49:50] 10Toolforge (Toolforge iteration 00), 10cloud-services-team (FY2023/2024-Q1), 10Cloud-Services-Origin-User, 10Cloud-Services-Worktype-Maintenance, 10User-dcaro: [webservice] Error shown when restarting buildpack-based tool - https://phabricator.wikimedia.org/T348312 (10dcaro) [09:01:48] (CodesearchConfigWriteFailed) firing: codesearch-write-config.service failed on codesearch8 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchConfigWriteFailed [09:02:18] (CodesearchBackendDown) firing: (2) Codesearch backend design is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchBackendDown [11:21:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [11:43:04] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 17 deleted instances on integration-puppetmaster-02 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [12:01:48] (CodesearchConfigWriteFailed) firing: codesearch-write-config.service failed on codesearch8 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchConfigWriteFailed [12:02:18] (CodesearchBackendDown) firing: (2) Codesearch backend design is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchBackendDown [12:33:19] 10Toolforge (Toolforge iteration 00), 10Toolforge Jobs framework, 10Patch-For-Review: jobs: Add option to disable NFS mounts - https://phabricator.wikimedia.org/T348250 (10CodeReviewBot) taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/39 kubernetes: Add MountO... [12:50:03] (InstanceDown) firing: Project toolsbeta instance toolsbeta-puppetdb-02 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [12:55:03] (InstanceDown) resolved: Project toolsbeta instance toolsbeta-puppetdb-02 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:29:14] (03PS1) 10Majavah: Add --mount option [cloud/toolforge/jobs-framework-cli] - 10https://gerrit.wikimedia.org/r/964009 (https://phabricator.wikimedia.org/T348250) [13:51:20] (ProbeDown) firing: Service toolserver-proxy-01:443 has failed probes (http_toolserver_org_redirects_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolserver-proxy-01:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:56:20] (ProbeDown) resolved: Service toolserver-proxy-01:443 has failed probes (http_toolserver_org_redirects_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolserver-proxy-01:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:21:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [14:36:57] 10Toolforge (Toolforge iteration 00), 10cloud-services-team (FY2023/2024-Q1), 10Cloud-Services-Origin-Team, 10Cloud-Services-Worktype-Project, and 3 others: [builds-api.start] Add statistics - https://phabricator.wikimedia.org/T337390 (10dcaro) >>! In T337390#9211280, @Raymond_Ndibe wrote: > @dcaro I'm hav... [14:39:40] 10Toolforge (Toolforge iteration 00), 10cloud-services-team (FY2023/2024-Q1), 10Cloud-Services-Origin-Team, 10Cloud-Services-Worktype-Project, and 3 others: [builds-api.start] Add statistics - https://phabricator.wikimedia.org/T337390 (10Raymond_Ndibe) oo thanks so much @dcaro this looks promising [14:43:04] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 17 deleted instances on integration-puppetmaster-02 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [14:59:37] 10cloud-services-team (Hardware), 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cloudvirt10[62-67] - https://phabricator.wikimedia.org/T342537 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host cloudvirt1063.eqiad.wmnet with OS bullseye [14:59:40] 10cloud-services-team (Hardware), 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cloudvirt10[62-67] - https://phabricator.wikimedia.org/T342537 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host cloudvirt1064.eqiad.wmnet with OS bullseye [14:59:44] 10cloud-services-team (Hardware), 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cloudvirt10[62-67] - https://phabricator.wikimedia.org/T342537 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host cloudvirt1067.eqiad.wmnet with OS bullseye [15:01:48] (CodesearchConfigWriteFailed) firing: codesearch-write-config.service failed on codesearch8 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchConfigWriteFailed [15:02:18] (CodesearchBackendDown) firing: (2) Codesearch backend design is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchBackendDown [15:19:25] (03CR) 10Arturo Borrero Gonzalez: [C: 03+1] cli: run: Use JSON in POST body [cloud/toolforge/jobs-framework-cli] - 10https://gerrit.wikimedia.org/r/963942 (owner: 10Majavah) [15:20:12] (03CR) 10Arturo Borrero Gonzalez: [C: 03+1] cli: Use new more structured URL routes [cloud/toolforge/jobs-framework-cli] - 10https://gerrit.wikimedia.org/r/963941 (owner: 10Majavah) [15:38:14] 10cloud-services-team, 10Data-Platform-SRE, 10Dumps-Generation, 10Patch-For-Review: clouddumps100[12] puppet alert: "Puppet performing a change on every puppet run" - https://phabricator.wikimedia.org/T346165 (10BTullis) Looks good to me too. Thanks again @jbond. I'll check back again next week, once it's... [15:38:56] (03CR) 10Majavah: [C: 03+2] cli: Use new more structured URL routes [cloud/toolforge/jobs-framework-cli] - 10https://gerrit.wikimedia.org/r/963941 (owner: 10Majavah) [15:39:13] (03CR) 10Majavah: [C: 03+2] cli: run: Use JSON in POST body [cloud/toolforge/jobs-framework-cli] - 10https://gerrit.wikimedia.org/r/963942 (owner: 10Majavah) [15:39:54] (03Merged) 10jenkins-bot: cli: Use new more structured URL routes [cloud/toolforge/jobs-framework-cli] - 10https://gerrit.wikimedia.org/r/963941 (owner: 10Majavah) [15:40:10] (03Merged) 10jenkins-bot: cli: run: Use JSON in POST body [cloud/toolforge/jobs-framework-cli] - 10https://gerrit.wikimedia.org/r/963942 (owner: 10Majavah) [15:57:19] 10Toolforge (Toolforge iteration 00), 10Toolforge Jobs framework, 10Patch-For-Review: jobs: Add option to disable NFS mounts - https://phabricator.wikimedia.org/T348250 (10CodeReviewBot) taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/39 kubernetes: Add MountO... [16:22:35] (HarborDown) firing: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown [16:32:30] 10Toolforge Build Service, 10cloud-services-team: toolsbeta harbor instance ran out of disk - https://phabricator.wikimedia.org/T348337 (10taavi) p:05Triage→03High [16:32:35] (HarborDown) resolved: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown [16:38:34] 10Toolforge Build Service, 10cloud-services-team: toolsbeta harbor instance ran out of disk - https://phabricator.wikimedia.org/T348337 (10taavi) I manually triggered Harbor garbage collection which freed up about two gigs of disk. That should last until Monday. [16:38:56] 10Toolforge (Toolforge iteration 00), 10Toolforge Build Service, 10cloud-services-team: toolsbeta harbor instance ran out of disk - https://phabricator.wikimedia.org/T348337 (10taavi) [17:20:21] 10Cloud-VPS, 10cloud-services-team (FY2023/2024-Q1), 10Goal, 10Patch-For-Review: Upgrade cloud-vps openstack to version 'Antelope' - https://phabricator.wikimedia.org/T341285 (10Andrew) So far all the cloudcontrol issues have been related to obsolete init scripts. I updated many (all?) of our init scripts... [17:21:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [17:31:25] 10Tools, 10translatewiki.net, 10Language-Team (Language-2023-July-September), 10Localization Infrastructure FY23-24, 10Unplanned-Sprint-Work: Wikidata Lexeme Forms uses inappropriate language names for translations - https://phabricator.wikimedia.org/T336606 (10LucasWerkmeister) Seems to be working, than... [17:35:59] 10cloud-services-team (FY2023/2024-Q1), 10wikitech.wikimedia.org: [wikitech] administrator rights for WMCS - https://phabricator.wikimedia.org/T347557 (10nskaggs) We need a bureaucrat to grant permissions: https://wikitech.wikimedia.org/wiki/Special:ListUsers/bureaucrat [17:43:04] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 17 deleted instances on integration-puppetmaster-02 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [18:01:48] (CodesearchConfigWriteFailed) firing: codesearch-write-config.service failed on codesearch8 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchConfigWriteFailed [18:02:18] (CodesearchBackendDown) firing: (2) Codesearch backend design is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchBackendDown [19:11:48] (03PS1) 10Majavah: pm-maintain-projects: fix dict loop [cloud/metricsinfra/prometheus-manager] - 10https://gerrit.wikimedia.org/r/964066 [19:12:00] (03CR) 10Majavah: [C: 03+2] pm-maintain-projects: fix dict loop [cloud/metricsinfra/prometheus-manager] - 10https://gerrit.wikimedia.org/r/964066 (owner: 10Majavah) [19:14:56] (03Merged) 10jenkins-bot: pm-maintain-projects: fix dict loop [cloud/metricsinfra/prometheus-manager] - 10https://gerrit.wikimedia.org/r/964066 (owner: 10Majavah) [19:35:00] 10Quarry: Add maintainers to quarry - https://phabricator.wikimedia.org/T348184 (10SD0001) @rook Is there any staging/beta/QA environment as well for quarry? [19:38:16] 10Quarry: Add maintainers to quarry - https://phabricator.wikimedia.org/T348184 (10rook) >>! In T348184#9231920, @SD0001 wrote: > @rook Is there any staging/beta/QA environment as well for quarry? A local dev environment can be setup as described in the README under "Setting up a local dev environment" As for... [19:48:04] 10Quarry: Add maintainers to quarry - https://phabricator.wikimedia.org/T348184 (10rook) >>! In T348184#9231920, @SD0001 wrote: > ~~@rook Is there any staging/beta/QA environment as well for quarry?~~ Never mind, got it: https://quarry-dev.wmflabs.org/ Oh yeah, quarry-dev. It's not the best representation of qu... [19:56:40] 10Quarry: Add maintainers to quarry - https://phabricator.wikimedia.org/T348184 (10Audiodude) FWIW I set up the dev environment without any issue and was able to run queries against mywiki. @rook is it possible to query replicas on toolforge from the dev environment if I use an SSH tunnel and change my config? [19:58:10] 10Quarry: Add maintainers to quarry - https://phabricator.wikimedia.org/T348184 (10rook) >>! In T348184#9232082, @Audiodude wrote: > FWIW I set up the dev environment without any issue and was able to run queries against mywiki. > > @rook is it possible to query replicas on toolforge from the dev environment if... [20:21:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [20:43:04] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 17 deleted instances on integration-puppetmaster-02 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [20:56:49] 10cloud-services-team: radosgw+keystone seem to choke on projects with '-' in their name - https://phabricator.wikimedia.org/T341509 (10Andrew) [20:56:56] 10Cloud-VPS, 10Data-Services, 10cloud-services-team, 10Patch-For-Review, 10User-Marostegui: Better swift error messages for projects with invalid chars - https://phabricator.wikimedia.org/T347927 (10Andrew) [20:57:00] 10cloud-services-team: radosgw+keystone seem to choke on projects with '-' in their name - https://phabricator.wikimedia.org/T341509 (10Andrew) [20:57:03] 10Cloud-VPS, 10Data-Services, 10cloud-services-team, 10User-Marostegui: Support Openstack Swift APIs via the radosgw - https://phabricator.wikimedia.org/T276961 (10Andrew) [20:57:18] 10Cloud-VPS, 10Data-Services, 10cloud-services-team, 10Patch-For-Review, 10User-Marostegui: Better swift error messages for projects with invalid chars - https://phabricator.wikimedia.org/T347927 (10Andrew) 05Open→03Resolved a:03Andrew [21:01:48] (CodesearchConfigWriteFailed) firing: codesearch-write-config.service failed on codesearch8 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchConfigWriteFailed [21:02:18] (CodesearchBackendDown) firing: (2) Codesearch backend design is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchBackendDown [21:18:47] 10Quarry: Add maintainers to quarry - https://phabricator.wikimedia.org/T348184 (10SD0001) The networking limitation is overcome with an ssh tunnel. However, the code in replica.py to create the replica host URL is rather weird: ` repl_host = ( f"{self.database_name}.{self.config['REPLICA_DOM... [23:21:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [23:26:20] (03PS1) 10Krinkle: build: Upgrade dev packages, require PHP 7.4+ [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964095 [23:26:44] (03CR) 10CI reject: [V: 04-1] build: Upgrade dev packages, require PHP 7.4+ [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964095 (owner: 10Krinkle) [23:35:19] (03PS2) 10Krinkle: build: Upgrade dev packages, require PHP 7.4+ [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964095 [23:36:58] (03CR) 10Krinkle: [C: 03+2] build: Upgrade dev packages, require PHP 7.4+ [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964095 (owner: 10Krinkle) [23:37:36] (03Merged) 10jenkins-bot: build: Upgrade dev packages, require PHP 7.4+ [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964095 (owner: 10Krinkle) [23:38:34] (03CR) 10Krinkle: [C: 04-1] "> Patch Set 1:" [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/674848 (https://phabricator.wikimedia.org/T227631) (owner: 10Sahilgrewalhere) [23:43:04] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 17 deleted instances on integration-puppetmaster-02 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [23:51:58] 10Tool-Global-user-contributions, 10IP Masking, 10Stewards-and-global-tools, 10XTools, and 2 others: [Design research] Understand usage of current GUC tool - https://phabricator.wikimedia.org/T347618 (10KColeman-WMF)