[00:01:48] (CodesearchConfigWriteFailed) firing: codesearch-write-config.service failed on codesearch8 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchConfigWriteFailed [00:02:18] (CodesearchBackendDown) firing: (2) Codesearch backend design is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchBackendDown [00:10:03] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:11:17] (03PS1) 10Krinkle: App: Add formatSafeTrace() to error pages for unexpected errors [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964097 [00:11:38] (03CR) 10Krinkle: [C: 03+2] App: Add formatSafeTrace() to error pages for unexpected errors [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964097 (owner: 10Krinkle) [00:11:40] (03CR) 10CI reject: [V: 04-1] App: Add formatSafeTrace() to error pages for unexpected errors [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964097 (owner: 10Krinkle) [00:12:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [00:12:04] (03CR) 10CI reject: [V: 04-1] App: Add formatSafeTrace() to error pages for unexpected errors [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964097 (owner: 10Krinkle) [00:12:52] (03PS2) 10Krinkle: App: Add formatSafeTrace() to error pages for unexpected errors [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964097 [00:13:04] (03CR) 10Krinkle: [C: 03+2] App: Add formatSafeTrace() to error pages for unexpected errors [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964097 (owner: 10Krinkle) [00:13:14] (03CR) 10CI reject: [V: 04-1] App: Add formatSafeTrace() to error pages for unexpected errors [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964097 (owner: 10Krinkle) [00:17:20] (03PS3) 10Krinkle: App: Add formatSafeTrace() to error pages for unexpected errors [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964097 [00:17:29] (03CR) 10Krinkle: [C: 03+2] App: Add formatSafeTrace() to error pages for unexpected errors [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964097 (owner: 10Krinkle) [00:18:04] (03Merged) 10jenkins-bot: App: Add formatSafeTrace() to error pages for unexpected errors [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964097 (owner: 10Krinkle) [00:20:03] (InstanceDown) resolved: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:28:21] (03PS1) 10Krinkle: Main: fix php8.2 fatal "number of bound variables does not match" [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964098 [00:30:53] (03PS2) 10Krinkle: Main: fix php8.2 fatal "number of bound variables does not match" [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964098 [00:36:37] 10Quarry: Add maintainers to quarry - https://phabricator.wikimedia.org/T348184 (10Audiodude) Shouldn't REPLICA_DOMAIN be set to `analytics.db.svc.wikimedia.cloud` for this to work? I haven't tried it myself yet. Then you would get `enwiki.analytics.db.svc.wikimedia.cloud` which would be correct right? [00:43:50] (03PS3) 10Krinkle: Main: fix php8.2 fatal "number of bound variables does not match" [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964098 [00:45:12] 10Quarry: Allow Quarry to use arbitrary hostnames for the replica DB - https://phabricator.wikimedia.org/T348364 (10Audiodude) [00:45:46] 10Quarry: Add maintainers to quarry - https://phabricator.wikimedia.org/T348184 (10Audiodude) Forked discussion to T348364 [00:47:39] (03CR) 10Krinkle: [C: 03+2] Main: fix php8.2 fatal "number of bound variables does not match" [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964098 (owner: 10Krinkle) [00:48:14] (03Merged) 10jenkins-bot: Main: fix php8.2 fatal "number of bound variables does not match" [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964098 (owner: 10Krinkle) [00:49:13] (03PS1) 10Krinkle: Main: Fix query for user "0" [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964100 [00:49:15] (03PS1) 10Krinkle: build: Upgrade service from php7.4 to php8.2 [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964101 [00:49:33] (03PS2) 10Krinkle: Main: Fix query for user "0" [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964100 [00:49:36] (03CR) 10Krinkle: [C: 03+2] Main: Fix query for user "0" [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964100 (owner: 10Krinkle) [00:49:40] (03PS2) 10Krinkle: build: Upgrade service from php7.4 to php8.2 [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964101 [00:49:42] (03CR) 10Krinkle: [C: 03+2] build: Upgrade service from php7.4 to php8.2 [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964101 (owner: 10Krinkle) [00:50:14] (03Merged) 10jenkins-bot: Main: Fix query for user "0" [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964100 (owner: 10Krinkle) [00:50:27] (03Merged) 10jenkins-bot: build: Upgrade service from php7.4 to php8.2 [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964101 (owner: 10Krinkle) [00:52:33] 10Quarry: Allow Quarry to use arbitrary hostnames for the replica DB - https://phabricator.wikimedia.org/T348364 (10Audiodude) [00:53:58] (03PS1) 10Krinkle: Main: Actually fix query for user "0" [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964102 [00:54:14] (03CR) 10Krinkle: [C: 03+2] Main: Actually fix query for user "0" [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964102 (owner: 10Krinkle) [00:54:49] (03Merged) 10jenkins-bot: Main: Actually fix query for user "0" [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/964102 (owner: 10Krinkle) [01:16:13] 10Quarry: Allow Quarry to use arbitrary hostnames for the replica DB - https://phabricator.wikimedia.org/T348364 (10Audiodude) [02:17:11] 10Quarry: Allow Quarry to use arbitrary hostnames for the replica DB - https://phabricator.wikimedia.org/T348364 (10Audiodude) I thought I was going crazy because `REPLICA_HOST` does in fact exist in `default_config.yaml`, but it turns out it isn't used anywhere in the repo so it must be a vestige of an old way... [02:21:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [02:43:04] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 17 deleted instances on integration-puppetmaster-02 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [03:01:48] (CodesearchConfigWriteFailed) firing: codesearch-write-config.service failed on codesearch8 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchConfigWriteFailed [03:02:18] (CodesearchBackendDown) firing: (2) Codesearch backend design is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchBackendDown [03:12:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [04:28:37] (CephClusterInWarning) firing: The ceph cluster in is in warning status, that means that it's high availability is compromised, things should still be working as expected. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWar [04:33:37] (CephClusterInWarning) resolved: The ceph cluster in is in warning status, that means that it's high availability is compromised, things should still be working as expected. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInW [04:38:37] (CephClusterInWarning) firing: The ceph cluster in is in warning status, that means that it's high availability is compromised, things should still be working as expected. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWar [05:21:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [05:43:04] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 17 deleted instances on integration-puppetmaster-02 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [06:01:48] (CodesearchConfigWriteFailed) firing: codesearch-write-config.service failed on codesearch8 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchConfigWriteFailed [06:02:18] (CodesearchBackendDown) firing: (2) Codesearch backend design is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchBackendDown [06:12:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [06:16:24] 10Tool-bub2: Write unit test cases - https://phabricator.wikimedia.org/T344117 (10Akanksha.t05) 05Open→03In progress [06:44:21] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [07:19:21] (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [07:38:32] (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [08:09:21] (OpenstackAPIResponse) resolved: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [08:14:21] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [08:19:21] (OpenstackAPIResponse) resolved: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [08:21:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [08:23:32] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [08:38:52] (CephClusterInWarning) firing: The ceph cluster in is in warning status, that means that it's high availability is compromised, things should still be working as expected. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWar [08:43:04] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 17 deleted instances on integration-puppetmaster-02 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [09:01:48] (CodesearchConfigWriteFailed) firing: codesearch-write-config.service failed on codesearch8 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchConfigWriteFailed [09:02:18] (CodesearchBackendDown) firing: (2) Codesearch backend design is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchBackendDown [09:12:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [09:18:37] (CephClusterInWarning) resolved: The ceph cluster in is in warning status, that means that it's high availability is compromised, things should still be working as expected. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInW [09:27:54] 10Tool-inteGraality: Add integraality column for sitelinks - https://phabricator.wikimedia.org/T312726 (10VIGNERON) Not exactly what you expect but entities about Wikimedia pages also have other triples like : - schema:inLanguage (that gives the languages code) - wikibase:wikiGroup (that gives the names of s... [09:52:14] 10Quarry: Quarry suggests invalid database names, and doesn't suggest some valid database names - https://phabricator.wikimedia.org/T289943 (10SD0001) All of the invalid database names can be removed from the suggestions if we limit them to only include db names against which successful queries have been run in... [10:02:09] siddharthvp opened https://github.com/toolforge/quarry/pull/24 [10:02:11] 10Quarry: Quarry suggests invalid database names, and doesn't suggest some valid database names - https://phabricator.wikimedia.org/T289943 (10github-toolforge-bot) siddharthvp opened https://github.com/toolforge/quarry/pull/24 [10:10:41] 10PAWS: New upstream release 8.4.0 for Pywikibot - https://phabricator.wikimedia.org/T348372 (10Xqt) [10:21:21] 10Quarry, 10cloud-services-team: Support queries against Quarry's own database and ToolsDB - https://phabricator.wikimedia.org/T151158 (10SD0001) I think letting users query public toolsdb databases has clear value (but not sure of the utility of querying quarry's internal db). As the database selector field... [11:09:36] 10Tool-bub2, 10Outreach-Programs-Projects, 10Outreachy (Round 27): Use API:EmailUser to send Emails to the users - https://phabricator.wikimedia.org/T338267 (10Okerekechinweotito) I have opened a PR to fix this issue. Opened here - [[ https://github.com/coderwassananmol/BUB2/pull/189 | Use API:EmailUser to... [11:21:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [11:43:04] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 17 deleted instances on integration-puppetmaster-02 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [11:54:06] 10Tool-bub2, 10Outreach-Programs-Projects, 10Outreachy (Round 27): Use SMTP to send Emails to the users - https://phabricator.wikimedia.org/T338267 (10wassan.anmol117) [12:01:48] (CodesearchConfigWriteFailed) firing: codesearch-write-config.service failed on codesearch8 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchConfigWriteFailed [12:02:18] (CodesearchBackendDown) firing: (2) Codesearch backend design is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchBackendDown [12:12:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [12:13:32] (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [14:21:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [14:43:04] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 17 deleted instances on integration-puppetmaster-02 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [15:01:34] 10Quarry: Setup an easy way to have Quarry dump information / results on a wiki page - https://phabricator.wikimedia.org/T137179 (10SD0001) Since last year, enwiki has a bot for this: https://en.wikipedia.org/wiki/Template:Database_report (it doesn't interact with Quarry - the bot runs the query directly on tool... [15:01:48] (CodesearchConfigWriteFailed) firing: codesearch-write-config.service failed on codesearch8 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchConfigWriteFailed [15:02:18] (CodesearchBackendDown) firing: (2) Codesearch backend design is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchBackendDown [15:12:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [16:13:32] (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [17:21:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [17:42:10] audiodude opened https://github.com/toolforge/quarry/pull/25 [17:43:04] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 17 deleted instances on integration-puppetmaster-02 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [17:45:06] 10Quarry: Add maintainers to quarry - https://phabricator.wikimedia.org/T348184 (10SD0001) @rook Are there any docs on how to do deployments once a GitHub PR gets merged? [18:01:30] 10Quarry: Add maintainers to quarry - https://phabricator.wikimedia.org/T348184 (10Audiodude) @SD0001 I found: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Quarry, but I don't think I fully understand it [18:01:48] (CodesearchConfigWriteFailed) firing: codesearch-write-config.service failed on codesearch8 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchConfigWriteFailed [18:02:18] (CodesearchBackendDown) firing: (2) Codesearch backend design is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchBackendDown [18:12:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [18:27:29] 10Quarry: Allow Quarry to use arbitrary hostnames for the replica DB - https://phabricator.wikimedia.org/T348364 (10Audiodude) https://github.com/toolforge/quarry/pull/25 [18:51:28] 10cloud-services-team (FY2023/2024-Q1), 10Goal, 10User-aborrero: cloudlb: review swift/radosgw status - https://phabricator.wikimedia.org/T338937 (10Andrew) [18:52:24] 10cloud-services-team, 10Patch-For-Review, 10User-aborrero: Open swift port (28080) to the public internet - https://phabricator.wikimedia.org/T341380 (10Andrew) 05Open→03Resolved >>! In T341380#9225985, @cmooney wrote: > Looking on one of the cloudlb hosts in codfw it doesn't look like port 443 is open... [20:13:32] (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [20:21:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [20:43:04] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 17 deleted instances on integration-puppetmaster-02 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [21:01:48] (CodesearchConfigWriteFailed) firing: codesearch-write-config.service failed on codesearch8 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchConfigWriteFailed [21:02:18] (CodesearchBackendDown) firing: (2) Codesearch backend design is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DCodesearchBackendDown [21:12:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [22:08:05] audiodude closed https://github.com/toolforge/quarry/pull/25 [22:09:24] 10Quarry: Allow Quarry to use arbitrary hostnames for the replica DB - https://phabricator.wikimedia.org/T348364 (10Audiodude) 05Open→03Resolved a:03Audiodude [23:21:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [23:43:04] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 17 deleted instances on integration-puppetmaster-02 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates