[00:24:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:25:03] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate etytree away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435770#12446254 (10komla) @Andrew @Epantaleo needs some assistance with their migration. Ester, Andrew is away and should be back later in the week. [00:30:01] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate etytree away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435770#12446256 (10komla) The repo with instructions are found here: https://github.com/esterpantaleo/etymology [02:04:22] FIRING: HAProxyBackendUnavailable: HAProxy service wikireplica-db-analytics-s3 backend clouddb1023.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [02:08:25] FIRING: [3x] SystemdUnitFailed: remove_dangling_cinder_snapshots.service on cloudbackup2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:45:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:50:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:56:31] andrew@cloudcumin1001 safe_reboot (PID 310520) is awaiting input [03:40:48] FIRING: PuppetFailure: Puppet has failed on clouddb1023:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [03:58:17] 10Cloud-VPS, 06tools-infrastructure-team, 07Security: CVE-2026-99999 'pwnescape' - https://phabricator.wikimedia.org/T441555#12446352 (10Andrew) All ceph-based cloudvirts are now running 6.12.113+deb13-amd64 which contains a fix for this exploit. cloudvirt-local100[123] should also be upgraded but it's les... [04:08:52] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph} AND NOT P{F:kernelversion = 6.12.113}' [04:10:11] PROBLEM - MariaDB memory on clouddb1022 is CRITICAL: CRIT Memory 98% used. Largest process: mysqld (93632) = 75.3% https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [04:10:38] 10Cloud-VPS, 06tools-infrastructure-team, 07Security: CVE-2026-99999 'pwnescape' - https://phabricator.wikimedia.org/T441555#12446380 (10Andrew) cc @MoritzMuehlenhoff -- I was nervous about this exploit so did upgrades and reboots over the weekend. Hopefully we're caught up and these hosts don't need anoth... [04:12:34] 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team: clouddb memory alert - https://phabricator.wikimedia.org/T438200#12446392 (10Andrew) clouddb1023 seems to have fired the oomkiller; clouddb1022 is now also alerting about being short on memory. [04:55:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:05:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:08:25] FIRING: [3x] SystemdUnitFailed: check-private-data.service on clouddb1023:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:26:11] RECOVERY - mysqld processes on clouddb1023 is OK: PROCS OK: 2 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [05:28:25] FIRING: [3x] SystemdUnitFailed: check-private-data.service on clouddb1023:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:29:22] FIRING: [2x] HAProxyBackendUnavailable: HAProxy service wikireplica-db-analytics-s3 backend clouddb1023.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [05:33:25] FIRING: [4x] SystemdUnitFailed: check-private-data.service on clouddb1023:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:34:22] RESOLVED: [2x] HAProxyBackendUnavailable: HAProxy service wikireplica-db-analytics-s3 backend clouddb1023.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [05:49:11] RECOVERY - MariaDB memory on clouddb1022 is OK: OK Memory 86% used https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [05:57:46] 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team: clouddb memory alert - https://phabricator.wikimedia.org/T438200#12446533 (10Marostegui) clouddb1023 is down for s3 - this came after a spike in reads a few hours before: ` [5300389.185139] mysqld invoked oom-killer: gfp_mask=0x140cca(GFP_H... [06:56:59] (03update) 10renovatebot: Update pnpm to v12.14.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/33 [06:57:01] (03open) 10renovatebot: Update pnpm to v12.14.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/33 [06:57:13] (03update) 10renovatebot: Update pnpm to v12.14.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/43 [06:57:15] (03open) 10renovatebot: Update pnpm to v12.14.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/43 [06:57:29] (03update) 10renovatebot: Update pnpm to v12.14.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/39 [06:57:35] (03open) 10renovatebot: Update pnpm to v12.14.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/39 [06:58:30] (03update) 10renovatebot: Update pnpm to v12.14.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/71 [06:58:32] (03open) 10renovatebot: Update pnpm to v12.14.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/71 [06:59:27] (03update) 10renovatebot: Update pnpm to v12.14.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/59 [06:59:29] (03open) 10renovatebot: Update pnpm to v12.14.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/59 [07:28:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:53:26] 10Tool-inteGraality: Syntax-check SPARQL queries before sending them to the endpoint - https://phabricator.wikimedia.org/T437894#12446770 (10JeanFred) [07:53:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:55:08] 10Tool-inteGraality: Syntax-check SPARQL queries before sending them to the endpoint - https://phabricator.wikimedia.org/T437894#12446773 (10JeanFred) [08:01:23] 10Tool-inteGraality: Reference column drill-down queries time out on QLever - https://phabricator.wikimedia.org/T435101#12446783 (10JeanFred) [08:01:32] 10Tool-inteGraality, 06Wikidata Platform Team: Syntax-check SPARQL queries before sending them to the endpoint - https://phabricator.wikimedia.org/T437894#12446784 (10gmodena) [08:02:14] 10Tool-inteGraality, 06Wikidata Platform Team: Reference column drill-down queries time out on QLever - https://phabricator.wikimedia.org/T435101#12446801 (10JeanFred) [08:06:58] 10Tool-inteGraality: Retire property_threshold parameter - https://phabricator.wikimedia.org/T440290#12446824 (10JeanFred) 05Open→03Resolved a:03JeanFred [08:26:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:36:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:36:52] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/368 [08:37:07] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/368 [08:37:29] (03merge) 10taavi: eqiad1: cloudinfra: Allow ingress to Hiddenparma service [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/368 (https://phabricator.wikimedia.org/T439418) [08:37:31] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [08:37:52] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [08:38:25] 10Tool-inteGraality: Qualifier columns: allow matching subclasses of the statement value - https://phabricator.wikimedia.org/T441574#12447002 (10JeanFred) Thanks for filing this. This makes sense. For a complete example, here’s the relevant query lifted from the unit-tests, “lighthouses that are French monuments... [09:33:25] FIRING: SystemdUnitFailed: check-private-data.service on clouddb1023:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:58:51] (03merge) 10countcount: Update pnpm to v12.14.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/59 (owner: 10renovatebot) [09:58:57] (03merge) 10countcount: Update pnpm to v12.14.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/71 (owner: 10renovatebot) [09:59:06] (03merge) 10countcount: Update pnpm to v12.14.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/33 (owner: 10renovatebot) [09:59:24] (03merge) 10countcount: Update pnpm to v12.14.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/39 (owner: 10renovatebot) [09:59:27] (03merge) 10countcount: Update pnpm to v12.14.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/43 (owner: 10renovatebot) [10:15:17] FIRING: JobUnavailable: Reduced availability for job pdns in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:15:46] 10Tool-inteGraality: Qualifier columns: allow matching subclasses of the statement value - https://phabricator.wikimedia.org/T441574#12447825 (10JeanFred) The ticket assumes a fairly specific subset of the potential use-cases (albeit the most common for sure): drilling the subclass tree via `P279*`. However, I w... [10:20:17] RESOLVED: [2x] JobUnavailable: Reduced availability for job pdns in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:20:57] 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team: clouddb memory alert - https://phabricator.wikimedia.org/T438200#12447838 (10fnegri) Thanks @Marostegui for looking into this! One thing I noticed is that both on 1022 and 1023 the available memory started decreasing more rapidly this weeke... [10:21:17] FIRING: [2x] JobUnavailable: Reduced availability for job pdns in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:25:32] RESOLVED: [2x] JobUnavailable: Reduced availability for job pdns in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:46:30] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 13Patch-For-Review: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12447956 (10Raine) >>! In T435393#12442942, @So... [11:31:26] (03PS2) 10Majavah: etcd: Update settings for production cloudinfra cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1355624 (https://phabricator.wikimedia.org/T410721) [11:31:26] (03PS5) 10Majavah: etcd: Support managing a SRV DNS record for etcd cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341953 [11:35:44] (03CR) 10CI reject: [V:04-1] etcd: Update settings for production cloudinfra cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1355624 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah) [11:35:56] (03CR) 10CI reject: [V:04-1] etcd: Support managing a SRV DNS record for etcd cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341953 (owner: 10Majavah) [11:37:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:39:14] (03PS3) 10Majavah: etcd: Update settings for production cloudinfra cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1355624 (https://phabricator.wikimedia.org/T410721) [11:39:14] (03PS6) 10Majavah: etcd: Support managing a SRV DNS record for etcd cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341953 (https://phabricator.wikimedia.org/T410721) [11:39:45] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 13Patch-For-Review: Deploy hiddenparma application - https://phabricator.wikimedia.org/T439418#12448143 (10taavi) 05Open→03Resolved [11:47:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:01:35] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1356827 (owner: 10L10n-bot) [12:06:30] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/80 [12:06:31] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/53 [12:06:45] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/45 [12:06:57] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/1356841 (owner: 10L10n-bot) [12:06:59] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/massmailer] - 10https://gerrit.wikimedia.org/r/1356846 (owner: 10L10n-bot) [12:06:59] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/1356845 (owner: 10L10n-bot) [12:06:59] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1356844 (owner: 10L10n-bot) [12:07:00] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1356849 (owner: 10L10n-bot) [12:07:01] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/watch-translations] - 10https://gerrit.wikimedia.org/r/1356848 (owner: 10L10n-bot) [12:07:04] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/xtools] - 10https://gerrit.wikimedia.org/r/1356851 (owner: 10L10n-bot) [12:12:05] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance cloudinfra-internal-puppetserver-1 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [12:13:24] (03merge) 10filippo: kubernetes: move ToolforgeKubernetesNodeNotReady to critical [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/73 (https://phabricator.wikimedia.org/T441285) [12:22:00] (03PS1) 10Majavah: Add fake conftool2git private key [labs/private] - 10https://gerrit.wikimedia.org/r/1356861 [12:22:05] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance cloudinfra-internal-puppetserver-1 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [12:22:49] (03CR) 10Majavah: [V:03+2 C:03+2] Add fake conftool2git private key [labs/private] - 10https://gerrit.wikimedia.org/r/1356861 (owner: 10Majavah) [12:27:50] 10Tool-Pageviews, 06Data-Engineering, 10Pageviews-API: Pageviews data may occasionally come in later than usual - https://phabricator.wikimedia.org/T441554#12448437 (10Hason-LEK-SIN) [12:33:55] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: cloudvirt1071 crash, again - https://phabricator.wikimedia.org/T431374#12448451 (10Jclark-ctr) Someone logged into server on Sunday and rebooted server while it was running Self test. unsure of any results or if anythi... [12:34:22] (03CR) 10Volans: [C:03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1355624 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah) [12:41:17] FIRING: [2x] JobUnavailable: Reduced availability for job prometheus_cardinality_exporter in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [12:45:00] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [12:46:17] RESOLVED: [2x] JobUnavailable: Reduced availability for job prometheus_cardinality_exporter in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [12:47:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:50:41] FIRING: PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=codfw%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [12:52:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:55:41] FIRING: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [12:55:45] (03CR) 10Majavah: [C:03+2] etcd: Update settings for production cloudinfra cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1355624 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah) [12:56:37] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: [hypothesis] ST5.4.1 Extend logging capabilities - https://phabricator.wikimedia.org/T432564#12448555 (10dcaro) Weekly asana report: === Summary === * Hypothesis text * If we make a subset of platform and system-level events available to tool main... [12:58:56] (03Merged) 10jenkins-bot: etcd: Update settings for production cloudinfra cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1355624 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah) [13:05:17] FIRING: JobUnavailable: Reduced availability for job prometheus_cardinality_exporter in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:07:37] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: cloudvirt1071 crash, again - https://phabricator.wikimedia.org/T431374#12448611 (10Andrew) >>! In T431374#12448451, @Jclark-ctr wrote: > Someone that was me! I needed to upgrade kernels over the weekend due to an urg... [13:08:40] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: cloudvirt1071 crash, again - https://phabricator.wikimedia.org/T431374#12448618 (10Jclark-ctr) Opened Sr233121803 i did not see any previous Service request opened. Or any other TSR report ran for this Server [13:10:17] FIRING: [2x] JobUnavailable: Reduced availability for job prometheus_cardinality_exporter in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:10:32] FIRING: [2x] JobUnavailable: Reduced availability for job prometheus_cardinality_exporter in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:15:17] RESOLVED: [2x] JobUnavailable: Reduced availability for job prometheus_cardinality_exporter in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:15:41] FIRING: [3x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [13:15:56] FIRING: [3x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [13:18:28] FIRING: KeyholderUnarmed: 2 unarmed Keyholder key(s) on cloudcumin2001:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [13:20:41] FIRING: [4x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [13:21:16] (03open) 10filippo: kubernetes: make ToolforgeKubernetesNodeNotReady percent-based [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/74 (https://phabricator.wikimedia.org/T441285) [13:23:28] FIRING: [2x] KeyholderUnarmed: 2 unarmed Keyholder key(s) on cloudcumin1001:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [13:27:06] FIRING: InstanceDown: Project toolsbeta instance toolsbeta-nfs-3 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:28:28] RESOLVED: [2x] KeyholderUnarmed: 2 unarmed Keyholder key(s) on cloudcumin1001:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [13:32:06] RESOLVED: InstanceDown: Project toolsbeta instance toolsbeta-nfs-3 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:33:25] FIRING: SystemdUnitFailed: check-private-data.service on clouddb1023:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:35:06] FIRING: InstanceDown: Project toolsbeta instance toolsbeta-nfs-5 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:39:06] (03CR) 10Majavah: etcd: Support managing a SRV DNS record for etcd cluster (033 comments) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341953 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah) [13:40:06] RESOLVED: [2x] InstanceDown: Project toolsbeta instance toolsbeta-nfs-3 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:40:41] FIRING: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [13:40:56] RESOLVED: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [13:47:13] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1356883 (owner: 10L10n-bot) [13:50:21] (03update) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/45 [13:50:23] (03update) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/53 [13:50:25] (03update) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/80 [13:51:20] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/1356891 (owner: 10L10n-bot) [13:51:22] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1356892 (owner: 10L10n-bot) [13:51:24] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/1356894 (owner: 10L10n-bot) [13:51:24] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1356896 (owner: 10L10n-bot) [13:51:27] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/xtools] - 10https://gerrit.wikimedia.org/r/1356898 (owner: 10L10n-bot) [13:51:29] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/massmailer] - 10https://gerrit.wikimedia.org/r/1356895 (owner: 10L10n-bot) [13:51:35] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/watch-translations] - 10https://gerrit.wikimedia.org/r/1356897 (owner: 10L10n-bot) [13:54:37] (03update) 10dcaro: rotationg_file_flag: allow creating a file to flag for cred rotation [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/3 (https://phabricator.wikimedia.org/T440952) [14:17:42] (03update) 10dcaro: rotationg_file_flag: allow creating a file to flag for cred rotation [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/3 (https://phabricator.wikimedia.org/T440952) [14:20:24] 10PAWS, 06tools-platform-team: [maintain-dbusers] Allow deleting PAWS credentials/accounts - https://phabricator.wikimedia.org/T441687 (10dcaro) 03NEW [14:22:28] 10Cloud-Services, 06Product Safety and Integrity, 06Security-Team: Request deletion of Toolforge project: toolforge-defectdojo-secteam - https://phabricator.wikimedia.org/T441688 (10sbassett) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricat... [14:24:04] 10Toolforge, 06tools-platform-team, 06Product Safety and Integrity, 06Security-Team: Request deletion of Toolforge project: toolforge-defectdojo-secteam - https://phabricator.wikimedia.org/T441688#12449257 (10sbassett) [14:27:46] 10Toolforge, 06tools-platform-team, 06Product Safety and Integrity, 06Security-Team: Request deletion of Toolforge project: toolforge-defectdojo-secteam - https://phabricator.wikimedia.org/T441688#12449274 (10taavi) As documented on https://wikitech.wikimedia.org/wiki/Help:Toolforge/Tool_accounts#Delete_a_... [14:30:32] (03approved) 10tlepage: rotationg_file_flag: allow creating a file to flag for cred rotation [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/3 (https://phabricator.wikimedia.org/T440952) (owner: 10dcaro) [14:35:35] 10Toolforge, 06tools-platform-team, 06Product Safety and Integrity, 06Security-Team: Request deletion of Toolforge project: toolforge-defectdojo-secteam - https://phabricator.wikimedia.org/T441688#12449331 (10sbassett) >>! In T441688#12449274, @taavi wrote: > As documented on https://wikitech.wikimedia.org... [14:47:04] (03approved) 10volans: kubernetes: make ToolforgeKubernetesNodeNotReady percent-based [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/74 (https://phabricator.wikimedia.org/T441285) (owner: 10filippo) [14:51:34] (03merge) 10filippo: kubernetes: make ToolforgeKubernetesNodeNotReady percent-based [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/74 (https://phabricator.wikimedia.org/T441285) [14:52:15] 06tools-infrastructure-team, 06tools-platform-team, 13Patch-For-Review: Evaluate ToolforgeKubernetesNodeNotReady page-ability - https://phabricator.wikimedia.org/T441285#12449417 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi Calling this one done! [14:53:31] 10Tool-inteGraality: Qualifier columns: allow matching subclasses of the statement value - https://phabricator.wikimedia.org/T441574#12449428 (10Don-vip) Thanks for the quick and thorough feedback! 1. **Full syntax: yes, let's start with it (option 2).** It covers P279 and other cases with one mechanism. The sh... [14:54:52] 10Toolforge, 06tools-platform-team, 06Product Safety and Integrity, 06Security-Team: Request deletion of Toolforge project: toolforge-defectdojo-secteam - https://phabricator.wikimedia.org/T441688#12449436 (10taavi) 05Open→03Resolved a:03taavi Ah, no, sorry, I missed the part where noone in your... [15:14:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:19:22] (03update) 10dcaro: rotationg_file_flag: allow creating a file to flag for cred rotation [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/3 (https://phabricator.wikimedia.org/T440952) [15:20:22] 10Toolforge, 06tools-platform-team: [maintain-dbusers] allow self-rotation of user db credentials - https://phabricator.wikimedia.org/T441699 (10dcaro) 03NEW [15:23:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:28:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:29:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:33:08] (03approved) 10fnegri: rotationg_file_flag: allow creating a file to flag for cred rotation [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/3 (https://phabricator.wikimedia.org/T440952) (owner: 10dcaro) [15:33:10] (03update) 10fnegri: rotationg_file_flag: allow creating a file to flag for cred rotation [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/3 (https://phabricator.wikimedia.org/T440952) (owner: 10dcaro) [15:33:50] (03update) 10fnegri: rotationg_file_flag: allow creating a file to flag for cred rotation [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/3 (https://phabricator.wikimedia.org/T440952) (owner: 10dcaro) [15:34:00] (03CR) 10Volans: [C:03+1] "To be tested but LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341953 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah) [15:38:06] (03merge) 10dcaro: rotationg_file_flag: allow creating a file to flag for cred rotation [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/3 (https://phabricator.wikimedia.org/T440952) [15:39:16] (03open) 10dcaro: d/changelog: bump to 0.1.1 [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/4 (https://phabricator.wikimedia.org/T440952) [15:39:57] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component replica-cnf [15:41:52] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component replica-cnf [15:42:36] !log dcaro@cloudcumin1001 paws START - Cookbook wmcs.toolforge.component.deploy for component replica-cnf [15:42:51] !log dcaro@cloudcumin1001 paws END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component replica-cnf [15:43:54] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component replica-cnf [15:44:22] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component replica-cnf [15:44:45] (03merge) 10dcaro: d/changelog: bump to 0.1.1 [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/4 (https://phabricator.wikimedia.org/T440952) [15:45:11] (03approved) 10dcaro: d/changelog: bump to 0.1.1 [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/4 (https://phabricator.wikimedia.org/T440952) [16:05:01] 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team: clouddb memory alert - https://phabricator.wikimedia.org/T438200#12450122 (10fnegri) 05Open→03In progress [16:16:42] 10Tool-wikinewsie: Live pageviews for Wikinewsie - https://phabricator.wikimedia.org/T432576#12450211 (10JMonton-WMF) Related to the work you mention on T432352, there is an internal stream [[ https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Traffic/Pageview_Trending_Relative#event.pageview_trending_r... [16:25:13] 10Toolforge, 06tools-platform-team: [maintain-dbusers] allow self-rotation of user db credentials - https://phabricator.wikimedia.org/T441699#12450251 (10dcaro) 05Open→03In progress [16:25:14] 10Toolforge, 06tools-platform-team: [maintain-dbusers] allow self-rotation of user db credentials - https://phabricator.wikimedia.org/T441699#12450252 (10dcaro) p:05Triage→03High [16:25:19] 10Toolforge, 06tools-platform-team: [maintain-dbusers] allow self-rotation of user db credentials - https://phabricator.wikimedia.org/T441699#12450253 (10dcaro) p:05High→03Triage [16:29:47] (03Abandoned) 10David Caro: gitlab: add a note of maybe using the wrong branch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1090431 (owner: 10David Caro) [16:32:01] (03Abandoned) 10David Caro: get_project_for_proxy: cookbook to get a project given a web proxy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1318716 (owner: 10David Caro) [16:44:28] 10Toolforge, 06tools-platform-team: [maintain-dbusers] Allow self-rotating credentials - https://phabricator.wikimedia.org/T440952#12450413 (10dcaro) [16:44:37] 10Toolforge, 06tools-platform-team: [maintain-dbusers] Allow self-rotating credentials - https://phabricator.wikimedia.org/T440952#12450415 (10dcaro) 05In progress→03Resolved [16:48:47] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [16:49:58] (03open) 10dcaro: sshfs: follow symlinks [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/347 (https://phabricator.wikimedia.org/T432276) [16:51:53] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [lima-kilo] building locally jobs-api when a .tox env exists in the host code dir errors out - https://phabricator.wikimedia.org/T432276#12450445 (10dcaro) a:03dcaro Sorry @Mahveotm for taking it from you, it's blocking me every time I refetch lima-ki... [16:59:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:04:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:33:47] FIRING: SystemdUnitFailed: check-private-data.service on clouddb1023:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:56:05] 10Tool-wsindex, 10Wikisource Reader App (Android), 13Patch-For-Review: WS Index thumbnail URLs are broken if the file is redirected - https://phabricator.wikimedia.org/T432367#12450766 (10RCodes) Hi @Bodhisattwa, will this task be added for Outreachy? as i already open PR for it, i was wondering if i can rec... [18:05:45] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/80 (owner: 10l10n-bot) [18:06:15] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/80 (owner: 10l10n-bot) [18:07:12] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/53 (owner: 10l10n-bot) [18:07:16] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/53 (owner: 10l10n-bot) [18:08:05] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/45 (owner: 10l10n-bot) [18:08:22] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/45 (owner: 10l10n-bot) [18:25:36] (03update) 10mugurolevy: feat:implement search by translator [toolforge-repos/wsindex] - 10https://gitlab.wikimedia.org/toolforge-repos/wsindex/-/merge_requests/6 (owner: 10mercyjae) [18:41:38] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 13Patch-For-Review: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12450885 (10Southparkfan) >>! In T435393#124479... [18:43:55] (03open) 10semajk: Draft: docs: fix errors and improve local setup instructions in README [toolforge-repos/wsindex] - 10https://gitlab.wikimedia.org/toolforge-repos/wsindex/-/merge_requests/8 [18:51:29] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 13Patch-For-Review: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12450909 (10Southparkfan) 05In progress→... [19:14:24] (03update) 10semajk: Draft: docs: fix errors and improve local setup instructions in README [toolforge-repos/wsindex] - 10https://gitlab.wikimedia.org/toolforge-repos/wsindex/-/merge_requests/8 [19:16:06] 10Tool-wsindex, 10Wikisource Reader App (Android), 13Patch-For-Review: WS Index thumbnail URLs are broken if the file is redirected - https://phabricator.wikimedia.org/T432367#12451019 (10Bodhisattwa) [19:18:25] 10Tool-wsindex, 10Wikisource Reader App (Android), 13Patch-For-Review: WS Index thumbnail URLs are broken if the file is redirected - https://phabricator.wikimedia.org/T432367#12451028 (10Bodhisattwa) It is now added. You can now reclaim the ticket and wait for review by @Muguro and @Saiphani02 [19:19:12] 10Tool-wsindex, 10Wikisource Reader App (Android), 13Patch-For-Review: WS Index thumbnail URLs are broken if the file is redirected - https://phabricator.wikimedia.org/T432367#12451040 (10RCodes) Thanks @Bodhisattwa [19:20:59] 10Tool-wsindex, 10Wikisource Reader App (Android), 13Patch-For-Review: WS Index thumbnail URLs are broken if the file is redirected - https://phabricator.wikimedia.org/T432367#12451045 (10RCodes) @Bodhisattwa, should i wait till new issues will be added? i have raised PR for all tasks i had on hand. [19:26:54] (03CR) 10Dzahn: [C:03+2] "thank you! sgtm" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1356522 (https://phabricator.wikimedia.org/T268199) (owner: 10Krinkle) [19:29:19] 10Tool-wsindex, 10Wikisource Reader App (Android), 13Patch-For-Review: WS Index thumbnail URLs are broken if the file is redirected - https://phabricator.wikimedia.org/T432367#12451096 (10RCodes) a:03RCodes Hi mentors, @Muguro, @Saiphani02 i have already raised MR for this (https://gitlab.wikimedia.org/to... [19:29:35] (03Merged) 10jenkins-bot: frontend: Make backend a required parameter for getHoundApi() [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1356522 (https://phabricator.wikimedia.org/T268199) (owner: 10Krinkle) [19:57:12] 10Tool-wsindex, 10Wikisource Reader App (Android): Values keep fluctuating betwen locale and English - https://phabricator.wikimedia.org/T426046#12451310 (10SemajK) Hello @Bodhisattwa , Kindly requesting to be assigned this issue, As soon as it hits the Outreachy issue board. Thanks [20:22:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:27:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:31:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:36:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:48:47] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [21:04:55] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [lima-kilo] building locally jobs-api when a .tox env exists in the host code dir errors out - https://phabricator.wikimedia.org/T432276#12451653 (10Mahveotm) Oh, no issues. Thanks! [21:27:29] 10Cloud-VPS (Project-requests): Request creation of SignWriting VPS project - https://phabricator.wikimedia.org/T441738 (10Etrapani) 03NEW [21:32:32] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.set_maintenance (T431374) [21:32:34] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) (T431374) [21:32:38] T431374: cloudvirt1071 crash, again - https://phabricator.wikimedia.org/T431374 [21:33:47] FIRING: SystemdUnitFailed: check-private-data.service on clouddb1023:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:35:56] 10Cloud-VPS (Project-requests): Request creation of SignWriting VPS project - https://phabricator.wikimedia.org/T441738#12451777 (10neriah) a:05Etrapani→03None