[00:02:34] FIRING: DiskSpace: Disk space cloudcontrol1011:9100:/ 6.946% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudcontrol1011 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [00:12:34] RESOLVED: DiskSpace: Disk space cloudcontrol1011:9100:/ 6.888% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudcontrol1011 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [00:17:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:22:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:24:24] FIRING: SystemdUnitFailed: kiwix-mirror-update.service on clouddumps1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:41:53] 10Cloud-VPS, 06tools-infrastructure-team: Trove instance nt7vvmigxlj (mna-db) stuck in RESIZE - https://phabricator.wikimedia.org/T440801#12428614 (10Andrew) 05Open→03Resolved a:03Andrew The state of this was odd -- the VM showed state 'active' and was still in the older, smaller size. So somehow Tro... [00:42:24] 10Cloud-VPS, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10Puppet CI: PCC: no facts for Cloud VPS hosts on the cloudinfra puppetserver (codesearch10) - https://phabricator.wikimedia.org/T441067 (10Dzahn) 03NEW [01:06:14] 10VPS-project-Codesearch, 06Collaboration-Services, 13Patch-For-Review: Graduate codesearch to production - https://phabricator.wikimedia.org/T268199#12428654 (10Dzahn) [01:06:44] 10Cloud-VPS, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10Puppet CI: PCC: no facts for Cloud VPS hosts on the cloudinfra puppetserver (codesearch10) - https://phabricator.wikimedia.org/T441067#12428655 (10Dzahn) [01:06:48] 10VPS-project-Codesearch, 06Collaboration-Services, 13Patch-For-Review: Graduate codesearch to production - https://phabricator.wikimedia.org/T268199#12428656 (10Dzahn) [01:19:01] (03PS5) 10Dzahn: frontend: build and publish a frontend image with blubber [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1354853 (https://phabricator.wikimedia.org/T268199) [01:21:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:24:44] (03PS6) 10Dzahn: frontend: build and publish a frontend image with blubber [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1354853 (https://phabricator.wikimedia.org/T268199) [01:26:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:27:00] (03PS2) 10Dzahn: frontend: make the public Hound URL configurable [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1354897 (https://phabricator.wikimedia.org/T268199) [01:27:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:32:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:03:32] (03PS3) 10Krinkle: write_config: Add mediawiki/vendor to "Everything" preset [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1336608 (https://phabricator.wikimedia.org/T227704) [02:08:58] (03CR) 10Krinkle: [C:03+2] write_config: Add mediawiki/vendor to "Everything" preset [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1336608 (https://phabricator.wikimedia.org/T227704) (owner: 10Krinkle) [02:09:44] (03Merged) 10jenkins-bot: write_config: Add mediawiki/vendor to "Everything" preset [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1336608 (https://phabricator.wikimedia.org/T227704) (owner: 10Krinkle) [02:53:17] 10Tool-wsindex, 10Wikisource Reader App (Android): Add descriptions for books where Wikipedia articles are available [API] - https://phabricator.wikimedia.org/T430400#12428686 (10Bodhisattwa) >>! In T430400#12427767, @RCodes wrote: > Before implementing it, could I clarify a few details? > > 1. Should the API... [03:43:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:48:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:49:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:54:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:02:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:07:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:24:25] FIRING: SystemdUnitFailed: kiwix-mirror-update.service on clouddumps1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:34:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:39:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:08:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:18:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:26:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:30:23] (03close) 10countcount: Update python Docker tag to v3.15 [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/47 (owner: 10renovatebot) [05:31:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:44:09] (03open) 10countcount: Update ruff to 0.17.0 and drop unused noqa comments [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/37 [05:44:15] (03open) 10countcount: Update ruff to 0.17.0 [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/48 [05:44:17] (03open) 10countcount: Update ruff to 0.17.0 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/36 [05:49:51] (03merge) 10countcount: Update ruff to 0.17.0 and drop unused noqa comments [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/37 [05:49:54] (03merge) 10countcount: Update ruff to 0.17.0 [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/48 [05:49:58] (03merge) 10countcount: Update ruff to 0.17.0 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/36 [06:17:15] (03update) 10renovatebot: Update python Docker tag to v3.15.0 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [06:28:43] (03merge) 10countcount: Update pnpm to v12.11.2 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/38 (owner: 10renovatebot) [06:28:45] (03merge) 10countcount: Update pnpm to v12.11.2 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/34 (owner: 10renovatebot) [06:28:50] (03merge) 10countcount: Update pnpm to v12.11.2 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/27 (owner: 10renovatebot) [06:28:51] (03merge) 10countcount: Update pnpm to v12.11.2 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/66 (owner: 10renovatebot) [06:28:56] (03merge) 10countcount: Update pnpm to v12.11.2 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/53 (owner: 10renovatebot) [06:38:37] (03close) 10countcount: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/36 (owner: 10renovatebot) [06:44:21] (03close) 10countcount: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/35 (owner: 10renovatebot) [06:44:21] (03update) 10countcount: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/35 (owner: 10renovatebot) [06:44:23] (03close) 10countcount: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/46 (owner: 10renovatebot) [06:44:25] (03update) 10countcount: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/46 (owner: 10renovatebot) [06:49:21] (03update) 10renovatebot: Update dependency happy-dom to v21 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/29 [06:49:28] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/28 [06:49:43] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/39 [06:49:55] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/35 [06:50:57] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/67 [06:52:32] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/54 [07:05:23] 10Toolforge, 10Tools, 06tools-platform-team: Mix'n'match: Temporary DB size increase to reduce DB size - https://phabricator.wikimedia.org/T440708#12429076 (10magnusmanske) OK, table is ready. Can you add the following indices: ` -- 7.1 SET SESSION lock_wait_timeout=10; SET SESSION max_statement_time=0; ALT... [07:11:02] 10Tool-wikicrawler: Wiki Crawler: Request OAuth 2.0 grants needed for authenticated API reads - https://phabricator.wikimedia.org/T440613#12429123 (10Adityakumar0545) 05Open→03In progress [07:49:29] 06tools-infrastructure-team: Enable perf_counters on cloudceph - https://phabricator.wikimedia.org/T440686#12429243 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi change applied, a little less than 3x more metrics and samples ingested for prometheus cloud instance. in the grand scheme of things this i... [08:19:25] RESOLVED: SystemdUnitFailed: kiwix-mirror-update.service on clouddumps1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:20:07] 10Cloud-VPS (Quota-requests): Quota increase request for project catalyst - https://phabricator.wikimedia.org/T441094 (10jnuche) 03NEW [08:38:00] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [08:39:01] 10Toolforge, 06tools-platform-team, 07OKR-Work: Striker: Show active tool alerts - https://phabricator.wikimedia.org/T441098 (10TLepage-WMF) 03NEW [08:40:05] 10Toolforge, 06tools-platform-team, 07OKR-Work: Striker: See past alerts - https://phabricator.wikimedia.org/T441099 (10TLepage-WMF) 03NEW [08:40:18] 10Toolforge, 06tools-platform-team: [replica-cnf] The uwsgi service stops sometimes without errors - https://phabricator.wikimedia.org/T440948#12429445 (10dcaro) This might be related, from `dmesg -T` on toolsbeta nfs: ` [08:40:24] 10Toolforge, 06tools-platform-team, 07OKR-Work: Striker: Show active tool alerts - https://phabricator.wikimedia.org/T441098#12429446 (10TLepage-WMF) [08:40:25] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.3 Implement Toolforge UI MVP - https://phabricator.wikimedia.org/T440589#12429447 (10TLepage-WMF) [08:41:05] 10Toolforge, 06tools-platform-team, 07OKR-Work: Striker: See past alerts - https://phabricator.wikimedia.org/T441099#12429448 (10TLepage-WMF) [08:41:06] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.3 Implement Toolforge UI MVP - https://phabricator.wikimedia.org/T440589#12429449 (10TLepage-WMF) [08:42:16] 10Toolforge, 06tools-platform-team, 07OKR-Work: Striker: See alert-related logs - https://phabricator.wikimedia.org/T441100 (10TLepage-WMF) 03NEW [08:42:29] 10Toolforge, 06tools-platform-team, 07OKR-Work: Striker: See alert-related logs - https://phabricator.wikimedia.org/T441100#12429463 (10TLepage-WMF) [08:42:30] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.3 Implement Toolforge UI MVP - https://phabricator.wikimedia.org/T440589#12429464 (10TLepage-WMF) [08:43:33] 10Toolforge, 06tools-platform-team, 07OKR-Work: Striker: See run logs - https://phabricator.wikimedia.org/T441101 (10TLepage-WMF) 03NEW [08:43:49] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.3 Implement Toolforge UI MVP - https://phabricator.wikimedia.org/T440589#12429476 (10TLepage-WMF) [08:43:50] 10Toolforge, 06tools-platform-team, 07OKR-Work: Striker: See run logs - https://phabricator.wikimedia.org/T441101#12429475 (10TLepage-WMF) [09:05:10] 10Quarry, 06tools-platform-team: Quarry: Access Denied Error - https://phabricator.wikimedia.org/T440966#12429544 (10VeritasSapientiae) 05In progress→03Resolved It works, thanks! [09:11:38] 10Toolforge, 06tools-platform-team: [replica-cnf] The uwsgi service stops sometimes without errors - https://phabricator.wikimedia.org/T440948#12429589 (10dcaro) Nope this seems to be the cause: ` Oct 13 17:47:19 toolsbeta-nfs-4 puppet-agent[4170458]: (/Stage[main]/Uwsgi/Service[uwsgi]/ensure) ensure changed '... [09:31:47] 10Toolforge, 06tools-platform-team, 07OKR-Work: Striker: See run logs - https://phabricator.wikimedia.org/T441101#12429668 (10TLepage-WMF) [09:31:48] 10Toolforge, 06tools-platform-team, 07OKR-Work: api-gateway: Support bearer-token auth for Striker - https://phabricator.wikimedia.org/T440736#12429667 (10TLepage-WMF) [09:31:49] 10Toolforge, 06tools-platform-team, 07OKR-Work: Striker: See alert-related logs - https://phabricator.wikimedia.org/T441100#12429669 (10TLepage-WMF) [09:31:50] 10Toolforge, 06tools-platform-team, 07OKR-Work: Striker: See past alerts - https://phabricator.wikimedia.org/T441099#12429670 (10TLepage-WMF) [09:31:52] 10Toolforge, 06tools-platform-team, 07OKR-Work: Striker: Show active tool alerts - https://phabricator.wikimedia.org/T441098#12429671 (10TLepage-WMF) [09:31:53] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.3 Implement Toolforge UI MVP - https://phabricator.wikimedia.org/T440589#12429672 (10TLepage-WMF) [09:33:19] 10Toolforge, 06tools-platform-team: [replica-cnf] The uwsgi service stops sometimes without errors - https://phabricator.wikimedia.org/T440948#12429674 (10dcaro) the right service is uwsgi-: ` root@toolsbeta-nfs-4:~# systemctl status uwsgi-toolsdb-replica-cnf-web.service ● uwsgi-toolsdb-replica... [09:33:29] 10Toolforge, 06tools-platform-team: [replica-cnf] The uwsgi service stops sometimes without errors - https://phabricator.wikimedia.org/T440948#12429675 (10dcaro) 05Open→03Invalid [09:38:00] FIRING: [2x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [09:54:22] 06cloud-services-team, 10Data-Services, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Move replica_cnf_api out of the Puppet repo - https://phabricator.wikimedia.org/T340754#12429709 (10dcaro) [09:56:38] 06cloud-services-team, 10Data-Services, 10Toolforge, 06tools-platform-team: [replica-cnf] Package the new component - https://phabricator.wikimedia.org/T441108 (10dcaro) 03NEW [09:56:54] 06cloud-services-team, 10Data-Services, 10Toolforge, 06tools-platform-team: [replica-cnf] Package the new component - https://phabricator.wikimedia.org/T441108#12429730 (10dcaro) p:05Triage→03High [10:04:35] 10Tool-wikicrawler: Wiki Crawler: Add Wiki Article Race page with split-screen layout - https://phabricator.wikimedia.org/T441109 (10Jaishsingh913) 03NEW [10:05:33] 10Tool-wikicrawler: Wiki Crawler: Add Wiki Article Race page with split-screen layout - https://phabricator.wikimedia.org/T441109#12429761 (10Adityakumar0545) a:03Adityakumar0545 [10:18:00] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [10:21:05] 10Toolforge, 10Tools, 06tools-platform-team: Mix'n'match: Temporary DB size increase to reduce DB size - https://phabricator.wikimedia.org/T440708#12429828 (10fnegri) Currently adding the first index, I'll let you know when all are done. [10:33:02] (03open) 10dcaro: package: add debian packaging bits [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/2 (https://phabricator.wikimedia.org/T441108) [10:33:03] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance toolsbeta-nfs-4 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [10:34:59] (03update) 10dcaro: package: add debian packaging bits [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/2 (https://phabricator.wikimedia.org/T441108) [10:38:10] (03update) 10dcaro: package: add debian packaging bits [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/2 (https://phabricator.wikimedia.org/T441108) [10:48:03] FIRING: [2x] PuppetAgentFailure: Puppet agent failure detected on instance toolsbeta-nfs-3 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [10:53:03] FIRING: [3x] PuppetAgentFailure: Puppet agent failure detected on instance toolsbeta-nfs-3 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [10:59:07] 10Tool-lingua-datasets, 10Toolforge (Quota-requests), 06tools-platform-team: Setup Lingua-datasets.toolforge.org with 400GB - https://phabricator.wikimedia.org/T440689#12429980 (10Yug) More generally speaking, why don't we have a solid system to export Wikimedia Commons' archives on a per-category basis ? [12:28:34] 10Tool-delintbot: Fix all sorts of bogus-image-options errors - https://phabricator.wikimedia.org/T421530#12430422 (10Redmin) [12:37:42] (03update) 10dcaro: package: add debian packaging bits [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/2 (https://phabricator.wikimedia.org/T441108) [12:45:11] 10Toolforge, 06tools-infrastructure-team, 13Patch-For-Review: Upgrade toolforge calico to v3.31 - https://phabricator.wikimedia.org/T441022#12430537 (10aputhin) [12:48:14] (03update) 10dcaro: package: add debian packaging bits [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/2 (https://phabricator.wikimedia.org/T441108) [12:57:48] (03open) 10dcaro: helpers: add replica-cnf package support [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/346 (https://phabricator.wikimedia.org/T441108) [12:58:37] (03update) 10dcaro: helpers: add replica-cnf package support [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/346 (https://phabricator.wikimedia.org/T441108) [13:10:58] 10Toolforge, 06tools-platform-team, 07OKR-Work: alerts-api: List all active alerts for a tool - https://phabricator.wikimedia.org/T441144 (10TLepage-WMF) 03NEW [13:11:04] (03open) 10dcaro: toolforge_get_versions: don't fail on missing package [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1466 [13:11:44] 10Toolforge, 06tools-platform-team, 07OKR-Work: alerts-api: List all active alerts for a tool - https://phabricator.wikimedia.org/T441144#12430671 (10TLepage-WMF) [13:11:45] 10Toolforge, 06tools-platform-team, 07OKR-Work: Striker: Show active tool alerts - https://phabricator.wikimedia.org/T441098#12430672 (10TLepage-WMF) [13:12:52] (03open) 10dcaro: replica_cnf: add to versions and empty tests [repos/cloud/toolforge/toolforge-deploy] (fix_missing_package) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1467 (https://phabricator.wikimedia.org/T441108) [13:19:20] 10Toolforge, 06tools-platform-team, 07OKR-Work: alerts-api: List a range of previously-active alerts for a tool - https://phabricator.wikimedia.org/T441145 (10TLepage-WMF) 03NEW [13:19:33] 10Toolforge, 06tools-platform-team, 07OKR-Work: alerts-api: List a range of previously-active alerts for a tool - https://phabricator.wikimedia.org/T441145#12430722 (10TLepage-WMF) [13:19:56] 10Toolforge, 06tools-platform-team, 07OKR-Work: alerts-api: List a range of previously-active alerts for a tool - https://phabricator.wikimedia.org/T441145#12430724 (10TLepage-WMF) [13:19:57] 10Toolforge, 06tools-platform-team, 07OKR-Work: Striker: See past alerts - https://phabricator.wikimedia.org/T441099#12430725 (10TLepage-WMF) [13:20:06] 10Toolforge, 06tools-platform-team, 07OKR-Work: alerts-api: List all active alerts for a tool - https://phabricator.wikimedia.org/T441144#12430728 (10TLepage-WMF) [13:31:11] (03update) 10dcaro: replica_cnf: add to versions and empty tests [repos/cloud/toolforge/toolforge-deploy] (fix_missing_package) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1467 (https://phabricator.wikimedia.org/T441108) [13:31:28] 10Tool-wsindex, 10Wikisource Reader App (Android): Add an option to find new books added to the app - https://phabricator.wikimedia.org/T410597#12430807 (10Mercyjae) Hi @Saiphani02 @Bodhisattwa @Muguro I've been reviewing T410597 (Add an option to find new books). I'm interested in contributing for this featu... [13:32:34] (03update) 10dcaro: replica_cnf: add to versions and empty tests [repos/cloud/toolforge/toolforge-deploy] (fix_missing_package) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1467 (https://phabricator.wikimedia.org/T441108) [13:34:52] (03PS1) 10David Caro: toolforge.deploy: add replica-cnf [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1355062 (https://phabricator.wikimedia.org/T441108) [13:46:34] (03PS2) 10David Caro: toolforge.deploy: add replica-cnf [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1355062 (https://phabricator.wikimedia.org/T441108) [13:46:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:51:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:55:40] 10Cloud-VPS (Quota-requests): Quota increase request for project catalyst - https://phabricator.wikimedia.org/T441094#12431099 (10Andrew) +1 [14:00:33] !log filippo@cloudcumin1001 catalyst START - Cookbook wmcs.openstack.quota_increase by 12 volumes [14:00:38] !log filippo@cloudcumin1001 catalyst END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 12 volumes [14:00:56] 10Cloud-VPS (Quota-requests): Quota increase request for project catalyst - https://phabricator.wikimedia.org/T441094#12431158 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi {{done}} [14:03:00] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [14:03:13] 10Tool-lingua-datasets, 10Toolforge (Quota-requests), 06tools-platform-team: Setup Lingua-datasets.toolforge.org with 400GB - https://phabricator.wikimedia.org/T440689#12431206 (10fgiunchedi) >>! In T440689#12427957, @Yug wrote: > Hello @fgiunchedi , my repository tree is like this so far : > > > ### Struc... [14:08:46] 10Tool-wsindex, 10Wikisource Reader App (Android): Add descriptions for books where Wikipedia articles are available [API] - https://phabricator.wikimedia.org/T430400#12431244 (10Slimfatima) Hello everyone! 👋🏽😊 My name is Fatima Aruna, and I’m excited to join the Wikimedia community! I’m a Systems Engineerin... [14:09:08] 10Tool-delintbot: Explore working with specific parts of page Wikitext - https://phabricator.wikimedia.org/T440911#12431246 (10Redmin) 05Open→03In progress p:05Triage→03High [14:12:09] 10Toolforge, 06tools-platform-team: Can't retrieve logs for wsstats tool - https://phabricator.wikimedia.org/T440714#12431259 (10aputhin) →14Duplicate dup:03T429265 [14:12:10] 10Toolforge, 06tools-platform-team: [logs-api] Using -f on a job with no recent logs fails with Internal Server Error - https://phabricator.wikimedia.org/T429265#12431261 (10aputhin) [14:14:57] 10Tool-wsindex, 10Wikisource Reader App (Android): Add descriptions for books where Wikipedia articles are available [API] - https://phabricator.wikimedia.org/T430400#12431269 (10Bodhisattwa) >>! In T430400#12431244, @Slimfatima wrote: > Hello everyone! 👋🏽😊 > > My name is Fatima Aruna, and I’m excited to join... [14:15:01] 10Toolforge, 06tools-platform-team: [replica_cnf,functional-tests] Run replica_cnf functional tests in lima-kilo with the rest of functional tests - https://phabricator.wikimedia.org/T369800#12431271 (10aputhin) p:05Medium→03High [14:20:32] 10Tool-wsindex, 10Wikisource Reader App (Android): Add descriptions for books where Wikipedia articles are available [API] - https://phabricator.wikimedia.org/T430400#12431311 (10RCodes) Thank you for the clarification, @Bodhisattwa! I am working on the implementation now and will open a Merge Request once I'm... [14:20:35] 10Tool-wsindex, 10Wikisource Reader App (Android): Add summary statistics to analytics page - https://phabricator.wikimedia.org/T430399#12431319 (10Bodhisattwa) Sure, please go ahead and claim the task. Happy coding! [14:21:53] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [toolforge,infra] Centralized logging for Toolforge infrastructure logs - https://phabricator.wikimedia.org/T97861#12431328 (10aputhin) p:05Medium→03High Recent issues have greatly increased the priority for this, so we could reduce our blind spot i... [14:22:35] 10Tool-wsindex, 10Wikisource Reader App (Android): Add descriptions for books where Wikipedia articles are available [API] - https://phabricator.wikimedia.org/T430400#12431335 (10Bodhisattwa) a:03RCodes >>! In T430400#12431311, @RCodes wrote: > Thank you for the clarification, @Bodhisattwa! I am working on t... [14:23:25] 10Tool-wsindex, 10Wikisource Reader App (Android): Add descriptions for books where Wikipedia articles are available [API] - https://phabricator.wikimedia.org/T430400#12431343 (10RCodes) Thank you @Bodhisattwa [14:24:33] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance toolsbeta-nfs-4 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [14:25:49] 10Tool-wsindex, 10Wikisource Reader App (Android): Add summary statistics to analytics page - https://phabricator.wikimedia.org/T430399#12431362 (10RCodes) a:03RCodes [14:27:01] !log filippo@cloudcumin1001 cloudinfra START - Cookbook wmcs.openstack.cloudvirt.vm_console [14:27:03] !log filippo@cloudcumin1001 cloudinfra END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) [14:27:14] !log filippo@cloudcumin1001 cloudinfra START - Cookbook wmcs.openstack.cloudvirt.vm_console [14:27:16] !log filippo@cloudcumin1001 cloudinfra END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) [14:30:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:35:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:35:58] 10Toolforge, 06tools-platform-team: Review ToolsDB / Replica credential creation process - https://phabricator.wikimedia.org/T441164 (10aputhin) 03NEW [14:36:31] 10Toolforge, 06tools-platform-team: Review ToolsDB / Replica credential creation process - https://phabricator.wikimedia.org/T441164#12431471 (10aputhin) [14:36:35] 06cloud-services-team, 10Toolforge, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 07Epic: [dbaas] Add DB as a service capabilities to toolforge - https://phabricator.wikimedia.org/T384586#12431472 (10aputhin) [14:36:38] 10Toolforge, 06tools-platform-team: Review ToolsDB / Replica credential creation process - https://phabricator.wikimedia.org/T441164#12431475 (10aputhin) p:05Triage→03High [14:37:16] 10Toolforge, 06tools-platform-team: [toolsdb] Enable db-level and user-level monitoring - https://phabricator.wikimedia.org/T428087#12431476 (10aputhin) p:05Triage→03Medium [14:38:29] (03update) 10andrew: kube-state-metrics: bump chart version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1463 (https://phabricator.wikimedia.org/T414595) [14:38:57] (03update) 10andrew: kube-state-metrics: bump chart version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1463 (https://phabricator.wikimedia.org/T414595) [14:39:08] 10Toolforge, 06tools-platform-team: Review ToolsDB / Replica credential creation process - https://phabricator.wikimedia.org/T441164#12431491 (10taavi) How does this relate to {T140832}? [14:41:12] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, 10ops-eqiad: cloudvirt1071 crash, again - https://phabricator.wikimedia.org/T431374#12431548 (10Andrew) [14:43:12] 10Toolforge, 06tools-platform-team: [logs-api] API returns 500 when a stream has no "job_name" - https://phabricator.wikimedia.org/T440718#12431569 (10aputhin) p:05Triage→03Medium [14:44:24] 10Toolforge, 06tools-platform-team: Review ToolsDB / Replica credential creation process - https://phabricator.wikimedia.org/T441164#12431578 (10aputhin) >>! In T441164#12431491, @taavi wrote: > How does this relate to {T140832}? This one does not necessarily specify UI features for credential management. We... [14:45:03] FIRING: PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-nfs-3 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [14:45:24] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, 10ops-eqiad: cloudvirt1071 crash, again - https://phabricator.wikimedia.org/T431374#12431581 (10RobH) Recommended next steps by onsite: * check idrac/system event logs for hw failure * run memtest and CPU test liveimage I've go... [14:46:41] 10Tool-wsindex, 10Wikisource Reader App (Android): Add an option to find new books added to the app - https://phabricator.wikimedia.org/T410597#12431585 (10Bodhisattwa) @Mercyjae - we have not decided to add this task for Outreachy. Please wait until we do that. In the meantime, you can select from the subtask... [14:49:36] 10Cloud-VPS, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10Puppet CI: PCC: no facts for Cloud VPS hosts on the cloudinfra puppetserver (codesearch10) - https://phabricator.wikimedia.org/T441067#12431590 (10fgiunchedi) I can confirm the facts file is on the server: ` root@cloudinfra-cloudvps... [14:50:03] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-nfs-3 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [14:52:57] (03open) 10r4356th: Remove file names passed as file options [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/38 (https://phabricator.wikimedia.org/T421530) [14:53:03] (03update) 10r4356th: Remove file names passed as file options [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/38 (https://phabricator.wikimedia.org/T421530) [14:54:09] (03merge) 10r4356th: Remove file names passed as file options [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/38 (https://phabricator.wikimedia.org/T421530) [15:04:45] 10Striker, 06tools-platform-team, 07OKR-Work: Striker: See run logs - https://phabricator.wikimedia.org/T441101#12431709 (10JJMC89) [15:05:00] 10Striker, 06tools-platform-team, 07OKR-Work: Striker: See alert-related logs - https://phabricator.wikimedia.org/T441100#12431710 (10JJMC89) [15:05:03] FIRING: PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-nfs-4 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [15:05:51] (03update) 10dcaro: package: add debian packaging bits [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/2 (https://phabricator.wikimedia.org/T441108) [15:07:28] 10Striker, 06tools-platform-team, 07OKR-Work: Striker: Show active tool alerts - https://phabricator.wikimedia.org/T441098#12431750 (10JJMC89) [15:08:06] 10Striker, 06tools-platform-team, 07OKR-Work: Striker: See past alerts - https://phabricator.wikimedia.org/T441099#12431786 (10JJMC89) [15:08:12] 10Tool-delintbot, 13Patch-For-Review: Fix all sorts of bogus-image-options errors - https://phabricator.wikimedia.org/T421530#12431787 (10Redmin) [15:11:40] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 05Cloud-Services-Origin-Team, and 2 others: [dbaas,toolsdb] Add support for management of toolsdb databases within toolforge - https://phabricator.wikimedia.org/T384591#12431928 (10dcaro) [15:14:34] 10Striker, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.3 Implement Toolforge UI MVP - https://phabricator.wikimedia.org/T440589#12431962 (10TLepage-WMF) [15:15:05] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.3 Implement Toolforge UI MVP - https://phabricator.wikimedia.org/T440589#12431964 (10TLepage-WMF) [15:18:00] FIRING: [2x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [15:20:03] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-nfs-4 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [15:50:03] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance toolsbeta-nfs-5 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [15:53:13] (03open) 10r4356th: Ensure empty nowiki tags are present as file options before removing them [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/39 (https://phabricator.wikimedia.org/T441008) [15:53:31] 10Cloud-VPS, 06tools-infrastructure-team, 10Cumin, 06Infrastructure-Foundations: Enable cumin hostfile backend on cloudcumin hosts - https://phabricator.wikimedia.org/T433916#12432154 (10elukey) a:05elukey→03None [15:54:23] (03update) 10r4356th: Ensure empty nowiki tags are present as file options before removing them [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/39 (https://phabricator.wikimedia.org/T441008) [15:54:48] (03merge) 10r4356th: Ensure empty nowiki tags are present as file options before removing them [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/39 (https://phabricator.wikimedia.org/T441008) [15:55:21] 10Tool-delintbot, 13Patch-For-Review: Ensure empty nowiki tags are present as file options - https://phabricator.wikimedia.org/T441008#12432166 (10Redmin) 05Open→03Resolved a:03Redmin [15:55:51] 10Tool-wikinewsie: Wikinewsie portals page translatable - https://phabricator.wikimedia.org/T441192 (10Pharos) 03NEW [16:01:59] (03open) 10rabuma: Add Wikipedia descriptions for books (T430400) [toolforge-repos/wsindex] - 10https://gitlab.wikimedia.org/toolforge-repos/wsindex/-/merge_requests/3 (https://phabricator.wikimedia.org/T430400) [16:06:30] 10Tool-delintbot: Document how to operate the bot - https://phabricator.wikimedia.org/T416070#12432228 (10Redmin) [16:08:13] 10Toolforge, 10Tools, 06tools-platform-team: Mix'n'match: Temporary DB size increase to reduce DB size - https://phabricator.wikimedia.org/T440708#12432246 (10aputhin) 05Open→03In progress [16:08:30] 10Toolforge, 10Tools, 06tools-platform-team: Mix'n'match: Temporary DB size increase to reduce DB size - https://phabricator.wikimedia.org/T440708#12432261 (10aputhin) p:05Triage→03Medium [16:11:12] 10Tool-wsindex, 10Wikisource Reader App (Android), 13Patch-For-Review: Add descriptions for books where Wikipedia articles are available [API] - https://phabricator.wikimedia.org/T430400#12432294 (10Slimfatima) Thank you @Bodhisattwa will do that [16:15:03] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance toolsbeta-nfs-3 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [16:18:00] RESOLVED: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [16:25:14] 10Tool-wsindex, 10Wikisource Reader App (Android), 13Patch-For-Review: Add descriptions for books where Wikipedia articles are available [API] - https://phabricator.wikimedia.org/T430400#12432336 (10RCodes) Hi @Bodhisattwa, I've opened a merge request for this task (https://gitlab.wikimedia.org/toolforge-rep... [16:26:17] 10Toolforge, 06tools-platform-team: [*-api] don't allow non-canonical urls anywhere - https://phabricator.wikimedia.org/T441195 (10dcaro) 03NEW [16:28:36] 10Tool-wsindex, 10Wikisource Reader App (Android), 13Patch-For-Review: Add descriptions for books where Wikipedia articles are available [API] - https://phabricator.wikimedia.org/T430400#12432356 (10Bodhisattwa) >>! In T430400#12432336, @RCodes wrote: > Hi @Bodhisattwa, I've opened a merge request for this t... [16:29:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:30:27] 10Tool-wsindex, 10Wikisource Reader App (Android), 13Patch-For-Review: Add descriptions for books where Wikipedia articles are available [API] - https://phabricator.wikimedia.org/T430400#12432361 (10RCodes) okay @Bodhisattwa, Thanks! [16:31:29] 10Toolforge, 06tools-platform-team: [*-api] Add a check for consistency between the tool in the path and X-Toolforge-Tool - https://phabricator.wikimedia.org/T441196 (10dcaro) 03NEW [16:38:49] (03open) 10rabuma: Add catalog summary statistics to analytics page (T430399) [toolforge-repos/wsindex] - 10https://gitlab.wikimedia.org/toolforge-repos/wsindex/-/merge_requests/4 (https://phabricator.wikimedia.org/T430399) [16:42:47] 10Tool-wsindex, 10Wikisource Reader App (Android), 13Patch-For-Review: Add summary statistics to analytics page - https://phabricator.wikimedia.org/T430399#12432570 (10RCodes) Hi @Saiphani02 and @Muguro, I've opened a merge request for this task (https://gitlab.wikimedia.org/toolforge-repos/wsindex/-/merge_... [16:52:03] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance toolsbeta-puppetserver-1 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [16:54:21] 06cloud-services-team (Hardware), 06tools-infrastructure-team, 06DC-Ops, 06SRE: How to configure storage on new dumps servers - https://phabricator.wikimedia.org/T441202 (10Andrew) 03NEW [16:54:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:58:09] (03CR) 10Thibaut Le Page: [C:03+1] toolforge.deploy: add replica-cnf [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1355062 (https://phabricator.wikimedia.org/T441108) (owner: 10David Caro) [16:59:10] (03approved) 10tlepage: toolforge_get_versions: don't fail on missing package [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1466 (owner: 10dcaro) [17:01:08] (03approved) 10tlepage: package: add debian packaging bits [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/2 (https://phabricator.wikimedia.org/T441108) (owner: 10dcaro) [17:03:47] 10VPS-project-Wikistats: Add wlxwiki to wikistats - https://phabricator.wikimedia.org/T441186#12432798 (10Dzahn) once T441168 is resolved [17:04:21] 10VPS-project-Wikistats: Add mlwikivoyage to wikistats - https://phabricator.wikimedia.org/T441178#12432803 (10Dzahn) once T441169 is resolved [17:04:49] 06cloud-services-team (Hardware), 06tools-infrastructure-team, 06DC-Ops, 06SRE: How to configure storage on new dumps servers - https://phabricator.wikimedia.org/T441202#12432810 (10Andrew) Currently one day of writes are about 700GB [17:10:43] 10Cloud-VPS, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10Puppet CI: PCC: no facts for Cloud VPS hosts on the cloudinfra puppetserver (codesearch10) - https://phabricator.wikimedia.org/T441067#12432847 (10Dzahn) Thank you! Hmm.. interesting, the issue must be elsewhere then between puppet... [17:11:15] 10Cloud-VPS, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10Puppet CI: PCC: no facts for Cloud VPS hosts on the cloudinfra puppetserver (codesearch10) - https://phabricator.wikimedia.org/T441067#12432850 (10Dzahn) a:03Dzahn [17:16:14] 06cloud-services-team (Hardware), 06tools-infrastructure-team, 06DC-Ops, 06SRE: How to configure storage on new dumps servers - https://phabricator.wikimedia.org/T441202#12432880 (10Andrew) There's quite a bit of growth here, so even the old storage size doesn't look all that sustainable. Another vote for... [18:05:32] (03PS1) 10Dzahn: write_config: add --wmf-only to index only Wikimedia-hosted repos [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1355143 (https://phabricator.wikimedia.org/T268199) [18:20:45] 10Quarry, 06tools-platform-team: [quarry] update the database password - https://phabricator.wikimedia.org/T440979#12433272 (10fnegri) 05Open→03In progress [18:32:47] 10Tool-delintbot: Fix cases of tags not being closed correctly - https://phabricator.wikimedia.org/T417483#12433312 (10Redmin) p:05Medium→03High [18:37:40] 10Cloud-VPS, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10Puppet CI: PCC: no facts for Cloud VPS hosts on the cloudinfra puppetserver (codesearch10) - https://phabricator.wikimedia.org/T441067#12433336 (10Dzahn) It seems the issue is that there are 2 cloudinfra puppetservers uploading facts... [18:38:59] 10Toolforge, 06tools-platform-team: [alerts-api] Add fake Prometheus for local testing - https://phabricator.wikimedia.org/T441222 (10fnegri) 03NEW [18:53:10] 10Tool-delintbot: Fix all sorts of bogus-image-options errors - https://phabricator.wikimedia.org/T421530#12433415 (10Redmin) [18:57:32] 10Tool-delintbot: Fix all sorts of bogus-image-options errors - https://phabricator.wikimedia.org/T421530#12433428 (10Redmin) [19:02:23] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 13Patch-For-Review: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12433475 (10Southparkfan) >>! In T435393#123574... [19:06:45] 10Toolforge, 06tools-platform-team: [alerts-api] Add more docs and e2e tests - https://phabricator.wikimedia.org/T441226 (10fnegri) 03NEW [19:08:05] 10Toolforge, 06tools-platform-team, 07OKR-Work: [alerts-api] List all active alerts for a tool - https://phabricator.wikimedia.org/T441144#12433514 (10fnegri) [19:08:16] 10Toolforge, 06tools-platform-team, 07OKR-Work: [alerts-api] List a range of previously-active alerts for a tool - https://phabricator.wikimedia.org/T441145#12433525 (10fnegri) [19:15:50] 10Toolforge, 06tools-platform-team: [alerts-api] Handle unexpected responses from a Prometheus host - https://phabricator.wikimedia.org/T441228 (10fnegri) 03NEW [19:54:19] 10Toolforge, 06tools-infrastructure-team: [infra] Upgrade Toolforge K8s etcd nodes to Trixie - https://phabricator.wikimedia.org/T432634#12433785 (10Andrew) I keep bouncing off this task. Questions that give me pause are: - does etcd 3.5.16 still speak the v2 API? - can etcd on Bookworm cluster with etcd on T... [20:01:46] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 10Wikimedia-Developer-Portal, 06MediaWiki-API-Platform-Team (MediaWiki-Interfaces-Sprints), 07OKR-Work: Create an extended property for UI friendly API module descriptions - https://phabricator.wikimedia.org/T434253#12433823 (10HCoplin-WMF) [20:16:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:21:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:45:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:49:29] 10Toolforge, 06tools-platform-team: [alerts-api] Simplify notifier params - https://phabricator.wikimedia.org/T441238 (10fnegri) 03NEW [20:49:36] 10Toolforge, 06tools-platform-team: [alerts-api] Keep defaults in a single place - https://phabricator.wikimedia.org/T441239 (10fnegri) 03NEW [20:49:45] 10Toolforge, 06tools-platform-team: [alerts-api] Improve debug logs - https://phabricator.wikimedia.org/T441240 (10fnegri) 03NEW [20:50:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:13:03] 06cloud-services-team (Hardware), 06tools-infrastructure-team, 06DC-Ops, 06SRE: How to configure storage on new dumps servers - https://phabricator.wikimedia.org/T441202#12434211 (10Andrew) Claude points out, reasonably, that taking an hour to write 700GB means we aren't limited by disk write speed anyway.... [21:19:06] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install (3) dumps100[123] - https://phabricator.wikimedia.org/T439754#12434268 (10Andrew) [21:34:32] 10Tool-wsindex, 10Wikisource Reader App (Android): WS Index thumbnail URLs are broken if the file is redirected - https://phabricator.wikimedia.org/T432367#12434351 (10RCodes) a:03RCodes [21:48:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:53:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:09:36] (03open) 10rabuma: Fix thumbnail URLs for renamed Commons files (T432367) [toolforge-repos/wsindex] - 10https://gitlab.wikimedia.org/toolforge-repos/wsindex/-/merge_requests/5 (https://phabricator.wikimedia.org/T432367) [22:18:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:33:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:43:38] (03update) 10project_5175_bot_1dc5cd2b4a7cadb6d1af8e29974ce1a7: Kurulum düzeltmeleri ve saatlik tarama işi [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/1 [22:43:38] (03update) 10project_5175_bot_1dc5cd2b4a7cadb6d1af8e29974ce1a7: Kurulum düzeltmeleri ve saatlik tarama işi [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/1 [22:48:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:53:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown