[00:16:15] (03update) 10renovatebot: Update python Docker tag to v3.15 [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/47 [00:16:40] (03update) 10renovatebot: Update python Docker tag to v3.15.0 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [00:52:42] (03approved) 10raymond-ndibe: jobs: skip artifacts that don't exist on deletion [repos/cloud/toolforge/maintain-harbor] (delete_untagged) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/76 (https://phabricator.wikimedia.org/T440343) (owner: 10dcaro) [00:54:07] (03update) 10raymond-ndibe: add pagination [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/75 (https://phabricator.wikimedia.org/T440294) (owner: 10dcaro) [01:09:57] (03approved) 10raymond-ndibe: add pagination [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/75 (https://phabricator.wikimedia.org/T440294) (owner: 10dcaro) [01:14:20] (03update) 10renovatebot: Update dependency happy-dom to v21 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/29 [01:14:22] (03open) 10renovatebot: Update dependency happy-dom to v21 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/29 [02:24:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:29:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:39:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:44:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:46:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:51:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:18:47] 10Tool-autoscout: NPP and admin rebuilds overwrite each other's AfD participation; shared afd_afc_cursor skips the second cohort - https://phabricator.wikimedia.org/T440928 (10Audiodude) 03NEW [03:18:48] 10Tool-autoscout: replace_snapshot deletes every snapshot_meta key, blanking /npp and /admin and resetting the AfD cursor - https://phabricator.wikimedia.org/T440929 (10Audiodude) 03NEW [03:18:49] 10Tool-autoscout: Shared DYK/FA bot rows saved by --daily never expire; the weekly refresh reuses rows up to a week old - https://phabricator.wikimedia.org/T440930 (10Audiodude) 03NEW [03:18:52] 10Tool-autoscout: One failed AI-edit API request aborts the whole full refresh; maxlag gives up after 7 s - https://phabricator.wikimedia.org/T440931 (10Audiodude) 03NEW [03:18:53] 10Tool-autoscout: --daily wipes an interrupted weekly run's checkpoint and leaves stale stages that a later daily can reuse - https://phabricator.wikimedia.org/T440932 (10Audiodude) 03NEW [03:20:03] 10Tool-autoscout: Weekly full refresh likely exceeds its 3600 s job timeout - https://phabricator.wikimedia.org/T440933 (10Audiodude) 03NEW [03:20:06] 10Tool-autoscout: Weekly full refresh likely exceeds its 3600 s job timeout - https://phabricator.wikimedia.org/T440933#12423421 (10Audiodude) [03:43:33] 10Tool-autoscout: NPP and admin rebuilds overwrite each other's AfD participation; shared afd_afc_cursor skips the second cohort - https://phabricator.wikimedia.org/T440928#12423427 (10Audiodude) Seen on live data too (//AI-assisted run, Claude//). `refresh_npp_and_admin()` on `main` @ c904d6d, fresh SQLite DB,... [04:29:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:33:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [04:34:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:46:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:56:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:33:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [07:47:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:57:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:08:41] 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, 13Patch-For-Review: Set cloudvirt fleet to 'performance' governor - https://phabricator.wikimedia.org/T439729#12423850 (10fgiunchedi) >>! In T439729#12414633, @Peter wrote: > I compared the performance and it made our CI test about 20% faster overall! Tha... [08:09:03] 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, 13Patch-For-Review: Set cloudvirt fleet to 'performance' governor - https://phabricator.wikimedia.org/T439729#12423851 (10fgiunchedi) [08:09:10] 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, 13Patch-For-Review: Set cloudvirt fleet to 'performance' governor - https://phabricator.wikimedia.org/T439729#12423852 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi This is done! [08:13:54] 06tools-infrastructure-team, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Test Platform: Add more CPUs to quibble jobs - https://phabricator.wikimedia.org/T439808#12423861 (10fgiunchedi) Something else I wanted to add: with more instances I think a test with two executors instead... [08:19:00] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [08:19:05] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [08:21:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:24:00] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [08:24:05] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [08:25:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [08:26:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:28:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:33:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:05:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:10:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:11:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:13:53] 10Toolforge, 06tools-platform-team: [replica-cnf] The uwsgi service stops sometimes without errors - https://phabricator.wikimedia.org/T440948 (10dcaro) 03NEW [09:16:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:43:25] 10Toolforge, 06tools-platform-team: [maintain-dbusers] Allow self-rotating credentials - https://phabricator.wikimedia.org/T440952 (10dcaro) 03NEW [09:46:46] 10Toolforge, 06tools-platform-team: [maintain-dbusers] Allow self-rotating credentials - https://phabricator.wikimedia.org/T440952#12424184 (10dcaro) [10:12:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:22:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:29:07] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [maintain-dbusers] Allow self-rotating credentials - https://phabricator.wikimedia.org/T440952#12424399 (10dcaro) 05Open→03In progress [10:29:09] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [maintain-dbusers] Allow self-rotating credentials - https://phabricator.wikimedia.org/T440952#12424400 (10dcaro) p:05Triage→03High [10:30:24] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [maintain-dbusers] Allow self-rotating credentials - https://phabricator.wikimedia.org/T440952#12424419 (10dcaro) [10:30:54] 10Toolforge, 06tools-platform-team: [replica-cnf] The uwsgi service stops sometimes without errors - https://phabricator.wikimedia.org/T440948#12424421 (10dcaro) [10:43:26] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [replica_cnf,functional-tests] Run replica_cnf functional tests in lima-kilo with the rest of functional tests - https://phabricator.wikimedia.org/T369800#12424459 (10dcaro) >>! In T369800#10490208, @Raymond_Ndibe wrote: > yeaaaa I think there is a t... [10:57:37] 10Quarry, 06tools-platform-team: Quarry: Access Denied Error - https://phabricator.wikimedia.org/T440966 (10HideonRosie) 03NEW [10:59:29] 10Quarry, 06tools-platform-team: Quarry: Access Denied Error - https://phabricator.wikimedia.org/T440966#12424558 (10HideonRosie) p:05Triage→03High [11:30:07] FIRING: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [11:40:21] 06tools-infrastructure-team, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Test Platform: Add more CPUs to quibble jobs - https://phabricator.wikimedia.org/T439808#12424760 (10Peter) I'm thinking like this: when we move away from Jenkins and only use Zuul on K8, then it makes sense... [11:45:07] RESOLVED: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [11:46:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:51:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:52:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:57:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:59:07] FIRING: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [12:11:43] 10Toolforge, 10Tools, 06tools-platform-team: Mix'n'match: Temporary DB size increase to reduce DB size - https://phabricator.wikimedia.org/T440708#12424967 (10magnusmanske) Duplication process is ongoing. Meanwhile, I have fixed the `person_dates` table, could you try again: ` ALTER TABLE `person_dates` DRO... [12:29:17] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [12:35:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:36:46] 10Quarry, 06tools-platform-team: Quarry: Access Denied Error - https://phabricator.wikimedia.org/T440966#12425046 (10Urbanecm) Confirmed, it's not working for some reason :). [12:40:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:51:34] 10Quarry, 06tools-platform-team: [quarry] update the database password - https://phabricator.wikimedia.org/T440979 (10dcaro) 03NEW [12:52:37] 10Quarry, 06tools-platform-team: Quarry: Access Denied Error - https://phabricator.wikimedia.org/T440966#12425099 (10aputhin) 05Open→03In progress a:03dcaro [12:53:27] 10Quarry, 06tools-platform-team: [quarry] update the database password - https://phabricator.wikimedia.org/T440979#12425104 (10dcaro) [12:53:30] 10Quarry, 06tools-platform-team: Quarry: Access Denied Error - https://phabricator.wikimedia.org/T440966#12425103 (10dcaro) [12:53:46] 10Quarry, 06tools-platform-team: Quarry: Access Denied Error - https://phabricator.wikimedia.org/T440966#12425105 (10dcaro) >>! In T440966#12425046, @Urbanecm wrote: > Confirmed, it's not working for some reason :). Should be working now, can you retry? [12:55:08] 10Quarry, 06tools-platform-team: [quarry] update the database password - https://phabricator.wikimedia.org/T440979#12425112 (10github-toolforge-bot) david-caro opened https://github.com/toolforge/quarry/pull/105 [12:55:12] 10Quarry, 06tools-platform-team: [quarry] update the database password - https://phabricator.wikimedia.org/T440979#12425113 (10dcaro) p:05Triage→03High [12:55:25] 10Quarry, 06tools-platform-team: [quarry] update the database password - https://phabricator.wikimedia.org/T440979#12425116 (10dcaro) a:03dcaro [13:04:15] 10Quarry, 06tools-platform-team: Quarry: Access Denied Error - https://phabricator.wikimedia.org/T440966#12425150 (10Urbanecm) It works, wonderful! Thank you! :) [13:23:31] 10Cloud-VPS (Debian Bullseye Deprecation), 10SRE-swift-storage: Migrate swift away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435783#12425238 (10Eevans) >>! In T435783#12385202, @komla wrote: > Will you be able to meet a deadline of October 31st? No, but I went ahead and dele... [13:37:40] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 07Sustainability (Incident Followup): wmcs-backup should not delete backups before taking the new one - https://phabricator.wikimedia.org/T429002#12425336 (10Andrew) a:03Andrew [13:42:00] RESOLVED: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [13:53:41] 06cloud-services-team, 10Data-Services, 10Toolforge, 06tools-platform-team: Move replica_cnf_api out of the Puppet repo - https://phabricator.wikimedia.org/T340754#12425392 (10dcaro) p:05Low→03High a:03dcaro [13:53:55] 06cloud-services-team, 10Data-Services, 10Toolforge, 06tools-platform-team: Move replica_cnf_api out of the Puppet repo - https://phabricator.wikimedia.org/T340754#12425398 (10dcaro) 05Open→03In progress [13:55:37] 06tools-infrastructure-team: Enable perf_counters on cloudceph - https://phabricator.wikimedia.org/T440686#12425405 (10Andrew) Sounds fine to me. [14:16:29] 10Tool-commons-delinquent: Tool-commons-delinquent: HTTP Code 500 error - https://phabricator.wikimedia.org/T437416#12425565 (10SD_hehua) Emmm, can anyone deal with this problem? [14:35:51] 10Tool-wsindex, 10Wikisource Reader App (Android): Add descriptions for books where Wikipedia articles are available [API] - https://phabricator.wikimedia.org/T430400#12425657 (10RCodes) Hi everyone! I'm Rabuma and I'm applying for the December 2026 Outreachy internship. I'd like to work on this task and help... [15:24:00] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Create code change and redeploy tutorial - https://phabricator.wikimedia.org/T437900#12425913 (10aputhin) p:05Triage→03Low [15:39:08] 06tools-infrastructure-team, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Test Platform: Add more CPUs to quibble jobs - https://phabricator.wikimedia.org/T439808#12426046 (10fgiunchedi) >>! In T439808#12424760, @Peter wrote: > I'm thinking like this: when we move away from Jenkin... [15:40:27] 10Toolforge, 10Tools, 06tools-platform-team: Mix'n'match: Temporary DB size increase to reduce DB size - https://phabricator.wikimedia.org/T440708#12426055 (10fnegri) Done: `lang=mysql MariaDB [s51434__mixnmatch_p]> ALTER TABLE `person_dates` DROP FOREIGN KEY `person_dates_ibfk_1`; Query OK, 26515874 rows a... [15:47:45] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Clean up users and manage as code - https://phabricator.wikimedia.org/T367772#12426132 (10fnegri) The outdated grants were removed in {T409563}. Remaining users are: `lang=mysql MariaDB [(none)]> SELECT user,host FROM mysql.user WHERE use... [15:49:07] RESOLVED: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [15:57:58] (03open) 10dcaro: First commit [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/1 [15:58:14] (03update) 10dcaro: First commit [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/1 (https://phabricator.wikimedia.org/T340754) [15:58:29] (03update) 10dcaro: First commit [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/1 (https://phabricator.wikimedia.org/T340754) [15:59:17] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [16:01:26] (03update) 10dcaro: First commit [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/1 (https://phabricator.wikimedia.org/T340754) [16:07:42] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Create scheduled job deployment tutorial - https://phabricator.wikimedia.org/T437901#12426273 (10aputhin) p:05Triage→03Low [16:10:27] 10Cloud-VPS, 06tools-infrastructure-team, 07Security: spectre and MDS vulnerabilities on cloud-vps? - https://phabricator.wikimedia.org/T440615#12426294 (10Andrew) Here's what a running VM reports: ` Vulnerabilities: Gather data sampling: Unknown: Dependent on hypervisor status Indirec... [16:24:37] (03update) 10dcaro: First commit [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/1 (https://phabricator.wikimedia.org/T340754) [16:31:21] (03update) 10dcaro: First commit [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/1 (https://phabricator.wikimedia.org/T340754) [16:36:18] 10Tool-delintbot: Ensure empty nowiki tags are present as file options - https://phabricator.wikimedia.org/T441008 (10Redmin) 03NEW [16:36:47] 10Tool-delintbot: Ensure empty nowiki tags are present as file options - https://phabricator.wikimedia.org/T441008#12426521 (10Redmin) p:05Triage→03High [16:37:08] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Keystone logs no longer appearing in logstash - https://phabricator.wikimedia.org/T421911#12426538 (10Andrew) 05Open→03Resolved [16:37:10] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: openstack: keystone may be failing to add users to the bastion project in Keystone and/or LDAP - https://phabricator.wikimedia.org/T379550#12426540 (10Andrew) [16:40:53] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.34 - https://phabricator.wikimedia.org/T414595#12426553 (10Andrew) [16:43:40] (03update) 10dcaro: First commit [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/1 (https://phabricator.wikimedia.org/T340754) [16:45:49] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12426570 (10Amdrel) >>! In T428062#12354238, @JSengupta-WMF wrote: > @Doc_James Here is a proposal for the new section on the right side toolbar after discussin... [16:49:48] (03approved) 10fnegri: First commit [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/1 (https://phabricator.wikimedia.org/T340754) (owner: 10dcaro) [16:52:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:55:34] (03merge) 10dcaro: First commit [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/1 (https://phabricator.wikimedia.org/T340754) [16:57:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:04:29] 06cloud-services-team, 10Data-Services, 10Quarry, 06tools-platform-team: Quarry runners cannot authenticate to Wiki Replicas (s52788, error 1045) - https://phabricator.wikimedia.org/T441012 (10Niinarh7102026) 03NEW [17:07:11] 06cloud-services-team, 10Data-Services, 10Quarry, 06tools-platform-team: Quarry runners cannot authenticate to Wiki Replicas (s52788, error 1045) - https://phabricator.wikimedia.org/T441012#12426719 (10dcaro) a:03dcaro [17:08:38] 06cloud-services-team, 10Data-Services, 10Quarry, 06tools-platform-team: Quarry runners cannot authenticate to Wiki Replicas (s52788, error 1045) - https://phabricator.wikimedia.org/T441012#12426725 (10Niinarh7102026) Recovery update: query 110274 (`enwiki_p`, `SELECT 1`) completed successfully in 0.02 sec... [17:08:49] 06cloud-services-team, 10Data-Services, 10Quarry, 06tools-platform-team: Quarry runners cannot authenticate to Wiki Replicas (s52788, error 1045) - https://phabricator.wikimedia.org/T441012#12426726 (10dcaro) @Niinarh7102026 that query shows as running right now, are you still having issues? [17:09:41] 06cloud-services-team, 10Data-Services, 10Quarry, 06tools-platform-team: Quarry runners cannot authenticate to Wiki Replicas (s52788, error 1045) - https://phabricator.wikimedia.org/T441012#12426731 (10dcaro) 05Open→03Resolved I fixed the credentials a few hours ago yep, I'll close, but let me know... [17:18:00] 10Tool-wsindex, 06MediaWiki-Media-Platform-Team, 10Wikipedia-iOS-App, 06WikipediaApp, and 2 others: Update apps to support for the move from upload.wikimedia.org to thumb.wikimedia.org - https://phabricator.wikimedia.org/T434821#12426822 (10HNordeenWMF) 05Open→03Resolved a:03HNordeenWMF [17:41:46] (03Abandoned) 10Krinkle: hound: Upgrade from golang1.15 to golang1.19 [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1343614 (https://phabricator.wikimedia.org/T438777) (owner: 10Krinkle) [17:45:03] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance toolsbeta-nfs-3 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [18:02:05] 06cloud-services-team, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: Upgrade toolforge calico to v3.31 - https://phabricator.wikimedia.org/T441022 (10Andrew) 03NEW [18:05:03] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance toolsbeta-nfs-3 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [18:19:22] 10VPS-project-Codesearch, 06Collaboration-Services, 13Patch-For-Review: Graduate codesearch to production - https://phabricator.wikimedia.org/T268199#12427156 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=649e3600-aefd-4fa5-a03b-11e14f4ef835) set by dzahn@cumin1004 for 7 days, 0:00:00 o... [18:23:16] (03PS1) 10Dzahn: frontend: build and publish a frontend image with blubber [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1354853 (https://phabricator.wikimedia.org/T268199) [18:25:22] (03CR) 10CI reject: [V:04-1] frontend: build and publish a frontend image with blubber [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1354853 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [20:36:02] 10Tool-wsindex, 10Wikisource Reader App (Android): Add descriptions for books where Wikipedia articles are available [API] - https://phabricator.wikimedia.org/T430400#12427767 (10RCodes) Before implementing it, could I clarify a few details? 1. Should the API field be called `description`? 2. Should we use th... [20:49:42] (03open) 10andrew: Update Calico to 3.31.7 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/23 (https://phabricator.wikimedia.org/T441022) [20:58:48] (03PS2) 10Dzahn: frontend: build and publish a frontend image with blubber [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1354853 (https://phabricator.wikimedia.org/T268199) [20:58:59] 10Tool-wsindex, 10Wikisource Reader App (Android): Add summary statistics to analytics page - https://phabricator.wikimedia.org/T430399#12427861 (10RCodes) Hi i am Rabuma! I'd like to work on this task as part of my Outreachy contribution. I'll add the total number of books, languages, and authors to the anal... [21:00:09] (03CR) 10CI reject: [V:04-1] frontend: build and publish a frontend image with blubber [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1354853 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [21:16:07] (03PS3) 10Dzahn: frontend: build and publish a frontend image with blubber [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1354853 (https://phabricator.wikimedia.org/T268199) [21:40:26] 10Tool-lingua-datasets, 10Toolforge (Quota-requests), 06tools-platform-team: Setup Lingua-datasets.toolforge.org with 400GB - https://phabricator.wikimedia.org/T440689#12427957 (10Yug) Hello @fgiunchedi , my repository tree is like this so far : [21:57:59] 10Tool-lingua-datasets, 10Toolforge (Quota-requests), 06tools-platform-team: Setup Lingua-datasets.toolforge.org with 400GB - https://phabricator.wikimedia.org/T440689#12428026 (10Yug) [22:03:23] (03PS4) 10Dzahn: frontend: build and publish a frontend image with blubber [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1354853 (https://phabricator.wikimedia.org/T268199) [22:18:11] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 13Patch-For-Review: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12428116 (10Southparkfan) The critical steps of... [22:19:47] (03PS1) 10Dzahn: frontend: make the public Hound URL configurable [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1354897 (https://phabricator.wikimedia.org/T268199) [23:11:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:16:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:23:24] 10Tool-Pageviews: Add job queue system to allow for longer-running processing - https://phabricator.wikimedia.org/T441065 (10MusikAnimal) 03NEW [23:40:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:45:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:52:34] FIRING: DiskSpace: Disk space cloudcontrol1011:9100:/ 6.952% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudcontrol1011 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [23:57:34] RESOLVED: DiskSpace: Disk space cloudcontrol1011:9100:/ 6.931% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudcontrol1011 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace