[00:47:00] 10Tool-paulina: Add Wikisource links to works (feature request) - https://phabricator.wikimedia.org/T376570#10551400 (10Pepe_piton) 05Open→03Resolved The issue has been resolved as described above, with this commit: https://gitlab.wikimedia.org/toolforge-repos/paulina/-/commit/01cfa7bbac81ab76d8e0516e401... [08:35:59] 10tool-wscontest: Move wscontest source code to GitLab - https://phabricator.wikimedia.org/T324410#10551807 (10Samwilson) [08:53:06] FIRING: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:58:06] RESOLVED: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:49:27] 06cloud-services-team, 10Toolforge, 06Design-Research, 07Design: Toolforge UI: Publish newcomer experience and recruitment survey - https://phabricator.wikimedia.org/T381266#10551939 (10Sarai-WMF) [09:49:34] 06cloud-services-team, 10Toolforge, 06Design-Research, 07Design: Toolforge UI: Publish newcomer experience and recruitment survey - https://phabricator.wikimedia.org/T381266#10551940 (10Sarai-WMF) 05Open→03Resolved [09:51:39] FIRING: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:56:39] RESOLVED: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:41:44] 10Tool-refill: reFill fails to load, errors out with "Access to storage is not allowed from this context" - https://phabricator.wikimedia.org/T386446 (10Jpatokal) 03NEW [10:56:39] 10Tool-refill: reFill fails to load, errors out with "Access to storage is not allowed from this context" - https://phabricator.wikimedia.org/T386446#10552053 (10Curb_Safe_Charmer) a:03Curb_Safe_Charmer I can reproduce this in Chrome, but not in incognito mode. The code the error occurs in is part of the LastP... [10:57:59] 10Tool-refill: reFill fails to load, errors out with "Access to storage is not allowed from this context" - https://phabricator.wikimedia.org/T386446#10552061 (10Curb_Safe_Charmer) p:05Triage→03Low [11:34:23] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge: [toolsdb] ToolsToolsDBReplicationLagIsTooHigh - 2025-02-12 - https://phabricator.wikimedia.org/T386240#10552182 (10fnegri) Replication caught up in about 2 hours, and seems to be working fine. {F58398454} [11:37:06] FIRING: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:42:06] RESOLVED: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:25:09] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Modernize openstack rbac - https://phabricator.wikimedia.org/T330759#10552510 (10Andrew) The above patch didn't break any of my known tests. I guess now we wait a bit and see if anyone runs into codfw1dev access surprises. [15:00:44] 06cloud-services-team, 10Toolforge, 10Tools: Flickr blocking image requests from Toolforge k8s, breaking multiple tools - https://phabricator.wikimedia.org/T384468#10552651 (10Andrew) I would very much like to follow up with flickr support about this. Can I please get some feedback? [15:12:03] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge: [toolsdb] ToolsToolsDBReplicationLagIsTooHigh - 2025-02-12 - https://phabricator.wikimedia.org/T386240#10552686 (10Ladsgroup) If the storage is over the network (you mentioned ceph) it's going to be quite a pain to maintain :( Maybe that can change or we... [15:20:13] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge: [toolsdb] ToolsToolsDBReplicationLagIsTooHigh - 2025-02-12 - https://phabricator.wikimedia.org/T386240#10552712 (10fnegri) Storage have been over Ceph for a few years and it's mostly worked fine, but I agree we should reduce the write volume. I think the... [15:22:31] FIRING: ToolsToolsDBReplicationLagIsTooHigh: ToolsDB replication on tools-db-5 is lagging behind the primary, the current lag is 3666 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationLagIsTooHigh [15:30:47] (03PS5) 10Andrew Bogott: Get openstack project list from keystone [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/1093997 (https://phabricator.wikimedia.org/T379030) [15:43:00] 06cloud-services-team, 10Toolforge, 07Epic: loki into lima-kilo - https://phabricator.wikimedia.org/T386480 (10rook) 03NEW [16:15:06] (03CR) 10Andrew Bogott: "briefly confused about 'reviewer' vs 'cc'" [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/1093997 (https://phabricator.wikimedia.org/T379030) (owner: 10Andrew Bogott) [16:43:01] RESOLVED: ToolsToolsDBReplicationLagIsTooHigh: ToolsDB replication on tools-db-5 is lagging behind the primary, the current lag is 3634 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationLagIsTooHigh [17:31:38] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge: [toolsdb] ToolsToolsDBReplicationLagIsTooHigh - 2025-02-12 - https://phabricator.wikimedia.org/T386240#10553226 (10fnegri) I tried [capturing backtraces](https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/#getting-full-backtraces-for... [18:07:17] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge: [toolsdb] ToolsToolsDBReplicationLagIsTooHigh - 2025-02-12 - https://phabricator.wikimedia.org/T386240#10553336 (10fnegri) It did get stuck again, and running the same `gdb` command got it unstuck again. `SHOW PROCESSLIST` before running `gdb`: {P73480} [18:46:36] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: openstack: wmfkeystonehooks: project ids rather than names are being used in LDAP group creation - https://phabricator.wikimedia.org/T379030#10553421 (10Andrew) I still haven't decided what to do about this. https://gerrit.wikimedia.org/r/c/labs/tools/... [20:37:53] 06cloud-services-team, 10Toolforge, 07Epic: loki into lima-kilo - https://phabricator.wikimedia.org/T386480#10553730 (10rook) I believe https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/tree/T386480?ref_type=heads and https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/tree/T3... [20:57:49] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: openstack: wmfkeystonehooks: project ids rather than names are being used in LDAP group creation - https://phabricator.wikimedia.org/T379030#10553804 (10bd808) >>! In {T274268}, @bstorm wrote: > This should include trying to make it so that we can stil... [23:28:16] FIRING: ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:33:16] RESOLVED: [3x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown