[02:27:57] 10Cloud-VPS (Project-requests), 10VPS-project-voterlists: Request creation of voterlists VPS project - https://phabricator.wikimedia.org/T399418#11827926 (10SD0001) Creating a web frontend is T404250. Generating lists takes 3 hours, so the current way to use it is by shell. Let me know if you want access. [02:39:48] (03open) 10raymond-ndibe: testing fix [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/287 [02:47:11] (03update) 10raymond-ndibe: testing fix [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/287 [03:12:55] 06tools-platform-team: jobs not getting loaded properly - https://phabricator.wikimedia.org/T423544 (10Raymond_Ndibe) 03NEW [03:14:11] 06tools-platform-team: jobs not getting loaded properly - https://phabricator.wikimedia.org/T423544#11827986 (10Raymond_Ndibe) [03:16:24] (03update) 10raymond-ndibe: testing fix [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/287 (https://phabricator.wikimedia.org/T423544) [03:17:18] (03update) 10raymond-ndibe: Draft: fix jobs load bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/287 (https://phabricator.wikimedia.org/T423544) [04:29:34] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 6.994% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [04:34:34] RESOLVED: DiskSpace: Disk space cloudbackup1004:9100:/srv 6.991% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [05:25:52] 06cloud-services-team, 10Data-Services, 06DBA, 10decommission-hardware, 13Patch-For-Review: decommission clouddb1019.eqiad.wmnet - https://phabricator.wikimedia.org/T423151#11828081 (10Marostegui) [05:26:36] 10Cloud-VPS (Project-requests), 10VPS-project-voterlists: Request creation of voterlists VPS project - https://phabricator.wikimedia.org/T399418#11828082 (10Novem_Linguae) Hey SD0001, Sure, feel free to add me for access to the wmcloud voter list generation tool. Thanks for offering. I would suggest addi... [05:28:53] 10Cloud-VPS (Project-requests), 10VPS-project-voterlists: Request creation of voterlists VPS project - https://phabricator.wikimedia.org/T399418#11828083 (10Novem_Linguae) Oh, and... * Create a maintenance script capable of importing the tool's output (1 file I assume?) into one SQL table somewhere [05:31:11] 06cloud-services-team, 10Data-Services, 06DBA, 10decommission-hardware: decommission clouddb1019.eqiad.wmnet - https://phabricator.wikimedia.org/T423151#11828091 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by marostegui@cumin1003 for hosts: `clouddb1019.eqiad.wmnet` - clouddb1019.eqiad... [05:33:08] 10VPS-project-voterlists, 06Moderator-Tools-Team, 10PageTriage: Switch global election workflow to use voterlists.wmcloud.org? - https://phabricator.wikimedia.org/T423547 (10Novem_Linguae) 03NEW [05:33:32] 10Cloud-VPS (Project-requests), 10VPS-project-voterlists: Request creation of voterlists VPS project - https://phabricator.wikimedia.org/T399418#11828106 (10Novem_Linguae) I went ahead and created {T423547} so I stop spamming this ticket :) [05:33:54] 06cloud-services-team, 10Data-Services, 06DBA, 06DC-Ops, and 2 others: decommission clouddb1019.eqiad.wmnet - https://phabricator.wikimedia.org/T423151#11828108 (10Marostegui) a:05Marostegui→03Jclark-ctr [05:34:07] 06cloud-services-team, 10Data-Services, 06DBA, 06DC-Ops, and 2 others: decommission clouddb1019.eqiad.wmnet - https://phabricator.wikimedia.org/T423151#11828114 (10Marostegui) This is ready for DC-Ops [05:34:20] 10VPS-project-voterlists, 10MediaWiki-extensions-SecurePoll, 06Product Safety and Integrity: Switch global election workflow to use voterlists.wmcloud.org? - https://phabricator.wikimedia.org/T423547#11828115 (10Novem_Linguae) [05:35:20] 06cloud-services-team, 10Data-Services, 06DBA, 06DC-Ops, and 2 others: decommission clouddb1019.eqiad.wmnet - https://phabricator.wikimedia.org/T423151#11828120 (10Marostegui) >>! In T423151#11828091, @ops-monitoring-bot wrote: > cookbooks.sre.hosts.decommission executed by marostegui@cumin1003 for hosts:... [06:22:31] 10Cloud-VPS (Quota-requests): Temporary instance increase for Bullseye servers deprecation - https://phabricator.wikimedia.org/T416803#11828159 (10YochayCO) I've deleted the old servers (a week ago, but now come to think of it I did not update here). It can be reduced back to 500GB (40GB decrease) Volume 20... [06:22:49] 10VPS-project-voterlists, 10MediaWiki-extensions-SecurePoll, 06Product Safety and Integrity: Switch global election workflow to use voterlists.wmcloud.org? - https://phabricator.wikimedia.org/T423547#11828161 (10Bugreporter) > Create a maintenance script capable of importing the tool's output Why we need a... [06:27:48] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [06:34:01] 10Toolforge, 06tools-platform-team: [Toolforge]: Automate daily count of successful SSH logins as a measure of user activity - https://phabricator.wikimedia.org/T423549 (10komla) 03NEW [06:39:46] 10VPS-project-voterlists, 10MediaWiki-extensions-SecurePoll, 06Product Safety and Integrity: Switch global election workflow to use voterlists.wmcloud.org? - https://phabricator.wikimedia.org/T423547#11828178 (10Novem_Linguae) You mean by pasting the voter list into Voter Eligibility screen? Would probably o... [06:41:19] 10Toolforge, 06tools-platform-team: [Toolforge]: Automate daily count of successful SSH logins as a measure of user activity - https://phabricator.wikimedia.org/T423549#11828181 (10komla) [06:44:01] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Engineering-Radar, 06DBA: Re-run maintainviews on all clouddb* and an-redacteddb1001.eqiad.wmnet - https://phabricator.wikimedia.org/T422459#11828202 (10Marostegui) [06:47:56] 10VPS-project-voterlists, 10MediaWiki-extensions-SecurePoll, 06Product Safety and Integrity: Switch global election workflow to use voterlists.wmcloud.org? - https://phabricator.wikimedia.org/T423547#11828212 (10Bugreporter) > You mean by pasting the voter list into Voter Eligibility screen? Instead the tool... [07:06:39] 10Toolforge, 06tools-platform-team: jobs not getting loaded properly - https://phabricator.wikimedia.org/T423544#11828224 (10taavi) [07:44:00] 06cloud-services-team, 10Data-Services, 06DBA: Table/View imagelinks in commonswiki_p throws an error - https://phabricator.wikimedia.org/T423550#11828328 (10JJMC89) [08:03:43] 10Tool-curator: Curator: more recent image sequences cannot be retrieved - https://phabricator.wikimedia.org/T423321#11828387 (10PantheraLeo1359531) Interesting, it looks like we have some days of delay... It may depend on the sequence size and date of upload how fast the sequence can be retrieved [08:04:01] 10Tool-curator: Curator: more recent image sequences cannot be retrieved - https://phabricator.wikimedia.org/T423321#11828388 (10PantheraLeo1359531) 05Open→03Resolved [08:04:39] 10Tool-curator: Curator: more recent image sequences cannot be retrieved - https://phabricator.wikimedia.org/T423321#11828390 (10PantheraLeo1359531) I see this issue as resolved, because the circumstances are known and it may be an issue on Mapillary's side... [08:08:51] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Improve linting - unify example validation - https://phabricator.wikimedia.org/T423411#11828395 (10KBach) [08:20:26] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Improve linting - unify example validation - https://phabricator.wikimedia.org/T423411#11828435 (10KBach) [08:21:35] 10Tool-wikimedia-attribution: Update data links to the Attribution API Beta - https://phabricator.wikimedia.org/T421385#11828437 (10Sarai-WMF) New copy proposals were documented and are being reviewed in [[ https://docs.google.com/spreadsheets/d/13AySMpuK3-ESqbHTrWtBGiJOqAPGnBrV1v8UQwgeAbM/edit?gid=0#gid=0 | thi... [08:22:52] 10Tools: CSP violations with known domains in the blocked-uri are not collected by csp-report - https://phabricator.wikimedia.org/T422916#11828441 (10fgiunchedi) That's fair re: user confusion concerns. From my SRE POV I was surprised to find that the CSP report url we announce filters the feed of legitimate, al... [09:04:24] 10VPS-project-voterlists, 10MediaWiki-extensions-SecurePoll, 06Product Safety and Integrity: Switch global election workflow to use voterlists.wmcloud.org? - https://phabricator.wikimedia.org/T423547#11828544 (10SD0001) > Create a maintenance script capable of importing the tool's output (1 file I assume?)... [09:06:04] 06cloud-services-team: Consider splitting tf-infra-test into multiple "test suites" - https://phabricator.wikimedia.org/T423560 (10fgiunchedi) 03NEW [09:06:44] 10VPS-project-voterlists, 10MediaWiki-extensions-SecurePoll, 06Product Safety and Integrity: Switch global election workflow to use voterlists.wmcloud.org? - https://phabricator.wikimedia.org/T423547#11828581 (10Novem_Linguae) [09:07:08] 06cloud-services-team: Consider splitting tf-infra-test into multiple "test suites" - https://phabricator.wikimedia.org/T423560#11828583 (10fgiunchedi) [09:19:07] FIRING: ToolsDBReplicationLagIsTooHigh: ToolsDB replication is lagging - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBReplicationLagIsTooHigh [09:28:26] 10Tool-refill, 07Regression: regression: diff feature is broken - https://phabricator.wikimedia.org/T423566 (10Novem_Linguae) 03NEW [09:29:07] RESOLVED: ToolsDBReplicationLagIsTooHigh: ToolsDB replication is lagging - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBReplicationLagIsTooHigh [09:36:23] 10Tool-wikimedia-attribution: [WAF·V1] Update site with reviewed content - https://phabricator.wikimedia.org/T416982#11828721 (10Sarai-WMF) [09:36:40] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24502903296 (https://github.com/cluebotng/component-configs/commits/09046ab7e1f4831d3acaf275abbb61a756291a24) [09:36:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [09:38:33] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24502989226 (https://github.com/cluebotng/component-configs/commits/d1caa7e619b2f6df59cbdb32d5bdf150ceacfa19) [09:38:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [10:05:02] 10Tool-wikimedia-attribution: Document referrer recommendations for Participation CTA - https://phabricator.wikimedia.org/T423567 (10Sarai-WMF) 03NEW [10:08:05] 10Tool-wikimedia-attribution: Document referrer recommendations for Participation CTA - https://phabricator.wikimedia.org/T423567#11828774 (10Sarai-WMF) [10:17:35] 10Cloud-VPS (Project-requests), 10VPS-project-voterlists: Request creation of voterlists VPS project - https://phabricator.wikimedia.org/T399418#11828815 (10SD0001) I've added access for you. @jrbs already has it. [10:18:59] 10VPS-project-voterlists, 10MediaWiki-extensions-SecurePoll, 06Product Safety and Integrity: Switch global election workflow to use voterlists.wmcloud.org? - https://phabricator.wikimedia.org/T423547#11828822 (10SD0001) > Grant Joe access to voterlists.wmcloud.org shell Joe already has the access. > Joe... [10:21:12] 10VPS-project-voterlists, 10MediaWiki-extensions-SecurePoll, 06Product Safety and Integrity: Switch global election workflow to use voterlists.wmcloud.org? - https://phabricator.wikimedia.org/T423547#11828825 (10Novem_Linguae) [10:25:28] 06cloud-services-team, 10Toolforge: Audit tools memory requests vs actual usage - https://phabricator.wikimedia.org/T420565#11828840 (10fnegri) I'm ok with testing a 128mb default request, instead of the current 256mb. Worst case we can revert it if we see this causes any issues, the only one I can think of is... [10:28:03] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [10:29:43] 06cloud-services-team, 10Toolforge: Audit tools memory requests vs actual usage - https://phabricator.wikimedia.org/T420565#11828853 (10fnegri) Note that for jobs, there is only one value `JOB_DEFAULT_MEMORY = "512Mi"` that is used as a memory limit, the memory request is set to limit/2. If we want to keep the... [10:30:47] 06cloud-services-team, 10Toolforge, 06tools-platform-team: Audit tools memory requests vs actual usage - https://phabricator.wikimedia.org/T420565#11828860 (10fnegri) [10:35:29] 10Tool-wikimedia-attribution: Document referrer recommendations for Participation CTA - https://phabricator.wikimedia.org/T423567#11828874 (10Sarai-WMF) Hey @Khantstop! We'd really appreciate your help crafting this recommendation, which we're hoping to include as a best implementation practice for Participation... [10:39:53] 10Tool-refill, 07Regression: regression: diff feature is broken - https://phabricator.wikimedia.org/T423566#11828878 (10Novem_Linguae) Things I've tried so far... - Look in backup and get some source code that might have the bug fixed in that source code. I tried the source code from /versions/test/web and `n... [10:54:12] 10Tool-wikimedia-attribution: Define attribution recommendations for Kahoot! - https://phabricator.wikimedia.org/T423571 (10Sarai-WMF) 03NEW [11:05:51] (03open) 10vriaa: fix: make text runs viewport-specific [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/56 [11:06:25] (03merge) 10vriaa: fix: make text runs viewport-specific [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/56 [11:09:23] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24506838480 (https://github.com/cluebotng/component-configs/commits/db30680b466b66aed23b2f87989cbe2e9fd90511) [11:09:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [12:41:59] !log tools.cluebotng-monitoring Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24510746155 (https://github.com/cluebotng/component-configs/commits/e927788672bbd71dc8a52dc4b34db54abe3995e9) [12:42:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [13:14:43] 10Cloud-Services, 06Infrastructure-Foundations, 06SRE: Adjust WMCS Gitlab CI/CD repo to stop using mirrors.wikimedia.org - https://phabricator.wikimedia.org/T423596 (10Jdforrester-WMF) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wiki... [13:38:04] 10Cloud-Services, 06SRE: Migrate our use of osbpo away from mirrors.wikimedia.org - https://phabricator.wikimedia.org/T423598 (10MoritzMuehlenhoff) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and repl... [13:44:31] 10Cloud-Services, 06SRE: Migrate our use of osbpo away from mirrors.wikimedia.org - https://phabricator.wikimedia.org/T423598#11829648 (10Andrew) a:03Andrew [13:44:55] 10Cloud-VPS, 06tools-infrastructure-team: Migrate our use of osbpo away from mirrors.wikimedia.org - https://phabricator.wikimedia.org/T423598#11829651 (10taavi) [14:15:03] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: Do not share shm files for openstack uwsgi processes - https://phabricator.wikimedia.org/T423378#11829779 (10fgiunchedi) 05Open→03Resolved This is completed [14:15:16] (03open) 10raymond-ndibe: core.py: skip one-off jobs when updating storage [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/288 (https://phabricator.wikimedia.org/T423544) [14:15:35] (03CR) 10Jforrester: [C:03+1] "Please do merge this. :-)" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1268246 (owner: 10Libraryupgrader) [14:16:42] (03update) 10raymond-ndibe: core.py: skip one-off jobs when updating storage [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/288 (https://phabricator.wikimedia.org/T423544) [14:21:47] (03CR) 10Novem Linguae: [C:03+1] build: Updating dependencies [labs/xtools] - 10https://gerrit.wikimedia.org/r/1268246 (owner: 10Libraryupgrader) [14:28:01] (03update) 10raymond-ndibe: Draft: fix jobs load bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/287 (https://phabricator.wikimedia.org/T423544) [14:28:03] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [14:33:56] 06cloud-services-team: Consider splitting tf-infra-test into multiple "test suites" - https://phabricator.wikimedia.org/T423560#11829974 (10fnegri) > This gave me the idea of having different tf suites testing different parts of the infrastructure. There are two ways we could do this: 1. split the tests in the... [14:37:48] RESOLVED: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [14:40:18] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: jobs not getting loaded properly - https://phabricator.wikimedia.org/T423544#11830048 (10Raymond_Ndibe) 05Open→03In progress [14:45:23] 06cloud-services-team, 10Toolforge: Cannot contact service with port from buildservice - https://phabricator.wikimedia.org/T423610 (10magnusmanske) 03NEW [14:54:54] 06cloud-services-team, 10Toolforge: Cannot contact service with port from buildservice - https://phabricator.wikimedia.org/T423610#11830170 (10magnusmanske) Ah, I understand this **only** works from within //other// jobs, not from the normal shell, which is a pity. Anyhow, I tried from a sehll webservice, and... [14:57:01] 06cloud-services-team, 10Toolforge: Cannot contact service with port from buildservice - https://phabricator.wikimedia.org/T423610#11830180 (10magnusmanske) If I change the manual `https://` to `http://` it works. These things should be clarified in the manual IMHO. Otherwise, this can be closed, works-for-me. [14:57:05] 06cloud-services-team, 10Toolforge: Cannot contact service with port from buildservice - https://phabricator.wikimedia.org/T423610#11830183 (10taavi) That seems to indicate that you're trying to talk HTTPS to a service that's only listening on plaintext? [14:58:49] 10Tool-paulina: Major improvements to country page - https://phabricator.wikimedia.org/T370179#11830196 (10Pepe_piton) [14:59:00] 06cloud-services-team, 10Toolforge: Cannot contact service with port from buildservice - https://phabricator.wikimedia.org/T423610#11830198 (10taavi) 05Open→03Invalid Clarified the docs with https://wikitech.wikimedia.org/w/index.php?diff=2403068. [14:59:52] 06cloud-services-team, 10Toolforge: Cannot contact service with port from buildservice - https://phabricator.wikimedia.org/T423610#11830202 (10magnusmanske) thanks! [15:54:20] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Improve linting - unify example validation - https://phabricator.wikimedia.org/T423411#11830493 (10HCoplin-WMF) Notes from estimation: * Should this delete the old rules? * Part of this task is determining the restructuring; some rules might... [15:57:40] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#11830510 (10fnegri) @Marostegui given clouddb1019 is dead ({T422813}) and we won't be getting its replacement clouddb1032 for a while, I was thi... [16:02:41] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#11830519 (10Marostegui) I can do that but I'll have to stop that host to clone the new one, would that be okay for the service? [16:11:15] 10Tool-paulina: Customize the work page according to the type of work - https://phabricator.wikimedia.org/T370166#11830571 (10Pepe_piton) [16:12:52] 10Tool-paulina: Improve the Paulina tool through different tasks during Wikimedia Hackathon 2026 - https://phabricator.wikimedia.org/T423625 (10Pepe_piton) 03NEW [16:21:38] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#11830665 (10fnegri) > I can do that but I'll have to stop that host to clone the new one, would that be okay for the service? Good point, I for... [16:50:31] 06cloud-services-team, 10Cloud-VPS: [tofuinfratest] Magnum cluster creation is failing - https://phabricator.wikimedia.org/T423393#11830773 (10fnegri) The Magnum cluster was created successfully today! I would wait for a couple days to see if it's consistently fixed. [16:58:05] 10Tools: CSP violations with known domains in the blocked-uri are not collected by csp-report - https://phabricator.wikimedia.org/T422916#11830798 (10bd808) >>! In T422916#11828441, @fgiunchedi wrote: > That's fair re: user confusion concerns. From my SRE POV I was surprised to find that the CSP report url we an... [17:16:28] 10Tool-paulina: Link to Cradle to create new item if no results - https://phabricator.wikimedia.org/T370223#11830837 (10Pepe_piton) This feature is not necessary anymore, since the new Paulina editing features allow users to create items from within Paulina. Anyway, I don't close it until we add the links to cre... [17:20:31] 10Tool-paulina: Link to create new item if no results - https://phabricator.wikimedia.org/T370223#11830855 (10Pepe_piton) [17:43:57] 10Tool-paulina: Translate Paulina into more languages and improve existing translations - https://phabricator.wikimedia.org/T392681#11830946 (10Pepe_piton) [17:46:22] 10Tool-paulina: Improve the Paulina tool through different tasks during Wikimedia Hackathon 2026 - https://phabricator.wikimedia.org/T423625#11830961 (10Pepe_piton) [17:46:45] 10Tool-paulina: Translate Paulina into more languages and improve existing translations - https://phabricator.wikimedia.org/T392681#11830966 (10Pepe_piton) 05Resolved→03Open [18:07:01] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24526007498 (https://github.com/cluebotng/component-configs/commits/37d3288f49b6baec42eb13403fb004de8b7e6271) [18:07:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [18:07:33] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24526007472 (https://github.com/cluebotng/component-configs/commits/37d3288f49b6baec42eb13403fb004de8b7e6271) [18:07:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [18:08:48] (03CR) 10Akoopal: [C:03+2] "all make sense, changes look good." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1271920 (owner: 10Lokal Profil) [18:10:24] 10Tool-paulina, 03Wikimedia-Hackathon-2026: Improve the Paulina tool through different tasks during Wikimedia Hackathon 2026 - https://phabricator.wikimedia.org/T423625#11831062 (10Pepe_piton) [18:10:56] (03Merged) 10jenkins-bot: Avoid fill_table_monuments_all timeout on large monuments_all [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1271920 (owner: 10Lokal Profil) [18:11:00] 10Tool-paulina, 03Wikimedia-Hackathon-2026: Wikimedia Hackathon 2026: Improve the Paulina tool through different tasks - https://phabricator.wikimedia.org/T423625#11831076 (10Pepe_piton) [18:14:14] 10Cloud-VPS, 06tools-infrastructure-team: Migrate our use of osbpo away from mirrors.wikimedia.org - https://phabricator.wikimedia.org/T423598#11831095 (10Andrew) We can get these same packages from osbpo.debian.net -- probably there's a bit of proxy config needed to make that work though. [18:29:10] (03CR) 10Akoopal: [C:03+2] "All make sense, paid attention to the parameters are still in the right order." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1271919 (owner: 10Lokal Profil) [18:31:16] (03Merged) 10jenkins-bot: Port cl_to lookups to linktarget join [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1271919 (owner: 10Lokal Profil) [18:35:29] (03CR) 10Akoopal: [C:03+2] "Bit more difficult query to understand, but all looks good." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1271928 (owner: 10Lokal Profil) [18:37:28] (03Merged) 10jenkins-bot: Port ContestStatsBuilder cl_to lookups to linktarget join [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1271928 (owner: 10Lokal Profil) [19:06:23] (03CR) 10Akoopal: [C:03+2] "makes sense to have clean logfiles." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1271933 (owner: 10Lokal Profil) [19:08:22] (03Merged) 10jenkins-bot: Silence curl progress meter and drop response body in create_toolforge_job [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1271933 (owner: 10Lokal Profil) [19:54:21] (03CR) 10Akoopal: [C:03+2] "Indeed varchar in mysql is in characters, not in bytes. So overall this makes sense, code changes looks good." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1271900 (owner: 10Lokal Profil) [19:56:49] (03Merged) 10jenkins-bot: Truncate muni on character boundary to avoid invalid UTF-8 [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1271900 (owner: 10Lokal Profil) [19:58:21] 06cloud-services-team, 10Toolforge: Toolforge reverse proxies stopped working - https://phabricator.wikimedia.org/T423652 (10GPSLeo) 03NEW [20:18:15] 06cloud-services-team, 10Toolforge: Toolforge reverse proxies stopped working - https://phabricator.wikimedia.org/T423652#11831542 (10JJMC89) 05Open→03Invalid This is intentional. T392356 I believe using the [[ https://wikitech.wikimedia.org/wiki/Tool:Containers#Reverse_proxy_container | reverse proxy... [20:22:52] 06cloud-services-team, 10Toolforge: Toolforge reverse proxies stopped working - https://phabricator.wikimedia.org/T423652#11831558 (10bd808) 05Invalid→03Open [20:23:46] 06cloud-services-team, 10Toolforge: uploadmap reverse proxy broken because of paltform changes - https://phabricator.wikimedia.org/T423652#11831562 (10bd808) [20:23:57] 06cloud-services-team, 10Toolforge: uploadmap reverse proxy broken because of platform changes - https://phabricator.wikimedia.org/T423652#11831563 (10bd808) [20:35:00] 06cloud-services-team, 10Toolforge, 07Documentation: uploadmap reverse proxy broken because of platform changes - https://phabricator.wikimedia.org/T423652#11831629 (10bd808) a:03bd808 I'm sorry that the change to the Toolforge ingress layer caught you unaware @GPSLeo. @JJMC89 is correct that the fix for t... [21:05:09] 06cloud-services-team, 10Toolforge, 07Documentation: uploadmap reverse proxy broken because of platform changes - https://phabricator.wikimedia.org/T423652#11831736 (10bd808) I have updated the instructions at https://wikitech.wikimedia.org/wiki/Tool:Containers#Reverse_proxy_container to use the newly requir... [21:10:12] (03open) 10aghirelli: fix(linter): fix five false positives on OAD example [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/15 [21:14:43] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge: Replace ingress-nginx before upstream EOL date - https://phabricator.wikimedia.org/T392356#11831771 (10bd808) >>! In T392356#11830166, @gerritbot wrote: > Change #1272714 **merged** by Majavah: > %%%[operations/puppet@production] P:toolforge::k8s::haproxy... [21:22:28] 06cloud-services-team, 10Toolforge, 07Documentation: uploadmap reverse proxy broken because of platform changes - https://phabricator.wikimedia.org/T423652#11831800 (10bd808) I think this should fix things for you @GPSLeo: `lang=shell-session $ toolforge envvars create RPROXY_CONFIG << '_EOF' --- routes: -... [21:24:41] 06cloud-services-team, 10Toolforge, 07Documentation: uploadmap reverse proxy broken because of platform changes - https://phabricator.wikimedia.org/T423652#11831806 (10bd808) I have applied the changes proposed in T423652#11831800 to the uplaodmap tool. @GPSLeo, can you check and see if everything is working... [21:38:01] (03update) 10apaskulin: Copyedits [toolforge-repos/wikimedia-attribution] - 10https://gitlab.wikimedia.org/toolforge-repos/wikimedia-attribution/-/merge_requests/2 [21:38:20] 10VPS-project-voterlists, 10MediaWiki-extensions-SecurePoll, 06Product Safety and Integrity: Switch global election workflow to use voterlists.wmcloud.org? - https://phabricator.wikimedia.org/T423547#11831845 (10jrbs) I'd like to try this for the U4C votes since we have the scripts as backup (T423528). We ca... [21:38:47] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24535245433 (https://github.com/cluebotng/component-configs/commits/6b66d29ba7f99387d0fed0292e21c235b3f9ab0b) [21:38:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [21:58:47] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24536002079 (https://github.com/cluebotng/component-configs/commits/46f94a272d26bcd13cf55d2a223ba74fde716d4e) [21:58:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [22:00:58] (03CR) 10Alien4444: [C:03+2] Localisation updates from https://translatewiki.net. [labs/xtools] - 10https://gerrit.wikimedia.org/r/1272692 (owner: 10L10n-bot) [22:18:42] (03CR) 10Alien4444: [C:03+2] build: Updating dependencies [labs/xtools] - 10https://gerrit.wikimedia.org/r/1268246 (owner: 10Libraryupgrader) [22:54:13] 10VPS-project-voterlists, 10MediaWiki-extensions-SecurePoll, 06Product Safety and Integrity: Switch global election workflow to use voterlists.wmcloud.org? - https://phabricator.wikimedia.org/T423547#11832012 (10Novem_Linguae) [23:45:56] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24539805608 (https://github.com/cluebotng/component-configs/commits/bef895ecceca2ac8c0bafe1e39e4680142771a39) [23:45:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL