[00:00:06] (03update) 10habs: T433052: implement RTL support via dynamic dir attribute [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/14 (owner: 10chimaifeanyi29) [00:01:47] (03approved) 10habs: T433052: implement RTL support via dynamic dir attribute [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/14 (owner: 10chimaifeanyi29) [00:02:03] (03merge) 10habs: T433052: implement RTL support via dynamic dir attribute [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/14 (owner: 10chimaifeanyi29) [00:08:02] (03update) 10habs: T437765: Add check for Category filter [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/15 (owner: 10chimaifeanyi29) [00:14:23] (03update) 10habs: T437765: Add check for Category filter [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/15 (owner: 10chimaifeanyi29) [00:14:28] (03approved) 10habs: Fix T436414: Prevent external link rendering in Wikinewsie AI text [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/12 (owner: 10akaza24) [00:14:32] (03unapproved) 10habs: Fix T436414: Prevent external link rendering in Wikinewsie AI text [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/12 (owner: 10akaza24) [00:14:38] (03approved) 10habs: T437765: Add check for Category filter [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/15 (owner: 10chimaifeanyi29) [00:14:46] (03merge) 10habs: T437765: Add check for Category filter [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/15 (owner: 10chimaifeanyi29) [00:15:43] (03update) 10habs: Add suggested headline button from AI recap [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/17 (owner: 10chimaifeanyi29) [00:38:09] (03approved) 10habs: Add suggested headline button from AI recap [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/17 (owner: 10chimaifeanyi29) [00:38:14] (03merge) 10habs: Add suggested headline button from AI recap [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/17 (owner: 10chimaifeanyi29) [00:40:11] 10Tool-wikinewsie: RTL Codex framework for Wikinewsie - https://phabricator.wikimedia.org/T433052#12362348 (10Habst) 05In progress→03Resolved [00:40:38] 10Tool-wikinewsie: Wikinewsie shouldn't use year categories for Wikipedia languages where they don't exist - https://phabricator.wikimedia.org/T437765#12362352 (10Habst) 05In progress→03Resolved [00:47:17] (03approved) 10habs: T437966: Add pagination to Recent Changes page [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/18 (https://phabricator.wikimedia.org/T437966) (owner: 10coderanimeshpathak04092007) [00:51:06] (03close) 10habs: T437966: Add pagination to Recent Changes page [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/18 (https://phabricator.wikimedia.org/T437966) (owner: 10coderanimeshpathak04092007) [00:51:51] 10Tool-wikinewsie, 07good first task, 13Patch-For-Review: Wikinewsie recent changes page should let you go back further - https://phabricator.wikimedia.org/T437966#12362354 (10Habst) 05In progress→03Resolved [00:52:31] (03update) 10habs: Centralize THUMB_WIDTH and apply it to override images too (T433758) [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/20 (owner: 10waldir) [00:53:35] 10Tool-wikinewsie: pl.wikipedia.org is not supported - https://phabricator.wikimedia.org/T438597#12362356 (10ZhaoFJx) 05Duplicate→03Resolved [00:59:40] 10Tool-wikinewsie: Wikinewsie recent changes should link to article card - https://phabricator.wikimedia.org/T438640#12362365 (10ZhaoFJx) p:05Triage→03High [01:00:10] 10Tool-wikinewsie, 07good first task: Wikinewsie recent changes should link to article card - https://phabricator.wikimedia.org/T438640#12362367 (10ZhaoFJx) Thank you for tagging this task with #good_first_task for Wikimedia newcomers! Newcomers often may not be aware of things that may seem obvious to season... [01:00:43] 10Tool-wikinewsie: UX Design Walkthrough For AI Features - https://phabricator.wikimedia.org/T435738#12362371 (10ZhaoFJx) p:05Triage→03Low [01:00:56] 10Tool-wikinewsie: [SPIKE] UX Design Walkthrough For AI Features - https://phabricator.wikimedia.org/T435738#12362374 (10ZhaoFJx) [01:01:34] 10Tool-wikinewsie, 07Spike: [SPIKE] UX Design Walkthrough For AI Features - https://phabricator.wikimedia.org/T435738#12362375 (10ZhaoFJx) [01:04:11] (03merge) 10habs: Centralize THUMB_WIDTH and apply it to override images too (T433758) [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/20 (owner: 10waldir) [01:04:54] (03update) 10habs: Apply blurred backdrop only to opaque images [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/19 (owner: 10waldir) [01:08:16] (03approved) 10habs: Apply blurred backdrop only to opaque images [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/19 (owner: 10waldir) [01:09:04] (03merge) 10habs: Apply blurred backdrop only to opaque images [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/19 (owner: 10waldir) [01:09:35] 10Tool-wikinewsie: Adjustable image alignment on Wikinewsie - https://phabricator.wikimedia.org/T434467#12362398 (10ZhaoFJx) Indeed a big work... Thanks all! [01:14:44] 10Tool-wikinewsie: Adjustable image alignment on Wikinewsie - https://phabricator.wikimedia.org/T434467#12362404 (10ZhaoFJx) 05In progress→03Resolved [01:16:04] 10Tool-wikinewsie: Dropdown selection type for flexible Wikinewsie subportals - https://phabricator.wikimedia.org/T434608#12362405 (10ZhaoFJx) a:03ZhaoFJx [01:16:09] 10Tool-wikinewsie: Dropdown selection type for flexible Wikinewsie subportals - https://phabricator.wikimedia.org/T434608#12362407 (10ZhaoFJx) 05Open→03In progress [01:27:15] (03update) 10habs: Fix T436414: Prevent external link rendering in Wikinewsie AI text [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/12 (owner: 10akaza24) [01:27:36] (03approved) 10habs: Fix T436414: Prevent external link rendering in Wikinewsie AI text [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/12 (owner: 10akaza24) [01:28:16] (03merge) 10habs: Fix T436414: Prevent external link rendering in Wikinewsie AI text [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/12 (owner: 10akaza24) [01:32:41] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster (T438255) [01:32:47] T438255: Can't deploy anymore on Toolforge (pod stays pending) - https://phabricator.wikimedia.org/T438255 [01:38:03] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-worker-nfs-89 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [01:41:05] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance tools-k8s-worker-115 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [01:43:40] !log andrew@cloudcumin1001 tools Added a new k8s worker tools-k8s-worker-115.tools.eqiad1.wikimedia.cloud to the cluster [01:43:40] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster [01:44:23] FIRING: ToolforgeKubernetesNodeNotReady: Kubernetes node tools-k8s-worker-115 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [01:45:24] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T438255) [01:45:28] T438255: Can't deploy anymore on Toolforge (pod stays pending) - https://phabricator.wikimedia.org/T438255 [01:46:05] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance tools-k8s-worker-115 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [01:49:23] FIRING: [2x] ToolforgeKubernetesNodeNotReady: Kubernetes node tools-k8s-worker-115 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [01:57:13] !log andrew@cloudcumin1001 tools Added a new k8s worker-nfs tools-k8s-worker-nfs-89.tools.eqiad1.wikimedia.cloud to the cluster [01:57:13] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster [01:59:23] RESOLVED: [2x] ToolforgeKubernetesNodeNotReady: Kubernetes node tools-k8s-worker-115 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [02:30:05] FIRING: PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-worker-115 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [04:20:07] 10Cloud-VPS (Debian Bullseye Deprecation), 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12362535 (10tchin) Created `deployment-schema04` to replace `deployment-schem... [06:15:09] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [06:33:45] (03update) 10waldir: Apply blurred backdrop only to opaque images [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/19 [06:34:46] (03update) 10waldir: Apply blurred backdrop only to opaque images [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/19 [07:48:51] (03update) 10chimaifeanyi29: Add suggest headline button from AI recap [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/17 [07:49:07] (03update) 10chimaifeanyi29: T436419: Add suggest headline button from AI recap [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/17 [07:54:28] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [08:01:06] 10Tool-spacemedia, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: Can't deploy anymore on Toolforge (pod stays pending) - https://phabricator.wikimedia.org/T438255#12362792 (10aputhin) 05Open→03Resolved p:05Triage→03High a:03Andrew [08:02:04] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [08:05:30] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [08:12:00] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [08:23:25] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [08:31:14] 06tools-infrastructure-team, 07Epic, 06Test Platform (Epics): Explore if there are ways to tune the underlying hardware and configuration for the CI agents - https://phabricator.wikimedia.org/T437407#12362930 (10fgiunchedi) I completed the first round of tests on cloudvirt1077 yesterday, and will be continui... [08:38:09] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [08:49:10] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [08:56:10] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [09:04:24] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [09:09:54] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [09:16:29] 10Tool-wikinewsie, 07good first task: Wikinewsie recent changes should link to article card - https://phabricator.wikimedia.org/T438640#12363059 (10Chimaifeanyi29) a:03Chimaifeanyi29 [09:32:02] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [09:36:48] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [09:42:57] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [09:47:50] 10Tool-delintbot, 13Patch-For-Review: Implement visual regression testing - https://phabricator.wikimedia.org/T415882#12363142 (10Redmin) [09:48:15] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [09:48:51] 06cloud-services-team, 10Toolforge, 06Product Safety and Integrity, 10Temporary accounts: Check impact of Temporary Accounts on Toolforge tools - https://phabricator.wikimedia.org/T378516#12363147 (10aputhin) [09:49:28] 10Data-Services: [wikireplicas] clouddb* free memory decreases over time - https://phabricator.wikimedia.org/T365164#12363151 (10aputhin) [10:18:43] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [10:20:45] 10Tool-delintbot: Document how to operate the bot - https://phabricator.wikimedia.org/T416070#12363312 (10Redmin) [10:24:25] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [10:26:54] 10Tool-delintbot: Document how to operate the bot - https://phabricator.wikimedia.org/T416070#12363343 (10Redmin) [11:27:46] 10Tool-spacemedia, 10Toolforge (Quota-requests): Request increased quota for spacemedia Toolforge tool - https://phabricator.wikimedia.org/T439237 (10Don-vip) 03NEW [13:32:08] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-115 (T438255) [13:32:13] T438255: Can't deploy anymore on Toolforge (pod stays pending) - https://phabricator.wikimedia.org/T438255 [13:33:19] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-115 (T438255) [13:33:57] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster (T438255) [13:40:03] (03PS1) 10Btullis: ceph: Add a dummy STS key for the rados gateway [labs/private] - 10https://gerrit.wikimedia.org/r/1345118 (https://phabricator.wikimedia.org/T435590) [13:40:44] (03CR) 10Btullis: [V:03+2 C:03+2] ceph: Add a dummy STS key for the rados gateway [labs/private] - 10https://gerrit.wikimedia.org/r/1345118 (https://phabricator.wikimedia.org/T435590) (owner: 10Btullis) [13:45:48] !log andrew@cloudcumin1001 tools Added a new k8s worker tools-k8s-worker-115.tools.eqiad1.wikimedia.cloud to the cluster [13:45:48] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster [14:07:35] (03open) 10waldir: Make language selector work on narrow screens [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/23 [14:09:49] (03update) 10waldir: Make language selector work on narrow screens [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/23 [14:10:01] (03update) 10waldir: Make language selector work on narrow screens [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/23 [14:10:47] (03update) 10waldir: Make language selector work on narrow screens [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/23 [14:13:24] (03update) 10waldir: Make language selector work on narrow screens [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/23 [14:15:57] 10Tool-wikinewsie, 07good first task: Wikinewsie i18n - https://phabricator.wikimedia.org/T432938#12364308 (10waldyrious) Heads-up: I added a new MR (https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/23) to fix an overflow issue in small screens with the language selector @Chimaifeanyi29... [14:16:13] (03update) 10waldir: T432938: Make language selector work on narrow screens [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/23 [14:35:41] 10Cloud-VPS (Debian Bullseye Deprecation), 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12364411 (10tchin) Created `deployment-eventgate05`, accessible at `https://i... [14:48:31] 10Tool-lexeme-forms, 10MediaWiki-extensions-OAuth, 06MediaWiki-Core-Platform-Team (Kanban): Error “The refresh token is invalid.” after migrating Wikidata Lexeme Forms to OAuth 2 - https://phabricator.wikimedia.org/T431146#12364449 (10Tgr) @LucasWerkmeister three months have passed, should we just assume {e4... [15:07:55] (03PS1) 10Btullis: ceph: Move the dummy STS key to the correct hiera file [labs/private] - 10https://gerrit.wikimedia.org/r/1345151 (https://phabricator.wikimedia.org/T435590) [15:09:33] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12364561 (10JSengupta-WMF) We could. My proposal would be to change the link name to something user understandable (a verb preferably) than the direct tool name... [15:11:45] 10Cloud-VPS (Debian Bullseye Deprecation), 06Collaboration-Services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12364571 (10Arnoldokoth) Hey @komla I'm currently working on this. There is an issue with the certificates on the host resulting in a failed puppet... [15:13:35] (03CR) 10Btullis: [V:03+2 C:03+2] ceph: Move the dummy STS key to the correct hiera file [labs/private] - 10https://gerrit.wikimedia.org/r/1345151 (https://phabricator.wikimedia.org/T435590) (owner: 10Btullis) [15:19:13] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12364642 (10Doc_James) Perfect... Do we have a time line of when this may roll out? [15:26:55] 10Cloud-VPS (Debian Bullseye Deprecation), 06Collaboration-Services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12364665 (10Dzahn) Hi @Arnoldokoth we have seen this type of error before. See for example T95586 and T187042. I think there are some hints in th... [16:16:03] 06tools-infrastructure-team: tools-k8s-worker-nfs going offline - https://phabricator.wikimedia.org/T439177#12364893 (10Andrew) My very first google search turned up this reddit post: "We've seen cases where having a container run the node out of memory can cause Linux to kill unrelated processes, such as the k... [16:25:23] 10Cloud-VPS (Debian Bullseye Deprecation), 06Collaboration-Services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12364938 (10Arnoldokoth) Thanks @Dzahn From what I've gathered it's using the global puppetmaster. I've tried to access it via `cloudcumin` since I... [16:27:49] 10Cloud-VPS (Debian Bullseye Deprecation), 06Collaboration-Services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12364945 (10taavi) >>! In T429494#12364938, @Arnoldokoth wrote: > From what I've gathered it's using the global puppetmaster. No, it is not: `lang=... [16:28:56] 10Cloud-VPS (Quota-requests), 06tools-infrastructure-team, 06tools-platform-team: Quota increase request for toolforge - https://phabricator.wikimedia.org/T439157#12364963 (10fnegri) @Andrew looks like you applied this change, can it be marked as Resolved? [16:29:37] 10Data-Services, 06tools-platform-team: Update replica transcode views which references invalid columns - https://phabricator.wikimedia.org/T439268 (10TheDJ) 03NEW [16:30:06] 10Data-Services, 06tools-platform-team: Update replica transcode views which references invalid columns - https://phabricator.wikimedia.org/T439268#12364978 (10TheDJ) [16:32:15] 06cloud-services-team, 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (San Miguel de Allende 31): Apply the 4x-iops Cinder volume type to the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T437127#12364997 (10fnegri) [16:34:16] 10Cloud-VPS (Debian Bullseye Deprecation), 06Collaboration-Services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12365000 (10Dzahn) Thanks taavi for pointing that out. I think in that case let's ask if we can't just switch it to use the global master and possib... [16:35:44] 10Toolforge, 06tools-platform-team: Make it possible to configure order of language-specific script execution during build - https://phabricator.wikimedia.org/T439269 (10Redmin) 03NEW [16:37:42] 10Toolforge, 06tools-platform-team: Make it possible to configure order of language-specific script execution during build - https://phabricator.wikimedia.org/T439269#12365017 (10Redmin) [16:38:24] 10Cloud-VPS (Quota-requests), 06tools-infrastructure-team, 06tools-platform-team: Quota increase request for toolforge - https://phabricator.wikimedia.org/T439157#12365020 (10Andrew) 05Open→03Resolved a:03Andrew yep [16:42:06] 10Cloud-VPS (Project-requests): Request creation of wikivisage VPS project - https://phabricator.wikimedia.org/T438842#12365039 (10Premeditated) I set up `wikivisage-face` as a dedicated tool on Toolforge. The buildpack build compiled cleanly with `project.toml`, and the service is running smoothly as a publishe... [16:44:44] 10Cloud-VPS (Debian Bullseye Deprecation), 06Collaboration-Services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12365045 (10Arnoldokoth) @taavi Thanks. That explains it. The first puppet run used the global puppet master and then it got configured (unclear to... [16:49:11] 06tools-infrastructure-team: tools-k8s-worker-nfs going offline - https://phabricator.wikimedia.org/T439177#12365049 (10Andrew) Some solutions: 1) Make sure every pod has a realistic memory limit, and don't overschedule RAM. Then the k8s oom-killer should take care of killing individual pods that run amock and... [16:49:36] 10Toolforge, 06tools-infrastructure-team: tools-k8s-worker-nfs going offline - https://phabricator.wikimedia.org/T439177#12365051 (10taavi) [17:04:54] 10Cloud-VPS (Debian Bullseye Deprecation), 06Collaboration-Services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12365078 (10Arnoldokoth) ` aokoth@deploy-1007:~$ host puppetmaster.cloudinfra.wmflabs.org puppetmaster.cloudinfra.wmflabs.org has address 172.16.7.12... [17:16:46] 10Cloud-VPS (Debian Bullseye Deprecation), 06Collaboration-Services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12365105 (10Dzahn) In `hieradata/cloud/eqiad1/devtools/common.yaml` while there is `puppetmaster: puppetmaster.cloudinfra.wmflabs.org` there is also... [17:19:52] 10Cloud-VPS (Debian Bullseye Deprecation), 06Collaboration-Services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12365118 (10Arnoldokoth) @Dzahn Yeah, I saw that too and it does look unrelated. [17:21:44] 10Cloud-VPS (Debian Bullseye Deprecation), 06Collaboration-Services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12365119 (10taavi) The devtools project has prefix-level hiera in Horizon configured on the `deploy` prefix to use the local puppetserver. [17:31:35] 10Data-Services, 06tools-platform-team: Update replica transcode views which references invalid columns - https://phabricator.wikimedia.org/T439268#12365134 (10TheDJ) p:05Triage→03Low [17:32:23] 10Cloud-VPS (Debian Bullseye Deprecation), 06Collaboration-Services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12365138 (10Arnoldokoth) Well, today I learned... and it uncovers a two month long mystery. Thank you @taavi I'll get rid of that... I'd rather the... [17:33:38] (03update) 10fnegri: Notify when an alert fires or is resolved [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/5 (https://phabricator.wikimedia.org/T432865) [17:34:00] (03update) 10fnegri: Notify when an alert fires or is resolved [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/5 (https://phabricator.wikimedia.org/T432865) [17:35:31] 10Tool-delintbot, 13Patch-For-Review: Implement visual regression testing - https://phabricator.wikimedia.org/T415882#12365142 (10Redmin) [17:47:33] 10Toolforge, 06tools-infrastructure-team: tools-k8s-worker-nfs going offline - https://phabricator.wikimedia.org/T439177#12365152 (10BLiviero-WMF) do we have an idea of what was running on that worker in the first place? (is there a way to tell) [17:57:04] (03update) 10fnegri: Notify when an alert fires or is resolved [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/5 (https://phabricator.wikimedia.org/T432865) [18:00:17] 10Tool-lexeme-forms, 10MediaWiki-extensions-OAuth, 06MediaWiki-Core-Platform-Team (Kanban): Error “The refresh token is invalid.” after migrating Wikidata Lexeme Forms to OAuth 2 - https://phabricator.wikimedia.org/T431146#12365179 (10LucasWerkmeister) 05Open→03Resolved a:03Tgr I guess so… I kinda... [18:00:36] (03update) 10fnegri: Notify when an alert fires or is resolved [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/5 (https://phabricator.wikimedia.org/T432865) [18:13:38] 10Tool-wsindex, 06MediaWiki-Media-Platform-Team, 10Wikipedia-iOS-App, 06WikipediaApp, and 2 others: Update apps to support for the move from upload.wikimedia.org to thumb.wikimedia.org - https://phabricator.wikimedia.org/T434821#12365203 (10GMikesell-WMF) @Ladsgroup @ABorbaWMF Please review AC1 and AC4, th... [18:15:05] 10Tool-wsindex, 06MediaWiki-Media-Platform-Team, 10Wikipedia-iOS-App, 06WikipediaApp, and 2 others: Update apps to support for the move from upload.wikimedia.org to thumb.wikimedia.org - https://phabricator.wikimedia.org/T434821#12365213 (10GMikesell-WMF) [18:18:43] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-87 [18:20:13] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-87 [18:22:15] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T438255) [18:22:19] T438255: Can't deploy anymore on Toolforge (pod stays pending) - https://phabricator.wikimedia.org/T438255 [18:22:32] 10Cloud-VPS (Debian Bullseye Deprecation), 06Collaboration-Services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12365216 (10Dzahn) I also prefer to have all Hiera settings in the repo, in a single place. The arguments that are mostly just valid for people who... [18:31:49] !log andrew@cloudcumin1001 tools Added a new k8s worker-nfs tools-k8s-worker-nfs-90.tools.eqiad1.wikimedia.cloud to the cluster [18:31:49] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster [18:33:04] (03update) 10fnegri: Notify when an alert fires or is resolved [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/5 (https://phabricator.wikimedia.org/T432865) [18:33:04] (03update) 10fnegri: Do not re-notify old alerts after a restart [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/6 (https://phabricator.wikimedia.org/T432865) [18:33:06] (03update) 10fnegri: Do not re-notify old alerts after a restart [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/6 (https://phabricator.wikimedia.org/T432865) [18:43:13] (03update) 10fnegri: Do not re-notify old alerts after a restart [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/6 (https://phabricator.wikimedia.org/T432865) [18:47:35] (03open) 10chimaifeanyi29: T438640: link recent changes to article card instead of Wikidata [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/24 [19:56:12] (03CR) 10Duytran888997@gmail.com: "quansat775@gmail.com" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1334027 (owner: 10Gergő Tisza) [20:13:59] 06tools-infrastructure-team, 07Epic, 06Test Platform (Epics): Explore if there are ways to tune the underlying hardware and configuration for the CI agents - https://phabricator.wikimedia.org/T437407#12365416 (10Krinkle) [20:18:47] 10Tool-Pageviews: Add PetScan as a source for Massviews - https://phabricator.wikimedia.org/T437012#12365425 (10MusikAnimal) 05In progress→03Resolved [20:25:39] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.2 Toolforge Alerting System - https://phabricator.wikimedia.org/T432860#12365431 (10fnegri) //Note: we are experimenting with cross-posting to Phabricator the weekly report that previously was only posted to Asana, where only few people have acc... [20:40:51] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: Email tool maintainers when an alert condition is detected - https://phabricator.wikimedia.org/T432865#12365450 (10fnegri) All the review comments in https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/5 and http... [20:59:11] 10Cloud-VPS (Debian Bullseye Deprecation): Requests to extend shutdown timeline for Bullseye deprecation - https://phabricator.wikimedia.org/T434103#12365477 (10Krinkle) [20:59:50] 10Cloud-VPS (Debian Bullseye Deprecation): Requests to extend shutdown timeline for Bullseye deprecation - https://phabricator.wikimedia.org/T434103#12365483 (10Krinkle) Marking `cvn` as done per T395164. Please comment there if I missed anything. [21:01:51] (03PS1) 10Andrea Denisse: alert: add a dummy Klaxon managers incident URL [labs/private] - 10https://gerrit.wikimedia.org/r/1345218 [21:01:51] (03CR) 10Andrea Denisse: "Noticed this secret was missing, PCC didn't complain because it's optional but it also won't detect any changes regarding it." [labs/private] - 10https://gerrit.wikimedia.org/r/1345218 (owner: 10Andrea Denisse) [21:07:13] (03PS2) 10Andrea Denisse: kubernetes: add dummy secrets for oncall [labs/private] - 10https://gerrit.wikimedia.org/r/1345219 (https://phabricator.wikimedia.org/T438803) [21:07:35] (03CR) 10Andrea Denisse: "Adding the same values as the prod alert host." [labs/private] - 10https://gerrit.wikimedia.org/r/1345219 (https://phabricator.wikimedia.org/T438803) (owner: 10Andrea Denisse) [21:46:52] (03CR) 10Andrea Denisse: [V:03+2 C:03+2] kubernetes: add dummy secrets for oncall [labs/private] - 10https://gerrit.wikimedia.org/r/1345219 (https://phabricator.wikimedia.org/T438803) (owner: 10Andrea Denisse) [21:49:18] (03PS1) 10Andrea Denisse: Revert "ceph: Move the dummy STS key to the correct hiera file" [labs/private] - 10https://gerrit.wikimedia.org/r/1345227 [21:49:31] (03CR) 10Andrea Denisse: [V:03+2 C:03+2] Revert "ceph: Move the dummy STS key to the correct hiera file" [labs/private] - 10https://gerrit.wikimedia.org/r/1345227 (owner: 10Andrea Denisse) [21:50:15] (03PS1) 10Andrea Denisse: Revert^2 "ceph: Move the dummy STS key to the correct hiera file" [labs/private] - 10https://gerrit.wikimedia.org/r/1345228 [21:51:55] (03PS2) 10Andrea Denisse: ceph: Move the dummy STS key to the correct hiera file [labs/private] - 10https://gerrit.wikimedia.org/r/1345228 (https://phabricator.wikimedia.org/T435590) [22:27:11] 10Toolforge, 06tools-infrastructure-team: tools-k8s-worker-nfs going offline - https://phabricator.wikimedia.org/T439177#12365769 (10Andrew) K8s eviction thresholds are described as doing what we want. My main concern is that they should already be present with the non-ridiculous cutoff of memory.available<100... [22:33:23] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12365789 (10Jclark-ctr) [22:39:25] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12365795 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1004 for host cloudcephosd1055.eqiad.... [23:12:57] 10Tool-techcontribs: gerrit contribs should include commits from svn era - https://phabricator.wikimedia.org/T439295 (10Bawolff) 03NEW [23:15:39] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12365899 (10Jclark-ctr) @fgiunchedi I have not had any luck so far. The other Dell cloudcephosd105 servers have RAID1 for the two sma... [23:15:47] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12365900 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1004 for host cloudcephosd1055.eqiad.wmne...