[00:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:49:07] 06cloud-services-team, 10Toolforge, 07Upstream: Python buildpack does not detect requirements from pyproject.toml - https://phabricator.wikimedia.org/T353762#10399829 (10dcaro) Related {T380127} [08:49:16] 06cloud-services-team, 10Toolforge: Upgrade golang buildpack to 1.22 - https://phabricator.wikimedia.org/T363854#10399833 (10dcaro) Related {T380127} [08:49:29] 06cloud-services-team, 10Toolforge: Upgrade python buildpack to v0.17.0 or newer for Poetry support - https://phabricator.wikimedia.org/T374056#10399837 (10dcaro) p:05Triage→03Medium Related {T380127} [08:49:31] 06cloud-services-team, 10Toolforge: Toolforge Build Service does not support .python-version - https://phabricator.wikimedia.org/T381923#10399845 (10dcaro) Yep, the documentation shifted and we did not update (yet) the buildpack underneath, this will be solved with the next buildpack upgrade, related tasks: {... [08:50:25] 06cloud-services-team, 10Toolforge: Add support for Python 3.13 - https://phabricator.wikimedia.org/T381899#10399850 (10dcaro) Related {T380127} [08:56:29] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10399869 (10Sarai-WMF) >>! In T376267#10394511, @Andrew wrote: > I was just looking over @Sarai-WMF's shoulder and wikitech is for some reason not editable for her. I gave her the comfirmed u... [09:58:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-5 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [10:47:04] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5 [10:47:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:52:51] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5 [10:52:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:55:38] 06cloud-services-team, 10Horizon, 10Toolforge: Trove DB for glamtools/baglama2 - https://phabricator.wikimedia.org/T382058 (10Magnus) 03NEW [10:55:53] 06cloud-services-team, 10Horizon, 10Toolforge: Trove DB for glamtools/baglama2 - https://phabricator.wikimedia.org/T382058#10400092 (10Magnus) See https://stackoverflow.com/questions/2428738/how-do-you-fix-a-mysql-incorrect-key-file-error-when-you-cant-repair-the-tabl [10:57:03] 06cloud-services-team, 10Horizon, 10Toolforge: Trove DB for glamtools/baglama2 - https://phabricator.wikimedia.org/T382058#10400098 (10Magnus) Trying to repair it manually: ` REPAIR TABLE group2view USE_FRM; ... The storage engine for the table doesn't support repair ` [11:12:33] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-5 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [11:27:43] 06cloud-services-team, 10Data-Services, 06DBA, 07Chinese-Sites, 13Patch-For-Review: Prepare and check storage layer for arbcom_zhwiki - https://phabricator.wikimedia.org/T381086#10400159 (10Marostegui) As part of the kernel reboots, I've restarted s5 sanitarium codfw (db2186). Strictly, still pending eqi... [11:36:21] 06cloud-services-team, 10Data-Services, 06DBA, 07Chinese-Sites, 13Patch-For-Review: Prepare and check storage layer for arbcom_zhwiki - https://phabricator.wikimedia.org/T381086#10400191 (10Marostegui) db2187 in codfw also restarted, rebooted and upgraded. [11:44:12] 06cloud-services-team, 10Data-Services, 06DBA, 07Chinese-Sites, 13Patch-For-Review: Prepare and check storage layer for arbcom_zhwiki - https://phabricator.wikimedia.org/T381086#10400215 (10Marostegui) a:05ABran-WMF→03None I have restarted s5 eqiad for now. Just that process. We have still to restart... [11:54:30] (03approved) 10dcaro: metrics: add prometheus instrumentation [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/46 (https://phabricator.wikimedia.org/T381249) (owner: 10sstefanova) [11:57:05] (03open) 10sstefanova: openapi: add description to all operations [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/50 [12:02:17] (03update) 10sstefanova: openapi: add description to all operations [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/50 [12:24:45] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/26 [12:59:12] 10Tool-lexeme-forms, 06translatewiki.net, 10LPL Essential (LPL Essential 2024 Nov-Dec), 13Patch-For-Review, 07Unplanned-Sprint-Work: translatewiki export for Wikidata Lexeme Forms tries to remove sh-latn translations - https://phabricator.wikimedia.org/T379188#10400393 (10Nikerabbit) https://github.com/l... [15:35:16] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for tigwiki - https://phabricator.wikimedia.org/T381378#10400894 (10Ladsgroup) The wiki is live [15:50:55] 10Tool-wikiqanda, 06Future-Audiences: Refactoring btw Slack & Discord - https://phabricator.wikimedia.org/T381795#10400910 (10Maryana) [15:50:56] 10Tool-wikiqanda, 06Future-Audiences, 07Epic: [Epic] Discord Q&A bot Milestone 3 - https://phabricator.wikimedia.org/T378126#10400911 (10Maryana) [15:51:41] 10Tool-wikiqanda, 06Future-Audiences: [Discord] Add provenance parameters to bot links - https://phabricator.wikimedia.org/T382019#10400924 (10Maryana) [15:52:03] 10Tool-wikiqanda, 06Future-Audiences: [Discord] Add provenance parameters to bot links - https://phabricator.wikimedia.org/T382019#10400925 (10Maryana) [15:52:04] 10Tool-wikiqanda, 06Future-Audiences, 07Epic: [Epic] Discord Q&A bot Milestone 3 - https://phabricator.wikimedia.org/T378126#10400926 (10Maryana) [16:01:43] 10VPS-project-Wikistats: Add tigwiki to wikistats - https://phabricator.wikimedia.org/T381383#10400961 (10Dzahn) a:03Dzahn [16:07:59] 06cloud-services-team, 10Cloud-VPS: Trove DB for glamtools/baglama2 - https://phabricator.wikimedia.org/T382058#10400971 (10JJMC89) [16:12:55] 10VPS-project-Wikistats: Add tigwiki to wikistats - https://phabricator.wikimedia.org/T381383#10400985 (10Dzahn) 05Open→03Resolved ` MariaDB [wikistats]> insert into wikipedias (prefix, lang, loclang, method) values ("tgi","Tigre","ትግሬ",8); ... MariaDB [wikistats]> update wikipedias se... [16:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:22:32] 10Tool-wikiqanda, 06Future-Audiences: Add provenance parameters to bot links - https://phabricator.wikimedia.org/T382019#10401007 (10DLin-WMF) [16:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:07:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-73 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [18:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:56:27] 06cloud-services-team, 10Toolforge, 10Codex, 06Design-System-Team: Add Codex to Toolforge cdnjs - https://phabricator.wikimedia.org/T338834#10401526 (10Pppery) This has clearly missed the mark for being reasonably announced in Tech News over a year and a half after the original change. [19:05:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:27:58] 10Tool-wikiqanda, 06Future-Audiences, 07Design: Bot user personas - https://phabricator.wikimedia.org/T381009#10401595 (10DLin-WMF) 05Open→03In progress [19:28:20] 10Tool-wikiqanda, 06Future-Audiences, 07Design: Bot user personas - https://phabricator.wikimedia.org/T381009#10401598 (10DLin-WMF) 05In progress→03Open [20:01:08] 06cloud-services-team, 10Toolforge, 10Codex, 06Design-System-Team: Add Codex to Toolforge cdnjs - https://phabricator.wikimedia.org/T338834#10401660 (10Quiddity) I believe we were still waiting for the sub-task to be resolved, of writing documentation on how to use it. I will move the draft-summary and... [20:33:47] 10Tool-wikiqanda, 06Future-Audiences: Refactoring btw Slack & Discord - https://phabricator.wikimedia.org/T381795#10401685 (10derenrich) a:03derenrich [22:14:43] 10Tool-lexeme-forms, 06translatewiki.net, 10LPL Essential (LPL Essential 2024 Nov-Dec), 13Patch-For-Review, 07Unplanned-Sprint-Work: translatewiki export for Wikidata Lexeme Forms tries to remove sh-latn translations - https://phabricator.wikimedia.org/T379188#10401890 (10LucasWerkmeister) 05Open→... [22:14:47] 10Tool-lexeme-forms, 06translatewiki.net, 10LPL Essential (LPL Essential 2024 Nov-Dec), 13Patch-For-Review, 07Unplanned-Sprint-Work: translatewiki export for Wikidata Lexeme Forms tries to remove sh-latn translations - https://phabricator.wikimedia.org/T379188#10401893 (10LucasWerkmeister) a:05Lucas...