[03:22:30] 06serviceops, 06Content-Transform-Team: Parsoid extension no longer loaded on parsoidtest1001 - https://phabricator.wikimedia.org/T388465#10621998 (10ssastry) Neat .. creating /etc/wikimedia-servergroup with contents "parsoid" does the trick there. We should just puppetize it. Help welcome with the puppetizati... [09:18:56] 06serviceops, 10CX-cxserver, 10LPL Essential (LPL Essential 2025 Feb-Mar), 13Patch-For-Review, 07Technical-Debt: Use openapi compliant examples in swagger spec - https://phabricator.wikimedia.org/T382294#10622562 (10Wangombe) https://gerrit.wikimedia.org/r/1122072 has been merged. We can do a sanity chec... [10:17:00] 06serviceops, 06Commons, 10SRE-swift-storage: Commons thumbnails are broken for certain large sizes of thumbnail images - https://phabricator.wikimedia.org/T358738#10622775 (10jijiki) 05Open→03Resolved a:03jijiki [10:17:50] Dear serviceops, are you using this mw infra window? [10:29:22] 06serviceops, 07Wikimedia-production-error: Misbehaving mw-api-ext pods serving 5xx - https://phabricator.wikimedia.org/T370425#10622835 (10jijiki) 05Open→03Resolved a:03jijiki Closing as this issue hasn't occurred since [10:31:08] 06serviceops, 10MW-on-K8s: mw-script fails to render in CI - https://phabricator.wikimedia.org/T367200#10622843 (10jijiki) @RLazarus @JMeybohm is this issue still present? [10:36:50] 06serviceops: Multiple images fail to build from sources - https://phabricator.wikimedia.org/T350366#10622863 (10jijiki) 05Open→03Resolved Looks like it is resolved, please reopen if needed [10:37:33] Amir1: it appears that no [10:37:44] sorry for the late reply, too much botnet traffic [10:37:46] bot* [10:37:56] awesome. I have like five deploys [11:00:41] 06serviceops, 07Datacenter-Switchover: SRE comms for March 2025 Datacentre switchover - https://phabricator.wikimedia.org/T385157#10622929 (10hnowlan) [11:04:15] 06serviceops, 06Content-Transform-Team, 13Patch-For-Review: Parsoid extension no longer loaded on parsoidtest1001 - https://phabricator.wikimedia.org/T388465#10622939 (10Clement_Goubert) 05Open→03In progress [11:15:02] 06serviceops, 06Infrastructure-Foundations, 10Packaging, 06SRE: Package php-ast in {stretch,buster}-wikimedia/component - https://phabricator.wikimedia.org/T280210#10622983 (10hashar) 05Open→03Resolved a:03Legoktm **TLDR: this was solved by @Legoktm in October 2021** I am adding back #serviceops... [11:23:29] 06serviceops, 10RESTBase Sunsetting, 07User-notice-archive: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10623045 (10daniel) >>! In T374683#10511063, @Legoktm wrote: > @HCoplin-WMF is there a reason no inc... [11:27:34] 06serviceops, 07Datacenter-Switchover, 07User-notice: MoveComms support for March 2025 Datacentre switchover - https://phabricator.wikimedia.org/T387444#10623058 (10Trizek-WMF) [11:28:22] 06serviceops, 07Datacenter-Switchover, 07User-notice: MoveComms support for March 2025 Datacentre switchover - https://phabricator.wikimedia.org/T387444#10623074 (10Trizek-WMF) I'm waiting for translators to check on the message before sending it to communities on Thursday. [11:46:03] Amir1: I'd like to verify that the temp accounts expiration cron script is working properly. Can I just run the `foreachwikiindblist` invocation that's used in the puppet script on mwmaint, to check that it's working? [11:46:24] and if so, should I then use a `!log` message to show that I've manually run it? [11:46:43] that should work [11:50:35] If you'd like you can test that it works in k8s by running it from deploy with mwscript-k8s :) [11:50:53] (that way when I migrate it to mw-cron in k8s I'll know it works :p) [11:57:56] claime: sure, will do that [11:58:09] kostajh: <3 [12:09:36] claime: is there documentation for how to run this with mwscript-k8s? I am not seeing it on https://wikitech.wikimedia.org/wiki/MediaWiki_On_Kubernetes [12:10:02] ah, it's here https://wikitech.wikimedia.org/wiki/Maintenance_scripts [12:10:03] https://wikitech.wikimedia.org/wiki/Maintenance_scripts [12:10:05] yeah [12:10:17] I'm gonna link it from the MediaWiki_On_Kubernetes though [12:10:33] thanks! [12:11:10] Oh it is already, https://wikitech.wikimedia.org/wiki/MediaWiki_On_Kubernetes#Maintenance [12:11:19] It may need to be more prominent though [12:11:27] I think I'll remove the "What traffic" header [12:17:57] kostajh: I've changed a bit the https://wikitech.wikimedia.org/wiki/MediaWiki_On_Kubernetes page, is the information you'd want to find as to what serves what more easily found this way? [12:19:04] lgtm! [12:19:18] should the text `mwscript-k8s` appear on that page somwhere? [12:19:28] good ide [12:19:30] a [12:20:18] Done [12:24:33] 06serviceops, 10MediaWiki-extensions-ReadingLists, 06MW-Interfaces-Team, 10RESTBase Sunsetting: Switchover plan from RESTbase to REST Gateway for Reading Lists endpoints - https://phabricator.wikimedia.org/T384891#10623211 (10HCoplin-WMF) [12:24:53] huh, roll_restart behaviour doesn't work like it used to in the thumbor helmfile dir [12:24:57] in ./helmfile.yaml: error during helmfile.yaml.part.1 parsing: template: stringTemplate:7:73: executing "stringTemplate" at : error calling eq: incompatible types for comparison [12:25:12] need to ensure it's handled as a string: `helmfile -e codfw --state-values-set 'roll_restart="1"' sync` [12:28:50] might be the latest version of helmfile that rolled out recently? [12:29:39] could be yes, I didn't see this before [12:30:19] yeah, we explicitly expect a string by default [12:30:44] so helmfile is just typing stuff correctly (as correctly as we can in the unholy marriage of yaml and gotpl) [12:35:22] a rather large series of fixes https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1126541 [12:42:20] 06serviceops, 10decommission-hardware: decommission kafka-main1001 / kafka-main1002 / kafka-main1003 / kafka-main1004 / kafka-main1005 - https://phabricator.wikimedia.org/T381593#10623259 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jiji@cumin1002 for hosts: `kafka-main[1001-1005].eqiad.... [12:54:11] claime: is `--comment=T388125` supposed to post a comment to that task? [12:55:07] kostajh: No, it adds a comment label on the k8s job [12:55:44] ok, thx [12:57:09] 06serviceops, 06Infrastructure-Foundations, 10Maps (Kartotherian), 13Patch-For-Review: Scale up Kartotherian on Wikikube and move live traffic to it - https://phabricator.wikimedia.org/T386926#10623328 (10elukey) Kartotherian is now fully served by K8s, all bare metal hosts have been depooled. Things to w... [14:13:30] 06serviceops, 10decommission-hardware, 13Patch-For-Review: decommission kafka-main1001 / kafka-main1002 / kafka-main1003 / kafka-main1004 / kafka-main1005 - https://phabricator.wikimedia.org/T381593#10623827 (10jijiki) [14:14:15] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-eqiad: decommission kafka-main1001 / kafka-main1002 / kafka-main1003 / kafka-main1004 / kafka-main1005 - https://phabricator.wikimedia.org/T381593#10623829 (10jijiki) [14:24:32] 06serviceops, 06Trust and Safety Product Team: Migrate trust_and_safety_product_team jobs to mw-cron - https://phabricator.wikimedia.org/T388542#10623877 (10Clement_Goubert) [14:24:43] 06serviceops, 06Community-Tech: Migrate community-tech jobs to mw-cron - https://phabricator.wikimedia.org/T388536#10623880 (10Clement_Goubert) [14:29:03] 06serviceops, 06Trust and Safety Product Team: Migrate trust_and_safety_product_team jobs to mw-cron - https://phabricator.wikimedia.org/T388542#10623891 (10Dreamy_Jazz) @Clement_Goubert I put the wrong name for the one owned by #community-tech. It's `mediawiki_job_pageassessments_cleanup.timer` which would be... [14:37:26] 06serviceops, 10MediaWiki-Page-derived-data: Migrate MediaWiki-Page-derived-data jobs to mw-cron - https://phabricator.wikimedia.org/T388530#10623948 (10Clement_Goubert) [14:38:09] 06serviceops, 06Security-Team: Migrate Security-Team jobs to mw-cron - https://phabricator.wikimedia.org/T388531#10623950 (10Clement_Goubert) [14:38:39] 06serviceops, 10MediaWiki-General: Migrate MediaWiki-General jobs to mw-cron - https://phabricator.wikimedia.org/T388532#10623952 (10Clement_Goubert) [14:39:01] 06serviceops, 10MediaWiki-Categories: Migrate MediaWiki-Categories jobs to mw-cron - https://phabricator.wikimedia.org/T388533#10623954 (10Clement_Goubert) [14:39:38] 06serviceops, 10MediaWiki-Special-pages: Migrate mediawiki-special-pages jobs to mw-cron - https://phabricator.wikimedia.org/T388534#10623956 (10Clement_Goubert) [14:40:13] 06serviceops, 10FlaggedRevs: Migrate flaggedrevs jobs to mw-cron - https://phabricator.wikimedia.org/T388535#10623959 (10Clement_Goubert) [14:40:42] 06serviceops, 06Data-Engineering: Migrate data-engineering jobs to mw-cron - https://phabricator.wikimedia.org/T388537#10623962 (10Clement_Goubert) [14:40:51] 06serviceops, 06Discovery-Search: Migrate discovery-search jobs to mw-cron - https://phabricator.wikimedia.org/T388538#10623969 (10Clement_Goubert) [14:41:44] 06serviceops, 06Language and Product Localization: Migrate language_and_product_localization jobs to mw-cron - https://phabricator.wikimedia.org/T388539#10623971 (10Clement_Goubert) [14:42:08] 06serviceops, 06MediaWiki-Platform-Team: Migrate mediawiki-platform-team jobs to mw-cron - https://phabricator.wikimedia.org/T388540#10623976 (10Clement_Goubert) [14:42:41] 06serviceops, 06MW-Interfaces-Team: Migrate mw-interfaces-team jobs to mw-cron - https://phabricator.wikimedia.org/T388541#10623980 (10Clement_Goubert) [14:43:11] 06serviceops, 10Wikidata, 06Wikidata Dev Team: Migrate wikidata_dev_team jobs to mw-cron - https://phabricator.wikimedia.org/T388543#10623983 (10Clement_Goubert) [14:51:04] 06serviceops, 10MediaWiki-extensions-CentralAuth, 10MW-on-K8s, 10MediaWiki-Platform-Team (Radar): Migrate CentralAuth maintenance jobs to mw-cron - https://phabricator.wikimedia.org/T385866#10624052 (10Clement_Goubert) >>! In T385866#10531647, @Tgr wrote: > All of these are low-risk (in the sense that if t... [14:52:23] 06serviceops, 10MediaWiki-File-management, 10MW-on-K8s: Migrate cleanupUploadStash job to mw-cron - https://phabricator.wikimedia.org/T385868#10624074 (10Clement_Goubert) [15:02:36] 06serviceops, 10CirrusSearch, 10Data-Platform-SRE (2025.03.01 - 2025.03.21), 10Discovery-Search (2025.03.01 - 2025.03.21): Repartition [eqiad|codfw].cirrussearch.update_pipeline.update.v1 topics in kafka-main@[eqiad|codfw] - https://phabricator.wikimedia.org/T387863#10624138 (10brouberol) @JMeybohm would y... [15:05:32] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission kafka-main1001 / kafka-main1002 / kafka-main1003 / kafka-main1004 / kafka-main1005 - https://phabricator.wikimedia.org/T381593#10624167 (10VRiley-WMF) a:03VRiley-WMF [15:12:29] 06serviceops, 10CirrusSearch, 10Data-Platform-SRE (2025.03.01 - 2025.03.21), 10Discovery-Search (2025.03.01 - 2025.03.21): Repartition [eqiad|codfw].cirrussearch.update_pipeline.update.v1 topics in kafka-main@[eqiad|codfw] - https://phabricator.wikimedia.org/T387863#10624216 (10JMeybohm) >>! In T387863#106... [15:17:59] 06serviceops, 07Datacenter-Switchover: SRE comms for March 2025 Datacentre switchover - https://phabricator.wikimedia.org/T385157#10624227 (10hnowlan) [15:19:03] 06serviceops, 06Content-Transform-Team, 13Patch-For-Review: Parsoid extension no longer loaded on parsoidtest1001 - https://phabricator.wikimedia.org/T388465#10624234 (10Clement_Goubert) I've merged the puppetization of the file, can you test that everything works ok on `parsoidtest1001.eqiad.wmnet` and `tes... [15:20:55] 06serviceops, 06Community-Tech: Migrate community-tech jobs to mw-cron - https://phabricator.wikimedia.org/T388536#10624238 (10Clement_Goubert) [15:21:15] 06serviceops, 06Trust and Safety Product Team: Migrate trust_and_safety_product_team jobs to mw-cron - https://phabricator.wikimedia.org/T388542#10624239 (10Clement_Goubert) [15:21:47] 06serviceops, 06Trust and Safety Product Team: Migrate trust_and_safety_product_team jobs to mw-cron - https://phabricator.wikimedia.org/T388542#10624241 (10Clement_Goubert) >>! In T388542#10623891, @Dreamy_Jazz wrote: > @Clement_Goubert I put the wrong name for the one owned by #community-tech. It's `mediawik... [15:22:37] 06serviceops, 06Content-Transform-Team, 13Patch-For-Review: Parsoid extension no longer loaded on parsoidtest1001 - https://phabricator.wikimedia.org/T388465#10624261 (10ssastry) 05In progress→03Resolved a:03Clement_Goubert parse.php on CLI works.Thanks! This shouldn't affect testreduce1002 .. so... [15:27:22] 06serviceops, 06Trust and Safety Product Team: Migrate trust_and_safety_product_team jobs to mw-cron - https://phabricator.wikimedia.org/T388542#10624300 (10Dreamy_Jazz) >>! In T388542#10624239, @Clement_Goubert wrote: >>>! In T388542#10623891, @Dreamy_Jazz wrote: >> @Clement_Goubert I put the wrong name for t... [15:49:22] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Update wikikube-staging-codfw to kubernetes 1.31 - https://phabricator.wikimedia.org/T384450#10624465 (10Jelto) @JMeybohm @kamila and I did a upgrade of `staging-codfw` to `1.31`. We used the `sre.k8s.wipe-cluste... [15:54:14] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, and 2 others: Ensure all required kubectl versions are installed on deploy hosts - https://phabricator.wikimedia.org/T388388#10624500 (10Jelto) a:05Jelto→03None I won't be able to finish this before my PTO, I did some exper... [16:08:09] 06serviceops, 06Infrastructure-Foundations, 10Maps (Kartotherian): Scale up Kartotherian on Wikikube and move live traffic to it - https://phabricator.wikimedia.org/T386926#10624612 (10elukey) I added two new graphs to the Kartotherian k8s dashboard, to monitor the heap used by Nodejs (avg and max between po... [16:24:21] 06serviceops, 10MediaWiki-extensions-CentralAuth, 10MW-on-K8s, 10MediaWiki-Platform-Team (Radar): Migrate CentralAuth maintenance jobs to mw-cron - https://phabricator.wikimedia.org/T385866#10624762 (10ArielGlenn) >>! In T385866#10624052, @Clement_Goubert wrote: ... > Cross-referencing with the list of tim... [16:27:07] 06serviceops, 10MediaWiki-extensions-CentralAuth, 10MW-on-K8s, 10MediaWiki-Platform-Team (Radar): Missing backfill_localaccounts periodic jobs - https://phabricator.wikimedia.org/T388564 (10Clement_Goubert) 03NEW [16:27:17] 06serviceops, 10MediaWiki-extensions-CentralAuth, 10MW-on-K8s, 10MediaWiki-Platform-Team (Radar): Missing backfill_localaccounts periodic jobs - https://phabricator.wikimedia.org/T388564#10624784 (10Clement_Goubert) 05Open→03In progress p:05Triage→03High [16:30:45] 06serviceops, 10MediaWiki-extensions-CentralAuth, 10MW-on-K8s, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Missing backfill_localaccounts periodic jobs - https://phabricator.wikimedia.org/T388564#10624810 (10Clement_Goubert) [16:31:31] 06serviceops, 13Patch-For-Review: Migrate mw-script to PHP 8.1 - https://phabricator.wikimedia.org/T387917#10624817 (10Scott_French) Alright, I've run puppet-agent on the deployment host and verified `mwscript-k8s` works as expected following the switch in release values file. Switching mw-web/main to PHP 8.1... [16:37:23] 06serviceops, 10MediaWiki-extensions-CentralAuth, 10MW-on-K8s, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Missing backfill_localaccounts periodic jobs - https://phabricator.wikimedia.org/T388564#10624852 (10Clement_Goubert) @ArielGlenn I've created the CR to add the jobs to production, howeve... [16:50:40] 06serviceops, 10MediaWiki-extensions-ReadingLists, 06MW-Interfaces-Team, 10RESTBase Sunsetting: Switchover plan from RESTbase to REST Gateway for Reading Lists endpoints - https://phabricator.wikimedia.org/T384891#10624911 (10HCoplin-WMF) Task itself looks good to me! One request is that we hold off on exe... [16:53:32] 06serviceops, 10Citoid, 06Editing QA, 06Editing-team, and 3 others: Switchover plan from restbase to api gateway for Citoid - https://phabricator.wikimedia.org/T361576#10624921 (10EAkinloose) [16:58:29] 06serviceops, 10MediaWiki-General: Migrate InitSiteStats job to mw-cron - https://phabricator.wikimedia.org/T388532#10624936 (10Pppery) [17:16:48] 06serviceops, 10MediaWiki-Special-pages: Migrate InitSiteStats job to mw-cron - https://phabricator.wikimedia.org/T388532#10625051 (10Pppery) (Sorry that this screws up your tracking, but the correct component for Special:Statistics is #mediawiki-special-pages) [17:18:12] 06serviceops, 10MediaWiki-Special-pages: Migrate updatequerypages/update_special_pages jobs to mw-cron - https://phabricator.wikimedia.org/T388534#10625056 (10Pppery) [17:18:43] 06serviceops, 10MediaWiki-Categories: Migrate recount_categories job to mw-cron - https://phabricator.wikimedia.org/T388533#10625086 (10Pppery) [17:20:51] 06serviceops, 10MediaWiki-Special-pages: Migrate updatequerypages/update_special_pages jobs to mw-cron - https://phabricator.wikimedia.org/T388534#10625111 (10Clement_Goubert) [17:21:42] 06serviceops, 10MediaWiki-Special-pages: Migrate updatequerypages/update_special_pages jobs to mw-cron - https://phabricator.wikimedia.org/T388534#10625122 (10Clement_Goubert) [17:29:44] 06serviceops, 10MediaWiki-Special-pages: Migrate InitSiteStats job to mw-cron - https://phabricator.wikimedia.org/T388532#10625175 (10Clement_Goubert) >>! In T388532#10625051, @Pppery wrote: > (Sorry that this screws up your tracking, but the correct component for Special:Statistics is #mediawiki-special-pages... [17:30:37] 06serviceops, 10MediaWiki-Special-pages: Migrate InitSiteStats job to mw-cron - https://phabricator.wikimedia.org/T388532#10625191 (10Pppery) Perhaps. No objection to closing as duplicate. [17:31:06] 06serviceops, 07Kubernetes: Add pod ip address blocks to staging-eqiad - https://phabricator.wikimedia.org/T386232#10625197 (10JMeybohm) staging-codfw switched to the new ip pool today (T384450) [17:31:56] 06serviceops, 10MediaWiki-Special-pages: Migrate InitSiteStats job to mw-cron - https://phabricator.wikimedia.org/T388532#10625212 (10Clement_Goubert) →14Duplicate dup:03T388534 [17:31:58] 06serviceops, 10MediaWiki-Special-pages: Migrate updatequerypages/update_special_pages jobs to mw-cron - https://phabricator.wikimedia.org/T388534#10625214 (10Clement_Goubert) [17:32:12] 06serviceops, 10MediaWiki-Special-pages: Migrate updatequerypages/update_special_pages/init_site_stats jobs to mw-cron - https://phabricator.wikimedia.org/T388534#10625219 (10Pppery) [17:32:32] 06serviceops, 10MediaWiki-Special-pages: Migrate updatequerypages/update_special_pages/initsitestats jobs to mw-cron - https://phabricator.wikimedia.org/T388534#10625231 (10Pppery) [20:02:26] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10626018 (10Jhancock.wm) [22:29:31] 06serviceops, 10MW-on-K8s: mw-script fails to render in CI - https://phabricator.wikimedia.org/T367200#10626704 (10RLazarus) 05Open→03Resolved a:03RLazarus No.