[07:11:20] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: mw1492.eqiad.wmnet is down - https://phabricator.wikimedia.org/T338566 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host mw1492.eqiad.wmnet with OS buster [07:45:52] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: mw1492.eqiad.wmnet is down - https://phabricator.wikimedia.org/T338566 (10elukey) @Papaul thanks! I confirm that it works :) I think that there is only one thing to do, namely update the documentation (https://wikitech.wikimedia.org/wiki/Management_Interfaces#Di... [08:52:59] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: mw1492.eqiad.wmnet is down - https://phabricator.wikimedia.org/T338566 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host mw1492.eqiad.wmnet with OS buster completed: - mw1492 (**WARN**) - Downtimed on Icinga/Alertm... [08:54:18] hello folks, mw1492 is ready [08:54:34] reimaged to buster, all good afaics [08:54:39] it needs to be repooled etc.. [08:54:45] do you want to check or should I proceed? [08:55:52] (IIRC scap pull + contctl pool) [08:56:08] elukey: You can even do everything from the server itself [08:56:16] ah yes right [08:56:16] scap pull; sudo pool [08:56:29] all right done :) [08:57:28] Thanks <3 [08:59:24] elukey: removing downtime [09:34:03] 10serviceops, 10API Platform (RESTbase Deprecation Roadmap), 10Patch-For-Review: Migrate node-based services in production to node16 - https://phabricator.wikimedia.org/T308371 (10Jgiannelos) [09:34:07] 10serviceops, 10SRE, 10API Platform (RESTbase Deprecation Roadmap), 10Patch-For-Review: Migrate node-based services in production to node14 - https://phabricator.wikimedia.org/T306995 (10Jgiannelos) [09:37:18] 10serviceops, 10MW-on-K8s: Coordinate testing of testwiki on kubernetes - https://phabricator.wikimedia.org/T337489 (10Clement_Goubert) @dom_walden Do you know how long you and your team would need `testwiki` to be on k8s to run your full test suite? [09:38:33] 10serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: Migrate Wikitech to Kubernetes - https://phabricator.wikimedia.org/T292707 (10Ladsgroup) To expand upon what Manuel said, we have manually special-case wikitech in a lot of places which is a lot of work and when we forget to, it usually end up as an incide... [09:38:37] 10serviceops, 10MW-on-K8s, 10Quality-and-Test-Engineering-Team (QTE): Coordinate testing of testwiki on kubernetes - https://phabricator.wikimedia.org/T337489 (10Clement_Goubert) [09:42:48] 10serviceops, 10MW-on-K8s, 10Quality-and-Test-Engineering-Team (QTE): Run QTE test suite on testwiki on kubernetes - https://phabricator.wikimedia.org/T337489 (10Clement_Goubert) a:03Clement_Goubert [10:38:12] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Backport mediawiki chart mesh changes - https://phabricator.wikimedia.org/T338210 (10Clement_Goubert) 05Open→03Resolved [10:38:19] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Gracefully handle pod termination in mw-on-k8s - https://phabricator.wikimedia.org/T331609 (10Clement_Goubert) [11:33:54] 10serviceops, 10SRE, 10Wikimedia-Site-requests, 10Performance-Team (Radar): Raise limit of $wgMaxArticleSize for Hebrew Wikisource - https://phabricator.wikimedia.org/T275319 (10Fuzzy) p:05Medium→03High We hit the fan once again with the Israeli [[ https://he.wikisource.org/wiki/פקודת_מס_הכנסה | Income... [11:42:28] FYI, parse1002 is feeling unwell and that's blocking scap, I pooled it inactive and am in the process of powercycling it (it wasn't responding to console either) [11:54:44] 10serviceops, 10RESTbase Sunsetting, 10Epic, 10Platform Engineering Roadmap: Replace usage of RESTbase parsoid endpoints - https://phabricator.wikimedia.org/T328559 (10daniel) [11:55:50] 10serviceops, 10RESTbase Sunsetting, 10Epic, 10Platform Engineering Roadmap: Replace usage of RESTbase parsoid endpoints - https://phabricator.wikimedia.org/T328559 (10daniel) [12:09:19] 10serviceops, 10MW-on-K8s, 10Quality-and-Test-Engineering-Team (QTE), 10Patch-For-Review: Run QTE test suite on testwiki on kubernetes - https://phabricator.wikimedia.org/T337489 (10dom_walden) >>! In T337489#8934086, @Clement_Goubert wrote: > @dom_walden Do you know how long you and your team would need `... [12:21:22] 10serviceops, 10MW-on-K8s, 10Quality-and-Test-Engineering-Team (QTE), 10Patch-For-Review: Run QTE test suite on testwiki on kubernetes - https://phabricator.wikimedia.org/T337489 (10Clement_Goubert) >>! In T337489#8934607, @dom_walden wrote: >>>! In T337489#8934086, @Clement_Goubert wrote: >> @dom_walden D... [13:19:26] 10serviceops, 10MW-on-K8s, 10Quality-and-Test-Engineering-Team (QTE), 10Patch-For-Review: Run QTE test suite on testwiki on kubernetes - https://phabricator.wikimedia.org/T337489 (10dom_walden) >>! In T337489#8934625, @Clement_Goubert wrote: >>>! In T337489#8934607, @dom_walden wrote: >>>>! In T337489#8934... [14:13:23] 10serviceops, 10RESTbase Sunsetting, 10Epic, 10Platform Engineering Roadmap: Replace usage of RESTbase parsoid endpoints - https://phabricator.wikimedia.org/T328559 (10daniel) [15:04:24] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: mw1492.eqiad.wmnet is down - https://phabricator.wikimedia.org/T338566 (10Clement_Goubert) 05Open→03Resolved Host is back in pool, resolving. [15:30:33] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: mw1492.eqiad.wmnet is down - https://phabricator.wikimedia.org/T338566 (10elukey) 05Resolved→03Open >>! In T338566#8933725, @elukey wrote: > @Papaul thanks! I confirm that it works :) > > I think that there is only one thing to do, namely update the document... [15:35:13] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: mw1492.eqiad.wmnet is down - https://phabricator.wikimedia.org/T338566 (10Clement_Goubert) >>! In T338566#8935469, @elukey wrote: >>>! In T338566#8933725, @elukey wrote: >> @Papaul thanks! I confirm that it works :) >> >> I think that there is only one thing to... [15:38:38] 10serviceops, 10SRE, 10Wikimedia-Site-requests, 10Performance-Team (Radar): Raise limit of $wgMaxArticleSize for Hebrew Wikisource - https://phabricator.wikimedia.org/T275319 (10Dovi) I concur with [[User:Fuzzy]]; a direct solution to this is needed on Hebrew Wikisource. [16:05:32] 10serviceops: Publish Wikimedia bookworm base Docker image - https://phabricator.wikimedia.org/T335560 (10MoritzMuehlenhoff) This still fails with errors related to snapshot.debian.org ` Jun 15 15:39:09 build2001 build-base-images[2139003]: W: Couldn't download package libxxhash0 (ver 0.8.1-1 arch amd64) at h... [16:26:00] 10serviceops, 10SRE, 10Wikimedia-Site-requests, 10Performance-Team (Radar): Raise limit of $wgMaxArticleSize for Hebrew Wikisource - https://phabricator.wikimedia.org/T275319 (10Nahum) The Income Tax Ordinance requires a temporrary immediate solution while we continue to ponder the best permanent one. [17:00:57] 10serviceops, 10GitLab (CI & Job Runners), 10Patch-For-Review, 10Release-Engineering-Team (They Live 🕶️🧟): Provide ability to tag GitLab CI built images with a datetime format, set as default in pipeline-to-gitlab conversion - https://phabricator.wikimedia.org/T338224 (10brennen) [17:17:03] 10serviceops, 10SRE, 10Wikimedia-Site-requests, 10Performance-Team (Radar): Raise limit of $wgMaxArticleSize for Hebrew Wikisource - https://phabricator.wikimedia.org/T275319 (10Aklapper) p:05High→03Medium [[ https://www.mediawiki.org/wiki/Phabricator/Project_management#Setting_task_priorities | The Pr... [22:23:02] 10serviceops, 10GitLab (CI & Job Runners), 10Patch-For-Review, 10Release-Engineering-Team (They Live 🕶️🧟): Provide ability to tag GitLab CI built images with a datetime format, set as default in pipeline-to-gitlab conversion - https://phabricator.wikimedia.org/T338224 (10CodeReviewBot) dancy updated https:... [22:28:18] 10serviceops, 10GitLab (CI & Job Runners), 10Patch-For-Review, 10Release-Engineering-Team (They Live 🕶️🧟): Provide ability to tag GitLab CI built images with a datetime format, set as default in pipeline-to-gitlab conversion - https://phabricator.wikimedia.org/T338224 (10dancy) 05Open→03In progress p:...