[00:09:28] (03PS1) 10Krinkle: zuul: Add CI jobs for cvn-clerkbot [integration/config] - 10https://gerrit.wikimedia.org/r/879972 [00:09:38] (03CR) 10Krinkle: [C: 03+2] zuul: Add CI jobs for cvn-clerkbot [integration/config] - 10https://gerrit.wikimedia.org/r/879972 (owner: 10Krinkle) [00:10:45] (03Merged) 10jenkins-bot: zuul: Add CI jobs for cvn-clerkbot [integration/config] - 10https://gerrit.wikimedia.org/r/879972 (owner: 10Krinkle) [00:19:23] !log Reloading Zuul [00:19:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:36:12] 10Release-Engineering-Team, 10Desktop Improvements (Vector 2022), 10Language-Team, 10SRE, and 3 others: Unexpected "Page contents not supported in other languages" in non-article namespace - https://phabricator.wikimedia.org/T326788 (10thcipriani) >>! In T326788#8526494, @RhinosF1 wrote: > Hi, > > Can som... [00:48:31] 10Release-Engineering-Team, 10Desktop Improvements (Vector 2022), 10Language-Team, 10SRE, and 3 others: Unexpected "Page contents not supported in other languages" in non-article namespace - https://phabricator.wikimedia.org/T326788 (10Ladsgroup) I'm around in my capacity as SRE. Going to help on this. [01:27:10] 10Release-Engineering-Team, 10Desktop Improvements (Vector 2022), 10Language-Team, 10SRE, and 4 others: Unexpected "Page contents not supported in other languages" in non-article namespace - https://phabricator.wikimedia.org/T326788 (10Ladsgroup) I'd appreciate an incident report on this: https://wikitech.... [02:06:33] 10Release-Engineering-Team, 10Desktop Improvements (Vector 2022), 10Language-Team, 10SRE, and 3 others: Unexpected "Page contents not supported in other languages" in non-article namespace - https://phabricator.wikimedia.org/T326788 (10Func) 05In progress→03Open [02:06:37] 10Release-Engineering-Team, 10Desktop Improvements (Vector 2022), 10Language-Team, 10SRE, and 3 others: Unexpected "Page contents not supported in other languages" in non-article namespace - https://phabricator.wikimedia.org/T326788 (10Func) p:05Unbreak!→03High Lower the priority since backported. [02:14:53] 10Release-Engineering-Team, 10Desktop Improvements (Vector 2022), 10Language-Team, 10SRE, and 3 others: Unexpected "Page contents not supported in other languages" in non-article namespace - https://phabricator.wikimedia.org/T326788 (10Ladsgroup) The deployment went way bumpier than anticipated. I assume a... [02:29:54] 10Scap, 10serviceops: Scap deploy failed to depool codfw servers - https://phabricator.wikimedia.org/T327041 (10thcipriani) [02:30:23] 10Scap, 10serviceops: Scap deploy failed to depool codfw servers - https://phabricator.wikimedia.org/T327041 (10thcipriani) p:05Triage→03Unbreak! Setting as UBN! since it blocks deploys. [02:32:54] 10Scap, 10conftool, 10serviceops: Scap deploy failed to depool codfw servers - https://phabricator.wikimedia.org/T327041 (10Ladsgroup) Tentatively [02:42:16] 10Scap, 10conftool, 10serviceops: Scap deploy failed to depool codfw servers - https://phabricator.wikimedia.org/T327041 (10Ladsgroup) Copying what I wrote in the ops@ email: This really looks like something like https://wikitech.wikimedia.org/wiki/Incidents/2022-09-08_codfw_appservers_degradation, even err... [03:10:42] 10Gerrit, 10SRE, 10serviceops-collab: move gerrit.wm.org SSH service to private/behind LVS like phab-vcs - https://phabricator.wikimedia.org/T165631 (10demon) >>! In T165631#8485123, @Dzahn wrote: > @ayounsi It would mean a considerable effort to recreate an entire LVS service, which we just recently shut do... [03:29:34] (03PS1) 10Krinkle: Add CI jobs for countervandalism/stillalive [integration/config] - 10https://gerrit.wikimedia.org/r/879975 [03:30:43] (03PS2) 10Krinkle: zuul: Add CI jobs for countervandalism/stillalive [integration/config] - 10https://gerrit.wikimedia.org/r/879975 [03:30:59] (03CR) 10Krinkle: [C: 03+2] zuul: Add CI jobs for countervandalism/stillalive [integration/config] - 10https://gerrit.wikimedia.org/r/879975 (owner: 10Krinkle) [03:32:04] (03Merged) 10jenkins-bot: zuul: Add CI jobs for countervandalism/stillalive [integration/config] - 10https://gerrit.wikimedia.org/r/879975 (owner: 10Krinkle) [06:19:24] 10Scap, 10conftool, 10serviceops: Scap deploy failed to depool codfw servers - https://phabricator.wikimedia.org/T327041 (10taavi) One of the codfw LVS servers is down due to {T327001}, maybe that is causing this? [06:36:38] 10Scap, 10conftool, 10serviceops: Scap deploy failed to depool codfw servers - https://phabricator.wikimedia.org/T327041 (10Joe) Yes, that's definitely the reason why this is happening: lvs2008 is down since saturday (along with all rack b2, see T327001). That's what's causing the issues as the script still... [07:00:18] 10Scap, 10conftool, 10serviceops: Scap deploy failed to depool codfw servers - https://phabricator.wikimedia.org/T327041 (10Joe) Sadly, the root cause is indeed b2 being down, but the problem is that **lvs2009 at the moment cannot reach any servers in row b**. This means that lvs2009 sees a good third of al... [08:26:19] (03CR) 10Hashar: "Note it still has a bunch of open changes which will need to be abandoned and moved https://gerrit.wikimedia.org/r/q/project:mediawiki/too" [integration/config] - 10https://gerrit.wikimedia.org/r/879907 (https://phabricator.wikimedia.org/T290260) (owner: 10Jeena Huneidi) [10:48:25] 10GitLab (Infrastructure), 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: GitLab minor version upgrade 15.7.3 - https://phabricator.wikimedia.org/T326815 (10Jelto) p:05Triage→03Medium [11:01:21] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10serviceops-collab: Create cleanup policy for buildkit cache - https://phabricator.wikimedia.org/T327060 (10Jelto) [11:05:25] 10GitLab (CI & Job Runners), 10serviceops-collab: runner-1026.gitlab-runners.eqiad1.wikimedia.cloud ran out of disk space - https://phabricator.wikimedia.org/T326378 (10Jelto) 05Open→03Resolved >>! In T326378#8509591, @dancy wrote: > @Jelto There's still an outstanding matter of tuning the size of buildkit... [11:09:06] 10Release-Engineering-Team, 10Desktop Improvements (Vector 2022), 10SRE, 10UniversalLanguageSelector, and 4 others: Unexpected "Page contents not supported in other languages" in non-article namespace - https://phabricator.wikimedia.org/T326788 (10Nikerabbit) [12:39:34] 10Scap, 10Parsoid, 10SRE, 10serviceops: scap groups on bastions still needed? - https://phabricator.wikimedia.org/T327066 (10MoritzMuehlenhoff) [13:40:01] 10Continuous-Integration-Infrastructure, 10serviceops-collab: migrate doc hosts to bullseye - https://phabricator.wikimedia.org/T319477 (10LSobanski) [13:48:56] 10GitLab (Auth & Access), 10Infrastructure-Foundations, 10SRE, 10serviceops-collab, 10CAS-SSO: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10Jelto) According to [GitLab deprecations list](https://docs.gitlab.com/ee/update/deprecations.html#cas-omniauth-provider)... [13:58:02] 10MediaWiki-Releasing, 10TemplateStyles, 10MW-1.40-release: Bundle Extension:TemplateStyles with MediaWiki core - https://phabricator.wikimedia.org/T327006 (10Bugreporter2) p:05Triage→03Medium [14:35:42] 10Diffusion: phabricator is slow to update mirrors - https://phabricator.wikimedia.org/T154172 (10Aklapper) 05Open→03Declined Declining nowadays, as we do not want any deployment repos to use Phab anymore as a canonical code location. [14:58:18] 10Continuous-Integration-Infrastructure, 10Jenkins, 10SRE, 10SRE-Access-Requests, and 2 others: New Keyholder identity for RelEng Jenkins service - https://phabricator.wikimedia.org/T324014 (10jnuche) Awesome, I think that covers everything we need. Should be able to try it out soon. Thanks a lot @dzahn! [15:27:14] 10Gerrit: ReplicationTasksStorage : Error while renaming task - https://phabricator.wikimedia.org/T327084 (10hashar) [16:22:56] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10serviceops-collab: Create cleanup policy for buildkit cache - https://phabricator.wikimedia.org/T327060 (10LSobanski) p:05Triage→03Medium [17:40:49] (03PS1) 10Pwangai: Zuul: [mediawiki/services/parsoid] Add service to codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/880503 (https://phabricator.wikimedia.org/T321837) [17:53:23] 10Phabricator: Incorrectly attributed as the author of a random commit I have nothing to do with in a repository I've never touched - https://phabricator.wikimedia.org/T327105 (10ashley) [19:04:44] 10GitLab (CI & Job Runners), 10mwbot-rs: Unable to run Rust coverage reports in GitLab CI - https://phabricator.wikimedia.org/T325947 (10Legoktm) [19:04:56] 10GitLab (CI & Job Runners): Allow variants of rustlang/rust:nightly images on GitLab CI - https://phabricator.wikimedia.org/T326515 (10Legoktm) 05Open→03Resolved a:03Legoktm [19:33:36] (03PS1) 10Ssingh: zuul: configure CI for operations/debs/cadvisor [integration/config] - 10https://gerrit.wikimedia.org/r/880529 (https://phabricator.wikimedia.org/T325557) [20:04:52] 10Scap, 10Parsoid, 10SRE, 10serviceops: scap groups on bastions still needed? - https://phabricator.wikimedia.org/T327066 (10Arlolra) > Bastionhosts are used by parsoid deployers to restart parsoid machines and they use the dsh groups that are maintained in scap::dsh. https://github.com/wikimedia/puppet/c... [20:35:04] 10Diffusion, 10Phabricator: Incorrectly attributed as the author of a random commit I have nothing to do with in a repository I've never touched - https://phabricator.wikimedia.org/T327105 (10Aklapper) Indeed. Corresponding (mirrored) Gerrit patch: https://gerrit.wikimedia.org/r/c/wikidata/query/gui/+/866822 [20:42:08] 10Diffusion, 10Phabricator: Incorrectly attributed as the author of a random commit I have nothing to do with in a repository I've never touched - https://phabricator.wikimedia.org/T327105 (10Legoktm) The commit has the metadata of: ` Parent: 6ec26b23 (Remove outdated comment for download()) Author: Ad... [21:53:49] 10Diffusion, 10Phabricator: Incorrectly attributed as the author of a random commit I have nothing to do with in a repository I've never touched - https://phabricator.wikimedia.org/T327105 (10Dylsss) 05Open→03Resolved a:03Dylsss It was assigned to the correct account, but a spam account changed it here:... [22:06:04] (03PS1) 10Zabe: Add AbuseFilter to phan dependencies of WikimediaMaintenance [integration/config] - 10https://gerrit.wikimedia.org/r/880540 [22:12:07] (03PS1) 10Krinkle: dockerfiles: Add dotnet-mono612 image for CVNBot CI job [integration/config] - 10https://gerrit.wikimedia.org/r/880541 [22:12:09] (03PS1) 10Krinkle: zuul: Add CI job for countervandalism/CVNBot [integration/config] - 10https://gerrit.wikimedia.org/r/880542 [22:13:08] (03CR) 10Krinkle: "@AntiComposite: For CVN-side awareness. @Taavi: For awareness in relation to https://phabricator.wikimedia.org/T311466 where you built a r" [integration/config] - 10https://gerrit.wikimedia.org/r/880541 (owner: 10Krinkle) [22:26:25] (03CR) 10Krinkle: [C: 03+2] dockerfiles: Add dotnet-mono612 image for CVNBot CI job [integration/config] - 10https://gerrit.wikimedia.org/r/880541 (owner: 10Krinkle) [22:27:33] (03Merged) 10jenkins-bot: dockerfiles: Add dotnet-mono612 image for CVNBot CI job [integration/config] - 10https://gerrit.wikimedia.org/r/880541 (owner: 10Krinkle) [22:29:31] !log Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/880541 [22:29:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:47:27] (03CR) 10Krinkle: [C: 03+2] "> INFO:jenkins_jobs.cli.subcommand.update:Updating jobs in ['jjb'] (['cvnbot*'])" [integration/config] - 10https://gerrit.wikimedia.org/r/880542 (owner: 10Krinkle) [22:48:34] (03Merged) 10jenkins-bot: zuul: Add CI job for countervandalism/CVNBot [integration/config] - 10https://gerrit.wikimedia.org/r/880542 (owner: 10Krinkle) [22:51:32] "SUCCESS in 26s" – https://gerrit.wikimedia.org/r/c/labs/countervandalism/CVNBot/+/880545 [22:51:36] and that's with a cold docker image cache [22:51:53] end to end in 26 seconds. Yeah, our CI is awesome [22:52:17] for a C# dotnet project with msbuild. It's supposed to be slow. [23:59:29] (03PS2) 10Jforrester: Zuul: [mediawiki/tools/release] Mark as archived, moved to GitLab [integration/config] - 10https://gerrit.wikimedia.org/r/879907 (https://phabricator.wikimedia.org/T290260) (owner: 10Jeena Huneidi)