[00:24:41] Project beta-update-databases-eqiad build #67055: 04FAILURE in 4 min 40 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/67055/ [00:29:45] ^ seems to be caused by https://gerrit.wikimedia.org/r/c/mediawiki/core/+/915849/ [01:24:41] Yippee, build fixed! [01:24:41] Project beta-update-databases-eqiad build #67056: 09FIXED in 4 min 40 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/67056/ [02:48:03] 01:24:41 [f12a1323bf2df43ba81f4991] [no req] TypeError: Argument 2 passed to FileBackendGroup::__construct() must be an instance of Wikimedia\Rdbms\ConfiguredReadOnlyMode, instance of MediaWiki\DB\ConfiguredReadOnlyMode given, called in /srv/mediawiki-staging/php-master/includes/ServiceWiring.php on line 696 [02:48:14] zabe: Im guessing some kind of race on the server [02:48:17] seems to work fine now [02:48:31] there was no fixup patch, nothing changed [02:48:46] it's CLI so a bit strange, it's not like there's opcache involved. [03:55:56] (03CR) 10Lens0021: "Thank you! @hashar" [integration/config] - 10https://gerrit.wikimedia.org/r/910848 (owner: 10Lens0021) [04:19:30] 10Gerrit, 10ExtensionDistributor: gerrit-replica seems out of date breaking extension distributor - https://phabricator.wikimedia.org/T335730 (10Dzahn) replication from gerrit1003 is stopped and puppet is running, we ran fresh replication from gerrit1001. This shouldn't happen again. Soon we will switch to 10... [05:50:07] 10Phabricator: Phabricator is auto claiming tasks when closing as resolved (again) - https://phabricator.wikimedia.org/T134106 (10valerio.bozzolan) Premising that I see a wide margin of possible side-improvements. Just a few: 1. for experts, when fixing the auto-assign, decrease the number if weird related warn... [05:56:04] 10Phabricator: Phabricator is auto claiming tasks when closing as resolved (again) - https://phabricator.wikimedia.org/T134106 (10valerio.bozzolan) >>! In T134106#8828782, @valerio.bozzolan wrote: But, with or without that, PLEASE SET A DAMN ASSIGNEE ihih (Not because I want it, but because Phabricator/Phorge... [07:20:46] 10Phabricator: Phabricator is auto claiming tasks when closing as resolved (again) - https://phabricator.wikimedia.org/T134106 (10Michael) a:03Peachey88 [07:59:40] (03CR) 10Hashar: [C: 03+2] build: Updating eslint-config-wikimedia to 0.25.0 [integration/docroot] - 10https://gerrit.wikimedia.org/r/915858 (owner: 10Libraryupgrader) [09:28:01] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [09:30:41] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [09:36:56] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Add traffic sampling support to mw-on-k8s.lua ATS script - https://phabricator.wikimedia.org/T336038 (10Clement_Goubert) [09:39:57] 10Phabricator, 10CommTech-Wishathon: Extract Cycle Time from Phab boards - https://phabricator.wikimedia.org/T336039 (10KSiebert) [10:20:54] 10Release-Engineering-Team, 10API Platform, 10AQS2.0, 10Platform Engineering, and 5 others: Define a procedure/pattern to populate test environments - https://phabricator.wikimedia.org/T334851 (10Sfaci) Thank you @Eevans for clarifying that!. @Htriedman We are discussing about creating generic patterns/pr... [11:22:43] 10Phabricator: Phabricator is auto claiming tasks when closing as resolved (again) - https://phabricator.wikimedia.org/T134106 (10Peachey88) a:05Peachey88→03None Unassigning - nothing was actioned. [11:28:04] 10Phabricator: Phabricator is auto claiming tasks when closing as resolved (again) - https://phabricator.wikimedia.org/T134106 (10thiemowmde) 05Declined→03Open [11:45:04] 10Phabricator: Phabricator is auto claiming tasks when closing as resolved (again) - https://phabricator.wikimedia.org/T134106 (10Aklapper) @valerio.bozzolan: UI changes sound rather off-topic here. And comments like T134106#8828784 are just noise which add no value to the conversation. Can you please move that... [11:46:46] 10Phabricator: Phabricator is auto claiming tasks when closing as resolved (again) - https://phabricator.wikimedia.org/T134106 (10Aklapper) 05Open→03Declined I'll boldly decline this task as I currently don't plan to change the current behavior, given the provided arguments in previous comments. [12:04:38] 10Phabricator, 10CommTech-Wishathon: Extract Cycle Time from Phab boards - https://phabricator.wikimedia.org/T336039 (10Aklapper) See https://phabricator.wikimedia.org/maniphest/?ids=280999,143348,255703,148805#R for existing tasks and code. Maybe this task should be merged? [12:05:44] 10Phabricator: Phabricator is auto claiming tasks when closing as resolved (again) - https://phabricator.wikimedia.org/T134106 (10thiemowmde) 05Declined→03Open What arguments? > the person who closes a task […] takes the responsibility of this status, and so is the correct assignee of the task. > … > Peache... [12:32:21] 10Phabricator: Phabricator is auto claiming tasks when closing as resolved (again) - https://phabricator.wikimedia.org/T134106 (10Aklapper) 05Open→03Declined As a co-maintainer of the Wikimedia Phabricator software, for the time being I'd like to keep the current status quo in the Phabricator configuration,... [12:41:58] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥), 10API Platform, 10Anti-Harassment, and 19 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10BTullis) [13:08:34] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥), 10API Platform, 10Anti-Harassment, and 19 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10BTullis) I've confirmed information related to [[https://gerrit.wikimedia.org/g/analytics/datah... [13:10:32] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: Technical documentation - https://phabricator.wikimedia.org/T336058 (10KBach) [15:03:21] 10Phabricator, 10Release-Engineering-Team, 10SecTeam-Processed: Make it more difficult to unintentionally paste secrets, passwords, and keys in Phabricator task descriptions, comments and pastes - https://phabricator.wikimedia.org/T332639 (10kostajh) > Premising that to prevent human errors we just need to p... [15:07:44] (03PS1) 10Sohom Datta: Add 2023 VideoCutTool GSoC students [integration/config] - 10https://gerrit.wikimedia.org/r/916513 [15:09:41] (03CR) 10Sohom Datta: "These students will be working on VideoCutTool as part of the 2023 GSOC cohort and will require access to jenkins CI as part of their cont" [integration/config] - 10https://gerrit.wikimedia.org/r/916513 (owner: 10Sohom Datta) [15:16:36] (03CR) 10Gopavasanth: [C: 03+1] Add 2023 VideoCutTool GSoC students [integration/config] - 10https://gerrit.wikimedia.org/r/916513 (owner: 10Sohom Datta) [15:24:01] 10Gerrit, 10ExtensionDistributor: gerrit-replica seems out of date breaking extension distributor - https://phabricator.wikimedia.org/T335730 (10hashar) 05Open→03Resolved >>! In T335730#8827843, @Dzahn wrote: > Sorry about this, 2 gerrit hosts were fighting over the replication. > > So, the new host gerri... [15:28:32] 10Gerrit, 10Release-Engineering-Team, 10SRE, 10serviceops-collab: gerrit1003 service implementation task - https://phabricator.wikimedia.org/T326368 (10hashar) I wasn't aware about this task until yesterday (via T335730). I'd like the new host to be added first as a replica rather than an entirely new pri... [15:34:24] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥), 10API Platform, 10Anti-Harassment, and 19 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10isarantopoulos) Wow, sorry for the noise by referencing wrong ticket :) [15:40:35] maintenance-disconnect-full-disks build 488500 integration-agent-docker-1035 (/: 31%, /srv: 100%, /var/lib/docker: 45%): OFFLINE due to disk space [15:43:35] 10GitLab (Infrastructure), 10serviceops-collab, 10Patch-For-Review: Cookbook sre.gitlab.upgrade fails when unpausing runners - https://phabricator.wikimedia.org/T335855 (10eoghan) 05Open→03In progress a:03eoghan [15:46:37] maintenance-disconnect-full-disks build 488501 integration-agent-docker-1035 (/: 31%, /srv: 40%, /var/lib/docker: 43%): RECOVERY disk space OK [16:22:19] is it possible to search through jenkins logs for a specific phrase? e.g. i want to find all test failures that mention "Clicking on a watchstar toggles the watchstar" (i've seen this error a few times and i want to see how common it is) [16:25:24] MatmaRex: I gave it a try by looking at contint2001:/var/log/jenkins/jenkins.log* ..but I don't have anything to report, the term "watchstar" does not appear in it [16:25:56] e.g. https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php81-docker/6387/console is an example failure [16:25:58] it might need verbose error logging or so [16:26:18] sorry, i wasn't clear what "jenkins logs" i wanted to search [16:26:31] i want to search the things it dumps on gerrit :) [16:26:41] i guess that's still not clear, heh [16:26:53] gotcha [16:27:03] but don't have the answer .. hmm [16:29:16] MatmaRex: if you could turn this into a ticket, that would be great and get the answer it deserves. EU people will be already in Friday dinner now [16:30:23] i can probably find enough examples to complain about it manually, i was just hoping there was a way to save 5 minutes :) thanks [16:31:05] alright, cheers [16:33:05] MatmaRex: I also checked /var/log/zuul/ since that is like the bridge between jenkins and gerrit, but I could not find that error message.. tried. [16:35:33] maintenance-disconnect-full-disks build 488511 integration-agent-docker-1027 (/: 29%, /srv: 96%, /var/lib/docker: 51%): OFFLINE due to disk space [16:40:34] maintenance-disconnect-full-disks build 488512 integration-agent-docker-1027 (/: 29%, /srv: 30%, /var/lib/docker: 48%): RECOVERY disk space OK [16:44:54] (03PS1) 10Cparle: Add CirrusSearch to CI so unit tests can run [integration/config] - 10https://gerrit.wikimedia.org/r/916527 (https://phabricator.wikimedia.org/T322872) [16:50:36] 10Gerrit, 10ExtensionDistributor: gerrit-replica seems out of date breaking extension distributor - https://phabricator.wikimedia.org/T335730 (10Dzahn) >>! In T335730#8829781, @hashar wrote: > I have seen the message yesterday and as a result did no further investigation. It definitely explains the issue, I g... [16:53:37] (03PS2) 10Cparle: Add CirrusSearch to CI so unit tests can run [integration/config] - 10https://gerrit.wikimedia.org/r/916527 (https://phabricator.wikimedia.org/T322872) [16:55:01] (03CR) 10CI reject: [V: 04-1] Add CirrusSearch to CI so unit tests can run [integration/config] - 10https://gerrit.wikimedia.org/r/916527 (https://phabricator.wikimedia.org/T322872) (owner: 10Cparle) [16:55:11] (03PS3) 10Cparle: Add CirrusSearch to CI so unit tests can run [integration/config] - 10https://gerrit.wikimedia.org/r/916527 (https://phabricator.wikimedia.org/T322872) [17:06:03] (my failing test issue issue turns out to be already filed as https://phabricator.wikimedia.org/T334634) [17:42:46] (03CR) 10Krinkle: [C: 03+2] Add CirrusSearch to CI so unit tests can run [integration/config] - 10https://gerrit.wikimedia.org/r/916527 (https://phabricator.wikimedia.org/T322872) (owner: 10Cparle) [17:43:55] (03Merged) 10jenkins-bot: Add CirrusSearch to CI so unit tests can run [integration/config] - 10https://gerrit.wikimedia.org/r/916527 (https://phabricator.wikimedia.org/T322872) (owner: 10Cparle) [17:45:34] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/916527 [17:45:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:45:40] cormacparle: ^ [17:57:17] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥), 10Abstract Wikipedia team, 10function-evaluator: Move the mediawiki/services/function-evaluator repo from gerrit+pipeline to GitLab - https://phabricator.wikimedia.org/T334398 (10SDunlap) [17:57:24] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥), 10Abstract Wikipedia team, 10function-orchestrator, 10Patch-For-Review: Move the mediawiki/services/function-orchestrator repo from gerrit+pipeline to GitLab - https://phabricator.wikimedia.org/T334397 (10SDunlap) [17:57:59] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥), 10Abstract Wikipedia team, 10function-evaluator: Move the mediawiki/services/function-evaluator repo from gerrit+pipeline to GitLab - https://phabricator.wikimedia.org/T334398 (10SDunlap) 05Open→03Resolved [17:58:33] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥), 10API Platform, 10Anti-Harassment, and 19 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10SDunlap) [18:36:19] 10Gerrit, 10Release-Engineering-Team, 10SRE, 10serviceops-collab: gerrit1003 service implementation task - https://phabricator.wikimedia.org/T326368 (10Dzahn) [18:37:29] 10Release-Engineering-Team, 10API Platform, 10AQS2.0, 10Platform Engineering, and 5 others: Define a procedure/pattern to populate test environments - https://phabricator.wikimedia.org/T334851 (10Htriedman) > if I have understood right, the library you mentioned would use source data to create synthetic on... [19:18:38] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T330213 (10brennen) [19:50:54] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥), 10Abstract Wikipedia team, 10function-evaluator: Move the mediawiki/services/function-evaluator repo from gerrit+pipeline to GitLab - https://phabricator.wikimedia.org/T334398 (10SDunlap) [20:32:36] 10GitLab (Auth & Access), 10Release-Engineering-Team: Provide Access Token to allow SonarQube to access GitLab - https://phabricator.wikimedia.org/T336014 (10thcipriani) [20:45:35] maintenance-disconnect-full-disks build 488561 integration-agent-docker-1023 (/: 29%, /srv: 100%, /var/lib/docker: 64%): OFFLINE due to disk space [20:50:34] maintenance-disconnect-full-disks build 488562 integration-agent-docker-1023 (/: 29%, /srv: 49%, /var/lib/docker: 63%): RECOVERY disk space OK [21:25:32] maintenance-disconnect-full-disks build 488569 integration-agent-docker-1038 (/: 29%, /srv: 97%, /var/lib/docker: 53%): OFFLINE due to disk space [21:30:36] maintenance-disconnect-full-disks build 488570 integration-agent-docker-1038 (/: 29%, /srv: 15%, /var/lib/docker: 50%): RECOVERY disk space OK [21:40:52] maintenance-disconnect-full-disks build 488572 integration-agent-docker-1039 (/: 29%, /srv: 96%, /var/lib/docker: 48%): OFFLINE due to disk space [21:45:39] maintenance-disconnect-full-disks build 488573 integration-agent-docker-1038 (/: 29%, /srv: 99%, /var/lib/docker: 53%): OFFLINE due to disk space [21:45:39] maintenance-disconnect-full-disks build 488573 integration-agent-docker-1039 (/: 29%, /srv: 35%, /var/lib/docker: 45%): RECOVERY disk space OK [21:50:43] maintenance-disconnect-full-disks build 488574 integration-agent-docker-1038 (/: 29%, /srv: 16%, /var/lib/docker: 50%): RECOVERY disk space OK [22:04:38] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [gadgets/ProveIt] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/916621 [22:04:40] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [gadgets/ProveIt] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/916621 (owner: 10QChris) [22:05:04] (03PS1) 10QChris: Import done. Revoke import grants [gadgets/ProveIt] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/916622 [22:05:06] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [gadgets/ProveIt] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/916622 (owner: 10QChris) [22:25:06] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T330213 (10Krinkle) [22:37:28] (03PS1) 10QChris: Zuul: Switch to new repo name of ProveIt gadget [integration/config] - 10https://gerrit.wikimedia.org/r/916634 [22:43:41] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T330213 (10brennen) 05Open→03In progress With blockers cleared, I'm going to say we'll go to all wikis first thing (MDT) Monday morning. [22:59:01] 10Gerrit, 10serviceops-collab, 10Patch-For-Review: add new and remove old Gerrit service IP from netbox and DNS - https://phabricator.wikimedia.org/T334524 (10Dzahn) sorry, wrong ticket linked [22:59:08] (03CR) 10Krinkle: [C: 03+2] Zuul: Switch to new repo name of ProveIt gadget [integration/config] - 10https://gerrit.wikimedia.org/r/916634 (owner: 10QChris) [23:00:15] (03Merged) 10jenkins-bot: Zuul: Switch to new repo name of ProveIt gadget [integration/config] - 10https://gerrit.wikimedia.org/r/916634 (owner: 10QChris) [23:39:04] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/916634 [23:39:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL