[08:18:10] 10Release-Engineering-Team, 10Quibble: Run Quibble Mariadb server with innodb_print_all_deadlocks - https://phabricator.wikimedia.org/T342088 (10hashar) a:03hashar This will be in the next Quibble version after 1.5.5. The deadlock will be emitted in the MySQL error log which will be attached to the builds as... [08:55:35] I'd like to reboot contint1002 in half an hour (9:30 UTC). This should be the passive instance as far as I can tell [09:18:27] jelto: it is actively running a zuul merger process, but iirc it gracefully shut down when the host is rebooted [09:18:39] at least I assume the systemd unit is made to properly shut down that service [09:19:06] worse case, some on the fly merge is aborted which leads to a CI build to fail and that can always be restarted [09:19:15] in short +1 on reboot [09:19:45] we can do contint2002 in our morning when CI is quiet ( https://integration.wikimedia.org/zuul/ gives a hint as how busy it is) [09:20:25] hashar: great thanks for the context. Then I'll proceed with the reboot in 10 minutes [09:20:32] +1 [09:20:58] hashar: regarding contint2002, yes I agree. How about tomorrow morning? Does that work for you? [09:21:23] sure :] [09:23:58] hashar: great thanks. There is a deploy window tomorrow 07:00-08:00 UTC. Do you prefere before or after that? [09:24:48] after the backport window, I am unlikely to be around at 7:00 UTC (kids are still at home this week :D ) [09:25:43] ok that works for me too [09:30:36] I'm going to reboot contint1002 now [09:35:18] (03CR) 10Hashar: [C: 03+2] "The way I do it is:" [integration/quibble] - 10https://gerrit.wikimedia.org/r/952224 (owner: 10Krinkle) [09:37:54] done [09:39:58] hashar: so we aim for 08:00-08:15 UTC for contint2002 reboot tomorrow? [09:40:13] sounds good to me thanks :) [09:41:32] great. I'll write a short mail to ops, wikitech-l and engineering-all? [09:42:16] if you want yes, doesn't hurt to inform everyone [09:55:41] (03Merged) 10jenkins-bot: docs: Document how to test Quibble changes [integration/quibble] - 10https://gerrit.wikimedia.org/r/952224 (owner: 10Krinkle) [10:12:10] 10Release-Engineering-Team (Priority Backlog 📥), 10collaboration-services: Adopt gitlab-cloud-runner Digital Ocean based cluster as the default for untagged jobs - https://phabricator.wikimedia.org/T344874 (10Jelto) [11:07:15] 10GitLab (Pipeline Services Migration🐤), 10Language-Team: Migrate machinetranslation/cxserver/apertium to GitLab - https://phabricator.wikimedia.org/T344737 (10Nikerabbit) [11:08:14] 10GitLab (Pipeline Services Migration🐤), 10Language-Team: Migrate machinetranslation/cxserver/apertium to GitLab - https://phabricator.wikimedia.org/T344737 (10Nikerabbit) p:05Triage→03Medium [12:06:20] 10Release-Engineering-Team (Priority Backlog 📥), 10Epic, 10Patch-For-Review, 10Release Pipeline (Blubber): Upgrade docker on integration hosts for fixes to BuildKit builder - https://phabricator.wikimedia.org/T318382 (10MoritzMuehlenhoff) >>! In T318382#9118776, @dduvall wrote: >>>! In T318382#8377321, @dd... [12:49:18] 10Continuous-Integration-Infrastructure, 10Abstract Wikipedia team, 10ci-test-error: CI for WikiLambda broken due to npm cache bug (?) on some @types packages - https://phabricator.wikimedia.org/T339812 (10hashar) Might be related, as part of T340092, we now run `npm cache verify` before saving the cache in... [13:05:26] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Quibble, 10castor: Figure out how to garbage collect the npm cache - https://phabricator.wikimedia.org/T340092 (10hashar) After a couple months: ` name=integration-castor05 $ du -s -m /srv/castor/*/master/*/npm|sort -n|tail -n10 390 /sr... [13:31:54] 10Continuous-Integration-Infrastructure, 10Abstract Wikipedia team, 10ci-test-error: CI for WikiLambda broken due to npm cache bug (?) on some @types packages - https://phabricator.wikimedia.org/T339812 (10Jdforrester-WMF) >>! In T339812#9123262, @hashar wrote: > Might be related, as part of T340092, we now... [13:50:44] 10Phabricator (Upstream), 10Upstream: Phabricator search dropdown results should open in new tab on middle-click - https://phabricator.wikimedia.org/T76812 (10Pols12) 05Open→03Resolved Fixed, maybe while migrating to Phorge. [14:05:51] 10Phabricator (Upstream), 10Upstream: Phabricator search dropdown results should open in new tab on middle-click - https://phabricator.wikimedia.org/T76812 (10Pols12) a:03valerio.bozzolan {T333885} included [[https://we.phorge.it/D25069?id=324 | a patch]] for [[https://we.phorge.it/T15149 | Phorge:T15149]] b... [14:11:05] 10Phabricator (Upstream), 10Upstream: Phabricator search dropdown results should open in new tab on middle-click - https://phabricator.wikimedia.org/T76812 (10valerio.bozzolan) You're welcome! Yeah that was really frustrating :) Just to clarify that this was the exact patch: https://we.phorge.it/D25069 - spoi... [14:48:58] 10GitLab (CI & Job Runners): Allow GitLab CI containers to connect to services - https://phabricator.wikimedia.org/T344818 (10kostajh) @dduvall @jnuche and I discussed this a bit on IRC last week, copying the relevant lines here: `lang=irc kostajh, jnuche: you can use gitlab ci services, but not with... [15:29:31] 10GitLab (CI & Job Runners): Allow GitLab CI containers to connect to services - https://phabricator.wikimedia.org/T344818 (10kostajh) Work-in-progress at https://gitlab.wikimedia.org/repos/mediawiki/services/ipoid/-/merge_requests/44 [15:39:05] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10SecTeam-Processed, 10Security, 10user-sbassett: [EPIC] Application Security Pipeline Components for Gitlab - Phase 2 Work - https://phabricator.wikimedia.org/T342177 (10sbassett) [15:39:07] 10Release-Engineering-Team (Radar), 10Anti-Harassment, 10Security-Team, 10iPoid-Service: Use Gitlab Security Pipeline for ipoid - https://phabricator.wikimedia.org/T338238 (10sbassett) [15:49:21] 10Phabricator: New post for Phabricating Phabricator - https://phabricator.wikimedia.org/T340223 (10valerio.bozzolan) [16:13:16] 10Phabricator: New post for Phabricating Phabricator - https://phabricator.wikimedia.org/T340223 (10valerio.bozzolan) [16:20:14] 10GitLab (Pipeline Services Migration🐤), 10Image-Suggestion-API, 10Structured-Data-Backlog: Migrate image-suggestion-api to GitLab - https://phabricator.wikimedia.org/T344740 (10Cparle) @BPirkle we can probably just archive this, right? [16:43:38] (03CR) 10Michael Große: Run EntitySchema CI on master once a day (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/948558 (https://phabricator.wikimedia.org/T333660) (owner: 10Michael Große) [16:52:42] 10GitLab (Pipeline Services Migration🐤), 10Growth-Team, 10Image-Suggestion-API, 10Structured-Data-Backlog: Migrate image-suggestion-api to GitLab - https://phabricator.wikimedia.org/T344740 (10BPirkle) No objection here. But [[ https://wikimedia.slack.com/archives/C01DFVAQRGA/p1682362901575589 | last time... [16:54:15] 10Release-Engineering-Team, 10Quibble: Update Quibble setuptools check to modern alternative - https://phabricator.wikimedia.org/T345093 (10hashar) [16:59:47] (03CR) 10Michael Große: Run EntitySchema CI on master once a day (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/948558 (https://phabricator.wikimedia.org/T333660) (owner: 10Michael Große) [17:04:14] (03CR) 10Michael Große: "On the other hand, now that I think about it, given that the goal is only "Run EntitySchema tests once a day and send an email if they fai" [integration/config] - 10https://gerrit.wikimedia.org/r/948558 (https://phabricator.wikimedia.org/T333660) (owner: 10Michael Große) [17:18:52] 10GitLab (Upstream pit of despair 🕳️), 10Release-Engineering-Team (Radar), 10Upstream, 10User-brennen: Investigate whether issues, operations, wikis, etc. can be disabled globally on GitLab - https://phabricator.wikimedia.org/T264231 (10hashar) >>! In T264231#7392442, @brennen wrote: >> The sidebar distrac... [17:23:40] 10Release-Engineering-Team (Priority Backlog 📥), 10Epic, 10Patch-For-Review, 10Release Pipeline (Blubber): Upgrade docker on integration hosts for fixes to BuildKit builder - https://phabricator.wikimedia.org/T318382 (10dduvall) 05Open→03Resolved thanks, @MoritzMuehlenhoff ! Closed [17:23:42] 10Release-Engineering-Team (Priority Backlog 📥), 10Epic, 10Release Pipeline (Blubber): Refactor PipelineLib to use blubber's buildkit frontend - https://phabricator.wikimedia.org/T318290 (10dduvall) [18:45:31] (Queue (Jenkins jobs + Zuul functions) alert) firing: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [18:50:41] 10Phabricator: Component or Project ConfirmAccount is gone - https://phabricator.wikimedia.org/T345106 (10Bugreporter) [18:53:45] PROBLEM - Work requests waiting in Zuul Gearman server on contint2002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [19:05:31] (Queue (Jenkins jobs + Zuul functions) alert) resolved: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [19:29:39] RECOVERY - Work requests waiting in Zuul Gearman server on contint2002 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [19:52:21] 10Phabricator: Component or Project ConfirmAccount is gone - https://phabricator.wikimedia.org/T345106 (10Peachey88) 05Open→03Resolved a:03Peachey88 Fixed, It's been renamed in the past so I suspect it was hit with {T224662} [19:59:26] 10GitLab (CI & Job Runners): Allow GitLab CI containers to connect to services - https://phabricator.wikimedia.org/T344818 (10CodeReviewBot) kharlan updated https://gitlab.wikimedia.org/repos/mediawiki/services/ipoid/-/merge_requests/44 gitlab: Build image to connect to mariadb service [20:46:12] 10GitLab: GitLab Private Repository Request for: - https://phabricator.wikimedia.org/T345113 (10calbon) [21:15:23] 10GitLab (Account Approval), 10Release-Engineering-Team: Requesting GitLab account activation for jdrewniak - https://phabricator.wikimedia.org/T344995 (10bd808) https://gitlab.wikimedia.org/jdrewniak is enabled per the admin UI and also is a member of https://gitlab.wikimedia.org/repos/feature-engineering/tak... [21:15:33] 10GitLab: GitLab Private Repository Request for: calbon - https://phabricator.wikimedia.org/T345113 (10Reedy) [21:17:28] 10GitLab (Account Approval), 10Release-Engineering-Team: Requesting GitLab account activation for jdrewniak - https://phabricator.wikimedia.org/T344995 (10Jdrewniak) 05Open→03Resolved a:03Jdrewniak @bd808 oh! thanks, I just wasn't sure if I had commit access, but if everything's in order then this task c... [21:18:57] 10GitLab (Account Approval), 10Release-Engineering-Team, 10User-bd808: Requesting GitLab account activation for matthias - https://phabricator.wikimedia.org/T345044 (10bd808) 05Open→03Resolved a:03bd808 https://gitlab.wikimedia.org/matthiasm has been approved. [22:21:09] 10GitLab: GitLab Private Repository Request for: calbon - https://phabricator.wikimedia.org/T345113 (10calbon) [23:23:07] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10Patch-For-Review: scap backport sometimes crashes when running with two stacked patches - https://phabricator.wikimedia.org/T343967 (10CodeReviewBot) dancy opened https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/24 verify-patches... [23:41:03] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10Patch-For-Review: scap backport sometimes crashes when running with two stacked patches - https://phabricator.wikimedia.org/T343967 (10CodeReviewBot) dancy merged https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/24 verify-patches...