[01:25:23] 10Beta-Cluster-Infrastructure, 10DiscussionTools, 10Editing-team, 10Verified: "Service Temporarily Unavailable" shows up when trying to add a new section/topic to a talk page on Beta cluster - https://phabricator.wikimedia.org/T312689 (10Ryasmeen) [04:36:14] (03CR) 10Abijeet Patro: "This change is ready for review." [integration/docroot] - 10https://gerrit.wikimedia.org/r/812948 (owner: 10Abijeet Patro) [04:36:47] (03CR) 10Abijeet Patro: Add language-data library (031 comment) [integration/docroot] - 10https://gerrit.wikimedia.org/r/812540 (owner: 10Abijeet Patro) [05:47:27] (03PS1) 10Physikerwelt: Make math tests dependent on popus [integration/config] - 10https://gerrit.wikimedia.org/r/813118 (https://phabricator.wikimedia.org/T288076) [07:04:21] 10Project-Admins, 10Design-Innovation-Team: Archive #Design-Innovation-Team project tag? - https://phabricator.wikimedia.org/T312831 (10Aklapper) [08:15:46] (03PS1) 10Kosta Harlan: zuul: Run non-voting phpbench job for MW core patches [integration/config] - 10https://gerrit.wikimedia.org/r/813194 (https://phabricator.wikimedia.org/T291549) [08:15:53] (03CR) 10CI reject: [V: 04-1] zuul: Run non-voting phpbench job for MW core patches [integration/config] - 10https://gerrit.wikimedia.org/r/813194 (https://phabricator.wikimedia.org/T291549) (owner: 10Kosta Harlan) [08:16:25] (03PS2) 10Kosta Harlan: zuul: Run non-voting phpbench job for MW core patches [integration/config] - 10https://gerrit.wikimedia.org/r/813194 (https://phabricator.wikimedia.org/T291549) [08:16:34] (03CR) 10CI reject: [V: 04-1] zuul: Run non-voting phpbench job for MW core patches [integration/config] - 10https://gerrit.wikimedia.org/r/813194 (https://phabricator.wikimedia.org/T291549) (owner: 10Kosta Harlan) [08:17:52] (03CR) 10Kosta Harlan: "> Merge Failed." [integration/config] - 10https://gerrit.wikimedia.org/r/813194 (https://phabricator.wikimedia.org/T291549) (owner: 10Kosta Harlan) [08:20:40] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10User-DannyS712: Gerrit: all patches are being reported as merge conflicts - https://phabricator.wikimedia.org/T309371 (10kostajh) 05Resolved→03Open p:05Triage→03Unbreak! @hashar this is happening again, it seems. [08:20:42] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: gerrit-bot holding open SSH sessions - https://phabricator.wikimedia.org/T309376 (10kostajh) [08:22:06] (03CR) 10Kosta Harlan: zuul: Run non-voting phpbench job for MW core patches (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/813194 (https://phabricator.wikimedia.org/T291549) (owner: 10Kosta Harlan) [08:23:47] (03CR) 10Hashar: [C: 03+2] "Thanks for catching that ;)" [integration/docroot] - 10https://gerrit.wikimedia.org/r/812948 (owner: 10Abijeet Patro) [08:24:36] (03Merged) 10jenkins-bot: Fix NPM URL for Wikimedia language-data library [integration/docroot] - 10https://gerrit.wikimedia.org/r/812948 (owner: 10Abijeet Patro) [08:26:56] (03PS3) 10Kosta Harlan: zuul: Run non-voting phpbench job for MW core patches [integration/config] - 10https://gerrit.wikimedia.org/r/813194 (https://phabricator.wikimedia.org/T291549) [08:27:03] (03CR) 10CI reject: [V: 04-1] zuul: Run non-voting phpbench job for MW core patches [integration/config] - 10https://gerrit.wikimedia.org/r/813194 (https://phabricator.wikimedia.org/T291549) (owner: 10Kosta Harlan) [08:34:44] I'm getting -1 from jenkins-bot saying that a puppet change is unable to be merged, but it's rebased on latest (according to gerrit, and myself), is there anything going on? (the patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/813197, re-uploaded three times already) [08:36:53] that sounds like T309371 [08:36:54] T309371: Gerrit: all patches are being reported as merge conflicts - https://phabricator.wikimedia.org/T309371 [08:37:04] yep [08:37:19] thanks! I'll keep an eye, let me know if I can help [08:37:41] I’d just seen the task ID in -operations, don’t know much about it I’m afraid [08:41:39] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10User-DannyS712: Gerrit: all patches are being reported as merge conflicts - https://phabricator.wikimedia.org/T309371 (10dcaro) Yep, [[ https://logstash.wikimedia.org/goto/7f444beea192b492214f6b10155d6b20 | same error logs in logstash aga... [08:41:41] 👍 it seems indeed the same [08:46:17] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10User-DannyS712: Gerrit: all patches are being reported as merge conflicts - https://phabricator.wikimedia.org/T309371 (10dcaro) And same three connections from zuul in contint2001 as before too: ` root@contint2001:~# netstat -tnp | grep 2... [08:46:57] should I just restart zuul on contint2001? that seemed to do the trick the last time (but I don't have any experience with the setup, so I'm hesitant xd) [08:50:10] restarting zuul is scary since it can miss some events, https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Restart [08:50:31] although I'm tempted to just do that here since I don't see other options to close these connections [08:52:17] hashar: if you are around, could you have a look please? [08:52:51] usualyl don't restart Zuul :) [08:53:00] it must be something else [08:55:13] I am checking [08:56:44] so yeah that is the same as T309371 [08:56:45] T309371: Gerrit: all patches are being reported as merge conflicts - https://phabricator.wikimedia.org/T309371 [08:56:49] I will restart Gerrit [08:58:19] which did not flush the connection :D [09:03:23] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10User-DannyS712: Gerrit: all patches are being reported as merge conflicts - https://phabricator.wikimedia.org/T309371 (10hashar) I have restarted Gerrit but that did not flush the Zuul server ssh connections to Gerrit. I took a stack dum... [09:03:59] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10User-DannyS712: Gerrit: all patches are being reported as merge conflicts - https://phabricator.wikimedia.org/T309371 (10hashar) And the three connections: ` lsof -p 15751|grep gerrit zuul-serv 15751 zuul 8u IPv6 295859537... [09:05:41] waiting for last job to complete and I will restart zuul [09:06:38] dcaro: Lucas_WMDE: taavi: kostajh: somehow a ssh connection get stuck between Zuul and Gerrit for whatever reason. Zuul apparently establishes another one and that goes against the 4 connections per user max limit [09:07:02] thank you hashar [09:17:58] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: gerrit-bot holding open SSH sessions - https://phabricator.wikimedia.org/T309376 (10hashar) [09:18:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10User-DannyS712: Gerrit: all patches are being reported as merge conflicts - https://phabricator.wikimedia.org/T309371 (10hashar) 05Open→03Resolved I have send `SIGUSR1` to Zuul to let it finishes processing the ongoing job then restar... [09:49:09] (03PS2) 10Hashar: Make math tests dependent on Popups [integration/config] - 10https://gerrit.wikimedia.org/r/813118 (https://phabricator.wikimedia.org/T288076) (owner: 10Physikerwelt) [09:49:14] (03CR) 10Hashar: [C: 03+2] Make math tests dependent on Popups [integration/config] - 10https://gerrit.wikimedia.org/r/813118 (https://phabricator.wikimedia.org/T288076) (owner: 10Physikerwelt) [09:50:20] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10Quibble, 10Patch-For-Review: Some unit tests are not executed with composer phpunit:unit - https://phabricator.wikimedia.org/T266441 (10kostajh) a:03kostajh [09:51:12] (03Merged) 10jenkins-bot: Make math tests dependent on Popups [integration/config] - 10https://gerrit.wikimedia.org/r/813118 (https://phabricator.wikimedia.org/T288076) (owner: 10Physikerwelt) [09:52:30] (03CR) 10Hashar: [C: 03+2] "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/813118 (https://phabricator.wikimedia.org/T288076) (owner: 10Physikerwelt) [10:07:42] (03CR) 10Physikerwelt: Make math tests dependent on Popups (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/813118 (https://phabricator.wikimedia.org/T288076) (owner: 10Physikerwelt) [10:38:39] (03CR) 10Abijeet Patro: "This change is ready for review." [integration/docroot] - 10https://gerrit.wikimedia.org/r/812953 (owner: 10Abijeet Patro) [10:41:26] (03PS1) 10Kosta Harlan: dockerfiles: Install pnpm in Quibble images [integration/config] - 10https://gerrit.wikimedia.org/r/813215 (https://phabricator.wikimedia.org/T305525) [11:18:45] (03PS4) 10Kosta Harlan: zuul: Run non-voting phpbench job for MW core patches [integration/config] - 10https://gerrit.wikimedia.org/r/813194 (https://phabricator.wikimedia.org/T291549) [11:21:28] (03CR) 10CI reject: [V: 04-1] zuul: Run non-voting phpbench job for MW core patches [integration/config] - 10https://gerrit.wikimedia.org/r/813194 (https://phabricator.wikimedia.org/T291549) (owner: 10Kosta Harlan) [11:40:01] (03PS2) 10Kosta Harlan: dockerfiles: Install pnpm in Quibble images [integration/config] - 10https://gerrit.wikimedia.org/r/813215 (https://phabricator.wikimedia.org/T305525) [11:41:59] hashar: thanks! [13:46:59] (03PS3) 10Jbond: WIP: add files for custom image for beaker builds [integration/config] - 10https://gerrit.wikimedia.org/r/812463 [13:49:17] (03CR) 10CI reject: [V: 04-1] WIP: add files for custom image for beaker builds [integration/config] - 10https://gerrit.wikimedia.org/r/812463 (owner: 10Jbond) [14:41:19] 10Phabricator: Grant access to the Wikimedia Security Team phame blog to more members of the Security Team - https://phabricator.wikimedia.org/T312860 (10sbassett) [14:44:28] 10Phabricator, 10Security-Team: Grant access to the Wikimedia Security Team phame blog to more members of the Security Team - https://phabricator.wikimedia.org/T312860 (10sbassett) [14:49:27] 10Phabricator, 10Security-Team: Grant access to the Wikimedia Security Team phame blog to more members of the Security Team - https://phabricator.wikimedia.org/T312860 (10sbassett) [15:20:37] Project beta-update-databases-eqiad build #59953: 04FAILURE in 36 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/59953/ [15:22:30] Amir1: ^ [15:23:25] Error 1265: Data truncated for column 'tl_target_id' at row 138124 [15:23:51] which I assume is a 3 hour late failure from https://github.com/wikimedia/mediawiki/commit/692dde00df1a513691a64e5a3cf77c67836fa8fc [15:24:10] hmm, I'm going to try to run it manually [15:28:23] somehow commons were not done fully, it should recover now [15:30:12] Project beta-update-databases-eqiad build #59954: 04STILL FAILING in 10 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/59954/ [15:32:20] ugh, it was done half-way, that's why it broke again [15:32:29] I do commons manually [15:35:52] Project beta-update-databases-eqiad build #59955: 04STILL FAILING in 2 min 36 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/59955/ [15:36:25] and enwiki is half-done too [15:41:35] let's try again [15:46:02] PROBLEM - Host doc2001 is DOWN: PING CRITICAL - Packet loss = 100% [15:46:28] doc2001 is maint [15:46:42] PROBLEM - Host contint2001 is DOWN: PING CRITICAL - Packet loss = 100% [15:47:38] and ^ [15:51:22] RECOVERY - Host contint2001 is UP: PING OK - Packet loss = 0%, RTA = 30.13 ms [15:51:29] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Upstream, 10User-brennen: Uploaded files via the drag-and-drop are defaulting to private-access - https://phabricator.wikimedia.org/T310833 (10DLynch) T312864 features a variant of this still happening. Jon added the second image in the description vi... [15:52:40] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Upstream, 10User-brennen: Uploaded files via the drag-and-drop are defaulting to private-access - https://phabricator.wikimedia.org/T310833 (10brennen) > Jon added the second image in the description via editing the description after the task was crea... [15:58:16] I got a -1 from jenkins-bot on https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/813262 because "This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.", but my change is up to do with the target branch already. What gives? [15:59:57] ori: codfw power went down so probably bad state [16:00:12] fun [16:00:14] thanks [16:00:14] maybe try again when it's not on fire or at least people aren't busy [16:00:22] ori: maint went wrong in A5 [16:11:13] ori: maybe try recheck and if not ping a relenger. Most things are settling. [16:11:34] yeah it worked [16:13:00] ori: great! [16:15:10] thanks again [16:15:44] np, i mainly sat here and watched [16:19:16] PROBLEM - Check systemd state on doc1002 is CRITICAL: CRITICAL - degraded: The following units failed: rsync-doc-doc2001.codfw.wmnet.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [16:23:12] RECOVERY - Host doc2001 is UP: PING OK - Packet loss = 0%, RTA = 30.39 ms [16:25:12] mutante: does ^ auto-recover? [16:25:28] Yippee, build fixed! [16:25:29] Project beta-update-databases-eqiad build #59957: 09FIXED in 5 min 28 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/59957/ [16:25:59] RhinosF1: I think herron actively fixed [16:26:28] mutante: the doc rsync? [16:27:24] that'll auto recover once the timer triggers next time [16:28:27] RhinosF1: I can't keep up because there are 5 channels and 15 people spread evenly [16:30:13] mutante: that was a fairly noisy incident [16:30:53] sorry, it's either -operations or the private channels but not also -releng:) [16:31:07] or I spend time just forwarding stuff [16:39:15] (03PS1) 10Subramanya Sastry: Add UI styling resource module [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/813272 [16:40:09] 10Continuous-Integration-Config: Please add mediawiki-i18n-check-docker to repository mediawiki/extensions/SemanticRESTAPI - https://phabricator.wikimedia.org/T311226 (10Jdforrester-WMF) >>! In T311226#8070512, @Umherirrender wrote: >>>! In T311226#8069736, @Jdforrester-WMF wrote: >> To confirm, @Sophivorus as t... [16:41:40] RhinosF1: people are getting on ganeti VMs via console and fix networking [16:41:47] that's why they come back [16:42:44] mutante: ah that [16:50:03] RECOVERY - Check systemd state on doc1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [17:29:16] !log dropping tl_namespace and tl_title from templatelinks in fawiki (T312865) [17:29:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:29:18] T312865: Turn off writing to the old columns of templatelinks in beta and production - https://phabricator.wikimedia.org/T312865 [17:44:30] (03CR) 10CI reject: [V: 04-1] Add CookieWarning as a dependency of Cosmos [integration/config] - 10https://gerrit.wikimedia.org/r/812963 (owner: 10Universal Omega) [17:47:21] (03PS3) 10Universal Omega: Add CookieWarning as a dependency of Cosmos [integration/config] - 10https://gerrit.wikimedia.org/r/812963 [19:31:53] 10Continuous-Integration-Config: Please add mediawiki-i18n-check-docker to repository mediawiki/extensions/SemanticRESTAPI - https://phabricator.wikimedia.org/T311226 (10Sophivorus) Hi! Yes I'm OK with this, thanks! [20:08:19] (03PS1) 10Kosta Harlan: jjb: [wkibase-kind-docker] Pass Wikibase directory to phpunit [integration/config] - 10https://gerrit.wikimedia.org/r/813304 [20:27:19] (03PS2) 10Kosta Harlan: jjb: [wkibase-kind-docker] Pass Wikibase directory to phpunit [integration/config] - 10https://gerrit.wikimedia.org/r/813304 (https://phabricator.wikimedia.org/T310255) [20:32:54] (03CR) 10Kosta Harlan: "This is pretty much a bandaid fix, as I can't reproduce the issue locally. But it seems like an improvement and would unblock other work (" [integration/config] - 10https://gerrit.wikimedia.org/r/813304 (https://phabricator.wikimedia.org/T310255) (owner: 10Kosta Harlan) [21:07:01] 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10user-sbassett: Fix a handful of minor bugs within the Semgrep Merge Tool that surfaced during production-deployment testing on toolforge - https://phabricator.wikimedia.org/T312807 (10sbassett) 05Open→03Resolved p:05Tri... [21:09:14] 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10user-sbassett: Fix a handful of minor bugs within the Semgrep Merge Tool that surfaced during production-deployment testing on toolforge - https://phabricator.wikimedia.org/T312807 (10sbassett) [21:13:11] 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10Security: Change default config args within semgrep ci include - https://phabricator.wikimedia.org/T312901 (10sbassett) [21:30:49] 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10Security: Change default config args within semgrep ci include - https://phabricator.wikimedia.org/T312901 (10sbassett) **fake-gitlab-bot:** new merge request: https://gitlab.wikimedia.org/repos/security/gitlab-ci-security-t... [21:31:11] 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10SecTeam-Processed, and 2 others: Change default config args within semgrep ci include - https://phabricator.wikimedia.org/T312901 (10sbassett) [21:31:28] 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10SecTeam-Processed, and 2 others: Change default config args within semgrep ci include - https://phabricator.wikimedia.org/T312901 (10sbassett) p:05Triage→03Medium [21:31:39] 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10SecTeam-Processed, and 2 others: Change default config args within semgrep ci include - https://phabricator.wikimedia.org/T312901 (10sbassett) 05Open→03In progress [21:54:14] 10Release-Engineering-Team (Next), 10tech-decision-forum, 10Code-Stewardship-Reviews, 10Documentation, 10User-AKlapper: Document checklist steps to undeploy / sunset a codebase on WMF servers (not: archiving) - https://phabricator.wikimedia.org/T294329 (10Aklapper) [21:54:53] 10Release-Engineering-Team (Next), 10tech-decision-forum, 10Code-Stewardship-Reviews, 10Documentation, 10User-AKlapper: Document checklist steps to undeploy / sunset a codebase on WMF servers (not: archiving) - https://phabricator.wikimedia.org/T294329 (10Aklapper) @LNguyen: Hi, an answer would be welcome. [22:05:58] (03PS1) 10Zabe: Stop branching CongressLookup for Wikimedia production [tools/release] - 10https://gerrit.wikimedia.org/r/813343 (https://phabricator.wikimedia.org/T312894) [23:22:42] 10Phabricator, 10Project-Admins, 10Codex: Create custom Phabricator Forms for Codex tasks - https://phabricator.wikimedia.org/T312232 (10bd808) I you can handle these being links on a wiki rather than a custom form here on phab, you can use https://phabulous.toolforge.org/ to create URLs which pre-populate t...