[01:20:02] Project beta-update-databases-eqiad build #82155: 04FAILURE in 1.6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/82155/ [02:31:23] Yippee, build fixed! [02:31:23] Project beta-update-databases-eqiad build #82156: 09FIXED in 11 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/82156/ [02:52:10] (03CR) 10Ejegg: "Ah, I thought it was for the fundraising/REL* branches of the mediawiki/core project - where do we configure that?" [integration/config] - 10https://gerrit.wikimedia.org/r/1114447 (owner: 10Ejegg) [03:10:24] 10GitLab, 06Release-Engineering-Team, 10mwcli: Possibly rename mwcli and mwcli gitlab repo - https://phabricator.wikimedia.org/T384781#10499415 (10Samwilson) I like `wikcli`, and `wmcli` sounds good too (short for `wikimedia-cli`). Do you think I should rename [[https://gitlab.wikimedia.org/toolforge-repos/m... [04:03:05] (03PS1) 10Chlod Alejandro: zuul: Add EventLogging as test/Phan dependency for Nuke [integration/config] - 10https://gerrit.wikimedia.org/r/1114489 (https://phabricator.wikimedia.org/T382149) [06:34:21] Is there something going on with gerrit again? I'm getting extreme slowness and timeouts again... I was having similar problems last night (10 hours ago). [06:35:54] Sometimes switching IPs helps, so I suppose I am getting hit by some IP based throttle... [06:36:37] Are there throttles on ranges? Or just individual IPs? I'm starting to wonder whether PhpStorm is talking a lot to the git repo in the background. I don't see why it would, though. [07:20:58] Gerrit metrics look normal and I'm not aware of any problems currently [07:27:31] IP based throttling is active since several weeks to protect gerrit. In the past there were cases with IDEs hammering gerrit and causing temporary throttling. [07:27:31] I went over the log of blocked IPs and did not found anything obvious. But I also don't know your IP or ISP [07:38:04] jelto: do ssh and http requests go into the same throttle bucket? or are they separate? [07:38:18] would it be possible to relax the http throttle for requests that have a session cookie? [07:40:27] it only applies to https. The throttling is working on layer 3 and 4, so information from the http request like the header is not available at the moment [07:41:53] hm, that doesn't explain why a simple git pull on the command line takes several minutes... [07:42:02] using ssh [07:42:35] my current IP is 83.135.15.221 and 2001:9e8:1d5c:8a00:997f:ff75:1823:6595 [07:43:42] ping shows no packet loss [07:43:47] this IPs are not in the list of throttled IPs [07:45:10] it'S really strange... is there anything I can do on my side to investigate the cause of this? It's really interfering with my work... [07:47:45] huh - actually, I *did* see severe packet loss just now. But only on the last hop to gerrit.wikimedia.org. Pings along the route are fine, and pinging 8.8.8.8 is fine as well. [07:47:53] Maybe it's a routing issue, then? [07:48:52] But a routing issue *after* xe-5-3-3-500.cr1-eqiad.wikimedia.org? Odd. [07:49:26] Now ping is fine again, and gerrit seems to be working too. [07:50:44] I can reproduce that and see occasionally packet loss of around 10% when doing a ping. But now the issue is gone [07:51:56] I'll keep an eye on the last-hop-packete-loss issue [08:08:58] duesen: Guten Tag [08:09:34] i can't see anything standing out in the graphs :( [08:10:14] though some packets got dropped eventually and there was some elevated cpu usage https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&var-server=gerrit1003 [08:25:38] [2025-01-28T06:29:41.716Z] [SSH git-receive-pack /mediawiki/core (daniel)] [08:25:38] INFO com.google.gerrit.server.git.MultiProgressMonitor : Processing changes: (\) [08:25:39] [CONTEXT ratelimit_period="1 MINUTES [skipped: 5]" ] [08:25:55] again at 7:46:22 and 7:55:51 [08:26:12] which oddly align with your messages here? [08:26:47] then I don't know what those messages are :) [08:37:04] duesen: I have reviewed the 9 sshd connection you have done this morning and none of the session took more than 3 seconds [08:37:20] though most probably your fetches/pull happen over https [08:52:36] and your http queries that took more than 5 seconds are all about finding the conflicting changes of the change you are currently browsing [08:52:44] which indeed takes time [08:53:00] but UI wise that just means the list of conflicting changes is not shown immediately [08:53:12] so essentially, I can't find anything [09:06:30] I don't pull over https, i use ssh for everything... [09:07:17] pulls over ssh take minutes. i'll try with -vv next time [09:07:34] in the browser, I get timesouts - for individual plugins, or sometimes for the entire page [09:08:36] 10Deployments, 06Release-Engineering-Team, 06SRE, 13Patch-For-Review, 07SRE-Unowned: Reduce automatic messages on #wikimedia-operations - https://phabricator.wikimedia.org/T384804#10499724 (10hashar) 05Open→03Declined I had enough push back that I am not interested in pursuing. I will keep using... [09:11:18] duesen: have you had a pull over ssh that took minutes this morning? [09:11:30] yes [09:11:41] well, a fetch [09:11:53] pull... not sure if this morning. last night, definitely [09:12:24] also pushes (via git-review) [09:13:28] The fetch was at 8:12, I think. [09:13:43] The one at 7:29 was quick, iirc [09:15:08] here are all your SSH sessions and the rough duration for each of them https://phabricator.wikimedia.org/P72573 [09:16:16] i have upgraded Gerrit a couple weeks ago which fixed some ssh timeout [09:16:25] though in practice they should be large enough [09:17:44] I think for the push the issue is that if you are not up-to-date with the large remote, git has a to exchange a lot of objects with the servers during the negotiation phase [09:18:03] so your git push / git-review looks slow cause under the hood there is lot of network traffic happening [09:18:50] they can be seen using `GIT_TRACE_PACKET=1 git push origin HEAD:refs/for/master` [09:21:21] also you can have some verbose ssh logging when using git with: `GIT_SSH_COMMAND='ssh -v' git push origin HEAD:refs/for/master` [09:21:37] (that causes git to use the given shell parsed string instead of just `ssh`) [09:25:14] so I don't know really :-( it is tough [09:35:02] maintenance-disconnect-full-disks build 671003 integration-agent-docker-1043 (/: 27%, /srv: 95%, /var/lib/docker: 29%): OFFLINE due to disk space [09:40:03] maintenance-disconnect-full-disks build 671004 integration-agent-docker-1043 (/: 27%, /srv: 40%, /var/lib/docker: 29%): RECOVERY disk space OK [09:53:20] hashar: yea, but all that shouldn't be different now than it was before. And it's not always like that, just sometimes, for a couple of hours. This isn't "regular" slowness. [09:53:43] Anything, different question... I get odd CI failures from selenium.... [09:53:49] https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php74/51524/console [09:54:02] Apparently related to wikibase... [09:54:07] https://www.irccloud.com/pastebin/EWILpF0H/ [09:54:24] Any ideas? This is blocking core patches. [09:54:44] Not sure if it happens always, but I got it several times since yesterday. [09:56:33] oh yay, and now the website is timing out again [10:00:06] duesen: my guess is that npm install failed due to the disk being full [10:03:10] that host had a disk alarm at that time [10:03:22] we need to rebuild them to larger size [10:07:20] !log Manually cleaned integration-agent-docker-1043 [10:07:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:37:06] 10Continuous-Integration-Config, 10Gerrit, 13Patch-For-Review: mediawiki/extensions/BlueSpice isn't an MW extension - https://phabricator.wikimedia.org/T378301#10499967 (10Reedy) I'm removing it in ^, but I'm guessing unless it's actually dealt with, it'll get re-added... [10:44:13] 10Continuous-Integration-Config, 10Gerrit, 10BlueSpice, 13Patch-For-Review: mediawiki/extensions/BlueSpice isn't an MW extension - https://phabricator.wikimedia.org/T378301#10499972 (10Reedy) [10:58:40] hashar: thanks! [11:23:32] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [11:23:59] (03merge) 10hashar: make-release: remove manual branch point [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/144 [11:24:33] FIRING: [2x] DatasourceNoData: - https://alerts.wikimedia.org/?q=alertname%3DDatasourceNoData [11:33:32] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [11:34:33] RESOLVED: [2x] DatasourceNoData: - https://alerts.wikimedia.org/?q=alertname%3DDatasourceNoData [12:16:05] (03open) 10jforrester: releases: Bump mediawiki-codesniffer to v46.0.0 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/59 [12:16:18] (03merge) 10jforrester: releases: Bump mediawiki-codesniffer to v46.0.0 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/59 [12:18:51] (03update) 10jforrester: Bump Vue to 3.5.13 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/56 (owner: 10catrope) [12:19:33] (03close) 10jforrester: Bump Vue to 3.5.13 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/56 (owner: 10catrope) [12:45:00] (03open) 10hashar: make-release: print revision of new branches [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/146 [13:06:40] 10Continuous-Integration-Config, 10Gerrit, 10BlueSpice, 13Patch-For-Review: mediawiki/extensions/BlueSpice isn't an MW extension - https://phabricator.wikimedia.org/T378301#10500456 (10hashar) a:03hashar I have created that repository with @Kizule , it is a permissions only repository and should be filt... [13:10:42] 10Continuous-Integration-Config, 10Gerrit, 10BlueSpice, 13Patch-For-Review: mediawiki/extensions/BlueSpice isn't an MW extension - https://phabricator.wikimedia.org/T378301#10500470 (10hashar) That was added by @Zabe in 2e446b4e5360fc15a5d3f8e1b526cbd0cccda9e3 :) [13:36:52] 10Continuous-Integration-Config, 10Gerrit, 10BlueSpice, 13Patch-For-Review: mediawiki/extensions/BlueSpice isn't an MW extension - https://phabricator.wikimedia.org/T378301#10500557 (10hashar) 05Open→03Resolved Done, thank you @zabe for the reviews/backports! [13:45:02] 10GitLab, 06Release-Engineering-Team, 10mwcli: Possibly rename mwcli and mwcli gitlab repo - https://phabricator.wikimedia.org/T384781#10500572 (10Addshore) >>! In T384781#10499415, @Samwilson wrote: > I like `wikcli`, and `wmcli` sounds good too (short for `wikimedia-cli`). Do you think I should rename [[ht... [14:28:13] 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review, 05Release, 05Train Deployments: 1.44.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T382365#10500727 (10Aklapper) For the records, {T384905} currently blocks CI [14:35:50] 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review, 05Release, 05Train Deployments: 1.44.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T382365#10500765 (10Daimona) [14:51:24] (03approved) 10jforrester: make-release: print revision of new branches [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/146 (owner: 10hashar) [15:25:54] (03CR) 10Hashar: [C:03+2] "That sounds exciting!! :)" [integration/config] - 10https://gerrit.wikimedia.org/r/1114489 (https://phabricator.wikimedia.org/T382149) (owner: 10Chlod Alejandro) [15:27:46] (03Merged) 10jenkins-bot: zuul: Add EventLogging as test/Phan dependency for Nuke [integration/config] - 10https://gerrit.wikimedia.org/r/1114489 (https://phabricator.wikimedia.org/T382149) (owner: 10Chlod Alejandro) [15:29:11] (03CR) 10Hashar: [C:03+2] "Deployed and I have triggered a `recheck` on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Nuke/+/1114487" [integration/config] - 10https://gerrit.wikimedia.org/r/1114489 (https://phabricator.wikimedia.org/T382149) (owner: 10Chlod Alejandro) [15:37:32] (03merge) 10dancy: make-release: print revision of new branches [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/146 (owner: 10hashar) [15:52:23] (03update) 10dancy: spiderpig: CAS auth integration [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/625 [16:02:29] (03PS2) 10Ejegg: Use new CiviCRM image (see parent patch) [integration/config] - 10https://gerrit.wikimedia.org/r/1113550 [16:14:36] hi releng! Looks like this patch to start testing CiviCRM against PHP 8.2 is now passing tests: https://gerrit.wikimedia.org/r/c/integration/config/+/1113550 [16:15:01] Would anyone be available to take a review and merge that? [16:46:27] (03PS3) 10Arlolra: Migrate cloudvps-configs/scripts to tools/ [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1114110 (owner: 10Subramanya Sastry) [16:46:27] (03PS2) 10Arlolra: Add "2nd opinion needed?" column to diffs output [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1114441 (owner: 10Subramanya Sastry) [16:46:44] (03CR) 10Arlolra: [C:03+2] Migrate cloudvps-configs/scripts to tools/ (031 comment) [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1114110 (owner: 10Subramanya Sastry) [16:47:22] (03Merged) 10jenkins-bot: Migrate cloudvps-configs/scripts to tools/ [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1114110 (owner: 10Subramanya Sastry) [16:47:23] (03Merged) 10jenkins-bot: Add "2nd opinion needed?" column to diffs output [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1114441 (owner: 10Subramanya Sastry) [17:23:08] ejegg: Taking a look [17:23:56] (03CR) 10Subramanya Sastry: Migrate cloudvps-configs/scripts to tools/ (031 comment) [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1114110 (owner: 10Subramanya Sastry) [17:24:17] thanks dancy [17:25:37] ejegg: Is "parent patch" referring to https://gerrit.wikimedia.org/g/integration/config/+/4fdef6ff76fbb0ff1ee98ffb64813183e63ba759 or something else? [17:26:39] oh oops dancy I rebased it over master [17:26:54] the parent patch was the one creating the civicrm-bookworm image [17:26:55] lemme see [17:27:08] (03CR) 10Ahmon Dancy: Use new CiviCRM image (see parent patch) (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/1113550 (owner: 10Ejegg) [17:27:36] it was this one: https://gerrit.wikimedia.org/r/c/integration/config/+/1113549 [17:27:59] thanks for the comments, will update the commit message! [17:30:27] (03PS3) 10Ejegg: Test CiviCRM against new bookworm image [integration/config] - 10https://gerrit.wikimedia.org/r/1113550 (https://phabricator.wikimedia.org/T383337) [17:30:44] ok dancy, that's the parent patch and bug linked ^^^ [17:30:59] thx. I'm testing the parent now [17:31:33] Ah yeah, I was able to 'docker run' it locally but couldn't fully execute the tests without all the zuul mounts and env variables [17:35:53] (03CR) 10Ahmon Dancy: [C:03+2] Docker: [civicrm] Add CiviCRM image v0.5 based on bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1113549 (https://phabricator.wikimedia.org/T383337) (owner: 10Ejegg) [17:37:29] (03Merged) 10jenkins-bot: Docker: [civicrm] Add CiviCRM image v0.5 based on bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1113549 (https://phabricator.wikimedia.org/T383337) (owner: 10Ejegg) [17:38:15] !log Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/1113549 (T383337) [17:38:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:38:17] T383337: Migrate all CI jobs from bullseye to bookworm or later and drop bullseye testing support - https://phabricator.wikimedia.org/T383337 [17:40:55] (03CR) 10Ahmon Dancy: Test CiviCRM against new bookworm image (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/1113550 (https://phabricator.wikimedia.org/T383337) (owner: 10Ejegg) [17:41:09] (03CR) 10Ahmon Dancy: [C:03+2] Test CiviCRM against new bookworm image [integration/config] - 10https://gerrit.wikimedia.org/r/1113550 (https://phabricator.wikimedia.org/T383337) (owner: 10Ejegg) [17:43:17] (03CR) 10CI reject: [V:04-1] Test CiviCRM against new bookworm image [integration/config] - 10https://gerrit.wikimedia.org/r/1113550 (https://phabricator.wikimedia.org/T383337) (owner: 10Ejegg) [17:44:29] (03CR) 10Ahmon Dancy: Test CiviCRM against new bookworm image [integration/config] - 10https://gerrit.wikimedia.org/r/1113550 (https://phabricator.wikimedia.org/T383337) (owner: 10Ejegg) [17:44:33] (03CR) 10Ahmon Dancy: [C:03+2] Test CiviCRM against new bookworm image [integration/config] - 10https://gerrit.wikimedia.org/r/1113550 (https://phabricator.wikimedia.org/T383337) (owner: 10Ejegg) [17:46:03] (03Merged) 10jenkins-bot: Test CiviCRM against new bookworm image [integration/config] - 10https://gerrit.wikimedia.org/r/1113550 (https://phabricator.wikimedia.org/T383337) (owner: 10Ejegg) [17:46:51] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1113550 (T383337) [17:46:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:46:53] T383337: Migrate all CI jobs from bullseye to bookworm or later and drop bullseye testing support - https://phabricator.wikimedia.org/T383337 [17:47:10] ejegg: Merged, deployed, and ready for testing [18:05:03] maintenance-disconnect-full-disks build 671105 integration-agent-docker-1043 (/: 27%, /srv: 97%, /var/lib/docker: 38%): OFFLINE due to disk space [18:10:03] maintenance-disconnect-full-disks build 671106 integration-agent-docker-1043 (/: 27%, /srv: 9%, /var/lib/docker: 35%): RECOVERY disk space OK [18:17:09] thanks dancy ! [18:19:49] 10Diffusion, 10Gerrit, 06collaboration-services, 06serviceops, and 2 others: Gerrit replication to codfw (gerrit-replica.wikimedia.org) stopped working after Gerrit 3.4.5 upgrade - https://phabricator.wikimedia.org/T315942#10501700 (10Dzahn) I checked replication log after merging the patch above and i... [20:05:49] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10MW-1.44-notes (1.44.0-wmf.14; 2025-01-28), 13Patch-For-Review: Migrate all CI jobs from bullseye to bookworm or later and drop bullseye testing support - https://phabricator.wikimedia.org/T383337#10502068 (10Jdforrester-WMF) [20:36:54] (03CR) 10C. Scott Ananian: Add canvas-based diff image browser (031 comment) [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1111379 (https://phabricator.wikimedia.org/T383166) (owner: 10C. Scott Ananian) [22:36:57] (03open) 10dancy: spiderpig: Make searchPatch non-async [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/653 [22:37:00] (03update) 10dancy: spiderpig: Make searchPatch non-async [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/653 [22:39:31] (03merge) 10dancy: spiderpig: Make searchPatch non-async [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/653 [22:41:13] (03update) 10dancy: spiderpig: CAS auth integration [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/625