[03:41:56] 10Continuous-Integration-Config, 10Release-Engineering-Team (Radar), 06Experimentation Lab, 10Metrics Platform, 13Patch-For-Review: Standardize Java build for metrics-platform - https://phabricator.wikimedia.org/T314630#10478180 (10VirginiaPoundstone) [03:48:48] 10Continuous-Integration-Config, 10Release-Engineering-Team (Radar), 06Experimentation Lab, 13Patch-For-Review: Standardize Java build for metrics-platform - https://phabricator.wikimedia.org/T314630#10478236 (10VirginiaPoundstone) [08:10:06] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 05Release, 05Train Deployments: 1.44.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T382364#10478527 (10hashar) a:05hasharβ†’03None [10:58:04] (03approved) 10addshore: Fix OTEL configuration [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/597 [10:58:06] (03merge) 10addshore: Fix OTEL configuration [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/597 [12:47:32] gerrit went AWOL? [12:47:42] Project beta-code-update-eqiad build #531667: 04FAILURE in 4 min 42 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/531667/ [12:47:48] or did i miss an upgrade? [12:48:48] its slow to reply but it ended up replying [12:52:00] I'm seeing timeouts [12:53:02] i received a 503 [12:53:43] It is back for me now [12:54:36] Dreamy_Jazz: i can reach it from inside the cluster, can't from my laptop [12:55:21] Yippee, build fixed! [12:55:22] Project beta-code-update-eqiad build #531668: 09FIXED in 2 min 21 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/531668/ [12:55:29] i just saw a different Gerrit interface for a moment, then it went down. now it's back up. i don't see anything in SAL, was that expected? [12:55:52] I don't think it was expected [12:55:58] Looking at https://grafana.wikimedia.org/d/L0-l1o0Mz/apache?var-host=gerrit1003&from=now-1h&to=now&orgId=1&refresh=1m [12:56:01] For a while, Gerrit integration in Phabricator was gone too. Seems to work now. [12:56:30] The worker statuses suggested all were busy for a bit [12:57:27] i am concerned that the interface changed. i stupidly didn't take a screenshot, but our logo and name in top-left was missing. did some of our config disappear? that seems bad [12:58:01] MatmaRex: i still have gerrit opened from prior the outage, wanna a screen? [12:58:11] or do you mean the different instance? [12:58:27] the interesting thing happened during the outage [12:58:35] aha [12:58:53] i'd expect the other instance to be at https://gerrit-replica.wikimedia.org/, but that doesn't seem to have any UI [12:59:01] maybe my expectation is wrong [12:59:23] instead of the logo and "Wikimedia Code Review" in top-left, it just said "Gerrit" [13:02:14] gerrit is still very slow for me [13:05:01] urbanecm: yeap replica doesn't have aui from memory [13:06:21] MatmaRex: now back to normal for me [13:06:25] what about you? [13:06:56] hm, yeah [13:08:18] well, not really. it was briefly normal, now slow again [13:09:09] raised a page [13:37:15] gerrit is really slot again today. I'm getting timeouts for bits and pieces of the web page. earlier today, it took several minutes to pull a patch for review... Any idea what's going on? [13:43:00] folks are looking into it at #_security now [13:43:42] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1113135 (if it loads for you, heh) [13:43:44] "gerrit: lower throttling threshold to 15 parallel connections [13:43:44] Scraping traffic is causing availability issues" [14:11:21] Gerrit should be responsive now [14:11:28] Please let me know if that is not the case [14:38:36] 06Release-Engineering-Team, 10wikimedia.biterg.io: Create account for LGoto on wikimedia.biterg.io - https://phabricator.wikimedia.org/T383951#10479869 (10Aklapper) @LGoto: You should have received credentials per email, coud you confirm please? [14:49:07] 10Phabricator: Changing the SUL account linked to a Phab account is not reflected in People search results - https://phabricator.wikimedia.org/T384328 (10Aklapper) 03NEW [14:54:56] 10Phabricator (Search): Changing the SUL account linked to a Phab account is not reflected in People search results - https://phabricator.wikimedia.org/T384328#10479946 (10Aklapper) [15:14:26] MatmaRex, jelto: ugh. Gerrit makes a *lot* of requests on every page load. more than 50. Limiting to 15 is hash - I have been struggling to get any work done today. But I guess the slowness I observed earlier was due to load caused by scrapers? [15:14:52] yes [15:15:15] Would it be possible to apply the throttle only to html pages? [15:15:40] s/hash/harsh [15:17:27] it's 15 parallel connections, I doubt your browser is doing all 50 requests in parallel tcp streams. And yes the availability was impacted by the scraping, so we installed more restrictive thresholds. [15:17:27] If you are affected by the throttling of 15 parallel connections (which I doubt) gerrit is not answering to you for 10 minutes (so just timeouts). [15:18:43] 10Gerrit, 07Upstream: Gerrit notification emails are missing the content of inline comments on unchanged files - https://phabricator.wikimedia.org/T355259#10480099 (10Paladox) Fixed with https://github.com/GerritCodeReview/gerrit/commit/689e240425f06b233c9641212af2a9a5fa0154ad [15:25:09] jelto: I often go over my dashboard and open a tab for each patch I want to look at, often five or more in parallel. That can easily trigger the threshold. Locking me out for 10 minutes is... not good? I doubt I'm the only one doing this. [15:26:08] I'll try to remember no to do this, but it's honestly hard to drop the habit. [15:26:25] I can revert the threshold change in a sec [15:27:11] Yea, but if scrapers keep causing problems, we do need something more restrictive. I'm trying to think of a rule that would slow down the scraper, but not me :) [15:49:28] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 05Release, 05Train Deployments: 1.44.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T382364#10480296 (10Aklapper) FYI, noise expected from {T384254} [15:50:14] duesen: thresholds are back to normal [15:58:35] jelto: thank you! [16:07:57] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 05Release, 05Train Deployments: 1.44.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T382364#10480368 (10brennen) a:03brennen [16:32:08] (03open) 10dancy: Changes to support SpiderPig UI development [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/646 [16:32:11] (03update) 10dancy: Changes to support SpiderPig UI development [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/646 [16:36:27] 06Release-Engineering-Team, 10wikimedia.biterg.io: Create account for LGoto on wikimedia.biterg.io - https://phabricator.wikimedia.org/T383951#10480477 (10LGoto) I didn't see anything but checked spam and found this: {F58239876} I think it's legit but please let me know if I should proceed! [16:37:28] (03PS1) 10Hoo man: Update fresh-node20, fresh-node22 image versions [fresh] - 10https://gerrit.wikimedia.org/r/1113175 (https://phabricator.wikimedia.org/T383337) [16:37:34] (03update) 10dancy: Changes to support SpiderPig UI development [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/646 [16:38:52] (03CR) 10Jforrester: Update fresh-node20, fresh-node22 image versions (031 comment) [fresh] - 10https://gerrit.wikimedia.org/r/1113175 (https://phabricator.wikimedia.org/T383337) (owner: 10Hoo man) [16:39:12] 06Release-Engineering-Team, 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Web-Team: Allow JavaScript errors to fail CI builds - https://phabricator.wikimedia.org/T318902#10480485 (10Ottomata) No opinion on who best suited to do task. Q: is using `mediawiki.client.error` to detect this the bes... [16:40:13] (03merge) 10dancy: Changes to support SpiderPig UI development [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/646 [16:42:54] (03CR) 10Hoo man: Update fresh-node20, fresh-node22 image versions (031 comment) [fresh] - 10https://gerrit.wikimedia.org/r/1113175 (https://phabricator.wikimedia.org/T383337) (owner: 10Hoo man) [16:43:28] (03update) 10dancy: styles: create a mixin for job grid styles [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/644 (https://phabricator.wikimedia.org/T379413) (owner: 10lwatson) [16:44:39] (03merge) 10dancy: styles: create a mixin for job grid styles [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/644 (https://phabricator.wikimedia.org/T379413) (owner: 10lwatson) [16:48:14] 10Phabricator, 07SecTeam-Processed: Change the dropdown in security ticket dropdown to not include WMF Product and WMF Technology as two separate departments - https://phabricator.wikimedia.org/T384243#10480539 (10sbassett) The #security-team would consider this nice, but fairly low-priority, especially if it'... [16:51:52] 10Fresh: Update Fresh's version of Node 20 to 20.18.1, Node 22 to 22.13.0 - https://phabricator.wikimedia.org/T384342 (10hoo) 03NEW [16:52:11] (03PS2) 10Hoo man: Update fresh-node20, fresh-node22 image versions [fresh] - 10https://gerrit.wikimedia.org/r/1113175 (https://phabricator.wikimedia.org/T384342) [17:00:41] (03CR) 10Jforrester: [C:03+2] Update fresh-node20, fresh-node22 image versions (031 comment) [fresh] - 10https://gerrit.wikimedia.org/r/1113175 (https://phabricator.wikimedia.org/T384342) (owner: 10Hoo man) [17:06:46] 06Release-Engineering-Team, 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Web-Team: Allow JavaScript errors to fail CI builds - https://phabricator.wikimedia.org/T318902#10480672 (10Jdlrobson) FWIW Right now, if code was committed that caused JS errors this would likely be picked up in log tri... [17:06:53] 06Release-Engineering-Team, 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Web-Team: Allow JavaScript errors to fail CI builds - https://phabricator.wikimedia.org/T318902#10480675 (10Jdlrobson) p:05Triageβ†’03Medium [17:08:08] (03update) 10dancy: spiderpig: CAS auth integration [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/625 [17:14:31] (03Merged) 10jenkins-bot: Update fresh-node20, fresh-node22 image versions [fresh] - 10https://gerrit.wikimedia.org/r/1113175 (https://phabricator.wikimedia.org/T384342) (owner: 10Hoo man) [17:15:07] 10GitLab (CI & Job Runners), 06Release-Engineering-Team: buildkit v0.19.0 released - https://phabricator.wikimedia.org/T384346 (10dancy) 03NEW [17:45:02] maintenance-disconnect-full-disks build 669085 integration-agent-docker-1046 (/: 28%, /srv: 100%, /var/lib/docker: 40%): OFFLINE due to disk space [17:50:03] maintenance-disconnect-full-disks build 669086 integration-agent-docker-1046 (/: 28%, /srv: 30%, /var/lib/docker: 37%): RECOVERY disk space OK [18:13:48] 06Release-Engineering-Team, 10wikimedia.biterg.io: Create account for LGoto on wikimedia.biterg.io - https://phabricator.wikimedia.org/T383951#10480972 (10Aklapper) @Lgoto: I think that is legit, yes. (Wasn't sure how Bitergia will send this exactly.) [18:20:43] 10GitLab (CI & Job Runners), 06Release-Engineering-Team: buildkit v0.19.0 released - https://phabricator.wikimedia.org/T384346#10481064 (10dancy) @brennen Looks like we're ready for a second attempt at updating buildkit. Lemme know if you need any assistance. [18:39:48] (03PS11) 10Subramanya Sastry: Commit Cloud VPS config files as our version of lazy puppetization [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1109127 (https://phabricator.wikimedia.org/T295907) [18:39:48] (03PS11) 10Subramanya Sastry: Poor man's puppetization of visual diffing vm and services [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1109519 (https://phabricator.wikimedia.org/T295907) [18:39:48] (03PS15) 10Subramanya Sastry: Add retry scripts to simplify retrying failures [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1109503 (https://phabricator.wikimedia.org/T383255) [18:40:05] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 05Release, 05Train Deployments: 1.44.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T382364#10481219 (10thcipriani) [18:53:50] 06Release-Engineering-Team, 10wikimedia.biterg.io: Create account for LGoto on wikimedia.biterg.io - https://phabricator.wikimedia.org/T383951#10481316 (10LGoto) 05Openβ†’03Resolved Have confirmed I can log in. Thanks! [18:56:08] 10GitLab (Administration, Settings & Policy): Create an instance-level npm package registry in Gitlab - https://phabricator.wikimedia.org/T384364#10481326 (10tchin) [19:04:25] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, and 2 others: Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup failures, often for our own hosts... - https://phabricator.wikimedia.org/T374830#10481364 [19:14:36] 06Release-Engineering-Team, 10wikimedia.biterg.io: Create account for LGoto on wikimedia.biterg.io - https://phabricator.wikimedia.org/T383951#10481424 (10Aklapper) Great! If you have any questions, see https://www.mediawiki.org/wiki/Community_metrics - thanks! [20:48:59] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Scap (SpiderPig πŸ•ΈοΈ), 06collaboration-services: Scap SpiderPig: Routing for the web frontend - https://phabricator.wikimedia.org/T383946#10481700 (10thcipriani) Tagging in #collaboration-services as our usual partners for these types of requests (if DNS addit... [21:00:33] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Scap (SpiderPig πŸ•ΈοΈ): Scap SpiderPig: user group/admin set up - https://phabricator.wikimedia.org/T383947#10481757 (10thcipriani) [22:01:32] 10Phabricator, 06collaboration-services: Experiencing Phabricator 429 errors - https://phabricator.wikimedia.org/T383435#10482044 (10Dominicbm) @Dzahn Sorry, for the late reply. I am doing nothing on Phabricator at all besides occasionally viewing it or making the occasional ticket/comment. Certainly nothing h... [22:07:21] 06Release-Engineering-Team, 06cloud-services-team: Kokkuri feature request: pipeline-configurable repo credentials - https://phabricator.wikimedia.org/T384396 (10Andrew) 03NEW [22:12:02] (03open) 10dancy: image.py: Don't pass KOKKURI_REGISTRY_PUBLIC to ensure_jwt_auth [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/108 [22:12:03] (03update) 10dancy: image.py: Don't pass KOKKURI_REGISTRY_PUBLIC to ensure_jwt_auth [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/108 [22:12:03] (03open) 10dancy: image.py: ensure_jwt_auth: Remove duplicate output [repos/releng/kokkuri] (main-Ic7bf0213cee0d7c152170bf2e169f3e85ea8308b) - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/109 [22:12:03] (03update) 10dancy: image.py: ensure_jwt_auth: Remove duplicate output [repos/releng/kokkuri] (main-Ic7bf0213cee0d7c152170bf2e169f3e85ea8308b) - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/109 [22:12:04] (03update) 10dancy: image.py: Support KOKKURI_REGISTRY_USER and KOKKURI_REGISTRY_PASSWORD [repos/releng/kokkuri] (main-Icb6d64a72d5145d54034a50d651c4172b03f764f) - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/110 [22:12:06] (03open) 10dancy: image.py: Support KOKKURI_REGISTRY_USER and KOKKURI_REGISTRY_PASSWORD [repos/releng/kokkuri] (main-Icb6d64a72d5145d54034a50d651c4172b03f764f) - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/110 [22:12:10] (03update) 10dancy: image.py: Don't pass KOKKURI_REGISTRY_PUBLIC to ensure_jwt_auth [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/108 [22:12:14] (03update) 10dancy: image.py: ensure_jwt_auth: Remove duplicate output [repos/releng/kokkuri] (main-Ic7bf0213cee0d7c152170bf2e169f3e85ea8308b) - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/109 [22:12:18] (03update) 10dancy: image.py: Support KOKKURI_REGISTRY_USER and KOKKURI_REGISTRY_PASSWORD [repos/releng/kokkuri] (main-Icb6d64a72d5145d54034a50d651c4172b03f764f) - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/110 [22:12:34] (03update) 10dancy: image.py: Don't pass KOKKURI_REGISTRY_PUBLIC to ensure_jwt_auth [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/108 [22:12:44] (03update) 10dancy: image.py: ensure_jwt_auth: Remove duplicate output [repos/releng/kokkuri] (main-Ic7bf0213cee0d7c152170bf2e169f3e85ea8308b) - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/109 [22:12:57] (03update) 10dancy: image.py: Support KOKKURI_REGISTRY_USER and KOKKURI_REGISTRY_PASSWORD [repos/releng/kokkuri] (main-Icb6d64a72d5145d54034a50d651c4172b03f764f) - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/110 [22:14:57] (03update) 10dancy: image.py: Support KOKKURI_REGISTRY_USER and KOKKURI_REGISTRY_PASSWORD [repos/releng/kokkuri] (main-Icb6d64a72d5145d54034a50d651c4172b03f764f) - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/110 (https://phabricator.wikimedia.org/T384396) [22:14:59] (03update) 10dancy: image.py: Support KOKKURI_REGISTRY_USER and KOKKURI_REGISTRY_PASSWORD [repos/releng/kokkuri] (main-Icb6d64a72d5145d54034a50d651c4172b03f764f) - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/110 (https://phabricator.wikimedia.org/T384396) [22:17:02] (03update) 10dancy: image.py: Support KOKKURI_REGISTRY_USER and KOKKURI_REGISTRY_PASSWORD [repos/releng/kokkuri] (main-Icb6d64a72d5145d54034a50d651c4172b03f764f) - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/110 (https://phabricator.wikimedia.org/T384396) [22:23:36] (03update) 10dancy: image.py: Support KOKKURI_REGISTRY_USER and KOKKURI_REGISTRY_PASSWORD [repos/releng/kokkuri] (main-Icb6d64a72d5145d54034a50d651c4172b03f764f) - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/110 (https://phabricator.wikimedia.org/T384396) [22:25:04] (03update) 10dancy: image.py: Support KOKKURI_REGISTRY_USER and KOKKURI_REGISTRY_PASSWORD [repos/releng/kokkuri] (main-Icb6d64a72d5145d54034a50d651c4172b03f764f) - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/110 (https://phabricator.wikimedia.org/T384396) [22:57:06] (03approved) 10dduvall: image.py: Don't pass KOKKURI_REGISTRY_PUBLIC to ensure_jwt_auth [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/108 (owner: 10dancy) [22:57:13] (03approved) 10dduvall: image.py: ensure_jwt_auth: Remove duplicate output [repos/releng/kokkuri] (main-Ic7bf0213cee0d7c152170bf2e169f3e85ea8308b) - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/109 (owner: 10dancy)