[00:46:18] <wikibugs>	 (03CR) 10Jeena Huneidi: "AFAIK the bug in the tests is fixed here https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikiLambda/+/1124538 and all that remains i" [integration/config] - 10https://gerrit.wikimedia.org/r/1135722 (https://phabricator.wikimedia.org/T368002) (owner: 10Jforrester)
[04:28:49] <wikibugs>	 (03CR) 10Vaughn Walters: "Once https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikiLambda/+/1136486 and https://gerrit.wikimedia.org/r/c/mediawiki/extensions/" [integration/config] - 10https://gerrit.wikimedia.org/r/1135722 (https://phabricator.wikimedia.org/T368002) (owner: 10Jforrester)
[05:59:38] <wikibugs>	 10MediaWiki-Releasing: Bundle Extension:CirrusSearch with MediaWiki core - https://phabricator.wikimedia.org/T391922#10742256 (10Bugreporter2)
[06:00:48] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 06MediaWiki-Stakeholders-Group, 07Epic, 05MW-1.44-release: Expand the set of bundled extensions and skins in MediaWiki 1.44 - https://phabricator.wikimedia.org/T333405#10742257 (10Bugreporter2)
[06:04:43] <wikibugs>	 10MediaWiki-Releasing, 10MobileFrontend (Tracking): Bundle MobileFrontend extension with MediaWiki - https://phabricator.wikimedia.org/T191734#10742260 (10Bugreporter2) p:05Triage→03Medium
[06:09:29] <wikibugs>	 10MediaWiki-Releasing, 10CirrusSearch, 06Discovery-Search: Bundle Extension:CirrusSearch with MediaWiki core - https://phabricator.wikimedia.org/T391922#10742263 (10Bugreporter2)
[06:29:49] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Disable "To:" and "CC:" Recipient Hint footers in mail notifications (metamta.recipients.show-hints) - https://phabricator.wikimedia.org/T391924 (10Aklapper) 03NEW p:05Triage→03Low
[06:35:44] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Disable "To:" and "CC:" Recipient Hint footers in mail notifications (metamta.recipients.show-hints) - https://phabricator.wikimedia.org/T391924#10742290 (10Aklapper)
[06:45:40] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 06MediaWiki-Stakeholders-Group, 07Epic, 05MW-1.44-release: Expand the set of bundled extensions and skins in MediaWiki 1.44 - https://phabricator.wikimedia.org/T333405#10742323 (10Bugreporter2)
[06:46:42] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 06MediaWiki-Stakeholders-Group, 07Epic, 05MW-1.44-release: Expand the set of bundled extensions and skins in MediaWiki 1.44 - https://phabricator.wikimedia.org/T333405#10742326 (10Bugreporter2)
[06:52:07] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 06MediaWiki-Stakeholders-Group, 07Epic, 05MW-1.44-release: Expand the set of bundled extensions and skins in MediaWiki 1.44 - https://phabricator.wikimedia.org/T333405#10742352 (10Bugreporter2)
[06:54:06] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 06MediaWiki-Stakeholders-Group, 07Epic, 05MW-1.44-release: Expand the set of bundled extensions and skins in MediaWiki 1.44 - https://phabricator.wikimedia.org/T333405#10742355 (10Bugreporter2) p:05Triage→03Medium
[06:55:55] <wikibugs>	 10MediaWiki-Releasing, 10MassMessage, 10MediaWiki-extensions-MassMessageEmail: Bundle Extension:MassMessage with MediaWiki core - https://phabricator.wikimedia.org/T391927#10742359 (10Bugreporter2)
[06:56:57] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 06MediaWiki-Stakeholders-Group, 07Epic, 05MW-1.44-release: Expand the set of bundled extensions and skins in MediaWiki 1.44 - https://phabricator.wikimedia.org/T333405#10742360 (10Bugreporter2)
[07:11:40] <wikibugs>	 10Phabricator: Dark mode (experimental) has contrast issues - https://phabricator.wikimedia.org/T391929 (10Danielyepezgarces) 03NEW
[07:48:00] <codders>	 seeing some cache corruption issues on CI jobs: https://integration.wikimedia.org/ci/job/mwext-node20-rundoc/7700/console
[07:48:17] <codders>	 should i delete the cache over at integration-castor05.integration.eqiad1.wikimedia.cloud ?
[07:57:19] <hashar>	 00:00:26.113 file has vanished: "/castor-mw-ext-and-skins/master/mwext-node20-rundoc/_logs/2025-04-15T07_14_17_845Z-debug-0.log" (in caches)
[07:57:20] <hashar>	 eek
[07:57:30] <hashar>	 makes you wonder why we have npm logs in the cache :/
[07:57:54] <hashar>	 and `file has vanished` is because another rsync is busy deleting the files at the same time
[07:58:02] <codders>	 fun!
[07:58:31] <hashar>	 that is a race condition that has been identified for a while
[07:58:31] <hashar>	 T295351
[07:58:32] <stashbot>	 T295351: npm cache saved by castor get corrupted for unknown reason - https://phabricator.wikimedia.org/T295351
[07:58:32] <hashar>	 :)
[08:00:10] <hashar>	 yeah confirmed
[08:01:06] <hashar>	 codders: so I think you can recheck
[08:01:11] <hashar>	 the cache is probably fine
[08:01:15] <codders>	 thanks!
[08:01:49] <codders>	 I've resubmitted. will see what happens
[08:13:45] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Seen): Phacility (Maintainer of Phabricator) is winding down. Upstream support ending. - https://phabricator.wikimedia.org/T283980#10742539 (10Caoduoclieu) CAO DƯỢC LIỆU CHẤT LƯỢNG UY TÍN NHẤT HIỆN NAY  1. [[ https://chietxuatduoclieu.com/chiet-xuat-cao-duoc-lieu...
[08:22:43] <wikibugs>	 10MediaWiki-Releasing: Bundle Skin:Chameleon with MediaWiki core - https://phabricator.wikimedia.org/T391928#10742586 (10Peachey88) The maintaining team of the skin will need to agree to move over to gerrit instead of github at a minimum
[08:24:35] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-extensions-CodeMirror: Bundle Extension:CodeMirror with MediaWiki core - https://phabricator.wikimedia.org/T391926#10742588 (10Peachey88)
[08:25:39] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-extensions-CodeMirror: Bundle Extension:CodeMirror with MediaWiki core - https://phabricator.wikimedia.org/T391926#10742590 (10Peachey88) `(subtask of T333405)`  Please actually set Subtask status, instead of just describing it in the text.
[08:27:22] <wikibugs>	 06Release-Engineering-Team, 06serviceops: scap train-presync failed to push image: blob upload unknown - https://phabricator.wikimedia.org/T391935 (10hashar) 03NEW
[08:27:50] <wikibugs>	 06Release-Engineering-Team, 06serviceops: scap train-presync failed to push image: blob upload unknown - https://phabricator.wikimedia.org/T391935#10742603 (10hashar)
[08:27:54] <wikibugs>	 10Release-Engineering-Team (Doing 😎), 07Essential-Work, 05Release, 05Train Deployments: 1.44.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T386220#10742604 (10hashar)
[08:28:15] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-extensions-CodeMirror: Bundle Extension:CodeMirror with MediaWiki core - https://phabricator.wikimedia.org/T391926#10742607 (10taavi) p:05Medium→03Triage Please do not set priorities unless you actually plan to work on something.
[08:30:24] <wikibugs>	 06Release-Engineering-Team, 06serviceops: scap train-presync failed to push image: blob upload unknown - https://phabricator.wikimedia.org/T391935#10742612 (10hashar)
[08:30:43] <wikibugs>	 06Release-Engineering-Team, 06serviceops: scap train-presync failed to push image: blob upload unknown - https://phabricator.wikimedia.org/T391935#10742619 (10hashar) p:05Triage→03Unbreak! That is blocking the train, thus setting up {nav Unbreak Now!}
[08:52:50] <wikibugs>	 10Scap, 10Wikidata, 10Wikidata-Query-Service, 10Data-Platform-SRE (2025-04-12 - 2025-05-02), 10Discovery-Search (2025.04.11 - 2025.05.02): scap service restarts for WDQS are inconsistent - https://phabricator.wikimedia.org/T221709#10742665 (10dcausse) 05Open→03Resolved tested wdqs & wcqs deploys...
[08:54:35] <jnuche>	 jouncebot: nowandnext
[08:55:28] <hashar>	 poor jouncebot
[08:55:44] <hashar>	 jnuche: the bot is in -operations :]
[08:56:00] <jnuche>	 oops, yeah :)
[08:56:52] <jnuche>	 hashar: I'm gonna try to run the presync again, there's a good chance the blob issue doesn't happen consistently and we can the the presync through
[08:57:23] <jnuche>	 *get the presync
[08:57:42] <hashar>	 I do not know :)
[08:58:03] <hashar>	 the thing is presync is going to rebuild a whole new image isn't it?
[08:58:05] <hashar>	 err scap
[08:58:40] <hashar>	 hmm or maybe given it already has build the image on the deployment server, it will just do the delta for changes that might have occurred since it last ran
[08:58:50] <hashar>	 and save you from a full image rebuild
[08:58:52] <jnuche>	 that's not deterministic anymore AFAIK. Scap will take a look at the image and decide whether to build from scratch
[08:59:02] <hashar>	 great
[08:59:17] <hashar>	 I am quite happy to see that one got addressed :)
[08:59:26] <hashar>	 I thought about doing the docker push
[08:59:33] <hashar>	 but I have no idea which credentials are needed
[08:59:42] <hashar>	 so yeah scap presync is probably the way to go (+ got us logging)
[08:59:57] <hashar>	 I am so unfamiliar with that whole stack
[09:00:48] <jnuche>	 I'm not that familiar myself either tbh
[09:00:52] <jnuche>	 ok, it's running now
[09:01:03] <hashar>	 hurrah
[09:03:20] <wikibugs>	 10WikimediaDebug, 07Documentation, 05MediaWiki-backport-deployments: X-Wikimedia-Debug page should have short video on how to use the extension - https://phabricator.wikimedia.org/T197230#10742674 (10hashar) 05Open→03Declined https://wikitech.wikimedia.org/wiki/WikimediaDebug#Browser_usage has the th...
[09:15:30] <jnuche>	 hashar: presync has been stuck at push for the last ~12m. I think it's pretty safe to assume the same problem is happening again: `09:02:15 [mediawiki-publish] docker-registry.discovery.wmnet/restricted/mediawiki-multiversion-cli:2025-04-15-090125-publish`
[09:16:07] <hashar>	 wonderful! make sure to report about it on https://phabricator.wikimedia.org/T391935
[09:16:29] <hashar>	 I was wondering earlier why it would be stuck for 30~ minutes 
[09:16:35] <hashar>	 maybe it is retrying 
[09:21:20] <wikibugs>	 06Release-Engineering-Team, 06serviceops: scap train-presync failed to push image: blob upload unknown - https://phabricator.wikimedia.org/T391935#10742697 (10jnuche) I tried running the presync again earlier today. The same problem seemed to happen, with the image push stuck for around 14m before I aborted ma...
[09:22:04] <hashar>	 https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&var-server=deploy1003&var-datasource=thanos&var-cluster=misc
[09:22:10] <hashar>	 it did ~ 9 minutes of disk cruniching
[09:22:28] <hashar>	 some transmit
[09:22:34] <hashar>	 and well. That is about it
[09:22:55] <hashar>	 maybe docker has some detailed logs
[09:34:15] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Testing Support, 10ci-test-error (WMF-deployed Build Failure), 10MW-1.44-notes (1.44.0-wmf.23; 2025-04-01), and 2 others: Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#10742736 (10zeljkofil...
[09:41:38] <wikibugs>	 06Release-Engineering-Team, 06serviceops: scap train-presync failed to push image: blob upload unknown - https://phabricator.wikimedia.org/T391935#10742760 (10Clement_Goubert) This is {T390251}, will take a look
[09:46:45] <wikibugs>	 10Phabricator: Dark mode (experimental) has contrast issues - https://phabricator.wikimedia.org/T391929#10742781 (10Aklapper) 05Open→03Declined Hi, thanks for reporting this! I do not plan to track [upstream](https://www.mediawiki.org/wiki/Upstream_projects) Dark Mode issues in Wikimedia Phabricator itse...
[09:52:09] <claime>	 looking the docker registry blocking the train, I don't have a good quick solution except retrying until it somehow works, not unless d.ancy has commited something we can finish for the image validation
[09:56:26] <wikibugs>	 10MediaWiki-Releasing: Bundle Skin:Chameleon with MediaWiki core - https://phabricator.wikimedia.org/T391928#10742793 (10Aklapper) 05Open→03Declined p:05Medium→03Triage Please do not set priorities unless you actually plan to work on something. Also, no.
[11:05:37] <wikibugs>	 06Project-Admins, 06Release-Engineering-Team, 05MediaWiki-backport-deployments: Is the #MediaWiki-backport-deployments project desired (& if so, how should it be used)? - https://phabricator.wikimedia.org/T391779#10743024 (10A_smart_kitten) 05Invalid→03Open
[11:25:15] <claime>	 I managed to push images (just a sync-world), it took 15 minutes...
[11:39:56] <jnuche>	 hum, no idea if that's a different issue or could be related to the bad blobs, we're gonna need some of the US folks to come online and take a look
[11:46:03] <claime>	 it is related
[11:46:05] <claime>	 /var/log/daemon.log.1:Apr 10 11:43:07 deploy1003 dockerd[3956212]: time="2025-04-10T11:43:07.994473043Z" level=error msg="Upload failed, retrying: blob upload unknown"
[11:46:42] <claime>	 basically it uploads, then tries to fetch to verify, but gets a bad blob
[11:46:44] <claime>	 so it reuploads
[11:46:48] <claime>	 (afaiu)
[12:05:47] <wikibugs>	 10MediaWiki-Releasing, 06Growth-Team, 10WikiLove: Bundle Extension:WikiLove with MediaWiki core - https://phabricator.wikimedia.org/T391952 (10Bugreporter2) 03NEW
[12:06:57] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 06MediaWiki-Stakeholders-Group, 07Epic, 05MW-1.44-release: Expand the set of bundled extensions and skins in MediaWiki 1.44 - https://phabricator.wikimedia.org/T333405#10743218 (10Bugreporter2)
[12:12:55] <wikibugs>	 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 2 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10743284 (10Jelto) There were some problem adding the Ceph apus credentials to puppet. It was mostly an i...
[12:14:17] <wikibugs>	 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 2 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10743317 (10Jelto)
[12:20:53] <wikibugs>	 (03open) 10cgoubert: build-images.py: Temp sleep for swift consistency [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/164 (https://phabricator.wikimedia.org/T390251)
[12:24:38] <wikibugs>	 (03merge) 10cgoubert: build-images.py: Temp sleep for swift consistency [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/164 (https://phabricator.wikimedia.org/T390251)
[12:29:22] <wikibugs>	 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 2 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10743380 (10Jelto)
[12:34:43] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 06MediaWiki-Stakeholders-Group, 07Epic, 05MW-1.44-release: Expand the set of bundled extensions and skins in MediaWiki 1.44 - https://phabricator.wikimedia.org/T333405#10743391 (10Aklapper) p:05Medium→03Triage
[12:35:29] <wikibugs>	 10MediaWiki-Releasing, 06Growth-Team, 10WikiLove: Bundle Extension:WikiLove with MediaWiki core - https://phabricator.wikimedia.org/T391952#10743392 (10Bugreporter2) p:05Triage→03Low
[12:39:48] <hashar>	 andre: I am not sure I get the intent behind adding bunch of extensions to the tarball :)
[12:40:07] <andre>	 hashar: me neither. wishlist days I guess.
[12:40:25] <hashar>	 I'll decline the one about WikiLove
[12:40:27] <andre>	 especially that late. but hey, if folks bump the MW version in the task every six months that's what you get.
[12:41:17] <andre>	 if there was only a PM saying No (the most important word a PM can say) :P
[12:49:13] <hashar>	 ah
[12:49:18] <hashar>	 so the user went on https://www.mediawiki.org/wiki/Suggestions_for_extensions_to_be_integrated
[12:49:25] <hashar>	 and copy pasted each entries to tasks 
[12:49:26] <hashar>	 :/
[12:49:44] <hashar>	 well at least an answer is given via Phabricator :)
[12:50:09] <wikibugs>	 10MediaWiki-Releasing, 06Growth-Team, 10WikiLove: Bundle Extension:WikiLove with MediaWiki core - https://phabricator.wikimedia.org/T391952#10743473 (10taavi) p:05Low→03Triage
[12:50:10] <hashar>	 https://phabricator.wikimedia.org/p/Bugreporter2/
[12:50:10] <hashar>	 :(
[12:50:26] <wikibugs>	 10MediaWiki-Releasing, 06Growth-Team, 10WikiLove: Bundle Extension:WikiLove with MediaWiki core - https://phabricator.wikimedia.org/T391952#10743481 (10hashar) WikiLove was a short WMF project from back in 2011. It was meant to easily thanks people for their contributions with the intent to increase editors...
[12:54:57] <wikibugs>	 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [Ironblade101] - https://phabricator.wikimedia.org/T391203#10743528 (10Ironblade101)
[12:55:12] <wikibugs>	 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [Ironblade101] - https://phabricator.wikimedia.org/T391203#10743529 (10Ironblade101)
[12:56:20] <wikibugs>	 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [Ironblade101] - https://phabricator.wikimedia.org/T391203#10743540 (10Ironblade101)
[13:09:36] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 06collaboration-services, 10Wikidata, 10Wikidata Query UI, and 2 others: move query.wikidata.org to kubernetes - https://phabricator.wikimedia.org/T350793#10743588 (10Jelto)
[13:30:51] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 06collaboration-services, 10Wikidata, 10Wikidata Query UI, and 2 others: move query.wikidata.org to kubernetes - https://phabricator.wikimedia.org/T350793#10743671 (10Jelto) All (known) queryservice frontends run in wikikube now (gui and querybuilder).  I prepared t...
[13:37:01] <wikibugs>	 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 2 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10743705 (10MatthewVernon) Looking at the Ceph metrics, it seems the packages were fewer larger objects,...
[14:01:21] <MichaelG_WMF>	 Not to be confused with https://phabricator.wikimedia.org/p/Bugreporter/
[14:03:14] <claime>	 that's :sus: tbh
[14:19:48] <wikibugs>	 (03CR) 10Hashar: "I am not sure why the image has a copy of `operations/puppet` since CI has to fetch the patches anyway and would clone from a git mirror o" [integration/config] - 10https://gerrit.wikimedia.org/r/1135116 (owner: 10JHathaway)
[14:25:57] <wikibugs>	 (03CR) 10Hashar: [C:03+2] "That one is good enough for a first pass" [integration/gerrit-admin] - 10https://gerrit.wikimedia.org/r/1134018 (https://phabricator.wikimedia.org/T390719) (owner: 10Hashar)
[14:31:49] <wikibugs>	 (03Merged) 10jenkins-bot: Tool to inspect projects submit strategies [integration/gerrit-admin] - 10https://gerrit.wikimedia.org/r/1134018 (https://phabricator.wikimedia.org/T390719) (owner: 10Hashar)
[14:54:46] <dancy>	 o/
[14:54:47] <dancy>	 Good morning
[14:58:24] <andre>	 o/
[14:59:38] <brennen>	 morning
[15:48:51] <wikibugs>	 10Release-Engineering-Team (Doing 😎), 10MediaWiki-Special-pages: Explain on Special:Statistics that "Content pages" by default excludes pages without links - https://phabricator.wikimedia.org/T386069#10744393 (10Aklapper)
[16:29:35] <wikibugs>	 10Phabricator: Dark mode (experimental) has contrast issues - https://phabricator.wikimedia.org/T391929#10744605 (10Danielyepezgarces) >>! El T391929#10742781, @Aklapper escribió: > Hi, thanks for reporting this! I do not plan to track [upstream](https://www.mediawiki.org/wiki/Upstream_projects) Dark Mode is...
[16:29:59] <wikibugs>	 10Release-Engineering-Team (Doing 😎), 10MediaWiki-Special-pages, 10MW-1.44-notes (1.44.0-wmf.26; 2025-04-22), 13Patch-For-Review: Explain on Special:Statistics that "Content pages" by default excludes pages without links - https://phabricator.wikimedia.org/T386069#10744607 (10Aklapper) 05Open→03Reso...
[16:45:03] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Testing Support, 10ci-test-error (WMF-deployed Build Failure), 10MW-1.44-notes (1.44.0-wmf.23; 2025-04-01), and 2 others: Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#10744649 (10Daimona)...
[17:13:30] <wikibugs>	 06Project-Admins, 06Release-Engineering-Team, 05MediaWiki-backport-deployments: Is the #MediaWiki-backport-deployments project desired (& if so, how should it be used)? - https://phabricator.wikimedia.org/T391779#10744711 (10Pppery) I would archive the project entirely.  - Tasks about the backport tooling ->...
[17:24:15] <MatmaRex>	 is beta cluster having a bad time, or is it just me?
[17:24:31] <MatmaRex>	 pages are taking forever to load
[17:24:49] <bd808>	 beta.wmflabs.org seems slow/stuck from my laptop
[17:26:35] <bd808>	 there are some tall spikes on the aggregated load graph for the Cloud VPS project in total
[17:27:13] <bd808>	 I finally got a timeout from deployment-cache-text08 trying to get data from the backing MediaWiki.
[17:30:01] <bd808>	 load on deployment-mediawiki14 is ~6. https://grafana.wmcloud.org/d/0g9N-7pVz/cloud-vps-project-board?orgId=1&var-project=deployment-prep&var-instance=All&from=now-2d&to=now&viewPanel=902
[17:30:56] <bd808>	 !log `shutdown -r now` on deployment-mediawiki14. Load has been growing for ~2 days.
[17:30:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:31:33] <bd808>	 MatmaRex: maybe better for a bit. ¯\_(ツ)_/¯
[17:32:56] <bd808>	 its popping right back up though. I need to find hashar's notes on looking at the inbound traffic.
[17:32:56] <MatmaRex>	 huh, interesting
[17:34:22] <bd808>	 yeah, it's right back up to 6 again with 9 parallel php processes at the top of the %CPU
[17:35:49] <MatmaRex>	 so, scraping, surely?
[17:36:42] <bd808>	 zuul is very quiet, so yeah I would guess some bots being agressive
[17:38:07] <wikibugs>	 10Beta-Cluster-Infrastructure: HIgh load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392003 (10bd808) 03NEW
[17:41:45] <wikibugs>	 10Beta-Cluster-Infrastructure: HIgh load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392003#10744829 (10bd808) {T389181} was a prior round of overload (sorry, private task because of lots of IPv4 addresses in it)
[17:46:37] <MatmaRex>	 thanks for filing it
[17:46:43] <bd808>	 looks like some bots running in Azure may have decide to crawl beta agressively
[17:53:58] <wikibugs>	 10Beta-Cluster-Infrastructure: HIgh load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392003#10744847 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/f2d68f95b1405955014af32d4eb6b7c83d201083%5E%21/ ` diff --git a/deployment-prep/_.yaml b/deplo...
[17:56:24] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-extensions-CodeMirror: Bundle Extension:CodeMirror with MediaWiki core - https://phabricator.wikimedia.org/T391926#10744855 (10Izno) >>! In T391926#10742409, @Bugreporter wrote: > We should wait until {T373711} is resolved. There is no reason to bundle both CodeEditor and Cod...
[18:04:35] <wikibugs>	 10Beta-Cluster-Infrastructure: HIgh load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392003#10744926 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/290e8a5c6939740c79abbe71546be60178ac8f1c%5E%21/#F0 ` diff --git a/deployment-prep/_.yaml b/de...
[18:06:15] <bd808>	 !log `sudo puppet agent -tv` on deployment-cache-text08 to update varnish deny list (T392003)
[18:06:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[18:06:17] <stashbot>	 T392003: HIgh load on deployment-mediawiki14 and slow responses  - https://phabricator.wikimedia.org/T392003
[18:11:26] <bd808>	 !log `bd808@deployment-cache-text08:~$ sudo service varnish-frontend restart` (T392003)
[18:11:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[18:11:28] <stashbot>	 T392003: HIgh load on deployment-mediawiki14 and slow responses  - https://phabricator.wikimedia.org/T392003
[18:17:56] <bd808>	 so. many. bots.
[18:18:59] <Daimona>	 Sigh.
[18:19:48] <bd808>	 I whacked a bunch from Azure, but there are still a flood from OVH and a couple of other big IP blocks
[18:20:33] * bd808 decides to eat before he just blocks the whole world
[19:18:12] <wikibugs>	 10Release-Engineering-Team (Doing 😎), 07Essential-Work, 05Release, 05Train Deployments: 1.44.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T386220#10745209 (10Jdforrester-WMF)
[19:37:44] <wikibugs>	 10Beta-Cluster-Infrastructure: HIgh load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392003#10745273 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/44875f6c7725d413a0071da0f18cec6f1bbf749d%5E%21/#F0 ` diff --git a/deployment-prep/_.yaml b/de...
[19:40:10] <bd808>	 !log Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks (T392003)
[19:40:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[19:40:13] <stashbot>	 T392003: HIgh load on deployment-mediawiki14 and slow responses  - https://phabricator.wikimedia.org/T392003
[19:42:42] <bd808>	 I think I may have whacked enough network blocks to slow the bots down for a while.
[19:51:03] <wikibugs>	 10Beta-Cluster-Infrastructure: HIgh load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392003#10745322 (10bd808) The last hiera change I made for this is https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/8697c499f4c8e91428329ea98b4e519de03e3507%5E%21/#F0....
[20:01:03] <wikibugs>	 10Beta-Cluster-Infrastructure: HIgh load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392003#10745370 (10bd808) 05Open→03Resolved a:03bd808
[21:15:43] <swfrench-wmf>	 I feel like I should know the answer to this, but alas I do not: how does one re-trigger gate-and-submit on a patch that already has +2/+2?
[21:18:04] <swfrench-wmf>	 I can recheck, which confirms that the job _should_ succeed (now that the broken test has been marked skipped in a separate patch), but I have no idea how to re-trigger the gate-and-submit itself
[21:20:53] <bd808>	 swfrench-wmf: remove  +2 and then add it again
[21:24:44] <swfrench-wmf>	 bd808: thanks! I was guessing it might come to that :) it's not a repo where I would generally be comfortable self-+2'ing, but it seems that should actually be fine in this case
[21:32:08] <bd808>	 yeah, there will be a log on the commit so folks will be able to tell what happened. I think if someone else has given the prior CR:+2 you can actually add your own CR:+2 without removing theirs... maybe?
[21:33:34] <swfrench-wmf>	 got it, yeah that's what I realized after the fact that I could probably do - i.e., provide my own review +2 in addition (since there was no change to the patchset, the earlier votes was retained)
[22:00:01] <wikibugs>	 (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/WikiLambda] Make Catalyst job voting [integration/config] - 10https://gerrit.wikimedia.org/r/1135722 (https://phabricator.wikimedia.org/T368002)
[22:00:26] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/WikiLambda] Make Catalyst job voting [integration/config] - 10https://gerrit.wikimedia.org/r/1135722 (https://phabricator.wikimedia.org/T368002) (owner: 10Jforrester)
[22:02:12] <wikibugs>	 (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WikiLambda] Make Catalyst job voting [integration/config] - 10https://gerrit.wikimedia.org/r/1135722 (https://phabricator.wikimedia.org/T368002) (owner: 10Jforrester)
[22:02:21] <James_F>	 swfrench-wmf, bd808: You don't need to remove and add, if you've already C+2'ed just comment and that comment will be a C+2-plus-comment in CI's eyes.
[22:02:35] <James_F>	 !log Zuul: [mediawiki/extensions/WikiLambda] Make Catalyst job voting, for T368002
[22:02:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[22:02:37] <stashbot>	 T368002: Test Wikifunctions Services with 'Catalyst' - https://phabricator.wikimedia.org/T368002
[22:05:52] <bd808>	 James_F: TIL. Thanks!
[22:07:03] <bd808>	 I would guess in swfrench-wmf's case this time that means he would have needed to add a second CR:+2 since the original wasn't his.
[22:19:31] <swfrench-wmf>	 Thank you both! This answers my question, and explains why even though recheck itself does not actually trigger these jobs specifically, commenting _anything_ in conjunction with having already CR:+2’d would have triggered them again.
[23:14:05] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-extensions-CodeMirror: Bundle Extension:CodeMirror with MediaWiki core - https://phabricator.wikimedia.org/T391926#10745943 (10MusikAnimal) Thanks for filing this task! I was going to create one as well, eventually. We are still in "beta" for CodeMirror 6. I'd say it makes mo...
[23:33:42] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 06MediaWiki-Stakeholders-Group, 07Epic, 05MW-1.44-release: Expand the set of bundled extensions and skins in MediaWiki 1.44 - https://phabricator.wikimedia.org/T333405#10745968 (10Bugreporter2)
[23:42:49] <wikibugs>	 10Beta-Cluster-Infrastructure, 13Patch-For-Review, 07Wikimedia-production-error: PHP Warning: include(/srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Exception/ConnectException.php): Failed to open stream: Too many open files - https://phabricator.wikimedia.org/T389422#10745982 (10bd808) 05In pro...
[23:58:57] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Disable "To:" and "CC:" Recipient Hint footers in mail notifications (metamta.recipients.show-hints) - https://phabricator.wikimedia.org/T391924#10745990 (10bd808) I vaguely remember these being enabled to give folks more ways to filter and tag em...