[06:33:01] 10Gerrit, 07Upstream: Gerrit's syntax highlighting for PHP code breaks when encountering an apostrophe in a // comment in a function call - https://phabricator.wikimedia.org/T372404#10759375 (10hashar) https://github.com/highlightjs/highlight.js/pull/4230 has been merged by #upstream. That would be in highligh... [06:37:06] 10Continuous-Integration-Config, 10ci-test-error (WMF-deployed Build Failure), 05MW-1.42-release, 05MW-1.43-release: Extensions that pull in WebAuthn for their CI can support PHP 7.4 or PHP 8.2+, but not both (was: quibble-composer-mysql-php8[23] fail sin... - https://phabricator.wikimedia.org/T390754#10759382 [07:24:40] 10Gerrit (Gerrit 3.12): Upgrade to Gerrit 3.12 - https://phabricator.wikimedia.org/T392448 (10hashar) 03NEW [07:28:13] 10Gerrit (Gerrit 3.12): Upgrade to Gerrit 3.12 - https://phabricator.wikimedia.org/T392448#10759556 (10hashar) Since: * Gerrit 3.12 drops support for Java 17 and requires Java 21 * Debian Bookworm provides Java 17 * Debian Trixie will provide Java 21 My guess is we will have to either: * reimage the Gerrit serv... [07:43:04] 10Gerrit (Gerrit 3.12): Upgrade to Gerrit 3.12 - https://phabricator.wikimedia.org/T392448#10759611 (10MoritzMuehlenhoff) We have a maintained Java 21 backport for Bookworm! It was needed for Apereo CAS which also requires Java 21 and can be found in component/jdk21 As such, I'd propose the following plan inste... [09:11:19] 10Gerrit (Gerrit 3.12): Upgrade Gerrit hosts to Bookworm - https://phabricator.wikimedia.org/T392464 (10hashar) 03NEW [09:14:21] 10Gerrit (Gerrit 3.12): Upgrade to Gerrit 3.12 - https://phabricator.wikimedia.org/T392448#10759973 (10hashar) I have filed {T392464} to upgrade to Bookworm :-] Before or after that we can upgrade to Gerrit 3.11 T379714. Once we have both 3.11 and Bookworm we can switch to Java 21 and after that upgrade to 3.1... [09:23:18] 10Gerrit (Gerrit 3.12): Upgrade Gerrit hosts to Bookworm - https://phabricator.wikimedia.org/T392464#10760028 (10MoritzMuehlenhoff) [09:25:31] 10Gerrit: Switch Gerrit from Java 17 to Java 21 - https://phabricator.wikimedia.org/T392465#10760036 (10hashar) [09:26:00] 10Gerrit (Gerrit 3.12): Upgrade to Gerrit 3.12 - https://phabricator.wikimedia.org/T392448#10760039 (10hashar) [09:26:02] 10Gerrit: Switch Gerrit from Java 17 to Java 21 - https://phabricator.wikimedia.org/T392465#10760040 (10hashar) [09:26:36] 10Gerrit: Switch Gerrit from Java 17 to Java 21 - https://phabricator.wikimedia.org/T392465#10760046 (10hashar) [09:26:38] 10Gerrit (Gerrit 3.11), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services: Upgrade to Gerrit 3.11 - https://phabricator.wikimedia.org/T379714#10760047 (10hashar) [09:28:12] 10Gerrit: Switch Gerrit from Java 17 to Java 21 - https://phabricator.wikimedia.org/T392465#10760061 (10hashar) [09:28:15] 10Gerrit (Gerrit 3.12): Upgrade Gerrit hosts to Bookworm - https://phabricator.wikimedia.org/T392464#10760062 (10hashar) [09:34:06] (03PS1) 10Hashar: jjb: publish gearman using Java 17 [integration/config] - 10https://gerrit.wikimedia.org/r/1138299 [09:35:31] FIRING: [2x] ProbeDown: Service gerrit2002:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [09:35:38] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T392467 (10phaultfinder) 03NEW [09:37:56] (03PS1) 10CrioCz: Review access change [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1138300 [09:38:15] (03PS1) 10Hashar: jjb: remove Java 8 & 11 from gearman [integration/config] - 10https://gerrit.wikimedia.org/r/1138301 [09:38:30] 06Release-Engineering-Team, 06collaboration-services: ProbeDown (gerrit2002) - https://phabricator.wikimedia.org/T392467#10760089 (10Jelto) [09:38:51] (03Abandoned) 10CrioCz: Review access change [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1138300 (owner: 10CrioCz) [09:39:02] (03PS1) 10Hashar: dockerfiles: remove gearman-java8 and gearman-java11 [integration/config] - 10https://gerrit.wikimedia.org/r/1138302 [09:39:15] 06Release-Engineering-Team, 06collaboration-services: ProbeDown (gerrit2002) - https://phabricator.wikimedia.org/T392467#10760098 (10Jelto) 05Openβ†’03In progress p:05Triageβ†’03High a:03Jelto Gerrit is noticable slow, the nftables tracking list is increasing since one week: https://grafana.wikimedia.org... [09:40:31] RESOLVED: [2x] ProbeDown: Service gerrit2002:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:04:59] (03PS2) 10Hashar: jjb: publish gearman-java using Java 17 [integration/config] - 10https://gerrit.wikimedia.org/r/1138299 [10:05:50] (03CR) 10Hashar: [C:03+2] jjb: publish gearman-java using Java 17 [integration/config] - 10https://gerrit.wikimedia.org/r/1138299 (owner: 10Hashar) [10:06:08] (03PS2) 10Hashar: jjb: remove Java 8 & 11 from gearman-java [integration/config] - 10https://gerrit.wikimedia.org/r/1138301 [10:06:14] (03PS3) 10Hashar: jjb: remove Java 8 & 11 from gearman-java [integration/config] - 10https://gerrit.wikimedia.org/r/1138301 [10:07:41] (03Merged) 10jenkins-bot: jjb: publish gearman-java using Java 17 [integration/config] - 10https://gerrit.wikimedia.org/r/1138299 (owner: 10Hashar) [10:08:09] (03CR) 10Hashar: [C:03+2] jjb: remove Java 8 & 11 from gearman-java [integration/config] - 10https://gerrit.wikimedia.org/r/1138301 (owner: 10Hashar) [10:08:18] (03CR) 10Hashar: [C:03+2] dockerfiles: remove gearman-java8 and gearman-java11 [integration/config] - 10https://gerrit.wikimedia.org/r/1138302 (owner: 10Hashar) [10:09:53] (03Merged) 10jenkins-bot: jjb: remove Java 8 & 11 from gearman-java [integration/config] - 10https://gerrit.wikimedia.org/r/1138301 (owner: 10Hashar) [10:16:12] Project beta-code-update-eqiad build #544864: 04FAILURE in 3 min 11 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/544864/ [10:18:09] Project mediawiki-core-doxygen build #9889: 04FAILURE in 6.2 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/9889/ [11:15:37] 10GitLab (Pipeline Services Migration🐀), 06collaboration-services, 10Wikidata, 10Wikidata Query UI, 10wmde-wikidata-tech: move query.wikidata.org to kubernetes - https://phabricator.wikimedia.org/T350793#10760284 (10Jelto) [11:21:18] 10GitLab (Pipeline Services Migration🐀), 06collaboration-services, 10Wikidata, 10Wikidata Query UI, 10wmde-wikidata-tech: move query.wikidata.org to kubernetes - https://phabricator.wikimedia.org/T350793#10760306 (10Jelto) 05In progressβ†’03Resolved All query gui services are running on wikikube no... [11:21:48] 10GitLab (Pipeline Services Migration🐀), 06collaboration-services, 13Patch-For-Review: Move micro sites from Ganeti to Kubernetes and from Gerrit to GitLab - https://phabricator.wikimedia.org/T300171#10760309 (10Jelto) [11:31:18] Yippee, build fixed! [11:31:19] Project mediawiki-core-doxygen build #9890: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/9890/ [12:02:26] (03CR) 10Jforrester: "<3" [integration/config] - 10https://gerrit.wikimedia.org/r/1138301 (owner: 10Hashar) [12:12:32] (03CR) 10Hashar: [C:03+2] "Eventually we will get rid of everything! πŸ‘" [integration/config] - 10https://gerrit.wikimedia.org/r/1138301 (owner: 10Hashar) [12:26:06] 10Gerrit: Background image of login page is served with a short TTL - https://phabricator.wikimedia.org/T392479 (10hashar) 03NEW [13:10:31] hashar: Is gerrit really still running on Node 18? Its CI is about to be the last thing using it. [13:45:58] 10Scap (SpiderPig πŸ•ΈοΈ): Add browser notification when deployment is awaiting user interaction - https://phabricator.wikimedia.org/T392487 (10TheresNoTime) 03NEW [14:11:55] 10WikimediaDebug, 10Arc-Lamp, 10observability, 07Regression: Daily flame graph for "fn-EditAction" missing since 28 March 2025 - https://phabricator.wikimedia.org/T391517#10760919 (10lmata) Looks like things are working again: https://performance.wikimedia.org/arclamp/svgs/daily/2025-04-23.excimer-k8s-php8... [14:14:10] 10WikimediaDebug, 10Arc-Lamp, 10observability, 07Regression: Daily flame graph for "fn-EditAction" missing since 28 March 2025 - https://phabricator.wikimedia.org/T391517#10760932 (10Krinkle) The directory was never empty. This is about `fn-EditAction` which is still missing. Other datasets are unaffected. [14:14:41] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Migrate mwdebug* hosts to PHP8.1 - https://phabricator.wikimedia.org/T391452#10760935 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1002 for host mwdebug1001.eqiad.wmnet wi... [14:21:35] (03PS1) 10Jforrester: jjb: [wikimedia-portals-build] Switch to Node 20 [integration/config] - 10https://gerrit.wikimedia.org/r/1138375 (https://phabricator.wikimedia.org/T382179) [14:21:37] (03PS1) 10Jforrester: Zuul: [wikimedia/portals] Switch to Node 20 [integration/config] - 10https://gerrit.wikimedia.org/r/1138376 (https://phabricator.wikimedia.org/T382179) [14:24:44] (03CR) 10Jforrester: [C:03+2] jjb: [wikimedia-portals-build] Switch to Node 20 [integration/config] - 10https://gerrit.wikimedia.org/r/1138375 (https://phabricator.wikimedia.org/T382179) (owner: 10Jforrester) [14:25:00] (03CR) 10Jforrester: [C:03+2] Zuul: [wikimedia/portals] Switch to Node 20 [integration/config] - 10https://gerrit.wikimedia.org/r/1138376 (https://phabricator.wikimedia.org/T382179) (owner: 10Jforrester) [14:25:53] !log Zuul: [wikimedia/portals] Switch to Node 20, for T382179 [14:25:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:25:55] T382179: wikimedia/portals fails CI on Node 20 - https://phabricator.wikimedia.org/T382179 [14:26:05] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 10Cloud-VPS (Quota-requests), 13Patch-For-Review: Quota increase for zuul3 project - https://phabricator.wikimedia.org/T392294#10760977 (10fnegri) > I'm going to assume that @fnegri is seeing flavors that are hidden from mor... [14:26:22] (03Merged) 10jenkins-bot: jjb: [wikimedia-portals-build] Switch to Node 20 [integration/config] - 10https://gerrit.wikimedia.org/r/1138375 (https://phabricator.wikimedia.org/T382179) (owner: 10Jforrester) [14:26:34] (03Merged) 10jenkins-bot: Zuul: [wikimedia/portals] Switch to Node 20 [integration/config] - 10https://gerrit.wikimedia.org/r/1138376 (https://phabricator.wikimedia.org/T382179) (owner: 10Jforrester) [14:41:02] (03update) 10dancy: spiderpig: Add means to prepopulate backport change numbers [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/760 [14:41:08] (03update) 10dancy: spiderpig: Add means to prepopulate backport change numbers [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/760 [14:55:54] (03CR) 10Jforrester: "check experimental" [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1111944 (https://phabricator.wikimedia.org/T383597) (owner: 10Hashar) [15:00:22] (03PS1) 10Jforrester: Zuul: [mediawiki/services/parsoid/testreduce] Switch to Node 20 CI [integration/config] - 10https://gerrit.wikimedia.org/r/1138383 [15:00:22] (03PS1) 10Jforrester: jjb: Drop legacy-node18 and legacy-node18-browser, unused [integration/config] - 10https://gerrit.wikimedia.org/r/1138384 [15:00:22] (03PS1) 10Jforrester: Zuul: [operations/software/gerrit] Switch to Node 20 CI [integration/config] - 10https://gerrit.wikimedia.org/r/1138385 [15:00:23] (03PS1) 10Jforrester: jjb: Drop gerrit-node18, unused [integration/config] - 10https://gerrit.wikimedia.org/r/1138386 [15:00:24] (03PS1) 10Jforrester: Docker: Drop Node 18 images, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/1138387 [15:01:54] (03PS2) 10Hashar: dockerfiles: remove gearman-java8 and gearman-java11 [integration/config] - 10https://gerrit.wikimedia.org/r/1138302 [15:03:16] (03PS1) 10Peter Fischer: maven: update to 3.9.9 [integration/config] - 10https://gerrit.wikimedia.org/r/1138388 (https://phabricator.wikimedia.org/T367405) [15:04:30] (03CR) 10Jforrester: [C:03+2] "Let's actually land this. :-)" [integration/config] - 10https://gerrit.wikimedia.org/r/1138302 (owner: 10Hashar) [15:05:10] (03CR) 10CI reject: [V:04-1] maven: update to 3.9.9 [integration/config] - 10https://gerrit.wikimedia.org/r/1138388 (https://phabricator.wikimedia.org/T367405) (owner: 10Peter Fischer) [15:05:21] (03CR) 10Peter Fischer: "I am not quite sure about the versioning. I only updated the maven-java versions manually, everything else uses the auto-increment value o" [integration/config] - 10https://gerrit.wikimedia.org/r/1138388 (https://phabricator.wikimedia.org/T367405) (owner: 10Peter Fischer) [15:06:05] (03Merged) 10jenkins-bot: dockerfiles: remove gearman-java8 and gearman-java11 [integration/config] - 10https://gerrit.wikimedia.org/r/1138302 (owner: 10Hashar) [15:09:21] 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’): Add tooling to pgp-public-keys.git to make using and maintaining it easier - https://phabricator.wikimedia.org/T279051#10761191 (10thcipriani) 05Openβ†’03Declined [15:09:50] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Migrate mwdebug* hosts to PHP8.1 - https://phabricator.wikimedia.org/T391452#10761195 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1002 for host mwdebug1001.eqiad.wmnet with O... [15:09:52] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen): Have linters/tests results show up as comments in files on gerrit - https://phabricator.wikimedia.org/T209149#10761197 (10thcipriani) [15:09:57] (03PS2) 10Peter Fischer: maven: update to 3.9.9 [integration/config] - 10https://gerrit.wikimedia.org/r/1138388 (https://phabricator.wikimedia.org/T367405) [15:10:08] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Scap, 07Developer Productivity: Fix mwdebug dashboard to show mw and scap messages again - https://phabricator.wikimedia.org/T288576#10761199 (10thcipriani) [15:10:23] 10Release-Engineering-Team (Seen): Add local-charts to mw-cli - https://phabricator.wikimedia.org/T281391#10761216 (10thcipriani) p:05Triageβ†’03Low [15:10:55] 10Phabricator, 10Release-Engineering-Team (Seen): On project workboard cards, show status of associated Gerrit patches - https://phabricator.wikimedia.org/T215148#10761219 (10thcipriani) [15:12:36] 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10Scap (SpiderPig πŸ•ΈοΈ): Add browser notification when deployment is awaiting user interaction - https://phabricator.wikimedia.org/T392487#10761222 (10thcipriani) [15:12:43] 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10Scap (SpiderPig πŸ•ΈοΈ): Integrate mwscript-k8s with SpiderPig - https://phabricator.wikimedia.org/T392275#10761223 (10thcipriani) [15:12:58] 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10Scap (SpiderPig πŸ•ΈοΈ), 10logspam-watch, 10observability, 10Observability-Logging: Add a log view to SpiderPig - https://phabricator.wikimedia.org/T391005#10761229 (10thcipriani) [15:14:37] 10Scap (SpiderPig πŸ•ΈοΈ): SpiderPig UI: improve job status - https://phabricator.wikimedia.org/T383853#10761238 (10thcipriani) 05Openβ†’03Resolved a:03dancy Added some progress to the status: works well, seem prominent. Claiming success :) [15:16:16] 10Phabricator, 06Security-Team, 07SecTeam-Processed: Audit Phabricator security policies and groups membership - https://phabricator.wikimedia.org/T391150#10761270 (10sbassett) @jly has refreshed the previous audit task (see: T305577#10758812). We've already gotten a few responses, which is great. Unfortun... [15:16:47] 10Scap (SpiderPig πŸ•ΈοΈ): SpiderPig UI: investigate Login console warnings - https://phabricator.wikimedia.org/T385506#10761319 (10thcipriani) 05Openβ†’03Declined Login is via IDP now. Not seeing this on the TOTP screen. [15:22:43] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 10Cloud-VPS (Quota-requests), 13Patch-For-Review: Quota increase for zuul3 project - https://phabricator.wikimedia.org/T392294#10761335 (10Andrew) Here's my mortal user adding a VM in zuul3. The user is a member and reader b... [15:24:26] 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10Scap (SpiderPig πŸ•ΈοΈ): Integrate SpiderPig with the Deployment calendar - https://phabricator.wikimedia.org/T392507 (10thcipriani) 03NEW [15:33:14] (03open) 10fceratto: Add data persistence service: zarcillo [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/112 [15:36:23] (03merge) 10dancy: Add data persistence service: zarcillo [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/112 (owner: 10fceratto) [15:39:11] 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10Scap (SpiderPig πŸ•ΈοΈ): Add browser notification when deployment is awaiting user interaction - https://phabricator.wikimedia.org/T392487#10761465 (10bd808) I think this one was on a notes doc from earlier testing too. Thanks for getting it into Phab. [15:45:04] 10Scap (SpiderPig πŸ•ΈοΈ): Automatically select first backport search match - https://phabricator.wikimedia.org/T392508 (10bd808) 03NEW [15:45:58] 10Scap (SpiderPig πŸ•ΈοΈ): Automatically select first backport search match - https://phabricator.wikimedia.org/T392508#10761523 (10bd808) 05Openβ†’03In progress p:05Triageβ†’03Medium a:03bd808 I have been poking at this for a while at https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/731 but jus... [15:46:04] (03merge) 10dancy: spiderpig: Add means to prepopulate backport change numbers [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/760 [15:46:54] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Migrate mwdebug* hosts to PHP8.1 - https://phabricator.wikimedia.org/T391452#10761535 (10jijiki) 05In progressβ†’03Resolved [15:47:58] (03open) 10dancy: Release 4.154.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/761 [15:50:02] (03merge) 10dancy: Release 4.154.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/761 [15:54:29] 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10Scap (SpiderPig πŸ•ΈοΈ): Automatically select first backport search match - https://phabricator.wikimedia.org/T392508#10761566 (10bd808) [16:12:56] Hi releng folks! [16:13:10] Could someone please publish the latest dev-images to the docker registry? [16:14:02] This patch merged a couple weeks ago and the corresponding image still doesn't seem to be available [16:14:06] https://gitlab.wikimedia.org/repos/releng/dev-images/-/commit/a80e5211100f1cc42e4ae020d4266ea22938eb5a [16:20:38] ejegg: Who merged it? Did they not follow the publication steps? [16:29:21] 10MediaWiki-Releasing, 05MW-1.44-notes, 05MW-1.44-release, 10MW-Interfaces-Team (MW-Sprint-8 (2025-04-23 to 2025-05-06)), 13Patch-For-Review: Update CREDITS in MW master - https://phabricator.wikimedia.org/T390700#10761799 (10HCoplin-WMF) [16:31:24] 10Gerrit: Background image of login page is served with a short TTL - https://phabricator.wikimedia.org/T392479#10761823 (10hashar) The one year max age is only set when one requests a static file with the `e` parameter set to an ETag matching the requested file. Eg: Request file: ` $ curl --head https://gerrit... [16:38:05] 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10Scap (SpiderPig πŸ•ΈοΈ): Integrate SpiderPig with the Deployment calendar - https://phabricator.wikimedia.org/T392507#10761857 (10bd808) https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/760 has landed such that `https://spiderpig.wikimedia.org?backport=... [16:39:07] 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10Scap (SpiderPig πŸ•ΈοΈ): Add browser notification when deployment is awaiting user interaction - https://phabricator.wikimedia.org/T392487#10761869 (10jnuche) a:03jnuche [16:40:49] 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10Scap (SpiderPig πŸ•ΈοΈ): Add browser notification when deployment is awaiting user interaction - https://phabricator.wikimedia.org/T392487#10761880 (10dancy) Question to consider: Who should receive a browser notification? All users logged into spiderpig? Just the u... [16:48:11] 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10Scap (SpiderPig πŸ•ΈοΈ): Add browser notification when deployment is awaiting user interaction - https://phabricator.wikimedia.org/T392487#10761905 (10jnuche) >Question to consider: Who should receive a browser notification? All users logged into spiderpig? Just the u... [16:49:14] 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’): Yak: Gerrit user page graph - https://phabricator.wikimedia.org/T280091#10761907 (10taavi) 05Openβ†’03Resolved a:03Chlod @chlod's https://techcontribs.toolforge.org/ does this. [16:50:00] ejegg, James_F: looking. [16:53:52] thanks brennen ! [16:57:18] !log Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/commit/a80e5211100f1cc42e4ae020d4266ea22938eb5a (T383097) [16:57:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:57:20] T383097: Add wmf-cv to our dev environments - https://phabricator.wikimedia.org/T383097 [16:58:27] 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’): Yak: Gerrit user page graph - https://phabricator.wikimedia.org/T280091#10761953 (10thcipriani) >>! In T280091#10761907, @taavi wrote: > @chlod's https://techcontribs.toolforge.org/ does this. whoa! This is really cool! Nice work, @Chlod [17:06:36] 10MediaWiki-Releasing, 10MediaWiki-Installer, 05MW-1.45-release: Expand the set of bundled extensions and skins in MediaWiki 1.45 - https://phabricator.wikimedia.org/T392080#10762007 (10Pppery) [17:06:39] ejegg: Successfully published image docker-registry.discovery.wmnet/dev/fundraising-civicrm-bookworm-php82-apache2:1.1.0 [17:07:07] sidebar: there's... a bunch of stuff in here that no longer builds and i'm guessing is safe to remove. [17:07:08] 10MediaWiki-Releasing, 10MediaWiki-extensions-ContactPage: Bundle Extension:ContactForm with MediaWiki core - https://phabricator.wikimedia.org/T392444#10762009 (10Pppery) [17:07:12] 10MediaWiki-Releasing, 10MediaWiki-extensions-ContactPage: Bundle Extension:ContactPage with MediaWiki core - https://phabricator.wikimedia.org/T392444#10762010 (10Pppery) [17:10:06] 10MediaWiki-Releasing, 10MediaWiki-extensions-ContactPage: Bundle Extension:ContactPage with MediaWiki core - https://phabricator.wikimedia.org/T392444#10762027 (10Dreamy_Jazz) [17:11:31] 10MediaWiki-Releasing, 10MediaWiki-extensions-ContactPage: Bundle Extension:ContactPage with MediaWiki core - https://phabricator.wikimedia.org/T392444#10762031 (10Reedy) p:05Triageβ†’03Low [17:11:55] 10MediaWiki-Releasing, 10MediaWiki-extensions-ContactPage: Bundle Extension:ContactPage with MediaWiki core - https://phabricator.wikimedia.org/T392444#10762037 (10Dreamy_Jazz) [17:13:13] 10MediaWiki-Releasing, 10MediaWiki-extensions-ContactPage: Bundle Extension:ContactPage with MediaWiki core - https://phabricator.wikimedia.org/T392444#10762046 (10Dreamy_Jazz) > Extension's default configuration provides optimal experience I would disagree with this. For a wiki to define a #mediawiki-extensio... [17:23:51] 10Phabricator, 06Security-Team, 07SecTeam-Processed: Audit Phabricator security policies and groups membership - https://phabricator.wikimedia.org/T391150#10762129 (10Jly) Thanks Scott! Folks, I'd like to propose a cut-off date for this in 3 months, which I will notify in #T306708. Users who still have not... [17:37:22] thank you brennen ! [17:37:43] sure thing [17:37:58] right, I think we can remove the bullseye images [17:40:07] (03PS1) 10Jforrester: Zuul: [mediawiki/services/parsoid/testreduce] Disable CI for now [integration/config] - 10https://gerrit.wikimedia.org/r/1138421 (https://phabricator.wikimedia.org/T382177) [17:41:43] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/services/parsoid/testreduce] Disable CI for now [integration/config] - 10https://gerrit.wikimedia.org/r/1138421 (https://phabricator.wikimedia.org/T382177) (owner: 10Jforrester) [17:42:23] 10Continuous-Integration-Config, 13Patch-For-Review: Upgrade all CI jobs for WMF-deployed projects from Node 18 to Node 20 - https://phabricator.wikimedia.org/T343827#10762269 (10Jdforrester-WMF) [17:43:14] (03Merged) 10jenkins-bot: Zuul: [mediawiki/services/parsoid/testreduce] Disable CI for now [integration/config] - 10https://gerrit.wikimedia.org/r/1138421 (https://phabricator.wikimedia.org/T382177) (owner: 10Jforrester) [17:43:27] !log Zuul: [mediawiki/services/parsoid/testreduce] Disable CI for now, for T382177 [17:43:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:43:30] T382177: mediawiki/services/parsoid/testreduce fails CI on Node 20 - https://phabricator.wikimedia.org/T382177 [17:54:59] 10Gerrit, 07Upstream: Background image of login page is served with a short TTL - https://phabricator.wikimedia.org/T392479#10762366 (10hashar) I have sent a couple patches upstream to update the doc and remove the `e` parameter which has no usage in the existing code. * https://gerrit-review.googlesource.com/... [18:38:46] 10Release-Engineering-Team (Priority Backlog πŸ“₯): Refactor `build-images.py` to use a common code image and `docker buildx` - https://phabricator.wikimedia.org/T392526 (10dduvall) 03NEW [18:44:11] 10Release-Engineering-Team (Priority Backlog πŸ“₯): Refactor `build-images.py` to use a common code image and `docker buildx` - https://phabricator.wikimedia.org/T392526#10762482 (10dduvall) [18:55:28] (03open) 10dduvall: build-images.py: Refactor using `docker buildx bake` [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/167 (https://phabricator.wikimedia.org/T392526) [18:56:02] (03CR) 10Hashar: "In MariaDB I went to update the `log_url` of all builds that happened before this change:" [integration/config] (zuul3) - 10https://gerrit.wikimedia.org/r/1137545 (owner: 10Hashar) [18:59:46] (03update) 10dduvall: build-images.py: Refactor using `docker buildx bake` [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/167 (https://phabricator.wikimedia.org/T392526) [18:59:50] (03update) 10dduvall: build-images.py: Refactor using `docker buildx bake` [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/167 (https://phabricator.wikimedia.org/T392526) [18:59:57] (03update) 10dduvall: build-images.py: Refactor using `docker buildx bake` [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/167 (https://phabricator.wikimedia.org/T392526) [19:35:12] (03open) 10dancy: local-dev/cas_server.py: Fix delete expired tickets bug [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/762 [19:35:15] (03update) 10dancy: local-dev/cas_server.py: Fix delete expired tickets bug [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/762 [19:35:22] (03open) 10dancy: spiderpig: Fix auth loop bug [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/763 [19:35:25] (03update) 10dancy: spiderpig: Fix auth loop bug [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/763 [19:37:59] (03update) 10dancy: local-dev/cas_server.py: Fix delete expired tickets bug [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/762 [19:39:20] (03update) 10dancy: spiderpig: Fix auth loop bug [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/763 [19:40:27] (03merge) 10dancy: local-dev/cas_server.py: Fix delete expired tickets bug [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/762 [19:48:17] (03open) 10dancy: cli.py: Allow None return from Application main function [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/764 [19:48:19] (03update) 10dancy: cli.py: Allow None return from Application main function [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/764 [19:49:03] (03update) 10dancy: cli.py: Allow None return from Application main function [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/764 [19:49:09] (03update) 10dancy: cli.py: Allow None return from Application main function [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/764 [19:50:02] (03update) 10dancy: cli.py: Allow None return from Application main function [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/764 [19:51:03] (03update) 10dancy: cli.py: Allow None return from Application main function [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/764 [20:02:44] (03approved) 10bd808: spiderpig: Fix auth loop bug [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/763 (owner: 10dancy) [20:03:41] (03update) 10dancy: spiderpig: Fix auth loop bug [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/763 [20:13:37] 10Beta-Cluster-Infrastructure: Move `tmpdir` on deployment-db instances - https://phabricator.wikimedia.org/T117446#10762804 (10bd808) 05Openβ†’03Declined Everything in this layer has changed a zillion times in the last 10 years. These days the database servers use [[https://wikitech.wikimedia.org/wiki/Hel... [20:13:42] (03merge) 10dancy: spiderpig: Fix auth loop bug [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/763 [20:28:45] scrapers have the load average on mw14 back up to 7 again :old-man-yells-at-ai: [20:29:38] bd808: Ban-time again? [20:29:52] Likely same scraper, or another one? [20:29:58] yeah, I'm getting the windows open to find the jerks [20:30:02] <3 [20:32:06] 10Beta-Cluster-Infrastructure: HIgh load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392534 (10bd808) 03NEW [20:32:27] 10Beta-Cluster-Infrastructure: HIgh load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392534#10762845 (10bd808) 05Openβ†’03In progress p:05Triageβ†’03High a:03bd808 [20:43:18] 10Beta-Cluster-Infrastructure: HIgh load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392534#10762944 (10bd808) I am again using variations of `tail -50000 /var/log/apache2/other_vhosts_access-json.log | grep -oP '"X-Client-IP": "\d+\.\d+\.\d+\.' |sort|uniq -c|sort -nr|head -... [20:44:53] 10Beta-Cluster-Infrastructure: HIgh load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392534#10762956 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/d8b94389e713db63d2964c42e5c1591e15a303f8 ` diff --git a/deployment-prep/_.yaml b/deployment-p... [20:46:34] (03update) 10dancy: cli.py: Allow None return from Application main function [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/764 [20:46:58] !log Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks (T392534) [20:47:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:47:01] T392534: HIgh load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392534 [20:47:16] 10Beta-Cluster-Infrastructure: High load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392534#10763007 (10Reedy) [20:47:47] (03merge) 10dancy: cli.py: Allow None return from Application main function [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/764 [20:48:43] (03open) 10dancy: Release 4.155.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/765 [20:49:51] (03merge) 10dancy: Release 4.155.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/765 [20:52:22] brazil seems to be our big botnet winner today [21:04:04] (03open) 10dancy: pyproject.toml: Modernize license specification [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/766 [21:04:06] (03update) 10dancy: pyproject.toml: Modernize license specification [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/766 [21:04:30] (03update) 10dancy: pyproject.toml: Modernize license specification [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/766 [21:04:34] (03update) 10dancy: pyproject.toml: Modernize license specification [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/766 [21:05:07] 10Beta-Cluster-Infrastructure: High load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392534#10763046 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/ef2212ca1aeee50d3277b4813d60acb6742aec73 ` diff --git a/deployment-prep/_.yaml b/deployment-p... [21:05:52] !log Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks (T392534) [21:16:06] (03update) 10dancy: pyproject.toml: Modernize license specification [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/766 [21:19:51] 10Beta-Cluster-Infrastructure: High load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392534#10763059 (10bd808) I made some quick helper scripts in deployment-mediawiki14/root for this stuff: `name=find-top-class-b.sh,lang=bash #!/usr/bin/env bash exec tail -50000 /var/log/ap... [21:21:07] TFW you are considering blocking the entire country of Brazil for a couple of hours... [21:25:03] I can't imagine there's much legit traffic... [21:28:23] 10Beta-Cluster-Infrastructure: High load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392534#10763089 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/80718740d397e7a27c90e9749c65527c9c75ec4e ` diff --git a/deployment-prep/_.yaml b/deployment-p... [21:29:01] !log Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks (T392534) [21:29:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:29:03] T392534: High load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392534 [21:45:31] 10Beta-Cluster-Infrastructure: High load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392534#10763134 (10bd808) Getting more aggressive. This list was made by comparing `./find-top-class-b.sh` output with ranges listed at https://ipnetinfo.com/country/BR. https://gerrit.wiki... [21:46:58] 10Beta-Cluster-Infrastructure: High load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392534#10763135 (10bd808) [21:47:10] !log Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks (T392534) [21:47:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:47:12] T392534: High load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392534 [21:47:23] 99% of the bots probably won't respect it, but should we make sure all the beta wikis have a suitable robots.txt set? [21:48:51] bots that read robots.txt are almost certainly not the problem, but sure it wouldn't hurt [21:56:18] (03PS2) 10Jforrester: jjb: Drop legacy-node18 and legacy-node18-browser, unused [integration/config] - 10https://gerrit.wikimedia.org/r/1138384 [21:56:19] (03CR) 10Jforrester: [C:03+2] jjb: Drop legacy-node18 and legacy-node18-browser, unused [integration/config] - 10https://gerrit.wikimedia.org/r/1138384 (owner: 10Jforrester) [21:58:16] (03Merged) 10jenkins-bot: jjb: Drop legacy-node18 and legacy-node18-browser, unused [integration/config] - 10https://gerrit.wikimedia.org/r/1138384 (owner: 10Jforrester) [22:07:18] (03PS2) 10Jforrester: Zuul: [mediawiki/services/parsoid/testreduce] Switch Node 20 CI on [integration/config] - 10https://gerrit.wikimedia.org/r/1138383 [22:07:23] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/services/parsoid/testreduce] Switch Node 20 CI on [integration/config] - 10https://gerrit.wikimedia.org/r/1138383 (owner: 10Jforrester) [22:10:35] (03Merged) 10jenkins-bot: Zuul: [mediawiki/services/parsoid/testreduce] Switch Node 20 CI on [integration/config] - 10https://gerrit.wikimedia.org/r/1138383 (owner: 10Jforrester) [22:11:14] !log Zuul: [mediawiki/services/parsoid/testreduce] Switch Node 20 CI on, for T382177 [22:11:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:11:16] T382177: mediawiki/services/parsoid/testreduce fails CI on Node 20 - https://phabricator.wikimedia.org/T382177 [22:11:56] 10Continuous-Integration-Config, 13Patch-For-Review: Upgrade all CI jobs for WMF-deployed projects from Node 18 to Node 20 - https://phabricator.wikimedia.org/T343827#10763196 (10Jdforrester-WMF) [22:14:04] 10Beta-Cluster-Infrastructure: High load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392534#10763197 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/daee6f5f7d94c4952db8bf7e5753aca584c90102 ` diff --git a/deployment-prep/_.yaml b/deployment-p... [22:15:07] !log Forced puppet run and restarted varnish on deployment-cache-text08 to pick up a huge pile of new blocks (T392534) [22:15:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:15:17] T392534: High load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392534 [22:41:02] 10Beta-Cluster-Infrastructure: High load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392534#10763235 (10bd808) Even blocking by class B is not making much of a dent. I am going to try an even bigger stick and block all class A CIDR blocks (`/8`) that had more than 500 reques... [22:43:26] 10Beta-Cluster-Infrastructure: High load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392534#10763236 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/5666f32ebf5e2c6f4a62999ef6911d71081c1b48 `lines=10 diff --git a/deployment-prep/_.yaml b/depl... [22:43:38] !log Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks (T392534) [22:43:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:43:40] T392534: High load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392534 [22:50:30] well there you go. banning 16 Class A ranges (X.0.0.0/8) "fixed" things for now [22:55:29] (03open) 10brennen: Draft: SpiderPig log view [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/767 (https://phabricator.wikimedia.org/T391005) [22:59:51] !log Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks (T392534) [22:59:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:59:53] T392534: High load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392534 [22:59:53] 10Beta-Cluster-Infrastructure: High load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392534#10763260 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/f36af87c3ae9bcb2f7c0a78ecf6bdd21586687e9 ` diff --git a/deployment-prep/_.yaml b/deployment-p... [23:01:12] p858snake|cloud: the robots.txt for beta meta and simple both start with `Disallow: /` and I'm pretty sure that means they all do. [23:03:39] 10Beta-Cluster-Infrastructure: High load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392534#10763279 (10bd808) https://grafana.wmcloud.org/d/0g9N-7pVz/cloud-vps-project-board?orgId=1&var-project=deployment-prep&var-instance=All&viewPanel=902&from=now-24h&to=now {F59365910,si... [23:05:07] (03approved) 10bd808: pyproject.toml: Modernize license specification [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/766 (owner: 10dancy) [23:05:39] (03merge) 10bd808: pyproject.toml: Modernize license specification [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/766 (owner: 10dancy) [23:06:37] 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10Scap (SpiderPig πŸ•ΈοΈ), 10logspam-watch, 10observability, and 2 others: Add a log view to SpiderPig - https://phabricator.wikimedia.org/T391005#10763281 (10brennen) 05Openβ†’03In progress a:03brennen [23:23:13] 10Beta-Cluster-Infrastructure: High load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392534#10763296 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/d9f0e372ce622f8e53a84b2dae9c99069ef8b54e `lines=10 diff --git a/deployment-prep/_.yaml b/depl... [23:41:00] 10Beta-Cluster-Infrastructure: High load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392534#10763319 (10bd808) Announced again at https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/message/B7RKAXXIFVDWOA4VRTFPR3AFS2JNAHSE/ as I fear the Class A blocks... [23:44:28] 10Beta-Cluster-Infrastructure: High load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392534#10763323 (10bd808) 05In progressβ†’03Resolved I think using new tasks each time will be better than keeping one open one in the longer term. Maybe after #wikimedia-hackathon-202...