[01:47:24] (03PS3) 10NMW03: [Zuul] Update email of Nemoralis [integration/config] - 10https://gerrit.wikimedia.org/r/1175976 [03:00:35] Project mwcore-phpunit-coverage-master build #4499: 04FAILURE in 34 sec: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/4499/ [06:00:14] (03PS3) 10Jforrester: Zuul: Archive operations/debs/trafficserver, again [integration/config] - 10https://gerrit.wikimedia.org/r/1175944 (owner: 10BCornwall) [06:00:18] (03CR) 10Jforrester: [C:03+2] Zuul: Archive operations/debs/trafficserver, again [integration/config] - 10https://gerrit.wikimedia.org/r/1175944 (owner: 10BCornwall) [06:01:39] (03PS3) 10Jforrester: Zuul: Archive operations/software/prometheus-rdkafka-exporter [integration/config] - 10https://gerrit.wikimedia.org/r/1175946 (owner: 10BCornwall) [06:01:39] (03Merged) 10jenkins-bot: Zuul: Archive operations/debs/trafficserver, again [integration/config] - 10https://gerrit.wikimedia.org/r/1175944 (owner: 10BCornwall) [06:02:37] !log Zuul: Archive operations/debs/trafficserver, again [06:02:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [06:02:55] (03CR) 10Jforrester: [C:03+2] Zuul: Archive operations/software/prometheus-rdkafka-exporter [integration/config] - 10https://gerrit.wikimedia.org/r/1175946 (owner: 10BCornwall) [06:04:29] (03Merged) 10jenkins-bot: Zuul: Archive operations/software/prometheus-rdkafka-exporter [integration/config] - 10https://gerrit.wikimedia.org/r/1175946 (owner: 10BCornwall) [06:08:23] !log Zuul: Archive operations/software/prometheus-rdkafka-exporter [06:08:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:19:33] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: upgrade gitlab hosts to bookworm - https://phabricator.wikimedia.org/T399306#11063415 (10Jelto) 05Open→03Resolved All gitlab hosts are on bookworm now, I'll close the task. [07:25:36] 06Project-Admins, 06Data-Engineering: Consider archiving `Q1 FY2025/26 July 1st - September 30th` - https://phabricator.wikimedia.org/T401018#11063442 (10Aklapper) [07:26:23] 06Project-Admins, 06Data-Engineering: Consider archiving `Q1 FY2025/26 July 1st - September 30th` - https://phabricator.wikimedia.org/T401018#11063445 (10Aklapper) [07:52:06] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: upgrade gitlab hosts to bookworm - https://phabricator.wikimedia.org/T399306#11063609 (10Jelto) [07:56:49] o/ [08:29:59] bd808: thanks! I don’t mind ^^ [08:39:42] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.45.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T396374#11063847 (10hashar) [10:42:18] (03open) 10jnuche: jenkins-rel: add dpkg options to jenkins package installation [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/103 (https://phabricator.wikimedia.org/T400645) [10:47:47] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: dpkg error when deploying Jenkins to releases2003.codfw.wmnet - https://phabricator.wikimedia.org/T400645#11064432 (10jnuche) @Dzahn you're right the manual installation shoul... [10:49:49] (03PS1) 10Cyndywikime: Remove GrowthExperiments dependency on TimedMediaHandler in CI [integration/config] - 10https://gerrit.wikimedia.org/r/1176200 (https://phabricator.wikimedia.org/T400701) [10:52:56] (03CR) 10Cyndywikime: "This patch is ready for review" [integration/config] - 10https://gerrit.wikimedia.org/r/1176200 (https://phabricator.wikimedia.org/T400701) (owner: 10Cyndywikime) [11:15:20] 10Phabricator: Rename Phabricator account username ‘R4356th’ to ‘Redmin’ - https://phabricator.wikimedia.org/T401306 (10R4356th) 03NEW [11:17:58] (03CR) 10Urbanecm: [C:03+1] "LGTM, we no longer depend on TimedMediaHandler (removed in Ic08cca893a1d4595fc1b587d210c62930b94e206, fully removed in I2c8c0f2b2e6f8bd12e" [integration/config] - 10https://gerrit.wikimedia.org/r/1176200 (https://phabricator.wikimedia.org/T400701) (owner: 10Cyndywikime) [11:51:21] 10Release-Engineering-Team (Radar), 10Similarusers, 06Trust and Safety Product Team: Configure SimilarEditors in production with Similarusers credentials - https://phabricator.wikimedia.org/T308670#11064629 (10Dreamy_Jazz) 05Open→03Declined See T309125#11064561 for the reasoning. [12:48:21] 10GitLab (Pipeline Services Migration🐤), 10Similarusers, 06Trust and Safety Product Team: Migrate mediawiki/services/similar-users to GitLab - https://phabricator.wikimedia.org/T344742#11064983 (10Aklapper) 05Open→03Declined Declining per {T368269} [12:51:15] 10Phabricator, 10Release-Engineering-Team (Doing 😎): Rename Phabricator account username ‘R4356th’ to ‘Redmin’ - https://phabricator.wikimedia.org/T401306#11065004 (10Aklapper) 05Open→03Resolved a:03Aklapper Done. [12:55:03] maintenance-disconnect-full-disks build 725749 integration-agent-docker-1045 (/: 26%, /srv: 98%, /var/lib/docker: 42%): OFFLINE due to disk space [13:00:04] maintenance-disconnect-full-disks build 725750 integration-agent-docker-1045 (/: 26%, /srv: 42%, /var/lib/docker: 41%): RECOVERY disk space OK [13:29:27] (03CR) 10Hashar: [C:03+2] Remove GrowthExperiments dependency on TimedMediaHandler in CI [integration/config] - 10https://gerrit.wikimedia.org/r/1176200 (https://phabricator.wikimedia.org/T400701) (owner: 10Cyndywikime) [13:31:19] (03Merged) 10jenkins-bot: Remove GrowthExperiments dependency on TimedMediaHandler in CI [integration/config] - 10https://gerrit.wikimedia.org/r/1176200 (https://phabricator.wikimedia.org/T400701) (owner: 10Cyndywikime) [13:36:31] (03CR) 10Hashar: [C:03+2] "Deployed :)" [integration/config] - 10https://gerrit.wikimedia.org/r/1176200 (https://phabricator.wikimedia.org/T400701) (owner: 10Cyndywikime) [13:38:00] (03CR) 10Hashar: [C:03+2] [Zuul] Update email of Nemoralis [integration/config] - 10https://gerrit.wikimedia.org/r/1175976 (owner: 10NMW03) [13:39:45] (03Merged) 10jenkins-bot: [Zuul] Update email of Nemoralis [integration/config] - 10https://gerrit.wikimedia.org/r/1175976 (owner: 10NMW03) [13:59:44] (03PS3) 10Hashar: build: upgrade QUnit [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1175475 [14:01:08] (03CR) 10Hashar: "I have made some code adjustment after `QUnit.test.each` learned to output nice labels when being fed an array ( https://github.com/qunitj" [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1175475 (owner: 10Hashar) [14:07:17] (03update) 10dancy: Bump dependencies based on Dependabot recommendations [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/939 [14:19:02] (03open) 10dancy: requirements.txt: Bump Jinja2 to 2.11.3 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/941 [14:19:07] (03update) 10dancy: requirements.txt: Bump Jinja2 to 2.11.3 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/941 [14:21:41] (03merge) 10dancy: requirements.txt: Bump Jinja2 to 2.11.3 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/941 [14:21:48] (03close) 10dancy: Bump dependencies based on Dependabot recommendations [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/939 [14:34:22] (03open) 10dancy: requirements.txt: Bump PyYAML to 6.0.2 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/942 [14:34:26] (03update) 10dancy: requirements.txt: Bump PyYAML to 6.0.2 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/942 [14:36:15] (03merge) 10dancy: requirements.txt: Bump PyYAML to 6.0.2 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/942 [14:37:25] (03open) 10dancy: requirements.txt: Bump certifi to 2022.12.07 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/943 [14:37:29] (03update) 10dancy: requirements.txt: Bump certifi to 2022.12.07 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/943 [14:39:09] (03merge) 10dancy: requirements.txt: Bump certifi to 2022.12.07 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/943 [14:40:21] (03open) 10dancy: requirements.txt: Bump urllib3 to 2.0.6 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/944 [14:40:25] (03update) 10dancy: requirements.txt: Bump urllib3 to 2.0.6 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/944 [14:42:10] (03merge) 10dancy: requirements.txt: Bump urllib3 to 2.0.6 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/944 [14:49:04] (03open) 10dancy: requirements.txt: Bump idna to 3.7 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/945 [14:49:09] (03update) 10dancy: requirements.txt: Bump idna to 3.7 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/945 [14:50:40] (03merge) 10dancy: requirements.txt: Bump idna to 3.7 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/945 [14:51:00] (03open) 10dancy: requirements.txt: Bump requests to 2.32.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/946 [14:51:04] (03update) 10dancy: requirements.txt: Bump requests to 2.32.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/946 [14:52:36] (03merge) 10dancy: requirements.txt: Bump requests to 2.32.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/946 [14:54:27] (03open) 10dancy: requirements.txt: Bump setuptools to 2.32.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/947 [14:54:30] (03update) 10dancy: requirements.txt: Bump setuptools to 2.32.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/947 [14:55:59] (03merge) 10dancy: requirements.txt: Bump setuptools to 2.32.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/947 [14:56:59] (03open) 10dancy: requirements.txt: Remove Python 3.8 conditionalization [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/948 [14:57:03] (03update) 10dancy: requirements.txt: Remove Python 3.8 conditionalization [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/948 [14:59:01] (03merge) 10dancy: requirements.txt: Remove Python 3.8 conditionalization [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/948 [14:59:18] (03open) 10dancy: verify-requirements.txt: Bump virtualenv to 20.26.6 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/949 [14:59:22] (03update) 10dancy: verify-requirements.txt: Bump virtualenv to 20.26.6 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/949 [15:13:10] (03merge) 10dancy: verify-requirements.txt: Bump virtualenv to 20.26.6 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/949 [15:17:23] (03open) 10dancy: web/package-lock.json: Update all packages [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/950 [15:17:27] (03update) 10dancy: web/package-lock.json: Update all packages [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/950 [15:22:51] (03merge) 10dancy: web/package-lock.json: Update all packages [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/950 [15:37:25] (03open) 10dancy: Release 4.197.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/951 [15:38:50] (03merge) 10dancy: Release 4.197.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/951 [15:49:24] Yippee, build fixed! [15:49:24] Project mwcore-phpunit-coverage-master build #4500: 09FIXED in 49 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/4500/ [15:57:20] (03open) 10dancy: requirements.txt: Bump certifi to 2024.7.4 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/952 [15:57:24] (03update) 10dancy: requirements.txt: Bump certifi to 2024.7.4 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/952 [15:58:57] (03merge) 10dancy: requirements.txt: Bump certifi to 2024.7.4 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/952 [15:59:07] (03open) 10dancy: requirements.txt: Bump setuptools to 78.1.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/953 [15:59:11] (03update) 10dancy: requirements.txt: Bump setuptools to 78.1.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/953 [16:01:17] (03merge) 10dancy: requirements.txt: Bump setuptools to 78.1.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/953 [16:06:04] (03PS1) 10Pppery: Zuul: Add Aasim to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1176269 [16:12:12] 06Project-Admins, 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Consider archiving `Q1 FY2025/26 July 1st - September 30th` - https://phabricator.wikimedia.org/T401018#11065906 (10Ottomata) a:03Ahoelzl [16:13:28] 06Project-Admins, 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Consider archiving `Q1 FY2025/26 July 1st - September 30th` - https://phabricator.wikimedia.org/T401018#11065920 (10Ottomata) a:05Ahoelzl→03Milimetric [16:13:37] 06Project-Admins, 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Consider archiving `Q1 FY2025/26 July 1st - September 30th` - https://phabricator.wikimedia.org/T401018#11065921 (10Ottomata) p:05Triage→03Low [16:26:50] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: dpkg error when deploying Jenkins to releases2003.codfw.wmnet - https://phabricator.wikimedia.org/T400645#11065973 (10Dzahn) @jnuche Thank you:) I checked the dpkg options an... [16:29:29] (03open) 10dancy: requirements.txt: Bump starlette to 0.47.2 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/954 [16:29:32] (03update) 10dancy: requirements.txt: Bump starlette to 0.47.2 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/954 [16:33:24] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: dpkg error when deploying Jenkins to releases2003.codfw.wmnet - https://phabricator.wikimedia.org/T400645#11065994 (10Dzahn) 05Resolved→03Open The dpkg install looks fine, but... [16:36:22] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: dpkg error when deploying Jenkins to releases2003.codfw.wmnet - https://phabricator.wikimedia.org/T400645#11065996 (10Dzahn) when running the same command manually shortly after t... [16:38:52] (03PS1) 10Zoranzoki21: Zuul: [mediawiki/extensions/ForceSkin] Enable CI [integration/config] - 10https://gerrit.wikimedia.org/r/1176271 [16:39:45] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: dpkg error when deploying Jenkins to releases2003.codfw.wmnet - https://phabricator.wikimedia.org/T400645#11066008 (10Dzahn) My guess it is about the quoting and having to escape... [16:45:31] FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [16:45:31] (03open) 10dancy: requirements.txt: Bump MarkupSafe to 3.0.2 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/955 [16:45:35] (03update) 10dancy: requirements.txt: Bump MarkupSafe to 3.0.2 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/955 [16:45:39] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T401340 (10phaultfinder) 03NEW [16:49:20] Project beta-code-update-eqiad build #559976: 04FAILURE in 3 min 50 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/559976/ [16:49:40] (03update) 10dancy: requirements.txt: Bump Jinja2 to 3.1.6 and MarkupSafe to 3.0.2 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/955 [16:49:45] (03update) 10dancy: requirements.txt: Bump Jinja2 to 3.1.6 and MarkupSafe to 3.0.2 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/955 [16:51:33] (03merge) 10dancy: requirements.txt: Bump Jinja2 to 3.1.6 and MarkupSafe to 3.0.2 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/955 [16:52:09] 17:49:19 fatal: unable to access 'https://gerrit.wikimedia.org/r/mediawiki/core/': Failed to connect to gerrit.wikimedia.org port 443: Connection timed out [16:52:19] (03update) 10dancy: requirements.txt: Bump starlette to 0.47.2 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/954 [16:52:24] (03open) 10dancy: requirements.txt: Bump fastapi to 0.116.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/956 [16:52:28] (03update) 10dancy: requirements.txt: Bump fastapi to 0.116.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/956 [16:52:56] !log restarting gerrit [16:52:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:53:03] that'd explain it :P [16:53:22] doing that as a reaction [16:53:37] because status showed java jgit error [16:54:12] (03merge) 10dancy: requirements.txt: Bump fastapi to 0.116.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/956 [16:55:01] (03update) 10dancy: requirements.txt: Bump starlette to 0.47.2 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/954 [16:57:11] Project beta-code-update-eqiad build #559977: 04STILL FAILING in 4 min 10 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/559977/ [16:57:15] (03merge) 10dancy: requirements.txt: Bump starlette to 0.47.2 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/954 [17:00:31] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:05:12] Yippee, build fixed! [17:05:12] Project beta-code-update-eqiad build #559978: 09FIXED in 2 min 11 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/559978/ [17:09:35] (03open) 10dancy: requirements.txt: Bump urllib3 to 2.5.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/957 [17:09:39] (03update) 10dancy: requirements.txt: Bump urllib3 to 2.5.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/957 [17:11:25] (03merge) 10dancy: requirements.txt: Bump urllib3 to 2.5.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/957 [17:11:41] (03open) 10dancy: requirements.txt: Bump requests to 2.32.4 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/958 [17:11:44] (03update) 10dancy: requirements.txt: Bump requests to 2.32.4 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/958 [17:14:24] (03merge) 10dancy: requirements.txt: Bump requests to 2.32.4 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/958 [17:22:16] (03open) 10dancy: tox.ini: Bump black to 24.3.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/959 [17:22:20] (03update) 10dancy: tox.ini: Bump black to 24.3.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/959 [17:22:50] mutante: that was not Gerrit. That was Apache being saturated due to, I believe, a massive crawling [17:22:54] (03update) 10dancy: tox.ini: Bump black to 24.3.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/959 [17:23:03] (03update) 10dancy: tox.ini: Bump black to 24.3.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/959 [17:23:35] see Apache workers in reply/closing raising at 16:45 https://grafana.wikimedia.org/d/L0-l1o0Mz/apache?orgId=1&from=now-3h&to=now&timezone=utc&var-host=gerrit1003&var-port=9117&refresh=1m&viewPanel=panel-2 [17:24:18] and something massively spammed it at https://grafana.wikimedia.org/d/nX8li17Sk/overview-sre-collab?orgId=1&from=now-3h&to=now&timezone=utc&var-node=gerrit1003&var-instance=gerrit.wikimedia.org:443&viewPanel=panel-44 [17:24:37] (03merge) 10dancy: tox.ini: Bump black to 24.3.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/959 [17:24:56] hashar: yea, we already now the bad guy [17:24:56] I bet that is all traffic made to gitiles [17:25:13] a student from a German university [17:25:20] so potentially you can just restart Apache which should clear the workers [17:25:23] and indeed I restarted apache for the first fix [17:25:28] blocking IP right now [17:25:32] cool [17:25:33] already done :) [17:25:36] thanks -:] [17:25:38] :) [17:25:52] I think the Gerrit daemon would recover by itself [17:27:44] I only did that because the first thing I saw was a java exception in systemctl status gerrit [17:27:48] but yes [17:38:31] FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:41:35] :/ [17:43:07] Project beta-code-update-eqiad build #559982: 04FAILURE in 6.9 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/559982/ [17:53:50] Project beta-code-update-eqiad build #559983: 04STILL FAILING in 50 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/559983/ [17:55:47] What we've got here is failure to commit [17:58:31] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:05:34] Yippee, build fixed! [18:05:34] Project beta-code-update-eqiad build #559984: 09FIXED in 2 min 33 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/559984/ [18:18:31] FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:33:31] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:40:48] Daimona: lol [18:42:33] ;) [18:45:34] (03open) 10dancy: mediawiki-base/Dockerfile: Fix PHP_VERSION in single-version-base [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/201 [18:45:37] (03update) 10dancy: mediawiki-base/Dockerfile: Fix PHP_VERSION in single-version-base [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/201 [18:54:41] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - gerrit1003 - https://phabricator.wikimedia.org/T401340#11066429 (10Dzahn) [18:56:23] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - gerrit1003 - https://phabricator.wikimedia.org/T401340#11066434 (10Dzahn) https://gerrit.wikimedia.org/r/c/operations/puppet/+/1176288 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1176297 https://gerrit.wikimedia.org/r/c/operations/pup... [19:01:09] (03approved) 10dancy: make-container-image: add mediawiki publish-83 flavour [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/194 (https://phabricator.wikimedia.org/T399884) (owner: 10swfrench) [19:03:11] (03update) 10swfrench: make-container-image: add mediawiki publish-83 flavour [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/194 (https://phabricator.wikimedia.org/T399884) [19:03:26] (03merge) 10swfrench: make-container-image: add mediawiki publish-83 flavour [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/194 (https://phabricator.wikimedia.org/T399884) [19:03:32] (03update) 10dancy: mediawiki-base/Dockerfile: Fix PHP_VERSION in single-version-base [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/201 [19:03:37] (03update) 10dancy: mediawiki-base/Dockerfile: Fix PHP_VERSION in single-version-base [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/201 [19:04:53] (03update) 10dancy: mediawiki-base/Dockerfile: Fix PHP_VERSION in single-version-base [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/201 [19:05:52] (03merge) 10dancy: mediawiki-base/Dockerfile: Fix PHP_VERSION in single-version-base [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/201 [20:13:55] hi, i want to try out an unmerged patch on the beta cluster (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/1176311). i think i know what to do, but if anything breaks, it's probably my fault [20:18:12] MatmaRex: you might need to disable https://integration.wikimedia.org/ci/view/Beta/job/beta-code-update-eqiad/ while you mess about. [20:19:21] marking https://integration.wikimedia.org/ci/computer/deployment%2Ddeploy04/ as offline is probably the easiest on/off switch for beta automation [20:21:08] when i did this in the past, i just "raced" with the automated job, i should need less than 10 minutes ;) [20:23:00] i'm currently trying to figure out what user i need to be in order to do a git cherry-pick in deployment-deploy04:/srv/mediawiki-staging/php-master/extensions/CentralAuth… i don't have permissions as myself, and i worry that if i just `sudo` it, the automated job won't have permissions to undo it afterwards [20:23:29] there should be a bot that owns things I think... [20:25:08] jenkins-deploy is the bot. In theory there is are the right permissions in that directory for anyone in the wikidev group to make updates. If it gets messed up I'm sure we can figure out how to fix it too. [20:27:31] i think i did it right [20:27:58] running scap now [20:27:59] (03open) 10dancy: spiderpig-apiserver: Invoke uvicorn directly [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/960 [20:28:03] (03update) 10dancy: spiderpig-apiserver: Invoke uvicorn directly [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/960 [20:29:39] (03merge) 10dancy: spiderpig-apiserver: Invoke uvicorn directly [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/960 [20:30:03] alright, i'm done, i assume the automated job will undo my cherry-pick the next time it runs [20:30:35] (03open) 10dancy: Release 4.197.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/961 [20:30:48] (btw i was a bit surprised that i could just do `sudo -u jenkins-deploy git cherry-pick ...`, it just seemed too easy) [20:31:38] and thanks :) [20:31:47] glad you got what you needed [20:31:50] (03merge) 10dancy: Release 4.197.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/961 [20:32:17] and yeah, messing with MediaWiki is surprisingly simple when scap is doing most of the ugly bits :) [21:33:55] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: allow new zuul executor VMs in prod to talk to cloud VPS - https://phabricator.wikimedia.org/T394838#11066756 (10bd808) 05Open→03Resolved a:03bd808 I think updating the squid config has resolved this one. When we config... [22:09:44] (03open) 10dancy: docs: Drop sphinxcontrib.blockdiag, use sphinx.ext.graphviz [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/962 [22:09:47] (03update) 10dancy: docs: Drop sphinxcontrib.blockdiag, use sphinx.ext.graphviz [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/962 [22:14:40] (03update) 10dancy: docs: Drop sphinxcontrib.blockdiag, use sphinx.ext.graphviz [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/962 [22:14:45] (03update) 10dancy: docs: Drop sphinxcontrib.blockdiag, use sphinx.ext.graphviz [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/962 [22:17:29] (03open) 10dancy: Dockerfile.base: Remove python3-sphinx* packages [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/152 [22:17:32] (03update) 10dancy: Dockerfile.base: Remove python3-sphinx* packages [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/152 [22:18:35] (03merge) 10dancy: Dockerfile.base: Remove python3-sphinx* packages [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/152 [22:24:06] (03approved) 10thcipriani: docs: Drop sphinxcontrib.blockdiag, use sphinx.ext.graphviz [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/962 (owner: 10dancy) [23:37:03] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: dpkg error when deploying Jenkins to releases2003.codfw.wmnet - https://phabricator.wikimedia.org/T400645#11067008 (10Dzahn) It's a syntax error because of the `:` in it: ` /usr... [23:48:22] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: dpkg error when deploying Jenkins to releases2003.codfw.wmnet - https://phabricator.wikimedia.org/T400645#11067031 (10Dzahn) 05Open→03Resolved