[00:46:50] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.48.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T442223#12542671 (10SecurityPatchBot) [01:44:42] 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#12542901 (10bd808) I have [[https://github.com/commons-app/apps-android-commons/issues/7304|opened a task with the Commons Android app]] to see if we ca... [04:01:37] 10Beta-Cluster-Infrastructure, 10Pywikibot, 07Pywikibot-tests: Find a way for pywikibot GitHub Actions to avoid IP range blocks of Microsoft Azure hosted runners - https://phabricator.wikimedia.org/T399485#12543037 (10Xqt) 05Open→03Stalled [07:59:06] (03CR) 10Hashar: "Well done, thank you @snwachukwu@wikimedia.org" [integration/config] - 10https://gerrit.wikimedia.org/r/1368275 (https://phabricator.wikimedia.org/T441426) (owner: 10STran) [09:09:24] GitLab needs a short maintenance break in 20 minutes [09:20:34] (03open) 10jelto: gitlab-runner: update gitlab-runner image to alpine-v19.5.3 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/665 (https://phabricator.wikimedia.org/T444442) [09:42:26] GitLab maintenance finished [09:45:11] 06Release-Engineering-Team (Seen), 10phan, 06Security-Team, 07Documentation, 07Wikimedia-extension-review-queue: Document that Phan is required for WMF deployed and MW bundled extensions - https://phabricator.wikimedia.org/T250038#12543551 (10Nikerabbit) Phan is completely missing from https://www.mediaw... [10:17:42] 10GitLab (CI & Job Runners), 06Release-Engineering-Team: runner-1036 & runner-1034.gitlab-runners.eqiad1.wikimedia.cloud running out of space - https://phabricator.wikimedia.org/T443833#12543648 (10Addshore) Still happening https://gitlab.wikimedia.org/repos/releng/cli/-/jobs/1038339 on 1036 ` FATAL: write /... [10:18:44] 10GitLab, 06Release-Engineering-Team: runner-1036 & runner-1034.gitlab-runners.eqiad1.wikimedia.cloud running out of space - https://phabricator.wikimedia.org/T443833#12543649 (10Addshore) [10:18:52] 10GitLab (CI & Job Runners), 06Release-Engineering-Team: runner-1036 & runner-1034.gitlab-runners.eqiad1.wikimedia.cloud running out of space - https://phabricator.wikimedia.org/T443833#12543653 (10Addshore) [10:19:05] now sure if anyone can help with that one? ^^ [10:27:00] 10MediaWiki-Releasing, 06MediaWiki-Core-Platform-Team, 10MediaWiki-General: Remove unused deprecated methods after 1.47 release - https://phabricator.wikimedia.org/T444454 (10daniel) 03NEW [10:29:29] I'm seeing a message I don't really understand from docker build for termbox: https://integration.wikimedia.org/ci/job/termbox-pipeline-test/445/console [10:29:41] ERROR: failed to solve: error writing data blob sha256:2f5aae89f9a78c14160bd18fb65e8dbb4b18aadf67d6e20e6bc5af6da0bfba65: failed to copy: failed to send write: trying to send message larger than max (19888840 vs. 16777216): unknown [10:29:51] started this morning - anyone here have any clues what that might be about? [10:33:58] 06Gerrit-Privilege-Requests: Request membership in extension-S5SlideShow group for seppl2017 - https://phabricator.wikimedia.org/T444455 (10Seppl2013) 03NEW [10:34:06] 06Release-Engineering-Team, 10Wikidata, 06Wikidata-Omega, 07ci-test-error: termbox CI: docker-build "trying to send message larger than max (19888840 vs. 16777216)" - https://phabricator.wikimedia.org/T444456 (10ArthurTaylor) 03NEW [10:42:49] 06Release-Engineering-Team, 10Wikidata, 06Wikidata-Omega, 07ci-test-error: termbox CI: docker-build "trying to send message larger than max (19888840 vs. 16777216)" - https://phabricator.wikimedia.org/T444456#12543796 (10ArthurTaylor) See also https://integration.wikimedia.org/ci/job/termbox-pipeline-test/... [10:49:52] 06Release-Engineering-Team, 10Wikidata, 06Wikidata-Omega, 07ci-test-error: termbox CI: docker-build "trying to send message larger than max (19888840 vs. 16777216)" - https://phabricator.wikimedia.org/T444456#12543847 (10awight) This similar upstream bug has some links: https://github.com/moby/buildkit/iss... [10:54:40] 06Release-Engineering-Team, 10Wikidata, 06Wikidata-Omega, 07ci-test-error: termbox CI: docker-build "trying to send message larger than max (19888840 vs. 16777216)" - https://phabricator.wikimedia.org/T444456#12543861 (10awight) I'm not familiar with the CI runners but https://debmonitor.wikimedia.org/host... [10:55:10] 06Release-Engineering-Team, 10Wikidata, 06Wikidata-Omega, 07ci-test-error: termbox CI: docker-build "trying to send message larger than max (19888840 vs. 16777216)" - https://phabricator.wikimedia.org/T444456#12543863 (10awight) @dduvall this may be of interest! [11:39:25] (03Abandoned) 10Matthias Mullie: Add Thanks dep for ReaderExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/1368163 (owner: 10Matthias Mullie) [14:07:16] 10MediaWiki-Releasing, 06MediaWiki-Core-Platform-Team, 10MediaWiki-General: Remove unused deprecated methods after 1.47 release - https://phabricator.wikimedia.org/T444454#12544980 (10Bugreporter) There are entire file that is deprecated too, such as https://phabricator.wikimedia.org/source/mediawiki/browse/... [14:53:13] 10MediaWiki-Releasing, 06MediaWiki-Core-Platform-Team, 10MediaWiki-General: Remove unused deprecated methods after 1.47 release - https://phabricator.wikimedia.org/T444454#12545227 (10Jdforrester-WMF) Historically we've fought really quite hard to explicitly *not* make these kinds of tasks part of the #media... [15:04:09] 10Continuous-Integration-Config, 10MediaWiki-Docker, 10MediaWiki-Vendor, 07Composer, 13Patch-For-Review: Update composer from 2.10.3 to 2.11.0 - https://phabricator.wikimedia.org/T442976#12545335 (10Jdforrester-WMF) [15:26:53] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 10a Wikimedia CLI: runner-1036 & runner-1034.gitlab-runners.eqiad1.wikimedia.cloud running out of space - https://phabricator.wikimedia.org/T443833#12545522 (10Addshore) p:05Triage→03High Going to go ahead and mark it as high prio (as it is for me)... [15:36:26] (03PS1) 10Gkyziridis: inference-services: Remove llm, qwen36 and gpt-oss-safeguard jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1368701 (https://phabricator.wikimedia.org/T444471) [16:05:15] (03Abandoned) 10STran: Add Thanks dependency to WikimediaCustomizations [integration/config] - 10https://gerrit.wikimedia.org/r/1368275 (https://phabricator.wikimedia.org/T441426) (owner: 10STran) [16:45:49] 10MediaWiki-Releasing, 06MediaWiki-Core-Platform-Team, 10MediaWiki-General: Remove unused deprecated methods after 1.47 release - https://phabricator.wikimedia.org/T444454#12545868 (10daniel) [19:40:00] (03CR) 10Hashar: "And verified on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaCustomizations/+/1368279/comments/abd3c695_62c5c789 😎" [integration/config] - 10https://gerrit.wikimedia.org/r/1368275 (https://phabricator.wikimedia.org/T441426) (owner: 10STran) [19:40:42] (03CR) 10Hashar: [C:03+2] dockerfiles: rebuild all base images [integration/config] - 10https://gerrit.wikimedia.org/r/1368395 (https://phabricator.wikimedia.org/T444391) (owner: 10Hashar) [19:41:46] (03CR) 10Hashar: [C:03+2] "I am rebuilding them now in order in the interest of time and will switch the Jenkins jobs tomorrow 😊" [integration/config] - 10https://gerrit.wikimedia.org/r/1368395 (https://phabricator.wikimedia.org/T444391) (owner: 10Hashar) [19:42:42] (03Merged) 10jenkins-bot: dockerfiles: rebuild all base images [integration/config] - 10https://gerrit.wikimedia.org/r/1368395 (https://phabricator.wikimedia.org/T444391) (owner: 10Hashar) [20:05:08] FIRING: [3x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cp-text09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [20:49:00] 10MediaWiki-Releasing, 06MediaWiki-Core-Platform-Team, 10MediaWiki-General, 13Patch-For-Review: Remove unused deprecated methods after 1.47 release - https://phabricator.wikimedia.org/T444454#12546463 (10matmarex) >>! In T444454#12545227, @Jdforrester-WMF wrote: > Historically we've fought really quite har... [21:23:06] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-webperf31 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [21:56:44] (03PS1) 10Wandji collins: Use generic-node26 for WdTmCollab lab tool [integration/config] - 10https://gerrit.wikimedia.org/r/1368765 [22:04:06] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-webperf31 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:05:08] (03PS2) 10Wandji collins: Use generic-node26 for the WdTmCollab lab tool [integration/config] - 10https://gerrit.wikimedia.org/r/1368765 (https://phabricator.wikimedia.org/T444548) [22:05:52] (03CR) 10Wandji collins: "As a patch of Bug: T444548" [integration/config] - 10https://gerrit.wikimedia.org/r/1368765 (https://phabricator.wikimedia.org/T444548) (owner: 10Wandji collins) [22:39:18] (03PS1) 10Hashar: jjb: update jobs for "rebuild all base images" [integration/config] - 10https://gerrit.wikimedia.org/r/1368780 (https://phabricator.wikimedia.org/T444391) [22:39:30] that is for tomorrow [22:51:07] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on deployment-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates