[00:25:22] (03PS2) 10Jeena Huneidi: Update wikilambda daily selenium job [integration/config] - 10https://gerrit.wikimedia.org/r/1179783 (https://phabricator.wikimedia.org/T388056) [00:25:42] (03CR) 10Jeena Huneidi: "https://integration.wikimedia.org/ci/view/All/job/wikilambda-catalyst-end-to-end-daily/15/console working now" [integration/config] - 10https://gerrit.wikimedia.org/r/1179783 (https://phabricator.wikimedia.org/T388056) (owner: 10Jeena Huneidi) [00:28:58] (03CR) 10Jeena Huneidi: "Thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/1179783 (https://phabricator.wikimedia.org/T388056) (owner: 10Jeena Huneidi) [01:12:01] (03open) 10swfrench: Draft: make-container-image: clean up unused build-report attributes [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/208 (https://phabricator.wikimedia.org/T401721) [02:17:08] Project beta-scap-sync-world build #220250: 04FAILURE in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/220250/ [02:27:22] Yippee, build fixed! [02:27:22] Project beta-scap-sync-world build #220251: 09FIXED in 2 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/220251/ [02:39:48] (03merge) 10thcipriani: Added Metrics Platform Experimentation Lab deployment window to deployments calendar [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/186 (https://phabricator.wikimedia.org/T396742) (owner: 10sfaci) [07:33:05] !log Restarted CI Jenkins to update plugins [07:33:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:33:57] agan [07:33:59] again [07:37:34] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Release-Engineering-Team (Seen): Migrate CI Jenkins artifacts to a non wikimedia.org domain - https://phabricator.wikimedia.org/T248696#11101026 (10hashar) 05Open→03Declined This will be done as part of the #zuul-upgrade , the artifacts are stored... [07:44:22] 10Continuous-Integration-Infrastructure, 07Jenkins: Spontaneous “No valid crumb was included in the request” Jetty error when loading Jenkins logs - https://phabricator.wikimedia.org/T326970#11101030 (10hashar) 05Open→03Declined I had, and still have, no idea how to reproduce or debug the issue at the... [07:44:44] 10Release-Engineering-Team (Seen), 10Quibble: Jenkins collapsible sections are deeply nested in Quibble output - https://phabricator.wikimedia.org/T303056#11101032 (10hashar) [07:51:54] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 07Jenkins, 10Quality-and-Test-Engineering-Team (Test Infrastructure): Move beta cluster automatic deployment to a dedicated infrastructure - https://phabricator.wikimedia.org/T256168#11101039 (10hashar) [08:03:35] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services: Phorge setup check caching is misbehaving, leading to many duck-sound=quack requests - https://phabricator.wikimedia.org/T401157#11101048 (10hashar) The APC cache has been raised to 256MB. I thought `phabricator:PhabricatorRepositoryGraphCa... [08:20:30] 10MediaWiki-Releasing, 10MediaWiki-Release-Tools: Convert old release branches to tags - https://phabricator.wikimedia.org/T380841#11101072 (10hashar) I forgot to clean the superprojects `mediawiki/skins` and `mediawiki/extensions`. I did it with: ` $ TAG=REL1_27 bash -c 'git tag -s "$TAG" "remotes/origin/... [08:22:08] 10Release-Engineering-Team (Radar), 10MediaWiki-Release-Tools: mediawiki/extensions and mediawiki/skins missing the REL1_XX branches - https://phabricator.wikimedia.org/T264365#11101083 (10hashar) 05Open→03Declined I have converted the legacy `RELX_XX` branches to tags as I did for the other repositori... [08:25:10] !log deployment-prep: pruning old branches for /srv/mediawiki-staging/php-master/extensions and /srv/mediawiki-staging/php-master/skins [08:25:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:31:30] (03CR) 10Hashar: [C:04-1] Update wikilambda daily selenium job (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1179783 (https://phabricator.wikimedia.org/T388056) (owner: 10Jeena Huneidi) [08:33:19] Project mediawiki-core-doxygen build #12987: 04FAILURE in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/12987/ [09:06:57] (03PS1) 10Hslater: Add new extensions [integration/config] - 10https://gerrit.wikimedia.org/r/1180508 [09:08:22] (03PS2) 10Hashar: Zuul: Add new extensions [integration/config] - 10https://gerrit.wikimedia.org/r/1180508 (owner: 10Hslater) [09:08:27] (03CR) 10Hashar: [C:03+2] Zuul: Add new extensions [integration/config] - 10https://gerrit.wikimedia.org/r/1180508 (owner: 10Hslater) [09:09:47] (03Merged) 10jenkins-bot: Zuul: Add new extensions [integration/config] - 10https://gerrit.wikimedia.org/r/1180508 (owner: 10Hslater) [09:11:09] (03CR) 10Hashar: [C:03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/1180508 (owner: 10Hslater) [09:15:50] 10Continuous-Integration-Infrastructure (Zuul upgrade): Investigate zuul migrate script used for the Zuul v3 OpenStack migration - https://phabricator.wikimedia.org/T383273#11101303 (10hashar) 05Open→03Resolved a:03hashar I have done the discovery a few months ago, it is unlikely we can use much of it... [09:16:02] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: Troubleshoot GitLab nftables throttling after switchover - https://phabricator.wikimedia.org/T400971#11101306 (10ABran-WMF) the last iteration does not seem to have a negative impact on Gerrit and has reduced `gitlab1004` DENYLIST to a... [09:29:20] (03PS1) 10Hashar: jjb: remove assert-lock experimental jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1180513 (https://phabricator.wikimedia.org/T74952) [09:33:28] FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance deployment-ircd03 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [09:38:28] FIRING: [11x] PuppetAgentNoResources: No Puppet resources found on instance deployment-cache-text08 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [09:41:48] 10Continuous-Integration-Config, 10Pywikibot, 13Patch-For-Review, 07Pywikibot-tests: Enable doctest for multiple Python versions - https://phabricator.wikimedia.org/T348905#11101424 (10Xqt) 05Open→03Resolved a:03Xqt [09:43:28] FIRING: [19x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief06 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [09:46:37] (03PS1) 10Hashar: Archive eventlogging repository [integration/config] - 10https://gerrit.wikimedia.org/r/1180519 [09:48:28] FIRING: [30x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief06 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [09:49:28] FIRING: [32x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief06 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [09:49:33] (03PS1) 10Hashar: Archive analytics/gobblin-wmf: moved to GitLab [integration/config] - 10https://gerrit.wikimedia.org/r/1180521 (https://phabricator.wikimedia.org/T392854) [09:51:17] (03CR) 10Hashar: "Done and I have deleted the jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/1180519 (owner: 10Hashar) [09:51:21] (03CR) 10Hashar: "Done and I have deleted the jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/1180521 (https://phabricator.wikimedia.org/T392854) (owner: 10Hashar) [09:51:28] (03CR) 10Hashar: [C:03+2] Archive analytics/gobblin-wmf: moved to GitLab [integration/config] - 10https://gerrit.wikimedia.org/r/1180521 (https://phabricator.wikimedia.org/T392854) (owner: 10Hashar) [09:51:30] (03CR) 10Hashar: [C:03+2] Archive eventlogging repository [integration/config] - 10https://gerrit.wikimedia.org/r/1180519 (owner: 10Hashar) [09:52:54] (03CR) 10CI reject: [V:04-1] Archive eventlogging repository [integration/config] - 10https://gerrit.wikimedia.org/r/1180519 (owner: 10Hashar) [09:52:54] (03CR) 10CI reject: [V:04-1] Archive analytics/gobblin-wmf: moved to GitLab [integration/config] - 10https://gerrit.wikimedia.org/r/1180521 (https://phabricator.wikimedia.org/T392854) (owner: 10Hashar) [09:53:28] FIRING: [41x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [09:54:29] (03CR) 10Hashar: [C:03+2] "CI complained because I already deleted the jobs for I1722988d9e3fbabf947e80da1159e5ebd67ed411" [integration/config] - 10https://gerrit.wikimedia.org/r/1180519 (owner: 10Hashar) [09:56:01] (03Merged) 10jenkins-bot: Archive eventlogging repository [integration/config] - 10https://gerrit.wikimedia.org/r/1180519 (owner: 10Hashar) [09:56:02] (03Merged) 10jenkins-bot: Archive analytics/gobblin-wmf: moved to GitLab [integration/config] - 10https://gerrit.wikimedia.org/r/1180521 (https://phabricator.wikimedia.org/T392854) (owner: 10Hashar) [09:58:28] FIRING: [53x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [09:59:58] FIRING: [57x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:03:28] FIRING: [59x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:05:13] FIRING: [59x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:08:28] FIRING: [59x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:10:28] FIRING: [59x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:13:28] FIRING: [59x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:15:43] FIRING: [59x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:18:28] FIRING: [59x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:20:58] FIRING: [59x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:23:28] FIRING: [59x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:26:13] FIRING: [59x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:28:28] FIRING: [59x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:31:28] RESOLVED: [59x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:50:15] (03update) 10jnuche: patches: enhance patch failure notifications [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/986 (https://phabricator.wikimedia.org/T402262) [11:01:30] (03merge) 10jnuche: patches: enhance patch failure notifications [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/986 (https://phabricator.wikimedia.org/T402262) [11:02:33] (03open) 10jnuche: Release 4.207.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/989 [11:05:01] (03merge) 10jnuche: Release 4.207.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/989 [11:08:29] 10GitLab: GitLab Private Repository Request for: repos/sre/XCHEESESCORE - https://phabricator.wikimedia.org/T401921#11101832 (10ABran-WMF) Empty private repository created: [[ https://gitlab.wikimedia.org/repos/sre/XCHEESESCORE | repos/sre/XCHEESESCORE ]] [11:08:46] 10GitLab, 06collaboration-services: GitLab Private Repository Request for: repos/sre/XCHEESESCORE - https://phabricator.wikimedia.org/T401921#11101837 (10ABran-WMF) p:05Triage→03Medium a:03ABran-WMF [11:31:54] Yippee, build fixed! [11:31:54] Project mediawiki-core-doxygen build #12988: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/12988/ [13:00:13] (03PS1) 10Hashar: jjb: simplify branchdeploy jobs definition [integration/config] - 10https://gerrit.wikimedia.org/r/1180552 [13:00:41] (03CR) 10Hashar: "Do we even need the node20/node22/node24 variants for the branchdeploy jobs?" [integration/config] - 10https://gerrit.wikimedia.org/r/1180552 (owner: 10Hashar) [13:08:38] (03PS1) 10Hashar: Archive labs/tools/ci [integration/config] - 10https://gerrit.wikimedia.org/r/1180554 [13:09:00] (03CR) 10Hashar: [C:03+2] Archive labs/tools/ci [integration/config] - 10https://gerrit.wikimedia.org/r/1180554 (owner: 10Hashar) [13:10:24] (03Merged) 10jenkins-bot: Archive labs/tools/ci [integration/config] - 10https://gerrit.wikimedia.org/r/1180554 (owner: 10Hashar) [13:14:07] !log gerrit: deleted labs/tools/ScrotBot which has never been used [13:14:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:14:25] !log gerrit: archived labs/tools/SuchABot [13:14:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:23:17] !log gerrit: deleted labs/tools/Wiki2Reveal , it is hosted at https://github.com/niebert/Wiki2Reveal [13:23:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:28:25] this gitlab MR got stuck during merging: https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/3 [13:28:27] (03PS1) 10Hashar: Archive labs/tools/bodh and labs/tools/bodh-backend [integration/config] - 10https://gerrit.wikimedia.org/r/1180560 [13:28:35] anything I can do to get it unstuck? [13:29:57] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services: Phab Aphlict Notification Servers config shows Connection error - https://phabricator.wikimedia.org/T402396 (10Aklapper) 03NEW [13:30:51] I have noticed also pipelines not triggering (ex for https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/203/pipelines?diff_id=126864) [13:30:56] for ~1hour now [13:36:44] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services: Phab Aphlict Notification Servers config shows Connection error - https://phabricator.wikimedia.org/T402396#11102260 (10Dzahn) I think it's ignorable because we are not using phabricator.wikimedia.org:443 as an aphlict... [13:39:59] 10GitLab, 06collaboration-services: GitLab Private Repository Request for: repos/sre/XCHEESESCORE - https://phabricator.wikimedia.org/T401921#11102269 (10ABran-WMF) 05Open→03Resolved closing this, let me know if there is any issue [13:40:09] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services: Phab Aphlict Notification Servers config shows Connection error - https://phabricator.wikimedia.org/T402396#11102271 (10Dzahn) Generally, if desktop notifications still work then aphlict is fine. [13:43:52] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services: Phab Aphlict Notification Servers config shows Connection error - https://phabricator.wikimedia.org/T402396#11102275 (10Aklapper) I do not know at all if this is new, sorry. I'm quite sure though it is **not** new, give... [13:44:06] (03CR) 10Hashar: [C:03+2] Archive labs/tools/bodh and labs/tools/bodh-backend [integration/config] - 10https://gerrit.wikimedia.org/r/1180560 (owner: 10Hashar) [13:45:41] (03Merged) 10jenkins-bot: Archive labs/tools/bodh and labs/tools/bodh-backend [integration/config] - 10https://gerrit.wikimedia.org/r/1180560 (owner: 10Hashar) [13:52:54] 10Scap, 07Essential-Work: SecurityPatchBot: Notify task when patch fails to apply during train deployment - https://phabricator.wikimedia.org/T402262#11102322 (10jnuche) 05Open→03Resolved a:03jnuche Deployed to prod. Going to close this for now, but will monitor in the coming weeks [13:58:41] 10Continuous-Integration-Config: Phase out "skin" terminology from CI config, use "extension" instead - https://phabricator.wikimedia.org/T402398 (10hashar) 03NEW [14:04:30] (03PS1) 10Hashar: Zuul, jjb: rename mwskin phan jobs to mwext [integration/config] - 10https://gerrit.wikimedia.org/r/1180572 (https://phabricator.wikimedia.org/T402398) [14:26:55] 06Project-Admins: Create project tag for Wikinews volunteer developed software - https://phabricator.wikimedia.org/T402150#11102441 (10Gryllida) Hi @Aklapper I would like to request that you add me to 'Trusted contributors', for purposes of managing the project that was created. If I understood correctly, th... [14:30:41] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 06serviceops, 06SRE, 06Traffic: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536#11102470 (10Clement_Goubert) 05In progress→03Resolved [14:46:43] 10Gerrit, 10ProveIt-Gadget: Users of ProvieIt gadget get a 403 Forbidden fetching i18n files from Gerrit/Gitiles - https://phabricator.wikimedia.org/T394916#11102654 (10Sophivorus) Done! Translations are now hosted at https://commons.wikimedia.org/wiki/Data:Gadget-ProveIt.tab and fetched from there. Can be tes... [14:46:50] 10Gerrit, 10ProveIt-Gadget: Users of ProvieIt gadget get a 403 Forbidden fetching i18n files from Gerrit/Gitiles - https://phabricator.wikimedia.org/T394916#11102655 (10Sophivorus) 05Open→03Resolved a:03Sophivorus [14:59:18] dhinus: pipelines are running again [15:04:44] jnuche: thank you! [15:39:54] (03PS1) 10Krinkle: zuul: Add Jenkins jobs for labs/tools/toolbase repo [integration/config] - 10https://gerrit.wikimedia.org/r/1180588 [15:49:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-etcd05 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [15:49:34] (03CR) 10Hashar: [C:03+2] zuul: Add Jenkins jobs for labs/tools/toolbase repo [integration/config] - 10https://gerrit.wikimedia.org/r/1180588 (owner: 10Krinkle) [15:49:36] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-etcd05 on project deployment-prep - https://phabricator.wikimedia.org/T402410 (10wmcs-alerts) 03NEW [15:51:01] (03Merged) 10jenkins-bot: zuul: Add Jenkins jobs for labs/tools/toolbase repo [integration/config] - 10https://gerrit.wikimedia.org/r/1180588 (owner: 10Krinkle) [15:51:21] Krinkle: [15:51:24] Krinkle: deployed! :) [15:51:28] well almost [15:51:41] now it is [15:56:34] hashar: thank you [15:58:10] Krinkle: and I don't know what to do with the scripts used for the MediaWiki coverage :/ Each time I look at them I get issues and they lack some test coverage :] [15:58:52] anyway I am ranting [16:08:20] hashar: which scripts? [16:09:30] (03update) 10swfrench: Draft: make-container-image: clean up unused build-report attributes [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/208 (https://phabricator.wikimedia.org/T401721) [16:09:38] (03update) 10swfrench: make-container-image: clean up unused build-report attributes [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/208 (https://phabricator.wikimedia.org/T401721) [16:10:44] (03approved) 10dancy: make-container-image: clean up unused build-report attributes [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/208 (https://phabricator.wikimedia.org/T401721) (owner: 10swfrench) [16:10:56] (03update) 10swfrench: make-container-image: clean up unused build-report attributes [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/208 (https://phabricator.wikimedia.org/T401721) [16:19:28] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance deployment-etcd02 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [16:27:09] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services: Phorge setup check caching is misbehaving, leading to many duck-sound=quack requests - https://phabricator.wikimedia.org/T401157#11103032 (10hashar) The 256MB cache ends up being filed as well: | Pattern | Count | Largest | Total | Usage |... [16:28:13] 10Phabricator (2025-08-19), 06Release-Engineering-Team, 06collaboration-services: Deploy Phabricator/Phorge 2025-08-19 - https://phabricator.wikimedia.org/T402309#11103035 (10brennen) 05Open→03Resolved [16:52:44] (03PS1) 10Hashar: dockerfiles: use os.path.join() in PHPUnit suite edit [integration/config] - 10https://gerrit.wikimedia.org/r/1180603 (https://phabricator.wikimedia.org/T402398) [16:54:05] (03CR) 10CI reject: [V:04-1] dockerfiles: use os.path.join() in PHPUnit suite edit [integration/config] - 10https://gerrit.wikimedia.org/r/1180603 (https://phabricator.wikimedia.org/T402398) (owner: 10Hashar) [16:54:50] (03update) 10dancy: deploy_promote.py: Attribute wikiversions.json updates to the initiating user [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/990 (https://phabricator.wikimedia.org/T401779) [16:54:52] (03open) 10dancy: deploy_promote.py: Attribute wikiversions.json updates to the initiating user [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/990 (https://phabricator.wikimedia.org/T401779) [16:56:30] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-etcd05 on project deployment-prep - https://phabricator.wikimedia.org/T402410#11103125 (10bd808) `COUNTEREXAMPLE Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while e... [16:59:06] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services: Phab Aphlict Notification Servers config shows Connection error - https://phabricator.wikimedia.org/T402396#11103134 (10Dzahn) @Aklapper I checked the actual live config in the filesystem and I see this: ` notific... [17:01:08] (03PS2) 10Hashar: dockerfiles: use os.path.join() in PHPUnit suite edit [integration/config] - 10https://gerrit.wikimedia.org/r/1180603 (https://phabricator.wikimedia.org/T402398) [17:02:50] (03merge) 10swfrench: make-container-image: clean up unused build-report attributes [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/208 (https://phabricator.wikimedia.org/T401721) [17:07:20] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services: Phab Aphlict Notification Servers config shows Connection error - https://phabricator.wikimedia.org/T402396#11103161 (10Dzahn) Oh, of course.. so.. the configuration in the CDN has this: ` - type: map target... [17:12:11] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-etcd05 on project deployment-prep - https://phabricator.wikimedia.org/T402410#11103186 (10bd808) 05Open→03In progress p:05Triage→03High a:03bd808 Potentially related to {T395446}. These were merged today: * https://gerrit.w... [17:14:46] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-etcd05 on project deployment-prep - https://phabricator.wikimedia.org/T402410#11103204 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/f21538d1f32640d19240a4f92b1626250b60f563%5E%21/#F0 ` diff --git a... [17:22:35] (03PS1) 10Hashar: quibble-coverage: Fix `--path-to-mw` arg for skins [integration/config] - 10https://gerrit.wikimedia.org/r/1180606 (https://phabricator.wikimedia.org/T395470) [17:23:47] (03PS3) 10Hashar: quibble-coverage: use os.path.join() in PHPUnit suite edit [integration/config] - 10https://gerrit.wikimedia.org/r/1180603 (https://phabricator.wikimedia.org/T402398) [17:23:48] (03PS1) 10Hashar: quibble-coverage: make phpunit-suite-edit skin agnostic [integration/config] - 10https://gerrit.wikimedia.org/r/1180607 (https://phabricator.wikimedia.org/T402398) [17:23:58] (03CR) 10CI reject: [V:04-1] quibble-coverage: Fix `--path-to-mw` arg for skins [integration/config] - 10https://gerrit.wikimedia.org/r/1180606 (https://phabricator.wikimedia.org/T395470) (owner: 10Hashar) [17:24:49] (03CR) 10Hashar: "I guess the same fix is needed for the `mwskin-phpunit-coverage.sh` script? I4e4848991f878ab8ad7e82011dca7e9017a2376a" [integration/config] - 10https://gerrit.wikimedia.org/r/1178071 (https://phabricator.wikimedia.org/T395470) (owner: 10Krinkle) [17:25:06] (03CR) 10CI reject: [V:04-1] quibble-coverage: use os.path.join() in PHPUnit suite edit [integration/config] - 10https://gerrit.wikimedia.org/r/1180603 (https://phabricator.wikimedia.org/T402398) (owner: 10Hashar) [17:25:17] (03CR) 10CI reject: [V:04-1] quibble-coverage: make phpunit-suite-edit skin agnostic [integration/config] - 10https://gerrit.wikimedia.org/r/1180607 (https://phabricator.wikimedia.org/T402398) (owner: 10Hashar) [17:26:06] (03PS2) 10Hashar: quibble-coverage: Fix `--path-to-mw` arg for skins [integration/config] - 10https://gerrit.wikimedia.org/r/1180606 (https://phabricator.wikimedia.org/T395470) [17:26:06] (03PS4) 10Hashar: quibble-coverage: use os.path.join() in PHPUnit suite edit [integration/config] - 10https://gerrit.wikimedia.org/r/1180603 (https://phabricator.wikimedia.org/T402398) [17:26:06] (03PS2) 10Hashar: quibble-coverage: make phpunit-suite-edit skin agnostic [integration/config] - 10https://gerrit.wikimedia.org/r/1180607 (https://phabricator.wikimedia.org/T402398) [17:26:09] I am off [17:26:45] (03update) 10dancy: deploy_promote.py: Attribute wikiversions.json updates to the initiating user [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/990 (https://phabricator.wikimedia.org/T401779) [17:27:35] (03CR) 10CI reject: [V:04-1] quibble-coverage: make phpunit-suite-edit skin agnostic [integration/config] - 10https://gerrit.wikimedia.org/r/1180607 (https://phabricator.wikimedia.org/T402398) (owner: 10Hashar) [17:28:49] (03update) 10dancy: deploy_promote.py: Attribute wikiversions.json updates to the initiating user [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/990 (https://phabricator.wikimedia.org/T401779) [17:31:34] (03update) 10dancy: deploy_promote.py: Attribute wikiversions.json updates to the initiating user [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/990 (https://phabricator.wikimedia.org/T401779) [17:32:53] (03CR) 10Krinkle: [C:04-1] quibble-coverage: Fix `--path-to-mw` arg for skins (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1180606 (https://phabricator.wikimedia.org/T395470) (owner: 10Hashar) [17:33:13] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance deployment-etcd02 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [17:33:41] 10Beta-Cluster-Infrastructure, 07Beta-Cluster-reproducible: HyperSwitch/errors/not found (404) on beta cluster: There was an issue displaying this preview - https://phabricator.wikimedia.org/T402206#11103283 (10bd808) [17:34:28] RESOLVED: [2x] PuppetAgentNoResources: No Puppet resources found on instance deployment-etcd02 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [17:36:13] 10Beta-Cluster-Infrastructure, 07Beta-Cluster-reproducible: HyperSwitch/errors/not found (404) on beta cluster: There was an issue displaying this preview - https://phabricator.wikimedia.org/T402206#11103307 (10bd808) [18:00:34] (03CR) 10Krinkle: [C:03+1] quibble-coverage: use os.path.join() in PHPUnit suite edit [integration/config] - 10https://gerrit.wikimedia.org/r/1180603 (https://phabricator.wikimedia.org/T402398) (owner: 10Hashar) [18:09:09] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 07Jenkins, 10Quality-and-Test-Engineering-Team (Test Infrastructure): Move beta cluster automatic deployment to a dedicated infrastructure - https://phabricator.wikimedia.org/T256168#11103426 (10bd808... [19:02:42] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 07Essential-Work, 13Patch-For-Review: Puppetize systemd unit for zuul's nodepool - https://phabricator.wikimedia.org/T401614#11103560 (10Dzahn) re: TLS certs. We now have ou... [19:11:20] 10Beta-Cluster-Infrastructure, 06Traffic: Beta cluster is slow. ISDN slow. - https://phabricator.wikimedia.org/T402430 (10AlexisJazz) 03NEW [19:14:26] 10Beta-Cluster-Infrastructure, 06Traffic: Beta cluster is slow. ISDN slow. - https://phabricator.wikimedia.org/T402430#11103654 (10AlexisJazz) [19:18:52] 10Beta-Cluster-Infrastructure, 06Traffic: Beta cluster is slow. ISDN slow. - https://phabricator.wikimedia.org/T402430#11103707 (10cmooney) I don't believe this is the beta cluster itself. ` root@nyc ~ # time wget "https://upload.wikimedia.beta.wmcloud.org/wikipedia/commons/7/75/Cute_grey_kitten.jpg" --2025-... [19:27:09] 10Beta-Cluster-Infrastructure, 10RESTBase, 10API Platform (API Gateway Roadmap), 07Beta-Cluster-reproducible: HyperSwitch/errors/not found (404) on beta cluster: There was an issue displaying this preview - https://phabricator.wikimedia.org/T402206#11103829 (10Krinkle) I'm guessing that in production, even... [19:27:44] 10Beta-Cluster-Infrastructure, 10RESTBase, 07Beta-Cluster-reproducible: HyperSwitch/errors/not found (404) on beta cluster: There was an issue displaying this preview - https://phabricator.wikimedia.org/T402206#11103846 (10Krinkle) //There is apparenlty no "REST Gateway" tag in Phabricator.// [19:28:33] 10Beta-Cluster-Infrastructure, 06Traffic: Beta cluster is slow. ISDN slow. - https://phabricator.wikimedia.org/T402430#11103848 (10AlexisJazz) 05Open→03Invalid Looks like it's an issue with my provider. Using a VPN (without changing country) fixes it. [19:53:05] 10Beta-Cluster-Infrastructure, 10RESTBase, 07Beta-Cluster-reproducible: HyperSwitch/errors/not found (404) on beta cluster: There was an issue displaying this preview - https://phabricator.wikimedia.org/T402206#11103991 (10Krinkle) >>! In T402206#11095526, @Jgiannelos wrote: > I think the problem might be th... [20:14:52] 10Phabricator, 10ReleaseTaggerBot: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T402453 (10Aklapper) 03NEW p:05Triage→03Low [20:14:56] 10Phabricator, 10Wikibugs: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T402454 (10Aklapper) 03NEW [20:25:27] 10Beta-Cluster-Infrastructure, 10RESTBase, 07Beta-Cluster-reproducible: HyperSwitch/errors/not found (404) on beta cluster: There was an issue displaying this preview - https://phabricator.wikimedia.org/T402206#11104138 (10Tgr) ` tgr@deployment-restbase05:~$ sudo systemctl status restbase.service ...... [20:27:56] (03open) 10dancy: tests/scap/integration/test_backport.py: Add check for unexpected state [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/991 [20:28:00] (03update) 10dancy: tests/scap/integration/test_backport.py: Add check for unexpected state [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/991 [20:31:36] (03merge) 10dancy: tests/scap/integration/test_backport.py: Add check for unexpected state [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/991 [20:32:26] (03update) 10dancy: deploy_promote.py: Attribute wikiversions.json updates to the initiating user [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/990 (https://phabricator.wikimedia.org/T401779) [20:32:43] (03update) 10dancy: deploy_promote.py: Attribute wikiversions.json updates to the initiating user [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/990 (https://phabricator.wikimedia.org/T401779) [20:44:26] 10Gerrit, 10ProveIt-Gadget: Users of ProvieIt gadget get a 403 Forbidden fetching i18n files from Gerrit/Gitiles - https://phabricator.wikimedia.org/T394916#11104210 (10Iniquity) >>! In T394916#11102654, @Sophivorus wrote: > Done! Translations are now hosted at https://commons.wikimedia.org/wiki/Data:Gadge... [20:51:44] (03PS3) 10Jeena Huneidi: Update wikilambda daily selenium job [integration/config] - 10https://gerrit.wikimedia.org/r/1179783 (https://phabricator.wikimedia.org/T388056) [20:52:28] (03CR) 10Jeena Huneidi: Update wikilambda daily selenium job (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1179783 (https://phabricator.wikimedia.org/T388056) (owner: 10Jeena Huneidi) [20:57:31] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-etcd05 on project deployment-prep - https://phabricator.wikimedia.org/T402410#11104253 (10bd808) 05In progress→03Resolved Huh. I left this open because in the past I have seen the @wmcs-alerts bot open a new ticket if the ori... [21:10:20] 06Project-Admins: Create project tag for Wikinews volunteer developed software - https://phabricator.wikimedia.org/T402150#11104312 (10Dzahn) 05Resolved→03Open a:05thcipriani→03Aklapper [21:27:13] 10WikimediaDebug: Make WikimediaDebug status more sticky - https://phabricator.wikimedia.org/T315111#11104360 (10Reedy) With the rollout of PHP 8.3... Especially in these very early stages, I'd love to be able to stick my activity onto the PHP 8.3 hosts, and frankly leave it there for the near term future. [21:30:41] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 07Essential-Work, 13Patch-For-Review: Puppetize systemd unit for zuul's nodepool - https://phabricator.wikimedia.org/T401614#11104368 (10Dzahn) >>! In T401614#11099815, @bd80... [21:33:14] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 07Essential-Work, 13Patch-For-Review: Puppetize systemd unit for zuul's nodepool - https://phabricator.wikimedia.org/T401614#11104400 (10Dzahn) 05Open→03In progress p:05... [21:56:15] 10Scap, 06serviceops: Provide MediaWiki app image PHP version in helm values - https://phabricator.wikimedia.org/T401721#11104468 (10Scott_French) [21:58:35] 10Scap, 06serviceops: Provide MediaWiki app image PHP version in helm values - https://phabricator.wikimedia.org/T401721#11104485 (10Scott_French) At this point, the last step is to clean up the now-unused `vnd.wikimedia.mediawiki.php.version` label. Since this will result in a full image build, it would be ni... [22:06:27] (03open) 10swfrench: make-container-image: remove php version metadata label [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/209 (https://phabricator.wikimedia.org/T401721) [22:09:46] 10Continuous-Integration-Config, 10Metrics Platform, 05MW-1.39-release: MetricsPlatform breaks CI on REL1_39 - https://phabricator.wikimedia.org/T402463 (10Reedy) 03NEW [22:09:48] 10Continuous-Integration-Config, 10Metrics Platform, 05MW-1.39-release: MetricsPlatform breaks CI on REL1_39 - https://phabricator.wikimedia.org/T402463#11104539 (10Reedy) p:05Triage→03High [22:14:32] (03PS1) 10Reedy: zuul: Simplify REL1_39 ext_deps removals [integration/config] - 10https://gerrit.wikimedia.org/r/1180668 [22:14:32] (03PS1) 10Reedy: zuul: Remove MetricsPlatform from REL1_39 ext_deps [integration/config] - 10https://gerrit.wikimedia.org/r/1180669 (https://phabricator.wikimedia.org/T402463) [22:16:56] (03CR) 10Reedy: [C:03+2] zuul: Simplify REL1_39 ext_deps removals [integration/config] - 10https://gerrit.wikimedia.org/r/1180668 (owner: 10Reedy) [22:16:59] (03CR) 10Reedy: [C:03+2] zuul: Remove MetricsPlatform from REL1_39 ext_deps [integration/config] - 10https://gerrit.wikimedia.org/r/1180669 (https://phabricator.wikimedia.org/T402463) (owner: 10Reedy) [22:19:02] (03Merged) 10jenkins-bot: zuul: Simplify REL1_39 ext_deps removals [integration/config] - 10https://gerrit.wikimedia.org/r/1180668 (owner: 10Reedy) [22:19:03] (03Merged) 10jenkins-bot: zuul: Remove MetricsPlatform from REL1_39 ext_deps [integration/config] - 10https://gerrit.wikimedia.org/r/1180669 (https://phabricator.wikimedia.org/T402463) (owner: 10Reedy) [22:19:54] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1180668 https://gerrit.wikimedia.org/r/1180669 T402463 [22:19:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:19:57] T402463: MetricsPlatform breaks CI on REL1_39 - https://phabricator.wikimedia.org/T402463 [22:22:48] 10Continuous-Integration-Config, 10Metrics Platform, 05MW-1.39-release, 13Patch-For-Review: MetricsPlatform breaks CI on REL1_39 - https://phabricator.wikimedia.org/T402463#11104616 (10Reedy) 05Open→03Resolved a:03Reedy [22:27:00] 06Project-Admins: Create project tag for Wikinews volunteer developed software - https://phabricator.wikimedia.org/T402150#11104629 (10Gryllida) 05Open→03Resolved Resolved as I was added to the trusted contributors group and now am able to manage the project. Thanks to Bawolff 😊 [22:27:33] (03CR) 10Reedy: "I suspect this broke CI on REL1_39; filed in T402463 and fixed." [integration/config] - 10https://gerrit.wikimedia.org/r/1177978 (https://phabricator.wikimedia.org/T401308) (owner: 10Cyndywikime) [22:29:17] (03CR) 10Krinkle: [C:03+2] "No. This patch fixes a bug introduced to mwext- with commit I835e5cac85." [integration/config] - 10https://gerrit.wikimedia.org/r/1178071 (https://phabricator.wikimedia.org/T395470) (owner: 10Krinkle) [22:48:08] (03approved) 10thcipriani: deploy_promote.py: Attribute wikiversions.json updates to the initiating user [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/990 (https://phabricator.wikimedia.org/T401779) (owner: 10dancy) [23:20:03] maintenance-disconnect-full-disks build 729906 integration-agent-docker-1044 (/: 24%, /srv: 96%, /var/lib/docker: 27%): OFFLINE due to disk space [23:25:03] maintenance-disconnect-full-disks build 729907 integration-agent-docker-1044 (/: 24%, /srv: 11%, /var/lib/docker: 26%): RECOVERY disk space OK