[00:59:24] (03CR) 10Abijeet Patro: "Done, thanks" [integration/config] - 10https://gerrit.wikimedia.org/r/1370134 (https://phabricator.wikimedia.org/T442735) (owner: 10Abijeet Patro) [00:59:30] (03PS3) 10Abijeet Patro: Zuul: [mediawiki/extensions/Translate] Add ConfirmEdit dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1370134 (https://phabricator.wikimedia.org/T442735) [01:10:20] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/Translate] Add ConfirmEdit dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1370134 (https://phabricator.wikimedia.org/T442735) (owner: 10Abijeet Patro) [01:12:13] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Translate] Add ConfirmEdit dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1370134 (https://phabricator.wikimedia.org/T442735) (owner: 10Abijeet Patro) [02:59:59] 10Continuous-Integration-Infrastructure, 10Castor, 13Patch-For-Review, 06Test Platform (San Miguel de Allende 31): Widen the cache proxy npm trial from UploadWizard to six repos - https://phabricator.wikimedia.org/T444254#12569177 (10Mhurd) 05Openβ†’03Declined Declining. The six-repo trial (https://g... [08:06:23] GitLab needs a short maintenance break in a few minutes [09:10:29] 06Release-Engineering-Team, 10MediaWiki-Codesniffer, 07ci-test-error, 13Patch-For-Review: PHPCS seems to fail when the changed files list includes a filename without an extension - https://phabricator.wikimedia.org/T437269#12569698 (10ArthurTaylor) [09:28:45] 10Gerrit (Gerrit 3.11): Enforce project configuration changes for code review - https://phabricator.wikimedia.org/T445067#12569812 (10QChris) Thanks for the heads up! ❀ My scripts should be ready for that. So if you want to enable that setting, :+1: from me. [10:07:04] 10Gerrit (Gerrit 3.11): Enforce project configuration changes for code review - https://phabricator.wikimedia.org/T445067#12569993 (10hashar) Great thank you @qchris! I spoke with Tyler yesterday about it and I am not entirely sure whether I want that feature to be enabled. I guess I'll look at the history of c... [10:26:09] 06Release-Engineering-Team, 10MediaWiki-Codesniffer, 07ci-test-error, 13Patch-For-Review: PHPCS seems to fail when the changed files list includes a filename without an extension - https://phabricator.wikimedia.org/T437269#12570048 (10hashar) I see it is already fixed by https://gerrit.wikimedia.org/r/c/me... [10:30:08] codders: https://gerrit.wikimedia.org/r/c/mediawiki/tools/codesniffer/+/1370453 :) [10:31:55] awesome - thank you! [10:32:54] codders: I have tagged it [10:33:12] I'll push a patch to upgrade Wikibase and then repush my termbox patch to depend on it [10:34:49] 06Release-Engineering-Team, 10MediaWiki-Codesniffer, 07ci-test-error, 13Patch-For-Review: PHPCS seems to fail when the changed files list includes a filename without an extension - https://phabricator.wikimedia.org/T437269#12570079 (10hashar) I have crafted the changelog entry, https://gerrit.wikimedia.org... [10:36:45] codders: also mediawiki/tools/codesniffer inherits rights from mediawiki and thus you should be able to +2 and push signed tags (hopefully I hvaen't checked) [10:36:47] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/1369905 . let's see [10:36:59] 🀞 [10:37:43] I am happy to +2 the parent if that passes [10:42:48] looking good - please do! [10:43:37] codders: ! [10:43:39] done [10:43:45] I can't tell about termbox update ( https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/1369905/ ) [10:43:50] thanks so much! and thanks for the quick response [10:43:54] yeah - I'll take care of that [10:44:10] πŸŽ‰ [10:44:46] 06Release-Engineering-Team, 10MediaWiki-Codesniffer, 07ci-test-error, 13Patch-For-Review: PHPCS seems to fail when the changed files list includes a filename without an extension - https://phabricator.wikimedia.org/T437269#12570106 (10ArthurTaylor) p:05Unbreak!β†’03Triage Downgrading from UBN as CI is un... [10:45:27] hashar: Please don’t release MW-CS without the team. [10:45:39] I am not sure why we did not made the release earlier (thoguh we did cut intermediate releases to fix the REL* branches) [10:45:49] hashar: Especially, mis-numbered releases with breaking changes (new CI rules) tagged as not even minor but patch releases. [10:46:11] * 413b8b8 - Handle files without extension when linting only changed files (2 months ago) [10:46:14] that is the sole change there [10:46:25] * Upgrade phpcsstandards/phpcsextra (1.5.0 => 1.5.1) (Anomie) [10:46:35] That was your release too. [10:46:45] but [10:47:11] The uptreams for CI tools *always* lie about semver. [10:47:11] ah yeah that one was the previous release [10:47:34] The reason we didn’t release it is because I was going to do it next week, as part of our normal schedule. [10:47:38] Oh well. [10:47:56] (But with proper new fixes in.) [10:48:09] arf sorry mate -:-\ [10:48:25] It happens. [10:49:24] when reviewing the changelog I merelyl skipped the commits marked build: [10:50:53] anyway point taken, next time I ll ask for reviews! :) [10:53:10] 10Continuous-Integration-Config, 06MediaWiki-Core-Platform-Team, 10OOjs core, 07ci-test-error, 13Patch-For-Review: oojs-ci: New oojs-core-node26-runjenkins tasks FAILS due to timeouts on chrome - https://phabricator.wikimedia.org/T444995#12570138 (10Les4353) a:03Esanders [10:53:31] 10Continuous-Integration-Config, 06MediaWiki-Core-Platform-Team, 10OOjs core, 07ci-test-error, 13Patch-For-Review: oojs-ci: New oojs-core-node26-runjenkins tasks FAILS due to WebDriver error on firefox - https://phabricator.wikimedia.org/T444996#12570139 (10Les4353) a:03Esanders [11:23:29] 10Phabricator: phabricator: Tasks with multiple assignees have anomaly behavior in the workboard screen - https://phabricator.wikimedia.org/T445200 (10Les4353) 03NEW [11:23:45] (03open) 10brouberol: Add repos/data-engineering/hive-metastore to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/199 [13:23:37] 10Gerrit, 06Collaboration-Services, 07Incident Severity 1, 13Patch-For-Review, 07Sustainability (Incident Followup): Add the ability to warm caches to Gerrit switchover cookbook - https://phabricator.wikimedia.org/T440177#12570869 (10ABran-WMF) I re-tested the dry-runs on cumin1004 at the current patch s... [13:38:26] 10Continuous-Integration-Config, 06MediaWiki-Core-Platform-Team, 10OOjs core, 07ci-test-error: oojs-ci: New oojs-core-node26-runjenkins tasks FAILS due to timeouts on chrome - https://phabricator.wikimedia.org/T444995#12570948 (10Esanders) 05Openβ†’03Resolved [13:38:28] 10Continuous-Integration-Config, 06MediaWiki-Core-Platform-Team, 10OOjs core, 07ci-test-error: oojs-ci: New oojs-core-node26-runjenkins tasks FAILS due to WebDriver error on firefox - https://phabricator.wikimedia.org/T444996#12570950 (10Esanders) 05Openβ†’03Resolved [13:53:22] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 10a Wikimedia CLI: runner-1036 & runner-1034.gitlab-runners.eqiad1.wikimedia.cloud running out of space - https://phabricator.wikimedia.org/T443833#12571008 (10Addshore) @brennen heads up, this has just started happening on runner-1037.gitlab-runners.e... [13:56:32] 10Beta-Cluster-Infrastructure: navtiming is broken on the Beta Cluster because kafka-jumbo has been removed - https://phabricator.wikimedia.org/T445011#12571011 (10Ottomata) > File a proper patch to make the Kafka cluster in profile::webperf::processors configurable - we can reuse kafka-main. This seems just as... [13:57:44] 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12571017 (10ABran-WMF) [13:58:47] 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12571021 (10ABran-WMF) 05Openβ†’03Resolved Marking this `Resolved` πŸŽ‰. The remaining bit is being tracked in {T430113}. [14:08:06] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:27:16] 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: Update gitlab cookbooks and processes - https://phabricator.wikimedia.org/T430655#12571125 (10ABran-WMF) 05Openβ†’03Resolved [14:48:06] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [15:30:11] Yippee, build fixed! [15:30:11] Project mwcore-phpunit-coverage-master build #5801: 09FIXED in 30 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/5801/ [15:41:09] (03update) 10dancy: Add repos/data-engineering/hive-metastore to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/199 (owner: 10brouberol) [15:41:11] (03update) 10dancy: Add repos/data-engineering/hive-metastore to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/199 (owner: 10brouberol) [15:41:31] (03merge) 10dancy: Add repos/data-engineering/hive-metastore to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/199 (owner: 10brouberol) [17:10:01] hashar: Can you determine why https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1359373 isn’t triggering G&S? [17:10:35] depends-on? [17:10:41] None. [17:10:45] dependency cycle (iirc Zuul just discard it) [17:11:10] It is depended upon. [17:11:11] and the thing is the event got lost becase the Gerrit > Zuul stream events got disconnected/reconnected [17:11:14] so bad luck [17:11:29] Each time? I’ve C+2-commented twice. [17:13:09] yeah that is it [17:13:11] dependency cycle [17:13:25] on contint1002.wikimedia.org in /var/log/zuul/error.log [17:13:32] Exception: Dependency cycle detected: 1359384 in [1359393, 1359384] [17:13:32] Exception: Dependency cycle detected: 1359384 in [1359393, 1359384] [17:14:00] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ReadingLists/+/1359393 [17:14:07] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ReadingLists/+/1359384 depended on it, not the other way around. [17:14:10] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ReadingLists/+/1359384 [17:14:37] Should I just V+2 it into MW? [17:15:59] I think you can delete the change https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ReadingLists/+/1359384 [17:16:16] Can you delete it? [17:16:21] I don’t have that right. [17:16:23] Zuul is probably confused because the depends-on points back to the same change-id [17:16:40] I deleted all the depends-on directives in every repo. [17:16:45] But Zuul is still confused by them. [17:20:20] hashar: Aha, with it deleted it’s now going through G&S, thank you! [17:37:47] :) [17:38:14] I think what happens is the readinglists change 1359384 depends on itself [17:38:20] which is a dependency loop [17:39:04] not sure why Core was involved, but if it shares the same change-id that might explains it [17:39:18] aka Zuul looks at https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1359373 [17:39:32] extract the change-id, does a search query for anything that might depends on it [17:39:40] reach the ReadingLists change (which has the same change-id) [17:39:51] retrieve that REadingLists change, hit the dependency loop [17:39:56] and zuul discard the events [17:40:40] and the error only shows up in /var/log/zuul/error.log [17:40:42] :) [17:41:06] I am off!!!! [17:42:43] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Release-Engineering-Team (Doing 😎), 10Openstack-Magnum, 06tools-infrastructure-team: Rebuild the cloud-runner-staging magnum cluster in the zuul3 project with the new magnum-cluster-api driver - https://phabricator.wikimedia.org/T437498#12572429 (10d... [17:58:49] 10Continuous-Integration-Infrastructure (Zuul upgrade): Build and deploy Zuul 14.3.0 - https://phabricator.wikimedia.org/T445248 (10dduvall) 03NEW [17:59:10] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Release-Engineering-Team (Doing 😎): Build and deploy Zuul 14.3.0 - https://phabricator.wikimedia.org/T445248#12572572 (10dduvall) 05Openβ†’03In progress p:05Triageβ†’03Medium [18:09:45] 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 06tools-infrastructure-team, 07Puppet: Cannot rollout Trixie hosts in deployment-prep without rm -rf /var/lib/puppet/ssl - https://phabricator.wikimedia.org/T443373#12572634 (10dancy) 05Openβ†’03Resolved a:03dancy After letting https://gerrit.wikimedia.or... [18:11:08] for awareness: deploying https://gerrit.wikimedia.org/r/1370218 now which removes the /srv/deployment/mediawiki-staging symlink (see https://phabricator.wikimedia.org/T443851) -- feel free to revert in case of problems [18:11:55] πŸ‘πŸΎ [18:11:56] taavi has https://gerrit.wikimedia.org/r/1370579 for the rsync side too, will deploy it shortly [18:30:11] FIRING: [4x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cp-text09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [19:16:53] 10WikimediaDebug, 10CirrusSearch, 06Discovery-Search, 10RemexHtml, and 4 others: Function spl_object_hash is deprecated in php8.6 - https://phabricator.wikimedia.org/T445253 (10Umherirrender) 03NEW [19:29:23] 06Release-Engineering-Team, 10Scap (SpiderPig πŸ•ΈοΈ): Tighten spiderpig security - https://phabricator.wikimedia.org/T445255 (10dancy) 03NEW [19:30:53] 06Release-Engineering-Team, 10Scap (SpiderPig πŸ•ΈοΈ): Tighten spiderpig security - https://phabricator.wikimedia.org/T445255#12573001 (10dancy) [21:29:49] 06Release-Engineering-Team, 10Scap (SpiderPig πŸ•ΈοΈ), 13Patch-For-Review: Tighten spiderpig security - https://phabricator.wikimedia.org/T445255#12573514 (10dancy) [21:33:06] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [22:15:11] FIRING: [4x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cp-text09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [22:15:21] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cp-upload09 in project deployment-prep - https://phabricator.wikimedia.org/T445275 (10wmcs-alerts) 03NEW [22:16:05] (03open) 10addshore: Draft: Agents, guardrails, and command metadata [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/821 [22:16:28] 10Continuous-Integration-Infrastructure, 10Castor, 06Test Platform (San Miguel de Allende 31): Let CI jobs fetch npm packages through the cache proxy, with Castor fallback - https://phabricator.wikimedia.org/T444894#12573682 (10Mhurd) [22:20:11] RESOLVED: [2x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cp-text09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [22:27:29] (03PS1) 10Mhurd: jjb: point npm at integration-cache-proxy when Zuul asks [integration/config] - 10https://gerrit.wikimedia.org/r/1370655 (https://phabricator.wikimedia.org/T444894) [22:29:28] !log deployment-puppetserver-1: Vacuum systemd journal and remove pacct file, disk exhausted [22:29:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:33:06] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [22:40:32] (03PS2) 10Mhurd: jjb: point npm at integration-cache-proxy when Zuul asks [integration/config] - 10https://gerrit.wikimedia.org/r/1370655 (https://phabricator.wikimedia.org/T444894) [22:46:10] !log Restart cassandra and kafka instances in Beta Cluster [22:46:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:51:08] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 2 deleted instances on deployment-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [23:03:06] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-webperf31 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:13:06] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance deployment-webperf31 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:23:19] (03update) 10mstyles: Add scripts for reserving CVE IDs and listing the tasks of a security release [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/285 (https://phabricator.wikimedia.org/T443837) [23:23:35] (03update) 10mstyles: Add scripts for reserving CVE IDs and listing the tasks of a security release [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/285 (https://phabricator.wikimedia.org/T443837) [23:23:51] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance deployment-webperf31 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:26:35] 10Beta-Cluster-Infrastructure: Found non-revoked Puppet certificates for 2 deleted instances on deployment-puppetserver-1 - https://phabricator.wikimedia.org/T445278 (10wmcs-alerts) 03NEW [23:30:05] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-cp-text09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [23:33:51] (03update) 10mstyles: Add scripts for reserving CVE IDs and listing the tasks of a security release [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/285 (https://phabricator.wikimedia.org/T443837) [23:34:12] (03update) 10mstyles: Add scripts for reserving CVE IDs and listing the tasks of a security release [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/285 (https://phabricator.wikimedia.org/T443837) [23:35:06] FIRING: [2x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cp-text09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [23:43:00] !log Decommission deployment-zookeeper-3 - T436481 [23:43:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:43:03] T436481: Replace deployment-zookeeper-3 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436481 [23:44:14] dancy: Can I remove the stale puppet certs for `deployment-dancytest0{1,3}`? [23:44:50] Yes please! [23:45:07] Thanks, will remove :) [23:45:12] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Replace deployment-zookeeper-3 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436481#12573925 (10Southparkfan) 05In progressβ†’03Resolved [23:46:19] !log Cleaned stale puppet certs for `deployment-dancytest0{1,3}` [23:46:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:46:47] Well, with deployment-zookeeper-3 gone, RESTBase is the only thing on the Beta Cluster still not running on bookworm or trixie [23:54:43] 10Beta-Cluster-Infrastructure: Found non-revoked Puppet certificates for 2 deleted instances on deployment-puppetserver-1 - https://phabricator.wikimedia.org/T445278#12573975 (10Southparkfan) 05Openβ†’03Resolved a:03Southparkfan https://sal.toolforge.org/log/YHXn66EBDAZZyZXngGAO [23:55:13] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T445160#12573979 (10Southparkfan) 05Openβ†’03Resolved a:03Southparkfan https://sal.toolforge.org/log/VHSh66EBDA... [23:55:43] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-zookeeper04 in project deployment-prep - https://phabricator.wikimedia.org/T445012#12573996 (10Southparkfan) 05Openβ†’03Resolved a:03Southparkfan Runs fine now: T436481. [23:56:08] RESOLVED: PuppetStaleCertificates: Found non-revoked Puppet certificates for 2 deleted instances on deployment-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [23:56:16] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-zookeeper-3 in project deployment-prep - https://phabricator.wikimedia.org/T443134#12574006 (10Southparkfan) 05Openβ†’03Resolved a:03Southparkfan Decommissioned: T436481. [23:56:37] 10Beta-Cluster-Infrastructure: Widespread instances down in project deployment-prep - https://phabricator.wikimedia.org/T442126#12574012 (10Southparkfan) 05Openβ†’03Resolved a:03Southparkfan Beta Cluster is already back online: T442482. [23:57:03] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-kafka-jumbo-9 is down - https://phabricator.wikimedia.org/T440387#12574021 (10Southparkfan) 05Openβ†’03Invalid kafka-jumbo is no longer: T437541. [23:57:12] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-kafka-jumbo-5 is down - https://phabricator.wikimedia.org/T440388#12574028 (10Southparkfan) 05Openβ†’03Invalid kafka-jumbo is no longer: T437541. [23:57:44] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-ms-fe04 on project deployment-prep - https://phabricator.wikimedia.org/T440205#12574033 (10Southparkfan) 05Openβ†’03Resolved a:03Southparkfan Instance no longer exists: T436476.