[07:35:51] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Juniper: use export-format state-data json compact - https://phabricator.wikimedia.org/T362523#9833931 (10ayounsi) JTAC was able to confirm/duplicate the bug on 22.3R3-S2.4, they're escalating it to their engineering team. [08:50:05] 06Traffic, 06Data-Engineering, 10Observability-Logging: Install benthos on single esams host to check performances under higher load - https://phabricator.wikimedia.org/T365968 (10Fabfur) 03NEW [10:22:50] godog: BTW, it looks like mtail just merged a patch for the stdin slowness issue (https://github.com/google/mtail/pull/869) should we get that version in bookworm-wikimedia under the main component or create a new one just for the CDN? [10:31:18] 06Traffic, 10MoveComms-Support, 10MW-on-K8s, 06serviceops, and 2 others: Move 100% of external traffic to Kubernetes (excluding Votewiki) - https://phabricator.wikimedia.org/T362323#9834404 (10Clement_Goubert) As far as mediawiki calling itself goes (I see it was removed from the task description, but it i... [11:46:28] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f5-eqiad - https://phabricator.wikimedia.org/T365982 (10ABran-WMF) 03NEW [11:48:08] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9834595 (10ABran-WMF) [11:49:06] 10netops, 06Infrastructure-Foundations, 06SRE: T348977: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f6-eqiad - https://phabricator.wikimedia.org/T365983 (10ABran-WMF) 03NEW [11:50:10] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9834613 (10ABran-WMF) [11:50:58] 10netops, 06Infrastructure-Foundations, 06SRE: T348977: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad - https://phabricator.wikimedia.org/T365984 (10ABran-WMF) 03NEW [11:51:53] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9834628 (10ABran-WMF) [11:58:11] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e5-eqiad - https://phabricator.wikimedia.org/T365986 (10ABran-WMF) 03NEW [11:58:49] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e6-eqiad - https://phabricator.wikimedia.org/T365987 (10ABran-WMF) 03NEW [11:58:50] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9834657 (10ABran-WMF) [11:59:29] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e6-eqiad - https://phabricator.wikimedia.org/T365988 (10ABran-WMF) 03NEW [11:59:58] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e7-eqiad - https://phabricator.wikimedia.org/T365988#9834682 (10ABran-WMF) [12:01:28] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9834684 (10ABran-WMF) [12:37:55] vgutierrez: I'm only briefly at the keyboard today due to cold, though +1 to main [12:38:10] godog: ouch, sorry about the ping :) [12:38:57] vgutierrez: sure no worries, just passing by and I'm going afk again [13:01:55] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f5-eqiad - https://phabricator.wikimedia.org/T365982#9834830 (10ABran-WMF) [13:02:05] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f5-eqiad - https://phabricator.wikimedia.org/T365982#9834831 (10ABran-WMF) [13:03:41] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e7-eqiad - https://phabricator.wikimedia.org/T365988#9834834 (10ABran-WMF) [13:04:12] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e6-eqiad - https://phabricator.wikimedia.org/T365987#9834836 (10ABran-WMF) [13:05:40] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e5-eqiad - https://phabricator.wikimedia.org/T365986#9834838 (10ABran-WMF) [13:05:44] 10netops, 06Infrastructure-Foundations, 06SRE: T348977: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad - https://phabricator.wikimedia.org/T365984#9834840 (10ABran-WMF) [13:06:32] 10netops, 06Infrastructure-Foundations, 06SRE: T348977: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f6-eqiad - https://phabricator.wikimedia.org/T365983#9834842 (10ABran-WMF) [13:07:38] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e1-eqiad - https://phabricator.wikimedia.org/T365993 (10ABran-WMF) 03NEW [13:08:04] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e1-eqiad - https://phabricator.wikimedia.org/T365993#9834861 (10ABran-WMF) [13:09:11] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e2-eqiad - https://phabricator.wikimedia.org/T365994 (10ABran-WMF) 03NEW [13:09:27] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e2-eqiad - https://phabricator.wikimedia.org/T365994#9834874 (10ABran-WMF) [13:11:04] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e3-eqiad - https://phabricator.wikimedia.org/T365995 (10ABran-WMF) 03NEW [13:11:33] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e3-eqiad - https://phabricator.wikimedia.org/T365995#9834886 (10ABran-WMF) [13:13:14] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9834888 (10ABran-WMF) [13:16:03] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f6-eqiad - https://phabricator.wikimedia.org/T365983#9834899 (10ABran-WMF) [13:16:13] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad - https://phabricator.wikimedia.org/T365984#9834905 (10ABran-WMF) [13:16:41] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e7-eqiad - https://phabricator.wikimedia.org/T365988#9834911 (10ABran-WMF) a:05cmooney→03MatthewVernon [13:16:58] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e6-eqiad - https://phabricator.wikimedia.org/T365987#9834908 (10ABran-WMF) a:05cmooney→03ABran-WMF [13:19:37] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f1-eqiad - https://phabricator.wikimedia.org/T365996 (10ABran-WMF) 03NEW [13:21:15] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f2-eqiad - https://phabricator.wikimedia.org/T365997 (10ABran-WMF) 03NEW [13:21:44] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f2-eqiad - https://phabricator.wikimedia.org/T365997#9834941 (10ABran-WMF) [13:22:35] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f3-eqiad - https://phabricator.wikimedia.org/T365998 (10ABran-WMF) 03NEW [13:22:48] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f3-eqiad - https://phabricator.wikimedia.org/T365998#9834959 (10ABran-WMF) [13:24:08] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9834960 (10ABran-WMF) [14:27:33] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Arelion IPv6 transit renumbering - https://phabricator.wikimedia.org/T365697#9835124 (10ayounsi) p:05Triage→03Medium a:03ayounsi [14:28:13] 10Acme-chief, 06Traffic, 06Infrastructure-Foundations, 10Puppet-Infrastructure, 06SRE: Revert back to fleet-wide acmechief config once all ACME consumers are on Puppet 7 - https://phabricator.wikimedia.org/T365799#9835132 (10MoritzMuehlenhoff) p:05Triage→03High [14:28:42] are we in a hurry moritzm? :) [14:30:28] not urgent, but it's part of the wider "get rid of Puppet 5" work, which is also tagged as high :-) [14:32:42] do you have tasks for the remaining hosts? those two are effectively blocking T365799 [14:32:43] T365799: Revert back to fleet-wide acmechief config once all ACME consumers are on Puppet 7 - https://phabricator.wikimedia.org/T365799 [14:58:40] both are dependant on being moved off Buster first, lists1001 is being actively worked on ATM and archiva is https://phabricator.wikimedia.org/T349292 [14:59:17] although Archiva development has stopped and things are moved to gitlab as replacement, so it might also get decommed rather than upgraded [15:01:27] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f5-eqiad - https://phabricator.wikimedia.org/T365982#9835264 (10ABran-WMF) [15:01:36] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f6-eqiad - https://phabricator.wikimedia.org/T365983#9835266 (10ABran-WMF) [15:01:49] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad - https://phabricator.wikimedia.org/T365984#9835267 (10ABran-WMF) [15:01:58] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e5-eqiad - https://phabricator.wikimedia.org/T365986#9835268 (10ABran-WMF) [15:02:07] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e6-eqiad - https://phabricator.wikimedia.org/T365987#9835269 (10ABran-WMF) [15:02:17] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e7-eqiad - https://phabricator.wikimedia.org/T365988#9835270 (10ABran-WMF) [15:02:25] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e1-eqiad - https://phabricator.wikimedia.org/T365993#9835271 (10ABran-WMF) [15:02:34] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e2-eqiad - https://phabricator.wikimedia.org/T365994#9835272 (10ABran-WMF) [15:02:55] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e3-eqiad - https://phabricator.wikimedia.org/T365995#9835273 (10ABran-WMF) [15:03:02] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f1-eqiad - https://phabricator.wikimedia.org/T365996#9835274 (10ABran-WMF) [15:03:16] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f2-eqiad - https://phabricator.wikimedia.org/T365997#9835275 (10ABran-WMF) [15:03:25] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f3-eqiad - https://phabricator.wikimedia.org/T365998#9835276 (10ABran-WMF) [15:16:07] 10Wikimedia-Apache-configuration, 10MediaWiki-Configuration, 06Security-Team, 07Security: Remove docroot/wikimediafoundation.org/ folder from mediawiki-config - https://phabricator.wikimedia.org/T366005 (10mmartorana) 03NEW [15:17:08] 10Wikimedia-Apache-configuration, 10MediaWiki-Configuration, 06Security-Team, 07Security: Remove docroot/wikimediafoundation.org/ folder from mediawiki-config - https://phabricator.wikimedia.org/T366005#9835329 (10mmartorana) [16:10:26] FIRING: SystemdUnitFailed: ncmonitor.service on ncmonitor1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:21:57] extending it [20:19:44] 06Traffic, 06Data-Engineering, 10Observability-Logging: Upgrade Benthos package on cp hosts - https://phabricator.wikimedia.org/T366031 (10Fabfur) 03NEW [20:20:41] 06Traffic, 06Data-Engineering, 10Observability-Logging: Upgrade Benthos package on cp hosts - https://phabricator.wikimedia.org/T366031#9836148 (10Fabfur)