[08:10:41] <wikibugs>	 06serviceops, 10Language-Technical Support, 06SRE, 10Wikimedia-Site-requests, 13Patch-For-Review: Change $wgMaxArticleSize limit from byte-based to character-based - https://phabricator.wikimedia.org/T275319#9878379 (10Fuzzy) What would you suggest to reduce the template size? The external `<span class="...
[08:23:56] <godog>	 kamila_: FYI I noticed benthos mw-accesslog-metrics is lagging, not sure what's going on tho https://phabricator.wikimedia.org/T367076
[08:24:14] <godog>	 would you mind taking a look? thank you!
[08:26:06] <kamila_>	 Oh, yeah, will check, thanks godog 
[08:26:23] <godog>	 sure np, let me know how it goes kamila_ 
[08:27:25] <kamila_>	 Will do 
[09:51:17] <wikibugs>	 06serviceops, 10MW-on-K8s, 07Datacenter-Switchover: Control mw-on-k8s periodic maintenance jobs with an etcd value - https://phabricator.wikimedia.org/T367118#9878699 (10Clement_Goubert) #3 sounds like the most sane option to me as well, and enables us to halt maintenance for other reasons than a DC switchov...
[10:31:24] <kamila_>	 godog: looking at the benthos thing, the errors are a separate problem (mw access log encoding issues I'm guessing), other than that I think benthos is just running out of CPU now that we have so much traffic on k8s
[10:31:26] <wikibugs>	 06serviceops, 10MW-on-K8s: glogger crashes regularly in mw-on-k8s containers - https://phabricator.wikimedia.org/T363342#9878886 (10Clement_Goubert) I've tried to debug this unsuccessfully, but just realized that #serviceops wasn't tagged. Can someone in the team with better go than me have a look?
[10:33:31] <wikibugs>	 06serviceops, 10Wikimedia-Apache-configuration: 2030.wikimedia.org is a double redirect - https://phabricator.wikimedia.org/T367013#9878895 (10akosiaris) Thanks for the historical aspect @Dzahn. Given that perspective and the fact the double redirect isn't apparently considered a problem, my opinion is that it...
[10:33:32] <kamila_>	 why it coincides with the reboots I'm not sure, maybe our steadily moving traffic hit some sort of a limit 
[10:35:33] <godog>	 kamila_: ah yeah certainly possible there isn't enough cpu
[10:36:10] <godog>	 I'm puzzled too re: the reboot, double checking that's the case
[10:36:11] <kamila_>	 godog: just staring at htop doesn't seem immediately problematic though, I'll look at benthos's parallelism configs
[10:37:09] <godog>	 yeah by default all cpus are used unless specified otherwise
[10:37:52] <godog>	 for reference the other benthos instance on centrallog1002 consumes half of the webrequest firehose which is ~60k msg/s
[10:38:07] <godog>	 total being ~150k msg/s max
[10:39:37] <kamila_>	 where do I check the throughput for the mediawiki.httpd.accesslog topic? 
[10:40:17] <godog>	 the kafka by topic dashboard e.g https://grafana.wikimedia.org/d/000000234/kafka-by-topic?orgId=1&refresh=5m&var-datasource=eqiad+prometheus%2Fops&var-kafka_cluster=logging-eqiad&var-kafka_broker=All&var-topic=mediawiki.httpd.accesslog&from=1717929599448&to=1718102399448
[10:40:22] <kamila_>	 oh, thanks
[10:40:40] <claime>	 godog was faster :P
[10:40:53] <kamila_>	 :D
[10:41:10] <godog>	 hehe I was just looking at it too
[10:41:13] <claime>	 btw we're going to have to roll-reboot the kafka servers soon
[10:41:29] <kamila_>	 so it's not huge... then I wonder if the problematic messages now stay in the queue for some reason, but that was not happening in the past 
[10:43:34] <godog>	 kamila_: possible though I doubt it, benthos should drop everything if I'm reading the config correctly
[10:43:37] <godog>	 errors or not
[10:43:54] <godog>	 claime: FWIW I used sre.kafka.roll-restart-reboot-brokers this morning for kafka-logging and worked pretty well
[10:43:55] <kamila_>	 yeah, and we've had funky messages before 
[10:44:19] <claime>	 godog: cool, that's what I was planning on
[10:44:23] <kamila_>	 I really think the errors are unrelated, and then I'm not sure what's happening
[10:44:44] <claime>	 "pretty well" meaning you didn't encounter any issue, or you had some things to do outside the cookbook?
[10:45:35] <godog>	 claime: the former, did what it said on the tin
[10:45:42] <claime>	 nice
[10:46:25] <godog>	 kamila_: yeah I'm looking at the benthos dashboard for that instance and don't see any obvious difference https://grafana.wikimedia.org/d/V0TSK7O4z/benthos?from=now-30d&orgId=1&to=now&var-port=4152
[10:48:48] <wikibugs>	 06serviceops, 06Machine-Learning-Team, 13Patch-For-Review: Rename the envoy's uses_ingress option to sets_sni - https://phabricator.wikimedia.org/T346638#9879003 (10JMeybohm)
[10:53:48] <wikibugs>	 06serviceops, 10MW-on-K8s, 10Observability-Logging, 06SRE: benthos mw-accesslog-metrics kafka lag and interpolation errors - https://phabricator.wikimedia.org/T367076#9879026 (10fgiunchedi)
[10:55:18] <godog>	 I don't think the reboot is implicated btw
[10:56:37] <kamila_>	 Thanks for checking godog 
[10:57:00] <kamila_>	 We also moved commons to k8s around that time, but I need to check the exact time, and I need to have lunch first, bbiab
[10:57:36] <godog>	 ok thanks
[10:58:42] <claime>	 I'm going to start rebooting kafka-main in codfw 
[11:03:11] <jayme>	 dcausse: the consumer-search-backfill release in staging is still using an older version of the flink-app chart. Is that expected?
[11:24:36] <wikibugs>	 06serviceops, 10Similarusers, 13Patch-For-Review: Remove similar-users service from k8s - https://phabricator.wikimedia.org/T345274#9879146 (10JMeybohm)
[11:25:34] <wikibugs>	 06serviceops, 10Similarusers, 13Patch-For-Review: Remove similar-users service from k8s - https://phabricator.wikimedia.org/T345274#9879148 (10JMeybohm)
[11:45:50] <wikibugs>	 06serviceops, 06Infrastructure-Foundations, 06Release-Engineering-Team, 13Patch-For-Review: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9879216 (10Clement_Goubert) Deleted `docker-registry.wikimedia.org/wikimedia/analytics-datahub`
[11:49:13] <wikibugs>	 06serviceops, 10Similarusers, 13Patch-For-Review: Remove similar-users service from k8s - https://phabricator.wikimedia.org/T345274#9879225 (10JMeybohm)
[12:04:39] <dcausse>	 jayme: looking, this one should be short lived 
[12:43:38] <jayme>	 dcausse: okay, cool. thanks
[12:59:12] <wikibugs>	 06serviceops, 06collaboration-services, 10GitLab, 06Infrastructure-Foundations, 13Patch-For-Review: Container image reports in debmonitor are broken - https://phabricator.wikimedia.org/T348876#9879555 (10elukey)
[13:00:58] <wikibugs>	 06serviceops, 10Language-Technical Support, 06SRE, 10Wikimedia-Site-requests, 13Patch-For-Review: Change $wgMaxArticleSize limit from byte-based to character-based - https://phabricator.wikimedia.org/T275319#9879565 (10stjn) >>! In T275319#9878379, @Fuzzy wrote: > The external `<span class="...">` is nec...
[13:01:45] <wikibugs>	 06serviceops, 10Similarusers, 13Patch-For-Review: Remove similar-users service from k8s - https://phabricator.wikimedia.org/T345274#9879590 (10JMeybohm)
[13:11:31] <moritzm>	 kubetcd1005 will go down for a few minutes for a reboot of the underlying Ganeti node
[13:12:22] <wikibugs>	 06serviceops, 06Machine-Learning-Team, 13Patch-For-Review: Rename the envoy's uses_ingress option to sets_sni - https://phabricator.wikimedia.org/T346638#9879651 (10JMeybohm)
[13:12:24] <wikibugs>	 06serviceops, 10Similarusers, 13Patch-For-Review: Remove similar-users service from k8s - https://phabricator.wikimedia.org/T345274#9879648 (10JMeybohm) 05Open→03Resolved Finally gone...
[13:24:54] <wikibugs>	 06serviceops, 10Language-Technical Support, 06SRE, 10Wikimedia-Site-requests, 13Patch-For-Review: Change $wgMaxArticleSize limit from byte-based to character-based - https://phabricator.wikimedia.org/T275319#9879697 (10Fuzzy) > I get the point of external class, but I don’t get the point of the internal...
[13:25:24] <wikibugs>	 06serviceops, 06Infrastructure-Foundations, 06Release-Engineering-Team, 13Patch-For-Review: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9879702 (10Clement_Goubert) Deleted `docker-registry.wikimedia.org/wikimedia/mediawiki-services-similar-users` following resolution of T345274
[13:26:36] <wikibugs>	 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE-OnFire, 10Sustainability (Incident Followup): codfw:(3) wikikube-ctrl NIC upgrade to 10G - https://phabricator.wikimedia.org/T366205#9879703 (10Jhancock.wm) proposed relocations for each server. lemme know if that works for you @Papaul   wikikube-ctrl2001 curren...
[13:40:24] <wikibugs>	 06serviceops, 10Language-Technical Support, 06SRE, 10Wikimedia-Site-requests, 13Patch-For-Review: Change $wgMaxArticleSize limit from byte-based to character-based - https://phabricator.wikimedia.org/T275319#9879781 (10stjn) Then it should be added only where anchors are involved. I suggest converting to...
[13:46:17] <wikibugs>	 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE-OnFire, 10Sustainability (Incident Followup): codfw:(3) wikikube-ctrl NIC upgrade to 10G - https://phabricator.wikimedia.org/T366205#9879840 (10kamila) @Papaul Thanks for the additional details!  I think moving to the new per-rack VLAN shouldn't be a problem for...
[14:44:31] <wikibugs>	 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE-OnFire, 10Sustainability (Incident Followup): eqiad:(3) wikikube-ctrl NIC upgrade to 10G - https://phabricator.wikimedia.org/T366204#9880133 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by kamila@cumin1002 for hosts: `wikikube-ctrl1002.eqiad....
[14:50:32] <wikibugs>	 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9880170 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by cgoubert@cumin1002 from mw1402 to wikikube-worker1013 completed: - mw1402 (**PASS*...
[14:51:43] <wikibugs>	 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9880185 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by cgoubert@cumin1002 from mw1403 to wikikube-worker1014.eqiad.wmnet completed: - mw1...
[14:53:23] <wikibugs>	 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9880193 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1013.eqiad.wmnet with OS bullseye
[16:21:25] <wikibugs>	 06serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9880675 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1014.eqiad.wmnet with OS bullseye completed: - wikikube-work...
[16:22:00] <wikibugs>	 06serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9880700 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1018.eqiad.wmnet with OS bullseye completed: - wikikube-work...
[16:22:42] <wikibugs>	 06serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9880720 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1017.eqiad.wmnet with OS bullseye completed: - wikikube-work...
[16:29:37] <wikibugs>	 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE-OnFire, 10Sustainability (Incident Followup): eqiad:(3) wikikube-ctrl NIC upgrade to 10G - https://phabricator.wikimedia.org/T366204#9880802 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-ctrl1002.eq...
[16:36:52] <wikibugs>	 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE-OnFire, 10Sustainability (Incident Followup): eqiad:(3) wikikube-ctrl NIC upgrade to 10G - https://phabricator.wikimedia.org/T366204#9880830 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-ctrl1002.eqiad....
[16:39:47] <wikibugs>	 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE-OnFire, 10Sustainability (Incident Followup): eqiad:(3) wikikube-ctrl NIC upgrade to 10G - https://phabricator.wikimedia.org/T366204#9880863 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-ctrl1002.eq...
[16:44:11] <wikibugs>	 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE-OnFire, 10Sustainability (Incident Followup): eqiad:(3) wikikube-ctrl NIC upgrade to 10G - https://phabricator.wikimedia.org/T366204#9880919 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-ctrl1002.eqiad....
[16:52:34] <wikibugs>	 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE-OnFire, 10Sustainability (Incident Followup): eqiad:(3) wikikube-ctrl NIC upgrade to 10G - https://phabricator.wikimedia.org/T366204#9880955 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-ctrl1002.eq...
[16:57:07] <wikibugs>	 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE-OnFire, 10Sustainability (Incident Followup): eqiad:(3) wikikube-ctrl NIC upgrade to 10G - https://phabricator.wikimedia.org/T366204#9880980 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-ctrl1002.eqiad....
[16:59:41] <wikibugs>	 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE-OnFire, 10Sustainability (Incident Followup): eqiad:(3) wikikube-ctrl NIC upgrade to 10G - https://phabricator.wikimedia.org/T366204#9880993 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-ctrl1002.eq...
[17:09:26] <wikibugs>	 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE-OnFire, 10Sustainability (Incident Followup): eqiad:(3) wikikube-ctrl NIC upgrade to 10G - https://phabricator.wikimedia.org/T366204#9881034 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-ctrl1002.eqiad....
[19:23:39] <wikibugs>	 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE-OnFire, 10Sustainability (Incident Followup): eqiad:(3) wikikube-ctrl NIC upgrade to 10G - https://phabricator.wikimedia.org/T366204#9881923 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-ctrl1002.eq...
[19:33:55] <wikibugs>	 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE-OnFire, 10Sustainability (Incident Followup): eqiad:(3) wikikube-ctrl NIC upgrade to 10G - https://phabricator.wikimedia.org/T366204#9881979 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-ctrl1002.eqiad....