[10:00:55] 06serviceops, 06Discovery-Search, 10Data-Platform-SRE (2025.03.22 - 2025.04.11): Search Update Pipeline requests to Action API are logged as coming from 127.0.0.1 - https://phabricator.wikimedia.org/T388855#10661123 (10Gehel) [10:09:47] 06serviceops, 10Wikidata, 06Wikidata Dev Team, 10Wikidata Integration in Wikimedia projects: Migrate wikidata_dev_team jobs to mw-cron - https://phabricator.wikimedia.org/T388543#10661179 (10karapayneWMDE) @Ifeatu_Nnaobi_WMDE , @WMDECyn , this is related to change dispatch so falls into your team's scope :) [10:17:30] 06serviceops, 10Observability-Logging, 13Patch-For-Review: Logs from containers sometimes not visible in logstash - https://phabricator.wikimedia.org/T357616#10661197 (10fgiunchedi) 05Resolved→03Open Sadly I have to reopen this, not sure yet it is the same issue though while investigating {T389603} I fou... [10:45:38] 06serviceops, 06SRE, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 10Wikimedia-Site-requests: Increase entityAccessLimit for WikibaseClient wikis - https://phabricator.wikimedia.org/T384455#10661306 (10seanleong-WMDE) Hii, got it! We will create a patch to bump the limit for other wikis to 50... [10:56:06] 06serviceops, 07sre-alert-triage: Alert in need of triage: Postgres Replication Lag (instance maps-test2002) - https://phabricator.wikimedia.org/T388782#10661327 (10LSobanski) There are three overdue alerts for maps-test, two of which are critical. Can these be disabled or downgraded? [11:38:48] 06serviceops, 10MW-on-K8s, 06Release-Engineering-Team: Refactor scap's kubernetes DeploymentsConfig to support selection of image kinds - https://phabricator.wikimedia.org/T389499#10661423 (10Clement_Goubert) I think the second option is sound and would greatly simplify adding new images. If we go with thi... [11:42:07] 06serviceops, 07sre-alert-triage: Alert in need of triage: Postgres Replication Lag (instance maps-test2002) - https://phabricator.wikimedia.org/T388782#10661429 (10Clement_Goubert) 05Open→03Resolved a:03Clement_Goubert I think @elukey fixed those, they appear green in icinga now. Feel free to reopen... [11:43:18] 06serviceops, 07sre-alert-triage: Alert in need of triage: Postgres Replication Lag (instance maps-test2002) - https://phabricator.wikimedia.org/T388782#10661433 (10MoritzMuehlenhoff) You're wrong, but it's still fine to close :-) Luca fixed the replication lag for the main maps cluster, this is for the W... [14:43:15] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10662183 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-ctrl2004.codfw.wmnet with O... [14:43:19] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10662187 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2249.codfw.wmnet with... [14:43:22] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10662188 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2250.codfw.wmnet with... [14:43:25] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10662189 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2251.codfw.wmnet with... [14:43:26] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10662190 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2252.codfw.wmnet with... [15:14:40] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10662358 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2250.codfw.wmnet with OS... [15:17:09] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10662387 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2252.codfw.wmnet with OS... [15:20:04] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10662398 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-ctrl2004.codfw.wmnet with OS bo... [15:28:11] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10662447 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2251.codfw.wmnet with OS... [15:39:43] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10662504 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2249.codfw.wmnet with OS... [15:53:49] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10662586 (10Jhancock.wm) [16:47:20] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10662946 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2254.codfw.wmnet with... [16:47:22] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10662947 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2255.codfw.wmnet with... [16:47:25] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10662948 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2256.codfw.wmnet with... [16:47:27] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10662949 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2258.codfw.wmnet with... [16:47:29] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10662950 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2257.codfw.wmnet with... [17:29:32] 06serviceops, 10MW-on-K8s, 06Release-Engineering-Team: Refactor scap's kubernetes DeploymentsConfig to support selection of image kinds - https://phabricator.wikimedia.org/T389499#10663237 (10Scott_French) Thanks, @Clement_Goubert. So, I realized in retrospect that adding a new image kind of course //alread... [17:44:10] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10663370 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2253.codfw.wmnet with OS... [17:46:17] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10663386 (10Jhancock.wm) [17:47:25] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10663406 (10Jhancock.wm) wikikube-worker2254-58 did not ping on the network address after dhcp. need to investigate. [18:07:45] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10663496 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2256.codfw.wmnet with OS... [18:07:46] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10663497 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2257.codfw.wmnet with OS... [18:07:50] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10663498 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2254.codfw.wmnet with OS... [18:07:56] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10663509 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2258.codfw.wmnet with OS... [18:08:07] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10663510 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2255.codfw.wmnet with OS... [18:15:25] 06serviceops, 10Excimer, 06MediaWiki-Platform-Team: Update Excimer to 1.2.3 in production - https://phabricator.wikimedia.org/T389243#10663533 (10Scott_French) Following up, two of the three package updates I was hoping to consolidate together (to minimize production image upgrades during the switchover week... [21:47:33] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, and 2 others: Ensure all required kubectl versions are installed on deploy hosts - https://phabricator.wikimedia.org/T388388#10664229 (10jhathaway) Unfortunately, it appears that Puppet made the unwise decision to combine stder... [23:11:02] 06serviceops, 06Commons, 10MediaWiki-File-management, 06SRE, and 2 others: Frequent "Error: 429, Too Many Requests" errors on pages with many (>50) thumbnails - https://phabricator.wikimedia.org/T266155#10664451 (10zdev) >>! In T266155#9766334, @Bawolff wrote: > I think if we did deliver the wrong thumbsiz...