[00:05:25] RESOLVED: [6x] SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:19:40] 06Traffic, 06Data-Persistence, 06MediaWiki-Media-Platform-Team, 13Patch-For-Review, 07User-notice: Move thumbnail caching from upload cluster to text - https://phabricator.wikimedia.org/T427465#12273386 (10PacmanD) The ProofreadPage tool is broken on zh.wikisource.org. See: https://zh.wikisource.org/w/... [00:23:20] 06Traffic, 06Data-Persistence, 06MediaWiki-Media-Platform-Team, 13Patch-For-Review, 07User-notice: Move thumbnail caching from upload cluster to text - https://phabricator.wikimedia.org/T427465#12273387 (10Ladsgroup) That actually has nothing to do with this change (and it was only codfw), I just pushed... [03:44:54] 06Traffic, 06Data-Persistence, 06MediaWiki-Media-Platform-Team, 13Patch-For-Review, 07User-notice: Move thumbnail caching from upload cluster to text - https://phabricator.wikimedia.org/T427465#12273655 (10valerio.bozzolan) (@Ladsgroup Maybe last file F101159597 is private) [07:24:32] 06Traffic, 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Turnilo: X-Is-Browser filter doesn't work (always HTTP 500) - https://phabricator.wikimedia.org/T435152#12274079 (10JAllemandou) Ok, applying the same new field to the main turnilo config. Thanks @CDanis . [07:53:29] 06Traffic, 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Turnilo: X-Is-Browser filter doesn't work (always HTTP 500) - https://phabricator.wikimedia.org/T435152#12274164 (10JAllemandou) Actually I didn't realize but my patch changed the conf for both turnilo-next and turnilo. I just applied the he... [07:55:53] hey folks [07:56:01] as FYI I am working on cp5022 (depooled) for https://phabricator.wikimedia.org/T436386 [08:15:21] 06Traffic, 06Infrastructure-Foundations, 06ServiceOps, 06SRE, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12274234 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2003 for host urldownlo... [08:24:53] 06Traffic, 10Pywikibot, 06SRE, 10Wikidata, and 3 others: Pywikibot reports maxlag retry error on Wikidata - https://phabricator.wikimedia.org/T421642#12274279 (10Xqt) >>! In T421642#12271821, @JeanFred wrote: > As I wrote back in T244030 (which was {T242081} back then): >>>! In T244030#5842377, @JeanFred w... [08:50:50] 06Traffic, 06Infrastructure-Foundations, 06ServiceOps, 06SRE, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12274338 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2003 for host urldownloader... [09:00:23] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Alert on packet loss to core sites from POPs - https://phabricator.wikimedia.org/T435592#12274367 (10ayounsi) Should that be abandoned in favor of RPM probes alerting ? As the RPM probes are more accurate, and generic services monitoring ar... [09:01:41] 10netops, 06Infrastructure-Foundations, 10Observability-Metrics, 06SRE: Expand blackbox icmp probes to ping specific router interfaces/circuits - https://phabricator.wikimedia.org/T435855#12274375 (10ayounsi) Nice work !! >>! In T435855#12253094, @cmooney wrote: > The stats for the magru HE circuits are w... [09:35:13] 06Traffic, 06Data-Persistence, 06MediaWiki-Media-Platform-Team, 13Patch-For-Review, 07User-notice: Move thumbnail caching from upload cluster to text - https://phabricator.wikimedia.org/T427465#12274478 (10DerHexer) [[ https://meta.wikimedia.org/wiki/Meta:Central_notice_administrators | CentralNotice adm... [09:38:06] 06Traffic, 10Pywikibot, 06SRE, 10Wikidata, and 3 others: Pywikibot reports maxlag retry error on Wikidata - https://phabricator.wikimedia.org/T421642#12274488 (10Ladsgroup) It depends on what kind of load and what is causing the lag to go up. e.g. If it's a gigantic write, reducing reads on replicas won't... [10:25:43] 06Traffic, 06Data-Persistence, 06MediaWiki-Media-Platform-Team, 13Patch-For-Review, 07User-notice: Move thumbnail caching from upload cluster to text - https://phabricator.wikimedia.org/T427465#12274696 (10Ladsgroup) >>! In T427465#12274478, @DerHexer wrote: > [[ https://meta.wikimedia.org/wiki/Meta:Cent... [10:38:06] 06Traffic, 06Infrastructure-Foundations, 06ServiceOps, 06SRE, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12274739 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2003 for host urldownlo... [11:10:36] 06Traffic, 10Citoid, 06MediaWiki-API-Platform-Team, 06ServiceOps, 06Editing-team (FY2026-27 Q1 sprint 5): Consider adding a php interface over the citoid api in the Citoid extension - https://phabricator.wikimedia.org/T435916#12274891 (10Clement_Goubert) [11:15:50] 06Traffic, 06Infrastructure-Foundations, 06ServiceOps, 06SRE, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12274901 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2003 for host urldownloader... [11:26:30] 06Traffic, 06Infrastructure-Foundations, 06ServiceOps, 06SRE, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12274953 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2003 for host urldownlo... [12:01:45] 06Traffic, 10Citoid, 06MediaWiki-API-Platform-Team, 06ServiceOps, 06Editing-team (FY2026-27 Q1 sprint 5): Consider adding a php interface over the citoid api in the Citoid extension - https://phabricator.wikimedia.org/T435916#12275153 (10Mvolz) Yes, all out-going requests pass through url-downloader. The... [12:06:02] 06Traffic, 06Infrastructure-Foundations, 06ServiceOps, 06SRE, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12275173 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2003 for host urldownloader... [12:08:17] 06Traffic, 06SRE, 07Wikimedia-Incident: Wikipedia and Wiktionary pages returning "upstream connect error or disconnect/reset before headers" - https://phabricator.wikimedia.org/T436004#12275178 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF T436001 is now public with a report. Let's call... [12:08:27] 06Traffic, 06SRE, 07Wikimedia-Incident: Wikipedia and Wiktionary pages returning "upstream connect error or disconnect/reset before headers" - https://phabricator.wikimedia.org/T436004#12275184 (10Jdforrester-WMF) a:05Jdforrester-WMF→03None [12:15:20] 06Traffic, 06Infrastructure-Foundations, 06ServiceOps, 06SRE, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12275219 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2003 for host urldownlo... [12:22:45] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Create a 3rd Netflow instance in eqiad/codfw - https://phabricator.wikimedia.org/T436517#12275270 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by ayounsi@cumin1003 for host netflow1004.eqiad.wmnet with OS trixie [12:52:52] 10netops, 06Infrastructure-Foundations: Create a 3rd Netflow instance in eqiad/codfw - https://phabricator.wikimedia.org/T436517#12275429 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by ayounsi@cumin1003 for host netflow1004.eqiad.wmnet with OS trixie completed: - netflow1004 (**PASS**)... [12:57:58] 06Traffic, 06Infrastructure-Foundations, 06ServiceOps, 06SRE, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12275445 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2003 for host urldownloader... [13:15:48] 10netops, 06Infrastructure-Foundations: Upgrade netflow hosts to Trixie - https://phabricator.wikimedia.org/T424478#12275541 (10ayounsi) Fastnetmon .service file changed in Trixie, the process doesn't start anmymore: `name=trixie,counterexample netflow1004:~$ sudo systemctl status fastnetmon ● fastnetmon.serv... [13:16:54] 06Traffic, 06Infrastructure-Foundations, 06ServiceOps, 06SRE, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12275546 (10ops-monitoring-bot) VM urldownloader1003.wikimedia.org rebooted by jmm@cumin2003 with reason: expand RAM [13:23:56] 10netops, 06Infrastructure-Foundations: Upgrade netflow hosts to Trixie - https://phabricator.wikimedia.org/T424478#12275566 (10MoritzMuehlenhoff) That's because in Trixie fastnetmon runs under the fastnetmon system user (while as root in bookworm) and our /etc/fastnetmon.conf is only readable by root. Fixing... [13:24:30] 10netops, 06Infrastructure-Foundations, 06SRE: Missing series for BGP session_state from eqiad CRs since upgrade to 23.4R2-S8.7 - https://phabricator.wikimedia.org/T435909#12275570 (10ayounsi) Adding a 3rd netflow host in eqiad shuffled the targets around, it caused the data to be collected for a short while... [13:25:19] 06Traffic, 06Infrastructure-Foundations, 06ServiceOps, 06SRE, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12275571 (10ops-monitoring-bot) VM urldownloader1004.wikimedia.org rebooted by jmm@cumin2003 with reason: expand RAM [13:34:56] 06Traffic, 06Infrastructure-Foundations, 06ServiceOps, 06SRE, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12275628 (10ops-monitoring-bot) VM urldownloader2003.wikimedia.org rebooted by jmm@cumin2003 with reason: expand RAM [13:39:27] 06Traffic, 06Infrastructure-Foundations, 06ServiceOps, 06SRE, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12275636 (10ops-monitoring-bot) VM urldownloader2004.wikimedia.org rebooted by jmm@cumin2003 with reason: expand RAM [13:59:35] 10netops, 06Infrastructure-Foundations: Create a 3rd Netflow instance in eqiad/codfw - https://phabricator.wikimedia.org/T436517#12275711 (10ayounsi) Trixie instance is live in eqiad, unfortunately we're still missing metrics on some devices. [14:18:33] 06Traffic, 06Infrastructure-Foundations, 06ServiceOps, 06SRE, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12275821 (10MoritzMuehlenhoff) I've extended the RAM of the old nodes (to be repurposed as Bookworm nodes) also to 8... [15:04:23] 10netops, 06Infrastructure-Foundations: Upgrade netflow hosts to Trixie - https://phabricator.wikimedia.org/T424478#12276053 (10MoritzMuehlenhoff) [16:29:43] 06Traffic, 06DC-Ops, 10ops-drmrs: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet - https://phabricator.wikimedia.org/T431651#12276666 (10RobH) The ram landed yesterday, filing remote hands shortly to work on this host. [16:40:59] 06Traffic, 06DC-Ops, 10ops-drmrs: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet - https://phabricator.wikimedia.org/T431651#12276711 (10RobH) There is a 48 hour minimum window to have remote hands work on things without an urgency fee applied. I picked Friday with a note that the work can actual... [16:42:11] 06Traffic, 06DC-Ops, 10ops-drmrs: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet - https://phabricator.wikimedia.org/T431651#12276714 (10RobH) [16:46:36] 06Traffic, 10Maps (Kartographer): Geoshape rate limiting breaks the live maps - https://phabricator.wikimedia.org/T435741#12276751 (10TheDJ) >>! In T435741#12265641, @CDanis wrote: > I did a little looking at the cacheability angle with some simulations. Ideally if you add/remove/change only a few ids on a ma... [17:04:50] 06Traffic, 06SRE Observability, 13Patch-For-Review: Move wikimediastatus.net 301 to ncredir - https://phabricator.wikimedia.org/T419887#12276835 (10BCornwall) 05Open→03In progress p:05Triage→03High a:03BCornwall [17:05:37] 06Traffic, 06SRE Observability, 13Patch-For-Review: Redirect http(s?)://wikimediastatus.net to https://www.wikimediastatus.net - https://phabricator.wikimedia.org/T419887#12276839 (10BCornwall) [17:16:22] 06Traffic, 06SRE Observability, 13Patch-For-Review: Redirect http(s?)://wikimediastatus.net to https://www.wikimediastatus.net - https://phabricator.wikimedia.org/T419887#12276914 (10BCornwall) [19:49:27] 06Traffic, 06SRE Observability, 13Patch-For-Review: Redirect http(s?)://wikimediastatus.net to https://www.wikimediastatus.net - https://phabricator.wikimedia.org/T419887#12277638 (10BCornwall) MarkMonitor offers our edge case as a solution for $3,500 a year. Yikes. We'll be exploring other avenues - My reco...