[05:30:30] 10Traffic, 10PyBal, 10SRE, 10Scap, and 3 others: High rate of errors and increased latency on uncached MediaWiki requests due to infrastructure outage - https://phabricator.wikimedia.org/T337497 (10Joe) 05Open→03Resolved a:03Joe The problem that caused this outage has been fixed. [05:50:49] 10Traffic: Q1:unified decommission task for old esams hosts (knams migration) - https://phabricator.wikimedia.org/T344363 (10MoritzMuehlenhoff) [05:53:42] 10Traffic: Q1:unified decommission task for old esams hosts (knams migration) - https://phabricator.wikimedia.org/T344363 (10MoritzMuehlenhoff) [08:41:06] 10Traffic, 10Patch-For-Review: Q1:unified decommission task for old esams hosts (knams migration) - https://phabricator.wikimedia.org/T344363 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: `ganeti3001.esams.wmnet` - ganeti3001.esams.wmnet (**FAIL**) - Downtimed h... [08:51:08] 10Traffic, 10Patch-For-Review: Q1:unified decommission task for old esams hosts (knams migration) - https://phabricator.wikimedia.org/T344363 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: `ganeti3002.esams.wmnet` - ganeti3002.esams.wmnet (**FAIL**) - Downtimed h... [08:56:55] 10Traffic, 10Patch-For-Review: Q1:unified decommission task for old esams hosts (knams migration) - https://phabricator.wikimedia.org/T344363 (10MoritzMuehlenhoff) [09:03:56] 10Traffic, 10Patch-For-Review: Q1:unified decommission task for old esams hosts (knams migration) - https://phabricator.wikimedia.org/T344363 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: `ganeti3003.esams.wmnet` - ganeti3003.esams.wmnet (**FAIL**) - Downtimed h... [10:16:47] 10Traffic: Q1:unified decommission task for old esams hosts (knams migration) - https://phabricator.wikimedia.org/T344363 (10Fabfur) [11:42:35] 10Traffic, 10Lingua Libre: Network issue between LinguaLibre and Wikimedia Commons - https://phabricator.wikimedia.org/T344421 (10Tgr) [14:14:29] 10Acme-chief, 10Traffic, 10SRE: acme-chief should support debian bookworm - https://phabricator.wikimedia.org/T344330 (10Vgutierrez) [14:16:32] 10Traffic, 10SRE, 10SRE-swift-storage, 10MediaWiki-Platform-Team (Radar), and 2 others: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10Krinkle) [14:46:22] 10Traffic, 10Lingua Libre, 10SRE: Network issue between LinguaLibre and Wikimedia Commons - https://phabricator.wikimedia.org/T344421 (10Yug) Given the bug is persisting and preventing loggin, we may want to use the sitenotice to gently announce a pause in contributions / log in. [14:49:40] 10Traffic, 10Lingua Libre, 10SRE: Network issue between LinguaLibre and Wikimedia Commons - https://phabricator.wikimedia.org/T344421 (10Joe) It would be useful if you could follow https://wikitech.wikimedia.org/wiki/Reporting_a_connectivity_issue to give us a bit more details to go by. [14:51:34] 10Traffic, 10Lingua Libre, 10SRE: Network issue between LinguaLibre and Wikimedia Commons - https://phabricator.wikimedia.org/T344421 (10ayounsi) Hi, see https://wikitech.wikimedia.org/wiki/Reporting_a_connectivity_issue for more troubleshooting commands, but to start with could you provide the output of: `... [16:03:05] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10BCornwall) [19:24:48] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10BCornwall) [20:12:58] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10BCornwall)