[06:37:38] 10serviceops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 9 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10Marostegui) @Ladsgroup During this operation, replication codfw -> eqiad is still active, so as there are codfw masters involved (even if codfw wi... [07:49:18] 10serviceops, 10MW-on-K8s, 10SRE, 10Release-Engineering-Team (Priority Backlog 📥): Automated validation of mediawiki-multiversion images - https://phabricator.wikimedia.org/T288629 (10JMeybohm) >>! In T288629#8808899, @dancy wrote: >>>! In T288629#8807158, @JMeybohm wrote: >> I don't see helm defaults bein... [11:28:24] 10serviceops, 10Shellbox, 10SyntaxHighlight, 10Patch-For-Review, 10User-bd808: Install pygments in Shellbox container with pip, not a Debian package - https://phabricator.wikimedia.org/T320848 (10akosiaris) >>! In T320848#8776934, @Legoktm wrote: > Could we get a +1 from someone in serviceops on the gene... [11:31:42] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover: Post March 2023 Datacenter Switchover Tasks - https://phabricator.wikimedia.org/T328907 (10Clement_Goubert) [11:31:50] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: sre.discovery.datacenter should support switching the active/passive services to the other datacenter - https://phabricator.wikimedia.org/T335364 (10Clement_Goubert) 05Open→03In progress p:05Triage→03Me... [12:12:00] 10serviceops, 10SRE: keyholder on inactive deployment server - https://phabricator.wikimedia.org/T335435 (10Clement_Goubert) I would like @MoritzMuehlenhoff and other serviceops (@Joe, @akosiaris ?) input on this. I think it's sound, but maybe in case of the main deployment server being down, we don't want to... [12:57:33] 10serviceops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 9 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10fgiunchedi) [13:00:46] 10serviceops, 10SRE: keyholder on inactive deployment server - https://phabricator.wikimedia.org/T335435 (10akosiaris) The motd states: ` While it is perfectly working, this is not the active deployment server. If you want to deploy software, you should /not/ do it from here; it will probably work, but the n... [13:03:15] 10serviceops, 10SRE: keyholder on inactive deployment server - https://phabricator.wikimedia.org/T335435 (10MoritzMuehlenhoff) Yeah, what Alex said. In addition, if we really want to prevent deployers from using the inactive servers, the better fix would be to have scap check/prevent this. [13:42:39] akosiaris: is it okay if multiple people are deploying k8s services at the same time or should we be going one at a time? [13:46:21] legoktm: as long as it's not the same service, that shpuld be fine I'd say [13:46:40] gotcha, ty :) [14:13:13] 10serviceops, 10ChangeProp, 10Content-Transform-Team-WIP, 10Page Content Service, and 3 others: Parsoid cache invalidation for mobile-sections seems not reliable - https://phabricator.wikimedia.org/T226931 (10Jaifroid) Just as a follow-up, the Kiwix English-language [[ https://download.kiwix.org/zim/wikivo... [14:14:55] jayme: can you remind me what the URL for hitting the staging cluster is? can't find it on wikitech [14:16:01] nvm, found it [14:16:02] will document [14:16:12] staging.svc.eqiad.wmnet [14:16:30] (just so you can confirm what you found) [14:18:16] yep, ty :D [14:19:47] https://wikitech.wikimedia.org/w/index.php?title=Kubernetes/Clusters&diff=prev&oldid=2072166 [14:20:14] <3 [14:34:53] legoktm: was that post merge test failure noise anything to worry about? All the "Shellbox server returned status code 500" look like they would be unrelated to what we were changing, but I haven't dug into the test suite at all. https://integration.wikimedia.org/ci/job/phpunit-coverage-php74-docker-publish/845/console [14:36:57] I think it's unrelated, most likely the custom stuff we do to get code coverage broke in a PHPUnit update: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/libs/Shellbox/+/refs/heads/master/tests/ClientServerTestCase.php [14:37:58] staging is updated, I'm waiting for the MW deploys to finish so I can live hack MW config to point at the staging cluster and just check that it all works [14:38:18] there were some other envoy and chart updates that got pulled in too [14:38:20] awesome :) [15:04:25] legoktm: sorry, was in a meeting [15:04:34] no worries [15:16:59] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Priority Backlog 📥): Automated validation of mediawiki-multiversion images - https://phabricator.wikimedia.org/T288629 (10dancy) >>! In T288629#8809671, @JMeybohm wrote: > Can you elaborate/point me to the discussion on how... [15:33:00] shellbox deployment to eqiad looks good, finishing up codfw now [15:47:00] 10serviceops, 10Shellbox, 10SyntaxHighlight, 10Patch-For-Review, 10User-bd808: Install pygments in Shellbox container with pip, not a Debian package - https://phabricator.wikimedia.org/T320848 (10Legoktm) The new `/srv/app/pygmentize` entrypoint is now in place, but that was also the first Shellbox deplo... [16:31:27] 10serviceops, 10Wikimedia Enterprise, 10Performance-Team (Radar), 10affects-Kiwix-and-openZIM: large amount of traffic to the action=parse API from MWOffliner - https://phabricator.wikimedia.org/T324866 (10daniel) The using the Enterprise streams is not a good fit (T329779), Kiwix could also start using th... [17:07:15] I armed the keyholder on deploy2002 based on your comments on the ticket [17:08:37] 10serviceops, 10SRE: keyholder on inactive deployment server - https://phabricator.wikimedia.org/T335435 (10Dzahn) Thanks all for the input. In that case.. I think all that was left to do here was to arm the keyholder again, after the server reboot. And I just did that above. Monitoring can stay as it is th... [17:09:48] 10serviceops, 10SRE: keyholder on inactive deployment server - https://phabricator.wikimedia.org/T335435 (10Dzahn) 05Open→03Resolved a:03Dzahn https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [17:35:13] 10serviceops, 10RESTbase Sunsetting, 10Parsoid (Tracking), 10Patch-For-Review: Enable WarmParsoidParserCache on all wikis - https://phabricator.wikimedia.org/T329366 (10jijiki) After chatting with @daniel, either #serviceops merges [[ https://gerrit.wikimedia.org/r/912929 | 912929 ]] on Tuesday, if we fee...