[08:51:09] 10serviceops, 10MW-on-K8s: mw-debug-repl: Make it possible to trigger verbose logging in shell.php - https://phabricator.wikimedia.org/T344323 (10Urbanecm_WMF) [08:59:12] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: mw-debug-repl: Make it possible to trigger verbose logging in shell.php - https://phabricator.wikimedia.org/T344323 (10Urbanecm_WMF) [08:59:33] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: mw-debug-repl: Make it possible to trigger verbose logging in shell.php - https://phabricator.wikimedia.org/T344323 (10Urbanecm_WMF) [09:05:33] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: mw-debug-repl: Make it possible to trigger verbose logging in shell.php - https://phabricator.wikimedia.org/T344323 (10Joe) 05Open→03Resolved a:03Joe Patch merged, applied and I also verified it works. Thanks @Urbanecm_WMF :) [09:14:06] 10serviceops, 10Data Products, 10RESTbase Sunsetting, 10Code-Health-Objective, 10Patch-For-Review: Route to new AQS Knowledge Gaps endpoint - https://phabricator.wikimedia.org/T342213 (10hnowlan) >>! In T342213#9093461, @fkaelin wrote: > Question: once the machines are updated with https://gerrit.wikimed... [09:54:07] 10serviceops, 10Abstract Wikipedia team, 10Patch-For-Review, 10Service-deployment-requests: New Service Request memcached-wikifunctions - https://phabricator.wikimedia.org/T297815 (10Jdforrester-WMF) 05In progress→03Resolved a:03Jdforrester-WMF I'll call this Resolved for now; though no doubt we'll w... [10:14:12] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Reserve resources for system daemons on kubernetes nodes - https://phabricator.wikimedia.org/T277876 (10JMeybohm) I looked at the data collected from workers since yesterday and they are (to no surprise) well below what the GKE formula would spit out - so I trie... [14:26:38] 10serviceops, 10MW-on-K8s, 10Observability-Logging, 10Patch-For-Review: Some apache access logs are invalid json - https://phabricator.wikimedia.org/T340935 (10CodeReviewBot) oblivian merged https://gitlab.wikimedia.org/repos/sre/glogger/-/merge_requests/1 Introduce glogger [14:35:45] hello, I'd like to upgrade jwt-authorizer from v1.1.0 to v1.1.1 on the registry nodes. I'll proceed soon if there are no objections :) [14:39:30] 10serviceops, 10MW-on-K8s, 10MediaWiki-Platform-Team, 10MediaWiki-Shell, and 2 others: User logging in on mw-on-k8s triggers "RuntimeException: firejail is enabled, but cannot be found" - https://phabricator.wikimedia.org/T344223 (10dancy) Hi @Krinkle. Can you give me a status report on this ticket? [14:59:44] 10serviceops, 10RESTBase, 10Platform Team Initiatives (API Gateway), 10Platform Team Workboards (Platform Engineering Reliability): Add support in REST gateway to simplify edge logic - https://phabricator.wikimedia.org/T344358 (10hnowlan) [15:43:23] 10serviceops, 10collaboration-services, 10GitLab (CI & Job Runners): Standardize Debian package builds on GitLab CI - https://phabricator.wikimedia.org/T304491 (10dancy) [16:11:05] 10serviceops, 10MW-on-K8s, 10MediaWiki-Platform-Team, 10MediaWiki-Shell, and 2 others: User logging in on mw-on-k8s triggers "RuntimeException: firejail is enabled, but cannot be found" - https://phabricator.wikimedia.org/T344223 (10Krinkle) 05Open→03Resolved Fixed. {F37512360 height=200} [16:16:27] 10serviceops, 10MW-on-K8s, 10MediaWiki-Platform-Team, 10MediaWiki-Shell, and 2 others: User logging in on mw-on-k8s triggers "RuntimeException: firejail is enabled, but cannot be found" - https://phabricator.wikimedia.org/T344223 (10dancy) Thanks @Krinkle ! [16:23:38] Does anyone know a process for verifying a shellbox deployment on the staging cluster? https://wikitech.wikimedia.org/wiki/Shellbox#Deploying_a_new_version helpfully says "If you want a quick way to cycle through them, once you've tested how they perform on staging:" but does not seem to include any documentation of testing on staging. [16:23:41] Context is https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/914014 [16:24:36] * bd808 is ok with the answer being YOLO, but hopes for maybe better [16:26:15] Errr.. not 914014, but https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/949548 [16:29:43] 10serviceops, 10Scap, 10Release-Engineering-Team (Priority Backlog 📥): Re-imaged mw app servers can end up with missing l10n cache for old versions of MW needed for rollback - https://phabricator.wikimedia.org/T273334 (10thcipriani) 05Open→03Declined The move to mw-on-k8s should obviate the need for this... [16:39:25] 10serviceops, 10PyBal, 10SRE, 10Scap, and 3 others: High rate of errors and increased latency on uncached MediaWiki requests due to infrastructure outage - https://phabricator.wikimedia.org/T337497 (10thcipriani)