[00:11:25] 06serviceops, 10MW-on-K8s: Support passing env variables to maintenance scripts in mwscript-k8s - https://phabricator.wikimedia.org/T380925#10360192 (10RLazarus) >>! In T380925#10360022, @Tgr wrote: > You can definitely break maintenance scripts by changing the right env variable, but as long as the user needs... [08:30:48] 06serviceops, 10MediaWiki-extensions-PropertySuggester, 10MW-on-K8s, 10Wikidata, and 3 others: [PS] Update PropertySuggester update process for mwscript-k8s - https://phabricator.wikimedia.org/T376604#10360558 (10ArthurTaylor) a:05ArthurTaylor→03hoo [08:51:27] 06serviceops, 06Infrastructure-Foundations, 06SRE: Clean up the Docker Registry catalog and Swift storage from old images - https://phabricator.wikimedia.org/T375645#10360573 (10elukey) 05Open→03Declined The K8s SIG reviewed this proposal and for the moment it was decided not to proceed with anything... [09:34:47] 06serviceops, 10Deployments, 06Release-Engineering-Team, 07Wikimedia-production-error: httpb fails upon deployment of 1.44.0-wmf.5 - https://phabricator.wikimedia.org/T380958 (10hashar) 03NEW p:05Triage→03Medium [09:57:21] 06serviceops, 10Deployments, 06Release-Engineering-Team, 07Wikimedia-production-error: httpb fails upon deployment of 1.44.0-wmf.5 - https://phabricator.wikimedia.org/T380958#10360675 (10hashar) Some logs from php-fpm being restarted | Time | type | level | message | program |------|------|-------|--------... [10:16:30] 06serviceops, 10Deployments, 06Release-Engineering-Team, 07Wikimedia-production-error: httpb fails upon deployment of 1.44.0-wmf.5 - https://phabricator.wikimedia.org/T380958#10360717 (10akosiaris) This is probably what you are looking for ` [2024-11-27T09:25:11.887Z] "GET /wiki/Wikidata:Main_Page HTTP/1.... [10:34:18] 06serviceops, 10Deployments, 06Release-Engineering-Team, 07Wikimedia-production-error: httpb fails upon deployment of 1.44.0-wmf.5 - https://phabricator.wikimedia.org/T380958#10360770 (10Lucas_Werkmeister_WMDE) FWIW, I had two similar errors on Monday, no idea if the logs are still around. [IRC](https://wm... [10:37:13] 06serviceops, 10MediaWiki-extensions-PropertySuggester, 10MW-on-K8s, 10Wikidata, and 3 others: [PS] Update PropertySuggester update process for mwscript-k8s - https://phabricator.wikimedia.org/T376604#10360786 (10ArthurTaylor) Updated the [docs](https://wikitech.wikimedia.org/wiki/WMDE/Wikidata/PropertySug... [10:46:05] 06serviceops, 10Deployments, 06Release-Engineering-Team, 07Wikimedia-production-error: httpb fails upon deployment of 1.44.0-wmf.5 - https://phabricator.wikimedia.org/T380958#10360825 (10hashar) > How many times has it been reproduced up to now? I had one yesterday when promoting group 0 which came from b... [10:48:51] 06serviceops, 10Deployments, 06Release-Engineering-Team, 07Wikimedia-production-error: httpb fails upon deployment of 1.44.0-wmf.5 - https://phabricator.wikimedia.org/T380958#10360844 (10hashar) [10:50:02] 06serviceops, 10Deployments, 06Release-Engineering-Team, 07Wikimedia-production-error: httpb fails upon deployment of 1.44.0-wmf.5 - https://phabricator.wikimedia.org/T380958#10360861 (10hashar) I have removed this from the list of train subtasks, it only got there cause I have used {nav Create Subtask} to... [11:14:05] 06serviceops, 10Deployments, 06Release-Engineering-Team, 07Wikimedia-production-error: httpb fails upon deployment of 1.44.0-wmf.5 - https://phabricator.wikimedia.org/T380958#10360917 (10Tgr) Happened twice to me on Monday: https://pastebin.com/yV7PLW5P https://pastebin.com/H86xifkd [13:53:48] 06serviceops, 10Deployments, 06Release-Engineering-Team, 07Wikimedia-production-error: httpb fails upon deployment of 1.44.0-wmf.5 - https://phabricator.wikimedia.org/T380958#10361543 (10akosiaris) This [grafana explore panel](https://w.wiki/CDKe) aligns pretty well with the errors in the last 3 days. {F... [14:25:13] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes: Update Kubernetes clusters to >1.25 - https://phabricator.wikimedia.org/T341984#10361685 (10Jelto) [14:27:33] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes: Update Kubernetes clusters to >1.25 - https://phabricator.wikimedia.org/T341984#10361695 (10Jelto) [14:30:06] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes: Update Kubernetes clusters to >1.25 - https://phabricator.wikimedia.org/T341984#10361722 (10Jelto) [14:33:09] 06serviceops, 10Deployments, 06Release-Engineering-Team, 07Wikimedia-production-error: httpb fails upon deployment of 1.44.0-wmf.5 - https://phabricator.wikimedia.org/T380958#10361743 (10Aklapper) >>! In T380958#10360717, @akosiaris wrote: > How many times has it been reproduced up to now? I also experienc... [14:52:21] 06serviceops, 06Infrastructure-Foundations, 10netops, 10Prod-Kubernetes: WikiKube clusters close to exhausting Calico IPPool allocations - https://phabricator.wikimedia.org/T375845#10361841 (10JMeybohm) We're not expecting any more replacements/expansions for wikikube this FY. So we can switch to the `/17`... [15:31:27] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes: Update Kubernetes clusters to >1.25 - https://phabricator.wikimedia.org/T341984#10362091 (10JMeybohm) [15:37:28] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes: Update Kubernetes clusters to >1.25 - https://phabricator.wikimedia.org/T341984#10362130 (10JMeybohm) [16:06:25] 06serviceops, 10MW-on-K8s: Support passing env variables to maintenance scripts in mwscript-k8s - https://phabricator.wikimedia.org/T380925#10362255 (10Urbanecm_WMF) >>! In T380925#10360022, @Tgr wrote: > (and mwscript also lets you run ad hoc PHP files). For now. In mwscript-k8s, you can only run //deployed/... [16:17:20] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#10362280 (10Urbanecm_WMF) Usually, I find the output of `kubectl get job` unhelpful. For example, for T379146, I just executed `userOptions.php` with various arguments. When I req... [16:25:07] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#10362360 (10Urbanecm_WMF) Also... > mw-script.codfw.4hzzmsrm 0/1 32d 32d shell.php I apparently managed to create a hanging `shell.php` job that is waitin... [17:14:13] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#10362702 (10Clement_Goubert) >>! In T341553#10362280, @Urbanecm_WMF wrote: > Usually, I find the output of `kubectl get job` unhelpful. For example, for T379146, I just executed `... [17:51:25] 06serviceops: kafka-main100[6789] and kafka-main1010 implementation tracking - https://phabricator.wikimedia.org/T363214#10362841 (10jijiki) [18:03:18] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#10362865 (10RLazarus) Thanks @Clement_Goubert! Yeah, `--sort-by=.metadata.creationTimestamp` is my go-to for ordering. For pulling out specific details like args, there are two a... [18:11:22] 06serviceops, 10MW-on-K8s: Support passing env variables to maintenance scripts in mwscript-k8s - https://phabricator.wikimedia.org/T380925#10362904 (10Tgr) >>! In T380925#10362255, @Urbanecm_WMF wrote: > In mwscript-k8s, you can only run //deployed// files. I never tried, but I assume you can just use `--fil... [18:14:25] 06serviceops, 10MW-on-K8s: Support bringing text files into the container for one-off maintenance scripts - https://phabricator.wikimedia.org/T376230#10362908 (10Urbanecm_WMF) Seems to work perfectly now, thanks! ` [urbanecm@deploy2002 ~]$ cat > users.txt MediaWiki message delivery [urbanecm@deploy2002 ~]... [18:37:05] 06serviceops, 10MW-on-K8s: Support passing env variables to maintenance scripts in mwscript-k8s - https://phabricator.wikimedia.org/T380925#10363000 (10Urbanecm_WMF) >>! In T380925#10362904, @Tgr wrote: >>>! In T380925#10362255, @Urbanecm_WMF wrote: >> In mwscript-k8s, you can only run //deployed// files. > >... [18:55:27] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#10363064 (10Urbanecm_WMF) Thanks both! This is helpful, but it will take some time to get accustomed with. >>! In T341553#10362865, @RLazarus wrote: >> I apparently managed to c...