[00:07:13] 06serviceops: Regenerate UcfirstOverrides.php for PHP 7.4 -> 8.1 transition - https://phabricator.wikimedia.org/T372603#10094570 (10Scott_French) Alright, so there is //at least// one tricky bit to this: How do we run `generateUpperCharTable.php` on 8.1 without also installing 8.1 on maintenance hosts? Although... [07:02:23] 👋 hnowlan I updated this patch yesterday, are you available to take a look: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1066711 [07:03:47] ? [09:03:37] 06serviceops, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate charts to Calico Network Policies - https://phabricator.wikimedia.org/T359423#10095014 (10klausman) [09:12:18] 06serviceops, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate charts to Calico Network Policies - https://phabricator.wikimedia.org/T359423#10095032 (10klausman) [09:38:51] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10095106 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host w... [09:45:09] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10095118 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host w... [09:52:27] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10095141 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host w... [10:14:29] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10095259 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host w... [10:24:58] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10095281 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikik... [10:29:56] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10095291 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikik... [10:30:50] 06serviceops, 06Release-Engineering-Team: scap fails to create /srv/mediawiki-staging/php-1.43.0-wmf.20/cache/l10n - https://phabricator.wikimedia.org/T373425 (10hashar) 03NEW [10:31:01] 06serviceops, 06Release-Engineering-Team: scap fails to create /srv/mediawiki-staging/php-1.43.0-wmf.20/cache/l10n - https://phabricator.wikimedia.org/T373425#10095304 (10hashar) p:05Triage→03Unbreak! [10:33:36] 06serviceops, 06Release-Engineering-Team: scap fails to create /srv/mediawiki-staging/php-1.43.0-wmf.20/cache/l10n - https://phabricator.wikimedia.org/T373425#10095311 (10hashar) `/srv/mediawiki-staging/php-1.43.0-wmf.20/cache/` is owned by `mwpresync:deployment` with permissions `rwxrwsr-x` / `2775`. That di... [10:37:32] 06serviceops, 10decommission-hardware: decommission kafka-main2001.codfw.wmnet - https://phabricator.wikimedia.org/T373428 (10JMeybohm) 03NEW [10:37:44] 06serviceops, 10decommission-hardware: decommission kafka-main2001.codfw.wmnet - https://phabricator.wikimedia.org/T373428#10095355 (10JMeybohm) [10:37:46] 06serviceops, 13Patch-For-Review: kafka-main200[6789] and kafka-main2010 implementation tracking - https://phabricator.wikimedia.org/T363210#10095356 (10JMeybohm) [10:41:30] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10095368 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikik... [10:42:53] !log Running homer 'lsw1-a5-codfw*' commit 'T372878' [10:47:19] 06serviceops, 06Release-Engineering-Team: scap fails to create /srv/mediawiki-staging/php-1.43.0-wmf.20/cache/l10n - https://phabricator.wikimedia.org/T373425#10095375 (10hashar) 05Open→03Resolved a:03hashar `php-1.43.0-wmf.18/cache/` and `php-1.43.0-wmf.19/cache/` were created at 3:02am UTC in the l... [10:49:31] 06serviceops, 06Release-Engineering-Team: scap fails to create /srv/mediawiki-staging/php-1.43.0-wmf.20/cache/l10n - https://phabricator.wikimedia.org/T373425#10095392 (10hashar) [11:01:21] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10095415 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikik... [11:14:09] 06serviceops, 10decommission-hardware, 13Patch-For-Review: decommission kafka-main2001.codfw.wmnet - https://phabricator.wikimedia.org/T373428#10095445 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jayme@cumin1002 for hosts: `kafka-main2001.codfw.wmnet` - kafka-main2001.codfw.wmnet (**P... [11:19:11] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 13Patch-For-Review: decommission kafka-main2001.codfw.wmnet - https://phabricator.wikimedia.org/T373428#10095452 (10JMeybohm) [12:01:03] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10095590 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by akosiaris@cumin1002 from mw2292 to... [12:01:51] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10095592 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host... [12:24:01] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 13Patch-For-Review: decommission kafka-main2001.codfw.wmnet - https://phabricator.wikimedia.org/T373428#10095626 (10Jhancock.wm) a:03Jhancock.wm [12:47:06] 06serviceops: Unexpected helmfile changes when attempting a k8s deployment for a miscweb site - https://phabricator.wikimedia.org/T372825#10095735 (10elukey) @sbassett Hi! Please subscribe to the ops mailing list so you can get notified by these changes, usually we post a message there to warn users. In this cas... [12:56:28] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10095765 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host wiki... [14:02:51] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, 07Kubernetes: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T373401#10096117 (10Jhancock.wm) 05Open→03Resolved [14:10:14] nemo-yiannis: I will try to get to that today [14:10:23] 👍 [14:13:30] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 13Patch-For-Review: decommission kafka-main2001.codfw.wmnet - https://phabricator.wikimedia.org/T373428#10096188 (10Jhancock.wm) 05Open→03Resolved [14:32:56] hello folks, if everything is good I'd depool wikikube-ctrl2003 and run the provision cookbook [14:36:34] elukey: what's up with it? [14:37:23] claime: when the provision cookbook was run there was a bug, there is the high chance that not all BIOS parameters were set correctly etc.. (all info in https://phabricator.wikimedia.org/T371132). So I'd need to re-run the cookbook, but it causes the host to reboot [14:37:47] so I need to depool etc.. [14:38:01] Right yeah [14:38:22] I think you can go ahead, jayme don't do the vlan move for the other wikikube-ctrl please :D [14:38:47] eheh, nono. I had the reboot in mind for today :) [14:40:05] all right doing it :) [14:41:57] so I depooled it and downtimed it, anything else worth to do on the host itself? [14:42:54] elukey: no, thats sufficient [14:42:56] I don't think so [14:44:21] all right starting [15:02:35] all done! [15:02:39] repolled [15:02:42] *repooled [15:04:03] <3 [15:14:34] hnowlan: o/ you won't believe it but I'd need to build and deploy thumbor again :D [15:14:47] if you are free I'll send some patches to you in a bit :D [15:14:49] lolol [15:14:52] sounds good, more updates? [15:15:03] yep https://phabricator.wikimedia.org/T373363 [15:15:26] I am not sure if this is a big concern or not, buuut better be safe :D [15:15:48] yeah fair enough [15:20:25] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10096550 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from kubernetes20... [15:22:59] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10096557 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wik... [15:32:40] 06serviceops: Unexpected helmfile changes when attempting a k8s deployment for a miscweb site - https://phabricator.wikimedia.org/T372825#10096596 (10sbassett) 05Open→03Resolved p:05Triage→03Low a:03sbassett >>! In T372825#10095735, @elukey wrote: > @sbassett Hi! Please subscribe to the ops mailing... [15:43:23] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10096619 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by akosiaris@cumin1002 from mw2293 to... [15:44:01] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10096620 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host... [15:46:35] there you go https://gerrit.wikimedia.org/r/c/operations/software/thumbor-plugins/+/1067379 [15:46:41] verified that the package is updated [16:05:39] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10096726 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikub... [16:14:29] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, 07Kubernetes: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T373457 (10kamila) 03NEW [16:16:40] 06serviceops, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate charts to Calico Network Policies - https://phabricator.wikimedia.org/T359423#10096821 (10akosiaris) [17:07:21] 06serviceops, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate charts to Calico Network Policies - https://phabricator.wikimedia.org/T359423#10097088 (10BTullis) [17:09:15] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10097092 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host wiki... [21:13:14] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10098130 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by swfrench@cumin2002 from kubernetes... [21:15:08] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10098132 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by swfrench@cumin2002 for host w... [22:02:06] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10098199 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by swfrench@cumin2002 for host wikik... [22:08:58] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, 07Kubernetes: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T373491 (10Scott_French) 03NEW [22:52:00] 06serviceops, 07Essential-Work, 13Patch-For-Review, 07Wikimedia-production-error: TypeError: Argument 4 passed to Wikimedia\Parsoid\Utils\Title::__construct() must be of the type string, null given, called in /srv/mediawiki/php-1.42.0-wmf.15/vendor/wikimed... - https://phabricator.wikimedia.org/T356024#10098347 [22:56:02] 06serviceops, 07Essential-Work, 13Patch-For-Review, 07Wikimedia-production-error: TypeError: Argument 4 passed to Wikimedia\Parsoid\Utils\Title::__construct() must be of the type string, null given, called in /srv/mediawiki/php-1.42.0-wmf.15/vendor/wikime... - https://phabricator.wikimedia.org/T356024#10098367