[05:29:08] 06serviceops, 10MW-on-K8s, 10Shellbox, 10SRE-swift-storage: Support large files in Shellbox - https://phabricator.wikimedia.org/T292322#10090462 (10Joe) For the record, the reason we wanted to support large file uploads was not to worsen the performance of upload-by-url, which has since been fixed by makin... [06:20:57] 06serviceops, 06DBA, 06SRE, 10MediaWiki-Platform-Team (Radar), 10Sustainability (Incident Followup): In the aftermath of T370304: Brainstorming of short- and medium-term observability / quality-of-life production changes - https://phabricator.wikimedia.org/T372943#10090530 (10ABran-WMF) p:05High→03Med... [06:21:04] 06serviceops, 06DBA, 06SRE, 10MediaWiki-Platform-Team (Radar), 10Sustainability (Incident Followup): In the aftermath of T370304: Brainstorming of short- and medium-term observability / quality-of-life production changes - https://phabricator.wikimedia.org/T372943#10090532 (10ABran-WMF) p:05Medium→03H... [09:51:47] 06serviceops: deploy1003 implementation tracking - https://phabricator.wikimedia.org/T364417#10091174 (10Clement_Goubert) >>! In T364417#10089240, @jwang wrote: > Hi, I failed to ssh deployment.eqiad.wmnet. The message I got is `deployment.eqiad.wmnet: Permission denied (publickey)`. I was able to ssh a cou... [09:55:18] 👋 hnowlan, heads up I am deploying the caching changes on mobileapps [09:57:56] nemo-yiannis: ack [10:01:28] failed with a bunch of: `.ipBlock.cidr: Invalid value: "restbase2021-a.codfw.wmnet/32` [10:01:46] does it require the IPs one by one / [10:02:41] 06serviceops, 06Content-Transform-Team: Failed mobileapps deployment - https://phabricator.wikimedia.org/T373314 (10Jgiannelos) 03NEW [10:03:42] hnowlan: ^ [10:05:49] huh I thought this was solved in the cassandra module [10:09:10] nemo-yiannis: oh, no - the hosts all need to be listed as IP addresses [10:09:39] no worries, i am sending a patch now [10:31:03] 06serviceops, 06Content-Transform-Team, 06Content-Transform-Team-WIP, 13Patch-For-Review: Failed mobileapps deployment - https://phabricator.wikimedia.org/T373314#10091291 (10Jgiannelos) [10:32:54] 06serviceops, 10iPoid-Service (iPoid 1.0), 10Trust and Safety Product Sprint (Sprint Koto (July 15 - July 26)): Define service level indicators and service level objectives - https://phabricator.wikimedia.org/T348935#10091294 (10kostajh) I would suggest we delay this exercise until #ipoid-opensearch work is... [10:33:10] 06serviceops, 10iPoid-Service (iPoid 1.0), 03Trust and Safety Product Sprint: Define service level indicators and service level objectives - https://phabricator.wikimedia.org/T348935#10091295 (10kostajh) [12:53:32] 06serviceops, 06cloud-services-team, 06Infrastructure-Foundations, 10wikitech.wikimedia.org: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10091775 (10akosiaris) >>! In T371592#10081906, @bd808 wrote: >>>! In T371592#10081520, @Andrew wrote: >> Will this be... [13:36:09] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10091977 (10Clement_Goubert) 05Open→03In progress [13:53:22] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10092081 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host w... [14:02:51] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10092094 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host w... [14:08:11] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10092103 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host w... [14:17:00] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10092168 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host w... [14:17:21] 06serviceops, 06Content-Transform-Team-WIP, 13Patch-For-Review: Failed mobileapps deployment - https://phabricator.wikimedia.org/T373314#10092169 (10MSantos) [14:19:04] 06serviceops, 10MW-on-K8s, 10TimedMediaHandler, 13Patch-For-Review, 07Video: Move video transcoding to use Shellbox - https://phabricator.wikimedia.org/T356241#10092124 (10hnowlan) 05Open→03Resolved a:03hnowlan TimedMediaHandler now uses shellbox by default. [14:25:18] 06serviceops, 06Content-Transform-Team-WIP, 07Essential-Work, 13Patch-For-Review: Failed mobileapps deployment - https://phabricator.wikimedia.org/T373314#10092222 (10MSantos) [14:39:54] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10092313 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikik... [14:44:39] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10092328 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host w... [14:46:35] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10092348 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikik... [14:54:19] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10092370 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikik... [15:00:24] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10092408 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikik... [15:07:32] 06serviceops, 06Content-Transform-Team-WIP, 10RESTBase Sunsetting, 07Essential-Work, 13Patch-For-Review: Failed mobileapps deployment - https://phabricator.wikimedia.org/T373314#10092440 (10Jgiannelos) [15:07:50] 06serviceops, 06Content-Transform-Team-WIP, 10RESTBase Sunsetting, 07Essential-Work, 13Patch-For-Review: Failed mobileapps deployment - https://phabricator.wikimedia.org/T373314#10092447 (10Jgiannelos) a:03Jgiannelos [15:28:59] hello folks! [15:29:27] I'd need to run the provision cookbook for wikikube-ctrl2003 (see https://phabricator.wikimedia.org/T371132) and the node gets rebooted [15:29:45] I recall that somebody told me to just go ahead when a deployment wasn't scheduled etc.. [15:30:04] is it still the case? [15:31:21] elukey: I think I said that. Still true as long as you depool and downtime ofc [15:31:56] jayme: ack yes yes :) [15:32:07] ok thanks! Will try to do it tomorrow [15:32:31] 06serviceops, 10LPL Essential, 10MinT, 10Community Wishlist (Translations), 10Community-Tech (Gray Fox (Aug 26 - Sept 6)): Caching service request for MinT - https://phabricator.wikimedia.org/T370755#10092675 (10JWheeler-WMF) [15:32:49] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10092677 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikik... [15:41:13] 06serviceops, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes: Migrate Search Platform-owned helm charts to Calico Network Policies - https://phabricator.wikimedia.org/T373195#10092757 (10Gehel) p:05Triage→03Medium [16:00:24] 06serviceops, 10MW-on-K8s, 10TimedMediaHandler, 13Patch-For-Review, 07Video: Move video transcoding to use Shellbox - https://phabricator.wikimedia.org/T356241#10092838 (10Kappakayala) Awesome! Thanks for taking care of this. [16:01:33] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10092863 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host w... [16:01:54] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10092864 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikik... [16:05:21] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10092871 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host w... [16:10:59] !log homer 'cr*codfw*' commit 'T372878' [16:20:40] 06serviceops, 06cloud-services-team, 06Infrastructure-Foundations, 10wikitech.wikimedia.org: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10092958 (10bd808) >>! In T371592#10091775, @akosiaris wrote: >>>! In T371592#10081906, @bd808 wrote: >> This is block... [16:52:55] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10093070 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikik... [16:53:01] 06serviceops: deploy1003 implementation tracking - https://phabricator.wikimedia.org/T364417#10093071 (10Dzahn) Because users have been asking about the changed host key, I dumped the fingerprints for deploy1003 on https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/deploy1003.eqiad.wmnet It didn't ha... [16:58:51] 06serviceops, 10Scap, 13Patch-For-Review, 10Release-Engineering-Team (Priority Backlog 📥), 10Sustainability (Incident Followup): scap should check if it is running within a tmux/screen - https://phabricator.wikimedia.org/T361724#10093111 (10dancy) 05Open→03Resolved Handled in scap 4.100.0 which w... [17:49:42] 06serviceops: deploy1003 implementation tracking - https://phabricator.wikimedia.org/T364417#10093438 (10jwang) > Because users have been asking about the changed host key, Hi @Dzahn, thanks for the reply. Is it something I need to update on my end? If so, could you provide more instructions on how to do th... [17:56:37] 06serviceops: deploy1003 implementation tracking - https://phabricator.wikimedia.org/T364417#10093466 (10Dzahn) @jwang My comment was a general comment about the new deployment server and the replacement process. It shouldn't be related to your access question. It wouldn't manifest as "Permission denied (pu... [17:56:50] 06serviceops: deploy1003 implementation tracking - https://phabricator.wikimedia.org/T364417#10093464 (10jwang) @Clement_Goubert, thanks for the reply. I have created a ticket about it: T373379. [18:18:18] 06serviceops, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate charts to Calico Network Policies - https://phabricator.wikimedia.org/T359423#10093537 (10Scott_French) [18:37:58] 06serviceops, 10Scap, 13Patch-For-Review, 10Release-Engineering-Team (Priority Backlog 📥), 10Sustainability (Incident Followup): scap should check if it is running within a tmux/screen - https://phabricator.wikimedia.org/T361724#10093613 (10dancy) 05Resolved→03Open [18:40:03] 06serviceops: deploy1003 implementation tracking - https://phabricator.wikimedia.org/T364417#10093621 (10jwang) Thanks @Dzahn, I guess I mixed up with the other servers. I will create a request for the `deployment` group. [19:05:48] 06serviceops, 10Scap, 13Patch-For-Review, 10Release-Engineering-Team (Priority Backlog 📥), 10Sustainability (Incident Followup): scap should check if it is running within a tmux/screen - https://phabricator.wikimedia.org/T361724#10093733 (10dancy) [19:13:43] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10093775 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from kubernetes20... [19:14:48] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10093792 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wik... [19:22:14] 06serviceops, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Prepare WMF PHP 8.1 packages for Bullseye - https://phabricator.wikimedia.org/T372507#10093823 (10Scott_French) One last test and point-of-note: The suite of packages in T372507#10082651 //appear// to be co-installable with their 7.4 equiv... [19:59:55] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10093998 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikub... [21:09:53] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10094187 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by swfrench@cumin2002 from kubernetes... [21:17:25] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10094193 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by swfrench@cumin2002 for host w... [21:33:11] 06serviceops, 10Charts, 10Wikimedia-Extension-setup, 07Epic, 07Wikimedia-extension-review-queue: Epic: Deploy Chart extension in production - https://phabricator.wikimedia.org/T369944#10094237 (10Catrope) [22:05:56] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10094339 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by swfrench@cumin2002 for host wikik...