[08:30:24] 10serviceops, 10SRE: Clean up old Docker images on deneb - https://phabricator.wikimedia.org/T287222 (10JMeybohm) There now is a prune timer, see T304644 [08:34:24] 10serviceops, 10GitLab (Infrastructure): GitLab minor version upgrade: 14.9.x - https://phabricator.wikimedia.org/T304622 (10Jelto) @Arnoldokoth and I will do the upgrade of GitLab + Runners on Monday after 4pm UTC. [09:06:16] jayme: o/ going to merge the change to clean up the bullseye/overlay configs (puppet disabled, just one node at the time) [09:06:38] elukey: ack [09:27:07] mmm I am not getting the V:+2 from jenkins in https://gerrit.wikimedia.org/r/c/operations/puppet/+/773520 [09:31:08] elukey: it seems to not have started a job [09:32:28] https://integration.wikimedia.org/zuul/ looks relatively clean, I don't see problems (or clear ones) [09:55:10] ah snap of course kubernetes[12]00[1-4] are still on stretch [09:55:17] so puppet tries to apply new settings [09:55:26] I am going to revert their settings for the moment [09:55:53] or better, they didn't have overrides [09:56:12] otherwise we can merge the decom patch that akosiaris created :D [10:09:06] jayme: if you have a min https://gerrit.wikimedia.org/r/c/operations/puppet/+/773751/ [10:12:32] elukey: +1ed [10:12:58] thanks! [10:13:40] I am going to remove myself as assignee of the task, since there are only the decom tasks to do and Alex has already a patch for them [10:14:58] sgtm [10:16:06] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Move kubernetes workers to bullseye and docker to overlayfs - https://phabricator.wikimedia.org/T300744 (10elukey) a:05elukeyโ†’03None Removing myself as assignee of the task, all the reimages completed. The remaining steps are: * Merge... [10:16:15] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Move kubernetes workers to bullseye and docker to overlayfs - https://phabricator.wikimedia.org/T300744 (10elukey) [10:19:15] <_joe_> great job folks :) [16:23:50] 10serviceops, 10SRE, 10Wikimedia-Etherpad: Etherpads corrupted - https://phabricator.wikimedia.org/T304005 (10Dzahn) a:03Dzahn [16:24:06] 10serviceops, 10SRE, 10ops-eqiad: mc1053 PS redundancy alert - https://phabricator.wikimedia.org/T304477 (10Cmjohnson) 05Openโ†’03Resolved a:03Cmjohnson Fixed [16:43:28] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week): Need a service account on deploy servers - https://phabricator.wikimedia.org/T303857 (10jcrespo) Normally this kind of tasks would be routed by the person in clinic duty, but I just happened to see him g... [16:45:21] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10SRE-Access-Requests, 10Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week): Need a service account on deploy servers - https://phabricator.wikimedia.org/T303857 (10Dzahn) [16:48:34] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10SRE-Access-Requests, 10Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week): Need a service account on deploy servers - https://phabricator.wikimedia.org/T303857 (10Joe) a:03Joe I'll take care of this, as I assume it's not urgent to be completed before... [17:32:34] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10SRE-Access-Requests, 10Release-Engineering-Team (๐Ÿš‚๐Ÿงช Trainsperiment Week): Need a service account on deploy servers - https://phabricator.wikimedia.org/T303857 (10dancy) Hi @Joe. This request is not urgent so it can wait until next week. The plan is f... [20:42:01] 10serviceops, 10SRE, 10Wikimedia-Etherpad: Etherpads corrupted - https://phabricator.wikimedia.org/T304005 (10Dzahn) Hello @Zapipedia-WMF so I think what happened here is, the first case was likely caused by me doing the maintenance because I had the service running from 2 servers at the same time. I knew... [21:01:52] 10serviceops, 10SRE, 10Wikimedia-Etherpad: Etherpads corrupted - https://phabricator.wikimedia.org/T304005 (10Dzahn) I documented this at https://wikitech.wikimedia.org/wiki/Etherpad.wikimedia.org#Restoring_a_pad_to_a_previous_revision [21:03:02] 10serviceops, 10SRE, 10Wikimedia-Etherpad: Etherpads corrupted - https://phabricator.wikimedia.org/T304005 (10Dzahn) 05Openโ†’03Resolved claiming resolved, let me know if you agree [21:06:28] 10serviceops, 10Data-Engineering, 10SRE, 10Traffic, and 2 others: Disable GeoIP Legacy Download - https://phabricator.wikimedia.org/T303464 (10Dzahn) a:03Dzahn