[06:22:09] 10serviceops, 10Infrastructure-Foundations, 10SRE-tools, 10Wikimedia-Mailing-lists: Support services VIPs with not marked as VIP in Netbox - https://phabricator.wikimedia.org/T295793 (10ayounsi) [07:23:37] 10serviceops, 10Infrastructure-Foundations, 10SRE-tools, 10Wikimedia-Mailing-lists: Support services VIPs with not marked as VIP in Netbox - https://phabricator.wikimedia.org/T295793 (10Jelto) `gitlab1001` and `gitlab2001` will be decommissioned soon in T307142. So regarding GitLab this should be resolved... [10:15:13] 10serviceops, 10Data-Persistence-Backup, 10serviceops-collab, 10GitLab (Infrastructure), and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Jelto) >>! In T274463#8045578, @Dzahn wrote: > unfortunately just noticed an Icinga alert for gitlab1003 (nothing mails us about this, th... [10:35:47] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10JMeybohm) [10:49:01] 10serviceops, 10Release-Engineering-Team, 10Patch-For-Review: PendingDeprecationWarning on update_version.py - https://phabricator.wikimedia.org/T310133 (10TheresNoTime) 05Open→03Stalled Stalling on figuring out resolution to [[ https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/803886/1#mes... [10:55:02] 10serviceops, 10Infrastructure-Foundations, 10SRE-tools, 10Wikimedia-Mailing-lists: Support services VIPs with not marked as VIP in Netbox - https://phabricator.wikimedia.org/T295793 (10Ladsgroup) Regarding mailman, yes, Kunal and I didn't touch those settings [1] (I couldn't as I didn't have access to net... [11:03:11] 10serviceops, 10Release-Engineering-Team, 10Patch-For-Review: PendingDeprecationWarning on update_version.py - https://phabricator.wikimedia.org/T310133 (10RhinosF1) >>! In T310133#8051077, @TheresNoTime wrote: > Stalling on figuring out resolution to [[ https://gerrit.wikimedia.org/r/c/operations/deployment... [13:20:31] 10serviceops, 10Infrastructure-Foundations, 10SRE-tools, 10Wikimedia-Mailing-lists: Support services VIPs with not marked as VIP in Netbox - https://phabricator.wikimedia.org/T295793 (10akosiaris) Didn't need a reboot after all. I fixed /etc/network/interfaces configuration and issued a `systemctl restart... [13:37:42] 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops-collab, and 2 others: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Volker_E) Same, thanks @Dzahn for making the OOUIPHP demos work again! [13:47:21] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Migrate kubernetes apiserver alerts away from icinga - https://phabricator.wikimedia.org/T311251 (10JMeybohm) [14:04:31] 10serviceops, 10SRE, 10Thumbor, 10Thumbor Migration: Migrate thumbor Dockerfile to blubber - https://phabricator.wikimedia.org/T312104 (10hnowlan) [14:39:51] FYI, I'm switching kubetcd2004 to DRBD for the ganeti/codfw bullseye update, so there might be latency issues. It will probably be reverted to "plain" by end of this week [14:54:36] 10serviceops, 10serviceops-collab, 10GitLab (CI & Job Runners): DNS/networking not working on Trusted Runners - https://phabricator.wikimedia.org/T311241 (10Jelto) >>! In T311241#8042109, @Dzahn wrote: > currently the issue here is not DNS anymore. > > but it is now: 'This job is stuck because you don't ha... [14:57:49] 10serviceops, 10Patch-For-Review, 10Performance-Team (Radar): Install php 7.4 in production - https://phabricator.wikimedia.org/T311386 (10Joe) [15:20:38] 10serviceops, 10serviceops-collab, 10GitLab (CI & Job Runners): DNS/networking not working on Trusted Runners - https://phabricator.wikimedia.org/T311241 (10Jelto) I run the script in [repos/releng/gitlab-trusted-runner/](https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/) manually: ` $ add-pr... [15:57:46] 10serviceops, 10SRE, 10SRE-OnFire, 10Shellbox, 10Sustainability (Incident Followup): Shellbox resource management - https://phabricator.wikimedia.org/T310557 (10matmarex) [15:58:01] 10serviceops, 10SRE, 10SRE-OnFire, 10Shellbox, 10Sustainability (Incident Followup): Shellbox resource management - https://phabricator.wikimedia.org/T310557 (10matmarex) Cross-referencing: https://wikitech.wikimedia.org/wiki/Incidents/2022-07-03_shellbox_request_spike (I think the tasks I merged are re... [17:19:57] 10serviceops, 10Patch-For-Review, 10Performance-Team (Radar): Install php 7.4 in production - https://phabricator.wikimedia.org/T311386 (10Joe) After the installation on the mediawiki canaries, I ran our standard "benchmw" set of benchmarks on both php 7.2 and 7.4 in codfw. In general we see better performan... [18:15:56] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Dzahn) gitlab2001 has been removed from the acme_chief yaml, that allowed it to request certs but it's still up and has the puppet role applied. This... [18:32:02] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Dzahn) >>! In T307142#7967245, @Jelto wrote: > https://gitlab-replica.wikimedia.org/ will be migrated from `gitlab2001` (the old ganeti VM) to `gitlab1... [18:39:49] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin2002 for hosts: `gitlab2001.codfw.wmnet` - gitlab2001.codfw.wmnet (**FAIL**... [18:48:28] 10serviceops, 10Thumbor, 10TimedMediaHandler: Create thumbnail for MPEG (.mpg) videos - https://phabricator.wikimedia.org/T244570 (10brion) a:03brion Adding a note to this ticket: thumbor should apply deinterlacing on files that indicate interlacing in their video metadata; this will require a quick call i... [18:52:24] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin2002 for hosts: `gitlab2001.wikimedia.org` - gitlab2001.wikimedia.org (**PA... [18:56:04] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Dzahn) [18:56:51] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Dzahn) [18:57:11] 10serviceops, 10Release-Engineering-Team, 10Patch-For-Review: PendingDeprecationWarning on update_version.py - https://phabricator.wikimedia.org/T310133 (10jeena) I saw a question on IRC about which version is needed. According to the docs, https://pypi.org/project/ruamel.yaml/ version 0.16.0 or greater woul... [20:25:09] 10serviceops, 10Data-Persistence-Backup, 10serviceops-collab, 10GitLab (Infrastructure), and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Dzahn) >>! In T274463#8051032, @Jelto wrote: > I reverted the change which introduced the Blackbox exporter in https://gerrit.wikimedia.or... [21:03:40] 10serviceops, 10Release-Engineering-Team, 10Patch-For-Review: PendingDeprecationWarning on update_version.py - https://phabricator.wikimedia.org/T310133 (10Dzahn) @jeena Debian buster has 0.15.34 and Debian bullseye has 0.16.12. The puppet line "ensure_packages('python3-ruamel.yaml')" just tells it to make... [23:46:13] 10serviceops, 10Release-Engineering-Team, 10Patch-For-Review: PendingDeprecationWarning on update_version.py - https://phabricator.wikimedia.org/T310133 (10jeena) @Dzahn I was only commenting to provide information that in order to merge the patch in question, a different version would be needed, and finding... [23:54:22] 10serviceops, 10Release-Engineering-Team, 10Patch-For-Review: PendingDeprecationWarning on update_version.py - https://phabricator.wikimedia.org/T310133 (10Dzahn) @jeena And I just wanted to answer the question I saw "I'm not sure how to update it." literally. That is what is needed to update it. Also doesn'... [23:58:45] 10serviceops, 10Release-Engineering-Team, 10Patch-For-Review: PendingDeprecationWarning on update_version.py - https://phabricator.wikimedia.org/T310133 (10jeena) 👍 thanks @Dzahn, sorry for the misunderstanding