[08:16:03] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q4: (Need By: TBD) rack/setup/install mw14[57-98] - https://phabricator.wikimedia.org/T306121 (10ayounsi) [08:16:09] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Decommission mw13[07-48] - https://phabricator.wikimedia.org/T306162 (10ayounsi) [08:18:46] new release of kserve (https://github.com/kserve/kserve/releases/download/v0.8.0/kserve.yaml) - 15k yaml file [08:18:51] * elukey cries in a corner [08:38:47] oof [08:42:27] most of it is a horrible API conversion webhook, hiding it in my editor removes 10k+ lines [09:42:44] 10serviceops, 10GitLab (Project Migration): import subversion repos from Phabricator into Gitlab - https://phabricator.wikimedia.org/T308061 (10Aklapper) (Whoever recently created the saved Diffusion query called "svn-repos" at https://phabricator.wikimedia.org/diffusion/query/6WUBLfM9eS2R/ , it was the "defau... [09:51:18] 10serviceops, 10Patch-For-Review, 10Performance-Team (Radar): Properly support php7.4 across the observability stack - https://phabricator.wikimedia.org/T312634 (10Joe) There is a plot twist: I realized that anything currently offered by php-fpm-exporter of any value to us is also exported by the php-fpm-sta... [09:52:40] 10serviceops, 10Patch-For-Review, 10Performance-Team (Radar): Properly support php7.4 across the observability stack - https://phabricator.wikimedia.org/T312634 (10Joe) [10:29:39] FYI, I'm temporarily switching kubestagetcd2003 to DRBD to allow moving it to another host for the Ganeti bullseye update, latency will increase for a bit [10:52:30] ack, thanks [11:09:07] 10serviceops, 10GitLab (Project Migration): import subversion repos from Phabricator into Gitlab - https://phabricator.wikimedia.org/T308061 (10hashar) Subversion got removed from Phabricator by mistake in https://gerrit.wikimedia.org/r/c/operations/puppet/+/785149 which has lead to T307889 and now this task t... [11:47:36] 10serviceops, 10Phabricator, 10serviceops-collab, 10Release-Engineering-Team (The Decommission Mission πŸ’€): Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10LSobanski) @thcipriani if that particular bash history is critical, should we persist it somewhere more resilient? [12:13:06] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10JMeybohm) [12:19:42] 10serviceops, 10GitLab (Project Migration): import subversion repos from Phabricator into Gitlab - https://phabricator.wikimedia.org/T308061 (10Aklapper) I propose to decline this task, and not to care about these SVN repos. Not every ancient code base must be archived //by us// forever.`{{Citation needed}}`... [13:07:53] 10serviceops, 10serviceops-collab, 10GitLab (CI & Job Runners): DNS/networking not working on Trusted Runners - https://phabricator.wikimedia.org/T311241 (10Jelto) DNS issues on Trusted Runners should be fixed now. Mocked/test Trusted Runners in WMCS devtools still have DNS issues. When buildkitd was added... [13:49:43] 10serviceops, 10SRE, 10Traffic, 10Abstract Wikipedia team (Phase Ξ» – Launch), 10HTTPS: Get new edge & internal HTTPS certificates expanded to add wikifunctions.org and *.wikifunctions.org - https://phabricator.wikimedia.org/T313227 (10BBlack) a:03Jdforrester-WMF Hi - the process for the public certs+DN... [14:01:12] 10serviceops, 10SRE, 10Traffic, 10Abstract Wikipedia team (Phase Ξ» – Launch), 10HTTPS: Get new edge & internal HTTPS certificates expanded to add wikifunctions.org and *.wikifunctions.org - https://phabricator.wikimedia.org/T313227 (10Jdforrester-WMF) >>! In T313227#8091301, @BBlack wrote: > Hi - the pro... [14:27:18] best k8s version changelog so far: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md#changelog-since-v1203 [14:34:23] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10JMeybohm) [15:21:10] <_joe_> jayme: eheheh [15:27:25] 10serviceops, 10Phabricator, 10serviceops-collab, 10Release-Engineering-Team (The Decommission Mission πŸ’€): Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10thcipriani) >>! In T313360#8090919, @LSobanski wrote: > @thcipriani if that particular bash history is critical, should... [16:42:04] 10serviceops, 10serviceops-collab, 10GitLab (CI & Job Runners): DNS/networking not working on Trusted Runners - https://phabricator.wikimedia.org/T311241 (10dduvall) >>! In T311241#8086793, @Jelto wrote: >>>! In T311241#8065350, @Jelto wrote: >> [...] >> Would it be possible to use the docker default network... [16:48:32] 10serviceops, 10Release-Engineering-Team, 10serviceops-collab, 10GitLab (CI & Job Runners), 10User-dduvall: Changes to modules/gitlab_runner/templates/config-template.toml.erb have no effect on existing runners - https://phabricator.wikimedia.org/T311746 (10brennen) [17:28:31] 10serviceops, 10Phabricator, 10serviceops-collab, 10Release-Engineering-Team (The Decommission Mission πŸ’€): Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10hashar) Earlier this week I have mentioned `rsync` issue about username/group names being mixed up. That caused us a lo... [17:29:58] 10serviceops, 10Phabricator, 10serviceops-collab, 10Release-Engineering-Team (The Decommission Mission πŸ’€): Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10thcipriani) [17:34:08] 10serviceops, 10Phabricator, 10serviceops-collab, 10Release-Engineering-Team (The Decommission Mission πŸ’€): Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10thcipriani) >>! In T313359#8089535, @bd808 wrote: >>>! In T313359#8089476, @thcipriani wr... [17:36:37] 10serviceops, 10Phabricator, 10serviceops-collab, 10Release-Engineering-Team (The Decommission Mission πŸ’€): Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10hashar) For git over http one can issue a 301 redirect and avoid breaking the world. For... [17:44:51] 10serviceops, 10GitLab (Project Migration): import subversion repos from Phabricator into Gitlab - https://phabricator.wikimedia.org/T308061 (10demon) I don't think we need the Subversion data itself -- definitely don't need to go through the hassle of porting it all over to Gitlab. I proposed we just tar+gzip... [18:36:59] 10serviceops, 10Phabricator, 10serviceops-collab, 10Release-Engineering-Team (The Decommission Mission πŸ’€): Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10Aklapper) > do you have thoughts about how easy this query would be? Čüş, not really...... [18:53:45] 10serviceops, 10SRE, 10Patch-For-Review: Migrate node-based services in production to node12 - https://phabricator.wikimedia.org/T290750 (10Jdforrester-WMF) [19:36:19] 10serviceops, 10Znuny, 10serviceops-collab, 10vrts, 10Patch-For-Review: VRTS monitoring- re-activate new blackbox check (was: 'ProbeDown') - https://phabricator.wikimedia.org/T312194 (10Dzahn) This is now fixed after a couple follow-ups. Last one was that to make sure to only check via IPv4 for the envo... [19:36:59] 10serviceops, 10Znuny, 10serviceops-collab, 10vrts, 10Patch-For-Review: VRTS monitoring- re-activate new blackbox check (was: 'ProbeDown') - https://phabricator.wikimedia.org/T312194 (10Dzahn) 05Openβ†’03Resolved [19:37:10] 10serviceops, 10SRE, 10Znuny, 10serviceops-collab, 10Sustainability (Incident Followup): enhance Znuny (otrs) alerting - https://phabricator.wikimedia.org/T303190 (10Dzahn) [19:43:30] 10serviceops, 10SRE, 10Znuny, 10serviceops-collab, 10Sustainability (Incident Followup): enhance Znuny (otrs) alerting - https://phabricator.wikimedia.org/T303190 (10Dzahn) also see T312194#8092388 We now have working checks. Here you can see how it is working: https://grafana-rw.wikimedia.org/explore... [20:33:07] 10serviceops, 10Performance-Team, 10Scap: MW wmf-config tmp cache stays outdated after Scap deploy (opcache revalidation is off) - https://phabricator.wikimedia.org/T311788 (10dancy) 05Resolvedβ†’03Open The same situation happened today during a deployment, so we're not out of the woods yet. [21:04:36] 10serviceops, 10GitLab (Project Migration): import subversion repos from Phabricator into Gitlab - https://phabricator.wikimedia.org/T308061 (10Dzahn) I would also say we can make a .tar.gz and then I would suggest to put it here, next to these: https://dumps.wikimedia.org/other/misc/ .oh..look..there is alr... [21:15:52] 10serviceops, 10Performance-Team, 10Scap: MW wmf-config tmp cache stays outdated after Scap deploy (opcache revalidation is off) - https://phabricator.wikimedia.org/T311788 (10dancy) Clarifying for the record: Opcache revalidation is turned off by setting `opcache.validate_timestamps` to 0. `opcache.revali... [21:48:47] 10serviceops, 10GitLab (Project Migration): import subversion repos from Phabricator into Gitlab - https://phabricator.wikimedia.org/T308061 (10hashar) There are some dumps yes. For toolserver I previously traced the history to T60801. My point above was that `subversion` was erroneously removed and I would... [22:13:37] 10serviceops, 10GitLab (Project Migration): import subversion repos from Phabricator into Gitlab - https://phabricator.wikimedia.org/T308061 (10RhinosF1) The removal wasn't an accident or erroneous. It just didn't quite consider that some docs for old config variables and stuff doesn't point to SVN. I'm not q... [22:26:56] 10serviceops, 10SRE, 10Znuny, 10serviceops-collab, 10Sustainability (Incident Followup): enhance Znuny (otrs) alerting - https://phabricator.wikimedia.org/T303190 (10Dzahn) here is a screenshot that shows how to get this on https://grafana-rw.wikimedia.org {F35327060} [22:38:02] 10serviceops, 10Phabricator, 10serviceops-collab, 10Release-Engineering-Team (The Decommission Mission πŸ’€): Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10Dzahn) >>! In T313359#8092203, @Aklapper wrote: > Čüş https://en.wiktionary.org/wiki/%C4... [23:34:12] 10serviceops, 10Phabricator, 10serviceops-collab, 10Release-Engineering-Team (The Decommission Mission πŸ’€): Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10Dzahn) When I used the "metrics_user" I only got a limited set of phabricator databases b... [23:41:57] 10serviceops, 10Phabricator, 10serviceops-collab, 10Release-Engineering-Team (The Decommission Mission πŸ’€): Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10Dzahn) Now in this table there are also "viewPolicy", "editPolicy" and "pushPolicy" field... [23:54:20] 10serviceops, 10Phabricator, 10serviceops-collab, 10Release-Engineering-Team (The Decommission Mission πŸ’€): Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10Dzahn) total git repos starting with tool- : 445 repos without commits, starting with t...