[05:13:10] <_joe_> sigh, we have 40 appservers to put into rotation in eqiad [05:28:33] 10serviceops, 10Dumps-Generation, 10Patch-For-Review, 10Performance-Team (Radar): Migrate WMF production from PHP 7.2 to PHP 7.4 - https://phabricator.wikimedia.org/T271736 (10Joe) [06:15:45] _joe_: good morning. I could use some help for `docker-pkg` the repo lacks a 3.0.3 tag and that did not get deployed :-\ [06:16:17] I am notably looking for the commit [66606ae] Builder: use the full image tag, not just the name when pulling [06:16:23] <_joe_> hashar: I'm in the middle of a) a netowrk outage b) moving production to php 7.4; I saw your ping yesterday, either I or someone else will get to it [06:16:23] it is crippling us on the contint machines :] [06:16:38] awesome :) [06:16:54] I was merely looking at some acknowledgement the task got seen :-] [06:17:37] <_joe_> "someone else" because it's good for the rookies to learn of our best delicacies, like scap3 [06:18:34] I can surely pair on that with whoever ends up tasting the delicacy [06:19:32] <_joe_> you might be able to speak your native language then :P [06:19:56] * hashar grins at claime [06:19:57] :D [06:50:56] 10serviceops, 10Dumps-Generation, 10Patch-For-Review, 10Performance-Team (Radar): Migrate WMF production from PHP 7.2 to PHP 7.4 - https://phabricator.wikimedia.org/T271736 (10Joe) [06:51:54] 10serviceops, 10Dumps-Generation, 10Patch-For-Review, 10Performance-Team (Radar): Migrate WMF production from PHP 7.2 to PHP 7.4 - https://phabricator.wikimedia.org/T271736 (10Joe) [08:04:15] * claime just woke up and asks for a grace period [08:16:20] <_joe_> claime: scap will eagerly wait for you don't worry [08:16:46] * claime shivers [08:20:57] 10serviceops, 10GitLab, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: Disable email notifications from GitLab replicas - https://phabricator.wikimedia.org/T318682 (10Jelto) 05Open→03Resolved Should be fixed now. Feel free to re-open if you find any more mails from the replica. [08:36:53] got in sync with claime thanks _joe_ ! [08:42:14] 10serviceops, 10Observability-Metrics, 10Kubernetes: Don't scrape every containerPort for metrics - https://phabricator.wikimedia.org/T318707 (10JMeybohm) [10:33:59] 10serviceops, 10Observability-Metrics, 10Kubernetes: Don't scrape every containerPort for metrics - https://phabricator.wikimedia.org/T318707 (10hnowlan) I've just clarified the current behaviours [[ https://wikitech.wikimedia.org/wiki/Kubernetes/Metrics#Workload%2FPod_metrics | on Wikitech ]] - please updat... [10:39:30] 10serviceops, 10Dumps-Generation, 10Patch-For-Review, 10Performance-Team (Radar): Migrate WMF production from PHP 7.2 to PHP 7.4 - https://phabricator.wikimedia.org/T271736 (10Joe) [12:43:50] 10serviceops, 10PHP 7.3 support, 10PHP 7.4 support, 10Platform Team Workboards (Clinic Duty Team): Rename articles and users to prepare for PHP 7.3 unicode changes - https://phabricator.wikimedia.org/T292552 (10Jdforrester-WMF) [13:23:08] 10serviceops, 10Dumps-Generation, 10Patch-For-Review, 10Performance-Team (Radar): Migrate WMF production from PHP 7.2 to PHP 7.4 - https://phabricator.wikimedia.org/T271736 (10Jdforrester-WMF) [13:23:18] 10serviceops, 10PHP 7.3 support, 10PHP 7.4 support, 10Platform Team Workboards (Clinic Duty Team): Rename articles and users to prepare for PHP 7.3 unicode changes - https://phabricator.wikimedia.org/T292552 (10Jdforrester-WMF) [13:23:24] 10serviceops, 10Dumps-Generation, 10Patch-For-Review, 10Performance-Team (Radar): Migrate WMF production from PHP 7.2 to PHP 7.4 - https://phabricator.wikimedia.org/T271736 (10Jdforrester-WMF) [13:24:06] 10serviceops, 10PHP 7.3 support, 10PHP 7.4 support, 10Platform Team Workboards (Clinic Duty Team): Rename articles and users to prepare for PHP 7.3 unicode changes - https://phabricator.wikimedia.org/T292552 (10Jdforrester-WMF) Given we're doing the final parts of this after we've fully migrated, I've swit... [13:24:38] 10serviceops: Put parse parse10[01-24] in production - https://phabricator.wikimedia.org/T307219 (10Jdforrester-WMF) [13:25:00] 10serviceops: Put parse parse10[01-24] in production - https://phabricator.wikimedia.org/T307219 (10Jdforrester-WMF) [14:13:39] jayme I think I addressed your concerns in https://gerrit.wikimedia.org/r/c/operations/puppet/+/835691 but let me know if not, we can do a Meet if that's faster [14:17:52] inflatador: I think that's still the wrong place. "relabel_configs" deals with creating prometheus targets to scrape. That happens before actually scraping metrics [14:18:33] "metrics_relabel_configs" is the config structure that is applied to the actual metrics scraped from targets [14:19:38] happy to chat on meet if that will be of help to you! [14:27:41] 10serviceops, 10SRE, 10Wikidata, 10Wikidata-Termbox, 10wdwb-tech: Plan to scale up termbox service to be able to render the termbox for desktop pageviews - https://phabricator.wikimedia.org/T261486 (10jijiki) [14:29:56] 10serviceops, 10SRE, 10Wikidata, 10Wikidata-Termbox, 10wdwb-tech: Plan to scale up termbox service to be able to render the termbox for desktop pageviews - https://phabricator.wikimedia.org/T261486 (10Addshore) 05Open→03Invalid Didnt happen in 2020 / since, so closing this now [14:40:03] jayme sure, I'm up at https://meet.google.com/khy-nfoz-wab if you have time to drop in [15:44:40] 10serviceops, 10SRE, 10Thumbor, 10Thumbor Migration, 10Platform Team Workboards (Platform Engineering Reliability): Replace nutcracker with mcrouter - https://phabricator.wikimedia.org/T318695 (10jijiki) [15:44:46] 10serviceops, 10SRE, 10Thumbor, 10User-jijiki: Replace nutcracker with mcrouter on thumbor* - https://phabricator.wikimedia.org/T221081 (10jijiki) [15:48:23] 10serviceops, 10Thumbor, 10Thumbor Migration, 10Performance-Team (Radar), 10User-jijiki: Terminate Thumbor with SSL - https://phabricator.wikimedia.org/T180696 (10hnowlan) 05Open→03Invalid [15:48:40] 10serviceops, 10Thumbor, 10Thumbor Migration, 10Performance-Team (Radar), 10User-jijiki: Terminate Thumbor with SSL - https://phabricator.wikimedia.org/T180696 (10hnowlan) Closing this ticket as we will get it automatically as part of the Kubernetes migration. [16:08:15] 10serviceops, 10Observability-Metrics, 10Kubernetes, 10Patch-For-Review: Limit the envoy metrics scraped from k8s - https://phabricator.wikimedia.org/T318705 (10bking) How can we tell if this is working? Use an envoy metric that doesn't match the regex, such as `rate(envoy_cluster_default_total_match_co... [16:18:53] 10serviceops, 10Observability-Metrics, 10Kubernetes, 10Patch-For-Review: Limit the envoy metrics scraped from k8s - https://phabricator.wikimedia.org/T318705 (10JMeybohm) >>! In T318705#8269372, @bking wrote: > How can we tell if this is working? > > Use an envoy metric that doesn't match the regex, such... [16:21:16] 10serviceops, 10SRE: Update conf1* servers - https://phabricator.wikimedia.org/T310062 (10akosiaris) 05Open→03Resolved Yup, resolving. Thanks! [16:35:51] 10serviceops, 10Platform Engineering, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Phase out "redis_sessions" cluster and away from memcached cluster - https://phabricator.wikimedia.org/T267581 (10jijiki) a:03jijiki [16:36:16] 10serviceops, 10SRE, 10Performance-Team (Radar): Phase out nutcracker for connecting to redis - https://phabricator.wikimedia.org/T277183 (10jijiki) a:05Joe→03jijiki [16:40:51] 10serviceops, 10SRE, 10Performance-Team (Radar): Phase out nutcracker from mediawiki servers - https://phabricator.wikimedia.org/T277183 (10jijiki) [16:42:29] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install kubernetes102[34] - https://phabricator.wikimedia.org/T313873 (10Cmjohnson) a:05Cmjohnson→03Jclark-ctr @jclark-ctr can you verify the port for kubernetes1023, looks like something is already in c6/port 36 [16:43:57] 10serviceops, 10SRE, 10Performance-Team (Radar): Phase out nutcracker from mediawiki servers - https://phabricator.wikimedia.org/T277183 (10jijiki) [16:44:19] 10serviceops, 10Platform Engineering, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Phase out "redis_sessions" cluster and away from memcached cluster - https://phabricator.wikimedia.org/T267581 (10jijiki) [17:12:17] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install kubernetes102[34] - https://phabricator.wikimedia.org/T313873 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host kubernetes1024.eqiad.wmnet with OS bullseye [17:13:26] lunch/workout, back in ~1h [17:16:25] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install kubernetes102[34] - https://phabricator.wikimedia.org/T313873 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host kubernetes1024.eqiad.wmnet with OS bullseye executed with errors: - kubernetes1... [17:17:06] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install kubernetes102[34] - https://phabricator.wikimedia.org/T313873 (10Cmjohnson) Also, @jclark-ctr please check the network cables are in the correct port. 1024 is giving me a cable failure PXE-E61: Media test failure, check cable [17:17:29] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install kubernetes102[34] - https://phabricator.wikimedia.org/T313873 (10Cmjohnson) [18:09:06] 10serviceops, 10Data-Engineering, 10SRE, 10Event-Platform Value Stream (Sprint 02), 10Patch-For-Review: eventstreams chart should use latest common_templates - https://phabricator.wikimedia.org/T310721 (10lbowmaker) [18:09:17] 10serviceops, 10Data Engineering Planning, 10SRE, 10Event-Platform Value Stream (Sprint 02), 10Patch-For-Review: eventgate chart should use common_templates - https://phabricator.wikimedia.org/T303543 (10lbowmaker) [18:22:47] back [19:35:21] 10serviceops, 10Prod-Kubernetes, 10Wikidata, 10Wikidata-Query-Service, and 2 others: Write and adapt Runbooks and cookbooks related to the WDQS Streaming Updater and kubernetes - https://phabricator.wikimedia.org/T293063 (10bking)