[00:55:15] 10serviceops, 10Logging, 10Wikimedia-General-or-Unknown, 10Performance-Team (Radar), 10Sustainability (Incident Followup): Ingest logs from scheduled maintenance scripts at WMF in Logstash - https://phabricator.wikimedia.org/T285896 (10lmata) [02:40:14] 10serviceops, 10Logging, 10SRE, 10Datacenter-Switchover: Figure out switchover steps for mwlog hosts - https://phabricator.wikimedia.org/T261274 (10lmata) [02:47:06] 10serviceops, 10Alerting, 10Icinga, 10SRE, 10observability: incident 20170323-wikibase did not trigger Icinga paging - https://phabricator.wikimedia.org/T161528 (10lmata) [03:07:58] 10serviceops, 10Citoid, 10Logging, 10SRE, and 3 others: Citoid is logging all request / response headers as separate fields - https://phabricator.wikimedia.org/T239713 (10lmata) [03:16:21] 10serviceops, 10Logging, 10MW-on-K8s, 10SRE: Keep calculating latencies for MediaWiki requests that happen k8s - https://phabricator.wikimedia.org/T276095 (10lmata) [03:24:11] 10serviceops, 10Logging, 10SRE: rsyslogd: omkafka: action will suspended due to kafka error -187: Local: All broker connections are down - https://phabricator.wikimedia.org/T240560 (10lmata) [08:17:32] _joe_: Good morning, We deployed shellbox fully and I see that this WAN cache is got fixed: https://grafana.wikimedia.org/d/lqE4lcGWz/wanobjectcache-key-group?orgId=1&var-kClass=WikibaseQualityConstraints&from=now-3h&to=now but I can't see any massive impact on memcached itself :/ [08:17:49] Some other nice metrics https://grafana.wikimedia.org/d/000000378/ladsgroup-test?viewPanel=10&orgId=1&from=now-12h&to=now [13:16:17] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Add kubernetes 1.17+ topology annotations - https://phabricator.wikimedia.org/T270191 (10JMeybohm) the typology annotations where integrated with Kubernetes 1.17. Earlier kubelet versions reject the labels with: > --node-labels in the 'kubernetes.io' namespace m... [13:30:48] 10serviceops, 10Wikidata, 10Wikidata-Query-Service, 10wdwb-tech, and 2 others: Deploy Flink (rdf-streaming-updater) to kubernetes (k8s) - https://phabricator.wikimedia.org/T264006 (10So9q) [14:08:22] 10serviceops, 10MW-on-K8s, 10Kubernetes, 10Patch-For-Review: Only schedule mediawiki pods on nodes with non-spinning disks - https://phabricator.wikimedia.org/T288345 (10JMeybohm) That was a bit of a pain.... Anyways: All nodes do have proper `node.kubernetes.io/disk-type` labels now. [14:08:37] _joe_: ^ [14:09:28] * jayme away quick errand, back ~14:30Z [14:09:39] oops [14:31:22] welcome arnoldokoth! 👋 [14:32:38] Thanks Reuven! 😀 [14:32:39] Hey arnoldokoth 👋 [14:41:26] welcome arnoldokoth! o/ [15:03:32] <_joe_> welcome arnoldokoth!!! [15:03:42] <_joe_> (sorry I was in multiple meetings in a row) [15:34:08] _joe_: I btw. graduated the dragonfly dashboard last week as well https://grafana-rw.wikimedia.org/d/CmbiPADWz/dragonfly [15:34:22] <_joe_> oh nice! [15:36:28] I'm still puzzled about some of the metrics, so I left them as-is (based on the defaults from upstream) [15:37:06] supernode peers for example. I'm not sure that that should say exactly :) [15:44:13] welcome arnoldokoth :) (I'm Kunal) [15:46:12] Amir1: how did the shellbox-constraints full enabling go? [16:18:21] 10serviceops, 10MediaWiki-extensions-Score, 10Shellbox: Score ocassionally gets a 503 response from Shellbox - https://phabricator.wikimedia.org/T287288 (10Legoktm) 05Open→03Resolved a:03Joe {F34588678} Calling this fixed. [17:39:54] legoktm: dropped median of the time of doing the job to half [17:40:23] legoktm: https://phabricator.wikimedia.org/T176312#7269637 [17:40:25] onwards [17:41:38] nothing broke, etc. [17:48:09] legoktm: _joe_ when you both have time, let me know to discuss the systemd::timer /service for dispatching [17:48:28] <_joe_> Amir1: probably tomorrow [17:48:38] SGTM [20:04:04] 10serviceops, 10SRE, 10Traffic: Unexpected upload speed to commons - https://phabricator.wikimedia.org/T288481 (10aborrero) [20:58:24] 10serviceops, 10SRE, 10Traffic: Unexpected upload speed to commons - https://phabricator.wikimedia.org/T288481 (10aborrero)