[07:53:55] 10serviceops, 10MW-on-K8s, 10SRE-swift-storage, 10Shellbox, and 2 others: Support large files in Shellbox - https://phabricator.wikimedia.org/T292322 (10Joe) To further clarify how I conduct the no-timeout tests: - `sudo puppet agent --disable "T292322 --$USER"` `/etc/envoy/envoy.yaml`: - Change the timeou... [08:58:46] 10serviceops, 10MW-on-K8s, 10SRE-swift-storage, 10Shellbox, and 2 others: Support large files in Shellbox - https://phabricator.wikimedia.org/T292322 (10Joe) So apparently my small fixes to PagedTiffHandler shaved off about 100 seconds off of the shellbox-based request, see: https://performance.wikimedia.... [12:34:15] \o is it normal to see a lot of "Config file changed" messages for prometheus on linkrecommendation_staging-metrics-exporter container? I see this every ~2 minutes in the logs [12:50:58] <_joe_> kostajh: I wouldn't say so, no [12:52:02] <_joe_> but I don't have time to look into it right now, can you open a task if you need our help? [12:56:45] ok, will do [13:00:35] according to logstash it's actually pretty common [13:01:11] I'd guess that this has to do with how kubernetes links mounted configmaps for dynamic updates [13:01:26] ftr: the full message is Config file changed (\"/etc/monitoring/prometheus-statsd.conf\": MODIFY|ATTRIB), attempting reload" source="main.go:105 [13:06:11] bot im wondering kostajh: Does linkrecommendation even emit statsd metrics? [13:07:13] <_joe_> uhm it's frankly not what I'd expect, given configmaps are just bind mounts [13:07:26] yeah but o [13:07:29] *no [13:08:24] 10serviceops, 10Add-Link, 10Growth-Team: linkrecommendation: Many repeated config file changed / config file reloaded messages - https://phabricator.wikimedia.org/T300629 (10kostajh) [13:08:37] ^ the task for this [13:10:36] how unfortunate it actually uses statsd metrics - I was about to say: just drop the exporter as this is no legacy code :) [13:12:21] 10serviceops, 10Add-Link, 10Growth-Team: Many repeated config file changed / config file reloaded messages - https://phabricator.wikimedia.org/T300629 (10JMeybohm) [13:17:06] 10serviceops, 10Add-Link, 10Growth-Team: Many repeated config file changed / config file reloaded messages - https://phabricator.wikimedia.org/T300629 (10JMeybohm) This is actually common across all services that use prometheus-statsd exporter. While I do think it's not an issue in general it causes a lot of... [13:45:33] (big thank you to jayme for helping diagnose & unblock the linkrecommendation app deployments!) [15:05:22] 10serviceops, 10Add-Link, 10Growth-Team: Many repeated config file changed / config file reloaded messages - https://phabricator.wikimedia.org/T300629 (10JMeybohm) p:05Triage→03Low [15:13:04] 10serviceops, 10Add-Link, 10Growth-Team, 10Patch-For-Review: Many repeated config file changed / config file reloaded messages - https://phabricator.wikimedia.org/T300629 (10JMeybohm) a:03JMeybohm [15:17:52] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.2.2 - https://phabricator.wikimedia.org/T300392 (10Jelto) 05Open→03Resolved scap `4.2.2` is deployed on all machines. I'm closing this task. [16:46:10] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) tested applying the puppet role after floating IP was added. we will need "profile::gitlab::monitoring_whitelist" in Hiera next [16:55:02] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) The merge above fixed: did not find a value for the name 'profile::gitlab::monitoring_whitelist' next issue is: parameter 'exporters' expects a Hash v... [16:58:31] 10serviceops, 10Security-Team, 10GitLab (CI & Job Runners), 10Patch-For-Review, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Jelto) [17:02:46] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) [17:31:55] 10serviceops, 10SRE, 10Sustainability (Incident Followup): High latency on appservers - https://phabricator.wikimedia.org/T272215 (10lmata) 05Open→03Declined closing this documentation task as it is unlikely the documentation will be completed further [18:15:48] 10serviceops, 10MW-on-K8s, 10SRE-swift-storage, 10Shellbox, and 2 others: Support large files in Shellbox - https://phabricator.wikimedia.org/T292322 (10Legoktm) >>! In T292322#7666754, @Joe wrote: > I think this is still way too slow compared to the traditional request time. > > I'll update this task lat... [22:40:14] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) After small fixes above the "role::gitlab" (except same thing as prod) class is now applied on gitlab-prod-1001.devtools in cloud VPS andDOES NOT FAIL anym... [22:47:15] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) [] installed gitlab-ce package post-installation script subprocess returned error exit status 1 [] nginx initial setup needs race condition? [] Checking i... [23:04:10] 10serviceops, 10Add-Link, 10Growth-Team (Current Sprint), 10Patch-For-Review: Many repeated config file changed / config file reloaded messages - https://phabricator.wikimedia.org/T300629 (10mewoph)