[09:56:35] 10serviceops, 10SRE, 10Patch-For-Review: Publish wikimedia-bullseye base docker image - https://phabricator.wikimedia.org/T281596 (10Joe) 05Open→03Resolved [09:56:43] 10serviceops, 10cloud-services-team (Kanban): Replace bootstrap-vz to generate Docker base images - https://phabricator.wikimedia.org/T281984 (10Joe) 05Open→03Resolved [12:19:35] 10serviceops, 10Add-Link, 10Data-Persistence (Consultation), 10Growth-Team (Current Sprint), 10Patch-For-Review: Determine why service responses are slow and what we can do about it - https://phabricator.wikimedia.org/T279411 (10kostajh) 05Open→03Resolved Thank you to everyone for your help with this... [12:46:07] 10serviceops, 10SRE, 10Thumbor, 10User-jijiki: Replace nutcracker with mcrouter on thumbor* - https://phabricator.wikimedia.org/T221081 (10jijiki) a:05jijiki→03None [16:55:48] 10serviceops, 10SRE: codfw appserver latency alerts flapping - https://phabricator.wikimedia.org/T283744 (10CDanis) [16:55:52] 10serviceops, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar), 10User-jijiki: Enable TLS on memcached for cross-dc replication - https://phabricator.wikimedia.org/T271967 (10jijiki) >>! In T271967#7112446, @jbond wrote: >> i wonder if we have considered just having the TLS port every where accept l... [18:03:15] Error: container has runAsNonRoot and image has non-numeric user (nobody), cannot verify user is non-root [18:03:23] jayme: still around? [18:04:48] legoktm: yep [18:05:02] where would you get that? [18:05:08] jayme: I'm in the process of deploying shellbox to staging-eqiad and running into the non-numeric issues [18:05:20] for nobody and www-data [18:05:42] can you point me to the dockerfile (template)? [18:06:20] https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/libs/Shellbox/+/refs/heads/master/.pipeline/blubber.yaml [18:06:38] uh, that's weird [18:06:44] Container image "docker-registry.wikimedia.org/wikimedia/mediawiki-libs-shellbox:score" already present on machine is a bit fishy, it's not explicitly versioned [18:06:59] oh [18:07:01] hmm [18:07:43] maybe https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/charts/shellbox/values.yaml#12 is wrong? [18:08:33] you should be overwriting that with a properly versioned tag in helmfile.d I suppose [18:09:05] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/helmfile.d/services/shellbox/values.yaml has it [18:09:38] hm. Give me a minute and I'll take a look [18:09:48] image: "{{ .Values.docker.registry }}/{{ .Values.main_app.image }}:{{ .Values.main_app.version }}" [18:10:32] jayme: btw, mutante and I are sitting in https://meet.google.com/inb-dmch-ycg?authuser=0 while we debug my and his problem if you want to join [19:38:14] legoktm: mutante: https://logstash.wikimedia.org/goto/e44d8d30f31b1e57051f60bd68cd2b4b [19:51:25] I did try to search calico logs on kubernetes2005 but the one in /var/log/containers/ is already rotated and in logstash I dont get anything if i exclude loglevel: NOTICE so did not find an obvious crash/error [21:18:25] 10serviceops, 10SRE: Redirect docker-registry URLs with tags in them to the static /tags/ HTML page - https://phabricator.wikimedia.org/T283764 (10Legoktm) [21:34:08] 10serviceops, 10SRE: codfw appserver latency alerts flapping - https://phabricator.wikimedia.org/T283744 (10jijiki) Ι think there is a correlation between the latency spikes and TCP retransmits [[https://grafana-rw.wikimedia.org/d/5E7tdiGWz/xxxx-effie?viewPanel=34&orgId=1&var-datasource=codfw%20prometheus%2Fo... [22:56:56] 10serviceops, 10SRE: httpd-fcgi image is not using numerical UIDs - https://phabricator.wikimedia.org/T283774 (10Legoktm) [22:59:10] 10serviceops, 10SRE: httpd-fcgi image is not using numerical UIDs - https://phabricator.wikimedia.org/T283774 (10Legoktm) [23:02:02] 10serviceops, 10SRE: httpd-fcgi image is not using numerical UIDs - https://phabricator.wikimedia.org/T283774 (10Legoktm) [23:02:39] 10serviceops, 10SRE: httpd-fcgi image is not using numerical UIDs - https://phabricator.wikimedia.org/T283774 (10Legoktm)