[03:00:31] 10serviceops, 10SRE, 10Patch-For-Review: make noc.wikimedia.org active/active (was: improve mw maintenance server switch over and discovery names) - https://phabricator.wikimedia.org/T265936 (10Dzahn) . [09:19:42] 10serviceops, 10DNS, 10SRE, 10Traffic, and 2 others: DNS for GitLab - https://phabricator.wikimedia.org/T276170 (10jbond) [10:03:05] 10serviceops, 10SRE, 10Patch-For-Review, 10User-jbond: docker-reporter-releng-images failed on deneb - https://phabricator.wikimedia.org/T251918 (10jbond) [10:03:33] 10serviceops, 10SRE, 10Patch-For-Review, 10User-jbond: docker-reporter-releng-images failed on deneb - https://phabricator.wikimedia.org/T251918 (10jbond) 05Resolved→03Open [10:06:49] 10serviceops, 10SRE, 10Patch-For-Review, 10User-jbond: docker-reporter-releng-images failed on deneb - https://phabricator.wikimedia.org/T251918 (10jbond) now seeing `lsb_release: command not found` on the docker buster instances ` var/lib/dpkg/info/debmonitor-client.postinst: line 16: lsb_release: comman... [10:22:26] 10serviceops, 10SRE, 10Patch-For-Review, 10User-jbond: docker-reporter-releng-images failed on deneb - https://phabricator.wikimedia.org/T251918 (10JMeybohm) >>! In T251918#7133852, @jbond wrote: > now seeing `lsb_release: command not found` on the docker buster instances > > ` > var/lib/dpkg/info/debmoni... [10:24:56] 10serviceops, 10SRE, 10Patch-For-Review, 10User-jbond: docker-reporter-releng-images failed on deneb - https://phabricator.wikimedia.org/T251918 (10jbond) > This is a packaging error I guess. The script tries to install debmonitor-client inside of the container. Exactly, see https://gerrit.wikimedia.org/r/... [10:57:11] 10serviceops, 10SRE, 10Patch-For-Review, 10User-jbond: docker-reporter-releng-images failed on deneb - https://phabricator.wikimedia.org/T251918 (10Joe) should we ensure lsb_release is installed in the base images? [12:29:06] 10serviceops, 10SRE, 10Patch-For-Review, 10User-jbond: docker-reporter-releng-images failed on deneb - https://phabricator.wikimedia.org/T251918 (10JMeybohm) >>! In T251918#7133990, @Joe wrote: > should we ensure lsb_release is installed in the base images? I don't think we should. If it is really needed... [13:23:55] 10serviceops, 10SRE, 10Patch-For-Review, 10User-jbond: docker-reporter-releng-images failed on deneb - https://phabricator.wikimedia.org/T251918 (10jbond) I now see an error with docker-registry.wikimedia.org/kubernetes-fluentd-daemonset:0.0.1-20190122 ` E: Failed to fetch http://mirrors.wikimedia.org/deb... [13:26:26] 10serviceops, 10SRE, 10Patch-For-Review, 10User-jbond: docker-reporter-releng-images failed on deneb - https://phabricator.wikimedia.org/T251918 (10Joe) The "last updated" on that page means nothing - it barely tells you when the script ran the last time. That image is old and should really be retired fro... [13:27:16] 10serviceops, 10SRE, 10Patch-For-Review, 10User-jbond: docker-reporter-releng-images failed on deneb - https://phabricator.wikimedia.org/T251918 (10jbond) >The "last updated" on that page means nothing - it barely tells you when the script ran the last time. I was beginning to wonder :) > That image is ol... [13:27:35] _joe_: i allready have a change out to filter jessie images i can add it to that (unless yu have allready done it) [13:27:41] https://gerrit.wikimedia.org/r/c/operations/puppet/+/698164/1 [13:28:44] <_joe_> jbond: yeah you can add the name to the unmaintained fileter [13:28:46] <_joe_> *filter [13:28:54] yes will do thanks [13:46:09] 10serviceops, 10observability: Outdated link in "MediaWiki exceptions and fatals" alert - https://phabricator.wikimedia.org/T284301 (10Krinkle) [14:28:56] <_joe_> jayme: any idea of why a dns name resolution would fail inside k8s? [14:29:11] <_joe_> oh I know why, nevermind, sorry [14:29:25] do tell :-) [14:29:31] <_joe_> it's not k8s [14:29:34] hrhr [14:29:37] <_joe_> it's mediawiki-config's code [14:29:40] <_joe_> doing something funny [14:29:49] <_joe_> yes, I have mediawiki running on staging :) [14:29:57] nice [14:30:02] <_joe_> "running" [14:30:10] <_joe_> we're collecting metrics [14:30:16] yeah. Containers not crashing I guess [14:30:28] <_joe_> and php is correctly reached [14:30:46] <_joe_> so yeah, we got to camp 2 [14:32:52] nice! [14:46:49] <_joe_> oh and ofc [18:24:26] 10serviceops, 10DNS, 10SRE, 10Traffic, and 2 others: DNS for GitLab - https://phabricator.wikimedia.org/T276170 (10Dzahn) [18:35:09] 10serviceops, 10Packaging, 10SRE, 10Design-Systems-team-board (Vue.js Migration Team Radar): Create Debian packages for Node.js 14 upgrade - https://phabricator.wikimedia.org/T267891 (10egardner) [19:00:12] 10serviceops, 10Release Pipeline, 10Services: Provide a node 12 production image (based on bullseye?) - https://phabricator.wikimedia.org/T284346 (10Jdforrester-WMF) [19:17:18] 10serviceops: `Failed to fetch http://mirrors.wikimedia.org/debian/pool/main/z/zvbi/libzvbi0_0.2.35-18_amd64.deb Error reading from server. Remote end closed connection [IP: 208.80.154.15 80]` on build of a bullseye CI image - https://phabricator.wikimedia.org/T284197 (10Jdforrester-WMF) Well, it finally passed... [19:21:29] 10serviceops: `Failed to fetch http://mirrors.wikimedia.org/debian/pool/main/z/zvbi/libzvbi0_0.2.35-18_amd64.deb Error reading from server. Remote end closed connection [IP: 208.80.154.15 80]` on build of a bullseye CI image - https://phabricator.wikimedia.org/T284197 (10Dzahn) 05Open→03Invalid I can also w...