[05:32:11] <legoktm>	 !issync
[05:32:11] <wm-ircservserv_>	 Syncing #wikimedia-serviceops
[05:32:13] <wm-ircservserv_>	 Set /cs flags #wikimedia-serviceops effie -ARefiorstv
[05:32:15] <wm-ircservserv_>	 Set /cs flags #wikimedia-serviceops jayme -ARefiorstv
[05:32:50] <legoktm>	 sigh
[05:48:45] <apergos>	 and its back
[05:48:58] <apergos>	 but what is "syncing", anyways?
[05:52:11] <legoktm>	 apergos: theoretically it makes sure the channel state matches https://gerrit.wikimedia.org/r/plugins/gitiles/wikimedia/irc/ircservserv-config/+/refs/heads/master/channels/wikimedia-serviceops.toml 
[05:52:16] <legoktm>	 no more, no less
[05:52:21] <legoktm>	 https://meta.wikimedia.org/wiki/IRC/Bots/ircservserv has some docs
[05:52:34] <legoktm>	 GitOps for IRC or something :p
[05:52:39] <apergos>	 ah very good
[05:52:48] <legoktm>	 !isspull
[05:52:50] <wm-ircservserv_>	 Pulled changes for: #mediawiki, #pywikibot, #wikimedia-dev, #wikimedia-hackathon, #wikimedia-kawaii, #wikimedia-operations, #wikimedia-serviceops, #wikimedia-sre
[05:52:57] <apergos>	 oh yes, I recall the bikeshedding of the name
[05:54:00] <legoktm>	 fun fact, nicks cannot end with "serv"
[05:54:26] <apergos>	 lol
[05:54:42] <apergos>	 in fact the internet is held together with duct tape and edge cases...
[05:54:54] <legoktm>	 !issync
[05:54:54] <wm-ircservserv_>	 Syncing #wikimedia-serviceops
[05:54:56] <wm-ircservserv_>	 Set /cs flags #wikimedia-serviceops rzl -ARefiorstv
[05:54:58] <wm-ircservserv_>	 Set /cs flags #wikimedia-serviceops jayme -ARefiorstv
[05:55:00] <wm-ircservserv_>	 Set /cs flags #wikimedia-serviceops effie -ARefiorstv
[05:55:02] <wm-ircservserv_>	 Set /cs flags #wikimedia-serviceops mutante -ARefiorstv
[05:55:04] <wm-ircservserv_>	 Set /cs flags #wikimedia-serviceops akosiaris -ARefiorstv
[05:55:06] <wm-ircservserv_>	 Set /cs flags #wikimedia-serviceops rzl +Afiortv
[05:55:08] <wm-ircservserv_>	 Set /cs flags #wikimedia-serviceops mutante +Afiortv
[05:55:10] <wm-ircservserv_>	 Set /cs flags #wikimedia-serviceops jayme +Afiortv
[05:55:12] <wm-ircservserv_>	 Set /cs flags #wikimedia-serviceops akosiaris +Afiortv
[05:55:14] <wm-ircservserv_>	 Set /cs flags #wikimedia-serviceops effie +Afiortv
[05:55:30] <legoktm>	 !issync
[05:55:30] <wm-ircservserv_>	 Syncing #wikimedia-serviceops
[05:55:31] <wm-ircservserv_>	 No updates for #wikimedia-serviceops
[05:55:38] <legoktm>	 woot
[05:55:52] <apergos>	 \o/
[06:04:16] <wikibugs>	 10serviceops: Add IRC SRE bot for SAL !log actions to #wikimedia-serviceops - https://phabricator.wikimedia.org/T213196 (10Marostegui)
[06:05:34] <wikibugs>	 10serviceops: SRE FY2019-20 Q3 goal: Increase reach of deployment pipeline - https://phabricator.wikimedia.org/T212935 (10Marostegui)
[06:07:48] <wikibugs>	 10serviceops, 10SRE, 10Performance-Team (Radar): Increased POST latency for MW app servers (Oct 2019) - https://phabricator.wikimedia.org/T235755 (10Marostegui) It's been almost 2 years, any point on keeping this task open?
[06:15:57] <wikibugs>	 10serviceops, 10SRE, 10Performance-Team (Radar): Increased POST latency for MW app servers (Oct 2019) - https://phabricator.wikimedia.org/T235755 (10jijiki) 05Open→03Resolved a:03jijiki No!
[07:52:47] <_joe_>	 [spec.egress.18.ports.0.endPort] unknown key: endPort
[07:53:02] <_joe_>	 always remember to check the kubernetes version of the docs
[12:29:53] <wikibugs>	 10serviceops, 10Continuous-Integration-Infrastructure, 10netops: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ) - https://phabricator.wikimedia.org/T283582 (10Marostegui) p:05Triage→03Medium Tagging also #netops in case they can help out
[13:13:22] <_joe_>	 legoktm, mutante_ let's schedule some time tomorrow to discuss progress and next steps
[13:38:16] <wikibugs>	 10serviceops, 10Continuous-Integration-Infrastructure, 10netops: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ) - https://phabricator.wikimedia.org/T283582 (10ayounsi) I'd defer to DCops mgmt is connected to un-managed switches, so we don't have much visibility on this side. Either the...
[13:41:31] <wikibugs>	 10serviceops, 10Continuous-Integration-Infrastructure, 10DC-Ops, 10netops: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ) - https://phabricator.wikimedia.org/T283582 (10Marostegui)
[14:39:40] <wikibugs>	 10serviceops, 10SRE, 10MW-1.35-notes (1.35.0-wmf.34; 2020-05-26), 10Patch-For-Review, 10Platform Engineering (Icebox): Undeploy graphoid - https://phabricator.wikimedia.org/T242855 (10Seddon)
[14:53:13] <wikibugs>	 10serviceops, 10SRE, 10MW-1.35-notes (1.35.0-wmf.34; 2020-05-26), 10Patch-For-Review, 10Platform Engineering (Icebox): Undeploy graphoid - https://phabricator.wikimedia.org/T242855 (10Seddon) a:05Seddon→03None
[16:15:28] <legoktm>	 _joe_, mutante: ok, same time as last time?
[16:15:56] <_joe_>	 legoktm: i'm open to any time before 18:00Z
[16:16:04] <legoktm>	 so far I have all the puppet patches prepped for LVS, I think the helmfile.d stuff is ready, I just need to do the secrets now, which is generating the certs and secret key
[16:16:32] <_joe_>	 that seems correct, yes
[16:17:36] <legoktm>	 sent invite for 16:00
[16:17:49] <jayme>	 for all of you dealing with docker-registry in some sort. I wrote a initial set of httpbb tests https://gerrit.wikimedia.org/r/c/operations/puppet/+/694552
[16:18:38] <_joe_>	 jayme: so I finally tricked a poor soul into doing it
[16:18:42] <jayme>	 because it's tedious and a mess and nobody remembers what the "right" answer is
[16:18:45] <_joe_>	 \o/
[16:19:05] <_joe_>	 kudos :)
[16:19:30] <jayme>	 nah, you did not trick me. I already said I would do it the next time I need to work on the registry :P
[16:20:22] <jayme>	 unfortunately some things we're not able to test (like positive tests for the restricted stuff). Or at least I have no idea how one would do theat
[16:20:24] <jayme>	 *that
[16:20:40] <_joe_>	 we can work on that tomorrow :)
[16:20:42] <jayme>	 and that weird restricted IPs trickery
[16:21:10] <jayme>	 shit, I thought it's simply not possible and I'm done with this
[16:21:12] <_joe_>	 and that might need some additional patches to httpbb and the registry (adding the cumin hosts as authorized to upload there)
[16:21:44] <_joe_>	 to httpbb because we want to pass the registry credentials to it from a secret file :)
[16:21:48] <jayme>	 ah, yeah. With patching httpbb it is possible ofc
[16:22:12] <_joe_>	 rzl: ^^
[16:23:07] <jayme>	 he does not have a v- ;olans like highlight on httpbb? :P
[16:31:11] <_joe_>	 we also hire people who're not off the charts on the OCD scale, yes.
[16:35:29] <rzl>	 ahaha reading back
[16:36:36] <rzl>	 jayme: ah, sorry about the static timeout -- I can make that a command-line flag or something, if you'd prefer?
[16:37:05] <jayme>	 rzl: don't worry. It's not critical and I already prepared a patch...I guess
[16:37:37] <rzl>	 haha "# TODO: Consider making this number configurable." I see past-me left this as a gift for now-me
[16:37:55] <rzl>	 I think I couldn't decide if I wanted it as a flag, or a test-case parameter
[16:37:59] <jayme>	 indeed
[16:39:47] <rzl>	 and, for the registry credentials, sounds good to me but I probably won't have time to learn how to do it soon :) patches welcdome
[16:39:50] <rzl>	 *welcome
[18:23:06] <wikibugs>	 10serviceops, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar), 10User-jijiki: Enable TLS on memcached for cross-dc replication - https://phabricator.wikimedia.org/T271967 (10jijiki)
[19:43:55] <wikibugs>	 10serviceops, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar), 10User-jijiki: Enable TLS on memcached for cross-dc replication - https://phabricator.wikimedia.org/T271967 (10jbond) i wonder if we have considered just having the TLS port every where accept localhost?
[19:45:58] <wikibugs>	 10serviceops, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar), 10User-jijiki: Enable TLS on memcached for cross-dc replication - https://phabricator.wikimedia.org/T271967 (10jbond) > i wonder if we have considered just having the TLS port every where accept localhost? regardless i guess we need a t...