[08:20:13] <dpogorzelski>	 hmmm it seems that the deployment-charts repo is dirty on the deployment host, is it ok to git reset?
[08:20:18] <dpogorzelski>	 https://www.irccloud.com/pastebin/GhLS8EHH/
[08:22:48] <Emperor>	 ottomata: I'm guessing those changes were by you? ^--
[08:57:44] <_joe_>	 dpogorzelski: given the changes are just positional, I'd assume so
[08:59:01] <_joe_>	 but also: yes, unless the repo has been made dirty in the last few hours, assume it is in general :)
[09:32:41] <btullis>	 This might be related to work done on T417407 by brouberol. I think he's afk for a little while.
[09:32:41] <stashbot>	 T417407: Evacuate all kafka-mirrormaker instances to Kubernetes - https://phabricator.wikimedia.org/T417407
[09:35:05] <brouberol>	 sorry. that was me, and shouldn't have stayed dirty except my daughter barged into my office
[09:35:17] <brouberol>	 temporary mayhem ensured and I forgot 
[09:39:46] <_joe_>	 brouberol: good root cause analysis
[09:40:18] <_joe_>	 have you set up the proper failsafes for the future, like electrifying your offices' door?
[09:40:41] <brouberol>	 I was thinking about hiring a Real Chaos Gorilla as a bodyguard 
[09:40:52] <_joe_>	 that would also work I guess
[09:41:03] <brouberol>	 or printing "YOU SHALL NOT PASS 🧙" on my door
[09:42:09] <_joe_>	 brouberol: you probably just need to master the "annoyed joe stare", though. It worked pretty well with my daughter when she was a kid
[09:42:40] <_joe_>	 it just takes some practice not to let the fact you're delighted to leak out
[09:42:56] <p858snake|cloud>	 gorillas? train a pool of blåhaj's and place lasers on their heads?
[09:43:15] <p858snake|cloud>	 or is it a pod when it comes to sharks?
[09:50:13] <claime>	 https://clip.cafe/beneath-the-planet-of-the-apes-1970/if-are-caught-by-the-gorillas-s2/
[10:18:35] <Emperor>	 I thought that after https://gitlab.wikimedia.org/repos/sre/vopsbot/-/merge_requests/20 was merged, saying !ack would ack all unacked incidents. But when we had 3 p.ages fire in quick succession this morning and I said !ack, sirenbot said 'Could not ack the alert. Please check the parameters.' ... ?
[13:44:12] <matthieulec>	 Emperor: the pages of this morning are acknowledged but not resolved, should they be resolved?
[14:02:47] <Emperor>	 matthieulec: oh, good catch, yes, I'll do that
[14:02:49] <Emperor>	 !incidents
[14:02:49] <sirenbot>	 7784 (ACKED)  db1253 (paged)/MariaDB Replica IO: s7 (paged)
[14:02:50] <sirenbot>	 7785 (ACKED)  db1253 (paged)/MariaDB Replica Lag: s7 (paged)
[14:02:50] <sirenbot>	 7786 (ACKED)  db1253 (paged)/MariaDB Replica SQL: s7 (paged)
[14:02:52] <Emperor>	 !resolve
[14:02:53] <sirenbot>	 Could not resolve the alert. Please check the parameters.
[14:02:58] <Emperor>	 I'm sure that's meant to work now :(
[14:03:06] <Emperor>	 !resolve 7784-7786
[14:03:06] <sirenbot>	 Incident id must be an integer
[14:03:10] <Emperor>	 !resolve 7784
[14:03:10] <sirenbot>	 7784 (RESOLVED)  db1253 (paged)/MariaDB Replica IO: s7 (paged)
[14:03:12] <Emperor>	 !resolve 7785
[14:03:12] <sirenbot>	 7785 (RESOLVED)  db1253 (paged)/MariaDB Replica Lag: s7 (paged)
[14:03:13] <Emperor>	 !resolve 7786
[14:03:14] <sirenbot>	 7786 (RESOLVED)  db1253 (paged)/MariaDB Replica SQL: s7 (paged)
[14:03:45] <volans>	 7784-7786 is an integer, it's -2 :-P
[14:11:32] <Raine>	 Emperor: pretty sure it worked for a while, no idea what broke it
[14:37:04] <Emperor>	 nice to know it's not just me going mad :)
[14:40:05] <taavi>	 looking at the logs that apparently fails with 'error="invalid character 's' looking for beginning of value"' so seems like a sirenbot bug?
[15:02:33] <rzl>	 yeah I was looking at that on Friday, I got as far as tracking it down to "incidentIds is [], which should never happen" but not as far as why it does anyway :)
[16:48:54] <topranks>	 sukhe: we can re-enable doh in magru and esams when you have a minute 
[16:51:02] <sukhe>	 topranks: ok thank you
[16:51:08] <sukhe>	 I see the updated comment on the task
[16:51:15] <sukhe>	 I can re-enable it and we can observe for a few hours I guess?
[16:52:46] <topranks>	 yep, worth taking a look at the conntack numbers in a few hours 
[16:53:16] <topranks>	 I tested thoroughly so I'm pretty sure everything will be ok 
[16:53:27] <sukhe>	 yep, enabling it now
[16:58:38] <sukhe>	 topranks: all done
[16:58:52] <topranks>	 lgtm 
[16:59:08] <topranks>	 https://www.irccloud.com/pastebin/NrwA4HX2/
[16:59:21] <sukhe>	 yep
[16:59:24] <topranks>	 https://www.irccloud.com/pastebin/bS1RQC5d/
[16:59:28] <sukhe>	 https://grafana.wikimedia.org/goto/ffgwib37fs16oc?orgId=1
[16:59:42] <topranks>	 can't test magru from here but I need to still undo the outbound announce policy there 
[17:00:25] <sukhe>	 ; NSID: 64 6f 68 37 30 30 33 ("doh7003")
[17:00:35] <sukhe>	 bast7003 fwiw :P
[17:00:49] <topranks>	 cool
[17:01:02] <topranks>	 one thing to watch out though - internet is not routing it to magru right now 
[17:01:08] <topranks>	 bast7003 will hide that 
[17:01:18] <topranks>	 useful nonetheless of course 
[17:01:54] <sukhe>	 ja
[17:02:02] <sukhe>	 ha true
[17:47:58] <cwhite>	 Grafana will go down for a short reboot at 1800Z.
[18:17:28] <rzl>	 opened https://phabricator.wikimedia.org/T420982 for that vopsbot bug
[19:31:14] <swfrench-wmf>	 inflatador: is your puppet patch good to merge?
[19:32:04] <swfrench-wmf>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1251117 specifically
[19:34:35] <inflatador>	 swfrench-wmf yes, please do
[19:34:42] <inflatador>	 sorry for the lag on that
[19:34:52] <swfrench-wmf>	 inflatador: ack, will do :)
[19:35:48] <inflatador>	 Unrelated, but it looks like one of our discovery certs will expire fairly soon: issuer `CN=Wikimedia_Internal_Root_CA,OU=Cloud Services,O=Wikimedia Foundation\, Inc,L=San Francisco,ST=California,C=US', serial 0x715331115b69e7112b0e3c7f8c89ce15c51a4639, EC/ECDSA key 528 bits, signed using ECDSA-SHA512, activated `2021-05-04 13:54:00 UTC', expires `2026-05-03 13:54:00 UTC', pin-sha256="PbgfDlEHVB4Zw0a42zNqqnEQbcYF9jYp/dbT4eSdOb8="`
[20:58:03] <andrewbogott>	 oops, was today 'no more apt mirrors' day?
[20:59:00] * inflatador noticed that too
[20:59:41] <andrewbogott>	 I knew it was coming and yet find myself unprepared
[21:11:27] <inflatador>	 Oh, were they supposed to go away? I just noticed I couldn't install a package on one of the cirrussearch hosts
[21:16:31] <andrewbogott>	 I don't know what's happening.  https://phabricator.wikimedia.org/T416707 predicts future demise but implies that it's still in the future
[21:16:53] <andrewbogott>	 but meanwhile...  no connectivity to mirrors.wikimedia.org
[21:17:08] <andrewbogott>	 so maybe something is broken by accident today and I mistook it for an early symptom of the future removal
[21:31:04] <inflatador>	 maybe so...regardless, I guess we would have to one-off the linked change onto our servers if we wanted to get a package
[22:23:51] <rzl>	 ^ looking at this
[22:25:24] <rzl>	 we haven't made any of the prerequisite changes yet (like https://gerrit.wikimedia.org/r/1256371, or updating modules/package_builder/manifests/pbuilder_base.pp or modules/docker/templates/images/sourceslist.base.erb) so this is an unrelated breakage
[22:42:41] <rzl>	 great news, I didn't do anything and it came back on its own
[22:42:51] <rzl>	 like and follow for more SRE life hacks
[23:56:10] <bd808>	 rzl: I !bash'ed you at https://bash.toolforge.org/quip/gA8gHZ0Bvg159pQrJzGF