[00:02:13] (DiskSpace) resolved: Disk space puppetmaster1001:9100:/ 5.105% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=puppetmaster1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [03:34:26] (SystemdUnitFailed) firing: docker-reporter-k8s-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:43:39] 10SRE-tools, 10Infrastructure-Foundations, 10Spicerack: Spicerack: add distributed locking support - https://phabricator.wikimedia.org/T341973 (10Volans) Disributed locking is now live in Spicerack and used by the Cookbooks. For a general overview see https://doc.wikimedia.org/spicerack/master/introduction.h... [06:53:33] FYI I'll retry a reimage of sretest1001 [07:10:01] volans: congrats on releasing the locks! [07:10:22] I'm acquiring them though :-P [07:10:43] reluctantly releasing them afterwards :D [07:24:08] :) [07:29:38] thx :D [07:31:08] volans: does it collect stats? who have the most locks, who have the longest combined lock duration, etc? [07:31:09] :) [07:31:38] rotfl, no [07:32:01] who's most locking other out :D [07:38:37] (SystemdUnitFailed) firing: docker-reporter-k8s-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:48:37] (SystemdUnitFailed) firing: (2) docker-reporter-k8s-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:48:38] (SystemdUnitFailed) firing: (2) docker-reporter-k8s-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:45:06] jbond: I rewrote my pcc-debug-presentation script last night so that it now generates the build index page (and against different kinds of hosts [cancelled, core_diffs, diff, noop etc]) [09:45:31] which led me to fix some issues in the HTML output like a lonely `` for cancelled hostnames [09:45:48] and the `

` elements which was not closed messing up with font size in the host listed underneath [09:45:49] :) [09:46:19] it is fragile still, but at least lets one "easily" get a rendering of the pages [09:46:58] no rush for merging the series though, I ultimately wanted to enhance the style of the PCC report :) [09:58:37] (SystemdUnitFailed) firing: (2) docker-reporter-k8s-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:59:42] hashar: thanks [10:00:26] ultimately I'd like the ppc command to output a json report [10:04:26] (SystemdUnitFailed) firing: (2) docker-reporter-k8s-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:16:23] hashar: im lookiung at your changes and the border looks a bit stranged when there are long lines https://puppet-compiler.wmflabs.org/output/966221/1/cp1080.eqiad.wmnet/index.html [10:16:35] ideally we would have it so those lines automatically wrap [10:17:46] however that text is in 
 tags so im not sure we can do that
[10:31:01] 	 ok i think i have improved things would like some niput on if this looks good 
[10:31:06] 	 this is what we have 
[10:31:06] 	 Current version: https://puppet-compiler.wmflabs.org/output/966221/81/cp1080.eqiad.wmnet/index.html
[10:31:10] 	 no overflow: https://puppet-compiler.wmflabs.org/output/966221/1/cp1080.eqiad.wmnet/index.html
[10:31:13] 	 with overflow: https://puppet-compiler.wmflabs.org/output/966221/1/cp1080.eqiad.wmnet/index.html
[10:31:33] 	 hashar: slyngs: volans: (anyone else) what do you think ^^^ pcc css changes 
[10:31:50] 	 jbond: overflow of what, long changes?
[10:31:59] 	 yes
[10:32:03] 	 the last 2 links are the same
[10:32:17] 	 oh damm
[10:32:23] 	 one sec i overwrote it
[10:32:26] 	 k
[10:33:03] 	 Honestly I actually like the original
[10:34:56] 	 ok here are the links https://phabricator.wikimedia.org/P53012
[10:34:56] 	 I like the new one tbh, just the readability of the Total Resources:    3339 is I think harder now than before
[10:36:07] 	 jbond: absolutely with overlow, but I'd add something visual that tells the user there is more text, because if the content is truncated at a space it might be easily missed
[10:36:25] 	 I'd trust the no overflow the most, in the sense that I'd feel more secure about the output being exactly what shows up on the server
[10:37:25] 	 at least in chrome the scrollbar (that is a visible clue) is hidden until you over on the text overflowed
[10:37:34] 	 and start scrolling
[10:37:38] 	 not even hovering
[10:37:40] 	 Same in Firefox
[10:37:57] 	 slyngs: im not sure i follow that, the data is the same regarless of the overflow, its just a matter of scrolling in a box or the page
[10:38:07] 	 volans: as to visual cluse any suggestions?
[10:38:35] 	 BLINKING ARROW :-)
[10:38:40] 	 lo
[10:38:41] 	 l
[10:39:18] 	 The no overflow I can see the text continuing of screen
[10:39:27] 	 So I know that I can scroll
[10:39:46] 	 ahh ok so the same as volans "we need some visual clue if we use overflow"
[10:39:47] 	 slyngs: what if there is a it of space? it's unlikely but you could miss it too 
[10:40:18] 	 have you tried overflow-x: scroll; ?
[10:40:26] 	 I guess we care only about the horizontal one
[10:40:51] 	 but I'm not sure if the browser will hide it anyway
[10:40:57] 	 volans: i will try, ftr assum i have tried nothing, i try to avoid css as much as possible ;)
[10:41:04] 	 me too
[10:41:06] 	 I google it :D
[10:41:07] 	 Or stuff in a unicode symbol like ⇨
[10:42:15] 	 ok, with overflow-x: scroll; it adds a small gray line a the bottom-left of the text area
[10:42:22] 	 not sure it too visual but it's there
[10:42:40] 	 I mean, not sure if as a visual is enough to be seen
[10:44:45] * jbond would have been quicker to just change the raw html
[10:44:57] 	 https://puppet-compiler.wmflabs.org/output/966221/2/cp1076.eqiad.wmnet/index.html
[10:44:59] 	 I use the inspector :D
[10:45:13] 	 fyi i also drop the serif-sans font
[10:45:29] 	 afaikt its the same as before
[10:45:29] 	 still overflow:auto AFAICS
[10:45:41] 	 doh!
[10:45:43] 	 font is beter, thx
[10:47:42] 	 still not much different to me https://puppet-compiler.wmflabs.org/output/966221/2/cp1076.eqiad.wmnet/index.html
[10:47:58] 	 Firefox still hides the scrollbar, but I don't think there's much that can be done about that
[10:48:19] 	 oh yes its more obvious in chrome
[10:48:33] 	 Firefox briefly shows it and then hides it
[10:48:51] 	 yeah same for chrome in most cases, weird...
[10:49:08] 	 I was looking and apparently there isn't a css-only way to apply a style *only* if the content is oveflowing
[10:50:06] 	 you could add autoscroll :-P
[10:50:08] * volans hides
[10:50:39] 	 ok i think im going to go with this, if people really hate it we can roll back
[10:50:43] 	 Just put it in a marquee tag :-)
[10:50:47] 	 ak
[10:50:49] 	 *ack
[10:52:03] 	 It will be fine, people normally have an expectation about line length I think, so they should notice
[10:52:15] 	 agree 
[10:52:51] 	 Still think that the  is the way of the future though
[11:15:31] 	 jbond: nice work on the ppc 2.x branch :)
[11:16:08] 	 pushing the merge commit might have caused Gerrit to automatically mark as merged the series of changes I had to master
[11:16:11] 	 but I am not so sure
[11:16:21] 	 the topology looks great in the end :)
[11:21:21] 	 hashar: thanks
[11:38:42] 	 volans: quick question about the lock, does it wait?
[11:39:01] 	 looking at https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/967167
[11:39:27] 	 XioNoX: it polls, see https://doc.wikimedia.org/spicerack/master/introduction.html#distributed-locking
[11:39:36] 	 up to ~30m
[11:39:47] 	 it's a usual retry
[11:40:07] 	 ok cool
[11:40:28] 	 then +1
[11:40:35] 	 and shows you the current proceeses holding the locks
[12:53:49] 	 topranks: there are pending dns changes for sretest2003
[12:53:56] 	 should I commit them? I'm running the dns cookbook
[12:57:16] 	 volans: sorry yep please do 
[12:57:32] 	 my bad was adding and removing the IP a bunch of times must have left out last run 
[12:58:45] 	 k doing
[12:59:29] 	 {done}
[12:59:42] 	 volans: thanks :)
[13:07:39] 	 topranks: https://prometheus-eqiad.wikimedia.org/ops/classic/graph?g0.range_input=1h&g0.expr=node_net_ethtool_info%7Binstance%3D%22sretest1003%3A9100%22%7D&g0.tab=1 DATA 
[13:09:09] 	 slyngs: ok nice!
[13:09:18] 	 I'll have a dig in a little later see what is there 
[13:09:51] 	 It's only for sretest for now, but you have node_net_ethtool and node_net_ethtool_info metrics
[13:11:01] 	 cool yeah - I see all the stats there under node_net_ethtool alright 
[13:11:17] 	 good stuff, I'll see if I can add some decent graphs and work on the alerting when I get a moment 
[13:12:02] 	 I'll leave it to our talented network administrators to figure out what kind of data i actually useful :-)
[13:13:36] 	 In true "talented network administrator" style I'm graphing this for my home pc and router so already halfway there :P
[13:15:46] 	 Should have asked you to do my network cables. I think I did one of them wrong, it only does 100Mbit for some reason
[13:29:37] 	 that's awesome
[13:29:46] 	 100% that's the cable alright, probably didn't push the wires all the way in fully 
[13:29:57] 	 (I mean ethtool exporter)
[13:29:58] 	 1G uses all 8 wires in the RJ45, 100Mb only uses middle 4
[13:32:02] 	 I never used them but recently seen these "pass through" RJ45 plugs you can get to ensure the wires all reach to the end and make contact 
[13:32:04] 	 https://usercontent.irccloud-cdn.com/file/mWQbKxGF/image.png
[13:34:48] 	 slyngs: if yuor crimping your oiwn cables/sockets etc id definetly recomend getting a cable tester, i have a cheep one like this which is fine for my needs https://www.amazon.es/deleyCON-Comprobador-Cables-Conexi%C3%B3n-Conductividad/dp/B09MQQR5LQ/
[13:35:13] 	 but yuo can get more expensive fgancy ones that do all sorts of magik
[13:36:39] 	 I have keystones, but maybe they where a bit to cheap. The brand named one for Schneider is 165DKK and the cheap ones are 25DKK, so I got the cheap ones for the end in the wiring closet
[13:37:32] 	 ack 
[13:39:03] 	 yeah the cheap testers are fine... the more expensive ones tend to also test the electrical characteristics (bandwidth, crosstalk, impedence or whatever) and certify you comply with cat5/6/7 standard or some ish 
[13:39:13] 	 for typical small cable jobs I don't think you'd really need one 
[13:39:37] 	 jbond: I actually have that exact tester :-)
[13:39:58] 	 about keystones I'm always dubious about the benefits of more expensive connectors (the audio world has probably made me sceptical) 
[13:40:41] 	 You mean like Denons "special" audio ehternet cable
[13:41:40] 	 yeah or these bargain kettle leads: https://www.hifihut.ie/collections/power-cables/products/audioquest-nrg-y3-ac-power-cable-uk-plug-iec-c-13?variant=32761477922919
[13:41:59] 	 lol 
[13:42:57] 	 You should get that and just use it for your kettle, you know, to get a cleaner boiling
[13:43:51] 	 Tea tastes sooo much better.. well worth it :P
[13:51:43] 	 :D hehehe
[13:52:05] * jbond wouldn;t mind having a bussiness with thier customers
[14:08:38] 	 (SystemdUnitFailed) firing: docker-reporter-k8s-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[15:22:13] 	 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: CRs ECMP traffic to LVS VIPs despite higher MED on backup route - https://phabricator.wikimedia.org/T348446 (10cmooney) This is now fixed in esams, solution that's been applied is to add a community on sessions to LVS servers if the MED is...
[15:26:52] 	 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: CRs ECMP traffic to LVS VIPs despite higher MED on backup route - https://phabricator.wikimedia.org/T348446 (10ssingh) Thanks, confirming this is working: https://grafana.wikimedia.org/d/000000343/load-balancers-lvs?orgId=1&viewPanel=33
[15:28:06] 	 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: CRs ECMP traffic to LVS VIPs despite higher MED on backup route - https://phabricator.wikimedia.org/T348446 (10cmooney) Actually there is a caveat, traffic from other servers on asw1-bw27-esams will still route out via lvs3010, until I impl...
[15:47:00] 	 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: CRs ECMP traffic to LVS VIPs despite higher MED on backup route - https://phabricator.wikimedia.org/T348446 (10cmooney)
[18:08:38] 	 (SystemdUnitFailed) firing: docker-reporter-k8s-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[22:08:38] 	 (SystemdUnitFailed) firing: docker-reporter-k8s-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[22:20:13] 	 (DiskSpace) firing: Disk space puppetmaster1001:9100:/ 5.943% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=puppetmaster1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace