[10:55:42] I think https://wikitech.wikimedia.org/wiki/Incident_status needs some lua expert - I am seeing 2023 incidents classified under the 2022 headline [11:00:26] I think the issue is https://wikitech.wikimedia.org/w/index.php?title=Incident_status&diff=prev&oldid=1951858 and https://phabricator.wikimedia.org/T49137 [12:54:40] fyi im going to disable puppet fleet wide to deploy a change [13:34:09] jbond: let me know when it's enabled back (no rush) [13:35:06] XioNoX: sorry should be aneabled now [13:35:10] (enabled [13:35:14] cool! thx [14:02:49] hello! does this mapping from team to phab tags sounds correct to people? The scope is network maintenance, so the target is SREs (or embedded SREs) [14:03:06] https://www.irccloud.com/pastebin/BVWutGI3/ [14:22:35] lgtm [14:24:27] If I want to paste logs with IPs into a phab ticket ( T327253 ) should I mark it as a security item since those IPs are personal data (they're not all internal)? I assume so... [14:24:29] T327253: >=27k objects listed in swift containers but not extant - https://phabricator.wikimedia.org/T327253 [14:24:52] [I don't really want to have to remember to remove IP addresses from logs, since I'm likely to forget at some point] [14:25:00] Emperor: yes, the other option is to use a private paste in Phab [14:25:16] https://phabricator.wikimedia.org/paste/edit/form/36/ [14:25:27] Oh, yes, maybe that's better, then I can keep the main ticket open [14:25:31] that would let you keep the task public but the data --- yeah :) [14:25:38] XioNoX: I don't think that #data-engineering-operations tag has ever been used in phabricator yet. It might be better just to drop the `-operations` suffix for our team. [14:25:54] ok! [14:26:26] Emperor: btw another Phab markup trick -- you can reference the paste id in the task with curly braces around it, and it will embed [14:26:35] like {P12345} [14:26:58] that is neat; does that DTRT for a protected paste ina public ticket? [14:27:04] yes [14:27:19] sweet [14:30:03] XioNoX: looks good, technically the traffic one is #Traffic but I guess that phabricator doesn't care too much about it [14:30:16] yeah [14:39:53] While I'm asking silly questions, does cumin have options for "I just want the output from my commands, none of the surrounding metadata"? e.g. to make it easier to do cumin hosts 'grep runes' | sort ... [14:40:03] is grafana working for others? [14:40:37] cdanis: WFM [14:41:12] cdanis: Also working for me. [14:41:15] cdanis: wfm too [14:41:33] https://i.imgur.com/vwxSIPr.png [14:42:11] :( [14:42:28] Ah [14:42:41] I have the same when I click Dashboards -> Home [14:42:51] But the actual dashboards work [14:43:01] https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red?orgId=1 [14:44:53] Emperor: You can do txt or json output, which is more suitable for parsing: https://wikitech.wikimedia.org/wiki/Cumin#Output_handling There are also the `--no-progress` and `--no-colors` options you can use. [15:08:11] Emperor: what btullis said, also there is an interactive option to get dropped into a REPL shell and mangle the data there in python directly [15:08:30] or use cumin as a library, or write a cookbook, depending what you need :) [15:12:06] so far, I need some log output that's actually useful :( [ P43346 distinctly un-useful ] [15:13:33] :( [15:23:05] we're restoring the home dashboard in -observability FWIW [15:26:27] do we know why it disappeared? [15:29:59] godog: just to make sure: did you see https://gerrit.wikimedia.org/r/c/operations/puppet/+/871290 already? [15:31:04] volans: yes it's been deleted [15:31:24] taavi: thank you! that wasn't the previous home for grafana.w.o though [15:43:34] !log netbios wins disabled on db1140 ilom and ilom reset T327877 [15:43:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:43:38] T327877: Disable NETBIOS on some IPMI - https://phabricator.wikimedia.org/T327877 [15:44:15] I opened https://phabricator.wikimedia.org/T327925 for codfw row A switch upgrade, I'll send more communication via email, etc. It will be the first of 8 significant upgrades [15:46:11] hey all, I am having a laptop drama [15:47:40] godog: I deleted the memcached-historic-data (or something like that) dashboard [15:48:21] at least, that was my intention [15:48:34] effie: maybe grafana barfed? I still see this https://grafana.wikimedia.org/d/000000586/memcache-historic-data?orgId=1 [15:49:05] this is quite odd, because I am 100% sure that is what I was looking at [15:49:10] when I hit delete [15:49:26] effie: was it perhaps the home dashboard instead? [15:49:35] no no [15:49:56] I mean, it would amek more sense for me to delete accidentally the memcached dashboard [15:50:07] or any other memcached* one, than the home one [15:50:22] !log db1139 ilom wins/netbios disabled and ilom reset T327877 [15:50:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:50:28] T327877: Disable NETBIOS on some IPMI - https://phabricator.wikimedia.org/T327877 [15:50:30] now, I had opened this dashboaerd yesterday, and my session on chrome was restored [15:50:42] from what was left of my home directory on my laptop [15:50:53] after some drama I am sparing you the details of [15:51:29] godog: sorry I put you lot in trouble [15:52:01] effie: no worries, but yeah maybe the reload/restore had sth to do with it, not sure [15:52:15] it is good to exercise backup restore [15:52:45] now, my firefox data were complete gone so [15:52:55] if I were using ff for work, this wouldnt happen [15:53:02] as ff started from scratch [15:53:22] I might be outdated on the topic... do we still have puppet-driven dashboards and UI-edited ones? [15:53:36] Emperor: You've gotten your share of nags regarding T306098 but there's another easy task you can do for that project... 'swift' is shown as potentially abandoned on https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge. [15:53:36] T306098: Cloud VPS "swift" project Stretch deprecation - https://phabricator.wikimedia.org/T306098 [15:53:39] yes that's correct volans [15:53:54] should the homepage be part of the puppet ones? [15:54:03] Emperor: I've been sending nags to cloud-announce about that Purge page, maybe you're not subscribed to that list? [15:54:04] I guess is not as youre restoring it from backups [15:54:27] (oops, sorry for talking in the middle of another conversation!) [15:54:58] volans: yeah we used to provision it from json, then switched to database with the change taavi mentioned earlier [15:55:32] I think it'd be better to store it in grizzly tbh, that would help avoid collisions between envs that we saw yeah what godog mentioned [15:56:25] k [15:56:29] thanks for the context [15:59:48] is there a plan to move most dashboards to grizzly? [16:03:43] not currently, although thinking about this today it'd make sense to consider moving important dashboards to grizzly even as static json [16:05:14] I'm thinking on a folder by folder basis. where UI is be used to prototype quickly, and json exported to grizzly for deployment/updates to the live dashboard [16:21:51] I did put some things in motion sooner than intended, didn't I [16:27:44] I have made some changes on bacula on command line- these persist unless bacula is restarted- so please ping me if you need to restart bacula before 4 am UTC [16:32:01] fyi I'm planning to start a rolling reboot of kafka-jumbo in about 5 minutes as part of T325132 [21:13:54] brett: heh, the manual page from yesterday had been acked but not resolved so it just re-triggered -- i resolved it [22:38:26] cdanis: oh shit! Thanks