[00:00:37] Thank you so much! (re @wmtelegram_bot: @ederporto: your tool has credentials now.) [12:03:42] Looks like I have to fix the kubernetes config for PAWS right away. [12:03:58] Error: Error waiting for openstack_containerinfra_cluster_v1 1739512b-de9d-4d49-afa1-6361efd1f689 to become ready: context deadline exceeded [12:05:29] │ with openstack_containerinfra_cluster_v1.k8s_127a,│ on 127a.tf line 1, in resource "openstack_containerinfra_cluster_v1" "k8s_127a":│ 1: resource "openstack_containerinfra_cluster_v1" "k8s_127a" { [12:14:28] atrawog: also, magnum is known to be misbehaving after the last openstack upgrade, so you may want to check with andrewbogott later for additional information regarding that error [12:44:35] arturu: Yes, it's either that the old PAWS config isn't working with magnum, or that there still some issues with the new deployment. [13:37:36] magnum was working properly for me in codfw1dev, but maybe 70% of the time? So it's always worth retrying. [14:13:29] atrawog: ^^^ try again? [20:25:25] jelto: Puppet is failing on (I think) all gitlab runners, due to a failed lookup for profile::gitlab::runner::docker_gc_interval. I don't see that anything has recently changed in that area, do you have a theory? [20:26:44] for now I'm going to set profile::gitlab::runner::docker_gc_interval: 5m in project-wide config on horizon [20:33:15] hm, didn't help, it's just failing on the next var [21:12:28] andrewbogott: i know what it is and I can fix it. the point was to remove that from cloud.yaml and not have it in multiple places, including Horizon [21:13:09] mutante: what changed? [21:14:42] andrewbogott: removed from cloud.yaml and host-level to avoid that settings are all over the place. we have an entire task about "Hiera cleanup" [21:14:54] the mistake here was to expect that puppet role level is applied in cloud [21:15:27] I am going to move it to "project level" at least. [21:15:36] oh /now/ I see the patch, I was looking in entirely the wrong place [21:15:37] thanks! [21:16:21] I am foreseeing a future where you can't do project-level without saying if it's eqiad or codfw.. but luckily not yet [21:17:43] sigh, ok.. adding 2 lines back to cloud.yaml .. because there will be a gitlab-runner in 2 projects [21:17:56] we are going to have an entire meeting to define which way we consider correct .. [21:28:09] andrewbogott: waiting for the local master to sync from the global masters..i hope it does it at some point without a manual pull. generally.. if only there was a fix to just use the normal puppet role hiera by default.. and only set different values for cloud where it actually is supposed to differ.. oh well [21:28:21] I think in the great majority of cases this is done in horizon, but of course that's not cross project at all [21:29:43] that would be a lot of manual copy/pasting of values you can't search for [21:31:16] the upside to using Horizon Hiera is that people can change it without needing +2 but it also has a bunch of downsides. [21:31:48] ironically this change was trying to fix the situation a little bit but here we are [21:33:06] alright, it's fixed now. I verified puppet run on random runner-1027 [21:33:40] you can expect the rest to recover, laters! [21:34:44] thx! [21:36:18] no no, I should have checked this after merging it. just for some reason I did not get the nagging emails for this [21:36:28] while I do get them for devtools project instances [21:37:00] normally I would react to broken puppet mails, so no worries. cu [22:50:42] I'm going to make a bunch of noise here cleaning up older wm-bot aliases. [22:51:00] !obs del [22:51:00] !osb is https://openstack-browser.toolforge.org/project/$1 [22:51:00] Unable to find the specified key in the database [22:51:01] This key already exists - remove it, if you want to change it [22:51:09] !obs del [22:51:09] Unable to find the specified key in the database [22:51:20] !osb del [22:51:20] Successfully removed osb [22:51:26] !osb is https://openstack-browser.toolforge.org/project/$1 [22:51:26] Key was added [22:51:48] !project alias osb [22:51:48] Created new alias for this key [22:52:00] !sal del [22:52:01] Successfully removed sal [22:52:02] !sal is https://sal.toolforge.org/$1 [22:52:02] Key was added [22:52:08] !intro del [22:52:08] Successfully removed intro [22:52:15] !intro is https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_introduction [22:52:15] Key was added [22:52:21] !drift del [22:52:21] Unable to find the specified key in the database [22:52:44] !drift del [22:52:44] Unable to find the specified key in the database [22:52:47] huh [22:52:54] !replag del [22:52:54] Successfully removed replag [22:53:00] !replag is https://replag.toolforge.org/ [22:53:00] Key was added [22:53:18] !toolinfo del [22:53:19] Successfully removed toolinfo [22:53:23] !toolinfo is https://toolsadmin.wikimedia.org/tools/id/$1 [22:53:23] Key was added [22:53:36] !toolinfo bd808-test | bd808 [22:53:37] bd808: https://toolsadmin.wikimedia.org/tools/id/bd808-test [22:53:59] !access del [22:53:59] Successfully removed access [22:54:05] !access is https://wikitech.wikimedia.org/wiki/Help:Accessing_Cloud_VPS_instances [22:54:05] Key was added [22:54:12] !bastion del [22:54:12] Successfully removed bastion [22:54:42] !bastion is http://en.wikipedia.org/wiki/Bastion_host; WMCS's specific bastion host is bastion.wmcloud.org which should resolve to 185.15.56.87 see !access [22:54:43] Key was added [22:54:49] !bigbrother del [22:54:49] Unable to find the specified key in the database [22:55:04] !clouddocs del [22:55:04] Successfully removed clouddocs [22:55:05] !clouddocs is https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS [22:55:05] Key was added [22:55:12] !taskinfo del [22:55:12] Unable to find the specified key in the database [22:55:27] !xy del [22:55:27] Successfully removed xy [22:55:29] !xy is The XY problem is asking about your attempted *solution* rather than your *actual problem*. https://en.wikipedia.org/wiki/XY_problem [22:55:29] Key was added [22:55:36] !XY del [22:55:36] Successfully removed XY [22:55:42] !XY alias xy [22:55:43] Created new alias for this key [22:55:48] !2fa del [22:55:48] Successfully removed 2fa [22:56:18] /me is amused by reference [6] on !xy [22:58:32] I will refrain from making a `!zalgo` key inspired by that class copypasta :) [22:58:39] *classic [22:58:45] :D [22:59:03] we can sell it as stress-testing the unicode handling in matterbridge [23:00:20] !zalgo is https://zalgo.org/ [23:00:20] Key was added [23:00:34] !zalgo | @lucaswerkmeister [23:00:34] @lucaswerkmeister: https://zalgo.org/ [23:01:01] less stressful, but hopefully still fun :) [23:01:29] (CW unreadable) t̸ ͅh̷̹̾ë̸̢́ ̷̝́l̷̔͜ȩ̶̈́g̷̡̾ä̶͖́c̸̼͝ỳ̷͙ ̶̖͊p̶̺̽a̷̮͊r̶͖̽s̵̯̎e̵̫͆ř̸̲ ̴̫̅s̸ ̗h̶̹̓ậ̷ľ̶͜l̵͓̓ ̴͖̀r̷̙͘ī̸̢ś̵̭e̷̼̕ ̸̢̑ả̸̙ǵ̶̯ä̵͍i̶͎͑n̶̹͒ [23:01:59] subbu may have something to say about that, but I can't rule it out [23:02:14] (tbf I think it’s actually been several years now since I saw matterbridge mangle unicode – seems to be solid these days) [23:02:44] “not if anything to say about it – I have!” *force pushes*