[01:45:17] * bd808 off [09:24:30] * arturo is back! [09:26:12] hello!!! [09:26:18] o/ [09:30:04] welcome back! [10:49:32] * arturo heads to gitlab [10:54:09] * taavi sends some invites to arturo [10:55:12] thanks, got them [10:55:16] it seems I can't access gerrit [10:55:35] hm, the account enabling hook probably did not work as expected. let me see [10:56:18] same for phabricator [10:58:00] your gerrit account should be back [10:58:16] what error do you get from phab? it seems like that was correctly re-enabled yesterday [11:01:11] one sec [11:02:51] https://usercontent.irccloud-cdn.com/file/E83Ji8uA/image.png [11:03:10] I'm back in gerrit [11:03:34] great, all my config and preferences has been retained [11:05:37] hey arturo welcome back! it feels like you never left :) [11:06:09] thanks dhinus !! I'm very happy to be back with you all [11:08:59] hm, I'm not immediately seeing what's wrong with the phabricator login. and you can log in to gerrit and other developer account backed services with the same credentials? [11:09:49] yes [11:11:15] taavi: wait, I used the `send me login link via email` button and now I'm in! [11:11:21] interesting [11:11:33] I tried disabling and re-enabling your account, hopefully I did not just kill your session [11:11:46] still inside [11:12:14] great [11:12:22] taavi: could you please un-archive this ? https://phabricator.wikimedia.org/project/manage/6583/ [11:12:35] yes, done [11:12:49] thanks! [11:16:34] arturo: any other access that I can grant you directly? I think the main thing you're missing at this point is shell access + LDAP and for those you need to go through the process [11:17:06] yeah [11:17:49] I'm missing access to office wiki [11:19:36] that goes through techsupport@ I believe [11:31:27] 👍 [11:37:37] We can link grafana to an arbitrary prometheus instance and make charts from that. But we don't have a central prometheus instance to store data from arbitrary sources (Such as kube-state-metrics) correct? As such if we want a dimension of time we will need to have the project with the desired data also maintain a prometheus instance? [11:39:23] Rook: yep, we do gather some data from the projects in the metricsinfra-prometheus, but the preference is that for more data, for each project to have their own prometheus (as that way they use their quotas, and the main metricsinfra setup keeps lighter for critical alerts) [11:39:45] Alrighty, thanks [12:40:48] ok to reboot cloudlb[2001-2003]-dev.codfw.wmnet ? probably so, given they are just test hosts, but wanted to doublecheck [12:41:52] Rook: ^ you were testing something there? [12:42:18] Probably, but they're fine to reboot [12:42:32] moritzm: yes, one at a time please [12:43:58] ack, going ahead now, one at a time [12:58:13] hmm, starting to think that a single binary for toolforge clis would ease many things: https://phabricator.wikimedia.org/T356377 [12:59:18] uh, if people want to programmatically call the toolforge apis they should use a http client or a wrapper library for that in their language instead of baking the CLIs in the image [12:59:53] cloudlb2001-dev failed to set up one VLAN as part of the reboot (journalctl -u networking), it's unlikely caused by the new kernel, but rather an issue which only got unveiled by the reboot? [13:00:00] the idea I had in mind back in the day was to explore how to generate such library from the openapi definition [13:01:11] we had the decision request yesterday in which we decided to generate python instead [13:02:06] taavi: would be nice to offer something easier, even if it's autogenerated libraries [13:02:21] yeah, saw the ticket [13:02:22] we decided to try generating a python wrapper for use in the `toolforge` CLI tool. that does not stop us or anyone else from generating other clients in other languages for other use cases [13:02:54] taavi: do you have the powers to add me to whatever group in phab that allows me to see hw procurement tickets? [13:03:04] moritzm: hmm, that interface is up properly [13:03:52] arturo: no :( that group is https://phabricator.wikimedia.org/project/members/29/ and apparently I'm not in it either [13:16:56] yeah, it's puzzling. might be simply some race [13:17:09] I'll proceed with 2002, then we know more [13:37:27] same error message on 2002 [14:15:38] Can we alter https://prometheus-paws.wmcloud.org/paws to https://prometheus-paws.wmcloud.org/ in grafana? I'm not sure if I permissions to do so [14:19:16] yes. the data source URLs are controlled by a file in the Puppet repository. [14:19:37] (I need to start heading towards my gate, otherwise I'd link it) [14:20:34] Sounds good, I'll put in a patch. maybe dcaro could you verify that I have https://prometheus-paws.wmcloud.org/ as the right link to update to? [14:21:38] Rook: that looks good yep https://prometheus-paws.wmcloud.org/api/v1/query?query=kubernetes_build_info&time=1706797265.357 works [14:24:11] hmm, manually testing the url on grafana seems to complain about internal plugin downstream error [14:26:13] https://www.irccloud.com/pastebin/mkJ62EgI/ [14:32:59] aahhh, even if I change the url on the UI, it's not really trying what I added xd [14:33:28] Though I may have saw the same error for toolforge, so there may still be something happening there [14:35:43] Could I get a +1 on https://gerrit.wikimedia.org/r/c/operations/puppet/+/995044 if it looks alright? [14:36:41] are you running prometheus by itself? or inside k8s?V [14:38:21] manually changed it and it seems to work, will be reverted by puppet until your change is merged though [14:39:32] It's inside of k8s. It will deploy as part of the blue green deploy [14:40:10] then all the paws/prometheus.pp part can be scratched when the VMs go away [14:40:37] That makes sense [16:53:58] hey andrewbogott [16:54:01] https://www.irccloud.com/pastebin/J4K7Px9I/ [16:54:49] note how the restricted bastion still needs my key [16:54:57] arturo: try now [16:55:33] dcaro: nope [16:56:01] hmm, it ran and modified the root sshkeys [16:56:12] https://www.irccloud.com/pastebin/GWzPqjlA/ [16:56:41] let me run the other bastions [16:58:46] arturo: did you ssh as root or as aborrero? [16:59:24] dcaro: I tried both, with same results [16:59:27] this returns empty: root@bastion-restricted-eqiad1-3:~# /usr/sbin/ssh-key-ldap-lookup aborrero [16:59:51] let me see if the wikitech account is online [17:00:20] * andrewbogott doing a few too many things at once, sorry [17:00:52] arturo: can you try again using root? I don't see any entries for those tries in the logs [17:01:32] I just imported the ssh key via wikitech -- was empty [17:01:45] xd [17:02:10] dcaro: [17:02:13] https://www.irccloud.com/pastebin/xJW9ygqo/ [17:02:14] that is probably related :) [17:03:23] mmm note how I use root but the proxy is still using aborrero [17:03:28] hmmm, that's ssh config I guess, your jump config [17:03:50] can you try sshing directly to the restricted bastion as root? [17:04:08] oh, ldap started showing up [17:04:20] https://www.irccloud.com/pastebin/RPVWrckK/ [17:04:59] this sounds different [17:05:01] Feb 01 17:03:32 bastion-restricted-eqiad1-3 sshd[2147536]: pam_access(sshd:account): access denied for user `aborrero' from `79.116.165.41' [17:05:16] groups and such [17:05:20] I may need to be added to projects [17:05:34] I added you as admin to toolforge [17:05:45] let me check if you're in bastion [17:06:10] I need to be in the bastion project [17:06:56] you're in bastion already [17:07:45] just added you to admin too [17:08:18] I tried + failed to add him to the admin tool in toolforge because I couldn't get the UI to work [17:09:05] it took a while, but he shows in the list [17:11:04] I'm inside login.toolforge.org now [17:11:20] \o/ [17:12:09] aborrero@tools-sgebastion-10:~$ sudo su [17:12:09] aborrero is not allowed to run sudo on tools-sgebastion-10. This incident will be reported. [17:14:17] we can sort my access tomorrow [17:15:12] I'll add you to sudo shortly [17:17:10] done, don't know how long it takes for the cache to refresh though [17:19:48] the sudo policy should be handled via tools.admin membership these days [17:22:24] arturo: you might have to create an account on toolforge then :) [17:23:19] * dcaro brb [17:23:26] I see my ssh key at https://toolsadmin.wikimedia.org/profile/settings/ssh-keys/ [17:27:20] hmm, you don't show up in the list of users when adding maintainers to tools.admin [17:27:33] (btw. I was going to say that that was my key xd) [17:28:01] mmmm I think there is something else to be done on the operations side to have me in the LDAP. Moritz mentioned something, but he is going to do it tomorrow [17:36:06] * bd808 is trying to figure out why Striker doesn't seem to see "Arturo Borrero Gonzalez" as a maintainer [17:46:01] * taavi assumes Arturo needs to log out of toolsadmin and back in for it to sync data from LDAP [17:58:25] Ok, "Arturo Borrero Gonzalez" is back in the tools.admin tool. Maybe a cache expiration? [17:58:55] I did the logout/login [18:30:59] * dcaro off [18:39:19] * arturo off [19:07:42] * bd808 lunch [19:38:48] I need to run a series of complicated errands so will be afk for a couple of hours. [20:01:53] I started https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/Onboarding to document everything a.rturo or anyone else needs to set up when onboarding [20:19:38] taavi: we have https://www.mediawiki.org/wiki/Wikimedia_Cloud_Services_team/Onboarding_template too [21:56:47] Do we really recommend build service images as the best practice for everything now? That seems like a pretty fast beta -> best practice cycle. https://wikitech.wikimedia.org/w/index.php?title=Help:Toolforge/Web&diff=prev&oldid=2142995