[11:44:10] !status upgrading toolforge k8s T390214 [11:44:10] T390214: Upgrade "tools" cluster to k8s 1.29.15 - https://phabricator.wikimedia.org/T390214 [12:55:37] woo [13:54:16] !status ok [14:15:57] Hi. Is there an ongoing problem with logins to toolsadmin, IDM, IDP, etc? I'm not able to login and get 500 Error. Same with password reset, I get invalid token on IDM. [14:21:29] DaxServer: something may be happening indeed. That's doesn't seem normal. Could you please open a phab ticket? [14:21:44] Is it related to https://phabricator.wikimedia.org/T390214 dhinus [14:21:57] DaxServer: unlikely [14:24:34] yeah, toolsadmin should not be affected by the upgrade [14:35:20] https://usercontent.irccloud-cdn.com/file/v0G4IHW1/image.png [14:37:47] I've filed a report on phab T391259 [14:37:48] T391259: Cannot login to toolsadmin, password reset link says invalid token at idm - https://phabricator.wikimedia.org/T391259 [14:38:19] arturo: can you share the request id as text? [14:38:59] taavi: 87c8a2e186974ad2898f3bdefc083c23 [14:40:59] django.db.utils.OperationalError: (1290, 'The MariaDB server is running with the --read-only option so it cannot execute this statement') [14:41:01] hrm [14:42:28] T391237 looks relevant [14:42:28] T391237: m5 master db1228 rebooted itself - https://phabricator.wikimedia.org/T391237 [14:44:36] it seems like the proxy (dbproxy1029) still sees db1228 as down? [15:30:30] !log testlabs create a bunch of VMs by hand, like `networktests-vlan-legacy-floating` T380728 [15:30:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Testlabs/SAL [15:30:33] T380728: openstack: network problems when introducing new networks - https://phabricator.wikimedia.org/T380728 [15:30:48] !log admin [codfw1dev] testlabs create a bunch of VMs by hand, like `networktests-vlan-legacy-floating` T380728 [15:30:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [17:32:34] !log lucaswerkmeister@tools-bastion-13 tools.ranker deployed 348dc8edc7 (l10n updates: es, zh-hant) [17:32:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ranker/SAL [18:01:32] !log lucaswerkmeister@tools-bastion-13 tools.lexeme-forms deployed e106b7b684 (Quechua verbs + l10n updates: es, pa, qu, zh-hant) [18:01:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [23:01:01] Something very strange happened to me today: I created a tool, and it seems it does not have a replica.my.cnf file? Does anyone know if it is taking some time to appear in the repository? Usually is right away. PS: My tool is called glamwikibrasil [23:10:59] occasionally a bug prevents it from being created, unfortunately I don’t know how to solve it… I expect someone from the cloud services team will get to it soon [23:42:38] !log `sudo service maintain-dbusers restart` on cloudcontrol1007 after reports of missing replica.my.cnf and finding the journal for the service empty. [23:42:40] bd808: Unknown project "`sudo" [23:43:45] !log admin `sudo service maintain-dbusers restart` on cloudcontrol1007 after reports of missing replica.my.cnf and finding the journal for the service empty. [23:43:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [23:47:43] @ederporto: your tool has credentials now. [23:49:12] @lucaswerkmeister: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Wiki_Replicas#Account_management_(maintain-dbusers) and https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin#Regenerate_replica.my.cnf but unfortunately you will need wmcsroots permissions to mess with the process on cloudcontrol1007 [23:50:09] yeah, I suspected as much based on https://phabricator.wikimedia.org/T382962#10429947. thanks!