[09:28:20] <arturo>	 !log toolsbeta disabled PodSecurityPolicy admission plugin from apiserver static pod manifests (T368142)
[09:28:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[09:28:24] <stashbot>	 T368142: Toolforge: drop PodSecurityPolicy - https://phabricator.wikimedia.org/T368142
[09:30:34] <arturo>	 !log toolsbeta disabled PodSecurityPolicy admission plugin from kubeadm configmap (T368142)
[09:30:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[10:02:08] <arturo>	 !log tools disabled PodSecurityPolicy admission plugin from kubeadm configmap (T368142)
[10:02:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[10:02:12] <stashbot>	 T368142: Toolforge: drop PodSecurityPolicy - https://phabricator.wikimedia.org/T368142
[10:02:27] <arturo>	 !log tools drop all PSP definitions for all accounts (T368142)
[10:02:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[15:47:17] <andrewbogott>	 Tools-db will flap briefly as I move it to the new network setup
[15:48:34] <andrewbogott>	 all done
[15:50:05] <andrewbogott>	 (will be another flap in a few minutes)
[16:02:08] <dhinus>	 ack
[16:02:23] <dhinus>	 there are some alerts, looking
[16:03:13] <dhinus>	 the systemctl "mariadb" service did not start in the replic
[16:03:48] <dhinus>	 not sure why, the logs show a clean shutdown
[16:05:06] <dhinus>	 started manually
[16:06:52] <andrewbogott>	 dhinus: can you stick around for another 10-15?  I need to migrate -1 too (which seems more serious)
[16:06:59] <dhinus>	 sure
[16:07:06] <dhinus>	 the alerts are gone
[16:07:46] <dhinus>	 to migrate -1, ideally we would failover to -3, but I haven't tested it yet, so let's just migrate and accept some downtime
[16:07:53] <bd808>	 if toolsdb uses the normal WMF Puppet bits for mariadb is won't start on boot on purpose.
[16:08:05] <dhinus>	 bd808: I was checking and i think that's the case
[16:08:06] <bd808>	 *it
[16:08:17] <dhinus>	 I was confused because it was running on -1, but that hasn't been rebooted yet
[16:08:36] <andrewbogott>	 yeah, I was confused when I said there would be a flap, the flap is yet to come :)
[16:09:09] <dhinus>	 andrewbogott: yeah, it's not automated, the next step is to test a "manual failover", which is tricky enough :)
[16:09:17] <dhinus>	 automated one is step 2 :P
[16:09:52] <dhinus>	 andrewbogott: for extra safety, I would do "systemctl stop mariadb" before migrating
[16:10:00] <andrewbogott>	 ok
[16:10:22] * andrewbogott waits for CI
[16:16:01] * andrewbogott continues to wait for CI
[16:20:49] <andrewbogott>	 dhinus: ok, now I'm going to stop mariadb and start the migration on -1
[16:22:17] <dhinus>	 let me know when it's done
[16:23:01] <andrewbogott>	 migration is done, it's starting up now
[16:23:16] <andrewbogott>	 I'll start mariadb
[16:23:26] <andrewbogott>	 ...as soon as I can ssh...
[16:23:40] <andrewbogott>	 ok, done
[16:23:44] <andrewbogott>	 dhinus, check my work?
[16:24:22] <dhinus>	 checking
[16:24:24] * taavi restarts replication on tools-db-3
[16:24:27] <taavi>	 -1 is still in read-only
[16:24:47] <andrewbogott>	 dang, ok -- you fix or shall I?
[16:24:48] <dhinus>	 yes, fixing
[16:24:57] <andrewbogott>	 thanks
[16:25:29] <dhinus>	 read-only set to off
[16:26:48] <andrewbogott>	 all good now?
[16:29:13] <dhinus>	 yep
[16:29:36] <andrewbogott>	 dhinus: I think I'm done breaking dbs for today. Thanks for sticking around.
[16:30:01] <dhinus>	 no problem, I was around playing with the wikireplicas...
[16:54:10] <Guest55>	 On Toolforge, I am suddenly getting "no python application found, check your startup logs for errors" in my ~/uwsgi.log. But I have an app.py that has been working fine until less than an hour ago. I haven't made any significant changes to the setup. I made a minor tweak to code in app.py itself, but I've done that many times before without error.
[16:54:11] <Guest55>	 Is there some kind of outage or could I be doing something wrong?
[16:55:19] <dcaro>	 not that we are aware of, what tool is it?
[16:55:32] <Guest55>	 "datalibrarian"
[16:55:43] <dcaro>	 👀
[16:55:47] <Guest55>	 thanks
[16:56:35] <dcaro>	 unable to find "application" callable in file app.py
[16:56:46] <dcaro>	 looking
[16:56:47] <taavi>	 Guest55: the actual error why it's failing can be found from higher up from the log. this looks like a bug in the app itself to me: https://phabricator.wikimedia.org/P65538
[16:57:10] <dcaro>	 https://www.irccloud.com/pastebin/9ZY1wVdb/
[16:57:28] <dcaro>	 yep
[16:57:44] <Guest55>	 Ok thanks. I'll try and investigate.
[16:57:52] <dcaro>	 👍
[18:45:26] <wm-bb>	 <121063> https://ads.google.com/aw/express/video/dashboard?campaignId=21373192692&ocid=6623206467&euid=1060546717&_u=9106847733&uscid=6623206467&_c=5528725483&authuser=0&enableAllBrowsers=1&mobile=1
[18:45:27] <wm-bb>	 <121063> https://youtube.com/shorts/Ido5h121Evs?si=YGgWW1TubH8Z9McH
[18:45:28] <wm-bb>	 <121063> #范孝宗專攻資安相關性課程繼續加強雲端運算GA4_Al_AIl_GPS_AIP