[05:01:00] !log tools.poty-stuff Updated from da9e44c to 992dcf6 [05:01:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.poty-stuff/SAL [06:25:54] !log tools.poty-stuff Updated from 992dcf6 to 8ea5d2e [06:25:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.poty-stuff/SAL [16:08:58] FYI everyone, we're starting the toolsdb cut-over now, most tools will break for a while and then need restarting. [16:10:35] Oh, my mistake, starting in 50 minutes or so [16:17:20] !log tools.quickcategories set EXPECTED_DATABASE_ERROR for upcoming ToolsDB maintenance [16:17:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL [17:03:53] !log admin running wmcs-wikireplica-dns on cloudcontrol1005 to update tools-db dns entries [17:03:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [18:45:35] hi, I changed my toolsbd hostname to `tools.db.svc.wikimedia.cloud` as suggested (previously it was `tools.db.svc.wikimedia.wmflabs`). It is working, but it is realllllllly slow. Like >10 seconds for a single INSERT. I believe this effects reads as well, given the errors I'm seeing from a cronjob for eventmetrics.wmflabs.org [18:45:51] all tools I speak are from Cloud VPS, not Toolforge [18:47:08] okay, so querying directly on Toolforge is just as fast as ever, so it's only my tools that are effected for some reason [18:48:57] (I meant to say I was previously using `tools.db.svc.eqiad.wmflabs`, not `wikimedia.wmflabs`) [18:49:14] I should debug a bit more before I bother you all... just noting here aloud in case others have similar problems [19:04:29] hmm so it's only queries originating from Cloud VPS that are really slow for me... everything is fine on my local and Toolforge [19:12:08] !log tools.quickcategories unset EXPECTED_DATABASE_ERROR again; also restarted both webservice and background-runner in case it was needed to pick up the new DB server (I didn’t check) [19:12:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL [19:30:25] okay, I've verified all hostnames used are correct, cleared the app cache, restarted Apache, and even hard rebooted the instances and the queries are still timing out :( [19:30:54] on my team this effects https://eventmetrics.wmflabs.org and https://ws-export.wmcloud.org [19:31:34] andrewbogott: ^ possible routing/firewall/something affecting musikanimal's things outside of Toolforge talking to the new ToolsDB [19:32:23] musikanimal: knowing the source projects + vms may be helpful in debugging [19:33:03] And a phab task is generally a good idea once you've don't as much troubleshooting as it sounds like you have [19:33:56] sure thing. I was trying to think of another example of a Cloud VPS tool that talks to toolsdb, just to further verify it's a broader issue [19:37:55] are you saying that writes from cloud vps vms are slow but writes from toolforge are not? or have you only tested access from cloud vps? [19:38:30] https://phabricator.wikimedia.org/T334255 [19:38:45] I've tested on my local, Toolforge and VPS. Only VPS is timing out [19:38:53] this is reads too, not just writes [19:40:19] at least the reason is obvious.. there are no firewall rules permitting traffic from outside toolforge [19:40:25] * taavi fixes [19:40:35] that would do it :) [19:40:38] bah! [19:40:41] lol [19:41:11] so it was probably always timing out and never succeeding, just some of my apps failed gracefully [19:41:11] try now? [19:41:16] yep [19:41:20] works! [19:41:21] yay! [19:41:23] thank you :0 [19:41:25] :) [19:41:39] in general I'm not a big fan of non-toolforge tools using toolsdb, but breaking it without any announcement is not reasonable [19:42:06] they all were originally on Toolforge, to be fair [19:42:27] but I've read about the new user db services you offer for VPS, we will have to give it a try [19:42:55] yeah. and please take that as me expressing my general dissatisfaction about the current state of things, not your tools specifically [19:48:00] taavi: thank you for fixing! "doesn't work at all" was a much better symptom than "works but is slower" [21:33:41] !log tools.isa updated `SQLALCHEMY_DATABASE_URI` reference from `clouddb1001` to `tools-db-1` for T334251 [21:33:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.isa/SAL [21:33:46] T334251: ISA Tool is down while our campaign is running, please unbreak this now - https://phabricator.wikimedia.org/T334251 [21:37:45] that’s awfully nice of you [21:38:53] seemed a lil' urgent with the edit-a-thon or whatever starting today ^^ [21:41:50] let’s hope that campaign leads to less edit spam than previous ones have apparently motivated :/ [21:41:58] https://commons.wikimedia.org/wiki/Commons_talk:Structured_data#Can_anyone_explain_what_is_going_on_here%3F [21:43:09] oh [21:43:12] (: [21:43:13] arturo: you wrote "let me try myself in your tool". is this still relevant? I added you as a maintainer so you can write "become bothasava" and then "toolforge-jobs restart monthly" . cheers [21:50:01] Kotz: a.rturo works from the CET timezone (UTC+2 in summer time), so he likely will not respond until sometime tomorrow in his timezone [21:51:24] bd808: thanks [22:08:44] I am trying to restart webservice of one of my tools, and I am getting the error "type must be one of: golang1.11, * jdk17, ...", my tool runs with python2 image [22:10:01] are you targeting the gridengine or kubernetes backend? [22:10:10] kubernetes [22:10:11] (also, mentioning the tool name would probably help me or someone else look into it) [22:10:21] wmopbot [22:10:45] /me looks if k8s is still supposed to support python2 [22:11:44] I have another tool running python2 webservice (ptwikis) and it is running [22:15:58] hm, python3.7 works but says it’s deprecated, pointing at https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes [22:16:06] and https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#Available_container_types does list python2 [22:18:41] danilo: try python2.7 [22:19:06] (found in kubectl get configmap image-config --namespace=tf-public -o yaml, though it does list an alias of python2 so idk why that’s not working anymore) [22:19:34] ah, nothing in toolsws/kubernetes.py seems to read an “aliases” key, I guess that would explain it [22:19:42] *toolsws/backends/kubernetes.py, sry [22:20:04] python2.7 worked, it is restarting, thank you! [22:20:09] yay [22:20:39] I’ll file a task [22:20:41] (unless you want to?) [22:24:51] filed T334262 [22:30:27] win 88 [23:21:24] should you get people complaining about `Invalid datetime format` errors: https://phabricator.wikimedia.org/T333471#8764387 [23:23:01] but muh standards! [23:44:03] musikanimal, AntiComposite: ISO 8601 has never been an officially accepted datetime literal syntax for mysql/mariadb. But in on-strict mode it will just emit a warning. The new toolsdb is running in strict mode, mostly because that is the default since 10.2.4. [23:44:16] *non-strict mode [23:47:36] dhinus: ^ that switch to strict mode will probably catch others out as well. We should probably do an announce specifically about that. The whole toolsdb migration likely deserves a page under https://wikitech.wikimedia.org/wiki/News too.