[03:37:21] 10Analytics, 10Event-Platform: jsonschema-tools: No resolver found for key http. - https://phabricator.wikimedia.org/T288321 (10Tgr) [03:41:34] 10Analytics, 10Event-Platform: jsonschema-tools: No resolver found for key http. - https://phabricator.wikimedia.org/T288321 (10Tgr) 05Open→03Invalid I think this was just the case of using an incompatible version. After making sure everything is up to date, I can't reproduce it again. [04:20:05] (03PS1) 10Gergő Tisza: homepagevisit: add 'contributelist' to 'referer_route' value list [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/710414 (https://phabricator.wikimedia.org/T287926) [04:49:32] 10Analytics, 10Dumps-Generation: xmldatadumps dumpstatus.json files only readable by root - https://phabricator.wikimedia.org/T287989 (10ArielGlenn) There are only 4 wikis with the weird permissions, I should have checked that at the same time as the ownership most likely. ` ariel@dumpsdata1003:/data/xmldatad... [05:04:13] RECOVERY - Check unit status of refinery-import-page-current-dumps on an-launcher1002 is OK: OK: Status of the systemd unit refinery-import-page-current-dumps https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [05:04:28] 10Analytics, 10Dumps-Generation: xmldatadumps dumpstatus.json files only readable by root - https://phabricator.wikimedia.org/T287989 (10ArielGlenn) Well, or not fixed. I found scattered files in some of the 20210801 subdirectories sitll with root and 600 perms. I've just done the entire tree, for all wiki sub... [07:06:13] Hi, I just started a scala kernel in jupyterHub. It seems to be stuck. Showing `Busy` for too long. I didn't run any code yet, just ran the cell with imports. It says `Waiting for a Spark session to start...` and thats all. Any ideas whats wrong? [07:06:42] Hi tanny411! On what host are you? [07:06:58] stat1008 [07:07:45] and your username is? [07:08:00] akhatun [07:08:08] ah okok perfect, I always forget :) [07:08:45] this is from journalctl (that for the moment only we can read) [07:08:45] [E 2021-08-06 07:05:36.601 SingleUserNotebookApp __init__:109] Notebook JSON is invalid: Additional properties are not allowed ('source' was unexpected) [07:09:19] What does that mean? [07:09:46] I hoped it was something that you tuned, checking in the logs more [07:10:59] tanny411: this is the other thing that I see https://phabricator.wikimedia.org/P16966 [07:11:58] but I see something similar even before, so it may be a red herring [07:12:20] does it ring a bell? [07:12:46] I don't see any other problem in the logs [07:12:48] hmm...okay I will try recreating the lernel. Not really, it couldnt find the jar I think. [07:12:54] kernel* [07:27:07] elukey: Nope, its not a jar/any import issue. This time I just started a new kernel and basically set a string with some value `val s="something..."`. Still same. [07:27:07] It seems to be running okay with scala kernel, stuck with scala-sql kernel. [08:00:37] no idea then, maybe let's open a task to investigate [08:13:12] okay [09:19:17] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: jupyter notebook causing syslog/etc.. to fill up with error messages - https://phabricator.wikimedia.org/T287339 (10BTullis) I have created an [[ https://github.com/jupyterhub/systemdspawner/issues/82 | issue ]] and a [[ https://github.com/jupyterhub/syste... [09:21:53] (03CR) 10David Caro: "I think that we should decide how to manage dependencies on this project before this. We are also blocked from merging this until we upgra" [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/710305 (https://phabricator.wikimedia.org/T288249) (owner: 10Michael DiPietro) [09:54:43] 10Analytics-Clusters, 10Analytics-Kanban: Deploy an-test-coord1002 as a Ganeti VM to facilitate failover testing of analytics coordinator role - https://phabricator.wikimedia.org/T287864 (10BTullis) [10:21:04] 10Analytics-Clusters, 10Analytics-Kanban: Deploy an-test-coord1002 as a Ganeti VM to facilitate failover testing of analytics coordinator role - https://phabricator.wikimedia.org/T287864 (10elukey) https://wikitech.wikimedia.org/wiki/Ganeti#VM_operations :) [11:07:58] elukey: tanny411: Just an FYI, I'm currently working to set up logging of Jupyter notebooks to go to Logstash. So this should allow people to look at their own logs from notebooks. T287339 [11:07:58] T287339: jupyter notebook causing syslog/etc.. to fill up with error messages - https://phabricator.wikimedia.org/T287339 [11:08:18] Will let you know when there's something helpful to look at. [11:09:23] btullis: Thanks! [12:59:40] btullis: great news! It is indeed really needed! [13:19:01] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: jupyter notebook causing syslog/etc.. to fill up with error messages - https://phabricator.wikimedia.org/T287339 (10BTullis) Ah, it's not making it through to Logstash because it gets caught by the following rsyslog snippet. ` btullis@an-test-client1001:/... [13:32:48] 10Analytics: Jupyter notebook logs should appear in Logstash - https://phabricator.wikimedia.org/T288348 (10BTullis) [13:34:10] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: jupyter notebook causing syslog/etc.. to fill up with error messages - https://phabricator.wikimedia.org/T287339 (10BTullis) Created follow-up task: {T288348} [13:49:02] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Add analytics-presto.eqiad.wmnet CNAME for Presto coordinator failover - https://phabricator.wikimedia.org/T273642 (10BTullis) I have merged this change, so the CNAME has been created. ` btullis@marlin:~$ for i in 0 1 2 ; do dig @ns${i}.wikimedia... [13:49:46] 10Analytics-Clusters, 10Analytics-Kanban: Deploy an-test-coord1002 as a Ganeti VM to facilitate failover testing of analytics coordinator role - https://phabricator.wikimedia.org/T287864 (10BTullis) [13:49:48] 10Analytics: Analytics coordinator failover improvements - https://phabricator.wikimedia.org/T280905 (10BTullis) [15:01:22] (03CR) 10Bstorm: "I will say it again. I don't think we should be upgrading to a version of OS that isn't released yet. Buster will be available as an optio" [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/710305 (https://phabricator.wikimedia.org/T288249) (owner: 10Michael DiPietro) [15:03:58] (03CR) 10Bstorm: [C: 04-1] "Until we are ready to release bullseye." [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/710305 (https://phabricator.wikimedia.org/T288249) (owner: 10Michael DiPietro) [15:06:26] (03CR) 10Bstorm: [C: 04-1] "On a more positive note: using slim containers in the dev environment might be awesome 😊" [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/710305 (https://phabricator.wikimedia.org/T288249) (owner: 10Michael DiPietro) [15:13:34] (03CR) 10Bstorm: [C: 04-1] upgrade quarry to python 3.9 (031 comment) [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/710305 (https://phabricator.wikimedia.org/T288249) (owner: 10Michael DiPietro) [15:16:32] 10Analytics-Clusters, 10Analytics-Kanban: Refresh Druid nodes (druid100[1-3]) - https://phabricator.wikimedia.org/T255148 (10BTullis) Unless I'm mistake, we can commission `an-druid100[3-5]` before having to worry about the zookeeper migration. Would you agree @elukey? We would just temporarily have a cluster... [15:36:03] (03CR) 10Mforns: [C: 04-2] "Lets review, but please not merge, we still need to determine where this code should live :]" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/702668 (https://phabricator.wikimedia.org/T285692) (owner: 10Mforns) [15:38:11] (03CR) 10Bstorm: [C: 03+1] docs: added docker compose link and minor rewording [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/709951 (owner: 10David Caro) [16:05:18] 10Analytics-Clusters, 10Analytics-Kanban: Refresh Druid nodes (druid100[1-3]) - https://phabricator.wikimedia.org/T255148 (10elukey) Adding new nodes to a Druid cluster causes a re-assignment in segments served by various daemons (mostly historical ones), that changes the data cached on the current nodes (ther... [19:26:58] 10Analytics-Radar, 10SRE, 10ops-eqiad: Try to move some new analytics worker nodes to different racks - https://phabricator.wikimedia.org/T276239 (10Cmjohnson) @Ottomata an-worker1139 is officially in rack A7. All cabled up and ready for OS install [19:30:28] 10Analytics-Radar, 10SRE, 10ops-eqiad: Try to move some new analytics worker nodes to different racks - https://phabricator.wikimedia.org/T276239 (10Cmjohnson) [19:36:47] (03PS2) 10Andrew Bogott: tox.ini: update to work with default buster tox version [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/710357 [19:36:49] (03PS1) 10Andrew Bogott: Make a 'tests' dir and move our one test file there [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/710603 (https://phabricator.wikimedia.org/T210359) [19:36:52] (03PS1) 10Andrew Bogott: Added test_webhelpers.py [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/710604 [19:37:23] (03CR) 10jerkins-bot: [V: 04-1] Added test_webhelpers.py [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/710604 (owner: 10Andrew Bogott) [20:00:43] (03CR) 10Bstorm: Added test_webhelpers.py (031 comment) [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/710604 (owner: 10Andrew Bogott) [20:01:44] (03CR) 10Bstorm: Added test_webhelpers.py (031 comment) [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/710604 (owner: 10Andrew Bogott) [20:07:29] (03CR) 10Andrew Bogott: Added test_webhelpers.py (032 comments) [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/710604 (owner: 10Andrew Bogott)