[06:18:14] nuria: is it possible to dig out specific metrics somehow? [07:04:35] btullis: o/ - morning! an-worker1106 showed as DOWN in icinga, I acked it so we'll be able to work on it when you are online :) [07:12:09] 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10MoritzMuehlenhoff) [07:31:40] elukey: thanks, I'm online now. [07:31:54] good morning :) [07:32:06] nothing on fire it is just a host! [07:32:19] but if you haven't connected to the serial console etc.. yet we can do it [07:33:34] Perfect, thanks. In a Meet, or in IRC chat? [07:35:36] we can do it in here if you are ok! [07:35:56] i can explain what I usually do [07:36:08] do you have your pwstore repo ready with gpg key? [07:36:09] Yep, cool. I see the host in Icinga. Do we make a ticket for it? [07:36:45] We can, I usually check quickly to see if anything is really broken or if it is transient (like if a reboot fixes) [07:36:50] Yes, I have pws ready I think. [07:37:57] perfect [07:38:11] so in there there are two files that contains passwords that we need [07:38:24] 1) 'management' -> this is the pass to connect to the serial console [07:38:40] 2) root_password -> this is to login as 'root' when you have a tty basically [07:39:09] I usually just gpg --decrypt management when I need the pass [07:39:14] but pws works as well [07:40:26] when I have the management pass what I usually do is: [07:40:35] 1) connect to cumin1001.eqiad.wmnet and tmux/screen [07:40:58] 2) do an ssh like: ssh root@an-worker1106.mgmt.eqiad.wmnet [07:41:18] that should lead to a password prompt, that requires the management password [07:41:45] OK, one sec. Trying to recall gpg passphrase under pressure now. :-) [07:42:02] once in, you are either in the Dell DRAC or in the HP iLO (IIRC the naming) [07:42:21] ahahhaha nono please the contrary of under pressure, nothing is really on fire [07:45:30] Phew! Got it. No, it would just be the embarrassment of it. [07:47:17] OK, I've got a `racadm>>` prompt. [07:47:20] nice! [07:47:51] so we have on wikitech a lot of infos about what commands you can write, but the most useful ones for DELL that I use are [07:48:05] 2) racadm getsel (to get various events etc.. like PSU failure, CPU broken, etc..) [07:48:11] err 1) sorry :D [07:48:15] 2) console com2 [07:48:48] the latter brings you to the serial console [07:49:03] and ctrl+\ should exit it [07:50:02] hi team! looking into webrequest alerts [07:51:01] Cool. Nothing relevant in the log. The latest is 09/15/2020. Will start a console now. [07:52:15] hola mforns [07:52:17] Kernel spinning. Showing soft CPU lockups. Scrolling a bit quickly to get any more at the moment. [07:52:28] :) [07:52:42] btullis: yeah exactly, the soft lockup sometimes happens for reasons that I never really understood [07:53:06] it is interesting to see the side effects on hadoop metrics too [07:53:15] https://grafana-rw.wikimedia.org/d/000000585/hadoop?orgId=1 -> Namenode panel (basically the HDFS master) [07:53:36] you will find a "Under replicate blocks" graph [07:53:40] *replicated [07:54:28] each HDFS block is replicated three times, so after a worker goes down for 10 mins (IIRC) the Namenode doesn't get health checks anymore and flags it as down [07:54:43] and asks to the other replicas to stream data to a new (live) one [07:55:10] this stops immediately when the node down gets back to life [07:55:30] in this case, the usual fix for me is 'racadm serveraction powercycle' [07:55:34] Yes I see. [07:55:48] I usually !log it on #operations for visibility [07:56:14] once the host gets back up, you'll see the replicated blocks metric recovering etc.. [07:56:44] that's it :) [07:58:15] Gotcha, thanks. Will the racadm powercycle do an ACPI graceful shutdown if it can, or is it simply off->on ? I've been more used to Supermicro IPMI and `ipmitool` recently. [07:58:53] Also, do we monitor under-replicated blocks in Icinga or anything? [08:03:15] Ah, seems my Icinga privileges need updating, because I couldn't add a comment there. I thought I'd done that. [08:06:42] btullis: yes it is a graceful shutdown, there is a more brutal restart but I don't recall the syntax [08:07:14] for the under-replicated blocks, we haven't in the past but we added a lot of monitors for Namenode metrics (you can find those in puppet) [08:07:35] all of them have a related grafana panel + https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts [08:07:53] so it is sufficient to click on the links provided by the alarm to know what's going on [08:09:31] we have alarms for the more problematic corrupt/missing blocks [08:09:56] in theory we can add an alarm for under-replicated blocks, but in my opinion it should be dynamic based on the total number of blocks [08:10:05] something like 30/40% or similar [08:10:31] but the only use case that I have in mind to trigger the alert would be a ton of workers down [08:10:42] and we'd notice it from a lot of other alarms [08:10:52] (this is probably why we never added one) [08:11:36] OK, fab. Thanks. It's booted cleanly. Under-replicated blocks has dropped immediately to zero, as you said it would. All looks OK. [08:13:13] So in this case no phab ticket required, but just mental note that this known problem has happened once again and required intervention. If it hadn't behaved as expected with a simple reboot, then I would create a ticket for triage. Something like that? [08:17:47] yes exactly [08:18:02] say a DIMM broken, cpu broken, etc.. [08:18:18] we have automation (SRE automation I mean) that creates tasks for broken disks [08:18:27] but not for the rest (it is a little more difficult) [08:21:30] re: ipmitool, I forgot to add a note that we can use it as well [08:21:41] there should be documentation on wikitech [08:27:49] re: the automation. Is this a cookbook, or something else? I can't seem to find such a script in the cookbooks, nor in the netbox scripts. Wondering if there is another place for sre automation scripts that I should know about. [08:28:48] btullis: ah yes it should be in puppet, it is a a check that fires when nagios detects a broken disk [08:29:19] https://phabricator.wikimedia.org/T285643 is an example [08:30:27] Ah nice. A fully automated automation. :-) Will check that out. Thanks again for all the info. [08:32:15] np! [08:32:32] !log restarted webrequest bundle (messed up a coord when trying to rerun some failed hours) [08:32:35] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:49:27] mforns: Hi! Thanks a lot for checking and rerunning - I think I have found the problem leading to more data-loss now that we use gobblin :) [08:49:41] Adding you as a reviewer mforns [09:07:03] this probably sound evil, but is it possible to run PHP code in hadoop cluster? [09:07:24] addshore: I'll fake not having seen that message :D [09:07:36] xD [09:07:44] hear me out! :P https://phabricator.wikimedia.org/T94019 [09:08:14] I definitely hear you - this would a hell lot of sense [09:08:21] I was generally thinking that if hadoop had up to date external JSON representations of Wikidata entites, 1) we could generate JSON dumps from there and 2) we could use the same json -> rdf mapping code to make the rdf fumps [09:08:30] while keeping the mapping code in Wikibase in PHP [09:09:26] I'm not sure about how we can get "up to date external JSON representations" - I assume this means getting the wikidata content, right? [09:09:34] yeah [09:09:51] I was thinking, some event post edit -> kafka -> hadoop? but not sure about that part either? [09:09:56] ok - we're not yet ready on that front, btu we have a plan (but no resource to put it in practice) [09:10:08] okay [09:10:37] the plan would be: edit--> kafka --> streaming job --> WMF-API(get content) --> store on HDFS [09:11:11] aaah okay, rather than having the content in the stream [09:11:16] And there is one issue: we have incomplete edits in kafka currently, and a task force issue (we don't have the bandwisth) [09:11:36] aah, this is the "reliable events" ticket right? [09:11:41] correct sir [09:11:50] it always comes back to that one! :P [09:12:25] So https://phabricator.wikimedia.org/T120242 ? [09:12:35] The WDQS-updater relies on the edits stream, so we could do it (they already do almost exactly what I describe), but IMO it's worth waiting the solution to reliable events (or more reliable should I sa) [09:12:50] okay *goes to write a comment* [09:13:53] this is the ticket yes - I think we could even not do what the ticket advertise as possible solutions as long as the missing events in streams is low enough (currentl at ~1%, far too high) [09:15:03] And then about running PHP on hadoop - I have not yet seen it done, but I don't see why it would be feasible - The java (or python) haddop worker starts an external PHP process, feeds it the content and gets the result back, then finalizes its process [09:15:13] not great but doable [09:15:34] It'll also require having the PHP libs etc on hadoop workers (not the case now) [09:15:55] but also with the text diagram you mentioned above, if the content comes from the MW api anyway, then the conversion happens in the app, rather than in haddop land [09:16:07] with #worksforme [09:16:08] *which [09:16:28] hm, which app? [09:16:48] well, mediawiki / wikibase [09:17:09] Ah - We can ask the wikibase-API to give us json is that rght? [09:17:29] Maybe even RDF? [09:17:53] json,. rdf, ttl, jsonld, etc [09:17:59] right [09:18:13] So a dedicated wikibase job extracting info in streaming could do [09:18:33] I think that's what the WDQS-updater does [09:18:37] indeed, we could even provide multiple formats via a single call for optimizations sake [09:18:47] yes, the streaming updater goes to wikibase and gets the TLL afaik [09:18:50] yup, call once, save multiple [09:18:51] *ttl [09:26:34] 10Analytics-EventLogging, 10Analytics-Radar, 10Discovery, 10Wikimedia-production-error: '.event.pageViewId' should be string, '.event.subTest' should be string, '.event.searchSessionId' should be string - https://phabricator.wikimedia.org/T286814 (10Aklapper) Please don't add project tags as subscribers bu... [09:26:45] 10Analytics-Radar, 10WMDE-Templates-FocusArea, 10WMDE-TechWish-Sprint-2021-07-07: Backfill metrics for TemplateWizard and VisualEditor - https://phabricator.wikimedia.org/T274988 (10WMDE-Fisch) 05Open→03Resolved [09:45:07] joal: sorryyyy I was in a meeting, added a comment to you code review for gobblin [09:48:28] np elukey :) thank you! [09:52:52] 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10aborrero) [09:54:27] 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10aborrero) [10:08:18] joal: deployed [10:08:21] \o/ [10:08:43] thanks a lot elukey - I'm gonna double check next hour [11:17:44] elukey: all good from camus - many thanks! [11:17:50] s/camus/gobblin [11:59:57] goooood [13:22:57] Hello. We need to make about ~450GB of data (currently on hdfs, non-PII) available for download for a competition. Can we use the dataset release instructions for this purpose? https://wikitech.wikimedia.org/wiki/Analytics/Web_publication [13:35:31] fab: o/ 450GB before replication right? [13:35:40] (asking to double check) [13:37:55] yes without replication. generating ~1gb files, so there would be about 450 files that would end up in the public html directory. [13:42:02] would it be for a limited amount of time? [13:42:07] I mean not months [13:42:36] we currently have space of thorium (where we serve files from) but the host will be decommed and replaced with another one with less capacity [13:47:17] It is for a limited amount of time, I will check the planned timeline - there are still some open questions on where the different type of datasets will be hosted [13:49:24] I am a little worried about network bandwidth for such a big file, if multiple requests come in at the same time thorium will surely go under a little stress easily [13:49:44] (plus downloads will become a lot slow) [13:50:58] ideally if the dataset is public we could serve it from something like commons/swift/s3, not sure if it is a valid use case or people did it before [13:59:50] 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10ArielGlenn) [14:03:05] 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10MoritzMuehlenhoff) [14:09:33] 10Analytics, 10Event-Platform, 10Wikidata, 10Wikidata-Query-Service, and 2 others: Automate event stream ingestion into HDFS for streams that don't use EventGate - https://phabricator.wikimedia.org/T273901 (10Zbyszko) @Ottomata can this be closed? [14:19:32] 10Analytics, 10SRE, 10Wikidata, 10Wikidata-Query-Service, 10wdwb-tech: Deployment strategy and hardware requirement for new Flink based WDQS updater - https://phabricator.wikimedia.org/T247058 (10Zbyszko) 05Open→03Resolved a:03Zbyszko Strategy was developed and is being implemented. [14:35:12] Hi team, going to start disabling jobs on an-launcher for the hadoop master debian upgrade. In about 30 minutes when it's time to reimage I'll hop in the batcave so btullis and anybody else who wants to can see the commands [14:36:55] razzi: good morning! +1 for the cluster drain, but in ~25 mins there will be network maintenance in row D so we need to wait for it to be finished [14:37:34] question elukey and razzi: wouldn't it be cool to have the cluster drained before the row D maintenance? [14:38:00] joal: IIUC this is what Razzi is going to do now [14:38:22] Great - was not sure if the idea was to postpone draining after the row D [14:38:48] nono what I suggested was to wait for the reimage [14:39:00] perfect thanks :) [14:40:37] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Add analytics-presto.eqiad.wmnet CNAME for Presto coordinator failover - https://phabricator.wikimedia.org/T273642 (10jbond) I'm curious why the intention is to configure this using a analytics-presto.eqiad.wmnet CNAME instead of a analytics-pre... [14:40:50] 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10ops-monitoring-bot) Icinga downtime set by vgutierrez@cumin1001 for 1:00:00 4 host(s) and their services with reason: eqiad row D maintenance ` cp[1... [14:42:21] razzi: I'm aware of: https://phabricator.wikimedia.org/T278423#7190372 but out of interest, how is the cluster drained? [14:42:39] 10Analytics-EventLogging, 10Analytics-Radar, 10Discovery, 10Wikimedia-production-error: '.event.pageViewId' should be string, '.event.subTest' should be string, '.event.searchSessionId' should be string - https://phabricator.wikimedia.org/T286814 (10mforns) Uou, yea, my bad. Thanks for the heads up! [14:43:05] btullis: first two steps basically [14:43:11] 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10Vgutierrez) [14:43:44] we stop all the systemd timers, including the ones importing data, and eventually most of the recurrent jobs will halt (including the oozie ones, since no new data will be available) [14:44:02] some other jobs will be left running (say user-create ones etc..) [14:45:19] Thanks for answering btullis 's question elukey, still getting my day started but now I'm here! [14:45:43] Oh I see. 3rd bullet point. "Wait 30 minutes for applications to gracefully exit". Gotcha, I missed that. [14:45:58] Going to start by disabling puppet on an-launcher. Already announced maintenance here and in product-analytics slack channel [14:46:30] !log razzi@an-launcher1002:~$ sudo puppet agent --disable 'razzi: upgrade hadoop masters to debian buster' [14:46:33] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:47:06] Now going to stop timers, excuse the !log spam [14:47:19] Well actually I don't need to !log every line this time [14:47:35] !log Disable jobs on an-launcher1002 (see https://phabricator.wikimedia.org/T278423#7190372) [14:47:39] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:48:27] Hmm I accidentally did: [14:48:28] razzi@an-launcher1002:~$ sudo systemctl stop drop_event [14:48:28] Warning: Stopping drop_event.service, but it can still be activated by: [14:48:28] drop_event.timer [14:48:34] 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10ops-monitoring-bot) Icinga downtime set by vgutierrez@cumin1001 for 1:00:00 1 host(s) and their services with reason: eqiad row D maintenance ` dns1... [14:48:59] razzi: you'd need to stop the .timer unit, not the .service ones [14:49:02] So then I ran sudo systemctl stop drop_event.timer [14:49:02] I hope I didn't mess anything up by stopping the service rather than the timer, but should b efine [14:49:03] yep [14:49:17] Now we wait :) [14:49:34] 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10Vgutierrez) [14:49:51] razzi: gobblin timers are still up :) [14:50:17] also best to stop analytics-reportupdater-logs-rsync.timer too [14:51:12] 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10ops-monitoring-bot) Icinga downtime set by vgutierrez@cumin1001 for 1:00:00 1 host(s) and their services with reason: eqiad row D maintenance ` lvs1... [14:51:14] oh welcome to the party gobblin! [14:51:37] !log sudo systemctl stop analytics-reportupdater-logs-rsync.timer [14:51:39] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:52:42] !log sudo systemctl stop 'gobblin-*.timer' [14:52:44] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:54:21] Here is the list of running apps, need this to get to 0: https://yarn.wikimedia.org/cluster/apps/RUNNING [14:54:25] Currently 18 [14:55:30] 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [14:56:33] network maintenance for row D is about to start [14:59:28] Alright, one more thing we should do before network maintenance: disable yarn queues https://gerrit.wikimedia.org/r/c/operations/puppet/+/705698 [14:59:39] Then we can chill out for a bit [15:00:13] elukey: do you mind reviewing that patch? [15:05:17] It's pretty low risk, I've done a patch like that before, I'm going to self +2 [15:05:32] hey all [15:05:42] I see you are doing things there - thanks! [15:05:49] let me know when we are good to go :) [15:06:08] 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [15:06:15] AFAIK we're good to go now, yarn queues are disabled [15:06:20] elukey: any opinion? [15:06:46] topranks: Agree. good to go. [15:07:11] razzi: sorry I was afk [15:07:21] +1 [15:07:24] Thanks elukey, that doesn't sound like a great solution for our use case. Can you elaborate on the bandwidth and the expected problems? Is the https://dumps.wikimedia.org/ hosted on the same servers, or could we possibly use that? A large part of the data are commons image bytes, which we eventually want to offer for bulk download on our servers as well. [15:07:25] cool cool [15:07:27] Ok great - thanks :) [15:07:39] Updates in #wikimedia-sre, hopefully be uneventful [15:08:21] !log sudo -u yarn kerberos-run-command yarn yarn rmadmin -refreshQueues [15:08:22] fab: the files rsynced from stat100x end up on a single node, called "thorium", that has a 1G NIC [15:08:24] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:08:28] Still had to apply puppet changes, all set now [15:08:41] thanks for checking in topranks ! [15:09:09] fab: the host is not designed to serve a lot of big files and scale accordingly, this is why I was mentioning the bw issue [15:10:09] razzi: all the applications running are user-related ones, so they will likely stay up unless you kill them [15:10:50] and mjolnir jobs may take a long time to complete sigh [15:10:57] see https://yarn.wikimedia.org/cluster/app/application_1623774792907_176079 [15:11:27] elukey: should we kill them now or wait for row D maintenance to end, or does it not really matter? [15:11:36] 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [15:11:43] elukey agreed, that is not a good solution for this use case. are you aware of any infrastructure that is suitable for this at wmf, maybe dumps.wikimedia.org? [15:11:56] razzi: it doesn't really matter, you can proceed with killing the jupyter notebooks in theory [15:12:55] Ok cool so just keep making progress so when maintenance is done we can reimage [15:13:08] Safe progress :) [15:14:01] fab: not sure what it would be best, do you have a tight deadline for this? I can ask if Swift may be used, or something similar.. [15:14:12] fab: best is probably to open a task [15:15:53] razzi: I'd also follow up in the #search channel about mjolnir [15:16:31] sg re: mjolnir [15:16:45] looking at the running applications, I don't see any jupyter ones... [15:17:10] unless wmfdata-yarn-regular is the generic name for that sort of thing [15:17:32] exactly yes, in theory it should be a notebook [15:18:54] Is there a way to know for sure? I guess I could stop the jupyter processes themselves [15:19:09] but I'd rather not, stick to the plan and just touch yarn jobs themselves [15:20:00] https://wikitech.wikimedia.org/wiki/Analytics/Systems/Jupyter-SWAP#The_easiest_way:_wmfdata [15:20:59] I am pretty sure those are notebooks [15:21:34] anyway, those are probably not actively running anything [15:21:44] fab: elukey: This might be a silly idea, but would bittorrent be useful? I can see that we *unofficially* seed some dumps from tools.wmflabs.org here: https://meta.wikimedia.org/wiki/Data_dump_torrents [15:21:46] the main problem are the search jobs [15:22:42] btullis: it looks a promising way, never heard/done it before! [15:24:35] * joal likes the bittorrent idea! [15:25:53] mjolnir* jobs are OK to be killed as they might still run for long [15:25:56] elukey: checked in with dcausse in search and we're good to stop mjolnir [15:25:57] yep yep [15:26:16] to your knowledge elukey can we kill everything and proceed? [15:26:19] going to stop flink [15:26:43] dcausse: <3 [15:26:45] I'm in the batcave now for anybody who wants to hang out [15:27:15] We're done with our change, negligible impact - "almost too quiet" - but looks like it went in ok. [15:27:53] fab: I'm assuming that dumps.wikimedia.org servers would be the easiest, but it also is limited in term of bandwidth - users can only download 2 files at a time [15:28:40] razzi: let's make sure that the non-notebook jobs are not running, the rest should be killable (but I'll defer the final word to joal) [15:28:48] fab: if the use case is to allow a lot (define a lot?) of people to download this amount of data fast (define fast), maybe a different approach is needed [15:29:22] flink is stopped, everything remaining on analytics-search can be killed [15:31:37] joal: can you comment on if any jobs in https://yarn.wikimedia.org/cluster/apps/RUNNING need to stay running? Feel free to join me and btullis in the batcave to discuss :) [15:31:49] joining! [15:37:00] !log kill yarn applications: for jobId in $(yarn application -list | awk 'NR > 2 { print $1 }'); do yarn application -kill $jobId; done [15:37:03] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:39:12] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Add analytics-presto.eqiad.wmnet CNAME for Presto coordinator failover - https://phabricator.wikimedia.org/T273642 (10BTullis) @jbond - I like the DNS Discovery idea in principle, but that page seems to me to suggest that it is more geared up for... [15:42:07] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Add analytics-presto.eqiad.wmnet CNAME for Presto coordinator failover - https://phabricator.wikimedia.org/T273642 (10BTullis) I like the look of the new PKI methods. I'll try that and tag you for code review @jbond. Thanks. [15:42:29] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Add analytics-presto.eqiad.wmnet CNAME for Presto coordinator failover - https://phabricator.wikimedia.org/T273642 (10jbond) @BTullis yes the page is definitely written with multi site in mind but AFAIK it works fine with just one site. Either w... [15:43:05] o/ any idea if we need to wait for https://phabricator.wikimedia.org/T286655 to be done before we could merge the refered to schemas? [15:44:18] All but 2 jobs have been killed; analytics-search ones are strangely not going away when we sudo -u analytics kerberos-run-command analytics yarn application -kill them [15:45:52] joal was able to kill them as a member of analytics-admin group :) [15:47:08] apparently "analytics" is not not a member of "analytics-admins" !? [15:47:13] We can circle back on this later [15:47:49] razzi: analytics is not a superuser [15:47:51] hdfs is [15:48:04] analytics is a member of analytics-privatedata-users [15:48:11] tried running the command as hdfs too, same issue [15:48:15] (and it shouldn't be a super user) [15:48:42] have you tried with 'yarn' ? [15:48:46] sudo -u yarn etc.. ? [15:49:01] it is probably due to the capacity scheduler's ACLs [15:49:40] joal tried with yarn, and I guess there's no yarn keytab so it failed [15:49:54] but yeah he showed me the xml and I guess we should make a patch for it [15:49:59] the config xml, where acls are [15:50:14] the yarn keytab is where yarn runs, an-master and workers [15:50:24] so kerberos-run-command for yarn needs to be execute on those nodes [15:50:50] gotcha [15:51:08] going to proceed with enabling safe mode, yarn is already empty [15:52:15] 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) All works complete, no signs of any issues really, I had no ping loss on 16 pings towards 2 hosts connected off each member switch. Very h... [15:52:35] !log sudo -u hdfs kerberos-run-command hdfs hdfs dfsadmin -safemode enter [15:52:37] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:52:43] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Add analytics-presto.eqiad.wmnet CNAME for Presto coordinator failover - https://phabricator.wikimedia.org/T273642 (10elukey) The discovery records may be a good path forward, this task is following what we did for analytics-hive.eqiad.wmnet. We h... [15:52:49] (03PS3) 10MewOphaswongse: Add a link: Update schema to support edit mode toggle [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/704402 (https://phabricator.wikimedia.org/T278115) [15:54:27] razzi: how did it go? [15:54:46] from the logs I didn't see explosions [15:55:04] oh !! I forgot to checkpoint :) [15:55:21] also, 1002 is active right now [15:55:31] did we ever fallback to 1001 ? [15:55:46] no, I never switched back [15:55:49] probably should have [15:56:09] let's saveNamespace [15:56:30] and then copy the data from 1002 to the backup host [15:56:50] yep yep [15:57:02] !log sudo -u hdfs kerberos-run-command hdfs hdfs dfsadmin -saveNamespace [15:57:04] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:59:20] Save namespace successful for an-master1001.eqiad.wmnet/10.64.5.26:8020 [15:59:22] \o/ [15:59:30] still running though [15:59:34] don't want to celebrate too soon... [15:59:50] ok! Save namespace successful for an-master1002.eqiad.wmnet/10.64.21.110:8020 [16:00:55] good [16:01:01] let's backup :) [16:03:14] yep! [16:03:15] !log root@an-master1002:/srv/hadoop/name# tar -czf /home/razzi/hdfs-namenode-snapshot-buster-reimage-$(date --iso-8601).tar.gz current [16:03:17] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:04:09] fdans: standup? [16:07:59] tar is done [16:10:53] !log razzi@cumin1001:~$ sudo transfer.py an-master1002.eqiad.wmnet:/home/razzi/hdfs-namenode-snapshot-buster-reimage-$(date --iso-8601).tar.gz stat1004.eqiad.wmnet:/home/razzi/hdfs-namenode-fsimage [16:10:55] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:11:09] I had the wrong host in the comment but I fixed it before running (should be 1002) [16:11:33] transfer should take a few minutes, will post when it's done [16:14:52] ok transfer is done [16:15:12] going to stop hadoop processes on an-master1001 [16:15:22] (after standup, which is just about done) [16:18:57] !log sudo systemctl stop hadoop-hdfs-namenode [16:19:01] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:19:03] on an-master1001 [16:19:31] !log razzi@an-master1001:~$ sudo systemctl stop hadoop-yarn-resourcemanager [16:19:33] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:19:49] !log razzi@an-master1001:~$ sudo systemctl stop hadoop-hdfs-zkfc [16:19:51] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:23:43] I think I forgot to disable puppet, so the namenode process started again; re-disabled puppet and stopped it again [16:23:56] !log sudo systemctl stop hadoop-hdfs-namenode on an-master1001 [16:23:58] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:25:01] !log sudo systemctl stop hadoop-yarn-resourcemanager on an-master1001 again [16:25:05] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:25:30] !log sudo systemctl stop hadoop-hdfs-zkfc.service on an-master1001 again [16:25:33] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:25:57] !log razzi@an-master1001:~$ sudo systemctl stop hadoop-mapreduce-historyserver [16:25:59] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:26:06] ok all the namenode processes are stopped again [16:26:56] PROBLEM - Hadoop ResourceManager on an-master1001 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.resourcemanager.ResourceManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Resourcemanager_process [16:27:13] no downtime :) [16:27:33] ok, icinga alert is fine [16:27:51] ps auxf | egrep 'hdfs|yarn|hadoop' came up empty! [16:27:51] yes but others will come soon if you don't downtime [16:27:57] gotcha! [16:28:06] Should have put that in the steps [16:28:14] PROBLEM - Hadoop HDFS Zookeeper failover controller on an-master1001 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.hdfs.tools.DFSZKFailoverController https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23HDFS_ZKFC_process [16:28:33] here come the alerts... [16:29:22] !log razzi@alert1001:~$ sudo icinga-downtime -h an-master1001 -d 7200 -r "an-master1001 debian upgrade" [16:29:24] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:29:33] milimetric: an example of working command for unique_devices_daily job --> https://gist.github.com/jobar/239b25c3d8ca9cdf26d51536d4f0208c [16:31:30] Running uid script on an-master1001 [16:31:58] !log sudo bash gid_script.bash on an-maseter1001 [16:32:00] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:33:03] joal: you were talking about 0 as the _tid yesterday, but what's up with 13814000-1dd2-11b2-8080-808080808080? [16:33:35] hehe milimetric :) `Uuid.fromStart(0) --> 13814000-1dd2-11b2-8080-808080808080` [16:34:27] PROBLEM - Hadoop NodeManager on analytics1075 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:34:38] PROBLEM - Hadoop NodeManager on an-worker1085 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:34:41] PROBLEM - Hadoop NodeManager on an-worker1135 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:01] PROBLEM - Hadoop NodeManager on an-worker1081 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:01] PROBLEM - Hadoop NodeManager on analytics1070 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:06] PROBLEM - Hadoop NodeManager on an-worker1091 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:06] PROBLEM - Hadoop NodeManager on an-worker1106 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:06] PROBLEM - Hadoop NodeManager on an-worker1116 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:08] PROBLEM - Hadoop NodeManager on an-worker1120 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:09] PROBLEM - Hadoop NodeManager on analytics1064 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:09] PROBLEM - Hadoop NodeManager on analytics1067 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:10] PROBLEM - Hadoop NodeManager on an-worker1088 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:11] PROBLEM - Hadoop NodeManager on analytics1059 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:12] PROBLEM - Hadoop NodeManager on an-worker1095 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:13] PROBLEM - Hadoop NodeManager on an-worker1092 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:14] PROBLEM - Hadoop NodeManager on analytics1063 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:15] PROBLEM - Hadoop NodeManager on an-worker1109 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:16] PROBLEM - Hadoop NodeManager on an-worker1101 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:17] PROBLEM - Hadoop NodeManager on an-worker1103 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:18] PROBLEM - Hadoop NodeManager on an-worker1084 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:19] PROBLEM - Hadoop NodeManager on an-worker1094 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:20] PROBLEM - Hadoop NodeManager on an-worker1125 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:21] PROBLEM - Hadoop NodeManager on analytics1076 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:22] PROBLEM - Hadoop NodeManager on analytics1058 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:23] PROBLEM - Hadoop NodeManager on an-worker1079 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:26] PROBLEM - Hadoop NodeManager on analytics1071 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:28] PROBLEM - Hadoop NodeManager on an-worker1108 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:28] PROBLEM - Hadoop NodeManager on an-worker1111 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:29] PROBLEM - Hadoop NodeManager on an-worker1114 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:38] PROBLEM - Hadoop NodeManager on an-worker1083 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:41] PROBLEM - Hadoop NodeManager on an-worker1104 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:41] PROBLEM - Hadoop NodeManager on an-worker1118 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:48] PROBLEM - Hadoop NodeManager on analytics1061 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:51] PROBLEM - Hadoop NodeManager on analytics1069 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:51] PROBLEM - Hadoop NodeManager on analytics1073 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:52] PROBLEM - Hadoop NodeManager on an-worker1096 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:54] PROBLEM - Hadoop NodeManager on an-worker1127 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:56] fdans: I have a couple questions about adding the languages to Wikistats2 [16:35:58] PROBLEM - Hadoop NodeManager on analytics1065 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:35:59] razzi: ---^ [16:36:01] PROBLEM - Hadoop NodeManager on an-worker1107 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:36:01] RECOVERY - Hadoop NodeManager on an-worker1085 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:36:01] PROBLEM - Hadoop NodeManager on an-worker1090 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:36:02] PROBLEM - Hadoop NodeManager on an-worker1078 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:36:03] PROBLEM - Hadoop NodeManager on an-worker1131 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:36:04] PROBLEM - Hadoop NodeManager on an-worker1123 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:36:05] PROBLEM - Hadoop NodeManager on an-worker1130 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:36:06] PROBLEM - Hadoop NodeManager on an-worker1098 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:36:11] PROBLEM - Hadoop NodeManager on an-worker1089 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:36:11] PROBLEM - Hadoop NodeManager on an-worker1099 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:36:14] elukey joal - after discussing we are now looking to use thorium for a transfer, i.e. there will be few (possibly only one) downloads per file - and after the transfer the data can be removed. Is this a workable solution from your perspective? [16:36:17] mforns: tardis thing? i can go if you pass me the link [16:36:24] https://meet.google.com/kti-iybt-ekv?pli=1&authuser=1 [16:36:27] https://meet.google.com/kti-iybt-ekv?pli=1 [16:36:31] PROBLEM - Hadoop NodeManager on an-worker1086 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:36:32] RECOVERY - Hadoop NodeManager on an-worker1120 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:36:33] fdans: ok! https://meet.google.com/kti-iybt-ekv [16:36:36] RECOVERY - Hadoop NodeManager on an-worker1095 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:36:52] RECOVERY - Hadoop NodeManager on an-worker1114 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:36:54] razzi: ping ? [16:37:15] btullis, thanks for the bittorrent recommendation, I do like that idea - for now we hope to avoid hosting the data ourselves. [16:37:18] RECOVERY - Hadoop NodeManager on an-worker1127 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:37:21] hmm [16:37:24] elukey: we're in the batcave talking about the issue [16:37:38] sudo -u hdfs /usr/bin/hdfs haadmin -getServiceState an-master1002-eqiad-wmnet comes back active, so I'm not sure what the problem is [16:37:39] RECOVERY - Hadoop NodeManager on an-worker1099 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:37:56] razzi: have you checked after stopping the yarn resource manager on 1001 that 1002 ended up active? [16:37:58] RECOVERY - Hadoop NodeManager on an-worker1116 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:38:04] elukey: an-master1001 processes are down, an-master1002 should have picked - there is something bizarre :( [16:38:11] RECOVERY - Hadoop NodeManager on analytics1076 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:38:38] PROBLEM - Hadoop NodeManager on an-worker1112 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:38:39] PROBLEM - Hadoop NodeManager on an-worker1138 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:39:01] RECOVERY - Hadoop NodeManager on an-worker1135 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:39:06] brb bathroom [16:39:11] there's a lot of noise but we're in safe mode [16:39:19] and backed up :) [16:39:51] RECOVERY - Hadoop NodeManager on an-worker1108 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:40:54] joal: the node manager failed to contact an-master1002 for some reasons [16:40:57] PROBLEM - Hadoop NodeManager on an-worker1080 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:40:57] PROBLEM - Hadoop NodeManager on an-worker1097 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:40:58] PROBLEM - Hadoop NodeManager on an-worker1128 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:40:58] PROBLEM - Hadoop NodeManager on analytics1077 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:41:01] yup elukey [16:41:09] RECOVERY - Hadoop NodeManager on an-worker1094 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:41:11] RECOVERY - Hadoop NodeManager on analytics1058 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:41:22] running "yarn rmadmin -getServiceState an-master1002-eqiad-wmnet" failed [16:41:50] when? [16:41:54] now [16:41:57] RECOVERY - Hadoop NodeManager on an-worker1090 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:42:31] RECOVERY - Hadoop NodeManager on an-worker1091 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:42:36] RECOVERY - Hadoop NodeManager on an-worker1088 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:42:38] RECOVERY - Hadoop NodeManager on analytics1059 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:42:38] RECOVERY - Hadoop NodeManager on an-worker1092 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:43:17] RECOVERY - Hadoop NodeManager on an-worker1138 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:43:18] PROBLEM - Hadoop NodeManager on an-worker1082 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:43:18] PROBLEM - Hadoop NodeManager on an-worker1087 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:43:18] PROBLEM - Hadoop NodeManager on an-worker1121 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:43:19] PROBLEM - Hadoop NodeManager on an-worker1126 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:43:58] elukey: feel free to join us in the batcave if you'd like to listen to us troubleshoot [16:44:03] RECOVERY - Hadoop NodeManager on an-worker1081 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:44:11] RECOVERY - Hadoop NodeManager on an-worker1097 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:44:29] razzi: I communicated the issue to SRE, please do it next time since people were wondering why we had a storm of alerts [16:44:38] RECOVERY - Hadoop NodeManager on an-worker1111 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:45:06] RECOVERY - Hadoop NodeManager on analytics1075 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:45:42] PROBLEM - Hadoop NodeManager on an-worker1110 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:45:42] PROBLEM - Hadoop NodeManager on analytics1062 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:45:43] PROBLEM - Hadoop NodeManager on analytics1072 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:45:54] elukey: problem is due to yarn trying to use HDFS while it is in safemode (for node-labels) [16:45:55] RECOVERY - Hadoop NodeManager on an-worker1128 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:46:16] elukey: I suggest moving out of safemode to let RM recover, then back again [16:46:26] elukey: any counter opinion? [16:46:48] RECOVERY - Hadoop NodeManager on analytics1069 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:46:55] joal: +! [16:46:56] +1 [16:47:14] RECOVERY - Hadoop NodeManager on an-worker1089 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:48:01] PROBLEM - Hadoop NodeManager on an-worker1124 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:49:01] joal: since alerts are spamming, let's either exit safe mode now or downtime hsots [16:49:04] *hosts [16:49:13] ack elukey - trying to solve [16:49:14] RECOVERY - Hadoop NodeManager on analytics1062 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:49:15] RECOVERY - Hadoop NodeManager on analytics1072 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:49:26] PROBLEM - Hadoop NodeManager on an-worker1119 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:50:12] PROBLEM - Hadoop NodeManager on an-worker1136 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:50:21] PROBLEM - Hadoop NodeManager on an-worker1093 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:50:21] PROBLEM - Hadoop NodeManager on an-worker1105 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:50:21] PROBLEM - Hadoop NodeManager on an-worker1115 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:50:32] PROBLEM - Hadoop NodeManager on analytics1068 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:50:36] RECOVERY - Hadoop NodeManager on an-worker1107 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:50:42] RECOVERY - Hadoop NodeManager on an-worker1098 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:51:36] PROBLEM - Hadoop NodeManager on an-worker1137 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:52:00] PROBLEM - Hadoop NodeManager on an-worker1085 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:52:15] PROBLEM - Hadoop NodeManager on an-worker1117 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:52:19] !log starting hadoop processes on an-master1001 since they didn't failover cleanly [16:52:21] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:52:42] PROBLEM - Hadoop NodeManager on analytics1066 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:52:48] PROBLEM - Hadoop NodeManager on an-worker1127 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:52:52] razzi: it is sufficient to just exit safemode [16:53:15] PROBLEM - Hadoop NodeManager on an-worker1099 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:53:15] PROBLEM - Hadoop NodeManager on an-worker1113 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:53:42] RECOVERY - Hadoop NodeManager on an-worker1105 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:54:05] elukey: exiting safemode didn't work [16:54:06] elukey: we're having issues exiting safe-mode [16:54:11] elukey: shall we force? [16:54:12] razzi@an-master1002:~$ sudo -u hdfs kerberos-run-command hdfs hdfs dfsadmin -safemode leave [16:54:12] safemode: Call From an-master1002/10.64.21.110 to an-master1001.eqiad.wmnet:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused [16:55:04] RECOVERY - Hadoop NodeManager on an-worker1131 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:55:05] RECOVERY - Hadoop NodeManager on an-worker1123 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:55:11] Ok, exiting safemode worked just now [16:55:28] RECOVERY - Hadoop NodeManager on an-worker1125 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:55:33] RECOVERY - Hadoop HDFS Zookeeper failover controller on an-master1001 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.hdfs.tools.DFSZKFailoverController https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23HDFS_ZKFC_process [16:55:35] RECOVERY - Hadoop NodeManager on an-worker1086 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:55:40] RECOVERY - Hadoop NodeManager on analytics1061 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:56:02] 10Analytics: Create aggregate alarms for Hadoop daemons running on worker nodes - https://phabricator.wikimedia.org/T287027 (10elukey) [16:56:29] razzi: ah ack I didn't know it [16:56:34] I created a follow up task --^ [16:56:45] RECOVERY - Hadoop NodeManager on analytics1073 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:57:43] RECOVERY - Hadoop NodeManager on analytics1071 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:58:01] elukey: new corner case with node-labels :S [16:58:33] joal: sneaky corner case, we can switch to files-on-host IIRC [16:58:38] RECOVERY - Hadoop NodeManager on an-worker1118 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:58:50] RECOVERY - Hadoop NodeManager on an-worker1115 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:58:54] elukey: I think that would be great! will try to read about that [16:59:10] RECOVERY - Hadoop NodeManager on analytics1065 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:59:56] RECOVERY - Hadoop NodeManager on an-worker1084 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:59:56] RECOVERY - Hadoop NodeManager on an-worker1101 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [16:59:57] RECOVERY - Hadoop NodeManager on an-worker1103 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:00:04] elukey: are you ok with us proceeding on reimaging while not in safemode (due to the problem)? [17:00:10] +1 [17:00:12] ack [17:00:15] thank you a lot :) [17:00:26] RECOVERY - Hadoop NodeManager on an-worker1078 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:00:52] joal: one thing though - on 1001 all the daemons are up, let's wait for full bootstrap before shutting all down again [17:01:05] RECOVERY - Hadoop NodeManager on analytics1077 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:01:08] ack elukey - will make sure this is the case [17:01:12] RECOVERY - Hadoop NodeManager on an-worker1096 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:01:38] RECOVERY - Hadoop NodeManager on an-worker1130 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:02:06] RECOVERY - Hadoop ResourceManager on an-master1001 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.resourcemanager.ResourceManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Resourcemanager_process [17:02:06] RECOVERY - Hadoop NodeManager on an-worker1082 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:02:07] RECOVERY - Hadoop NodeManager on an-worker1080 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:02:08] RECOVERY - Hadoop NodeManager on an-worker1121 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:02:09] RECOVERY - Hadoop NodeManager on analytics1067 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:02:44] btullis: if you want to run 'cumin 'A:hadoop-worker' 'run-puppet-agent' -b 10 to speed up recoveries you can do it [17:03:06] (forces a puppet run in batches of 10, that brings up the nodemanagers down) [17:03:32] RECOVERY - Hadoop NodeManager on an-worker1093 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:03:52] RECOVERY - Hadoop NodeManager on analytics1068 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:04:23] RECOVERY - Hadoop NodeManager on an-worker1136 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:04:23] RECOVERY - Hadoop NodeManager on analytics1063 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:04:42] RECOVERY - Hadoop NodeManager on an-worker1137 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:05:27] RECOVERY - Hadoop NodeManager on an-worker1085 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:05:42] RECOVERY - Hadoop NodeManager on an-worker1113 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:06:15] RECOVERY - Hadoop NodeManager on analytics1066 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:06:26] RECOVERY - Hadoop NodeManager on an-worker1127 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:06:35] RECOVERY - Hadoop NodeManager on an-worker1112 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:07:05] RECOVERY - Hadoop NodeManager on an-worker1117 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:07:30] razzi: https://grafana.wikimedia.org/d/000000585/hadoop?viewPanel=46&orgId=1&from=now-3h&to=now [17:08:28] RECOVERY - Hadoop NodeManager on an-worker1099 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:08:44] 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10Andrew) [17:08:50] RECOVERY - Hadoop NodeManager on an-worker1110 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:08:50] RECOVERY - Hadoop NodeManager on an-worker1119 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:08:51] RECOVERY - Hadoop NodeManager on an-worker1124 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:08:52] RECOVERY - Hadoop NodeManager on an-worker1126 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:11:02] RECOVERY - Hadoop NodeManager on an-worker1106 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:13:12] RECOVERY - Hadoop NodeManager on an-worker1087 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:13:12] RECOVERY - Hadoop NodeManager on analytics1070 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:15:26] RECOVERY - Hadoop NodeManager on an-worker1083 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:15:26] RECOVERY - Hadoop NodeManager on an-worker1104 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:17:02] !log stop all hadoop processes on an-master1001 [17:17:04] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:18:14] ok good, all processes are stopped, I don't see any alerts exploding [17:18:47] razzi@an-master1002:~$ sudo -u yarn kerberos-run-command yarn yarn rmadmin -getServiceState an-master1002-eqiad-wmnet [17:18:47] active [17:19:03] check also hdfs just to be sure [17:19:13] ack elukey [17:19:39] but I don't see any java process on 1001 so we should be good : [17:19:40] :) [17:19:52] RECOVERY - Hadoop NodeManager on an-worker1079 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:19:53] RECOVERY - Hadoop NodeManager on analytics1064 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:21:34] razzi@an-master1002:~$ sudo -u hdfs /usr/bin/hdfs haadmin -getServiceState an-master1002-eqiad-wmnet [17:21:34] active [17:21:59] RECOVERY - Hadoop NodeManager on an-worker1109 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [17:22:08] Going to run the gid script again, just in case the messiness earlier caused it to fail somehow [17:22:13] then should be good to reimage [17:22:35] razzi: not sure if it works after changing [17:23:02] if you ran the script before it should be fine [17:23:11] sure thing, I ran it again, and it caused the same output [17:23:26] next time I'll be more careful to check that such a script is explicitly idempotent, but I think we're ok this time [17:23:50] yes yes no problem, lemme check files [17:23:53] cool [17:24:09] I'm ready to kick off the reimage, the command will be: sudo -i wmf-auto-reimage-host -p T278423 an-master1001.eqiad.wmnet [17:24:09] T278423: Upgrade the Hadoop masters to Debian Buster - https://phabricator.wikimedia.org/T278423 [17:24:22] this is the dangerous one, so I'll wait for elukey and joal to +1 before I go for it [17:24:34] elukey@an-master1001:~$ sudo find /srv/hadoop/name ! -user 903 [17:24:34] elukey@an-master1001:~$ sudo find /srv/hadoop/name ! -group 903 [17:24:34] elukey@an-master1001:~$ sudo find /srv/hadoop/name -user 903 | wc -l [17:24:35] ack :) [17:24:37] 364 [17:24:39] elukey@an-master1001:~$ sudo find /srv/hadoop/name -group 903 | wc -l [17:24:42] 364 [17:24:45] looks good to me (uid/gid I mean) [17:25:06] awesome elukey - +1 for reimage? [17:25:14] checking one thing [17:25:17] sure [17:25:43] +1 should be ok [17:25:59] proceeding with reimage on cumin1001 [17:27:03] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review, 10User-razzi: Upgrade the Hadoop masters to Debian Buster - https://phabricator.wikimedia.org/T278423 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by razzi on cumin1001.eqiad.wmnet for hosts: ` an-master1001.eqiad.wmnet ` The log... [17:27:07] !log razzi@cumin1001:~$ sudo -i wmf-auto-reimage-host -p T278423 an-master1001.eqiad.wmnet [17:27:10] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:37:39] razzi: all good?? [17:38:05] elukey: I'm in the console for the reimage now, and it's asking me which disk to partition [17:38:16] elukey: I don't remember this step from last time, thought there were fewer prompts [17:38:24] wanna take a look? I'm still in the batcave [17:39:50] Partitioning method: [17:39:50] Guided - use entire disk [17:39:50] or [17:39:50] Guided - use entire disk and set up LVM [17:42:17] razzi: so it should be already pre-compiled, with partitions marked as "K" or "F" [17:42:32] hmm I'm not seeing that screen [17:42:46] https://usercontent.irccloud-cdn.com/file/1crxegLa/image.png [17:43:00] maybe I have to proceed through this menu first? [17:43:11] joining bc [17:43:15] thx! [17:53:14] (going afk for a bit but I'll check later! [17:53:31] 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10Bstorm) [17:56:15] Reimage is proceeding smoothly :) [17:56:26] Debian GNU/Linux 10 an-master1001 ttyS1 [18:05:27] an-master1001 is a Hadoop Master (NameNode & ResourceManager) (analytics_cluster::hadoop::master) [18:05:27] The last Puppet run was at Tue Jul 20 17:53:30 UTC 2021 (11 minutes ago). [18:05:27] Last puppet commit: (44723368f1) RLazarus - scap: Drop never-used 'sqldump' tool [18:05:27] Debian GNU/Linux 10 auto-installed on Tue Jul 20 17:51:03 UTC 2021. [18:05:27] razzi@an-master1001:~$ [18:05:39] \\\ \o/ //// [18:05:50] ok cool [18:11:43] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review, 10User-razzi: Upgrade the Hadoop masters to Debian Buster - https://phabricator.wikimedia.org/T278423 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['an-master1001.eqiad.wmnet'] ` and were **ALL** successful. [18:13:55] Patch to re-enable yarn queues once we're ready: https://gerrit.wikimedia.org/r/c/operations/puppet/+/705732 [18:21:46] !log re-enable yarn queues by merging puppet patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/705732 [18:21:48] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:22:25] !log sudo -u hdfs /usr/bin/hdfs haadmin -failover an-master1002-eqiad-wmnet an-master1001-eqiad-wmnet [18:22:27] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:22:59] Failover to NameNode at an-master1001.eqiad.wmnet/10.64.5.26:8040 successful [18:31:55] !log razzi@an-master1002:~$ sudo systemctl stop hadoop-yarn-resourcemanager.service [18:31:57] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:32:02] to manually transition 1001 to active [18:32:11] back! [18:32:23] !log razzi@an-master1002:~$ sudo systemctl start hadoop-yarn-resourcemanager.service [18:32:26] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:32:58] razzi@an-master1002:~$ sudo -u hdfs /usr/bin/hdfs haadmin -getServiceState an-master1001-eqiad-wmnet [18:32:58] active [18:32:58] razzi@an-master1002:~$ sudo -u yarn yarn rmadmin -getServiceState an-master1001-eqiad-wmnet [18:32:58] active [18:33:02] yayyy [18:33:26] going to enable and run puppet on an-launcher, and that's a wrap! [18:33:29] razzi: little nit - for yarn it is sufficient to restart the active one to trigger a failover [18:34:24] !log razzi@an-master1002:~$ sudo -u yarn kerberos-run-command yarn yarn rmadmin -refreshQueues [18:34:24] Oh ok, as opposed to stop / start huh [18:34:25] elukey: he's learnt it the hard wa ;) [18:34:25] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:35:18] I am checking metrics and an-master1001, so far all good [18:35:52] gid/uid all good, partitions good [18:37:37] razzi: spark-shell returns me error, the root queue is still stopped [18:37:42] !log razzi@an-master1002:~$ sudo -i puppet agent --enable [18:37:44] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:38:16] Running puppet on an-master1001 to apply the change to the xml [18:38:37] ah ok perfect [18:39:03] the refreshQueues should be sufficient for the active one [18:39:13] I just noticed that you ran it on 1002, perfect [18:39:47] !log razzi@an-master1001:/var/log/hadoop-hdfs$ sudo -u yarn kerberos-run-command yarn yarn rmadmin -refreshQueues [18:39:49] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:40:11] ok I think we're ready to enable puppet on an-launcher [18:40:21] good work team!!!!!! [18:40:37] !log razzi@an-launcher1002:~$ sudo puppet agent --enable [18:40:39] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:40:54] spark shell works for me, +1 [18:41:05] good job! Finall all on buster :) [18:41:10] just in time for bullseye :D [18:41:19] going to dinner, ttl! [18:42:15] bye elukey - thanks a lot for the help :) [18:42:21] Going to diner soon as well :) [18:42:23] bye elukey, great success! enjoy dinner [18:42:35] you too joal, thanks for all the support! [18:43:03] Happy to help razzi - great work :) [18:47:17] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review, 10User-razzi: Upgrade the Hadoop masters to Debian Buster - https://phabricator.wikimedia.org/T278423 (10razzi) Follow up: change partman to remove -test config, no need to manually confirm the partitions every time since there was no complication [18:49:21] Taking a short computer break, but I'll have my phone so ping me if you need me [18:50:08] Metrics look healthy and no alarms, but I'll keep checking in periodically for the next hour. Let me know if anything looks amiss! [18:51:12] 10Analytics, 10Analytics-Data-Quality, 10Datasets-Webstatscollector: Add alarms for high volume of views to pages with replacement characters - https://phabricator.wikimedia.org/T117945 (10Milimetric) p:05Low→03Medium [18:53:06] 10Analytics, 10Pageviews-API: Pageviews API should allow specifying a country - https://phabricator.wikimedia.org/T245968 (10Milimetric) Please see my question from T245968#5912524, without a good reason we default to not providing data. [18:58:14] !log starting refinery deployment [18:58:16] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:58:20] 10Analytics, 10Analytics-Kanban: Crunch and delete many old dumps logs - https://phabricator.wikimedia.org/T280678 (10Milimetric) Moving back to incoming to triage with Olja [19:00:39] joal: I don't see any further refinery changes merged or unmerged since last deploy on the 15th [19:00:49] Ah? [19:00:53] checking [19:01:28] there have been a deploy on the 15th? [19:01:48] yes, Andrew did I think [19:02:09] mforns: checking [19:03:22] mforns: I confirm the code in on an-launcher1002 - All good :) [19:03:48] so no deploy needed right? [19:04:09] if the only change was sqoop, no need :) [19:04:16] ok, thaaanks joal :] [19:04:22] thank you mforns :) [19:19:18] ok stopping for diner - first gobblin-webrequest job still running, so jobs not yet unlocked - should happend soon [19:30:16] PROBLEM - Check unit status of eventlogging_to_druid_navigationtiming_hourly on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit eventlogging_to_druid_navigationtiming_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [19:41:58] PROBLEM - Check unit status of eventlogging_to_druid_editattemptstep_hourly on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit eventlogging_to_druid_editattemptstep_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [19:46:30] PROBLEM - Check unit status of eventlogging_to_druid_prefupdate_hourly on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit eventlogging_to_druid_prefupdate_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [20:02:27] RECOVERY - Check unit status of eventlogging_to_druid_editattemptstep_hourly on an-launcher1002 is OK: OK: Status of the systemd unit eventlogging_to_druid_editattemptstep_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [20:07:07] RECOVERY - Check unit status of eventlogging_to_druid_prefupdate_hourly on an-launcher1002 is OK: OK: Status of the systemd unit eventlogging_to_druid_prefupdate_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [20:15:05] RECOVERY - Check unit status of eventlogging_to_druid_navigationtiming_hourly on an-launcher1002 is OK: OK: Status of the systemd unit eventlogging_to_druid_navigationtiming_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [20:23:27] systems have recovered after data ingestion caught up - almost done [20:30:12] !log rerun webrequest timed-out instances [20:30:14] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log