[09:56:06] hello, We're going to resume the Netbox upgrade, previous issues solved, let's hope it doesn't take too long this time :) So please don't use Netbox for the next hour or so (until I give an all clear) [11:14:55] there is something wrong with wikitech-static, it's not syncing properly with wikitech [11:15:12] I tried running the sync script manually and it hangs after a few seconds, no error messages [11:15:44] https://phabricator.wikimedia.org/P66893 [11:15:50] any ideas? [11:16:11] last change 21 July 2024 01:16 [11:16:47] dhinus: not sure, historically andrew is the one that handled that, not sure if in recent times soemone else is familiar with that too [11:16:51] that script runs every night on cron, but apparently it's not completing [11:17:02] running manually hangs at different points every time :D [11:17:08] I'll wait for andrew to be online [11:39:01] Netbox: try 2 was unsucessful, everything has been rolled back, you can resume using netbox [14:54:54] re: wikitech-static, andrewbogott got the answer on the first try: the disk is full :D [15:08:44] <_joe_> dhinus: ouch [15:08:53] <_joe_> there's a db there? [15:10:51] not sure, andrewbogott is looking at it [15:11:54] yes, a full mediawiki install. [15:12:31] <_joe_> oh damn [15:14:31] /srv/mediawiki/images/wikitech/archive is 39G... [15:14:48] over 76G used [15:14:54] yeah, it's pretty much always image spam piling up there [15:15:11] * andrewbogott puts down what he was doing [15:15:12] maybe the archived ones are not really needed? if the path matches the functionality [15:15:37] it's not a problem to clean up space, I just haven't started cleaning yet [15:16:34] hm, good job mediawiki [15:16:36] php maintenance/deleteArchivedFiles.php --delete [15:16:41] Done! [0 file(s)] [15:18:53] hello oncallers, I am going to test a puppet private commit from puppetserver1001 - for the moment please do not commit/merge anything private from puppetmaster1001 [15:19:09] * andrewbogott finds " find . -type f | xargs rm" to be super effective [15:29:52] find [options] -delete is safer ;) [15:30:42] you can restart committing to private (third attempt failed, forgot one ferm rule sigh) [15:32:38] klausman: tell me more about xargs vs -delete? I've always assumed it's just a question of whether or not I was going to overflow xargs [15:32:45] I guess there could be races as well [15:33:06] well, xargs (unless you use -print0 and xargs -0) can be a landmine [15:33:28] yeah, whitespace in filenames and other parsing issues [15:33:47] find's -delete avoids that, and auto-enables -depth, so if you delete dirs and files, it will do them in the right order. [15:34:09] oh, whitespace, good point [15:34:11] From painful past experience, I would never run "find ... | xargs rm". Too risky. Run "find ... | sed "s/.*/rm &/" > tempfile" then at least glance at tempfile to verify the list is sane, then you can so "sh tempfile". [15:34:18] touch "/ foo"; find .|xargs rm -rf [15:34:19] granted, just running `find . -delete` is nuclear. [15:34:42] Guess I've been lucky :( [15:34:52] roy, the "generate a shellscript, read it, and only execute if satisified" approach is very valid, yes [15:35:29] I definitely do that if the operations are more complicated (multiple commands, complex slicing and dicing of results) [15:41:17] Not to mention Fear Of Bobby Tables. What if you had a file whose name was "; sudo rm -rf /" ? [15:43:22] rm -rf / fears are grossly exaggerated since --preserve-root is default (almost?) everywhere now, but still [15:44:10] I usually do a find . -type f [whatever options] -print and give it a glance, then replace with -delete [15:44:23] 'rm -rf /usr' is a good alternative [15:44:47] :P [15:45:31] fair [15:48:57] @claime repeating a command from the command history with a trivial change to make it go live is a reasonable strategy. [15:51:37] When I write command-line utilities which could do a lot of damage, I often provide a boolean safety option which defaults to --dryrun (i.e. just print out what you would do, but don't actually do it). Then you can say --no-dryrun to actually pull the trigger. [15:52:17] Paranoid perhaps, but born of too many self-inflicted train wrecks. [16:57:55] <_joe_> roy649: most of the tools WMF SRE built either offer a dry run or a way to see what would change and ask for confirmation [16:58:15] this makes me happy! [16:58:34] <_joe_> (or both, I should add) [16:58:35] it makes us happy too :D [19:41:43] claime: if you're still around... I'm seeing failures in profile/spec/classes/profile_lvs_realserver_spec.rb which I suspect are due to recent appserver changes [19:42:01] andrewbogott: we are discussing in -ops and a fix is coming [19:42:06] ok! [19:42:13] too many channels [19:42:16] it's late for claim.e, swfrench-wmf is patching it [19:42:17] yeah [19:42:55] thanks for the redirect, and for your patience, andrewbogott [19:43:53] no worries! [21:04:24] FYI, LVS services for appservers-http and api-https have now been turned down. Nothing notable to report at this time (most issues have involved puppet, and are now fixed). [21:04:24] https://phabricator.wikimedia.org/T367949 tracks the changes made today and the current state. [21:04:24] my sincere thanks to everyone who helped get this out the door :) [21:26:14] #serverless