[07:26:56] MacFan4000, dmehus, Sario: looking at moving bots phab to Wikimedia fork [07:27:07] It might provide us cover for a bit [15:19:01] $log stopped phd [15:19:06] Saved item "stopped phd" [15:19:32] MacFan4000: no Notifico for puppet? [15:21:38] $log restart apache2 on tools1 [15:21:43] RhinosF1: it’s not working anywhere [15:21:45] Saved item "restart apache2 on tools1" [15:22:02] Ozone klined it the other day [15:22:27] I mentioned it in #notifico but haven’t gotten a response [15:22:35] ok [15:23:24] MacFan4000: how do I check apache2 picked up https://github.com/MirahezeBots/puppet/commit/616d02e351ff69f896c7cbf48bc385eb8c61ea29 [15:23:25] [ Update phab.mirahezebots.org-le-ssl.conf · MirahezeBots/puppet@616d02e · GitHub ] - github.com [15:24:29] is restarting apache enough? [15:24:42] It definitely did considering https://phab.mirahezebots.org has errors [15:24:43] [ Front Page ] - phab.mirahezebots.org [15:24:46] Yeah should be [15:25:10] MacFan4000: what error you see? [15:25:29] https://www.irccloud.com/pastebin/8PcJywFt [15:25:30] [ Snippet | IRCCloud ] - www.irccloud.com [15:25:44] https://www.irccloud.com/pastebin/mplKkh41 [15:25:44] [ Snippet | IRCCloud ] - www.irccloud.com [15:25:59] MacFan4000: where? [15:26:00] https://www.irccloud.com/pastebin/ZLG7IJfR [15:26:00] [ Snippet | IRCCloud ] - www.irccloud.com [15:26:15] Those three when trying to access phab [15:26:41] I just see read only [15:28:10] Now it says it can’t connect to MariaDB [15:28:19] yeah i saw [15:28:36] PROBLEM - Phabricator on tools1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 5195 bytes in 0.103 second response time [15:32:47] MacFan4000: where does it log too? [15:38:24] $log reboot tools1 [15:38:31] Saved item "reboot tools1" [15:39:22] Should be standerd Apache log (/var/log/apache2/error.log) [15:40:19] MacFan4000: for some reason, it's not picking up local config [15:48:04] Apache is pointing to the old die for some reason [15:48:09] dir* [15:48:24] MacFan4000: why [15:48:36] I updated the templates [15:48:49] No idea, puppet should have carried out the change [15:51:27] I just made a test change, and when running puppet, I don’t see the file being updated [15:51:51] The manifest entry is https://github.com/MirahezeBots/puppet/blob/master/modules/apache/manifests/init.pp#L79 [15:51:52] [ puppet/init.pp at master · MirahezeBots/puppet · GitHub ] - github.com [15:52:06] https://github.com/MirahezeBots/puppet/commit/9ea26d81a9dee206b3b4a169b5da2daeb74e60ea [15:52:06] [ Update init.pp · MirahezeBots/puppet@9ea26d8 · GitHub ] - github.com [15:52:16] Ah [15:53:25] nope [15:57:07] $log manually deploy apache changes [15:57:13] Saved item "manually deploy apache changes" [15:57:52] MacFan4000: It runs! [15:59:07] We still should figure out why puppet isn’t properly managing the files [15:59:47] yeah [15:59:49] file a ask [15:59:52] task [16:00:23] RhinosF1: daemons aren’t running [16:00:49] MacFan4000: should be [16:00:56] It says ok [16:01:07] Looks good now [16:01:52] $log clear phab cache, update done [16:01:57] Saved item "clear phab cache, update done" [16:06:02] $log restart icingabot [16:06:02] RECOVERY - Phabricator on tools1 is OK: HTTP OK: HTTP/1.1 200 OK - 22178 bytes in 0.329 second response time [16:06:08] Saved item "restart icingabot" [16:06:28] RECOVERY - Icinga IRC bot on tools1 is OK: PROCS OK: 1 process with command name 'irc.py' [16:13:48] MacFan4000: do you see login with mediawiki [16:17:53] It comes from phabricator-extensions [16:18:22] MacFan4000: which exist [16:18:33] I have no clue how it's setup [16:18:40] there's no config for it anywhere [16:20:22] I don’t see it in /var/phab-deploy [16:21:03] rhinosf1@tools1:/etc/puppet/private/hieradata$ ls /var/phab-deploy/libext/misc [16:21:12] I just found it [16:22:29] The library has to be loaded [16:22:54] can you? [16:23:58] https://phab.mirahezebots.org/T254 [16:23:59] [ ⚓ T254 Extensions on new phab aren't loaded ] - phab.mirahezebots.org [16:24:07] via puppet if possible [16:25:38] Wmf seem to symlink stuff [16:31:51] PROBLEM - Phabricator on tools1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4188 bytes in 0.104 second response time [16:35:49] RECOVERY - Phabricator on tools1 is OK: HTTP OK: HTTP/1.1 200 OK - 22186 bytes in 0.212 second response time [16:36:50] RhinosF1: fixed [16:37:19] MacFan4000: how? [16:37:44] RhinosF1: see my latest commits to puppet [16:39:17] MacFan4000: why not use load extensions like at first [16:39:48] It wasn’t setup correctly for that [16:40:25] Oh ok [16:40:45] Could we symlink libext with the extensions folder [16:41:19] It has to be setup like https://github.com/Test-Wiki/phabricator-extensions or https://github.com/wikimedia/phabricator-extensions for load-libraries to work [16:41:19] [ GitHub - Test-Wiki/phabricator-extensions: Phabricator customizations ] - github.com [16:41:21] [ GitHub - wikimedia/phabricator-extensions: Github mirror of "phabricator/extensions" - our actual code is hosted in phabricator ] - github.com [16:41:46] I could try to [16:43:33] See what works [16:43:43] I just don't want a copy in 2 places [16:43:55] You can turn on the others in libext if you want [16:45:42] No old MediaWiki logo now! [16:45:51] It uses wikimedia though [16:48:26] RhinosF1, oh, Wikimedia has forked Phabricator already? Where's the repo? [16:48:47] dmehus: they always have [16:48:51] It's not new [16:48:56] RhinosF1, ah didn't know that [16:49:00] Because they wanted a few extra changes [16:49:20] ah, so they haven't talked about migrating away from Phabricator yet then, I assume [16:49:34] since they're maintaining their own fork anyway [16:50:55] Not in the immediate [16:51:05] They're will be eventually [16:51:34] RhinosF1, ack, okay cool [16:52:01] But they're willing to keep it going for now [16:52:58] dmehus: there is very little from source installed on WMF servers that they don't control [16:53:36] They even have staff working with Debian to deploy stuff to Debian packages [16:59:26] ah [17:00:37] from /usr/lib/nagios/plugins/check_puppet_run:154:in `

' [17:01:37] RECOVERY - Puppet on bots1 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [17:01:59] RECOVERY - Puppet on tools1 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [17:02:45] RhinosF1: it’s switched to a symlink now [17:02:56] MacFan4000: nice! [17:03:38] MacFan4000: feel free to test loading the upstream extensions [17:03:45] From WMF [17:05:25] I’ve gotten some of them working before on TestWiki phab [17:05:55] I have a repo for it - I did rewrite their OAuth scripts though [17:07:48] Primarily what I didn’t try to get working is anything to do with LDAP or Herriot [17:07:53] gerrit* [17:08:32] honestly, I don’t see a need for further extensions [17:19:28] MacFan4000: see if anything in it useful [17:19:34] I didn't really look [17:20:19] That’s really what I did with https://github.com/Test-Wiki/phabricator-extensions [17:20:20] [ GitHub - Test-Wiki/phabricator-extensions: Phabricator customizations ] - github.com [17:20:38] Ok [17:21:13] The only other thing I ended up putting there was the protect as a security issue app [17:29:17] That would be nice [17:31:46] I’ll look at working on that when I’m back on my laptop [17:35:51] I could just use the test wiki phab-exts repo but because I rewrote the OAuth stuff to use OAuth 2 implementing that I believe would mean that everyone would have to re-link their miraheze account [17:36:14] Oh that not good