[00:00:13] Will merge in about 5 minutes. [00:00:24] Thank you, that was quick :) [00:00:31] Actually will merge now. [00:00:37] [02mw-config] 07Universal-Omega closed pull request 03#4172: Fix enabling CommentStreams - 13https://git.io/JiIlw [00:00:39] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 [00:00:40] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JiIlP [00:00:42] [02miraheze/mw-config] 07Universal-Omega 038d71fcb - Fix enabling CommentStreams (#4172) [00:00:43] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 - 13https://git.io/vbvb3 [00:00:46] miraheze/mw-config - Universal-Omega the build passed. [00:01:30] Agent: no problem, and I'm pretty sure this will work although I can't be 100% sure as it works on 1.37 so test3 can't be tested. But won't have negative effects. [00:01:40] miraheze/mw-config - Universal-Omega the build passed. [00:02:31] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.50, 20.40, 16.26 [00:02:59] PROBLEM - cp15 Current Load on cp15 is CRITICAL: CRITICAL - load average: 3.49, 2.85, 1.59 [00:05:07] !log [@test3] starting deploy of {'config': True} to skip [00:05:08] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [00:05:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:05:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:05:41] !log [@mw11] starting deploy of {'config': True} to all [00:05:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:06:04] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 22s [00:06:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:06:30] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.25, 6.31, 5.21 [00:06:59] PROBLEM - cp15 Current Load on cp15 is WARNING: WARNING - load average: 1.35, 1.86, 1.45 [00:08:30] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.73, 5.71, 5.13 [00:09:56] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.75, 6.87, 4.43 [00:10:59] RECOVERY - cp15 Current Load on cp15 is OK: OK - load average: 1.24, 1.49, 1.39 [00:11:52] Agent: let me know if it works please. [00:12:17] Sure, let me try it out [00:14:30] CosmicAlpha: it worked, thank you! [00:14:41] Agent: great! And no problem [00:33:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.72, 3.54, 3.02 [00:35:30] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.19, 3.32, 2.99 [00:39:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.27, 3.43, 3.12 [00:41:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.83, 3.31 [00:43:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.72, 3.45, 3.23 [00:45:15] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.18, 3.63, 3.32 [00:48:56] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 2.06, 3.96, 5.64 [00:49:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 4.00, 3.83, 3.48 [00:51:15] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.96, 4.09, 3.63 [00:53:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.80, 3.68, 3.54 [00:54:31] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.67, 18.49, 23.96 [00:54:56] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.56, 3.65, 4.95 [00:59:15] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.02, 3.22, 3.37 [01:02:31] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.77, 15.83, 20.37 [01:13:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 4.00, 3.51, 3.33 [01:15:15] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.30, 3.84, 3.48 [01:19:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.88, 3.65, 3.47 [01:23:15] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.04, 3.21, 3.33 [01:27:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.79, 3.58, 3.46 [01:29:15] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.73, 3.20, 3.33 [04:17:43] PROBLEM - wiki.thedev.gq - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.thedev.gq' expires in 7 day(s) (Sat 30 Oct 2021 04:16:12 GMT +0000). [05:16:08] PROBLEM - www.chinatech.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.chinatech.wiki' expires in 15 day(s) (Sun 07 Nov 2021 05:08:44 GMT +0000). [05:25:29] PROBLEM - rww2.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'rww2.org' expires in 15 day(s) (Sun 07 Nov 2021 05:19:14 GMT +0000). [05:26:48] PROBLEM - otcg.ml - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'otcg.ml' expires in 15 day(s) (Sun 07 Nov 2021 05:17:51 GMT +0000). [05:48:36] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 25.98, 23.11, 15.54 [05:48:38] PROBLEM - mwtask1 Disk Space on mwtask1 is CRITICAL: DISK CRITICAL - free space: / 104 MB (0% inode=90%); [05:50:35] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 18.58, 20.59, 15.49 [05:52:31] PROBLEM - mwtask1 Disk Space on mwtask1 is WARNING: DISK WARNING - free space: / 3346 MB (7% inode=90%); [05:52:35] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.90, 19.60, 15.70 [05:54:28] RECOVERY - mwtask1 Disk Space on mwtask1 is OK: DISK OK - free space: / 7433 MB (15% inode=90%); [06:03:34] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.78, 5.47, 3.37 [06:05:34] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.28, 5.69, 3.71 [06:09:34] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.70, 7.49, 4.92 [06:13:34] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.67, 7.90, 5.71 [06:15:34] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.90, 8.94, 6.38 [06:15:49] PROBLEM - db11 APT on db11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:19:57] RECOVERY - db11 APT on db11 is OK: APT OK: 71 packages available for upgrade (0 critical updates). [06:21:20] RECOVERY - db11 Disk Space on db11 is OK: DISK OK - free space: / 52026 MB (11% inode=97%); [06:21:33] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 3.35, 7.36, 6.73 [06:21:35] alerting : [FIRING:1] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:23:33] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 2.34, 5.66, 6.18 [06:26:35] ok : [RESOLVED] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:32:11] PROBLEM - www.arru.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'www.arru.xyz' expires in 7 day(s) (Sat 30 Oct 2021 06:30:00 GMT +0000). [06:36:27] PROBLEM - arru.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'www.arru.xyz' expires in 7 day(s) (Sat 30 Oct 2021 06:30:00 GMT +0000). [08:10:05] [02mw-config] 07ugochimobi commented on pull request 03#4171: otherProjectsLinks on FAMEPedia and FAMEData - 13https://git.io/JitYI [11:32:11] PROBLEM - wikimoma.art - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikimoma.art' expires in 15 day(s) (Sun 07 Nov 2021 11:27:59 GMT +0000). [12:13:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.91, 3.36, 2.74 [12:21:15] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.93, 3.32, 3.00 [12:27:10] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.69, 1.77, 1.05 [12:29:10] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.43, 1.28, 0.96 [14:04:04] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.78, 3.52, 3.04 [14:06:01] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.03, 3.31, 3.02 [15:11:52] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/Jiqye [15:11:54] [02miraheze/ssl] 07Reception123 0325e7d0e - add adanfime.com cert [15:41:19] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 3.66, 3.11 [15:45:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.80, 3.60, 3.19 [15:47:19] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.75, 3.86, 3.33 [15:48:38] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.98, 4.99, 3.54 [15:49:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.77, 3.36 [15:50:38] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.85, 4.02, 3.35 [15:53:19] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.38, 3.13, 3.20 [16:30:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JimJL [16:30:13] [02miraheze/services] 07MirahezeSSLBot 03aaf2dcd - BOT: Updating services config for wikis [16:38:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 3.72, 3.46 [16:40:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.16, 3.34, 3.34 [17:15:51] [02miraheze/ManageWiki] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/mediawiki/mediawiki-codesniffer-38.0.0 [+0/-0/±1] 13https://git.io/JimOa [17:15:53] [02miraheze/ManageWiki] 07dependabot[bot] 0393ecd20 - Update mediawiki/mediawiki-codesniffer requirement from 37.0.0 to 38.0.0 [17:15:54] [02ManageWiki] 07dependabot[bot] opened pull request 03#306: Update mediawiki/mediawiki-codesniffer requirement from 37.0.0 to 38.0.0 - 13https://git.io/JimOV [17:15:56] [02ManageWiki] 07dependabot[bot] created branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-38.0.0 - 13https://git.io/vpSns [17:15:57] [02ManageWiki] 07dependabot[bot] labeled pull request 03#306: Update mediawiki/mediawiki-codesniffer requirement from 37.0.0 to 38.0.0 - 13https://git.io/JimOV [17:15:59] [02ManageWiki] 07dependabot[bot] labeled pull request 03#306: Update mediawiki/mediawiki-codesniffer requirement from 37.0.0 to 38.0.0 - 13https://git.io/JimOV [17:19:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.86, 3.17, 2.94 [17:21:19] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.27, 3.20, 2.98 [17:22:06] miraheze/ManageWiki - dependabot[bot] the build passed. [17:52:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.09, 3.25, 2.84 [17:54:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.19, 3.09, 2.83 [18:25:58] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.39, 3.60, 3.35 [18:31:41] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.44, 3.21, 3.28 [18:31:59] @paladox can you check jobs for metawiki [18:32:10] Wiki creation seems slow [18:33:01] https://www.irccloud.com/pastebin/AJJbuI3r/ [18:33:29] Hmm weird [18:33:31] Thanks [19:00:49] !log [universalomega@mwtask1] Started running checkStorage.php --fix on all wikis [19:00:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:06:27] That worked [19:06:30] \o/ [19:06:56] CosmicAlpha: ty for fixing [19:08:48] Spookreeeno: No problem, this will take awhile to finish (probably at least the rest of today at current pace), but it has worked on wikis it's been ran on so far. checkStorage was not an easy script to get working though... [19:09:02] the shell script should log when done though. [19:10:34] CosmicAlpha: yep! [19:10:41] Not expecting it to be fast [19:11:20] CosmicAlpha: I notice some fails from the deploy script [19:12:09] Spookreeeno: Yeah and I had to add some sleep commands to shell script, otherwise it tries to gunzip, and run script, and remove at the same time it seems, sometimes wrong order, to it takes at least 6 seconds each wiki. [19:12:25] Ah! [19:12:26] And you mean for test3? [19:12:52] Looks like it [19:13:06] Just wondering why [19:13:20] In case I can tidy anything up [19:13:30] Spookreeno, yeah that was when I was deploying the 1.37 update, vendor was out-of-sync, so I had to reset vendor, run again, and it worked. [19:13:41] Spookreeeno: ^ [19:14:00] Ack [19:14:09] Vendor should never go out of sync [19:20:46] [02puppet] 07Universal-Omega opened pull request 03#2086: Add cdn.jsdelivr.net and storage.googleapis.com to `connect-src` - 13https://git.io/Jimz5 [19:35:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.48, 3.33, 3.09 [19:36:11] PROBLEM - ping4 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 164.30 ms [19:36:42] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 163.76 ms [19:37:19] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.29, 2.97, 2.99 [19:41:29] [02puppet] 07Reception123 closed pull request 03#2086: Add cdn.jsdelivr.net and storage.googleapis.com to `connect-src` - 13https://git.io/Jimz5 [19:41:31] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jim2z [19:41:32] [02miraheze/puppet] 07Universal-Omega 0396ccbc8 - Add cdn.jsdelivr.net and storage.googleapis.com to `connect-src` (#2086) [20:09:35] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.24, 3.12, 2.93 [20:11:29] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.23, 2.70, 2.80 [20:16:58] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+1/-0/±1] 13https://git.io/Jimwf [20:17:00] [02miraheze/mediawiki] 07Universal-Omega 0301eb336 - Install MixedNamespaceSearchSuggestions [20:19:23] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+1/-0/±1] 13https://git.io/Jimw3 [20:19:24] [02miraheze/mediawiki] 07Universal-Omega 0374bf462 - Install MixedNamespaceSearchSuggestions [20:19:44] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://git.io/Jimwn [20:19:46] [02miraheze/mw-config] 07Universal-Omega 03b843a32 - Update ManageWikiExtensions.php [20:19:47] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-5 - 13https://git.io/vbvb3 [20:20:06] [02mw-config] 07Universal-Omega opened pull request 03#4173: T7990: add MixedNamespaceSearchSuggestions - 13https://git.io/JimwC [20:21:09] miraheze/mw-config - Universal-Omega the build passed. [20:21:29] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://git.io/Jimwg [20:21:31] [02miraheze/mw-config] 07Universal-Omega 03073410f - Update extension-list [20:21:32] [02mw-config] 07Universal-Omega synchronize pull request 03#4173: T7990: add MixedNamespaceSearchSuggestions - 13https://git.io/JimwC [20:22:36] miraheze/mw-config - Universal-Omega the build passed. [20:23:30] We're seeing a WMF outage [20:23:36] Looks bad if people are having issues [20:23:39] CosmicAlpha: ^ [20:24:23] !log [universalomega@test3] starting deploy of {'world': True, 'gitinfo': True, 'extensionlist': True} to skip [20:24:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:25:30] CosmicAlpha: can we be careful deploying please in case instant commons or something bites us? [20:25:50] !log [universalomega@mw11] starting deploy of {'world': True, 'gitinfo': True, 'extensionlist': True} to all [20:25:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:26:19] Spookreeeno: Oh, just so your messages. Sorry. Want me to cancel? [20:26:24] *saw [20:26:36] CosmicAlpha: no go ahead with that [20:26:39] Just be aware [20:26:52] Only abort if issues crop up that don't look right [20:27:03] Alright will do. Thanks! [20:29:36] [02mw-config] 07Universal-Omega commented on pull request 03#4171: otherProjectsLinks on FAMEPedia and FAMEData - 13https://git.io/JimrZ [20:33:00] paladox: any issues with doing https://github.com/miraheze/mw-config/pull/4159, to enable some more logging on database stuff? [20:33:01] [url] enable/modify database logging channels by Universal-Omega · Pull Request #4159 · miraheze/mw-config · GitHub | github.com [20:33:18] sure [20:33:20] though [20:33:24] keep an eye on disk space! [20:33:56] CosmicAlpha: let's hold until we know wikimedia sites won't get worse [20:34:28] W [20:34:34] We have no cause or ETA yet [20:34:51] Spookreeeno, I cancelled deploy. It failed to connect to gerrit, it said. [20:35:01] it connects to gerrit? [20:35:10] oh right [20:35:14] CosmicAlpha: ye known [20:35:29] Gerrit doesn't go through cp afaik [20:35:35] You hit eqiad directly [20:35:49] paladox: I am right? ^ [20:35:58] yeh [20:36:33] !log mediawiki deploys stopped - ongoing incident cc !sre [20:36:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:38:16] .op [20:38:16] Attempting to OP... [20:38:56] Strange, I can reach gerrit without issues [20:39:07] I cannot reach gerrit [20:39:11] europe issue [20:39:37] I can't [20:40:17] I guess I can reach it since I’m in the US [20:42:51] Spookreeeno: If we set up a check, for wikimedia API, similar to how deploy-mediawiki checks ours, we can probably disable InstantCommons in the event it's unreachable, preventing any issues on our end besides commons itself, right? Or is that not a good idea. Though actually, I thought we fixed the issue a couple months ago, so we don't go down anymore, so we shouldn't anyways I guess. [20:43:37] CosmicAlpha: it's better [20:43:45] I don't think we'd have seen it today [20:43:56] I think gerrit was our main issue [20:44:05] Because of the nature today [20:44:10] As we'll hit essams [20:44:22] Give it 5 but we should be back [20:45:17] * Spookreeeno wonders if we should check_up gerrit before deploying [20:47:53] CosmicAlpha: should be safe in ~10 minutes [20:55:42] CosmicAlpha: can you test we can ping gerrit from mw11 [20:55:53] If you can then free to go ahead [20:55:59] !log deployments should be safe [20:56:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:56:29] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.77, 1.93, 1.19 [20:57:44] !log [universalomega@test3] starting deploy of {'world': True, 'gitinfo': True, 'extensionlist': True} to skip [20:57:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:57:54] !log [universalomega@mw11] starting deploy of {'world': True, 'gitinfo': True, 'extensionlist': True} to all [20:57:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:58:25] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.81, 1.51, 1.12 [20:58:33] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 - 13https://git.io/vbvb3 [20:58:35] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JimKV [20:58:36] [02miraheze/mw-config] 07Universal-Omega 03046cd05 - enable/modify database logging channels (#4159) [20:58:38] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 [20:58:39] [02mw-config] 07Universal-Omega closed pull request 03#4159: enable/modify database logging channels - 13https://git.io/JKi4C [20:59:37] miraheze/mw-config - Universal-Omega the build passed. [21:01:20] !log [universalomega@test3] finished deploy of {'world': True, 'gitinfo': True, 'extensionlist': True} to skip - SUCCESS in 216s [21:01:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:02:44] [02mw-config] 07ugochimobi commented on pull request 03#4171: otherProjectsLinks on FAMEPedia and FAMEData - 13https://git.io/Jim6e [21:03:59] !log [@test3] starting deploy of {'config': True} to skip [21:04:00] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [21:04:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:04:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:04:24] !log [universalomega@mw11] finished deploy of {'world': True, 'gitinfo': True, 'extensionlist': True} to all - SUCCESS in 389s [21:04:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:04:51] !log [@mw11] starting deploy of {'config': True} to all [21:04:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:04:56] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 5s [21:05:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:05:05] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jim6C [21:05:07] [02miraheze/mw-config] 07Universal-Omega 039ad71ab - T8196: set $wgDplSettings['maxResultCount'] = 2500 for constantnoblewiki [21:06:05] miraheze/mw-config - Universal-Omega the build passed. [21:07:47] [02mw-config] 07Universal-Omega closed pull request 03#4173: T7990: add MixedNamespaceSearchSuggestions - 13https://git.io/JimwC [21:07:49] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 - 13https://git.io/vbvb3 [21:07:50] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 [21:07:52] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jim6r [21:07:53] [02miraheze/mw-config] 07Universal-Omega 032f39259 - T7990: add MixedNamespaceSearchSuggestions (#4173) [21:08:55] miraheze/mw-config - Universal-Omega the build passed. [21:08:59] !log [@mw11] starting deploy of {'config': True} to all [21:09:04] !log [@test3] starting deploy of {'config': True} to skip [21:09:05] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [21:09:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:09:10] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 11s [21:09:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:09:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:09:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:09:41] !log [universalomega@mw11] starting deploy of {'l10n': True} to all [21:09:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:10:00] !log [universalomega@test3] starting deploy of {'l10n': True} to skip [21:10:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:15:32] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.88, 4.82, 3.14 [21:16:00] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 28.78, 22.96, 18.18 [21:17:32] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 3.06, 4.07, 3.07 [21:18:01] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.02, 21.90, 18.37 [21:18:54] !log [universalomega@test3] finished deploy of {'l10n': True} to skip - SUCCESS in 533s [21:18:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:20:00] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.40, 20.10, 18.16 [21:20:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.35, 20.37, 16.63 [21:22:49] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.40, 21.83, 17.58 [21:24:55] [02miraheze/WikiDiscover] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/mediawiki/mediawiki-codesniffer-38.0.0 [+0/-0/±1] 13https://git.io/JimiD [21:24:55] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.26, 20.63, 17.70 [21:24:56] [02miraheze/WikiDiscover] 07dependabot[bot] 0305b427a - Update mediawiki/mediawiki-codesniffer requirement from 37.0.0 to 38.0.0 [21:24:58] [02WikiDiscover] 07dependabot[bot] created branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-38.0.0 - 13https://git.io/vhUAp [21:24:59] [02WikiDiscover] 07dependabot[bot] opened pull request 03#72: Update mediawiki/mediawiki-codesniffer requirement from 37.0.0 to 38.0.0 - 13https://git.io/Jimiy [21:25:01] [02WikiDiscover] 07dependabot[bot] labeled pull request 03#72: Update mediawiki/mediawiki-codesniffer requirement from 37.0.0 to 38.0.0 - 13https://git.io/Jimiy [21:25:02] [02WikiDiscover] 07dependabot[bot] labeled pull request 03#72: Update mediawiki/mediawiki-codesniffer requirement from 37.0.0 to 38.0.0 - 13https://git.io/Jimiy [21:26:09] !log [universalomega@mw11] finished deploy of {'l10n': True} to all - SUCCESS in 988s [21:26:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:26:50] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.41, 19.42, 17.59 [21:26:53] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 30.44, 29.84, 23.03 [21:27:32] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.43, 5.75, 4.15 [21:29:32] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 3.73, 5.19, 4.15 [21:30:40] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 18.63, 23.79, 22.09 [21:31:16] miraheze/WikiDiscover - dependabot[bot] the build passed. [21:31:32] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 8.96, 6.37, 4.69 [21:31:42] [02WikiDiscover] 07RhinosF1 commented on pull request 03#72: Update mediawiki/mediawiki-codesniffer requirement from 37.0.0 to 38.0.0 - 13https://git.io/JimPq [21:31:47] [02WikiDiscover] 07RhinosF1 closed pull request 03#72: Update mediawiki/mediawiki-codesniffer requirement from 37.0.0 to 38.0.0 - 13https://git.io/Jimiy [21:31:49] [02miraheze/WikiDiscover] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JimPY [21:31:50] [02miraheze/WikiDiscover] 07dependabot[bot] 03c224d1c - Update mediawiki/mediawiki-codesniffer requirement from 37.0.0 to 38.0.0 (#72) [21:31:52] [02WikiDiscover] 07RhinosF1 deleted branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-38.0.0 - 13https://git.io/vhUAp [21:31:53] [02miraheze/WikiDiscover] 07RhinosF1 deleted branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-38.0.0 [21:33:24] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.48, 5.99, 4.44 [21:33:38] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 33.14, 24.64, 20.24 [21:35:20] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.21, 5.07, 4.27 [21:35:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.54, 22.98, 20.12 [21:37:32] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.36, 5.73, 5.02 [21:38:31] miraheze/WikiDiscover - RhinosF1 the build passed. [21:39:22] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.43, 17.90, 18.73 [21:39:32] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 2.31, 4.46, 4.64 [21:42:03] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 13.33, 17.26, 19.91 [21:58:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.45, 18.05, 17.27 [22:00:49] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.02, 17.92, 17.32 [22:06:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.04, 2.92 [22:09:58] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.66, 3.05, 2.96 [22:16:17] [02miraheze/CreateWiki] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/mediawiki/mediawiki-codesniffer-38.0.0 [+0/-0/±1] 13https://git.io/Jim1A [22:16:19] [02miraheze/CreateWiki] 07dependabot[bot] 03e041b22 - Update mediawiki/mediawiki-codesniffer requirement from 37.0.0 to 38.0.0 [22:16:20] [02CreateWiki] 07dependabot[bot] created branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-38.0.0 - 13https://git.io/vpJTL [22:16:22] [02CreateWiki] 07dependabot[bot] opened pull request 03#252: Update mediawiki/mediawiki-codesniffer requirement from 37.0.0 to 38.0.0 - 13https://git.io/Jim1x [22:16:23] [02CreateWiki] 07dependabot[bot] labeled pull request 03#252: Update mediawiki/mediawiki-codesniffer requirement from 37.0.0 to 38.0.0 - 13https://git.io/Jim1x [22:16:25] [02CreateWiki] 07dependabot[bot] labeled pull request 03#252: Update mediawiki/mediawiki-codesniffer requirement from 37.0.0 to 38.0.0 - 13https://git.io/Jim1x [22:23:39] miraheze/CreateWiki - dependabot[bot] the build passed. [22:32:01] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.72, 1.71, 1.24 [22:33:57] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.65, 1.32, 1.16 [22:48:13] The wiki DataDump is failing to generate the file for main and user namespace https://www.portalsofphereon.com/wiki/Special:DataDump This is probably related to previously enabled extensions on the wiki (i trimmed the activated extensions to the actually the ones we use), last time i did a dump i had problems with pages because of the content model log and had to enable some extensions back. I am just creating a Quarter/after a lot [22:48:13] of activity periodic backup. Could this get checked? Thanks. [22:48:14] [url] Permission error - Portals of Phereon Wiki | www.portalsofphereon.com [22:59:32] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 7.73, 4.79, 3.36 [23:00:00] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 29.62, 24.10, 18.28 [23:01:33] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 3.18, 4.15, 3.31 [23:05:38] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 18.65, 9.74, 5.79 [23:05:44] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.84, 8.39, 5.41 [23:05:46] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.38.69.175/cpweb, 167.114.2.161/cpweb [23:05:52] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 13.92, 8.41, 5.32 [23:05:53] PROBLEM - mw10 APT on mw10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:06:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.93, 18.72, 15.83 [23:07:44] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.60, 7.37, 5.42 [23:07:46] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:07:50] RECOVERY - mw10 APT on mw10 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [23:08:10] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.45, 7.14, 4.92 [23:08:49] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.12, 17.55, 15.77 [23:09:30] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.12, 8.00, 6.05 [23:09:44] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.78, 5.97, 5.13 [23:09:50] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.97, 7.10, 5.57 [23:10:10] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.85, 5.86, 4.71 [23:11:30] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.73, 6.75, 5.82 [23:11:49] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.15, 6.09, 5.38 [23:14:00] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 12.29, 20.79, 21.87 [23:14:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jimyw [23:14:27] [02miraheze/puppet] 07paladox 031ae7395 - mediawiki: Increase fpm min restart threshold to 14 [23:14:28] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [23:14:30] [02puppet] 07paladox opened pull request 03#2087: mediawiki: Increase fpm min restart threshold to 14 - 13https://git.io/Jimyr [23:14:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jimyo [23:14:45] [02miraheze/puppet] 07paladox 032ddcfc8 - Update mw10.yaml [23:14:46] [02puppet] 07paladox synchronize pull request 03#2087: mediawiki: Increase fpm min restart threshold to 14 - 13https://git.io/Jimyr [23:14:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jimy6 [23:14:54] [02miraheze/puppet] 07paladox 032008529 - Update mw11.yaml [23:14:56] [02puppet] 07paladox synchronize pull request 03#2087: mediawiki: Increase fpm min restart threshold to 14 - 13https://git.io/Jimyr [23:15:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jimyi [23:15:05] [02miraheze/puppet] 07paladox 0307720fa - Update mw12.yaml [23:15:06] [02puppet] 07paladox synchronize pull request 03#2087: mediawiki: Increase fpm min restart threshold to 14 - 13https://git.io/Jimyr [23:15:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JimyX [23:15:14] [02miraheze/puppet] 07paladox 03bec8a50 - Update mw13.yaml [23:15:16] [02puppet] 07paladox synchronize pull request 03#2087: mediawiki: Increase fpm min restart threshold to 14 - 13https://git.io/Jimyr [23:15:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JimyM [23:15:23] [02miraheze/puppet] 07paladox 03fc350e3 - Update mw9.yaml [23:15:25] [02puppet] 07paladox synchronize pull request 03#2087: mediawiki: Increase fpm min restart threshold to 14 - 13https://git.io/Jimyr [23:15:37] [02puppet] 07paladox closed pull request 03#2087: mediawiki: Increase fpm min restart threshold to 14 - 13https://git.io/Jimyr [23:15:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±6] 13https://git.io/JimyD [23:15:40] [02miraheze/puppet] 07paladox 030839abf - mediawiki: Increase fpm min restart threshold to 14 (#2087) [23:15:41] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [23:15:43] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [23:18:00] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 33.09, 23.42, 22.43 [23:19:44] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.99, 7.73, 6.09 [23:19:46] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.69, 8.42, 5.82 [23:20:00] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 15.97, 20.00, 21.29 [23:20:10] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.02, 9.07, 6.39 [23:20:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.34, 20.81, 17.64 [23:21:44] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.24, 6.32, 5.76 [23:21:46] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.48, 7.49, 5.82 [23:22:10] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.08, 7.07, 5.98 [23:22:49] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.00, 17.49, 16.79 [23:23:46] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.71, 6.70, 5.75 [23:24:00] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 14.32, 17.18, 19.88 [23:24:10] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.84, 6.14, 5.79 [23:29:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JimSW [23:29:09] [02miraheze/puppet] 07paladox 03c573e25 - mediawiki: Fix ferm rules not being sorted properly [23:29:10] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [23:29:12] [02puppet] 07paladox opened pull request 03#2088: mediawiki: Fix ferm rules not being sorted properly - 13https://git.io/JimSl [23:32:40] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:38:36] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Service[ferm] [23:40:05] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 45.05, 25.07, 20.79 [23:40:35] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:41:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JimSQ [23:41:59] [02miraheze/puppet] 07paladox 03cc01558 - Update mediawiki.pp [23:42:00] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 16.52, 21.37, 19.98 [23:42:00] [02puppet] 07paladox synchronize pull request 03#2088: mediawiki: Fix ferm rules not being sorted properly - 13https://git.io/JimSl [23:42:22] [02puppet] 07paladox closed pull request 03#2088: mediawiki: Fix ferm rules not being sorted properly - 13https://git.io/JimSl [23:42:24] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [23:42:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JimS5 [23:42:27] [02miraheze/puppet] 07paladox 03bc736ef - mediawiki: Fix ferm rules not being sorted properly (#2088) [23:42:28] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [23:46:00] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 10.01, 17.03, 18.68 [23:46:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jim9k [23:46:03] [02miraheze/puppet] 07paladox 03bdd4230 - irc: Fix sorting ferm rules [23:46:04] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [23:46:06] [02puppet] 07paladox opened pull request 03#2089: irc: Fix sorting ferm rules - 13https://git.io/Jim9I [23:46:27] [02puppet] 07paladox closed pull request 03#2089: irc: Fix sorting ferm rules - 13https://git.io/Jim9I [23:46:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jim9t [23:46:30]