[00:00:39] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.59, 20.40, 20.80 [00:02:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.63, 20.73, 20.86 [00:02:54] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:03:12] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.96, 3.71, 3.44 [00:05:11] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.41, 3.18, 3.27 [00:06:39] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.10, 19.73, 20.40 [00:10:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.68, 21.87, 21.22 [00:14:39] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 41.51, 29.57, 24.28 [00:16:16] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.89, 5.17, 4.64 [00:21:01] [02puppet] 07Universal-Omega synchronize pull request 03#2127: Handle CORS preflight requests - 13https://git.io/J10Ko [00:21:59] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.83, 6.68, 5.41 [00:22:25] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.89, 5.92, 5.39 [00:22:30] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.41, 6.82, 5.77 [00:24:25] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.67, 5.38, 5.25 [00:24:27] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.86, 6.22, 5.67 [00:28:25] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.19, 6.57, 5.76 [00:29:03] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.38.69.175/cpweb, 54.38.211.199/cpweb, 2001:41d0:800:170b::5/cpweb, 51.222.25.132/cpweb, 167.114.2.161/cpweb, 2607:5300:205:200::1c30/cpweb, 2607:5300:201:3100::1d3/cpweb [00:29:06] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:29:07] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 2607:5300:201:3100::1d3/cpweb [00:30:25] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 11.51, 8.62, 6.63 [00:31:04] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:31:07] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:32:25] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.85, 7.11, 6.32 [00:32:54] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [00:34:06] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:34:44] [02puppet] 07Universal-Omega edited pull request 03#2127: Handle CORS preflight requests - 13https://git.io/J10Ko [00:36:25] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.31, 6.42, 6.23 [00:39:46] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.38, 3.29 [00:41:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.91, 3.19, 3.23 [00:45:41] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.03, 3.91, 3.54 [00:47:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.69, 3.48, 3.43 [00:49:39] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.93, 4.13, 3.67 [00:53:37] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.98, 3.97, 3.72 [01:01:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.97, 3.14, 3.40 [01:01:48] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.42, 4.33, 5.84 [01:07:48] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.78, 3.50, 4.98 [01:08:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.29, 4.08, 3.68 [01:08:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 14.35, 17.63, 23.31 [01:12:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.94, 3.93, 3.71 [01:16:23] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.35, 3.76, 3.68 [01:18:22] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 1.87, 3.06, 3.43 [01:20:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.88, 3.90, 3.67 [01:22:40] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 35.69, 25.76, 23.17 [01:24:07] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.35, 5.23, 4.71 [01:24:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.04, 22.68, 22.33 [01:26:01] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.65, 4.76, 4.60 [01:27:57] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3010 (END - exit=256) [01:28:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:28:16] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.21, 3.90, 3.84 [01:32:39] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.42, 17.17, 19.85 [01:32:58] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3010 (START) [01:33:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:36:11] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.68, 2.85, 3.33 [02:10:53] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.50, 3.72, 3.25 [02:12:53] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.21, 3.61, 3.28 [02:18:53] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.93, 3.90, 3.51 [02:20:53] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.78, 3.61, 3.46 [02:22:53] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.38, 3.66, 3.48 [02:24:53] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.93, 3.66, 3.50 [02:30:53] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.38, 3.14, 3.34 [02:47:45] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.45, 7.41, 5.58 [02:49:45] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.85, 6.12, 5.33 [03:56:19] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.20, 3.44, 2.89 [03:58:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.67, 3.40, 2.94 [03:58:19] PROBLEM - wiki.ct777.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.ct777.cf' expires in 7 day(s) (Sat 27 Nov 2021 03:53:12 GMT +0000). [04:00:16] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.40, 3.65, 3.08 [04:02:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.65, 3.49, 3.08 [04:04:14] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.09, 3.06, 2.98 [04:19:25] PROBLEM - olwest.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'olwest.org' expires in 7 day(s) (Sat 27 Nov 2021 04:18:59 GMT +0000). [04:20:10] PROBLEM - wmworld.sktz.live - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wmworld.sktz.live' expires in 7 day(s) (Sat 27 Nov 2021 04:14:05 GMT +0000). [04:26:52] PROBLEM - you.r-fit.cc - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'you.r-fit.cc' expires in 7 day(s) (Sat 27 Nov 2021 04:20:57 GMT +0000). [04:31:07] PROBLEM - www.rangpurpedia.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'rangpurpedia.xyz' expires in 7 day(s) (Sat 27 Nov 2021 04:29:15 GMT +0000). [04:32:39] PROBLEM - rangpurpedia.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'rangpurpedia.xyz' expires in 7 day(s) (Sat 27 Nov 2021 04:29:15 GMT +0000). [06:06:02] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.86, 5.93, 3.82 [06:07:14] PROBLEM - db13 Current Load on db13 is CRITICAL: CRITICAL - load average: 25.86, 11.48, 5.20 [06:08:01] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 11.13, 7.76, 4.74 [06:08:14] PROBLEM - db12 Current Load on db12 is WARNING: WARNING - load average: 6.86, 5.72, 3.87 [06:09:41] alerting : [FIRING:1] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:11:14] RECOVERY - db13 Current Load on db13 is OK: OK - load average: 1.26, 5.70, 4.26 [06:12:02] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.45, 7.79, 5.47 [06:12:14] RECOVERY - db12 Current Load on db12 is OK: OK - load average: 5.96, 6.08, 4.46 [06:14:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 20.50, 10.62, 6.68 [06:14:41] ok : [RESOLVED] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:16:12] 503 Backend fetch failed Our servers are having issues at the moment. Try this page again [06:16:31] Error on wiki [06:16:41] alerting : [FIRING:1] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:18:02] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 2.40, 6.27, 5.85 [06:18:24] That's normal, it should go away pretty quickly [06:21:41] ok : [RESOLVED] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:28:19] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3010 (END - exit=256) [07:28:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:30:36] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3010 (START) [07:30:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:58:13] PROBLEM - ambient.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ambient.wiki' expires in 15 day(s) (Sun 05 Dec 2021 08:49:56 GMT +0000). [11:25:36] PROBLEM - puppet3 APT on puppet3 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (2 critical updates). [13:25:46] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.70, 3.68, 3.07 [13:33:41] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.12, 3.35, 3.15 [14:20:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.88, 3.49, 3.03 [14:22:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.72, 3.57, 3.11 [14:24:08] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.50, 3.82, 3.25 [14:26:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.06, 3.58, 3.24 [14:34:02] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.51, 3.15, 3.20 [14:39:57] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.06, 3.46, 3.33 [14:41:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.19, 3.00, 3.17 [15:14:38] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.78, 3.72, 3.29 [15:16:37] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.79, 3.62, 3.31 [15:18:36] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.88, 3.36, 3.25 [15:26:30] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.43, 3.51, 3.36 [15:28:29] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.15, 3.06, 3.22 [15:56:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.04, 3.61, 3.31 [15:58:08] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.53, 3.27, 3.22 [15:59:54] on gratisdata, when adding sitelinks to items, I can't find the gratidatawiki sitelink, I have no idea why? [16:01:00] When was it created [16:01:35] If it's new, my guess is that it's not been picked up yet by the scripts [16:03:08] ah, but gratispaideia is shown [16:03:32] okay, I guess so too, i've been observing it for some days now though [16:03:44] some new wikis doesn't show up, yeah. [16:05:13] Wikis won't show up until the scripts run [16:05:17] It's every 2 weeks [16:06:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.52, 3.50, 3.34 [16:08:01] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.23, 3.81, 3.47 [16:09:48] ah [16:10:00] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.16, 3.70, 3.48 [16:10:12] * ugochimobi wonders why gratispaideiawiki showed up even before 2 weeks [16:13:58] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.59, 3.46 [16:15:57] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.84, 3.30, 3.37 [16:17:30] ugochimobi: "every 2 weeks", not "exactly 2 weeks after wiki creation" [16:17:38] ugochimobi: well if your wiki is created 5 minutes before the script runs, it'll be shown in 5 minutes [16:18:06] ohhhh, majavah I get exactly now [16:18:28] RhinosF1 welp that's true :) yeah [17:20:40] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3010 (END - exit=256) [17:20:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:20:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.69, 18.96, 17.08 [17:22:48] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.69, 18.95, 17.31 [17:40:39] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.77, 22.11, 19.59 [17:42:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.81, 21.78, 19.83 [17:43:48] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.05, 5.36, 4.66 [17:45:48] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.31, 4.88, 4.56 [17:48:39] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.96, 19.21, 19.40 [17:52:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.52, 22.57, 20.90 [17:56:39] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.02, 17.28, 19.13 [18:07:43] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.90, 3.34, 3.06 [18:09:41] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.86, 3.13, 3.02 [18:22:09] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/J16lY [18:22:10] [02miraheze/IncidentReporting] 07Universal-Omega 03c4ae1ee - Fix fatal in less compiler on MediaWiki 1.37.0+ [18:22:12] [02IncidentReporting] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/fh5YJ [18:22:17] [02IncidentReporting] 07Universal-Omega opened pull request 03#26: Fix fatal in less compiler on MediaWiki 1.37.0+ - 13https://git.io/J16l3 [18:24:20] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J16l1 [18:24:21] [02miraheze/CreateWiki] 07Universal-Omega 03ae531c2 - Remove unused `mediawiki.mixins` import [18:24:29] miraheze/IncidentReporting - Universal-Omega the build passed. [18:25:23] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J168T [18:25:25] [02miraheze/CreateWiki] 07Universal-Omega 0329b0ba7 - Revert [18:30:02] miraheze/CreateWiki - Universal-Omega the build has errored. [18:32:28] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/J164K [18:32:29] [02miraheze/IncidentReporting] 07Universal-Omega 036f62bed - Update ext.incidentreporting.oouiform.ooui.less [18:32:31] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#26: Fix fatal in less compiler on MediaWiki 1.37.0+ - 13https://git.io/J16l3 [18:34:39] [02miraheze/MatomoAnalytics] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/J16BL [18:34:40] miraheze/CreateWiki - Universal-Omega the build passed. [18:34:40] [02miraheze/MatomoAnalytics] 07Universal-Omega 03332a233 - Fix fatal in less compiler on MediaWiki 1.37.0+ [18:34:42] [02MatomoAnalytics] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/fN4LT [18:34:43] miraheze/IncidentReporting - Universal-Omega the build passed. [18:34:43] [02MatomoAnalytics] 07Universal-Omega opened pull request 03#57: Fix fatal in less compiler on MediaWiki 1.37.0+ - 13https://git.io/J16Bq [18:35:12] [02IncidentReporting] 07Universal-Omega closed pull request 03#26: Fix fatal in less compiler on MediaWiki 1.37.0+ - 13https://git.io/J16l3 [18:35:13] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J16B8 [18:35:15] [02miraheze/IncidentReporting] 07Universal-Omega 03344c2a5 - Fix fatal in less compiler on MediaWiki 1.37.0+ (#26) [18:35:16] [02IncidentReporting] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/fh5YJ [18:35:18] [02miraheze/IncidentReporting] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [18:35:47] .op [18:35:47] Attempting to OP... [18:36:01] .deop [18:36:01] Attempting to OP... [18:37:05] miraheze/MatomoAnalytics - Universal-Omega the build passed. [18:37:22] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.10, 3.42, 3.08 [18:37:29] miraheze/IncidentReporting - Universal-Omega the build passed. [18:37:56] [02miraheze/MatomoAnalytics] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J16BP [18:37:57] [02miraheze/MatomoAnalytics] 07Universal-Omega 039528cc3 - Fix fatal in less compiler on MediaWiki 1.37.0+ (#57) [18:37:59] [02MatomoAnalytics] 07Universal-Omega closed pull request 03#57: Fix fatal in less compiler on MediaWiki 1.37.0+ - 13https://git.io/J16Bq [18:38:00] [02miraheze/MatomoAnalytics] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [18:38:02] [02MatomoAnalytics] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/fN4LT [18:39:21] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.98, 3.24, 3.06 [18:40:20] miraheze/MatomoAnalytics - Universal-Omega the build passed. [18:43:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.98, 4.05, 3.47 [18:47:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.20, 3.97, 3.59 [18:51:12] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.59, 4.02, 3.67 [18:53:12] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.66, 3.58 [18:55:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.02, 3.14, 3.40 [19:08:01] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.28, 3.49, 3.41 [19:10:00] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.75, 3.31, 3.36 [19:12:03] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3010 (START) [19:12:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:22:33] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 14.95, 11.47, 7.36 [19:22:39] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.36, 21.52, 18.46 [19:24:39] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.13, 19.42, 18.08 [19:26:28] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.07, 7.96, 6.83 [19:30:22] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.86, 6.24, 6.37 [19:31:09] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.70, 6.91, 6.15 [19:33:04] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.53, 6.63, 6.14 [19:33:46] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.40, 3.55, 3.20 [19:35:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.53, 3.23 [19:39:43] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.25, 3.89, 3.46 [19:44:41] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/findBadBlobs.php --wiki=polandballwiki --revisions=1290461 --mark=T8309 [19:44:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:45:39] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.02, 3.82, 3.63 [19:47:38] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.01, 3.74, 3.62 [19:49:37] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.61, 3.40, 3.51 [19:53:35] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.69, 3.14, 3.39 [19:55:52] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/findBadBlobs.php --wiki=polandballwiki --revisions=1165323 [19:55:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:55:57] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/findBadBlobs.php --wiki=polandballwiki --revisions=1165323 [19:56:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:23:26] [02puppet] 07Universal-Omega edited pull request 03#2127: Handle CORS preflight requests + fix `Access-Control-Allow-Origin` for api.php - 13https://git.io/J10Ko [20:30:45] [02puppet] 07Universal-Omega edited pull request 03#2127: Handle CORS preflight requests + fix `Access-Control-Allow-Origin` for api.php - 13https://git.io/J10Ko [20:31:00] [02puppet] 07Universal-Omega edited pull request 03#2127: Handle CORS preflight requests + fix `Access-Control-Allow-Origin` for api.php - 13https://git.io/J10Ko [20:49:02] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/findBadBlobs.php --wiki=polandballwiki --revisions=1290410,1290411,1290412,1290413,1290414,1290415,1290416,1290417,1290418,1290420,1290421,1290422,1290424,1290425,1290426,1290427,1290428,1290429,1290430,1290460,1290462,1290463,1290464,1290465,1290466,1290467,1290468,1290469,129047 [20:49:03] 0,1290471,1290472,1290473,1290474,1290475,1290476,1290477,1290478,1290479,1290480,1290481,1290482,1290483,1290484,1290485,1290486,1290487,1290488,1290489,1290490,1290491,1290492,1290493,1290494,1290495,1290496,1290497,1290498,1290499,1290500,1290501,1290502,1290503,1290504,1290505,1290506,1290507,1290510,1290511,1290512,1290513,1290514,1290515,1290516,1290517 [20:49:04] ,1290518,1290519,1290520,1290521,1290522,1290524,1290525,1290526,1290534,1290535,1290536,1290537 --mark=T8309 (END - exit=0) [20:49:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:14:42] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.22, 3.64, 3.17 [21:16:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.70, 3.26 [21:22:37] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.04, 3.81, 3.43 [21:26:35] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.83, 3.75, 3.51 [21:30:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.76, 3.05, 3.27 [21:46:24] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.73, 3.24, 3.06 [21:48:23] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.88, 2.73, 2.90 [22:19:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.07, 22.41, 18.97 [22:21:45] PROBLEM - mw9 APT on mw9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:22:19] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 14.68, 10.78, 7.06 [22:23:44] RECOVERY - mw9 APT on mw9 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:28:11] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.90, 7.79, 7.00 [22:28:19] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3010 (END - exit=2) [22:28:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:28:48] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3010 (START) [22:28:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:30:25] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.54, 7.03, 5.88 [22:31:10] CosmicAlpha: that you killing it? [22:31:23] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.06, 5.84, 4.62 [22:31:51] RhinosF1: yes, it is almost done now [22:31:56] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3010 (END - exit=0) [22:32:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:32:06] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.09, 6.36, 6.60 [22:32:14] CosmicAlpha: that's quicker than expected [22:32:16] I made a local change to the script and more deleted in 1 minute than the last 4 days. [22:32:23] Right [22:32:25] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.32, 6.75, 5.90 [22:32:28] What change [22:33:00] RhinosF1: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/740165 [22:33:17] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.25, 5.63, 4.69 [22:33:32] CosmicAlpha: ah, what we said earlier! [22:33:40] Knew that didn't make sense [22:34:01] yep! [22:34:22] more deleted in 1 second than 10 hours before. [22:34:25] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.67, 5.62, 5.59 [22:35:11] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.64, 4.48, 4.38 [22:35:31] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3008 (START) [22:35:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:38:59] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.08, 5.77, 4.96 [22:39:09] CosmicAlpha: see mediawiki irc chan [22:41:51] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.19, 3.48, 2.85 [22:43:33] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3008 (END - exit=0) [22:43:46] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 4.40, 3.69, 2.99 [22:44:00] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3006 (START) [22:44:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:44:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:45:41] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.06, 3.09, 2.86 [22:48:00] seen RhinosF1 [22:48:30] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.45, 5.80, 5.62 [22:50:24] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.34, 6.63, 5.92 [22:51:42] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.51, 3.22 [22:53:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.49, 3.25 [22:54:12] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.78, 5.66, 5.73 [22:57:38] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.17, 3.15, 3.20 [22:59:54] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.30, 5.88, 5.66 [23:00:04] !log [@test3] starting deploy of {'l10nupdate': True} to skip [23:00:06] !log [@mw11] starting deploy of {'l10nupdate': True} to all [23:00:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:25] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.54, 7.15, 6.08 [23:00:45] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3006 (END - exit=35072) [23:01:01] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 4.36, 3.47, 2.95 [23:01:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:02:26] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3004 (START) [23:02:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:03:01] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 3.02, 3.21, 2.91 [23:03:34] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.83, 3.49 [23:03:48] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.80, 5.44, 5.60 [23:03:56] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 14.16, 8.16, 5.61 [23:04:07] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 27.78, 20.14, 15.59 [23:04:26] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3004 (END - exit=0) [23:04:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:04:44] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.92, 8.34, 5.99 [23:05:00] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3002 (START) [23:05:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:05:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.57, 3.80, 3.53 [23:05:56] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.48, 7.69, 5.79 [23:06:07] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.60, 18.12, 15.39 [23:06:24] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.20, 7.34, 6.22 [23:06:28] PROBLEM - mw12 APT on mw12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:06:44] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.86, 7.67, 6.00 [23:07:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.45, 4.00, 3.63 [23:08:22] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.01, 7.07, 6.27 [23:08:28] RECOVERY - mw12 APT on mw12 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [23:08:44] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 12.07, 9.08, 6.70 [23:08:54] PROBLEM - mw11 APT on mw11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:09:57] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.82, 8.12, 6.31 [23:10:19] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.06, 6.28, 6.06 [23:10:25] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 3.34, 6.85, 6.84 [23:10:53] RECOVERY - mw11 APT on mw11 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [23:11:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.92, 3.69, 3.61 [23:11:37] !log [@mw11] finished deploy of {'l10nupdate': True} to all - FAIL: [35072, 0, 0, 0, 0, 0, 0, 0, 0] in 691s [23:11:48] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 10.55, 6.53, 5.78 [23:12:08] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.66, 20.88, 17.53 [23:12:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:12:25] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.08, 6.08, 6.56 [23:12:44] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.39, 7.97, 6.90 [23:13:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:14:07] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.86, 19.10, 17.27 [23:14:11] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.50, 7.27, 6.57 [23:14:44] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.51, 6.78, 6.59 [23:15:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.13, 3.01, 3.37 [23:15:48] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.27, 5.72, 5.70 [23:15:56] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 3.90, 7.10, 6.73 [23:16:08] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.71, 6.38, 6.33 [23:17:48] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 10.12, 7.08, 6.17 [23:17:56] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.06, 5.86, 6.31 [23:18:06] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:19:53] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.38.69.175/cpweb, 54.38.211.199/cpweb, 2001:41d0:801:2000::58af/cpweb, 2607:5300:205:200::1c30/cpweb [23:20:25] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.06, 7.88, 7.10 [23:20:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:21:01] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 4.21, 3.46, 2.95 [23:21:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:21:58] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.86, 7.59, 6.85 [23:22:28] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.30, 6.93, 6.86 [23:23:01] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 1.86, 2.80, 2.77 [23:23:55] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.20, 6.79, 6.65 [23:24:25] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.83, 6.35, 6.66 [23:25:36] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:29:18] !log [@test3] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 1753s [23:29:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:19] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3002 (END - exit=35072) [23:30:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:25] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.34, 7.74, 7.08 [23:30:28] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3002 (START) [23:30:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:32:25] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.68, 6.84, 6.82 [23:33:48] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.24, 4.77, 6.00 [23:33:57] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3002 (END - exit=0) [23:34:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:34:20] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3006 (START) [23:34:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:34:25] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.58, 6.20, 6.60 [23:38:57] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3006 (END - exit=0) [23:39:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:40:25] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.26, 7.98, 7.24 [23:41:48] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.49, 3.98, 5.06 [23:42:25] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.62, 6.72, 6.87 [23:44:25] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.86, 5.86, 6.54 [23:48:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.84, 20.29, 23.78 [23:54:03] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.80, 3.63, 3.06 [23:58:01] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.18, 2.94, 2.91