[00:01:30] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/refreshLinks.php --wiki=gratisdatawiki (END - exit=0) [00:01:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:01:45] PROBLEM - en.pornwiki.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.pornwiki.org reverse DNS resolves to a51062ecadbb5a26e.awsglobalaccelerator.com [00:01:56] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=gratisdatawiki --update --active (END - exit=0) [00:02:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:02:13] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.06, 20.84, 20.24 [00:02:22] PROBLEM - files.pornwiki.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['dns102.registrar-servers.com.', 'dns101.registrar-servers.com.', 'dns2.registrar-servers.com.', 'dns1.registrar-servers.com.'], 'CNAME': None} [00:02:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.59, 3.26, 3.01 [00:04:52] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.83, 4.08, 3.36 [00:04:52] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=callipediawiki /home/universalomega/Wikipedia1.xml --username-prefix=w --no-updates --report 1 (START) [00:05:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:06:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.63, 3.77, 3.33 [00:07:19] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 4.77, 6.36, 5.42 [00:08:52] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.32, 3.93, 3.43 [00:09:15] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.42, 5.70, 5.29 [00:09:16] PROBLEM - files.pornwiki.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - files.pornwiki.org reverse DNS resolves to a51062ecadbb5a26e.awsglobalaccelerator.com [00:13:06] PROBLEM - pornwiki.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - pornwiki.org reverse DNS resolves to a51062ecadbb5a26e.awsglobalaccelerator.com [00:13:07] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 10.85, 7.79, 6.16 [00:13:53] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=callipediawiki /home/universalomega/Wikipedia1.xml --username-prefix=w --no-updates --report 1 (END - exit=2) [00:14:15] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=callipediawiki /home/universalomega/Wikipedia1.xml --username-prefix=w --no-updates --report 1 (START) [00:14:30] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=callipediawiki /home/universalomega/Wikipedia2.xml --username-prefix=w --no-updates --report 1 (START) [00:14:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.62, 3.94, 3.69 [00:14:59] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=callipediawiki /home/universalomega/Wikipedia3.xml --username-prefix=w --no-updates --report 1 (START) [00:15:27] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=callipediawiki /home/universalomega/Wikipedia4.xml --username-prefix=w --no-updates --report 1 (START) [00:15:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:15:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:15:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:15:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:15:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:16:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.38.69.175/cpweb, 54.38.211.199/cpweb, 2001:41d0:801:2000::58af/cpweb, 2001:41d0:800:170b::5/cpweb, 167.114.2.161/cpweb, 2607:5300:205:200::1c30/cpweb, 2607:5300:201:3100::1d3/cpweb [00:16:52] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.35, 4.06, 3.78 [00:18:47] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:22:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.66, 3.50, 3.68 [00:27:05] RECOVERY - pornwiki.org - reverse DNS on sslhost is OK: SSL OK - pornwiki.org reverse DNS resolves to cp14.miraheze.org - NS RECORDS OK [00:28:03] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --all --ns=3002 (END - exit=2) [00:28:08] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=callipediawiki /home/universalomega/Wikipedia1.xml --username-prefix=w --no-updates --report 1 (END - exit=0) [00:28:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:28:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:29:57] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=callipediawiki /home/universalomega/Wikipedia4.xml --username-prefix=w --no-updates --report 1 (END - exit=0) [00:30:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:52] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.75, 2.85, 3.31 [00:32:54] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=callipediawiki /home/universalomega/Wikipedia2.xml --username-prefix=w --no-updates --report 1 (END - exit=0) [00:33:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:34:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:36:05] PROBLEM - pornwiki.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - pornwiki.org reverse DNS resolves to a51062ecadbb5a26e.awsglobalaccelerator.com [00:37:53] PROBLEM - db12 Current Load on db12 is WARNING: WARNING - load average: 7.04, 6.40, 4.25 [00:39:36] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:43:38] RECOVERY - db12 Current Load on db12 is OK: OK - load average: 6.67, 6.79, 5.12 [00:44:21] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3004 (END - exit=256) [00:44:22] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=callipediawiki /home/universalomega/Wikipedia3.xml --username-prefix=w --no-updates --report 1 (END - exit=256) [00:44:23] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3008 (END - exit=256) [00:44:24] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3002 (END - exit=256) [00:44:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:44:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:44:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:44:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:44:44] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3006 (END - exit=256) [00:44:46] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3010 (END - exit=256) [00:44:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:44:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:59:49] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=callipediawiki /home/universalomega/Wikipedia3.xml --username-prefix=w --no-updates --report 1 (START) [00:59:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:01:29] guess I tried doing to much at once... [01:04:44] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 0.81, 3.39, 5.92 [01:08:13] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 13.21, 14.11, 22.55 [01:08:44] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 1.72, 2.60, 5.04 [01:15:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.75, 3.14 [01:16:13] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.55, 16.00, 20.10 [01:17:52] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.09, 3.13, 3.00 [01:22:58] [02puppet] 07Universal-Omega synchronize pull request 03#2121: mwscript: rewrite to use argparse - 13https://git.io/JXQhx [01:23:25] [02puppet] 07Universal-Omega edited pull request 03#2121: mwscript: rewrite to use argparse - 13https://git.io/JXQhx [01:26:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.44, 6.42, 4.73 [01:28:04] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.05, 6.05, 4.80 [01:29:15] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.20, 4.98, 4.71 [01:32:03] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=callipediawiki /home/universalomega/Wikipedia3.xml --username-prefix=w --no-updates --report 1 (END - exit=0) [01:32:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:33:07] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.46, 4.19, 4.51 [01:50:03] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.17, 3.89, 3.32 [01:52:53] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.29, 2.52, 1.47 [01:53:58] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.76, 3.41 [01:54:53] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 0.96, 1.90, 1.36 [01:56:53] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.54, 1.41, 1.25 [01:59:50] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.73, 3.20, 3.28 [03:33:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.54, 3.10, 2.86 [03:35:39] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.37, 2.85, 2.79 [04:15:38] PROBLEM - wiki.thefactoryhka.com.pa - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.thefactoryhka.com.pa' expires in 15 day(s) (Sat 04 Dec 2021 04:08:18 GMT +0000). [06:06:12] PROBLEM - db13 Current Load on db13 is WARNING: WARNING - load average: 8.00, 5.58, 3.11 [06:06:48] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.50, 6.90, 4.52 [06:07:01] PROBLEM - db12 Current Load on db12 is WARNING: WARNING - load average: 7.24, 6.19, 4.65 [06:08:12] RECOVERY - db13 Current Load on db13 is OK: OK - load average: 6.68, 5.87, 3.51 [06:13:01] RECOVERY - db12 Current Load on db12 is OK: OK - load average: 5.71, 6.34, 5.26 [06:22:50] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3002 (END - exit=256) [06:22:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:23:03] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3010 (END - exit=256) [06:23:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:26:48] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.56, 7.85, 7.57 [06:36:48] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.30, 6.29, 6.80 [06:51:36] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.98, 6.77, 6.61 [06:53:30] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.92, 6.26, 6.42 [07:16:48] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.56, 6.82, 6.53 [07:20:48] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.73, 6.68, 6.56 [07:22:37] [02mw-config] 07AgentIsai opened pull request 03#4224: Add 'On hold pending response' canned response + standardization of a few terms - 13https://git.io/J1zfF [07:23:35] miraheze/mw-config - AgentIsai the build passed. [07:23:40] [02mw-config] 07AgentIsai synchronize pull request 03#4224: Add 'On hold pending response' canned response + standardization of a few terms - 13https://git.io/J1zfF [07:24:05] [02mw-config] 07AgentIsai edited pull request 03#4224: Add 'On hold pending response' canned response + standardization of a few terms - 13https://git.io/J1zfF [07:24:41] miraheze/mw-config - AgentIsai the build passed. [07:27:49] Agent: lmk if good to merge [07:28:03] PROBLEM - wiki.wikimedia.cat - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.wikimedia.cat' expires in 15 day(s) (Sat 04 Dec 2021 07:23:19 GMT +0000). [07:28:37] RhinosF1: Please do, thanks [07:28:43] [02mw-config] 07RhinosF1 closed pull request 03#4224: Add 'On hold pending response' canned response + standardization of a few terms - 13https://git.io/J1zfF [07:28:45] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J1zU8 [07:28:46] [02miraheze/mw-config] 07AgentIsai 031e812dd - Add 'On hold pending response' canned response + standardization of a few terms (#4224) [07:28:56] Thanks! [07:29:00] Agent: will let puppet deploy [07:29:14] Sounds good [07:29:49] miraheze/mw-config - RhinosF1 the build passed. [07:41:03] !log [@mw11] starting deploy of {'config': True} to all [07:41:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:41:16] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 12s [07:41:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:44:48] Agent: ^ is production done [07:45:46] RhinosF1: ty [07:45:47] I just test declined a dummy request and it seems to work [07:53:56] !log [@test3] starting deploy of {'config': True} to skip [07:53:57] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [07:54:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:54:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:06:48] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.77, 7.19, 6.56 [08:08:48] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.75, 7.25, 6.65 [08:16:48] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.27, 6.43, 6.60 [08:45:48] [02ssl] 07Reception123 closed pull request 03#439: Remove nocyclo.tk - 13https://git.io/J10u3 [08:45:49] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-4/±1] 13https://git.io/J1zBa [08:45:51] [02miraheze/ssl] 07Universal-Omega 03cbde371 - Remove nocyclo.tk (#439) [08:45:53] [02dns] 07Reception123 closed pull request 03#229: Remove nocyclo.tk - 13https://git.io/J10uG [08:45:55] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/J1zBw [08:45:56] [02miraheze/dns] 07Universal-Omega 0347b4562 - Remove nocyclo.tk (#229) [08:52:11] PROBLEM - mw13 Puppet on mw13 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 2 minutes ago with 4 failures. Failed resources (up to 3 shown): File[meta.nocyclo.tk],File[en.nocyclo.tk],File[es.nocyclo.tk],File[incubator.nocyclo.tk] [08:52:30] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[incubator.nocyclo.tk] [08:53:40] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [08:56:38] PROBLEM - cp12 Puppet on cp12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [08:57:33] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [08:59:54] PROBLEM - mw12 Puppet on mw12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [09:01:11] PROBLEM - cp13 Puppet on cp13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [09:04:08] [02puppet] 07Reception123 closed pull request 03#2121: mwscript: rewrite to use argparse - 13https://git.io/JXQhx [09:04:10] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J1zzK [09:04:11] [02miraheze/puppet] 07Universal-Omega 03f9feeac - mwscript: rewrite to use argparse (#2121) [09:10:40] PROBLEM - cp14 Puppet on cp14 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [09:11:42] PROBLEM - cp15 Puppet on cp15 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [09:13:46] PROBLEM - mwtask1 Puppet on mwtask1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [09:14:02] Reception123: please check certificates [09:14:28] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [09:14:33] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [09:15:10] RhinosF1: I just raised a task to UBN about expired ones also. I let reception already know via DM about that though. [09:15:34] CosmicAlpha: k [09:19:14] I meant the ones crashing nginx [09:19:17] But yes them too [09:22:03] RhinosF1: yeah I knew what you meant, was just mentioning the expired ones. [09:22:26] According to icinga there are 12 expired ones. [09:22:38] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:23:35] CosmicAlpha: :( [09:24:02] RECOVERY - cp12 Puppet on cp12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:25:18] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [09:28:38] RECOVERY - cp13 Puppet on cp13 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [09:28:43] RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:38:39] RECOVERY - cp14 Puppet on cp14 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:39:41] RECOVERY - cp15 Puppet on cp15 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:41:38] RECOVERY - mwtask1 Puppet on mwtask1 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [09:41:42] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [09:42:24] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [09:49:39] paladox: around? [09:50:12] RECOVERY - mw13 Puppet on mw13 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [09:50:15] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [09:50:26] Reception123: if you have broken mess from puppet failures when it mismatched then try nginx reload [09:50:40] Just make sure to do the test thing first to make sure it's still not broken [09:50:45] certs seem to be the problem - https://phabricator.miraheze.org/T8285#167630 [09:50:46] [url] ⚓ T8285 Custom Domain - Hypothetical Hurricanes Wiki | phabricator.miraheze.org [09:51:14] Reception123: yes broken mess from nginx not reloading when the service has refused because conftest failed [09:51:39] ah [09:52:00] !log sudo service nginx reload on mw*/cp* [09:52:01] Do conftest to make sure it's good [09:52:04] Then reload [09:52:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:52:44] Reception123: don't forget task/test [09:53:27] It's serving *.Miraheze.org still [09:53:33] You must have made a type [09:53:35] Typo [09:55:15] RhinosF1: nope seems I made a larger mistake, something with a private key mismatch [09:55:18] I thought I fixed it but no [09:55:35] Reception123: fun you! [09:56:18] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J1zME [09:56:20] [02miraheze/ssl] 07Reception123 037cc7af8 - Update wiki.wilderyogi.eu.crt [09:59:22] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.30, 18.73, 15.38 [10:00:25] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 13.89, 8.37, 5.15 [10:01:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.80, 20.07, 16.27 [10:02:25] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.36, 7.21, 5.14 [10:02:52] RECOVERY - www.fossilfighterswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.fossilfighterswiki.com' will expire on Mon 14 Feb 2022 15:18:35 GMT +0000. [10:02:52] RECOVERY - worldsanskrit.net - LetsEncrypt on sslhost is OK: OK - Certificate 'worldsanskrit.net' will expire on Mon 14 Feb 2022 15:20:31 GMT +0000. [10:03:15] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.00, 19.24, 16.46 [10:03:17] RECOVERY - ref.undu.st - LetsEncrypt on sslhost is OK: OK - Certificate 'ref.undu.st' will expire on Mon 14 Feb 2022 15:19:30 GMT +0000. [10:03:52] RECOVERY - allthetropes.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Mon 14 Feb 2022 15:37:40 GMT +0000. [10:04:25] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 2.92, 5.74, 4.85 [10:04:56] RECOVERY - isv.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Mon 14 Feb 2022 15:37:40 GMT +0000. [10:05:13] RECOVERY - www.ethene.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ethene.wiki' will expire on Mon 14 Feb 2022 15:17:46 GMT +0000. [10:05:32] RECOVERY - miraheze.com - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.com' will expire on Mon 14 Feb 2022 15:21:15 GMT +0000. [10:05:38] RECOVERY - celeste.ink - LetsEncrypt on sslhost is OK: OK - Certificate 'celeste.ink' will expire on Mon 14 Feb 2022 15:33:21 GMT +0000. [10:06:44] RECOVERY - meta.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Mon 14 Feb 2022 15:37:40 GMT +0000. [10:06:55] RECOVERY - poserdazfreebies.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Mon 14 Feb 2022 15:37:40 GMT +0000. [10:07:04] RECOVERY - ethene.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ethene.wiki' will expire on Mon 14 Feb 2022 15:17:46 GMT +0000. [10:07:19] RECOVERY - wiki.aridia.space - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.aridia.space' will expire on Mon 14 Feb 2022 15:22:05 GMT +0000. [10:07:35] RECOVERY - orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Mon 14 Feb 2022 15:37:40 GMT +0000. [10:08:35] RECOVERY - wiki.wilderyogi.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.wilderyogi.eu' will expire on Wed 16 Feb 2022 08:55:46 GMT +0000. [10:10:12] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.57, 6.82, 6.10 [10:12:07] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.84, 6.43, 6.04 [10:20:42] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.63, 21.99, 17.89 [10:22:30] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.68, 5.99, 4.15 [10:22:39] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 32.47, 26.04, 19.85 [10:26:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.71, 21.76, 19.61 [10:28:13] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.32, 4.78, 4.39 [10:28:39] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.67, 18.74, 18.77 [10:45:06] is something going on? pages loads slow or loose data before saving [10:46:43] saved now, third time the charm [10:50:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:50:39] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 31.61, 22.04, 18.59 [10:51:46] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.99, 7.83, 5.25 [10:51:48] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.64, 4.53, 3.71 [10:52:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.44, 20.73, 18.54 [10:53:46] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.92, 7.72, 5.52 [10:53:48] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.09, 3.94, 3.59 [10:53:56] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.16, 7.48, 5.16 [10:54:39] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.96, 22.74, 19.45 [10:54:44] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.00, 6.30, 4.64 [10:55:36] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:55:45] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.99, 8.43, 6.07 [10:55:56] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.14, 7.11, 5.32 [10:56:25] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.35, 7.04, 5.84 [10:56:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.66, 20.77, 19.11 [10:56:44] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.36, 7.67, 5.33 [10:57:56] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.02, 6.70, 5.41 [10:58:25] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 17.98, 10.21, 7.09 [10:58:44] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.09, 6.66, 5.25 [10:59:45] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.45, 7.77, 6.37 [11:00:39] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 31.52, 24.81, 21.03 [11:02:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.11, 23.81, 21.11 [11:03:46] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.60, 6.78, 6.35 [11:04:25] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.80, 7.65, 7.09 [11:05:54] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.43, 7.25, 5.74 [11:05:56] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 12.35, 9.00, 6.63 [11:07:54] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.81, 6.29, 5.57 [11:07:56] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.00, 7.65, 6.42 [11:08:25] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 11.50, 9.36, 7.85 [11:09:46] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.70, 8.43, 7.21 [11:09:56] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.99, 6.35, 6.09 [11:10:10] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.93, 7.07, 6.08 [11:10:39] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.58, 19.58, 20.27 [11:11:30] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.65, 7.77, 6.74 [11:11:45] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.57, 7.63, 7.06 [11:13:26] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.49, 6.78, 6.49 [11:14:25] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.23, 8.00, 7.70 [11:16:10] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.59, 8.20, 6.80 [11:16:25] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.02, 8.22, 7.84 [11:17:18] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.45, 7.11, 6.73 [11:17:46] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.68, 8.96, 7.66 [11:18:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:18:42] PROBLEM - mw9 APT on mw9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:19:13] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.31, 7.67, 6.99 [11:20:10] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.70, 7.93, 7.02 [11:20:40] RECOVERY - mw9 APT on mw9 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [11:21:09] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.39, 7.41, 6.96 [11:22:25] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.57, 8.00, 7.93 [11:23:36] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:24:10] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.77, 6.19, 6.57 [11:27:45] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.96, 7.10, 7.92 [11:28:50] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.57, 7.45, 7.06 [11:30:45] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.71, 6.56, 6.78 [11:32:25] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.87, 5.28, 6.54 [11:34:37] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.50, 6.99, 6.95 [11:35:46] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.17, 5.23, 6.59 [11:36:33] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.22, 7.48, 7.11 [11:38:28] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.45, 7.28, 7.10 [11:43:41] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=liberapediawiki (START) [11:43:42] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=liberapediawiki (END - exit=256) [11:43:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:43:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:44:09] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=liberapediawiki (START) [11:44:10] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=liberapediawiki (END - exit=256) [11:44:14] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.78, 6.29, 6.73 [11:44:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:44:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:44:47] RhinosF1: ^ hmm, why didn't it ask me to confirm anything? [11:44:56] and I did "mwscript importDrmp.php liberapediawiki '/home/reception/liberapedia_pages_full.xml --username-prefix "wikia:liberapedia" --report 1 --no-updates'" which should be right [11:45:26] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=liberapediawiki /home/reception/liberapedia_pages_full.xml --username-prefix="wikia:liberapedia" --report 1 --no-updates [11:45:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:50:02] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 5.90, 6.78, 6.82 [11:50:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.19, 19.29, 18.32 [11:51:11] !log DELETED and DROPPED spektrowiki (reset) [11:51:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:51:45] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.68, 6.91, 6.26 [11:52:02] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.24, 6.51, 6.71 [11:52:37] hmm, something unusual is going on - the drop database query doesn't seem to be working [11:52:39] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.66, 18.36, 18.14 [11:52:44] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.79, 7.50, 6.23 [11:53:45] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.10, 6.45, 6.17 [11:55:30] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.09, 8.96, 6.83 [11:56:02] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.14, 7.25, 6.94 [11:56:27] don't see the query in processlist either [11:57:14] paladox: around now? [11:57:24] not really urgent if you don't have time right now, but just a bit strange [11:57:26] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.77, 7.73, 6.64 [11:57:46] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.90, 6.74, 6.36 [11:58:02] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.08, 6.52, 6.72 [11:58:44] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.61, 6.46, 6.23 [11:59:04] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 13.43, 9.22, 7.16 [11:59:45] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.60, 6.65, 6.37 [12:01:18] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.62, 6.60, 6.41 [12:02:06] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:04:44] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.59, 7.15, 6.60 [12:06:42] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.41, 7.01, 7.26 [12:06:44] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.99, 6.68, 6.49 [12:07:06] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:09:01] I’m mobile Reception123 [12:10:02] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.37, 7.14, 6.80 [12:10:32] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.78, 5.77, 6.70 [12:10:57] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J1gCS [12:10:59] [02miraheze/MirahezeMagic] 07translatewiki 0360dbd19 - Localisation updates from https://translatewiki.net. [12:10:59] [url] Main page - translatewiki.net | translatewiki.net [12:12:00] miraheze/MirahezeMagic - translatewiki the build passed. [12:15:56] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.40, 7.68, 6.90 [12:16:02] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 4.57, 6.39, 6.66 [12:16:44] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.37, 7.50, 6.85 [12:17:39] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.24, 3.64, 3.11 [12:17:56] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.43, 6.92, 6.71 [12:20:02] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.94, 7.12, 6.92 [12:21:06] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:21:33] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.05, 7.08, 6.43 [12:21:56] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 14.98, 9.90, 7.88 [12:26:06] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:27:16] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.31, 7.62, 6.88 [12:27:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 1.63, 3.43, 3.49 [12:30:44] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.53, 6.97, 7.41 [12:31:02] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.29, 21.00, 19.54 [12:31:30] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.81, 3.19, 3.38 [12:31:56] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.40, 7.58, 7.97 [12:32:07] paladox: any idea why a DROP query just isn't working? [12:32:17] I've left if for a few mins and there's no result [12:32:25] Process list also didn't show it as far as I saw [12:32:58] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.88, 6.18, 6.54 [12:32:59] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.90, 19.21, 19.06 [12:35:13] Reception123: you pushed a change this morning [12:35:32] So I would guess the pr you merged has a bug [12:36:44] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.94, 5.90, 6.73 [12:38:02] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 4.75, 5.88, 6.58 [12:43:56] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.63, 6.19, 6.80 [12:44:32] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.77, 6.81, 6.37 [12:44:45] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.37, 6.36, 5.92 [12:45:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.14, 20.54, 19.52 [12:46:18] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.49, 6.75, 6.04 [12:46:27] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.41, 6.60, 6.34 [12:46:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.42, 6.02, 5.85 [12:47:29] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.39, 19.59, 19.28 [12:48:16] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.42, 6.72, 6.12 [12:50:25] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.97, 7.55, 6.75 [12:51:19] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.14, 23.42, 20.94 [12:52:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.92, 7.70, 6.66 [12:53:16] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.89, 21.33, 20.47 [12:53:56] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.50, 7.24, 6.86 [12:54:25] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.98, 7.61, 6.99 [12:54:40] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.21, 6.84, 6.63 [12:55:12] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.35, 19.34, 19.87 [12:55:54] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.38, 6.93, 6.57 [12:55:56] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.38, 6.44, 6.60 [12:56:27] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.75, 6.41, 6.39 [12:56:35] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.34, 6.66, 6.59 [12:57:54] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.37, 6.46, 6.44 [12:58:25] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 12.76, 8.66, 7.42 [13:00:27] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.84, 7.58, 6.97 [13:01:00] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.60, 22.24, 20.90 [13:01:06] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:02:23] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.65, 7.37, 6.98 [13:02:57] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.21, 20.29, 20.33 [13:03:07] RhinosF1: yeah, that's why I was asking since I'm quite sure it has to be something there [13:03:53] Reception123: mobile so can't really debug [13:04:16] If it's an issue, revert [13:04:18] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.58, 8.13, 7.30 [13:04:26] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.49, 7.97, 7.79 [13:04:46] it did seem to ignore the other params too, maybe CosmicAlpha can have a look [13:05:01] If it's not working at all [13:05:03] Rollback [13:05:59] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/nukeNS.php --wiki=polandballwiki --delete --all --ns=3010 (END - exit=256) [13:06:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:06:06] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:06:13] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.49, 7.79, 7.29 [13:08:25] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.41, 8.27, 7.89 [13:09:33] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.87, 6.40, 6.05 [13:09:56] CosmicAlpha: Reception123 thinks it's broken [13:10:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.69, 6.89, 6.42 [13:10:11] Today hasn't been fun so probably won't get time to review stuff [13:11:29] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.94, 6.03, 5.97 [13:12:45] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.05, 6.85, 6.39 [13:13:39]