[00:03:10] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 38.83, 26.41, 20.81 [00:03:42] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 3 minutes ago with 10 failures. Failed resources (up to 3 shown): File[/etc/update-motd.d/97-last-puppet-run],File[/etc/sudoers],File[/home/salt-user],File[/etc/nginx/mime.types] [00:04:51] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.71, 6.33, 5.06 [00:06:31] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 12.85, 9.78, 7.44 [00:06:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.94, 8.99, 6.84 [00:07:01] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.77, 8.59, 6.51 [00:08:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.39, 7.61, 6.62 [00:09:01] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.67, 7.58, 6.38 [00:10:40] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.18, 6.99, 6.89 [00:10:57] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.88, 6.67, 6.16 [00:12:39] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.55, 6.20, 6.61 [00:12:46] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.07, 5.88, 6.14 [00:20:54] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:21:11] PROBLEM - cp13 Stunnel Http for mw12 on cp13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:22:28] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 2001:41d0:801:2000::58af/cpweb, 167.114.2.161/cpweb, 2607:5300:205:200::1c30/cpweb, 2607:5300:201:3100::1d3/cpweb [00:23:10] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 21089 bytes in 0.218 second response time [00:23:23] RECOVERY - cp13 Stunnel Http for mw12 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 2.441 second response time [00:24:32] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:28:28] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:29:15] PROBLEM - ping4 on ns1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 161.89 ms [00:31:03] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [00:32:57] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:33:41] RECOVERY - ping4 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 109.54 ms [00:35:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JKlaC [00:35:16] [02miraheze/services] 07MirahezeSSLBot 03ebfc521 - BOT: Updating services config for wikis [00:56:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.80, 3.50, 3.10 [00:58:21] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.35, 3.12, 3.01 [01:02:45] !log reboot mw10, php-fpm got stuck [01:02:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:03:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.16, 20.27, 23.60 [01:03:30] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.32, 6.65, 5.56 [01:04:23] RECOVERY - cp12 Stunnel Http for mw10 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.355 second response time [01:04:44] RECOVERY - cp13 Stunnel Http for mw10 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15869 bytes in 0.005 second response time [01:05:13] RECOVERY - cp12 Varnish Backends on cp12 is OK: All 9 backends are healthy [01:05:30] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.66, 6.92, 5.79 [01:05:32] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 21089 bytes in 0.306 second response time [01:05:47] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.45, 6.52, 5.75 [01:05:49] RECOVERY - cp14 Stunnel Http for mw10 on cp14 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.039 second response time [01:05:53] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.37, 6.72, 5.54 [01:05:54] RECOVERY - cp15 Stunnel Http for mw10 on cp15 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.336 second response time [01:06:03] RECOVERY - cp14 Varnish Backends on cp14 is OK: All 9 backends are healthy [01:06:26] RECOVERY - cp13 Varnish Backends on cp13 is OK: All 9 backends are healthy [01:06:43] RECOVERY - cp15 Varnish Backends on cp15 is OK: All 9 backends are healthy [01:07:26] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.92, 6.51, 5.78 [01:07:50] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.80, 5.68, 5.31 [01:09:45] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.89, 5.76, 5.69 [01:13:24] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.23, 4.74, 5.74 [01:13:33] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.02, 15.93, 20.24 [01:15:24] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.82, 6.03, 6.09 [01:19:04] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.38, 3.18 [01:21:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.52, 3.83, 3.37 [01:22:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 15.57, 19.31, 20.42 [01:23:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.72, 3.68, 3.36 [01:24:52] PROBLEM - mwtask1 Disk Space on mwtask1 is CRITICAL: DISK CRITICAL - free space: / 2784 MB (5% inode=89%); [01:24:53] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.10, 19.48, 20.33 [01:25:04] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.32, 3.40, 3.29 [01:25:32] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.27, 5.62, 5.95 [01:27:32] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.25, 5.75, 5.95 [01:29:31] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.49, 5.53, 5.84 [01:30:53] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.57, 3.42, 3.32 [01:34:44] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.76, 3.86, 3.51 [01:36:44] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 4.00, 3.62 [01:37:52] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.10, 6.57, 5.16 [01:39:52] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.34, 5.64, 4.99 [01:43:41] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.01, 4.16, 4.98 [01:44:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.60, 3.74, 3.57 [01:46:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.25, 3.52, 3.51 [01:52:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.37, 3.60, 3.51 [01:54:19] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.27, 3.19, 3.38 [02:22:37] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.96, 3.39, 3.12 [02:26:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.81, 3.15, 3.09 [03:04:23] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.23, 3.25, 3.02 [03:06:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.53, 2.92, 2.93 [03:24:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.70, 3.39, 3.13 [03:26:19] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.45, 3.08, 3.06 [03:47:17] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.71, 1.86, 1.14 [03:49:18] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.70, 1.39, 1.06 [03:51:45] PROBLEM - wiki.jumpbound.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.jumpbound.com' expires in 7 day(s) (Sun 24 Oct 2021 03:49:03 GMT +0000). [04:05:22] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.54, 6.41, 4.65 [04:06:13] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.75, 7.34, 5.27 [04:07:07] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 24.92, 19.97, 16.64 [04:07:22] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.39, 5.47, 4.52 [04:07:25] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 13.72, 7.84, 5.09 [04:08:12] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.54, 6.65, 5.29 [04:09:02] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 13.22, 17.37, 16.09 [04:09:24] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.84, 6.49, 4.93 [04:17:34] PROBLEM - biblestrength.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'biblestrength.net' expires in 7 day(s) (Sun 24 Oct 2021 04:16:04 GMT +0000). [04:20:18] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.31, 5.25, 4.57 [04:24:18] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.42, 4.52, 4.45 [04:48:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.29, 3.01 [04:50:19] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.42, 3.10, 2.99 [04:56:59] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±19] 13https://git.io/JKljr [04:57:01] [02miraheze/ssl] 07Reception123 0381150b6 - renew SSL certs [05:01:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.31, 3.09 [05:01:25] PROBLEM - cp13 Puppet on cp13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [05:03:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 3.56, 3.20 [05:05:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.09, 3.37, 3.17 [05:06:54] RECOVERY - www.wikilukas.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wikilukas.tk' will expire on Fri 14 Jan 2022 03:38:50 GMT +0000. [05:07:05] RECOVERY - www.marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Fri 14 Jan 2022 03:43:20 GMT +0000. [05:07:15] RECOVERY - wiki.valkyrienskies.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.valkyrienskies.org' will expire on Fri 14 Jan 2022 03:55:49 GMT +0000. [05:07:35] RECOVERY - cp13 Puppet on cp13 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [05:07:43] RECOVERY - www.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Fri 14 Jan 2022 03:39:45 GMT +0000. [05:08:02] RECOVERY - with.cpt-ra.bid - LetsEncrypt on sslhost is OK: OK - Certificate 'with.cpt-ra.bid' will expire on Fri 14 Jan 2022 03:45:14 GMT +0000. [05:08:19] RECOVERY - wiki.xysspon.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.xysspon.com' will expire on Fri 14 Jan 2022 03:50:51 GMT +0000. [05:08:43] RECOVERY - wiki.warframestat.us - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.warframestat.us' will expire on Fri 14 Jan 2022 03:52:46 GMT +0000. [05:08:54] RECOVERY - thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Fri 14 Jan 2022 03:39:45 GMT +0000. [05:09:15] RECOVERY - www.opendatascot.org - LetsEncrypt on sslhost is OK: OK - Certificate 'opendatascot.org' will expire on Fri 14 Jan 2022 03:42:07 GMT +0000. [05:09:37] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 5.77, 8.55, 5.93 [05:09:45] RECOVERY - wiki.velaan.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.velaan.org' will expire on Fri 14 Jan 2022 03:54:58 GMT +0000. [05:10:08] RECOVERY - wiki.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Fri 14 Jan 2022 03:39:45 GMT +0000. [05:10:26] RECOVERY - wiki.zymonic.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zymonic.com' will expire on Fri 14 Jan 2022 03:49:14 GMT +0000. [05:10:32] RECOVERY - opendatascot.org - LetsEncrypt on sslhost is OK: OK - Certificate 'opendatascot.org' will expire on Fri 14 Jan 2022 03:42:07 GMT +0000. [05:10:48] RECOVERY - wiki.villagecollaborative.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.villagecollaborative.net' will expire on Fri 14 Jan 2022 03:53:59 GMT +0000. [05:10:50] RECOVERY - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Fri 14 Jan 2022 03:43:20 GMT +0000. [05:11:03] RECOVERY - wikien.wildterra2.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikien.wildterra2.com' will expire on Fri 14 Jan 2022 03:48:05 GMT +0000. [05:11:05] RECOVERY - journeytheword.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.journeytheword.wiki' will expire on Fri 14 Jan 2022 03:44:27 GMT +0000. [05:11:15] RECOVERY - wikipariksha.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipariksha.com' will expire on Fri 14 Jan 2022 03:47:14 GMT +0000. [05:11:32] RECOVERY - wikiru.wildterra2.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiru.wildterra2.com' will expire on Fri 14 Jan 2022 03:46:13 GMT +0000. [05:11:36] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 3.79, 6.85, 5.60 [05:11:48] RECOVERY - wiki.whentheycry.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.whentheycry.org' will expire on Fri 14 Jan 2022 03:51:48 GMT +0000. [05:11:51] RECOVERY - wikilukas.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wikilukas.tk' will expire on Fri 14 Jan 2022 03:38:50 GMT +0000. [05:12:17] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.04, 23.08, 20.44 [05:12:25] RECOVERY - wiki.yunachannel.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.yunachannel.com' will expire on Fri 14 Jan 2022 03:50:00 GMT +0000. [05:12:32] RECOVERY - zw.fontainebleau-avon.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'zw.fontainebleau-avon.fr' will expire on Fri 14 Jan 2022 03:37:10 GMT +0000. [05:13:17] RECOVERY - www.journeytheword.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.journeytheword.wiki' will expire on Fri 14 Jan 2022 03:44:27 GMT +0000. [05:13:18] RECOVERY - wiki.triplescripts.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.triplescripts.org' will expire on Fri 14 Jan 2022 03:56:31 GMT +0000. [05:13:31] PROBLEM - wiki-asterix.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki-asterix.cf' expires in 15 day(s) (Mon 01 Nov 2021 05:09:56 GMT +0000). [05:13:35] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.19, 5.94, 5.42 [05:16:07] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.32, 19.64, 19.60 [05:21:33] PROBLEM - housing.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:24:45] PROBLEM - es.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - es.gyaanipedia.com All nameservers failed to answer the query. [05:25:41] PROBLEM - zh.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - zh.gyaanipedia.com All nameservers failed to answer the query. [05:26:49] PROBLEM - en.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.gyaanipedia.com All nameservers failed to answer the query. [05:26:51] PROBLEM - meta.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - meta.gyaanipedia.com All nameservers failed to answer the query. [05:27:31] PROBLEM - commons.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - commons.gyaanipedia.com All nameservers failed to answer the query. [05:28:14] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±12] 13https://git.io/JK8Jz [05:28:16] [02miraheze/ssl] 07Reception123 031e9a59a - renew SSL certs [05:28:17] PROBLEM - de.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - de.gyaanipedia.com All nameservers failed to answer the query. [05:28:26] PROBLEM - halloween.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - halloween.gyaanipedia.com All nameservers failed to answer the query. [05:28:43] PROBLEM - mr.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mr.gyaanipedia.com All nameservers failed to answer the query. [05:29:09] PROBLEM - bn.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - bn.gyaanipedia.com All nameservers failed to answer the query. [05:30:34] PROBLEM - lg.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - lg.gyaanipedia.com All nameservers failed to answer the query. [05:33:05] PROBLEM - fr.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - fr.gyaanipedia.com All nameservers failed to answer the query. [05:33:36] PROBLEM - pa.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - pa.gyaanipedia.com All nameservers failed to answer the query. [05:35:54] PROBLEM - bn.gyaanipedia.com - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in check_records nameserversans = dns_resolver.query(root_domain, 'NS') File "/usr/lib/pyt [05:35:54] dist-packages/dns/resolver.py", line 898, in query raise NoNameservers(request=request, errors=errors)dns.resolver.NoNameservers: All nameservers failed to answer the query gyaanipedia.com. IN NS: Server 1.1.1.1 UDP port 53 answered SERVFAIL [05:37:15] RECOVERY - wiki.minkyu.kim - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minkyu.kim' will expire on Fri 14 Jan 2022 04:26:43 GMT +0000. [05:37:21] PROBLEM - lg.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lg.gyaanipedia.com could not be found [05:37:40] RECOVERY - wiki.minecraftathome.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minecraftathome.com' will expire on Fri 14 Jan 2022 04:27:31 GMT +0000. [05:37:40] RECOVERY - wiki.nevillepedia.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nevillepedia.eu' will expire on Fri 14 Jan 2022 04:23:56 GMT +0000. [05:37:50] RECOVERY - wiki.starship.digital - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.starship.digital' will expire on Fri 14 Jan 2022 04:17:12 GMT +0000. [05:38:05] RECOVERY - wiki.s23.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.s23.org' will expire on Fri 14 Jan 2022 04:19:04 GMT +0000. [05:39:07] RECOVERY - wiki.raghuveer.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.raghuveer.net' will expire on Fri 14 Jan 2022 04:21:50 GMT +0000. [05:39:16] PROBLEM - zh.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for zh.gyaanipedia.com could not be found [05:39:37] RECOVERY - wiki.mxlinuxusers.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mxlinuxusers.de' will expire on Fri 14 Jan 2022 04:24:59 GMT +0000. [05:39:49] PROBLEM - fr.gyaanipedia.com - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in check_records nameserversans = dns_resolver.query(root_domain, 'NS') File "/usr/lib/pyt [05:39:49] dist-packages/dns/resolver.py", line 898, in query raise NoNameservers(request=request, errors=errors)dns.resolver.NoNameservers: All nameservers failed to answer the query gyaanipedia.com. IN NS: Server 1.1.1.1 UDP port 53 answered SERVFAIL [05:39:51] RECOVERY - wiki.seredrau.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.seredrau.xyz' will expire on Fri 14 Jan 2022 04:18:13 GMT +0000. [05:40:12] RECOVERY - wiki.rebirthofthenight.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rebirthofthenight.com' will expire on Fri 14 Jan 2022 04:21:08 GMT +0000. [05:40:27] RECOVERY - wiki.nowchess.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nowchess.org' will expire on Fri 14 Jan 2022 04:22:56 GMT +0000. [05:40:50] PROBLEM - meta.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for meta.gyaanipedia.com could not be found [05:41:23] PROBLEM - commons.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for commons.gyaanipedia.com could not be found [05:41:53] PROBLEM - halloween.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for halloween.gyaanipedia.com could not be found [05:42:09] PROBLEM - de.gyaanipedia.com - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/pyth [05:42:09] ist-packages/dns/resolver.py", line 898, in query raise NoNameservers(request=request, errors=errors)dns.resolver.NoNameservers: All nameservers failed to answer the query de.gyaanipedia.com. IN CNAME: Server 1.1.1.1 UDP port 53 answered SERVFAIL [05:42:38] RECOVERY - bn.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - bn.gyaanipedia.com reverse DNS resolves to cp14.miraheze.org - CNAME OK [05:42:43] PROBLEM - mr.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mr.gyaanipedia.com could not be found [05:43:23] RECOVERY - wiki.mobilityengineer.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mobilityengineer.com' will expire on Fri 14 Jan 2022 04:26:02 GMT +0000. [05:43:47] RECOVERY - wiki.ripto.gq - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ripto.gq' will expire on Fri 14 Jan 2022 04:19:41 GMT +0000. [05:44:07] PROBLEM - lg.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - lg.gyaanipedia.com All nameservers failed to answer the query. [05:45:24] RECOVERY - es.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - es.gyaanipedia.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [05:46:32] RECOVERY - fr.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - fr.gyaanipedia.com reverse DNS resolves to cp14.miraheze.org - CNAME OK [05:47:24] RECOVERY - en.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - en.gyaanipedia.com reverse DNS resolves to cp14.miraheze.org - CNAME OK [05:47:25] PROBLEM - pa.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pa.gyaanipedia.com could not be found [05:48:35] PROBLEM - halloween.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - halloween.gyaanipedia.com All nameservers failed to answer the query. [05:48:48] RECOVERY - mwtask1 Disk Space on mwtask1 is OK: DISK OK - free space: / 15131 MB (32% inode=89%); [05:49:09] RECOVERY - de.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - de.gyaanipedia.com reverse DNS resolves to cp14.miraheze.org - CNAME OK [05:50:00] PROBLEM - housing.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - housing.wiki reverse DNS resolves to a51062ecadbb5a26e.awsglobalaccelerator.com [05:55:17] PROBLEM - commons.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - commons.gyaanipedia.com All nameservers failed to answer the query. [05:57:41] PROBLEM - lg.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lg.gyaanipedia.com could not be found [05:59:40] PROBLEM - zh.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - zh.gyaanipedia.com All nameservers failed to answer the query. [06:01:50] PROBLEM - meta.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - meta.gyaanipedia.com All nameservers failed to answer the query. [06:02:13] PROBLEM - commons.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for commons.gyaanipedia.com could not be found [06:03:40] PROBLEM - mr.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mr.gyaanipedia.com All nameservers failed to answer the query. [06:04:28] PROBLEM - lg.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - lg.gyaanipedia.com All nameservers failed to answer the query. [06:05:23] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 26.17, 10.32, 5.32 [06:06:30] PROBLEM - zh.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for zh.gyaanipedia.com could not be found [06:08:42] PROBLEM - halloween.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for halloween.gyaanipedia.com could not be found [06:08:50] RECOVERY - de.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'sni.cloudflaressl.com' will expire on Sat 15 Oct 2022 23:59:59 GMT +0000. [06:09:06] RECOVERY - es.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'sni.cloudflaressl.com' will expire on Sat 15 Oct 2022 23:59:59 GMT +0000. [06:09:17] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 3.16, 7.83, 5.67 [06:10:41] PROBLEM - mr.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mr.gyaanipedia.com could not be found [06:10:46] PROBLEM - fr.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for fr.gyaanipedia.com could not be found [06:11:00] PROBLEM - en.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.gyaanipedia.com could not be found [06:11:15] PROBLEM - lg.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lg.gyaanipedia.com could not be found [06:11:15] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 2.94, 6.23, 5.33 [06:13:11] PROBLEM - hi.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for hi.gyaanipedia.com could not be found [06:13:37] PROBLEM - bn.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bn.gyaanipedia.com could not be found [06:14:03] PROBLEM - de.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for de.gyaanipedia.com could not be found [06:15:04] PROBLEM - pa.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - pa.gyaanipedia.com All nameservers failed to answer the query. [06:15:07] PROBLEM - es.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for es.gyaanipedia.com could not be found [06:18:01] PROBLEM - lg.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - lg.gyaanipedia.com All nameservers failed to answer the query. [06:19:56] PROBLEM - hi.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - hi.gyaanipedia.com All nameservers failed to answer the query. [06:20:06] PROBLEM - zh.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - zh.gyaanipedia.com All nameservers failed to answer the query. [06:21:02] PROBLEM - de.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - de.gyaanipedia.com All nameservers failed to answer the query. [06:22:00] PROBLEM - es.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - es.gyaanipedia.com All nameservers failed to answer the query. [06:24:48] PROBLEM - en.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.gyaanipedia.com All nameservers failed to answer the query. [06:24:50] PROBLEM - lg.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lg.gyaanipedia.com could not be found [06:26:43] PROBLEM - hi.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for hi.gyaanipedia.com could not be found [06:26:53] PROBLEM - zh.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for zh.gyaanipedia.com could not be found [06:27:53] [02miraheze/ssl] 07Reception123 pushed 035 commits to 03master [+0/-4/±27] 13https://git.io/JK8qs [06:27:54] [02miraheze/ssl] 07Reception123 034f9927e - renew SSL certs [06:27:56] [02miraheze/ssl] 07Reception123 031bb0b42 - remove wiki.arnab.at (deleted wiki) [06:27:57] PROBLEM - de.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for de.gyaanipedia.com could not be found [06:27:57] [02miraheze/ssl] 07Reception123 0366fc68a - remove battlecats.f5.si cert (deleted wiki) [06:27:59] [02miraheze/ssl] ... and 2 more commits. [06:28:54] PROBLEM - es.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for es.gyaanipedia.com could not be found [06:29:23] PROBLEM - halloween.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - halloween.gyaanipedia.com All nameservers failed to answer the query. [06:30:59] PROBLEM - fr.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - fr.gyaanipedia.com All nameservers failed to answer the query. [06:31:40] PROBLEM - en.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.gyaanipedia.com could not be found [06:31:44] PROBLEM - cp13 Puppet on cp13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:33:42] PROBLEM - zh.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - zh.gyaanipedia.com All nameservers failed to answer the query. [06:35:43] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:35:46] PROBLEM - pa.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pa.gyaanipedia.com could not be found [06:36:01] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:36:25] PROBLEM - halloween.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for halloween.gyaanipedia.com could not be found [06:37:08] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:37:19] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:37:46] PROBLEM - fr.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for fr.gyaanipedia.com could not be found [06:38:00] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:38:04] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [06:39:06] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:39:14] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [06:39:58] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:40:43] PROBLEM - zh.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for zh.gyaanipedia.com could not be found [06:41:01] PROBLEM - cp14 Puppet on cp14 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:41:12] PROBLEM - mw12 Puppet on mw12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:41:42] PROBLEM - cp15 Puppet on cp15 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:41:49] PROBLEM - cp12 Puppet on cp12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:43:24] PROBLEM - halloween.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - halloween.gyaanipedia.com All nameservers failed to answer the query. [06:43:28] PROBLEM - mwtask1 Puppet on mwtask1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:43:46] PROBLEM - meta.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for meta.gyaanipedia.com could not be found [06:43:56] PROBLEM - commons.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - commons.gyaanipedia.com All nameservers failed to answer the query. [06:45:33] PROBLEM - mr.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mr.gyaanipedia.com All nameservers failed to answer the query. [06:47:41] PROBLEM - zh.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - zh.gyaanipedia.com All nameservers failed to answer the query. [06:49:37] PROBLEM - pa.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - pa.gyaanipedia.com All nameservers failed to answer the query. [06:51:07] PROBLEM - commons.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for commons.gyaanipedia.com could not be found [06:52:15] PROBLEM - en.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.gyaanipedia.com All nameservers failed to answer the query. [06:52:34] PROBLEM - lg.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - lg.gyaanipedia.com All nameservers failed to answer the query. [06:53:48] RECOVERY - cp12 Puppet on cp12 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:56:33] PROBLEM - pa.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pa.gyaanipedia.com could not be found [06:57:41] PROBLEM - halloween.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for halloween.gyaanipedia.com could not be found [06:57:53] PROBLEM - commons.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - commons.gyaanipedia.com All nameservers failed to answer the query. [06:58:16] PROBLEM - fr.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - fr.gyaanipedia.com All nameservers failed to answer the query. [06:58:57] PROBLEM - en.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.gyaanipedia.com could not be found [06:59:14] PROBLEM - mr.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mr.gyaanipedia.com could not be found [06:59:50] RECOVERY - cp13 Puppet on cp13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:01:33] PROBLEM - zh.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for zh.gyaanipedia.com could not be found [07:03:58] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 10.34, 6.37, 4.27 [07:04:09] PROBLEM - meta.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - meta.gyaanipedia.com All nameservers failed to answer the query. [07:04:38] PROBLEM - commons.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for commons.gyaanipedia.com could not be found [07:05:19] PROBLEM - fr.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for fr.gyaanipedia.com could not be found [07:05:33] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:05:57] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.48, 5.15, 4.08 [07:06:05] PROBLEM - lg.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lg.gyaanipedia.com could not be found [07:07:14] RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:07:55] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.67, 4.27, 3.88 [07:08:02] PROBLEM - bn.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - bn.gyaanipedia.com All nameservers failed to answer the query. [07:08:09] RECOVERY - cp14 Puppet on cp14 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:09:40] RECOVERY - cp15 Puppet on cp15 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:11:25] PROBLEM - commons.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - commons.gyaanipedia.com All nameservers failed to answer the query. [07:11:33] RECOVERY - mwtask1 Puppet on mwtask1 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:11:38] PROBLEM - halloween.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - halloween.gyaanipedia.com All nameservers failed to answer the query. [07:14:59] PROBLEM - bn.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bn.gyaanipedia.com could not be found [07:16:26] PROBLEM - db13 APT on db13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:16:56] PROBLEM - pa.gyaanipedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - pa.gyaanipedia.com All nameservers failed to answer the query. [07:17:51] PROBLEM - meta.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for meta.gyaanipedia.com could not be found [07:18:23] RECOVERY - db13 APT on db13 is OK: APT OK: 60 packages available for upgrade (0 critical updates). [07:18:38] PROBLEM - halloween.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for halloween.gyaanipedia.com could not be found [07:23:43] PROBLEM - pa.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pa.gyaanipedia.com could not be found [07:24:50] PROBLEM - commons.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for commons.gyaanipedia.com could not be found [07:39:34] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JK8c2 [07:39:35] [02miraheze/ssl] 07Reception123 0360744bb - Update wiki.cyberfurs.org.crt [07:42:53] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 11.51, 7.83, 5.07 [07:43:03] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.24, 18.39, 14.73 [07:43:18] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.86, 6.65, 4.62 [07:44:49] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.04, 6.79, 5.04 [07:45:03] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 11.35, 15.75, 14.23 [07:45:14] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.28, 5.31, 4.37 [08:07:39] RECOVERY - nonciclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nonciclopedia.org' will expire on Fri 14 Jan 2022 05:19:09 GMT +0000. [08:07:52] RECOVERY - wiki.climatechange.ai - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.climatechange.ai' will expire on Fri 14 Jan 2022 05:05:51 GMT +0000. [08:08:39] RECOVERY - savagepedia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'savagepedia.wiki' will expire on Fri 14 Jan 2022 05:11:33 GMT +0000. [08:08:45] RECOVERY - dc-multiverse.dcwikis.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dc-multiverse.dcwikis.com' will expire on Fri 14 Jan 2022 05:22:09 GMT +0000. [08:08:52] RECOVERY - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thelonsdalebattalion.co.uk' will expire on Fri 14 Jan 2022 05:04:27 GMT +0000. [08:08:53] RECOVERY - spiral.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'spiral.wiki' will expire on Fri 14 Jan 2022 04:59:51 GMT +0000. [08:09:15] RECOVERY - wiki.cyberfurs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cyberfurs.org' will expire on Fri 14 Jan 2022 06:38:38 GMT +0000. [08:09:15] RECOVERY - history.estill.org - LetsEncrypt on sslhost is OK: OK - Certificate 'history.estill.org' will expire on Fri 14 Jan 2022 05:23:22 GMT +0000. [08:09:40] RECOVERY - wiki.anglish.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.anglish.info' will expire on Fri 14 Jan 2022 05:00:51 GMT +0000. [08:09:46] RECOVERY - bebaskanpengetahuan.id - LetsEncrypt on sslhost is OK: OK - Certificate 'bebaskanpengetahuan.id' will expire on Fri 14 Jan 2022 05:25:05 GMT +0000. [08:09:52] RECOVERY - wiki.lefrenchmelee.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.lefrenchmelee.fr' will expire on Fri 14 Jan 2022 05:09:01 GMT +0000. [08:09:55] RECOVERY - runzeppelin.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'runzeppelin.ru' will expire on Fri 14 Jan 2022 05:15:23 GMT +0000. [08:10:06] RECOVERY - spcodex.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'spcodex.wiki' will expire on Fri 14 Jan 2022 05:10:40 GMT +0000. [08:10:13] RECOVERY - kunwok.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kunwok.org' will expire on Fri 14 Jan 2022 05:03:30 GMT +0000. [08:10:58] RECOVERY - wiki.mastodon.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mastodon.kr' will expire on Fri 14 Jan 2022 05:18:13 GMT +0000. [08:12:04] RECOVERY - vise.dayid.org - LetsEncrypt on sslhost is OK: OK - Certificate 'vise.dayid.org' will expire on Fri 14 Jan 2022 05:01:48 GMT +0000. [08:12:19] RECOVERY - storytime.jdstroy.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'storytime.jdstroy.cf' will expire on Fri 14 Jan 2022 05:17:17 GMT +0000. [08:12:19] RECOVERY - miraheze.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.ga' will expire on Fri 14 Jan 2022 05:21:35 GMT +0000. [08:12:21] RECOVERY - vedopedia.witches-empire.com - LetsEncrypt on sslhost is OK: OK - Certificate 'vedopedia.witches-empire.com' will expire on Fri 14 Jan 2022 05:02:37 GMT +0000. [08:12:35] RECOVERY - icclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'icclopedia.org' will expire on Fri 14 Jan 2022 05:20:22 GMT +0000. [08:12:57] RECOVERY - vmcodex.net - LetsEncrypt on sslhost is OK: OK - Certificate 'vmcodex.net' will expire on Fri 14 Jan 2022 05:24:18 GMT +0000. [08:13:50] RECOVERY - wiki.mcpirevival.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mcpirevival.tk' will expire on Fri 14 Jan 2022 05:14:08 GMT +0000. [08:14:05] RECOVERY - wiki.kourouklides.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.kourouklides.com' will expire on Fri 14 Jan 2022 05:12:55 GMT +0000. [09:05:17] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.50, 6.59, 4.32 [09:07:15] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.66, 5.70, 4.27 [09:20:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JK8Vw [09:20:13] [02miraheze/services] 07MirahezeSSLBot 03584bd8e - BOT: Updating services config for wikis [09:45:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.17, 5.11, 3.90 [09:47:01] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.70, 4.49, 3.82 [11:01:37] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03add-labeler [+1/-0/±0] 13https://git.io/JK8yI [11:01:39] [02miraheze/mediawiki] 07RhinosF1 03ad02abc - Create label.yml [11:01:40] [02mediawiki] 07RhinosF1 created branch 03add-labeler - 13https://git.io/vbL5b [11:01:42] [02mediawiki] 07RhinosF1 opened pull request 03#3718: Create label.yml - 13https://git.io/JK8yt [11:05:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.73, 5.81, 4.33 [11:06:24] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03add-labeler [+1/-0/±0] 13https://git.io/JK8yD [11:06:26] [02miraheze/mediawiki] 07RhinosF1 0385b9666 - Create labeler.yml [11:06:27] [02mediawiki] 07RhinosF1 synchronize pull request 03#3718: Create label.yml - 13https://git.io/JK8yt [11:07:14] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.30, 4.82, 4.15 [11:10:23] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03add-labeler [+0/-0/±1] 13https://git.io/JK8SB [11:10:24] [02miraheze/mediawiki] 07RhinosF1 037c79171 - Update label.yml [11:10:26] [02mediawiki] 07RhinosF1 synchronize pull request 03#3718: Create label.yml - 13https://git.io/JK8yt [11:12:43] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03add-labeler [+0/-0/±1] 13https://git.io/JK8SD [11:12:44] [02miraheze/mediawiki] 07RhinosF1 037701943 - Update label.yml [11:12:46] [02mediawiki] 07RhinosF1 synchronize pull request 03#3718: Create label.yml - 13https://git.io/JK8yt [11:12:52] [02mediawiki] 07RhinosF1 closed pull request 03#3718: Create label.yml - 13https://git.io/JK8yt [11:12:53] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_36 [+2/-0/±0] 13https://git.io/JK8Sy [11:12:55] [02miraheze/mediawiki] 07RhinosF1 03524620d - Create label.yml (#3718) [11:12:56] [02mediawiki] 07RhinosF1 deleted branch 03add-labeler - 13https://git.io/vbL5b [11:12:58] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03add-labeler [11:14:11] [02mediawiki] 07RhinosF1 commented on pull request 03#3413: Bump extensions/DataTransfer from `bf4801d` to `8f8cb23` - 13https://git.io/JK8Sx [11:14:13] [02mediawiki] 07dependabot[bot] edited pull request 03#3413: Bump extensions/DataTransfer from `bf4801d` to `8f8cb23` - 13https://git.io/JoSt5 [11:14:59] [02mediawiki] 07dependabot[bot] edited pull request 03#3413: Bump extensions/DataTransfer from `bf4801d` to `8f8cb23` - 13https://git.io/JoSt5 [11:15:01] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/DataTransfer-8f8cb23 [+0/-0/±1] 13https://git.io/JK89k [11:15:03] [02miraheze/mediawiki] 07dependabot[bot] 039e45932 - Bump extensions/DataTransfer from `bf4801d` to `8f8cb23` [11:15:04] [02mediawiki] 07dependabot[bot] synchronize pull request 03#3413: Bump extensions/DataTransfer from `bf4801d` to `8f8cb23` - 13https://git.io/JoSt5 [11:21:02] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_36 [+1/-0/±0] 13https://git.io/JK899 [11:21:03] [02miraheze/mediawiki] 07RhinosF1 0314a3eaf - Create test [11:21:11] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-1/±0] 13https://git.io/JK89H [11:21:12] [02miraheze/mediawiki] 07RhinosF1 039e0267b - Delete test [11:21:14] [02mediawiki] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/vbL5b [11:21:16] [02mediawiki] 07RhinosF1 opened pull request 03#3719: Delete test - 13https://git.io/JK89Q [11:29:53] !log [rhinos@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/populateContentTables.php --wiki=shortstorieswiki (END - exit=0) [11:29:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:31:39] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JK8QU [11:31:41] [02miraheze/puppet] 07JohnFLewis 03a0469d4 - add storage.googleapis.com [11:33:42] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JK8Q4 [11:33:44] [02miraheze/puppet] 07JohnFLewis 033934e3a - add youtube.com to frame-src [11:34:02] !log INSERT INTO content_models (model_id, model_name) VALUES ( 1, 'wikitext') INSERT INTO content_models (model_id, model_name) VALUES ( 2, 'css');INSERT INTO content_models (model_id, model_name) VALUES ( 3, 'javascript');INSERT INTO content_models (model_id, model_name) VALUES ( 4, 'Scribunto');INSERT INTO content_models (model_id, model_name) VALUES ( 5, 'json'); - shortstorieswiki [11:34:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:37:35] !log [rhinos@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/moveBatch.php --wiki=lhmnwiki --noredirects -i=1 -r=T8122 move.txt (END - exit=256) [11:37:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:39:02] !log rerun above script correcly [11:39:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:41:30] !log [rhinos@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/moveBatch.php --wiki=lhmnwiki --noredirects --i=1 --r=T8122 move.txt (END - exit=0) [11:41:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:56:50] [02miraheze/mediawiki] 07RhinosF1 pushed 039 commits to 03REL1_37 [+0/-0/±19] 13https://git.io/JK8F6 [11:56:51] [02miraheze/mediawiki] 07jenkins-bot 031f40972 - Update git submodules [11:56:53] [02miraheze/mediawiki] 07reedy 030b28588 - PHPVersionCheck: Mark PHP 7.4.0 - 7.4.2 as buggy [11:56:54] [02miraheze/mediawiki] 07jdforrester 03f707b33 - Update git submodules [11:56:56] [02miraheze/mediawiki] ... and 6 more commits. [11:58:17] !log [rhinos@test3] starting deploy of {'world': True} to skip [11:58:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:01:05] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JK8b4 [12:01:06] [02miraheze/mw-config] 07RhinosF1 034d43a78 - disable wgParserEnableLegacyMediaDOM on test3wiki [12:01:08] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/vbvb3 [12:01:09] Reception123: ^ [12:01:14] [02mw-config] 07RhinosF1 opened pull request 03#4153: disable wgParserEnableLegacyMediaDOM on test3wiki - 13https://git.io/JK8bR [12:02:07] !log [rhinos@test3] finished deploy of {'world': True} to skip - SUCCESS in 230s [12:02:09] miraheze/mw-config - RhinosF1 the build passed. [12:02:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:02:16] JohnLewis: depending on runtime, only compressing old could still be fairly useful [12:02:26] [02mw-config] 07RhinosF1 closed pull request 03#4153: disable wgParserEnableLegacyMediaDOM on test3wiki - 13https://git.io/JK8bR [12:02:27] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JK8br [12:02:29] [02miraheze/mw-config] 07RhinosF1 03a1055f8 - disable wgParserEnableLegacyMediaDOM on test3wiki (#4153) [12:02:30] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-1 - 13https://git.io/vbvb3 [12:02:32] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-1 [12:02:44] Spookreeeno: Yeah, that's what I'm thinking at the minute [12:03:14] !log [@test3] starting deploy of {'config': True} to skip [12:03:15] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [12:03:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:03:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:03:30] miraheze/mw-config - RhinosF1 the build passed. [12:03:46] JohnLewis: I can't imagine it taking that long [12:05:00] there's enddate too [12:05:18] !log [@mw11] starting deploy of {'config': True} to all [12:05:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:05:34] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 16s [12:05:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:08:09] !log [rhinos@test3] sudo -u www-data php /srv/mediawiki/w/maintenance/purgeLis.php --wiki=test3wiki --all-namespaces --db-touch --verbose --delay=1 (END - exit=256) [12:08:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:08:44] !log invalidate parsercache on test3wiki to deploy parser changes [12:08:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:15:51] Spookreeeno: compressOld on test3 increased the database size [12:16:16] JohnLewis: increased? [12:16:50] yeah, by 4MB apparently [12:17:16] That's definately not supposed to happen [12:17:55] It might be simply because it's a 'small' wiki [12:18:05] It's very small [12:18:20] Is it worth trying on maybe testwiki [12:18:26] That gets actual usage [12:18:31] Sure [12:19:29] That's 4 times the size [12:19:43] !log running compressOld.php on testwiki for concat compression [12:19:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:21:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.43, 2.97, 2.55 [12:21:50] testwiki, no impact in size at all [12:22:06] that's not fun [12:23:16] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.72, 2.94, 2.59 [12:23:27] !log running compressOld.php on polandballwiki for concat compression [12:23:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:26:05] PROBLEM - db12 Disk Space on db12 is CRITICAL: DISK CRITICAL - free space: / 26066 MB (5% inode=98%); [12:26:36] urgh [12:27:18] JohnLewis: ^ [12:27:22] I know [12:27:24] JohnLewis: yeh that happened for me. Database sizes on large wikis increased but when I moved the db between db servers, the size decreased [12:27:27] That's why I want DB compression [12:27:46] JohnLewis: maybe optimise needs running? [12:29:38] maybe [12:29:50] !log [rhinos@test3] sudo -u www-data php /srv/mediawiki/w/maintenance/purgeList.php --wiki=test3wiki --all-namespaces --db-touch --verbose --delay=1 (END - exit=0) [12:29:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:30:17] JohnLewis: we can only try [12:30:45] Couldn’t we move the backups to bacula2? [12:31:01] mysqlcheck test3wiki -o should work [12:31:19] Though couldn’t we even generate the backups on bacula2 without needing to do it on the dbs? [12:31:25] no -o test3wiki [12:32:14] paladox: no, otherwise a backup takes 24-30 hours, increases risk of OOMs, incomplete backups, increases CPU usage etc. [12:32:26] Oh hmm [12:32:26] plus retention periods become worse [12:32:51] That was the problem with SPF's approach and the reason no progress was made on it - because it was unworkable [12:32:58] I don’t think we can use the backups anyways because of disk space [12:33:07] We wouldn’t be able to extract [12:33:53] you wouldn't be able to extract a database table of a few GBs at max? [12:33:57] I find that hard to believe [12:34:05] Oh [12:34:10] I thought it was one zip file [12:34:21] nope, it's per table gzip [12:34:27] Oh [12:36:31] Spookreeeno: good shout on optimise, test3 has halved in size [12:36:46] JohnLewis: \o/ [12:37:10] JohnLewis: I was going to suggest after 1.37 as there should be some space gains [12:37:23] but we can do now if it's that good [12:39:45] only issue is it is loocking [12:39:51] s/oo/o [12:39:52] Spookreeeno meant to say: only issue is it is locking [12:41:30] JohnLewis: maybe see if we can do it on some big wikis? if testwiki shows as good? [12:41:39] Agent could reach out and ask them [12:41:40] 500MB off testwiki [12:42:10] We've not got an urgent urgent need, so we could perhaps do it with 1.37 [12:43:04] JohnLewis: it might not be same day as I've no idea how long the cleanup will take [12:44:04] dropping flaggedimages + image metadata from PHP -> JSON are the 2 big changes [12:45:55] the image metadata i was planning on starting as soon as the actual schema changes finished. It only took like a minute on test3 but that's rubbish for seeing size changes. [12:46:11] and timings [12:59:10] I'm going to have a look at how much space could be saved by running an optimise on polandballwiki once compressOld is done [12:59:40] (though I'm not going to actually run the optimise) [13:02:53] ok [13:05:26] !log running compressOld.php for loginwiki [13:05:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:21:24] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JK4XW [13:21:26] [02miraheze/puppet] 07JohnFLewis 034e7981c - mariadb: drop binlog storage to 1 day, not 4 [13:21:55] !log clearing binlogs older than 24 hours on db12 - saved 28GB [13:21:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:22:12] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 49327 MB (11% inode=98%); [13:22:35] !log clearing binlogs older than 24 hours on db13 - saved 6GB [13:22:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:23:19] !log clearing binlogs older than 24 hours on db11 - saved 12GB [13:23:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:24:00] and with that simple thing, we've gone from CRIT to OK [13:26:26] \o/ [13:30:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JK4Hd [13:30:12] [02miraheze/services] 07MirahezeSSLBot 03fd639fa - BOT: Updating services config for wikis [13:31:32] !log running compressOld on closinglogosgroupwiki [13:31:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:43:00] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.17, 3.41, 3.12 [13:43:38] !log running compressOld on all wikis for data analysis [13:43:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:48:18] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.78, 4.86, 3.97 [13:50:19] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 5.04, 4.93, 4.11 [13:52:40] JohnLewis: are we going ahead with the change then? [13:52:55] Spookreeeno: it's the only suitable outcome [13:52:59] ack [13:53:02] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.43, 3.22, 3.28 [13:53:04] JohnLewis: just for old? [13:53:17] if so I'll do an update to people [13:53:24] if Agent isn't around [13:53:43] Yeah just for old [13:53:50] ReplaceText will continue to work [13:53:57] Agent: ^ [13:54:40] JohnLewis: do we want a regular run [13:58:58] Yeah [14:00:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JKBOw [14:00:14] [02miraheze/services] 07MirahezeSSLBot 03358b9a9 - BOT: Updating services config for wikis [14:04:30] JohnLewis: how often [14:05:40] I don't know, weekly? I think every 24 hours might be too excessive [14:06:39] will do weekly [14:07:36] [02puppet] 07RhinosF1 opened pull request 03#2035: Update runner.pp - 13https://git.io/JKBc4 [14:07:41] ^ [14:08:03] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.74, 3.51, 3.40 [14:09:08] [02puppet] 07JohnFLewis closed pull request 03#2035: Update runner.pp - 13https://git.io/JKBc4 [14:09:09] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JKBCo [14:09:11] [02miraheze/puppet] 07RhinosF1 03ccd4fe9 - Run compressOld.php weekly (#2035) [14:10:02] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.49, 3.21, 3.30 [14:23:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-4/±0] 13https://git.io/JKBgX [14:23:14] [02miraheze/puppet] 07paladox 03ab80d59 - Remove mathoid [14:31:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.80, 3.57, 3.39 [14:33:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.04, 3.57, 3.40 [14:35:46] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JKBPo [14:35:47] [02miraheze/MirahezeMagic] 07paladox 032a796e7 - Add all wikis to services [14:35:49] [02MirahezeMagic] 07paladox created branch 03paladox-patch-2 - 13https://git.io/fQRGX [14:35:50] [02MirahezeMagic] 07paladox opened pull request 03#293: Add all wikis to services - 13https://git.io/JKBPi [14:35:59] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.48, 3.44, 3.37 [14:36:40] [02MirahezeMagic] 07RhinosF1 commented on pull request 03#293: Add all wikis to services - 13https://git.io/JKBXa [14:36:46] paladox: why? [14:36:53] miraheze/MirahezeMagic - paladox the build passed. [14:37:50] Because there's a wiki that is trying to use restbase but hasn't enabled the extensions that adds it to restbase. So i don't see the point of only enabling it for the wikis that have the extension. [14:37:57] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.86, 3.33, 3.35 [14:38:09] [02MirahezeMagic] 07paladox commented on pull request 03#293: Add all wikis to services - 13https://git.io/JKB1H [14:38:10] paladox: but why? [14:38:19] Spookreeeno: i just explained why [14:38:30] paladox: what's the wiki trying to do? [14:38:42] Without a linked extension, it's of no benefit [14:38:46] https://polcompball.miraheze.org/wiki/MediaWiki:Common.js [14:38:47] [url] MediaWiki:Common.js - Polcompball Wiki | polcompball.miraheze.org [14:39:07] see line 45 [14:39:22] paladox: does the top articles api even work? [14:41:57] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.58, 3.96, 3.61 [14:42:20] Ahh, no. [14:42:37] paladox: so no need [14:43:14] [02MirahezeMagic] 07RhinosF1 commented on pull request 03#293: Add all wikis to services - 13https://git.io/JKBS9 [14:43:15] [02MirahezeMagic] 07RhinosF1 closed pull request 03#293: Add all wikis to services - 13https://git.io/JKBPi [14:43:17] [02MirahezeMagic] 07RhinosF1 deleted branch 03paladox-patch-2 - 13https://git.io/fQRGX [14:43:19] [02miraheze/MirahezeMagic] 07RhinosF1 deleted branch 03paladox-patch-2 [14:43:29] i don't see the need to put it behind an extension flag [14:43:51] paladox: per conversation in security, not seeing the need to any wikis having it [14:43:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.57, 3.78, 3.59 [14:44:15] ok [14:44:52] paladox: my whole argument is we can decom the whole thing if there are minimal users [14:45:22] for the 1 working use today, i personally we think we can [14:45:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.81, 4.10, 3.73 [14:45:59] Ok, you'll need to open the community process to remove it. [14:46:19] also need to uninstall citoid (extension) and electron and collection. [14:46:53] paladox: that's what we have Agent for [14:47:53] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.07, 3.66, 3.61 [14:48:10] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.06, 5.12, 4.49 [14:49:20] indeed, that's why I'm here. I haven't received data on the top users yet but I've been using WikiApiary to see more or less who uses it and it's not being really used by the wikis I've visited [14:49:53] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.28, 3.91, 3.71 [14:50:03] Agent: https://phabricator.miraheze.org/T8180 [14:50:04] [url] ⚓ T8180 Remove Collection & Citioid | phabricator.miraheze.org [14:50:16] o [14:50:25] https://nonbinary.wiki/wiki/User:Amazingakita was the only non me use of either [14:50:26] [url] User:Amazingakita - Nonbinary Wiki | nonbinary.wiki [14:50:31] they used citoid today [14:51:04] that's since midnight UTC and now [14:52:09] Agent: ^ [14:52:12] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.16, 5.54, 4.80 [14:53:17] yeah no, it seems to have very minimal usage. I'll start a community discussion about removing Citoid/Collection with this info then [14:53:51] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.39, 3.67, 3.66 [14:54:11] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.35, 4.71, 4.58 [14:55:17] Agent: also, DB compression has been done without impacting ReplaceText [14:56:46] Spookreeeno: I saw. That's great, I guess we can move to close the Requests for Feedback as moot [14:57:48] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JKBph [14:57:50] [02miraheze/mw-config] 07RhinosF1 032af0107 - MWE: Prevent new installs of Citoid/Collection [14:57:51] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/vbvb3 [14:57:53] [02mw-config] 07RhinosF1 opened pull request 03#4154: MWE: Prevent new installs of Citoid/Collection - 13https://git.io/JKBhJ [14:58:02] Agent: yep! [14:58:12] Agent: does my change above sound ok? [14:58:54] miraheze/mw-config - RhinosF1 the build passed. [14:59:02] Spookreeeno: yep, looks good to me [14:59:21] [02mw-config] 07RhinosF1 closed pull request 03#4154: MWE: Prevent new installs of Citoid/Collection - 13https://git.io/JKBhJ [14:59:22] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JKBjZ [14:59:24] [02miraheze/mw-config] 07RhinosF1 03c881484 - MWE: Prevent new installs of Citoid/Collection (#4154) [14:59:25] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-1 [14:59:27] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-1 - 13https://git.io/vbvb3 [14:59:38] !log [rhinos@test3] starting deploy of {'config': True} to skip [14:59:39] !log [rhinos@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [14:59:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:59:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:00:23] miraheze/mw-config - RhinosF1 the build passed. [15:00:56] would help if i pulled the change [15:01:37] !log [@test3] starting deploy of {'config': True} to skip [15:01:38] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [15:01:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:01:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:01:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.58, 3.83, 3.69 [15:02:22] PROBLEM - bharatwiki.online - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [15:06:34] !log [@mw11] starting deploy of {'config': True} to all [15:06:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:06:55] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 21s [15:07:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:07:09] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.88, 6.25, 5.22 [15:07:18] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.57, 8.95, 6.32 [15:07:30] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.03, 6.62, 5.02 [15:07:30] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.31, 8.22, 6.42 [15:09:07] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.30, 5.76, 5.19 [15:09:18] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.11, 7.47, 6.10 [15:09:25] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.87, 5.93, 4.96 [15:09:26] [02puppet] 07RhinosF1 opened pull request 03#2036: services: decom - 13https://git.io/JKRky [15:09:33] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.06, 7.03, 6.20 [15:11:18] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.15, 6.74, 6.00 [15:11:18] [02dns] 07RhinosF1 opened pull request 03#223: services: drop - 13https://git.io/JKRLB [15:11:33] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.67, 6.43, 6.06 [15:11:51] [02dns] 07RhinosF1 synchronize pull request 03#223: services: drop - 13https://git.io/JKRLB [15:11:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.94, 3.90 [15:16:33] Spookreeeno: any stats on Citoid/Collection usage yesterday or all of last week? [15:17:48] JohnLewis: ^ [15:18:16] Agent: based on grafana, I'd guess very simmilar [15:24:28] PROBLEM - bharatwiki.online - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - bharatwiki.online reverse DNS resolves to 30.99.98.34.bc.googleusercontent.com [15:25:59] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.45, 3.72, 3.62 [15:27:59] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.29, 3.48, 3.54 [15:30:01] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.58, 3.87, 3.67 [15:31:59] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.49, 3.67, 3.62 [15:32:56] [02puppet] 07RhinosF1 opened pull request 03#2037: Update openldap.pp - 13https://git.io/JKR0R [15:34:58] bonjour [15:35:05] paladox: ^ [15:35:08] Hi Hispano76 [15:35:30] [02puppet] 07paladox closed pull request 03#2037: Update openldap.pp - 13https://git.io/JKR0R [15:35:32] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JKRuQ [15:35:33] [02miraheze/puppet] 07RhinosF1 034c30280 - Update openldap.pp [15:35:35] [02miraheze/puppet] 07paladox 034cd5324 - Merge pull request #2037 from RhinosF1/patch-103 [15:36:04] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.83, 3.92, 3.68 [15:37:56] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.60, 7.22, 6.09 [15:39:55] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.51, 6.64, 5.99 [15:40:01] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.88, 3.79, 3.67 [15:40:56] CosmicAlpha: just a quick observation from task reviewing, you comment a lot "I will do x" and then you unassign yourself - effectively stalling a tasks process. [15:41:59] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.19, 3.90, 3.73 [15:43:59] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.66, 3.57, 3.64 [15:50:08] PROBLEM - db13 Disk Space on db13 is WARNING: DISK WARNING - free space: / 48097 MB (10% inode=98%); [15:51:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.57, 3.10, 3.38 [16:05:29] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.39, 7.67, 5.16 [16:05:59] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.98, 18.86, 16.55 [16:06:07] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.04, 6.33, 5.37 [16:07:25] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.86, 6.61, 5.07 [16:07:48] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.61, 3.69, 3.38 [16:07:58] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.22, 17.75, 16.42 [16:08:06] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.01, 5.35, 5.13 [16:09:47] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.23, 3.08, 3.19 [16:15:29] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.59, 5.48, 4.91 [16:22:47] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.54, 3.66, 3.38 [16:23:42] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.09, 6.32, 5.38 [16:27:46] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.15, 5.48, 5.29 [16:28:42] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.67, 3.30, 3.33 [16:31:47] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.04, 4.46, 4.92 [16:34:43] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.38, 3.83, 3.55 [16:38:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.21, 3.86, 3.65 [16:39:04] [02puppet] 07RhinosF1 opened pull request 03#2038: mediawiki: disable restbase - 13https://git.io/JK0se [16:39:16] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48892 MB (10% inode=98%); [16:39:28] JohnLewis: I'm aware. I committed to tasks I shouldn't have. As I had to much to do at once. I won't do that again. So apologies for that. Since I already committed to them I likely should still do them I understand that but I unassigned myself so someone else could if wanted. [16:40:22] [02puppet] 07RhinosF1 synchronize pull request 03#2038: mediawiki: disable restbase - 13https://git.io/JK0se [16:41:11] [02puppet] 07RhinosF1 synchronize pull request 03#2038: mediawiki: disable restbase - 13https://git.io/JK0se [16:42:57] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JK0ZQ [16:42:59] [02miraheze/mw-config] 07RhinosF1 030ffb69a - Remove Collection & Citoid [16:43:00] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/vbvb3 [16:43:02] [02mw-config] 07RhinosF1 opened pull request 03#4155: Remove Collection & Citoid - 13https://git.io/JK0ZF [16:44:56] miraheze/mw-config - RhinosF1 the build passed. [16:46:41] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.90, 3.20, 3.37 [16:55:48] [02puppet] 07Universal-Omega commented on pull request 03#2003: Remove parsoid.js from miraheze_project_sys.yaml - 13https://git.io/JK0Eu [16:55:50] [02puppet] 07Universal-Omega closed pull request 03#2003: Remove parsoid.js from miraheze_project_sys.yaml - 13https://git.io/J212X [16:59:30] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JK0gS [16:59:31] [02miraheze/mediawiki] 07paladox 03e0efaac - Update medik [17:01:20] Why is Security Reviewer archived? [17:01:47] !log [paladox@test3] starting deploy of {'world': True, 'gitinfo': True} to skip [17:01:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:01:53] !log [paladox@test3] finished deploy of {'world': True, 'gitinfo': True} to skip - FAIL: [2, 2, 2, 2] in 6s [17:01:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:02:13] paladox: errr? [17:02:20] Fail not good [17:02:22] !log [paladox@test3] starting deploy of {'world': True, 'gitinfo': True} to skip [17:02:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:02:40] merge conflicts because i had a local hack [17:02:44] Ah [17:03:04] * Spookreeeno should know user abort is 2 [17:03:37] soemone want to fix liberty? [17:03:38] https://test3.miraheze.org/w/index.php?title=Main_Page&useskin=liberty [17:03:39] [url] Test3 | test3.miraheze.org [17:04:56] We probably should [17:05:08] I don't believe anyones tested it yet [17:06:23] !log [paladox@test3] finished deploy of {'world': True, 'gitinfo': True} to skip - SUCCESS in 240s [17:06:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:14:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JK06m [17:14:29] [02miraheze/puppet] 07paladox 0360a840d - monitoring: Update mariadb to 10.4 [17:16:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JK0iJ [17:16:14] [02miraheze/puppet] 07paladox 034ba4c12 - mon2: Lower MaxConcurrentChecks to 4 [17:17:12] Bukkit: No security reviewers anymore. There are now Security Engineers. https://meta.miraheze.org/wiki/Miraheze_Vacancies#Security_Engineers_(Infrastructure/MediaWiki) [17:17:13] [url] Miraheze Vacancies - Miraheze Meta | meta.miraheze.org [17:19:25] !log upgrade grafana on mon2 [17:19:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:25:04] !log upgrade puppet-agent puppetdb puppetserver on puppet3 [17:25:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:27:48] !log upgrade puppet-agent on mon2 [17:27:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:29:30] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JK0Xi [17:29:31] [02miraheze/mw-config] 07RhinosF1 0356a0917 - Update ManageWikiExtensions.php [17:29:33] [02mw-config] 07RhinosF1 synchronize pull request 03#4155: Remove Collection & Citoid - 13https://git.io/JK0ZF [17:30:06] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JK0X1 [17:30:07] [02miraheze/mw-config] 07RhinosF1 03d2f487a - Update Defines.php [17:30:09] [02mw-config] 07RhinosF1 synchronize pull request 03#4155: Remove Collection & Citoid - 13https://git.io/JK0ZF [17:30:32] miraheze/mw-config - RhinosF1 the build passed. [17:31:07] miraheze/mw-config - RhinosF1 the build passed. [17:31:13] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JK0X7 [17:31:15] [02miraheze/mw-config] 07RhinosF1 03eb08270 - Update LocalSettings.php [17:31:16] [02mw-config] 07RhinosF1 synchronize pull request 03#4155: Remove Collection & Citoid - 13https://git.io/JK0ZF [17:31:22] !log upgrade puppet-agent everywhere [17:31:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:32:10] miraheze/mw-config - RhinosF1 the build passed. [17:32:31] [02puppet] 07RhinosF1 synchronize pull request 03#2038: mediawiki: disable restbase - 13https://git.io/JK0se [17:33:57] [02puppet] 07RhinosF1 synchronize pull request 03#2038: mediawiki: disable restbase - 13https://git.io/JK0se [17:34:31] [02miraheze/MirahezeMagic] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-1/±0] 13https://git.io/JK01Y [17:34:33] [02miraheze/MirahezeMagic] 07RhinosF1 032dcb702 - Services: delete [17:34:34] [02MirahezeMagic] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/fQRGX [17:34:36] [02MirahezeMagic] 07RhinosF1 opened pull request 03#294: Services: delete - 13https://git.io/JK01O [17:35:13] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.33, 4.40, 3.76 [17:35:33] miraheze/MirahezeMagic - RhinosF1 the build passed. [17:39:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.76, 3.62 [17:39:29] PROBLEM - phab2 Current Load on phab2 is WARNING: WARNING - load average: 0.41, 1.74, 1.22 [17:39:41] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.40, 7.02, 5.86 [17:39:47] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.81, 7.33, 6.05 [17:40:36] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.53, 8.20, 6.28 [17:41:34] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 17.67, 25.80, 24.04 [17:41:46] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 22.80, 27.10, 23.36 [17:41:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.28, 7.88, 6.69 [17:41:56] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:42:32] RECOVERY - phab2 Current Load on phab2 is OK: OK - load average: 0.15, 1.01, 1.03 [17:44:17] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 18.79, 22.54, 23.04 [17:45:27] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.58, 6.49, 6.04 [17:45:44] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.57, 6.20, 6.00 [17:47:43] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 14.50, 20.44, 21.71 [17:47:45] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.30, 6.34, 6.36 [17:49:33] RhinosF1: removing services will also cause issues with Math and 3D extensions I think. (3D can probably do without, if 3d2png is put somewhere else) but math looks like it won't. [17:49:48] Spookreeeno: ^ [17:50:10] CosmicAlpha: both use cli [17:50:28] they don't go through services[34] and restbase [17:51:20] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.99, 3.76, 3.56 [17:51:20] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.96, 5.83, 6.00 [17:52:52] [02puppet] 07Universal-Omega opened pull request 03#2039: Absent services crons - 13https://git.io/JK0M1 [17:53:17] PROBLEM - cloud4 Puppet on cloud4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 9 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[puppet-agent] [17:53:52] CosmicAlpha: absent services crons can go now [17:54:08] it's mw-restricted anyway [17:58:09] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.98, 18.49, 20.28 [17:58:29] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.54, 3.18, 3.40 [17:58:32] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.60, 5.37, 5.73 [17:59:48] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.08, 18.21, 19.93 [18:04:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JK0D1 [18:04:59] [02miraheze/puppet] 07paladox 03aca42c7 - Remove test4 [18:05:00] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [18:05:02] [02puppet] 07paladox opened pull request 03#2040: Remove test4 - 13https://git.io/JK0DM [18:05:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/JK0D9 [18:05:27] [02miraheze/puppet] 07paladox 03c2a0692 - Delete nginx.conf.erb [18:05:28] [02puppet] 07paladox synchronize pull request 03#2040: Remove test4 - 13https://git.io/JK0DM [18:05:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JK0Dd [18:06:01] [02miraheze/puppet] 07paladox 035d97314 - Remove test4 [18:06:02] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [18:06:04] [02puppet] 07paladox opened pull request 03#2041: Remove test4 - 13https://git.io/JK0DF [18:06:06] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:06:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JK0DA [18:06:26] [02miraheze/puppet] 07paladox 0358dcc89 - Update nginx.conf.erb [18:06:28] [02puppet] 07paladox synchronize pull request 03#2041: Remove test4 - 13https://git.io/JK0DF [18:06:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JK0Dh [18:06:43] [02miraheze/puppet] 07paladox 03d532534 - Update site.pp [18:06:44] [02puppet] 07paladox synchronize pull request 03#2041: Remove test4 - 13https://git.io/JK0DF [18:06:52] [02puppet] 07paladox closed pull request 03#2041: Remove test4 - 13https://git.io/JK0DF [18:06:53] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [18:06:55] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [18:06:56] [02miraheze/puppet] 07paladox pushed 034 commits to 03master [+0/-0/±6] 13https://git.io/JK0yv [18:06:58] [02miraheze/puppet] 07paladox 035d97314 - Remove test4 [18:06:59] [02miraheze/puppet] 07paladox 0358dcc89 - Update nginx.conf.erb [18:07:01] [02miraheze/puppet] 07paladox 03d532534 - Update site.pp [18:07:02] [02miraheze/puppet] 07paladox 0393758c7 - Merge pull request #2041 from miraheze/paladox-patch-4 [18:07:37] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JK0yT [18:07:38] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.58, 4.34, 5.03 [18:07:39] [02miraheze/dns] 07paladox 03ab337f2 - Remove test4 from dns [18:07:40] [02dns] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbQXl [18:07:42] [02dns] 07paladox opened pull request 03#224: Remove test4 from dns - 13https://git.io/JK0yk [18:08:34] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 17.62, 12.36, 8.61 [18:10:19] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.21, 20.58, 20.34 [18:13:25] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.17, 20.32, 20.33 [18:13:50] [02puppet] 07paladox closed pull request 03#2040: Remove test4 - 13https://git.io/JK0DM [18:13:51] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [18:13:53] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [18:15:06] RECOVERY - cloud4 Puppet on cloud4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:17:27] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.60, 7.77, 7.90 [18:19:20] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JK0SL [18:19:21] [02miraheze/mw-config] 07RhinosF1 0347b5a80 - Update extension-list [18:19:23] [02mw-config] 07RhinosF1 synchronize pull request 03#4155: Remove Collection & Citoid - 13https://git.io/JK0ZF [18:20:26] miraheze/mw-config - RhinosF1 the build passed. [18:23:00] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.05, 3.48, 3.26 [18:25:57] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.80, 3.18, 3.18 [18:32:02] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.00, 5.46, 6.43 [18:34:11] [02puppet] 07Universal-Omega commented on pull request 03#2036: services: decom - 13https://git.io/JK090 [18:45:07] PROBLEM - db11 Disk Space on db11 is WARNING: DISK WARNING - free space: / 48029 MB (10% inode=97%); [18:49:52] Reception123: https://phabricator.miraheze.org/T7740 was a question posed by paladox on Monday [18:49:54] [url] ⚓ T7740 Reconsider production usage of Elastic/CirrusSearch | phabricator.miraheze.org [18:52:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03grafana-webhook [+2/-0/±1] 13https://git.io/JK0QE [18:52:41] [02miraheze/puppet] 07paladox 0332fb545 - Add a grafana webhook [18:52:42] [02puppet] 07paladox created branch 03grafana-webhook - 13https://git.io/vbiAS [18:52:57] [02puppet] 07paladox opened pull request 03#2042: Add a grafana webhook - 13https://git.io/JK0Qz [18:55:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03grafana-webhook [+0/-0/±1] 13https://git.io/JK0QX [18:55:49] [02miraheze/puppet] 07paladox 03d489379 - Update grafana-webhook.py [18:55:51] [02puppet] 07paladox synchronize pull request 03#2042: Add a grafana webhook - 13https://git.io/JK0Qz [18:57:07] [02puppet] 07paladox closed pull request 03#2042: Add a grafana webhook - 13https://git.io/JK0Qz [18:57:09] [02miraheze/puppet] 07paladox pushed 033 commits to 03master [+4/-0/±3] 13https://git.io/JK0Q5 [18:57:10] [02miraheze/puppet] 07paladox 0332fb545 - Add a grafana webhook [18:57:12] [02miraheze/puppet] 07paladox 03d489379 - Update grafana-webhook.py [18:57:13] [02miraheze/puppet] 07paladox 034f0e3ad - Merge pull request #2042 from miraheze/grafana-webhook [18:57:15] [02miraheze/puppet] 07paladox deleted branch 03grafana-webhook [18:57:16] [02puppet] 07paladox deleted branch 03grafana-webhook - 13https://git.io/vbiAS [18:57:18] [02miraheze/puppet] 07paladox created branch 03grafana-webhook 13https://git.io/JK0Qd [18:57:19] [02puppet] 07paladox created branch 03grafana-webhook - 13https://git.io/vbiAS [18:57:21] [02miraheze/puppet] 07paladox deleted branch 03grafana-webhook [18:57:22] [02puppet] 07paladox deleted branch 03grafana-webhook - 13https://git.io/vbiAS [19:02:19] [02puppet] 07Universal-Omega opened pull request 03#2043: Remove trafficserver - 13https://git.io/JK074 [19:12:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.70, 3.58, 3.44 [19:16:39] paladox: ^ feel free to close #2043 if you don't want to completely remove it. But without test4 and the task being declined I thought it might be a good idea to. If not then closing is fine. [19:17:09] [02puppet] 07paladox closed pull request 03#2043: Remove trafficserver - 13https://git.io/JK074 [19:17:11] [02miraheze/puppet] 07paladox pushed 0311 commits to 03master [+0/-82/±6] 13https://git.io/JK057 [19:17:12] [02miraheze/puppet] 07Universal-Omega 03b677e15 - Remove trafficserver [19:17:14] [02miraheze/puppet] 07Universal-Omega 032b39fb0 - Delete trafficserver.pp [19:17:15] [02miraheze/puppet] 07Universal-Omega 0395fdb44 - Delete csp.yaml [19:17:17] [02miraheze/puppet] ... and 8 more commits. [19:17:44] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.64, 3.19, 3.33 [19:25:28] paladox: Thanks! And also at some point do you mind reviewing https://github.com/miraheze/puppet/pull/2032? That PR should be complete now. Not urgent of course, so just whenever you have time. [19:25:29] [url] T7073: enable `prometheus-es-exporter` by Universal-Omega · Pull Request #2032 · miraheze/puppet · GitHub | github.com [19:26:07] [02puppet] 07paladox closed pull request 03#2032: T7073: enable `prometheus-es-exporter` - 13https://git.io/JKnGz [19:26:08] [02miraheze/puppet] 07paladox pushed 0311 commits to 03master [+2/-2/±14] 13https://git.io/JK0du [19:26:10] [02miraheze/puppet] 07Universal-Omega 0377b377b - Enable prometheus-es-exporter [19:26:11] [02miraheze/puppet] 07Universal-Omega 0314399d8 - Update init.pp [19:26:13] [02miraheze/puppet] 07Universal-Omega 03b80ce21 - Update site.pp [19:26:14] [02miraheze/puppet] ... and 8 more commits. [19:49:22] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.60, 4.72, 3.95 [19:52:26] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.19, 4.43, 3.97 [19:53:51] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.87, 3.59, 3.29 [19:56:48] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.29, 3.73, 3.38 [19:58:11] Paladox: can you start using squash and merge please? It’s making the history look messy [20:02:37] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.47, 3.34, 3.36 [20:21:42] Yes [20:24:57] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.82, 6.85, 6.20 [20:26:16] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.01, 6.95, 5.63 [20:28:09] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.94, 8.77, 7.07 [20:29:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.28, 3.92, 3.57 [20:31:28] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.25, 21.14, 19.41 [20:31:48] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.32, 8.47, 6.87 [20:35:34] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.66, 3.59, 3.56 [20:37:37] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.83, 7.87, 7.39 [20:38:12] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.93, 7.90, 7.14 [20:38:40] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.73, 7.15, 5.99 [20:39:09] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.17, 7.32, 7.04 [20:40:07] [02mediawiki] 07Universal-Omega commented on pull request 03#3352: Bump extensions/MirahezeMagic from `1fccf46` to `9c33cf8` - 13https://git.io/JK0x7 [20:40:09] [02mediawiki] 07dependabot[bot] edited pull request 03#3352: Bump extensions/MirahezeMagic from `1fccf46` to `9c33cf8` - 13https://git.io/JoSka [20:40:54] [02mediawiki] 07dependabot[bot] edited pull request 03#3352: Bump extensions/MirahezeMagic from `1fccf46` to `9c33cf8` - 13https://git.io/JoSka [20:41:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/MirahezeMagic-e5c14b0 [+0/-0/±1] 13https://git.io/JK0xA [20:41:01] [02miraheze/mediawiki] 07dependabot[bot] 03464dbd5 - Bump extensions/MirahezeMagic from `1fccf46` to `e5c14b0` [20:41:03] [02mediawiki] 07dependabot[bot] opened pull request 03#3720: Bump extensions/MirahezeMagic from `1fccf46` to `e5c14b0` - 13https://git.io/JK0xx [20:41:04] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/MirahezeMagic-e5c14b0 - 13https://git.io/vbL5b [20:41:06] [02mediawiki] 07dependabot[bot] labeled pull request 03#3720: Bump extensions/MirahezeMagic from `1fccf46` to `e5c14b0` - 13https://git.io/JK0xx [20:41:07] [02mediawiki] 07dependabot[bot] labeled pull request 03#3720: Bump extensions/MirahezeMagic from `1fccf46` to `e5c14b0` - 13https://git.io/JK0xx [20:41:09] [02mediawiki] 07dependabot[bot] commented on pull request 03#3352: Bump extensions/MirahezeMagic from `1fccf46` to `9c33cf8` - 13https://git.io/JK0xh [20:41:10] [02mediawiki] 07dependabot[bot] closed pull request 03#3352: Bump extensions/MirahezeMagic from `1fccf46` to `9c33cf8` - 13https://git.io/JoSka [20:41:12] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/MirahezeMagic-9c33cf8 - 13https://git.io/vbL5b [20:41:13] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/MirahezeMagic-9c33cf8 [20:41:15] [02mediawiki] 07dependabot[bot] assigned pull request 03#3720: Bump extensions/MirahezeMagic from `1fccf46` to `e5c14b0` - 13https://git.io/JK0xx [20:41:42] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.98, 5.83, 5.68 [20:42:40] [02mediawiki] 07Universal-Omega closed pull request 03#3720: Bump extensions/MirahezeMagic from `1fccf46` to `e5c14b0` - 13https://git.io/JK0xx [20:42:41] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JK0pI [20:42:43] [02miraheze/mediawiki] 07dependabot[bot] 0300821c0 - Bump extensions/MirahezeMagic from `1fccf46` to `e5c14b0` (#3720) [20:42:44] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/extensions/MirahezeMagic-e5c14b0 [20:42:46] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/extensions/MirahezeMagic-e5c14b0 - 13https://git.io/vbL5b [20:43:43] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.48, 19.23, 19.62 [20:44:59] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.92, 6.31, 6.70 [20:46:18] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.85, 5.63, 6.49 [20:46:59] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.75, 5.78, 6.48 [20:50:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.10, 3.57, 3.51 [20:53:35] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.50, 2.98, 3.28 [21:01:04] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 13.53, 9.52, 7.07 [21:05:39] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.99, 6.64, 6.39 [21:06:11] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.49, 22.45, 19.39 [21:07:18] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.85, 7.75, 6.96 [21:08:40] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.08, 6.03, 6.18 [21:09:20] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.97, 19.83, 18.99 [21:10:25] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.16, 6.09, 6.46 [21:13:27] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.76, 7.19, 7.76 [21:15:37] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.81, 7.72, 6.94 [21:18:38] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.80, 6.68, 6.70 [21:21:01] PROBLEM - db12 Disk Space on db12 is CRITICAL: DISK CRITICAL - free space: / 25280 MB (5% inode=98%); [21:21:09] JohnLewis: ^ [21:21:55] Aware [21:22:07] I think compressOld isn’t good on binlogs [21:22:32] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.13, 5.87, 6.77 [21:23:56] ~50G is binlogs [21:25:33] RECOVERY - db11 Disk Space on db11 is OK: DISK OK - free space: / 60320 MB (13% inode=97%); [21:26:49] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 52717 MB (11% inode=98%); [21:27:15] RECOVERY - db13 Disk Space on db13 is OK: DISK OK - free space: / 74351 MB (16% inode=98%); [21:29:44] [02puppet] 07Universal-Omega opened pull request 03#2044: prometheus-es-exporter: set job_name to elasticsearch - 13https://git.io/JKEf4 [22:03:31] JohnLewis: it will trigger a lot of transactions so no it won't be [22:08:10] !log cp12: sudo ufw allow out 80/tcp [22:08:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:08:38] !log revert that change [22:08:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:10:04] paladox: I made team=mediawiki go to irc too [22:10:12] So let's watch alerts roll [22:10:18] ok [22:10:33] CosmicAlpha: can you remember how you triggered a test? [22:17:45] Spookreeeno: I just modified the alert to set to something lower then it was then reverted change afterwards. [22:19:05] CosmicAlpha: ah! [22:19:11] Mind quickly doing it [22:19:29] Sure. [22:21:29] Spookreeeno: done, should alert in ~2 minutes. [22:25:19] Spookreeeno: it fired to email. [22:29:38] paladox: ^ [22:29:42] no irc? [22:29:59] hmm, i'm not sure [22:30:01] Spookreeeno: it might be because it won't fire to both. I'm not sure. [22:30:39] i see this [22:30:40] Oct 16 22:24:30 mon2 python3[19674]: 51.195.236.249 - - [16/Oct/2021 22:24:30] "POST / HTTP/1.1" 404 - [22:30:43] CosmicAlpha: it should [22:30:51] paladox: 404 ain't good [22:31:01] it means it didn't pick up the route [22:31:50] Spookreeeno: do you have fake json i can try [22:31:53] i found the error [22:32:13] oh? [22:32:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JKEkd [22:32:48] [02miraheze/puppet] 07paladox 036d26894 - grafana-webhook: Fix typo [22:32:50] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [22:32:51] [02puppet] 07paladox opened pull request 03#2045: grafana-webhook: Fix typo - 13https://git.io/JKEkb [22:33:02] calling that api at least does 500 now [22:33:16] yeah that's it [22:34:01] paladox: does that look like it'll work [22:34:47] [02puppet] 07paladox closed pull request 03#2045: grafana-webhook: Fix typo - 13https://git.io/JKEkb [22:34:48] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JKEIe [22:34:50] [02miraheze/puppet] 07paladox 036d26894 - grafana-webhook: Fix typo [22:34:51] [02miraheze/puppet] 07paladox 03d43fe09 - Merge pull request #2045 from miraheze/paladox-patch-1 [22:34:53] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [22:34:54] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [22:35:32] test [22:35:38] it works [22:35:42] \o/ [22:35:56] CosmicAlpha: retrigger a test pls [22:36:58] Spookreeeno: done. ~2 minutes it will alert [22:38:12] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.16, 3.96, 3.29 [22:40:00] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1 [22:40:01] [url] Grafana | grafana.miraheze.org [22:40:22] CosmicAlpha: \o/, mark as resolved? [22:40:51] i can add the !_sre ping if they not spammy [22:40:59] Spookreeeno: sure! [22:41:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.23, 3.21, 3.11 [22:41:35] paladox: the es-exporter is available in Grafana now, but it doesn't seem to be able to connect to graylog2.miraheze.org:9206 properly. I'm trying `up{instance="graylog2.miraheze.org:9206",job="es_exporter"}` in explore, which results in 0, and I thought it should be 1 if its properly working, could be totally wrong though. I'm probably doing something else wrong though, or missed something in PR. [22:43:46] hmm, prometheus-es-exporter isn't running [22:43:50] it says failed to start [22:44:13] https://www.irccloud.com/pastebin/8rlOfc2Z/ [22:45:00] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1 [22:45:01] [url] Grafana | grafana.miraheze.org [22:45:02] CosmicAlpha: ^ [22:46:02] paladox: oh I know why. [22:46:05] Thanks! [22:51:29] [02puppet] 07Universal-Omega opened pull request 03#2046: Fix syntax error - 13https://git.io/JKELf [22:52:27] [02puppet] 07Universal-Omega synchronize pull request 03#2046: Fix syntax error - 13https://git.io/JKELf [22:53:33] paladox: ^ will probably work. [22:53:42] Was wrong indentation. [22:53:49] Resulting in syntax error [22:54:23] still fails [22:54:26] https://www.irccloud.com/pastebin/a3r2UIGI/ [22:54:59] paladox: oh same thing on the other configuration file. [22:56:03] [02puppet] 07Universal-Omega synchronize pull request 03#2046: Fix syntax error - 13https://git.io/JKELf [22:56:46] [02puppet] 07Universal-Omega edited pull request 03#2046: prometheus-es-exporter: fix indentation syntax errors in config files - 13https://git.io/JKELf [22:56:59] paladox: ^ [22:57:39] now it starts [22:57:50] [02puppet] 07paladox closed pull request 03#2046: prometheus-es-exporter: fix indentation syntax errors in config files - 13https://git.io/JKELf [22:57:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JKELu [22:57:53] [02miraheze/puppet] 07Universal-Omega 033c328a0 - prometheus-es-exporter: fix indentation syntax errors in config files (#2046) [22:58:48] Thanks paladox! Also do you mind merging https://github.com/miraheze/puppet/pull/2044 to name the job similarly to all the other ones? [22:58:49] [url] prometheus-es-exporter: set job_name to elasticsearch by Universal-Omega · Pull Request #2044 · miraheze/puppet · GitHub | github.com [22:59:09] [02puppet] 07paladox closed pull request 03#2044: prometheus-es-exporter: set job_name to elasticsearch - 13https://git.io/JKEf4 [22:59:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JKELw [22:59:12] [02miraheze/puppet] 07Universal-Omega 032075807 - prometheus-es-exporter: set job_name to elasticsearch (#2044) [22:59:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1 [22:59:31] [url] Grafana | grafana.miraheze.org [22:59:34] CosmicAlpha: up{instance="graylog2.miraheze.org:9206",job="es_exporter"} works now [22:59:41] Thanks! [23:00:43] It works! [23:03:44] SRE, 502 bad gateway on `testwiki` [23:04:08] seems to be resolved / have been transitory [23:04:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1 [23:04:31] [url] Grafana | grafana.miraheze.org [23:06:00] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1 [23:06:01] [url] Grafana | grafana.miraheze.org [23:06:38] PROBLEM - mw9 APT on mw9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:07:16] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.66, 9.33, 6.13 [23:08:43] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.58, 21.24, 17.95 [23:09:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.77, 7.20, 5.68 [23:09:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 6.26, 8.49, 6.37 [23:09:43] RECOVERY - mw9 APT on mw9 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [23:10:17] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.59, 6.48, 5.59 [23:11:00] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1 [23:11:01] [url] Grafana | grafana.miraheze.org [23:11:35] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 14.60, 18.21, 17.32 [23:12:07] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 2.69, 5.25, 5.17 [23:12:20] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 2.81, 6.14, 5.84 [23:18:43] paladox, CosmicAlpha, JohnLewis, or Agent, any idea why the MassMessage extension is tripping the AbuseFilter (https://test.miraheze.org/wiki/Special:Log/massmessage)? I've checked the abuselog locally and globally and may not seeing any abuse log entries for this 🤔 [23:18:45] [url] Mass message log - TestWiki | test.miraheze.org [23:20:46] Nope :( [23:20:57] I seem to have found this from October 9th (https://publictestwiki.com/w/index.php?title=Special:AbuseLog&wpSearchFilter=51), but not from the 16th and curious they don't show in the AbuseLog :( [23:20:58] [url] Abuse filter log - TestWiki | publictestwiki.com [23:21:32] I don't know either. That's weird. [23:21:39] it's disabled anyway [23:21:40] yeah [23:25:40] now they show up [23:25:41] wtf [23:25:44] that is bizarre [23:26:38] oh those were from the 9th [23:26:41] I'm blind lol [23:26:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JKEtp [23:26:49] [02miraheze/mw-config] 07Universal-Omega 033fb8bd6 - Enable `api-request` logging [23:26:51] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [23:26:59] [02mw-config] 07Universal-Omega opened pull request 03#4156: Enable `api-request` logging - 13https://git.io/JKEtj [23:27:58] miraheze/mw-config - Universal-Omega the build passed. [23:28:00] paladox: ^ does that look correct to you? I've never set something to use a buffer like it mentioned was necessary so not sure if I did that correctly. [23:29:12] apparently no buffer logger is setup [23:29:16] oh [23:29:17] nvm [23:29:22] yeh, that looks fine i guess [23:30:12] [02mw-config] 07Universal-Omega closed pull request 03#4156: Enable `api-request` logging - 13https://git.io/JKEtj [23:30:14] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [23:30:15] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [23:30:17] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JKEqI [23:30:18] [02miraheze/mw-config] 07Universal-Omega 0359d959a - Enable `api-request` logging (#4156) [23:31:10] miraheze/mw-config - Universal-Omega the build passed. [23:31:18] Thanks! [23:33:31] !log [@test3] starting deploy of {'config': True} to skip [23:33:32] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 1s [23:33:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:33:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:36:32] !log [@mw11] starting deploy of {'config': True} to all [23:36:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:37:05] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 32s [23:37:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:37:28] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.69, 9.44, 6.82 [23:37:37] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.01, 22.66, 18.51 [23:39:06] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.64, 7.06, 6.06 [23:39:53] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.72, 20.63, 18.50 [23:40:36] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.59, 9.80, 7.36 [23:40:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 3.88, 6.82, 6.29 [23:40:47] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.45, 22.53, 19.29 [23:40:57] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 5.85, 8.65, 7.08 [23:42:01] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.06, 5.96, 5.83 [23:42:44] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.42, 19.04, 18.28 [23:43:26] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.50, 7.39, 6.84 [23:43:30] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.42, 5.27, 5.78 [23:43:37] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.63, 19.32, 18.56 [23:43:48] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.32, 6.22, 6.40 [23:46:13] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.85, 5.88, 6.36