[00:02:36] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown) [00:03:13] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 4.78, 3.17, 2.17 [00:04:06] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.80, 1.68, 1.34 [00:06:02] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.46, 1.61, 1.35 [00:06:19] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.23, 6.27, 4.51 [00:06:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.87, 19.85, 15.10 [00:06:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.28, 6.23, 4.34 [00:07:09] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.67, 3.36, 2.53 [00:08:17] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.14, 6.00, 4.63 [00:08:20] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.48, 18.99, 15.40 [00:08:27] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.69, 5.32, 4.24 [00:12:51] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=rainworldwiki /home/universalomega/rainworld_gamepedia_pages_full.xml --username-prefix=wikia:rainworld --report=1 (END - exit=0) [00:12:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:13:55] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.31, 2.00, 1.69 [00:14:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.47, 20.83, 17.29 [00:15:52] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.72, 1.54, 1.56 [00:18:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.45, 19.74, 17.69 [00:25:59] !log UPDATE image SET img_minor_mime = 'octet-stream' WHERE img_name = 'SDL_Retro_(30_Cores).opk'; on funkywiki [00:26:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:26:28] !log UPDATE image SET img_major_mime = 'application' WHERE img_name = 'SDL_Retro_(30_Cores).opk'; on funkywiki [00:26:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:31] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:22:23] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.38, 3.41, 2.95 [01:26:22] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.09, 2.98, 2.89 [02:31:26] PROBLEM - diamowiki.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:38:19] RECOVERY - diamowiki.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'diamowiki.ga' will expire on Sat 18 Sep 2021 14:34:47 GMT +0000. [03:06:45] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRb2d [03:06:46] [02miraheze/mw-config] 07paladox 0388240bc - Add cp14 to wgCdnServers [03:07:39] miraheze/mw-config - paladox the build passed. [03:15:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRbVX [03:15:50] [02miraheze/puppet] 07paladox 0335c9344 - redis: Log redis to /var/log/redis/redis-server.log [03:17:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JRbVx [03:17:19] [02miraheze/puppet] 07paladox 0360c064d - redis: Save every 900s if one change is made rather than 60s [03:17:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3-1 [+0/-0/±1] 13https://git.io/JRbVp [03:17:22] [02miraheze/puppet] 07paladox 037a05f45 - redis: Save every 900s if one change is made rather than 60s [03:17:23] [02puppet] 07paladox created branch 03paladox-patch-3-1 - 13https://git.io/vbiAS [03:17:25] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbiAS [03:17:26] [02puppet] 07paladox opened pull request 03#1881: redis: Save every 900s if one change is made rather than 60s - 13https://git.io/JRbVj [03:22:27] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.02, 1.68, 1.45 [03:24:23] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.05, 1.46, 1.40 [03:38:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.58, 3.60, 3.15 [03:40:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.97, 3.33, 3.10 [04:02:22] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.90, 3.29, 2.98 [04:04:01] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.76, 1.77, 1.43 [04:04:23] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 3.54, 3.11 [04:05:57] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.62, 1.89, 1.50 [04:06:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.81, 3.59, 3.17 [04:07:53] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.82, 1.44, 1.38 [04:10:22] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.55, 3.28, 3.16 [04:43:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.18, 3.01 [04:45:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.96, 3.02, 2.97 [04:57:14] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.18, 1.94, 1.45 [04:59:14] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.41, 1.63, 1.39 [05:13:07] PROBLEM - ping6 on ns1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 94.50 ms [05:15:07] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 91.36 ms [05:24:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRbpS [05:24:42] [02miraheze/puppet] 07paladox 038f46eed - Update matomo to 4.4.1 [05:28:38] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.52, 3.19 [05:30:35] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.06, 3.10, 3.08 [06:13:08] RECOVERY - test4 APT on test4 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:18:43] RECOVERY - mwtask1 APT on mwtask1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:24:10] RECOVERY - mw8 APT on mw8 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:29:39] RECOVERY - mw9 APT on mw9 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:39:55] RECOVERY - test3 APT on test3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:51:04] RECOVERY - mw10 APT on mw10 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:51:14] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [06:54:47] RECOVERY - mw13 APT on mw13 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:55:46] RECOVERY - mw11 APT on mw11 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:58:13] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns2.dreamhost.com.', 'ns1.dreamhost.com.', 'ns3.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [07:26:12] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [07:33:12] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns3.dreamhost.com.', 'ns1.dreamhost.com.', 'ns2.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [08:01:12] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [08:08:13] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns3.dreamhost.com.', 'ns1.dreamhost.com.', 'ns2.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [08:29:11] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [08:36:12] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns3.dreamhost.com.', 'ns1.dreamhost.com.', 'ns2.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [09:18:15] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [09:25:13] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns3.dreamhost.com.', 'ns1.dreamhost.com.', 'ns2.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [09:39:13] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [09:46:12] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns3.dreamhost.com.', 'ns1.dreamhost.com.', 'ns2.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [09:47:01] PROBLEM - diamowiki.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:53:55] RECOVERY - diamowiki.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'diamowiki.ga' will expire on Sat 18 Sep 2021 14:34:47 GMT +0000. [10:00:12] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [10:07:11] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns3.dreamhost.com.', 'ns1.dreamhost.com.', 'ns2.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [10:32:08] [02puppet] 07JohnFLewis commented on pull request 03#1881: redis: Save every 900s if one change is made rather than 60s - 13https://git.io/JRAY8 [10:32:43] [02puppet] 07JohnFLewis commented on commit 0335c93444605c62495bb99bda80855a5e8254e066 - 13https://git.io/JRAYo [10:39:36] paladox: ^ minor points on both [10:49:10] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [10:56:13] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns3.dreamhost.com.', 'ns1.dreamhost.com.', 'ns2.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [11:10:13] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [11:17:13] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns3.dreamhost.com.', 'ns1.dreamhost.com.', 'ns2.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [11:24:14] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [11:31:11] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns3.dreamhost.com.', 'ns1.dreamhost.com.', 'ns2.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [11:38:27] PROBLEM - diamowiki.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:45:20] RECOVERY - diamowiki.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'diamowiki.ga' will expire on Sat 18 Sep 2021 14:34:47 GMT +0000. [11:54:32] PROBLEM - diamowiki.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:06:12] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [12:08:18] RECOVERY - diamowiki.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'diamowiki.ga' will expire on Sat 18 Sep 2021 14:34:47 GMT +0000. [12:13:11] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns3.dreamhost.com.', 'ns1.dreamhost.com.', 'ns2.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [12:41:10] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [12:48:12] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns3.dreamhost.com.', 'ns2.dreamhost.com.', 'ns1.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [13:09:12] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [13:16:13] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns3.dreamhost.com.', 'ns1.dreamhost.com.', 'ns2.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [13:23:14] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [13:30:13] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns3.dreamhost.com.', 'ns1.dreamhost.com.', 'ns2.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [14:05:47] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.44, 6.10, 4.94 [14:07:47] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.20, 5.69, 4.93 [14:19:12] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [14:21:48] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.45, 7.08, 5.92 [14:23:46] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.63, 7.02, 6.04 [14:25:45] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.66, 6.65, 6.01 [14:26:12] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns3.dreamhost.com.', 'ns1.dreamhost.com.', 'ns2.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [14:29:29] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.91, 6.56, 5.64 [14:29:45] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.49, 6.99, 6.31 [14:31:28] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.84, 6.20, 5.61 [14:31:46] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.92, 6.34, 6.16 [14:34:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.74, 3.25 [14:35:12] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.16, 7.78, 6.38 [14:35:34] PROBLEM - diamowiki.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:36:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.00, 3.30, 3.14 [14:37:08] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.09, 6.62, 6.11 [14:40:12] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [14:47:12] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns3.dreamhost.com.', 'ns2.dreamhost.com.', 'ns1.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [15:03:07] RECOVERY - diamowiki.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'diamowiki.ga' will expire on Sat 18 Sep 2021 14:34:47 GMT +0000. [15:14:52] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JRxE0 [15:14:54] [02miraheze/MirahezeMagic] 07Universal-Omega 03e95048d - Add $wgReCaptchaMinimumScore config [15:14:55] [02MirahezeMagic] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/fQRGX [15:15:01] [02MirahezeMagic] 07Universal-Omega opened pull request 03#282: Add $wgReCaptchaMinimumScore config - 13https://git.io/JRxEz [15:16:00] miraheze/MirahezeMagic - Universal-Omega the build passed. [15:16:30] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JRxEd [15:16:32] [02miraheze/MirahezeMagic] 07Universal-Omega 035fccf9b - Update ReCaptchaNoCaptcha.php [15:16:33] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#282: Add $wgReCaptchaMinimumScore config - 13https://git.io/JRxEz [15:16:59] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JRxuU [15:17:00] [02miraheze/MirahezeMagic] 07Universal-Omega 0352789d5 - Update extension.json [15:17:02] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#282: Add $wgReCaptchaMinimumScore config - 13https://git.io/JRxEz [15:17:25] miraheze/MirahezeMagic - Universal-Omega the build passed. [15:17:51] miraheze/MirahezeMagic - Universal-Omega the build passed. [15:27:16] PROBLEM - diamowiki.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:30:34] [02MirahezeMagic] 07Universal-Omega closed pull request 03#282: Add $wgReCaptchaMinimumScore config - 13https://git.io/JRxEz [15:30:36] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JRxaB [15:30:37] [02miraheze/MirahezeMagic] 07Universal-Omega 037f95a7a - Add $wgReCaptchaMinimumScore config (#282) [15:30:39] [02MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/fQRGX [15:30:40] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [15:31:32] miraheze/MirahezeMagic - Universal-Omega the build passed. [15:41:03] RECOVERY - diamowiki.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'diamowiki.ga' will expire on Sat 18 Sep 2021 14:34:47 GMT +0000. [16:07:01] PROBLEM - phab2 Disk Space on phab2 is CRITICAL: DISK CRITICAL - free space: / 1377 MB (5% inode=87%); [16:11:11] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [16:12:22] JohnLewis, paladox: phab2 [16:14:49] Could be the surge in imports [16:15:02] https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=phab2.miraheze.org&var-port=9100&from=now-90d&to=now-1m&viewPanel=152 [16:15:03] [url] Grafana | grafana.miraheze.org [16:15:20] paladox: I can see the graph [16:15:28] In 2 months it went up 10% [16:15:30] Guess we can request a disk expansion though someone uploaded large files [16:15:36] Last week it's gone up 10% [16:15:41] It'll be imports [16:15:52] Normally we get 2-3 and delete when done [16:16:03] But fandom are mass purging [16:16:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.49, 3.10, 2.88 [16:17:14] Found a lot of MB files in https://phabricator.miraheze.org/file/query/all/?after=1482673 [16:17:15] [url] ⇪ Query: All | phabricator.miraheze.org [16:18:13] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns3.dreamhost.com.', 'ns2.dreamhost.com.', 'ns1.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [16:20:39] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.44, 2.70, 2.76 [16:31:58] [02puppet] 07paladox closed pull request 03#1881: redis: Save every 900s if one change is made rather than 60s - 13https://git.io/JRbVj [16:32:01] [02puppet] 07paladox deleted branch 03paladox-patch-3-1 - 13https://git.io/vbiAS [16:32:02] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3-1 [16:39:11] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [16:46:13] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns3.dreamhost.com.', 'ns2.dreamhost.com.', 'ns1.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [16:53:13] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [17:00:13] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns2.dreamhost.com.', 'ns1.dreamhost.com.', 'ns3.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [17:10:34] PROBLEM - diamowiki.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:27] RECOVERY - diamowiki.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'diamowiki.ga' will expire on Sat 18 Sep 2021 14:34:47 GMT +0000. [17:21:11] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [17:22:23] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.41, 3.48, 2.75 [17:22:33] RhinosF1: do you know if any of those files were used by you or reception to import? [17:24:23] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.18, 3.31, 2.78 [17:28:12] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns2.dreamhost.com.', 'ns1.dreamhost.com.', 'ns3.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [17:32:22] paladox: Reception might have but UO has started some today [17:32:35] Ok, thanks! [17:32:37] If on the ticket for the import it says done then delete it [17:32:53] Best is probably to match file to ticket [17:33:13] https://phabricator.miraheze.org/T7799 [17:33:14] [url] ⚓ T7799 Import XIP Files to Polandball Wiki | phabricator.miraheze.org [17:34:00] seems that it hasn't been done but closed as declined until the owner makes the request [17:35:12] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [17:35:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.31, 5.75, 4.34 [17:35:28] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.70, 6.07, 4.57 [17:37:23] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.20, 5.30, 4.47 [17:37:25] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.26, 5.20, 4.32 [17:37:41] paladox: could they be moved to mwtask maybe? [17:38:11] RhinosF1: the files on phab? Well the user should use an external cloud provider ideally. [17:39:10] Ok [17:39:18] What can we do now though? [17:42:12] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns2.dreamhost.com.', 'ns1.dreamhost.com.', 'ns3.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [17:53:15] I guess we can just delete the files and recommend they use a cloud provider [17:54:49] RhinosF1: ^ [17:55:02] Ok [17:55:11] !log deleting some files on phabricator to clean up the storage [17:55:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:56:13] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [18:03:11] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns3.dreamhost.com.', 'ns2.dreamhost.com.', 'ns1.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [18:04:56] PROBLEM - phab2 Disk Space on phab2 is WARNING: DISK WARNING - free space: / 1709 MB (7% inode=87%); [18:08:58] RECOVERY - phab2 Disk Space on phab2 is OK: DISK OK - free space: / 3803 MB (16% inode=87%); [18:13:48] RhinosF1: resolved [18:17:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JRp36 [18:17:37] [02miraheze/puppet] 07paladox 038bc8935 - phabricator: Limit how much you can upload to 10MB [18:17:39] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [18:17:40] [02puppet] 07paladox opened pull request 03#1882: phabricator: Limit how much you can upload to 10MB - 13https://git.io/JRp3i [18:18:13] [02puppet] 07paladox closed pull request 03#1882: phabricator: Limit how much you can upload to 10MB - 13https://git.io/JRp3i [18:18:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRp3S [18:18:16] [02miraheze/puppet] 07paladox 03ec63c39 - phabricator: Limit how much you can upload to 10MB (#1882) [18:18:17] RhinosF1: ^ [18:18:17] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [18:18:19] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [18:23:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRpsr [18:23:11] [02miraheze/puppet] 07paladox 0389aac14 - phabricator: Restart phd on failure [18:23:15] PROBLEM - phab2 Puppet on phab2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[phd] [18:25:14] RECOVERY - phab2 Puppet on phab2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:27:05] Ok paladox [18:27:18] Maybe make sure the forms for tasks say use a cloud provider [18:33:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRpZo [18:33:18] [02miraheze/puppet] 07paladox 0355ff6d5 - Revert "redis: Log redis to /var/log/redis/redis-server.log" [18:45:12] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [18:52:11] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns1.dreamhost.com.', 'ns2.dreamhost.com.', 'ns3.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [18:52:48] [02puppet] 07paladox closed pull request 03#1863: puppetdb: Set access-log-config to enable request logging - 13https://git.io/JRocr [18:52:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRpWE [18:52:51] [02miraheze/puppet] 07paladox 037c72190 - puppetdb: Set access-log-config to enable request logging (#1863) [18:52:52] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [18:52:54] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [19:00:43] PROBLEM - diamowiki.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:04:55] I noticed that pollandballwiki's Fandom wiki has over 1 million revisions which will be imported whenever the import to their Miraheze wiki concludes. Wouldn't it be advisable to turn on Compressed Revisions for that wiki to shrink their probably massive database? [19:07:37] RECOVERY - diamowiki.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'diamowiki.ga' will expire on Sat 18 Sep 2021 14:34:47 GMT +0000. [19:08:23] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/mnt/mediawiki-static] [19:09:38] Agent: yes [19:16:48] PROBLEM - diamowiki.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:20:14] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [19:20:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.55, 3.38, 2.93 [19:22:50] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.27, 3.29, 2.95 [19:23:41] RECOVERY - diamowiki.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'diamowiki.ga' will expire on Sat 18 Sep 2021 14:34:47 GMT +0000. [19:27:11] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns1.dreamhost.com.', 'ns3.dreamhost.com.', 'ns2.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [19:33:45] paladox: is mw10 ok? [19:33:57] First I've seen of that error in a while [19:34:12] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [19:34:22] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [19:34:31] Ah there it's ok [19:41:14] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns1.dreamhost.com.', 'ns2.dreamhost.com.', 'ns3.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [19:48:12] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [19:55:12] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns1.dreamhost.com.', 'ns2.dreamhost.com.', 'ns3.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [20:07:23] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.17, 6.18, 4.90 [20:09:23] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.99, 5.49, 4.79 [20:44:13] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [20:51:11] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns1.dreamhost.com.', 'ns3.dreamhost.com.', 'ns2.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [20:58:13] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [21:12:12] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns1.dreamhost.com.', 'ns2.dreamhost.com.', 'ns3.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [21:33:11] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [21:40:13] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns3.dreamhost.com.', 'ns2.dreamhost.com.', 'ns1.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [22:43:11] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [22:50:15] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns1.dreamhost.com.', 'ns2.dreamhost.com.', 'ns3.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [22:57:11] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [23:11:12] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns3.dreamhost.com.', 'ns2.dreamhost.com.', 'ns1.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'} [23:18:13] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query. [23:25:11] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns1.dreamhost.com.', 'ns3.dreamhost.com.', 'ns2.dreamhost.com.'], 'CNAME': 'wiki.seamly.net.'}