[00:02:35] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 35.92, 22.67, 17.21 [00:02:52] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.96, 1.43, 1.23 [00:03:57] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:04:25] RECOVERY - cp12 Disk Space on cp12 is OK: DISK OK - free space: / 7475 MB (19% inode=96%); [00:04:36] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.64, 22.43, 17.79 [00:04:37] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 13.41, 9.23, 6.91 [00:04:48] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.32, 1.37, 1.23 [00:04:54] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 14.18, 9.18, 6.87 [00:06:04] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 51.222.25.132/cpweb [00:06:34] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 31.17, 25.72, 19.55 [00:10:08] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:14:14] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.73, 5.41, 3.85 [00:14:38] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.51, 7.94, 7.46 [00:16:14] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.37, 4.59, 3.73 [00:16:35] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.19, 8.57, 7.76 [00:24:19] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.83, 7.54, 7.76 [00:24:36] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.23, 6.93, 7.71 [00:30:04] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.56, 7.92, 7.86 [00:30:31] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 15.79, 21.70, 23.78 [00:32:00] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:32:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.46, 7.25, 7.61 [00:32:30] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.72, 22.24, 23.67 [00:34:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 14.85, 9.98, 8.55 [00:34:32] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.72, 7.84, 7.59 [00:36:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.93, 21.84, 23.32 [00:36:31] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.71, 7.95, 7.68 [00:38:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.16, 7.58, 7.92 [00:38:29] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.14, 8.25, 7.82 [00:40:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 7.08, 7.90, 8.02 [00:40:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.50, 7.99, 7.78 [00:42:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.41, 7.39, 7.83 [00:44:25] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 7.69, 8.20, 7.92 [00:46:24] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.99, 7.84, 7.82 [00:52:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.04, 7.17, 7.45 [00:53:17] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 2.00, 1.69, 1.40 [00:54:22] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.59, 7.05, 7.38 [00:55:13] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.17, 1.51, 1.37 [00:55:36] [02puppet] 07Universal-Omega opened pull request 03#1855: Use --quiet for LocalisationUpdate cron - 13https://git.io/JRGWx [00:56:06] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.62, 17.69, 19.96 [01:02:02] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.84, 5.82, 6.71 [01:02:18] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.41, 5.88, 6.66 [01:06:08] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.70, 6.88, 7.00 [01:06:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.22, 7.22, 7.13 [01:08:03] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.90, 6.01, 6.66 [01:12:22] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.24, 5.68, 6.48 [01:35:12] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.57, 6.19, 5.76 [01:37:06] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.04, 5.80, 5.68 [03:04:50] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.34, 6.69, 5.81 [03:06:46] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.08, 5.36, 5.43 [03:17:45] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/python3/di [03:17:45] ckages/dns/resolver.py", line 992, in query timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 799, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 30.017160892486572 seconds [03:31:29] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JRGdA [03:31:31] [02miraheze/RemovePII] 07Universal-Omega 03b302a82 - Allow underscores to be used. [03:31:32] [02RemovePII] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/Jsfpj [03:31:34] [02RemovePII] 07Universal-Omega opened pull request 03#17: Allow underscores to be used. - 13https://git.io/JRGdx [03:31:49] [02RemovePII] 07Universal-Omega edited pull request 03#17: Allow underscores to be used with the RemovePII action - 13https://git.io/JRGdx [03:32:25] miraheze/RemovePII - Universal-Omega the build has errored. [03:34:59] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JRGFK [03:35:00] [02miraheze/RemovePII] 07Universal-Omega 0330b1bdc - Update SpecialRemovePII.php [03:35:02] [02RemovePII] 07Universal-Omega synchronize pull request 03#17: Allow underscores to be used with the RemovePII action - 13https://git.io/JRGdx [03:35:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.31, 6.82, 5.35 [03:35:50] miraheze/RemovePII - Universal-Omega the build passed. [03:37:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.92, 6.23, 5.30 [03:39:19] RECOVERY - diamowiki.ga - reverse DNS on sslhost is OK: SSL OK - diamowiki.ga reverse DNS resolves to cp13.miraheze.org - CNAME OK [04:05:48] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in check_records nameserversans = dns_resolver.query(root_domain, 'NS') File "/usr/lib/python3/d [04:05:48] ackages/dns/resolver.py", line 992, in query timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 799, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 30.001267433166504 seconds [04:12:48] RECOVERY - diamowiki.ga - reverse DNS on sslhost is OK: SSL OK - diamowiki.ga reverse DNS resolves to cp13.miraheze.org - CNAME OK [04:20:51] What's going on with diamowiki.ga today? [04:21:59] It doesn't even resolve anymore. [04:34:29] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.query(hostname, [04:34:29] 0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 992, in query timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 799, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 30.000279664993286 seconds [04:49:44] RECOVERY - diamowiki.ga - reverse DNS on sslhost is OK: SSL OK - diamowiki.ga reverse DNS resolves to cp13.miraheze.org - CNAME OK [08:48:32] !log manually runJobs.php on metawiki [08:48:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:17:56] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/python3/di [10:17:56] ckages/dns/resolver.py", line 992, in query timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 799, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 30.003543615341187 seconds [10:23:11] JohnLewis: replied on task [10:25:08] RECOVERY - diamowiki.ga - reverse DNS on sslhost is OK: SSL OK - diamowiki.ga reverse DNS resolves to cp13.miraheze.org - CNAME OK [10:28:23] Replied, but either way it's not your decision and the comments you're sharing are not unexpected given your lack of respect for other volunteers in the past - particularly towards myself and Paladox - which contributed to us leaving in the past. If the project does not wish to accept valuable support, that is fine but with no attempts to solve such a problem, rejecting help seems counterintuitive and self-destructive. [10:33:44] I intend to try and work with Adamw to see if we can actually if he does get elected put something in place. [10:34:02] That's what I thinks gonna be valuable to Miraheze now. [10:34:41] Both the people I've spoken to following collapse of said relationships have provided ways to work round it. [10:35:00] Other people have provided the solutions then, not yourself. [10:35:25] Following talks with me [10:36:31] And what was done in response? [10:37:18] We've had long conversations and it's no secret how we can restore those relationships [10:37:36] "can" suggests nothing has been done. [10:37:50] Considering these problems were months and years ago, that sounds problematic. [10:37:53] because those above me sat around and did nothing rather than listen [10:38:02] And now you know how I feel. [10:38:18] John? Haven't seen you in like a month and a half [10:38:19] But above you is simply the board and Ferran [10:38:35] Ferran might be leaving soon [10:38:36] Exactly. One of which was incredibly obstructive and ineffective [10:38:47] But the board isn't massively chasing to my knowledge [10:39:00] Chasing what? [10:39:06] Changing* [10:39:21] Not sure why that got auto corrected [10:40:05] The Board aren't line manages to anyone but the DSRE. A problem was identified and a solution to change that is now in place. I never really liaised with the Board because the DSRE was my manager and the one who I should go to with all problems. [10:40:18] I went above to the Board, and now things are happening with a new DSRE [10:40:43] So you're saying that Southparkfan was obstructive and ineffective [10:41:13] Yes, because of his activity levels. He knows this, it was never a surprise as I spoke to him many times about it [10:41:29] His activity hasn't been great but he's stepped down [10:41:40] Yes [10:42:04] I don't believe that magically you'll have the opportunity though [10:42:09] Because of conversation and problems raised by myself to the Board, he stepped down after that [10:42:14] Because you've said yourself you tried [10:42:20] Opportunity for what? [10:42:27] Things like Clinic duty were yours [10:42:32] And failed miserably [10:42:45] Because there were no volunteers to fill it consistently [10:42:58] The system worked as response times and resolution times went down massively [10:43:01] Opportunity to implement anything because I don't think you have any ideas that will work [10:43:03] Task numbers dropped. [10:43:32] The task but burden on members of SRE to review situations they had no knowledge of [10:44:02] And if I don't, at the very least I can support and ease pressure on those who have such ideas by reducing their requirements to do things [10:44:17] But then I go back to SSLRequest [10:44:23] You were the one blocking it [10:44:30] Void seems to think he has ideas, but he's single handledly managing a large and critical part of the infra he doesn't understand fully [10:44:37] How was I? [10:45:07] I could find the PR where you said you'd rather wait because you might change how certificates are issued to them being sent everywhere [10:45:25] You didn't provide any idea when it was questioned of how it would work [10:45:31] Void is new blood [10:45:50] I raised a valid counter proposal, and you said I'm blocking your work? [10:45:52] And so far he's caught a few things that were issues for months [10:46:04] Okay? [10:46:08] Did you ever identify how it would work [10:46:19] Issues that could have been caught by infra [10:46:24] If they'd have simply looked [10:46:41] Who couldn't, because they were dealing consistently with your rude behaviour - demanding others do your work for you. [10:46:43] The packets being dropped causing depools for example [10:46:50] Paladox raised this numerous times. [10:47:08] I've asked people for help. I've said before if you have higher priority infra work then do it. [10:47:13] I'm glad @Void stepped up to the plate in recent weeks, so it's not a total loss [10:47:30] But they couldn't, because you kept demanding they do it now [10:47:46] Who? Me, or RhinosF1? [10:47:55] Rhinos [10:47:59] @DarkMatterMan4500: obviously me [10:48:19] Yeah, that was a stupid question on my end. Sorry that I asked. [10:48:45] If you are seriously telling me Void has all the time and resources to complete 3 roles, and additional help is not wanted by SRE, then this conversation should end as it serves no point. [10:48:50] He doesn't [10:48:59] I've never said we don't need additional help [10:49:07] I've said I don't think you're the help we need [10:49:31] Go find the help you need then, and tell others not to ask me come here and support the project then [10:49:45] The environment though with Reception and Void is far better [10:49:46] I'm here, and filed that request, because I was asked to. [10:49:54] I don't know who asked you too [10:50:09] It's not a suggestion that's come from me and I doubt Universal_Omega [10:50:17] Okay, it sounds like you 2 are fighting like cats and dogs, but shouldn't you all be working together? [10:50:58] DarkMatterMan4500: It's hard to work together in an environment where you are unwelcome [10:50:58] No, John left, asked to come back and ive said I don't believe it's best for us. [10:51:29] I know. Even Wikipedia sometimes suffers the same problem. [10:52:17] Generally yes [10:52:26] For Miraheze that's another issue [10:52:39] RhinosF1: you might not think I am useful here, but those who make the decisions seem to disagree with you. Maybe that is why they are the ones making the decisions. [10:53:03] I have no further comments to make as I don't believe this discussion is useful or conducive to the environment. [10:53:22] They are welcome to consult with me and Universal_Omega and here the opinions we've given [12:14:19] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in check_records nameserversans = dns_resolver.query(root_domain, 'NS') File "/usr/lib/python3/d [12:14:20] ackages/dns/resolver.py", line 992, in query timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 799, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 30.00053334236145 seconds [12:21:37] RECOVERY - diamowiki.ga - reverse DNS on sslhost is OK: SSL OK - diamowiki.ga reverse DNS resolves to cp13.miraheze.org - CNAME OK [12:26:41] Urgh [12:39:07] Is good to see John Lewis is back and both RhinosF1 and John are talking about Miraheze 👍 [12:39:27] That was good? [12:44:26] Probably good that I'm back and willing to contribute constructively is the angle they are coming from [13:52:54] Willing to contribute constructively. That is good 😀 👍 [13:55:36] https://phabricator.miraheze.org/T7751 does however suggest there are objections to my willingness, insisting that the project is better off without me. [13:55:37] [url] ⚓ T7751 [Access Request] John for SRE/Infra | phabricator.miraheze.org [14:12:23] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in check_records nameserversans = dns_resolver.query(root_domain, 'NS') File "/usr/lib/python3/d [14:12:23] ackages/dns/resolver.py", line 992, in query timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 799, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 30.001219749450684 seconds [14:34:46] PROBLEM - diamowiki.ga - reverse DNS on sslhost is UNKNOWN: [14:41:56] RECOVERY - diamowiki.ga - reverse DNS on sslhost is OK: SSL OK - diamowiki.ga reverse DNS resolves to cp13.miraheze.org - CNAME OK [15:35:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRcrV [15:35:11] [02miraheze/services] 07MirahezeSSLBot 030a8591f - BOT: Updating services config for wikis [16:51:18] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRcAT [16:51:19] [02miraheze/mw-config] 07Reception123 03076f311 - set wgAbuseFilterEmergencyDisableThreshold to 0.40 per T7753 [16:51:49] Reception123: not the format I said and what about Count [16:52:07] [ 'default' => 0.40 ], [16:52:12] miraheze/mw-config - Reception123 the build passed. [16:52:15] Should be instead of 0.4 [16:52:30] oh there's two arrays? [16:52:34] Yes [16:52:44] The commit I linked also has the count variable [16:52:57] I see, will fix [16:54:00] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-2 [+0/-0/±1] 13https://git.io/JRcAQ [16:54:01] [02miraheze/mw-config] 07Reception123 03ebdd804 - fix previous commit and add count [16:54:03] [02mw-config] 07Reception123 created branch 03Reception123-patch-2 - 13https://git.io/vbvb3 [16:54:04] [02mw-config] 07Reception123 opened pull request 03#4039: fix previous commit and add count - 13https://git.io/JRcA7 [16:54:46] Reception123: missing [ on L98 [16:54:51] miraheze/mw-config - Reception123 the build has errored. [16:55:30] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-2 [+0/-0/±1] 13https://git.io/JRcxY [16:55:30] thanks [16:55:31] [02miraheze/mw-config] 07Reception123 03cd7ea69 - fix [16:55:33] [02mw-config] 07Reception123 synchronize pull request 03#4039: fix previous commit and add count - 13https://git.io/JRcA7 [16:55:52] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.67, 8.25, 6.44 [16:56:02] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.15, 8.22, 6.31 [16:56:14] Reception123: no '' to round numbers [16:56:24] miraheze/mw-config - Reception123 the build passed. [16:56:47] ah true [16:56:50] Just 0.40 and 25 rather than '0.40' and '25' [16:56:58] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-2 [+0/-0/±1] 13https://git.io/JRcx6 [16:57:00] [02miraheze/mw-config] 07Reception123 038d90b00 - Update LocalSettings.php [16:57:01] [02mw-config] 07Reception123 synchronize pull request 03#4039: fix previous commit and add count - 13https://git.io/JRcA7 [16:57:12] yeah, forgot it was like that as it's been a while [16:57:13] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.09, 1.72, 1.23 [16:57:37] Should be good to go [16:57:43] Then please run puppet [16:57:54] miraheze/mw-config - Reception123 the build passed. [16:57:56] already have puppet2 waiting, just waiting on the CI [16:57:59] ok, we're good to go then [16:58:03] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.64, 7.60, 6.33 [16:58:04] [02mw-config] 07Reception123 closed pull request 03#4039: fix previous commit and add count - 13https://git.io/JRcA7 [16:58:05] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRcxj [16:58:07] [02miraheze/mw-config] 07Reception123 035b988f6 - fix previous commit and add count (#4039) [16:58:08] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-2 - 13https://git.io/vbvb3 [16:58:10] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-2 [16:58:55] miraheze/mw-config - Reception123 the build passed. [16:59:13] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.45, 1.49, 1.20 [17:00:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.12, 8.42, 6.78 [17:02:03] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.31, 7.80, 6.77 [17:03:18] !log [@test3] starting deploy of {'config': True} to skip [17:03:19] !log [@test3] finished deploy of {'config': True} to skip - FAIL: [256] [17:03:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:03:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:04:04] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.87, 8.49, 7.14 [17:04:37] Reception123: can you mwdeploy --config --servers=skip on test3? [17:04:51] ok [17:04:52] deploy-mediawiki even [17:04:56] I should alias that [17:05:13] yeah I don't think mwdeploy would work [17:05:18] !log [reception@test3] starting deploy of {'config': True} to skip [17:05:19] !log [reception@test3] finished deploy of {'config': True} to skip - SUCCESS [17:05:21] [02mw-config] 07RhinosF1 commented on commit 035b988f677be65fcacbc269aeb8169fd4a550d56a - 13https://git.io/JRchV [17:05:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:05:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:05:30] Yey [17:06:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.34, 7.47, 6.93 [17:07:49] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.58, 7.76, 7.39 [17:10:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.80, 6.70, 6.74 [17:13:45] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.57, 5.71, 6.59 [17:35:33] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.59, 6.72, 5.97 [17:36:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.94, 7.45, 6.09 [17:37:32] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.41, 6.67, 6.05 [17:38:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.39, 7.52, 6.30 [17:40:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.22, 7.58, 6.45 [17:42:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.25, 7.87, 6.73 [17:44:02] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.32, 10.59, 7.92 [17:45:32] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.49, 7.79, 6.71 [17:59:13] PROBLEM - cp12 Disk Space on cp12 is WARNING: DISK WARNING - free space: / 4234 MB (10% inode=96%); [18:00:33] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.23, 7.46, 6.09 [18:01:31] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-3 [+0/-0/±1] 13https://git.io/JRCm0 [18:01:32] [02miraheze/mw-config] 07RhinosF1 0379ac6f6 - [EMERGENCY] Up threshold even further [18:01:34] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-3 - 13https://git.io/vbvb3 [18:01:35] [02mw-config] 07RhinosF1 opened pull request 03#4040: [EMERGENCY] Up threshold even further - 13https://git.io/JRCmz [18:02:23] miraheze/mw-config - RhinosF1 the build passed. [18:02:33] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.12, 7.06, 6.10 [18:03:55] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.84, 7.24, 6.54 [18:04:09] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.query(hostname, [18:04:09] 0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 992, in query timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 799, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 30.001055479049683 seconds [18:05:26] [02mw-config] 07RhinosF1 closed pull request 03#4040: [EMERGENCY] Up threshold even further - 13https://git.io/JRCmz [18:05:28] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRCY4 [18:05:29] [02miraheze/mw-config] 07RhinosF1 03e9c6a5d - [EMERGENCY] Up threshold even further (#4040) [18:05:31] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-3 [18:05:32] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-3 - 13https://git.io/vbvb3 [18:05:46] Voidwalker: ^ we're still over 40% on that filter [18:06:14] Might be worth forcing puppet [18:06:21] miraheze/mw-config - RhinosF1 the build passed. [18:06:34] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.84, 6.76, 6.23 [18:09:55] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.72, 7.96, 7.03 [18:10:34] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.52, 7.91, 6.78 [18:11:55] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.56, 7.67, 7.03 [18:12:32] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.03, 7.26, 6.69 [18:16:32] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.33, 6.41, 6.48 [18:19:54] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.56, 6.49, 6.71 [18:25:51] RECOVERY - diamowiki.ga - reverse DNS on sslhost is OK: SSL OK - diamowiki.ga reverse DNS resolves to cp13.miraheze.org - CNAME OK [18:26:04] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.43, 6.05, 7.71 [18:27:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.91, 6.02, 7.86 [18:30:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.69, 7.05, 7.66 [18:32:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.74, 6.86, 7.52 [18:33:24] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.55, 6.87, 7.56 [18:33:28] !log [@test3] starting deploy of {'config': True} to skip [18:33:29] !log [@test3] finished deploy of {'config': True} to skip - FAIL: [256] [18:33:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:33:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:34:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.43, 7.77, 7.77 [18:35:24] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.22, 6.94, 7.50 [18:43:18] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.76, 5.76, 6.78 [18:44:23] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/python3/di [18:44:23] ckages/dns/resolver.py", line 992, in query timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 799, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 30.003174543380737 seconds [18:45:57] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.16, 7.25, 6.49 [18:46:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.40, 6.28, 7.79 [18:47:56] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.49, 6.56, 6.33 [18:50:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.83, 6.80, 7.59 [18:51:36] RECOVERY - diamowiki.ga - reverse DNS on sslhost is OK: SSL OK - diamowiki.ga reverse DNS resolves to cp13.miraheze.org - CNAME OK [18:52:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.72, 6.53, 7.41 [18:58:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.74, 5.74, 6.77 [18:58:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://git.io/JRC8q [18:58:43] [02miraheze/mw-config] 07Universal-Omega 0360309e3 - Update LocalSettings.php [18:58:44] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-4 - 13https://git.io/vbvb3 [18:58:56] [02mw-config] 07Universal-Omega opened pull request 03#4041: Add 2 vector configs to ManageWiki - 13https://git.io/JRC83 [18:59:55] miraheze/mw-config - Universal-Omega the build passed. [19:02:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.77, 6.59, 6.90 [19:04:03] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 14.05, 8.94, 7.68 [19:05:11] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.14, 9.33, 7.62 [19:05:43] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://git.io/JRC4P [19:05:45] [02miraheze/mw-config] 07Universal-Omega 03b80ef7c - Update ManageWikiSettings.php [19:05:46] [02mw-config] 07Universal-Omega synchronize pull request 03#4041: Add 2 vector configs to ManageWiki - 13https://git.io/JRC83 [19:06:41] miraheze/mw-config - Universal-Omega the build passed. [19:08:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.45, 7.60, 7.46 [19:09:10] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.93, 7.10, 7.12 [19:13:10] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.97, 5.76, 6.55 [19:16:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.49, 5.91, 6.71 [19:35:59] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.13, 7.15, 6.35 [19:36:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.74, 7.21, 6.42 [19:38:00] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.96, 6.59, 6.23 [19:40:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.18, 6.59, 6.34 [19:57:56] PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 306.96 ms [20:05:59] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.48, 7.01, 6.01 [20:09:59] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.54, 6.33, 5.96 [20:35:59] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.94, 6.88, 5.96 [20:36:56] [02mw-config] 07Universal-Omega closed pull request 03#4041: Add 2 vector configs to ManageWiki - 13https://git.io/JRC83 [20:36:58] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JRCPs [20:36:59] [02miraheze/mw-config] 07Universal-Omega 0326377c0 - Add 2 vector configs to ManageWiki (#4041) [20:37:01] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 - 13https://git.io/vbvb3 [20:37:02] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 [20:37:47] miraheze/mw-config - Universal-Omega the build passed. [20:39:59] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.30, 7.48, 6.38 [20:41:42] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://git.io/JRCPA [20:41:44] [02miraheze/mw-config] 07Universal-Omega 03e341587 - Update ManageWikiNamespaces.php [20:41:45] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-4 - 13https://git.io/vbvb3 [20:42:05] [02mw-config] 07Universal-Omega opened pull request 03#4042: Add `$wgExemptFromUserRobotsControl` to ManageWikiNamespaces - 13https://git.io/JRCXk [20:42:59] miraheze/mw-config - Universal-Omega the build passed. [20:43:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.65, 6.70, 5.98 [20:43:31] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://git.io/JRCXC [20:43:33] [02miraheze/mw-config] 07Universal-Omega 031e1c3c5 - Update LocalSettings.php [20:43:34] [02mw-config] 07Universal-Omega synchronize pull request 03#4042: Add `$wgExemptFromUserRobotsControl` to ManageWikiNamespaces - 13https://git.io/JRCXk [20:44:20] miraheze/mw-config - Universal-Omega the build passed. [20:44:58] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.18, 7.19, 6.24 [20:45:40] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://git.io/JRCX7 [20:45:41] [02miraheze/mw-config] 07Universal-Omega 034a3001f - Update ManageWikiNamespaces.php [20:45:43] [02mw-config] 07Universal-Omega synchronize pull request 03#4042: Add `$wgExemptFromUserRobotsControl` to ManageWikiNamespaces - 13https://git.io/JRCXk [20:46:39] miraheze/mw-config - Universal-Omega the build passed. [20:46:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.83, 7.04, 6.30 [20:47:59] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.02, 7.93, 7.18 [20:50:42] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.38, 7.92, 6.78 [20:52:57] Voidwalker: filter is down to 18% from 33 [20:53:02] That's much better [20:53:10] Hopefully it stays that way [20:53:27] Ideally we need 5% but it's better [20:53:45] I'll lower thresholds down in morning if it's safe [20:54:31] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.99, 7.50, 6.86 [20:54:37] Overall we're hitting 27% across every filter [20:57:00] yeh, I've hit a few massive ranges [20:57:09] going to do some more tomorrow [20:57:47] Good [20:57:48] @Void Oh. So it looks like you've completed the hunt and bagged a lot of them. Great. [20:58:08] I might try and lower the limit to 30% in the morning [20:58:19] We need to try and keep it under control [20:59:18] As long as filters don't get edited we're fine [20:59:36] As the disable only kicks in the first day after an edit [20:59:48] It just didn't get reported [21:00:54] @Voidwalker You think you still have enough energy left to do anything else today, or is that it? [21:01:49] depends on what you need [21:02:10] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.14, 7.64, 7.18 [21:02:26] Ah, okay. [21:03:05] !log [@test3] starting deploy of {'config': True} to skip [21:03:06] !log [@test3] finished deploy of {'config': True} to skip - FAIL: [256] [21:03:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:03:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:03:22] I should disable that [21:04:04] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.94, 8.52, 7.55 [21:05:09] @Voidwalker I was hoping you'd look into Command, mainly due to his suspicious comments, and when I blocked the user on suspicion of sockpuppetry, he proceeded to get mad at me for it: https://freshwebsites.miraheze.org/w/index.php?title=Topic:We4sbkf4uumqb4rt&topic_showPostId=we4sbkf4uyksj8q1#flow-post-we4sbkf4uyksj8q1 [21:05:11] [url] If I was a "sock" of that user on User talk:Command | freshwebsites.miraheze.org [21:05:31] I know I'm sounding like a broken record at this point. [21:05:41] [02puppet] 07RhinosF1 opened pull request 03#1856: mediawiki: don't auto sync config - 13https://git.io/JRCyu [21:06:00] Voidwalker: ^ [21:07:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.11, 6.72, 5.78 [21:09:29] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.90, 6.42, 5.78 [21:10:01] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.45, 7.69, 7.58 [21:12:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.88, 7.74, 7.89 [21:15:59] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.15, 5.85, 6.76 [21:17:48] * RhinosF1 night [21:18:22] Good night to you too. [21:22:02]