[00:02:29] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 2 minutes ago with 6 failures. Failed resources (up to 3 shown) [00:05:59] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.73, 7.27, 5.99 [00:06:06] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.13, 8.54, 6.60 [00:07:59] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.19, 6.67, 5.94 [00:08:05] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.69, 7.57, 6.49 [00:08:21] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [00:10:06] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.63, 6.44, 6.22 [00:17:05] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [00:30:29] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [00:52:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.98, 18.49, 15.89 [00:54:30] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.19, 16.33, 15.42 [01:08:03] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:11:18] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [01:26:48] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [01:40:21] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [02:09:22] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [02:22:54] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [02:45:10] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [02:58:44] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [03:14:13] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [03:14:56] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.99, 6.79, 6.41 [03:16:55] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.40, 6.27, 6.26 [03:20:51] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.55, 6.71, 6.43 [03:22:50] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.45, 6.60, 6.43 [03:24:37] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 3.35, 2.22, 1.44 [03:25:46] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.25, 6.72, 6.12 [03:26:37] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.25, 1.82, 1.39 [03:26:44] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.37, 7.25, 6.79 [03:27:45] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.06, 6.49, 6.11 [03:28:37] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.93, 1.60, 1.36 [03:28:42] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.12, 6.75, 6.65 [03:32:37] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.40, 1.75, 1.49 [03:34:37] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.84, 7.77, 7.00 [03:34:37] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 3.36, 2.10, 1.63 [03:35:43] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.80, 7.13, 6.49 [03:36:37] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.52, 1.85, 1.60 [03:36:51] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.28, 6.85, 6.04 [03:38:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.75, 7.45, 7.10 [03:38:37] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.21, 1.61, 1.54 [03:38:51] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.23, 5.84, 5.77 [03:39:43] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.03, 6.18, 6.27 [03:44:29] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.70, 6.30, 6.69 [03:56:17] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.04, 6.71, 6.48 [03:58:15] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.70, 6.56, 6.44 [03:58:46] PROBLEM - cyberlaw.ccdcoe.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for cyberlaw.ccdcoe.org could not be found [03:58:46] PROBLEM - fr.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for fr.gyaanipedia.com could not be found [03:58:53] PROBLEM - n64brew.dev - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for n64brew.dev could not be found [04:05:33] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.97, 7.64, 6.47 [04:05:38] RECOVERY - cyberlaw.ccdcoe.org - reverse DNS on sslhost is OK: rDNS OK - cyberlaw.ccdcoe.org reverse DNS resolves to cp13.miraheze.org [04:05:38] RECOVERY - fr.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - fr.gyaanipedia.com reverse DNS resolves to cp13.miraheze.org [04:05:52] RECOVERY - n64brew.dev - reverse DNS on sslhost is OK: rDNS OK - n64brew.dev reverse DNS resolves to cp13.miraheze.org [04:06:08] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.69, 7.40, 6.75 [04:07:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.44, 7.43, 6.54 [04:08:06] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.74, 6.80, 6.61 [04:08:23] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [04:09:33] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.99, 6.79, 6.40 [04:23:58] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [04:29:05] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 1.31, 2.02, 1.59 [04:31:00] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 0.94, 1.75, 1.55 [04:32:55] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.69, 1.35, 1.42 [04:35:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.04, 6.36, 5.64 [04:37:05] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.88, 6.08, 5.62 [04:37:30] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [04:52:58] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [05:06:41] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [05:29:41] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [06:03:35] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [06:19:05] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [06:36:07] !log renamed ahinfoboxeswiki to althistorywiki [06:36:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:41:07] !log renamed infrawiki to carmechanicsimwiki [06:41:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:44:19] !log [reception@jobrunner3] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki noelsimssandboxwiki /home/reception/noelsimssandbox_pages_full.xml --username-prefix=wikia:noelsimssandboxwiki --report 1 [06:44:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:46:17] !log [reception@jobrunner3] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki mathorgwiki /home/reception/delbackups/mathorgwiki.xml --username-prefix=mathorgwiki [06:46:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:47:25] paladox: hmm seems like Notifico isn't working [06:51:47] PROBLEM - privacy-wiki.0x.no - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for privacy-wiki.0x.no could not be found [06:52:57] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [07:06:41] PROBLEM - privacy-wiki.0x.no - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'privacy-wiki.0x.no' expired on Tue 25 May 2021 00:51:52 GMT +0000. [07:22:34] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [07:36:27] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [07:45:19] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [07:59:10] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [08:21:52] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [08:35:42] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [08:54:08] RECOVERY - privacy-wiki.0x.no - reverse DNS on sslhost is OK: rDNS OK - privacy-wiki.0x.no reverse DNS resolves to cp13.miraheze.org [09:12:14] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [09:23:01] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.93, 17.26, 14.20 [09:25:00] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.15, 14.81, 13.66 [09:46:49] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [09:55:39] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [10:18:28] miraheze/ManageWiki - translatewiki the build passed. [10:18:30] miraheze/MirahezeMagic - translatewiki the build passed. [10:19:08] .op [10:19:09] Attempting to OP... [10:29:27] PROBLEM - test3 MediaWiki Rendering on test3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 324 bytes in 0.030 second response time [10:30:13] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [10:30:15] Reception|away: ^ [10:33:42] trying to fix puppet after all the mediawiki merges [10:35:04] fixed now [10:35:27] RECOVERY - test3 MediaWiki Rendering on test3 is OK: HTTP OK: HTTP/1.1 200 OK - 19910 bytes in 0.185 second response time [10:38:26] well the internal error, puppet still doesn't work [10:39:57] dmehus: do you stewards want CU to log logins? [10:43:17] .ask JohnLewis Hi, We noticed CU logging of successful bot logins can now be disabled so a quick question with your steward hat on. Would it be useful to log logins to CU ex successful bot ones? [10:43:17] RhinosF1: I'll pass that on when JohnLewis is around. [10:46:14] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [10:55:18] Southparkfan: that might make https://meta.miraheze.org/wiki/Tech:Data_Processing_Inventory slightly out [10:55:19] [ Tech:Data Processing Inventory - Miraheze Meta ] - meta.miraheze.org [10:55:40] CheckUser (registered editor) -> CheckUser (registered editor & attempted logins) [11:00:06] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [11:08:56] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [11:46:46] Reception|away: it’s known that it isn’t working - it’s not working anywhere on libera, it got klined by ozone (Sigyn) [11:52:01] MacFan4000: did staff say anything about it? [11:52:13] It shouldn't be hard for them to undo it and exempt the bot [11:52:37] It’s not from a miraheze channel, it’s from somebody else’s channel [11:53:10] I mentioned it in the upstream channel (#notifico) [11:53:31] Reception|away: also litharge needs +o in here [11:56:16] MacFan4000: what about Libera staff? [11:56:18] And will do [11:56:45] I haven’t spoken to them [11:57:04] Ah, we should probably try that [11:58:44] Do you want to or should I? [12:14:22] Reception|away: per -ops there isn’t an active ban relating to notifico - I pinged upstream again [12:15:03] Hm that's weird [12:25:05] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [12:33:55] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [13:02:23] miraheze/mw-config - paladox the build passed. [13:03:21] PROBLEM - cp10 Varnish Backends on cp10 is WARNING: No backends detected. If this is an error, see readme.txt [13:04:38] PROBLEM - cp10 HTTPS on cp10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:07:06] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.39, 20.19, 15.24 [13:08:21] PROBLEM - cp11 Varnish Backends on cp11 is WARNING: No backends detected. If this is an error, see readme.txt [13:09:08] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.81, 18.96, 15.44 [13:09:22] PROBLEM - www.puzzles.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.puzzles.wiki could not be found [13:09:29] PROBLEM - cp11 HTTPS on cp11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:09:29] PROBLEM - dreamsit.com.br - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:09:50] PROBLEM - cp10 Varnish Backends on cp10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:11:06] PROBLEM - cp11 Varnish Backends on cp11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:11:36] PROBLEM - cp10 Disk Space on cp10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:12:27] PROBLEM - cp11 Puppet on cp11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:12:37] PROBLEM - ping4 on cp10 is CRITICAL: PING CRITICAL - Packet loss = 100% [13:12:55] !log shutdown cp10/cp11 after removing from mw-config && also downtime [13:13:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:13:04] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:13:07] PROBLEM - cp10 Puppet on cp10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:13:20] PROBLEM - ping6 on cp11 is CRITICAL: CRITICAL - Destination Unreachable (2001:41d0:800:1bbd::4) [13:13:36] PROBLEM - cp11 APT on cp11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:13:38] PROBLEM - ping4 on cp11 is CRITICAL: PING CRITICAL - Packet loss = 100% [13:13:45] PROBLEM - ping6 on cp10 is CRITICAL: PING CRITICAL - Packet loss = 100% [13:13:48] PROBLEM - cp10 APT on cp10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:13:48] PROBLEM - cp10 SSH on cp10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:13:56] PROBLEM - cp11 Disk Space on cp11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:14:08] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:16:29] RECOVERY - www.puzzles.wiki - reverse DNS on sslhost is OK: rDNS OK - www.puzzles.wiki reverse DNS resolves to cp13.miraheze.org [13:21:45] !log restart graylog [13:21:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:22:52] !log restart elasticsearch [13:22:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:29:12] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [13:34:25] !log upgrade puppetserver to 6.15.3 [13:34:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:36:47] !log upgrade puppet-agent on puppet3 [13:36:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:38:20] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [13:38:24] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown) [13:38:28] !log upgrade puppetdb on puppet3 [13:38:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:51:18] paladox: cp10? [13:51:34] RhinosF1: yes? [13:51:48] paladox: what's wrong with it? [13:51:56] we are not using it? [13:52:00] was replaced with cp13 [13:52:06] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [13:52:41] paladox: can you downtime then [13:52:46] i have [13:52:50] Okay ty [13:52:55] Saw alerts above [13:55:12] PROBLEM - db13 Puppet on db13 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:56:54] yeh i downtimed after seeing them [13:57:04] PROBLEM - cp12 Puppet on cp12 is CRITICAL: CRITICAL: Puppet has 102 failures. Last run 4 minutes ago with 102 failures. Failed resources (up to 3 shown): File[icclopedia.org],File[icclopedia.org_private],File[ipv6bolivia.tk],File[ipv6bolivia.tk_private] [13:57:06] !log testing some gluster config changes [13:57:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:01:24] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [14:05:36] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.32, 6.76, 5.57 [14:06:03] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:07:44] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.30, 6.12, 5.48 [14:21:27] > CheckUser (registered editor) -> CheckUser (registered editor & attempted logins) [14:21:27] RhinosF1: what do you mean here about it being potentially out of date? [14:22:30] RECOVERY - db13 Puppet on db13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:22:58] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.38, 7.29, 6.23 [14:23:14] RhinosF1: what do you mean by wanting CU to log logins? It already did that, via `loginwiki`, if the user logged in with their username and password. :) [14:23:25] PROBLEM - mw10 Check Gluster Clients on mw10 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [14:25:02] RECOVERY - cp12 Puppet on cp12 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:25:12] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.01, 6.55, 6.09 [14:34:30] RECOVERY - mw10 Check Gluster Clients on mw10 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [14:36:26] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [14:50:43] !log set jobrunner4 cpu to host to match jobrunner3 [14:50:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:51:42] !log set mon2 cpu to host [14:53:13] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [14:53:23] !log set mon2 cpu to host [14:53:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:56:00] PROBLEM - jobrunner4 Check Gluster Clients on jobrunner4 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [14:57:41] miraheze/mw-config - paladox the build passed. [14:57:55] !log set gluster i/o threads to 16 [14:57:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:58:06] !log disabled gluster client i/o [14:58:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:00:12] RECOVERY - jobrunner4 Check Gluster Clients on jobrunner4 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [15:04:04] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [15:27:57] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [15:51:04] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.82, 7.30, 6.33 [15:51:10] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.75, 6.63, 5.87 [15:53:16] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.78, 6.76, 6.26 [15:53:18] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.42, 6.10, 5.77 [15:57:32] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.37, 7.06, 6.26 [15:57:57] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [15:59:39] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.48, 6.45, 6.14 [16:11:54] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [16:38:32] dmehus: you see logins in CheckUser? [16:41:07] RhinosF1, as far as I know, yes. I mean, they don't show up as a login, but if the user has logged in through login.miraheze.org, within the past 90 days, a result should be returned. As far as I know, this is not, strictly speaking, a new feature in MediaWiki 1.36, but if I'm wrong, then yes, that configuration variable should be enabled [16:41:40] dmehus: what do you mean they don't show up as a login? You mean if they've edited? [16:42:41] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [16:42:53] RhinosF1, I mean there's no specific log or edit entry, like there would be if there was an edit to their user page, but there's also no log entries for abuse filter hits, either (unless abuse filter hits don't show up in CheckUser) [16:43:42] dmehus: oh this would add specific entries for logging in to wikis (everyone but loginwiki because loginwiki gets too many logins) [16:43:47] basically, a login wouldn't return a log entry or an edit if using "get edits" [16:44:02] hrm [16:44:22] why not loginwiki, though? Arguably that would be the most beneficial [16:44:30] for cross-wiki spambots [16:44:47] though if CA auto-logins are now tracked locally, then yes, that would be a useful feature [16:44:56] Because the advice from upstream is it would kill the DB space [16:45:10] A little while back, I **WAS** thinking of it being changed to 6 months, but dropped that when Doug said the following phrase: "Account age doesn't matter at all. If two accounts edited or centrally logged in from the same IP within the past three months, it would pick it up". [16:45:23] Possibly, but consider that Wikimedia has a way larger active userbase [16:46:05] We can roll out more if it becomes needed on loginwiki [16:46:09] darkmatterman450, proposing to change to 6 months is defined in Privacy Policy, and would almost certainly need board approval, as that's a legal policy [16:46:16] ^ [16:46:37] That's why I dropped the whole idea, considering that it would be impossible. [16:47:05] Not impossible, but just would need a very good reason. I personally like the 90 day data retention period [16:47:20] dmehus: history is https://phabricator.wikimedia.org/T253802 [16:47:21] [ ⚓ T253802 Configure WMF wikis to log login attempts in CheckUser ] - phabricator.wikimedia.org [16:47:24] RhinosF1, okay, sounds good then [16:47:26] Yeah, it's better off being 3 months anyway. [16:50:10] RhinosF1, I think loginwiki did always "count" Special:UserLogin successful logins in terms of the CheckUser tool, but CentralAuth auto-logins were not tracked before now [16:50:43] and also now the Special:UserLogin successful logins are also tracked locally now [16:51:34] dmehus: the code says that is default false [16:51:38] So I'm confused [16:52:58] Dropped my freenode account now \o/ [16:53:39] paladox, cool :) [16:54:03] RhinosF1, hrm, what's the variable name? Maybe `false` is correct [16:54:37] dmehus: wgCheckUserLogLogins iirc [16:54:44] heh, a funny /leave message would've been "Join Libera Chat" :P [16:55:54] RhinosF1, oh the CheckUser doc is out of date [16:56:38] Reception|away, Agent did that when he left #miraheze on Freenode. He changed his name to ClownPrince, then used `/quit Join irc.libera.chat - Andrew Lee is a clown` [16:56:42] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [16:56:53] Oh I forgot that [16:57:13] paladox, forgot what? [16:57:23] dmehus: lol, that's funny [16:57:39] I think paladox means he forgot to exit with a funny message [16:57:48] ^ [16:58:23] Reception123|away, paladox, ah yeah [16:58:38] I'm definitely going to do that when I leave, waiting a few days though [16:58:52] yeah, I'll make sure to do it in #freenode too though :D [16:58:57] heh [16:59:44] RhinosF1: any idea why `$wgCheckUserLogLogins` isn't in the CheckUser documentation? Google search results indicate it *was*, but seems to have been removed [17:00:12] dmehus: ask AntiHarrasment [17:01:21] oh [17:01:51] Martin might know as well [17:01:53] It seems to be enabled on WMF wikis, with https://sal.toolforge.org/log/slvflHUBpU87LSFJzEXD. It's curious why it's set to `false`. Maybe the GitHub repository is just out of date [17:01:54] [ Log event - SAL ] - sal.toolforge.org [17:01:57] As he did the deploys for it [17:02:00] yeah [17:02:27] Yeah I know it was deployed [17:02:32] How I found out about it [17:02:36] Completely unrelated but I really do want us to figure out what to do about Notifico as it's quite useful [17:02:46] Reception|away: patience [17:03:01] RhinosF1, asked [17:03:08] Heh, my problem is that staff said they didn't see a ban on their side [17:03:17] ^ [17:03:41] Reception|away: did niko check if the range was banned? Maybe he just checked the single IP [17:03:42] Reception|away: probably needs restarting by its devs [17:03:42] So while maybe not true I feel like it's not on anyones radar even [17:03:58] dmehus: they have special tools for that [17:04:04] Ah that would make sense then [17:04:12] I think MacFan4000 has contacted the devs [17:04:12] RhinosF1: maybe, but it's worth asking nonetheless [17:04:32] Sometimes common oversights are the cause [17:04:36] The primary dev isn't responding to pings [17:04:44] Hm :( [17:06:06] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [17:20:05] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [17:29:12] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [17:43:06] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [17:52:17] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [18:06:10] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [18:36:36] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [18:50:35] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [18:54:22] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.87, 6.86, 6.31 [19:03:12] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.41, 6.33, 6.43 [19:07:06] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [19:33:46] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.85, 6.56, 6.02 [19:35:49] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.50, 5.98, 5.89 [19:42:21] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [19:49:10] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.50, 6.34, 6.06 [19:51:21] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.64, 6.12, 6.02 [19:57:20] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.22, 7.34, 6.59 [20:01:35] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.94, 6.27, 6.34 [20:05:10] Reception|away: upstream say the notifico issue is due to some of the libera servers not supporting older versions of ssl [20:05:28] so, they need to update things [20:05:44] oh, but how come it worked before then? [20:06:24] 15:59 <@TkTech> It's been working on some channels. From a quick skim it looks like some Libera servers on the network (which we get [randomly?] load balanced onto) have disabled older SSL ciphers, so it sometimes fails the handshake to connect. [20:09:59] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.50, 6.93, 6.57 [20:12:04] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.69, 6.38, 6.41 [20:26:21] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [20:33:32] MacFan4000, ah, interesting :) [20:40:19] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [20:56:27] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [21:31:25] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [21:34:04] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.98, 6.53, 5.90 [21:34:14] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.73, 6.78, 5.91 [21:36:14] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.50, 6.44, 5.94 [21:38:26] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.52, 6.98, 6.22 [21:40:32] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.85, 6.96, 6.28 [21:40:32] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [21:42:35] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.59, 6.15, 6.04 [21:44:50] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.84, 6.41, 6.28 [21:54:35] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [22:17:49] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [22:31:46] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [22:39:51] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.31, 8.29, 6.95 [22:41:59] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.68, 7.36, 6.76 [22:46:14] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.38, 6.43, 6.52 [22:46:20] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.39, 1.74, 1.25 [22:48:21] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.88, 1.45, 1.21 [22:54:48] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [23:07:57] PROBLEM - bacula2 Bacula Static on bacula2 is CRITICAL: connect to address 168.235.111.29 port 5666: Connection timed outconnect to host 168.235.111.29 port 5666: Connection timed out [23:08:24] PROBLEM - bacula2 Bacula Databases db13 on bacula2 is CRITICAL: connect to address 168.235.111.29 port 5666: Connection timed outconnect to host 168.235.111.29 port 5666: Connection timed out [23:10:28] PROBLEM - bacula2 Bacula Static on bacula2 is WARNING: WARNING: Full, 10219941 files, 1.220TB, 2021-05-07 04:23:00 (3.5 weeks ago) [23:10:40] PROBLEM - bacula2 Bacula Databases db13 on bacula2 is WARNING: WARNING: Full, 279527 files, 66.56GB, 2021-05-02 17:53:00 (4.2 weeks ago) [23:30:19] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [23:39:35] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found