[00:03:16] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:06:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.95, 3.62, 3.31 [00:07:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.97, 21.90, 19.80 [00:09:49] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.64, 24.59, 21.05 [00:10:19] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.36, 4.01, 3.49 [00:14:13] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.22, 3.63, 3.45 [00:15:57] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.04, 6.05, 4.76 [00:22:02] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.04, 3.87, 3.61 [00:23:59] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.46, 3.70, 3.58 [00:25:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.03, 3.64, 3.56 [00:27:53] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.67, 3.20, 3.40 [00:31:16] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [00:41:45] !log [@mw11] starting deploy of {'config': True} to all [00:41:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:41:57] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 11s [00:42:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:49:57] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.44, 5.12, 5.99 [00:57:57] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.40, 4.13, 5.04 [01:01:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 13.42, 19.42, 22.86 [01:01:57] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.59, 4.74, 5.17 [01:03:57] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 1.66, 3.70, 4.74 [01:09:49] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.52, 16.56, 19.92 [02:01:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.15, 2.99 [02:03:53] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.76, 3.02, 2.97 [02:21:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.47, 19.80, 17.70 [02:23:14] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.31, 19.49, 17.88 [02:50:08] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.42, 18.70, 16.13 [02:52:03] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.02, 19.82, 16.83 [02:55:53] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.21, 18.14, 16.99 [03:51:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.46, 3.71, 3.09 [03:55:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.30, 3.16, 3.01 [06:05:13] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.00, 5.76, 4.14 [06:07:13] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.78, 6.96, 4.77 [06:13:56] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 50393 MB (11% inode=98%); [06:19:57] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 47947 MB (10% inode=98%); [06:21:13] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.22, 7.90, 6.95 [06:23:13] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.35, 8.55, 7.32 [06:27:13] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.00, 7.50, 7.15 [06:31:13] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.63, 6.27, 6.71 [06:37:13] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.58, 7.02, 6.86 [06:41:13] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.74, 6.59, 6.73 [06:45:13] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.42, 7.35, 7.01 [06:47:13] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.80, 7.47, 7.09 [06:51:13] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.16, 6.47, 6.80 [06:55:13] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.12, 6.82, 6.87 [06:57:13] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.18, 6.55, 6.76 [07:06:24] PROBLEM - staff.polandballwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'staff.polandballwiki.com' expires in 7 day(s) (Mon 29 Nov 2021 07:06:05 GMT +0000). [07:11:16] !log [@mw11] starting deploy of {'config': True} to all [07:11:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:11:25] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 8s [07:11:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:25:13] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.93, 6.93, 6.70 [07:38:11] PROBLEM - wiki.worldbuilding.network - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.worldbuilding.network' expires in 15 day(s) (Tue 07 Dec 2021 07:35:43 GMT +0000). [07:45:13] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.18, 6.19, 6.70 [07:51:13] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.91, 7.55, 7.17 [07:53:13] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.39, 7.43, 7.14 [07:57:13] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.44, 7.76, 7.36 [07:57:56] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 51584 MB (11% inode=98%); [07:59:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.96, 20.55, 15.59 [08:01:49] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 10.64, 17.12, 14.97 [08:17:13] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.74, 6.35, 6.71 [08:17:56] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48593 MB (10% inode=98%); [09:20:13] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.56, 6.99, 6.66 [09:29:49] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.56, 6.44, 6.60 [09:51:49] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.11, 18.37, 14.52 [09:53:49] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 10.14, 15.02, 13.76 [10:58:35] [02miraheze/ssl] 07JohnFLewis pushed 031 commit to 03JohnFLewis-patch-1 [+0/-0/±1] 13https://git.io/J19on [10:58:37] [02miraheze/ssl] 07JohnFLewis 03e3537ea - Update privacy-wiki.0x.no.crt [10:58:38] [02ssl] 07JohnFLewis created branch 03JohnFLewis-patch-1 - 13https://git.io/vxP9L [11:00:20] [02ssl] 07JohnFLewis opened pull request 03#440: Update SSL certs - 13https://git.io/J19oD [11:02:19] [02miraheze/ssl] 07JohnFLewis pushed 031 commit to 03JohnFLewis-patch-1 [+0/-0/±1] 13https://git.io/J19K3 [11:02:21] [02miraheze/ssl] 07JohnFLewis 03ad3ed34 - Update famedata.org.crt [11:02:22] [02ssl] 07JohnFLewis synchronize pull request 03#440: Update SSL certs - 13https://git.io/J19oD [11:08:19] [02miraheze/ssl] 07JohnFLewis pushed 031 commit to 03JohnFLewis-patch-1 [+0/-0/±1] 13https://git.io/J196i [11:08:20] [02miraheze/ssl] 07JohnFLewis 034afcc2c - Update en.petrawiki.org.crt [11:08:22] [02ssl] 07JohnFLewis synchronize pull request 03#440: Update SSL certs - 13https://git.io/J19oD [11:11:24] [02miraheze/ssl] 07JohnFLewis pushed 031 commit to 03JohnFLewis-patch-1 [+0/-0/±1] 13https://git.io/J19iw [11:11:25] [02miraheze/ssl] 07JohnFLewis 032bcdafa - Update files.petrawiki.org.crt [11:11:27] [02ssl] 07JohnFLewis synchronize pull request 03#440: Update SSL certs - 13https://git.io/J19oD [11:13:05] [02miraheze/ssl] 07JohnFLewis pushed 031 commit to 03JohnFLewis-patch-1 [+0/-0/±1] 13https://git.io/J19Pt [11:13:06] [02miraheze/ssl] 07JohnFLewis 03dd3ce4e - Update translate.petrawiki.org.crt [11:13:08] [02ssl] 07JohnFLewis synchronize pull request 03#440: Update SSL certs - 13https://git.io/J19oD [11:15:27] [02miraheze/ssl] 07JohnFLewis pushed 031 commit to 03JohnFLewis-patch-1 [+0/-0/±1] 13https://git.io/J19PF [11:15:29] [02miraheze/ssl] 07JohnFLewis 03ceaf63b - Update hr.petrawiki.org.crt [11:15:30] [02ssl] 07JohnFLewis synchronize pull request 03#440: Update SSL certs - 13https://git.io/J19oD [11:18:15] [02miraheze/ssl] 07JohnFLewis pushed 031 commit to 03JohnFLewis-patch-1 [+0/-0/±1] 13https://git.io/J19Xz [11:18:17] [02miraheze/ssl] 07JohnFLewis 030864cb6 - Update vault.agentisai.com.crt [11:18:18] [02ssl] 07JohnFLewis synchronize pull request 03#440: Update SSL certs - 13https://git.io/J19oD [11:20:52] [02miraheze/ssl] 07JohnFLewis pushed 031 commit to 03JohnFLewis-patch-1 [+0/-0/±1] 13https://git.io/J191k [11:20:53] [02miraheze/ssl] 07JohnFLewis 03f689d94 - Update wiki.eversd.com.crt [11:20:55] [02ssl] 07JohnFLewis synchronize pull request 03#440: Update SSL certs - 13https://git.io/J19oD [11:22:45] This is strange. For some reason, I can't access the polandballwiki despite me using Windows 10, and this error has occurred. [11:22:45] https://cdn.discordapp.com/attachments/808001911868489748/911939686233157653/bandicam_2021-11-21_06-21-25-574.jpg [11:22:50] [02miraheze/ssl] 07JohnFLewis pushed 031 commit to 03JohnFLewis-patch-1 [+0/-0/±1] 13https://git.io/J191z [11:22:51] [02miraheze/ssl] 07JohnFLewis 034542652 - Update pt.polandballwiki.com.crt [11:22:53] [02ssl] 07JohnFLewis synchronize pull request 03#440: Update SSL certs - 13https://git.io/J19oD [11:24:55] [02miraheze/ssl] 07JohnFLewis pushed 031 commit to 03JohnFLewis-patch-1 [+0/-0/±1] 13https://git.io/J191F [11:24:57] [02miraheze/ssl] 07JohnFLewis 03be5940a - Update www.polandballwiki.com.crt [11:24:58] [02ssl] 07JohnFLewis synchronize pull request 03#440: Update SSL certs - 13https://git.io/J19oD [11:25:46] https://cdn.discordapp.com/attachments/808001911868489748/911940448153653289/unknown.png [11:26:44] Ah, I see. So it's inaccessible for right now. [11:30:53] [02miraheze/ssl] 07JohnFLewis pushed 031 commit to 03JohnFLewis-patch-1 [+0/-0/±1] 13https://git.io/J19DT [11:30:55] [02miraheze/ssl] 07JohnFLewis 03e4c48b7 - Update wiki.yapsavun.xyz.crt [11:30:56] [02ssl] 07JohnFLewis synchronize pull request 03#440: Update SSL certs - 13https://git.io/J19oD [11:33:33] [02miraheze/ssl] 07JohnFLewis pushed 031 commit to 03JohnFLewis-patch-1 [+0/-0/±1] 13https://git.io/J19DX [11:33:34] [02miraheze/ssl] 07JohnFLewis 0327048e2 - Update reviews.thejoshmeister.com.crt [11:33:36] [02ssl] 07JohnFLewis synchronize pull request 03#440: Update SSL certs - 13https://git.io/J19oD [11:34:57] [02ssl] 07JohnFLewis closed pull request 03#440: Update SSL certs - 13https://git.io/J19oD [11:34:59] [02miraheze/ssl] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±12] 13https://git.io/J19Dh [11:35:00] [02miraheze/ssl] 07JohnFLewis 037571f3d - Update SSL certs (#440) [11:39:27] RECOVERY - pt.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pt.polandballwiki.com' will expire on Sat 19 Feb 2022 10:22:16 GMT +0000. [11:39:46] RECOVERY - famedata.org - LetsEncrypt on sslhost is OK: OK - Certificate 'famedata.org' will expire on Sat 19 Feb 2022 10:01:26 GMT +0000. [11:39:58] RECOVERY - en.petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.petrawiki.org' will expire on Sat 19 Feb 2022 10:07:28 GMT +0000. [11:40:05] RECOVERY - wiki.eversd.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.eversd.com' will expire on Sat 19 Feb 2022 10:20:08 GMT +0000. [11:40:17] RECOVERY - petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.petrawiki.org' will expire on Sat 19 Feb 2022 10:07:28 GMT +0000. [11:40:55] RECOVERY - wiki.yapsavun.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.yapsavun.xyz' will expire on Sat 19 Feb 2022 10:26:20 GMT +0000. [11:41:05] RECOVERY - reviews.thejoshmeister.com - LetsEncrypt on sslhost is OK: OK - Certificate 'reviews.thejoshmeister.com' will expire on Sat 19 Feb 2022 10:32:42 GMT +0000. [11:41:06] RECOVERY - privacy-wiki.0x.no - LetsEncrypt on sslhost is OK: OK - Certificate 'privacy-wiki.0x.no' will expire on Sat 19 Feb 2022 09:54:59 GMT +0000. [11:41:40] RECOVERY - www.famedata.org - LetsEncrypt on sslhost is OK: OK - Certificate 'famedata.org' will expire on Sat 19 Feb 2022 10:01:26 GMT +0000. [11:41:56] RECOVERY - www.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.polandballwiki.com' will expire on Sat 19 Feb 2022 10:24:22 GMT +0000. [11:42:03] RECOVERY - polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.polandballwiki.com' will expire on Sat 19 Feb 2022 10:24:22 GMT +0000. [11:42:04] RECOVERY - hr.petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'hr.petrawiki.org' will expire on Sat 19 Feb 2022 10:14:58 GMT +0000. [11:42:56] RECOVERY - files.petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'files.petrawiki.org' will expire on Sat 19 Feb 2022 10:10:50 GMT +0000. [11:43:41] RECOVERY - www.petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.petrawiki.org' will expire on Sat 19 Feb 2022 10:07:28 GMT +0000. [11:43:57] darkmatterman450: should be resolved now [11:44:10] RECOVERY - vault.agentisai.com - LetsEncrypt on sslhost is OK: OK - Certificate 'vault.agentisai.com' will expire on Sat 19 Feb 2022 10:17:35 GMT +0000. [11:44:27] RECOVERY - translate.petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'translate.petrawiki.org' will expire on Sat 19 Feb 2022 10:12:30 GMT +0000. [11:45:46] Okay. [11:50:15] !log restart all daemons on bacula2 [11:50:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:51:49] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.40, 24.72, 18.49 [11:51:57] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.83, 6.60, 3.93 [11:53:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.12, 22.47, 18.46 [11:53:57] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.80, 5.81, 3.99 [11:55:57] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.90, 4.85, 3.86 [11:57:49] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.31, 18.11, 17.64 [12:03:34] [02miraheze/dns] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J195J [12:03:35] [02miraheze/dns] 07JohnFLewis 03a1e0369 - ACME challenge for miraheze.wiki [12:07:15] [02miraheze/dns] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J19dO [12:07:16] [02miraheze/dns] 07JohnFLewis 03e3a4b2b - new challenge [12:09:39] [02miraheze/ssl] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J19dh [12:09:40] [02miraheze/ssl] 07JohnFLewis 038b39e9b - deploy new miraheze.wiki wildcard [12:11:45] !log running bacula jobs manually as all failed overnight [12:11:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:42:14] RECOVERY - miraheze.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.wiki' will expire on Sat 19 Feb 2022 11:08:45 GMT +0000. [13:16:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.81, 3.33, 2.74 [13:18:07] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.32, 3.24, 2.77 [14:11:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.68, 20.81, 18.21 [14:13:49] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.31, 18.48, 17.68 [14:41:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.59, 18.43, 16.67 [14:43:49] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.98, 18.50, 16.90 [14:59:59] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.89, 3.36, 3.04 [15:05:50] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.36, 3.95, 3.42 [15:11:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.12, 3.63, 3.49 [15:15:35] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.17, 3.07, 3.30 [15:49:57] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.73, 6.67, 4.92 [15:49:58] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.35, 22.86, 19.43 [15:51:53] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 24.00, 23.40, 20.04 [15:51:57] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.25, 5.55, 4.71 [15:53:20] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.81, 6.77, 4.91 [15:53:49] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 23.81, 24.61, 20.94 [15:55:20] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.63, 6.01, 4.86 [15:55:57] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.28, 4.78, 4.61 [15:57:44] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.77, 3.77, 3.24 [15:57:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.58, 22.57, 21.07 [15:59:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.35, 3.67, 3.27 [16:01:49] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.37, 19.24, 20.00 [16:05:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.50, 2.97, 3.09 [16:12:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.05, 3.48, 3.31 [16:13:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.63, 21.21, 20.55 [16:16:19] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.05, 3.27, 3.27 [16:17:49] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.96, 18.34, 19.67 [16:27:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.66, 4.00, 3.53 [16:29:53] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.80, 3.88, 3.55 [16:31:50] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.17, 3.88, 3.58 [16:33:47] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.02, 3.54, 3.49 [16:37:41] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.09, 3.61, 3.53 [16:39:38] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.61, 3.54 [16:41:35] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.94, 4.01, 3.71 [16:43:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.30, 3.86, 3.70 [16:53:01] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.21, 22.42, 19.85 [16:53:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.09, 3.49, 3.53 [16:54:56] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.04, 20.49, 19.46 [16:55:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.77, 3.24, 3.43 [16:56:51] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.65, 18.53, 18.87 [17:01:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.06, 2.97, 3.27 [17:32:02] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.25, 3.54, 3.23 [17:33:59] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.29, 3.17 [17:35:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.45, 3.02, 3.08 [17:38:23] CosmicAlpha, looks like you ended up using Matrix as a temporary IRCCloud replacement? [17:39:31] Oh, nevermind, it adds `:libera.chat` to any IRC connected user [17:44:58] I got on matrix for now [17:46:27] MacFan4000, cool, Element.io as the client? [17:46:38] Yeah the iOS app [17:46:42] ah [17:46:47] yeah it's pretty decent [17:47:21] you can message `!nick ` to liberachat IRC bridge status to change your nick to MacFan4000 [17:48:38] Is the switch only temporary? [17:49:13] Yeah just while IRCCloud is down for maintenance [17:49:44] I'm going to guess that this will take hours for it to be completed. [17:49:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.90, 19.84, 18.19 [17:50:10] Up to four hours [17:50:28] 4 hours? Okay then. [17:51:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.45, 3.77, 3.41 [17:51:49] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.01, 18.77, 18.05 [17:55:14] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.65, 1.92, 1.13 [17:57:13] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.65, 1.43, 1.05 [17:57:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.88, 3.93, 3.63 [18:03:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.88, 2.93, 3.29 [18:13:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.61, 3.66, 3.38 [18:13:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.38, 20.11, 18.94 [18:15:49] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.71, 18.23, 18.36 [18:17:06] Doug: I am using Matrix for now, yes [18:17:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.47, 3.82, 3.52 [18:18:13] CosmicAlpha, cool 🙂 [18:18:16] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.20, 5.22, 4.28 [18:18:44] RECOVERY - bacula2 Bacula Private Git on bacula2 is OK: OK: Full, 6656 files, 53.35MB, 2021-11-21 18:17:00 (1.7 minutes ago) [18:19:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.64, 3.90, 3.57 [18:19:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.72, 21.47, 19.75 [18:20:15] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.41, 4.92, 4.28 [18:21:49] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.43, 22.44, 20.31 [18:23:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.33, 21.48, 20.29 [18:27:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.97, 3.77, 3.77 [18:29:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.03, 4.16, 3.90 [18:29:49] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.66, 18.87, 19.54 [18:33:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.54, 3.83, 3.83 [18:39:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.17, 4.13, 3.92 [18:41:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.99, 3.89 [18:43:23] RECOVERY - bacula2 Bacula Phabricator Static on bacula2 is OK: OK: Full, 88168 files, 9.442GB, 2021-11-21 18:43:00 (22.0 seconds ago) [18:43:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.01, 4.01, 3.91 [18:45:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.25, 3.77, 3.83 [18:51:14] PROBLEM - bacula2 Puppet on bacula2 is WARNING: WARNING: Puppet last ran 1 hour ago [18:52:07] RECOVERY - bacula2 Bacula Databases db11 on bacula2 is OK: OK: Full, 83186 files, 13.98GB, 2021-11-21 18:51:00 (1.1 minutes ago) [18:53:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.90, 3.83, 3.75 [18:55:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.12, 3.69, 3.72 [18:55:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.90, 21.58, 19.65 [18:57:49] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.06, 22.00, 20.00 [18:59:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.78, 20.50, 19.66 [19:01:49] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.66, 18.95, 19.21 [19:04:56] [02puppet] 07paladox closed pull request 03#2132: use `req.url` for undefined/api.php redirect again - 13https://git.io/J1DSA [19:04:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J1Q1V [19:04:59] [02miraheze/puppet] 07Universal-Omega 03c6af7d6 - use `req.url` for undefined/api.php redirect again (#2132) [19:09:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.67, 2.98, 3.33 [19:10:20] !sre `[ca1e583d7e0686f55bdcacc3] 2021-11-21 19:09:52: Fatal exception of type "Error"` on `uneasyalliancewiki` [19:10:30] RECOVERY - bacula2 Bacula Databases db12 on bacula2 is OK: OK: Full, 71925 files, 29.29GB, 2021-11-21 19:09:00 (1.5 minutes ago) [19:10:44] If possible to look into as soon as available, as it's required to revert vandalism by VOA ☹️ [19:11:33] * dmehus[m] thinks there's probably some content model problem on the wiki [19:12:10] CosmicAlpha: you around? [19:13:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.51, 19.54, 18.68 [19:13:54] Doug: I'm mobile right now. [19:15:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.04, 3.52, 3.46 [19:15:49] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.61, 19.21, 18.68 [19:19:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.49, 3.47, 3.46 [19:21:20] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.51, 6.75, 5.21 [19:21:43] PROBLEM - mw9 APT on mw9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:21:49] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.58, 23.35, 20.35 [19:23:20] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.02, 7.05, 5.52 [19:23:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.01, 3.63, 3.53 [19:23:42] RECOVERY - mw9 APT on mw9 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [19:25:08] RECOVERY - bacula2 Bacula Databases db13 on bacula2 is OK: OK: Full, 72508 files, 27.48GB, 2021-11-21 19:24:00 (1.1 minutes ago) [19:25:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.98, 3.60, 3.54 [19:26:44] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.91, 6.28, 4.86 [19:27:20] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.37, 6.11, 5.48 [19:31:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.25, 3.02, 3.34 [19:33:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.44, 23.27, 22.61 [19:35:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.53, 3.48, 3.49 [19:38:22] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.30, 5.93, 5.66 [19:39:49] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.92, 22.37, 22.20 [19:40:05] paladox, ty [19:40:17] I didn't realize I was DM'ing Agent instead of messaging it in here 😛 [19:40:18] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.87, 6.67, 5.99 [19:40:36] xD [19:40:45] @Doug Emergency, please see #cvt channel as soon as you can. [19:40:56] ok [19:41:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.35, 3.71, 3.56 [19:42:30] ergh now I'm getting 503s on `metawiki` [19:42:35] DarkMatterMan4500: You always ping Doug on Discord though I think it'd be smarter to ping dmehus/dmehus[m] to get his attention on IRC as pinging Doug doesn't ping him on IRC [19:43:14] I'm not home right now, but I'm at the mall. [19:43:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.89, 3.64, 3.55 [19:43:40] DarkMatterMan4500: You can always ping him by mentioning his IRC nick [19:43:50] I can ping myself on IRC by just saying "Agent" [19:44:01] in that same manner, you can ping him from Discord by just saying his IRC nick [19:44:32] Agent, yeah I've tried mentioning that to DarkMatterMan4500 several times 😛 [19:44:38] so thank you 🙂 [19:45:11] Basically, ping my Discord username when you want me on Discord or my IRC username when you want me on IRC (i.e., more likely to grab my attention) [19:45:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.53, 3.91, 3.65 [19:45:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.63, 23.70, 22.94 [19:45:50] ooh, mon2 squawking doesn't sound good ☹️ [19:45:56] [[Tech:Mon2]] [19:45:56] http://enwp.org/Tech:Mon2 [19:46:28] Permission denied [19:46:28] @configure default-link-wiki=miraheze [19:46:40] Agent, can you make that change for me [19:46:55] Let me see [19:47:01] Permission denied [19:47:01] @configure default-link-wiki=miraheze [19:47:04] Yay [19:47:09] oh [19:47:16] it said permission denied ☹️ [19:47:17] I trust: MacFan4000!.*@user/macfan4000 (2admin), .*@miraheze/.* (2trusted), .*@miraheze/Paladox (2admin), .*@miraheze/Universal-Omega (2admin), .*@miraheze/RhinosF1 (2admin), .*@miraheze/Southparkfan (2admin), .*@miraheze/Reception123 (2admin), [19:47:17] @trusted [19:47:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.08, 3.51, 3.53 [19:47:35] wm-bot, hello? [19:47:38] I trust: MacFan4000!.*@user/macfan4000 (2admin), .*@miraheze/.* (2trusted), .*@miraheze/Paladox (2admin), .*@miraheze/Universal-Omega (2admin), .*@miraheze/RhinosF1 (2admin), .*@miraheze/Southparkfan (2admin), .*@miraheze/Reception123 (2admin), [19:47:38] @trusted [19:47:56] looks like it needs paladox [19:48:12] paladox, mind running `@configure default-link-wiki=miraheze` for us? 🙂 [19:48:25] Permission denied [19:48:25] @configure default-link-wiki=miraheze` [19:48:25] /kick wm-bot for not listening >:( [19:48:30] oh [19:48:48] i use a wikimedia cloak not miraheze one [19:48:50] I think it didn't like the ` [19:48:52] try again [19:48:56] oh [19:49:08] lol! [19:49:13] that needs to get updated then heh [19:49:34] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.20, 3.77, 3.62 [19:49:34] MacFan4000, can you do that for us, since paladox has de facto authorized it (with his other cloak)? [19:50:00] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.42, 5.24, 5.87 [19:50:19] !link Tech:Mon2 [19:50:44] https://meta.miraheze.org/wiki/Tech:Mon2 [19:50:45] [url] Tech:Mon2 - Miraheze Meta | meta.miraheze.org [19:51:13] ah, that's what's on `mon2` [19:51:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.87, 3.69 [19:51:39] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.07, 7.68, 6.25 [19:51:49] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.92, 24.42, 23.21 [19:53:34] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.26, 7.41, 6.32 [19:53:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.37, 23.06, 22.88 [19:57:26] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.93, 5.87, 5.96 [19:59:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.77, 3.88, 3.72 [19:59:57] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.42, 6.32, 5.99 [20:01:49] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.55, 22.98, 22.63 [20:01:57] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.23, 5.85, 5.86 [20:03:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.31, 3.96, 3.81 [20:03:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.93, 20.75, 21.87 [20:07:57] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.38, 6.37, 5.96 [20:11:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.50, 4.06, 3.85 [20:11:57] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.46, 5.75, 5.82 [20:13:57] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.03, 5.85, 5.84 [20:15:49] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.30, 18.41, 19.90 [20:16:11] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/J1QA6 [20:16:13] [02miraheze/RemovePII] 07Universal-Omega 031fb3009 - Update RemovePIIGlobalRenameUserLogger.php [20:16:14] [02RemovePII] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/Jsfpj [20:17:57] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.51, 5.12, 5.58 [20:19:57] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.04, 6.07, 5.86 [20:21:49] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 33.82, 23.84, 21.29 [20:23:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.40, 3.56, 3.87 [20:23:51] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 167.114.2.161/cpweb [20:24:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:25:51] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:26:29] Successfully added .*@wikimedia/Paladox [20:26:29] @trustadd .*@wikimedia/Paladox admin [20:26:49] User not found, sorry [20:26:49] @trustdel .*@miraheze/paladox [20:26:59] I trust: MacFan4000!.*@user/macfan4000 (2admin), .*@miraheze/.* (2trusted), .*@miraheze/Paladox (2admin), .*@miraheze/Universal-Omega (2admin), .*@miraheze/RhinosF1 (2admin), .*@miraheze/Southparkfan (2admin), .*@miraheze/Reception123 (2admin), .*@wikimedia/Paladox (2admin), [20:26:59] @trusted [20:27:28] User was deleted from access list [20:27:28] @trustdel .*@miraheze/Paladox [20:27:49] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/J1Qhs [20:27:50] [02miraheze/RemovePII] 07Universal-Omega 03be449b2 - Update SpecialRemovePII.php [20:28:48] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.24, 6.76, 5.63 [20:28:49] User was deleted from access list [20:28:49] @trustdel MacFan4000!.*@user/macfan4000 [20:29:08] User was deleted from access list [20:29:08] @trustdel .*@wikimedia/Paladox [20:29:29] Wrong number of parameters, go fix it - example @trustadd regex (admin|trusted) [20:29:29] @trustadd .*@wikimedia/paladox [20:29:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.16, 3.54, 3.71 [20:29:36] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:29:52] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.75, 11.51, 7.97 [20:29:54] Successfully added .*@wikimedia/paladox [20:29:54] @trustadd .*@wikimedia/paladox admin [20:30:05] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.59, 10.89, 8.05 [20:30:15] dmehus: dmehus[m]: should be good now [20:30:28] ty MacFan4000 for fixing that [20:30:52] Value miraheze was stored into default-link-wiki in the config [20:30:52] @configure default-link-wiki=miraheze [20:31:24] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/J1QhH [20:31:26] [02miraheze/RemovePII] 07Universal-Omega 03763e40b - Update mediawiki-tests.yml [20:31:28] Successfully added .*@miraheze/Agent-Isai [20:31:28] @trustadd .*@miraheze/Agent-Isai admin [20:31:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.00, 3.47, 3.68 [20:31:35] [02RemovePII] 07Universal-Omega opened pull request 03#39: Add 1.37 compatability - 13https://git.io/J1Qhd [20:32:40] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.78, 6.53, 5.81 [20:33:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.70, 3.73 [20:33:52] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.68, 7.70, 7.21 [20:34:10] miraheze/RemovePII - Universal-Omega the build has errored. [20:34:23] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/J1QjG [20:34:25] [02miraheze/RemovePII] 07Universal-Omega 033d7c910 - Syntax [20:34:26]