[00:01:44] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [00:04:17] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 21489MiB (24% inode=98%); [00:04:51] RECOVERY - cp41 Disk Space on cp41 is OK: DISK OK - free space: / 19008MiB (20% inode=98%); [00:04:56] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 21179MiB (23% inode=98%); [01:04:55] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.70, 16.93, 10.70 [01:05:53] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.91, 17.82, 10.93 [01:06:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.79, 20.50, 13.45 [01:09:12] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 26.26, 18.90, 12.13 [01:09:53] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 19.78, 19.96, 13.54 [01:10:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.64, 22.37, 15.91 [01:12:35] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.gab.pt.eu.org could not be found [01:13:53] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 27.09, 22.65, 15.99 [01:16:28] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.87, 19.92, 16.99 [01:18:55] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.20, 23.55, 19.64 [01:19:12] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 19.25, 22.91, 18.20 [01:20:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.57, 23.82, 19.21 [01:20:55] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 23.57, 24.06, 20.31 [01:21:53] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.55, 23.96, 19.62 [01:22:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.15, 22.01, 19.13 [01:24:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.16, 23.19, 19.87 [01:25:12] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 26.17, 23.44, 19.88 [01:27:12] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.22, 22.92, 20.10 [01:28:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.70, 23.50, 20.88 [01:29:12] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.49, 22.87, 20.39 [01:29:25] shit didn't realize [01:29:53] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 15.56, 20.04, 19.64 [01:30:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.83, 24.99, 21.72 [01:38:09] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:39:53] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.48, 20.27, 19.66 [01:41:53] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 17.22, 18.82, 19.19 [01:44:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.03, 23.60, 23.41 [01:45:37] PROBLEM - www.durawiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'www.durawiki.com' expires in 7 day(s) (Mon 29 Apr 2024 01:39:02 AM GMT +0000). [01:46:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.09, 24.30, 23.64 [01:48:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.51, 23.83, 23.57 [01:50:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.15, 25.31, 24.13 [01:52:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 16.35, 22.73, 23.40 [01:52:49] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.46, 23.60, 21.04 [01:54:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.25, 23.10, 23.42 [01:56:50] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.97, 22.85, 21.43 [01:58:46] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 26.19, 24.21, 22.09 [02:02:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.09, 23.79, 23.99 [02:02:43] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 14.17, 20.67, 21.29 [02:06:12] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [02:10:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.37, 25.02, 24.26 [02:10:38] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.66, 22.47, 21.66 [02:12:37] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.27, 21.83, 21.52 [02:14:36] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 24.88, 22.82, 21.91 [02:20:27] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw151 [02:22:26] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [02:23:46] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw151 [02:25:41] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [02:31:23] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 2 backends are down. mw151 mw182 [02:32:20] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw151 [02:32:33] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw151 [02:34:19] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [02:34:29] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [02:35:22] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [02:37:48] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw151 [02:39:43] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [02:50:37] [02mw-config] 07anpang54 synchronize pull request 03#5541: Add configs for spacewiki - 13https://github.com/miraheze/mw-config/pull/5541 [03:00:02] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw151 [03:02:02] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [03:02:10] [1/2] hey guys [03:02:10] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1231439809600290936/Captura_de_Pantalla_2024-04-20_a_las_9.01.53_p.m..png?ex=6636f6b1&is=662481b1&hm=9bf05a92fdb07ee520dfb3b575a44487de8994d3612ce1190ca4894b0749e93f& [03:02:45] PROBLEM - mon181 Backups Grafana on mon181 is WARNING: FILE_AGE WARNING: /var/log/grafana-backup.log is 864146 seconds old and 93 bytes [03:06:32] @emicraftnoob loaded fine for me [03:06:59] came back normally, I was going to change the settings of a namespace [03:10:43] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw151 [03:12:43] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [03:24:56] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw151 [03:30:54] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [03:33:44] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw151 [03:37:44] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [03:41:12] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw151 [03:43:11] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [03:43:44] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw151 [03:44:33] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw151 [03:45:44] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [03:52:25] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:54:10] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [04:01:12] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 9.08, 16.19, 23.70 [04:03:53] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 9.64, 15.24, 22.46 [04:04:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 12.42, 17.48, 23.52 [04:04:55] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 13.97, 16.64, 23.00 [04:07:12] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 13.37, 13.71, 20.07 [04:11:53] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 15.39, 15.46, 19.73 [04:14:28] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.30, 16.85, 20.17 [04:14:55] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 15.81, 16.68, 20.20 [04:58:46] [02landing] 07koreirozzu opened pull request 03#147: Use jsdelivr - 13https://github.com/miraheze/landing/pull/147 [05:23:39] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.78, 20.33, 18.72 [05:29:23] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 16.09, 19.53, 19.01 [05:36:39] PROBLEM - cloud15 Puppet on cloud15 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [05:42:55] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.75, 20.59, 19.28 [05:46:55] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.21, 22.73, 20.35 [05:55:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.57, 23.64, 19.23 [05:57:18] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.14, 22.75, 19.42 [05:59:16] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.22, 23.90, 20.26 [06:02:10] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 20.89, 20.42, 16.80 [06:03:12] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.58, 23.69, 21.14 [06:03:53] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.99, 21.64, 18.25 [06:04:08] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query franchising.org.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:04:10] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 19.21, 19.67, 16.94 [06:04:39] RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:05:53] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 16.01, 19.54, 17.89 [06:06:55] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.06, 22.13, 23.10 [06:14:07] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.69, 20.79, 18.49 [06:14:55] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.46, 23.95, 23.09 [06:15:01] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.55, 23.13, 21.64 [06:16:06] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 15.59, 18.68, 17.99 [06:20:55] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.96, 23.46, 22.25 [06:26:55] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.06, 22.57, 23.60 [06:28:47] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.42, 17.74, 20.06 [06:42:29] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.18 ms [06:44:30] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 166.89 ms [06:46:55] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.08, 22.90, 22.07 [06:48:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.99, 20.08, 18.91 [06:50:31] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.87, 17.82, 14.87 [06:50:55] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.60, 23.50, 22.67 [06:52:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.44, 22.19, 20.08 [06:52:30] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 18.62, 17.92, 15.24 [06:56:55] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.81, 22.82, 22.23 [06:58:25] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 24.96, 20.03, 16.68 [07:00:26] @emicraftnoob 502/3 from us will show cf pages now I think [07:00:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.47, 24.18, 21.62 [07:00:32] Unless we've replaced them [07:05:53] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 20.67, 20.07, 17.67 [07:09:52] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.25, 22.67, 19.31 [07:34:02] RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp36.wikitide.net - CNAME OK [07:38:13] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [07:52:25] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:06:09] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [08:27:36] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.40, 22.05, 23.71 [08:29:53] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 9.61, 19.24, 23.75 [08:30:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 9.71, 19.33, 23.36 [08:32:55] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 9.42, 16.00, 22.84 [08:33:34] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 7.06, 12.28, 18.84 [08:33:53] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 6.43, 12.18, 19.81 [08:34:28] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 8.64, 13.12, 19.85 [08:36:55] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 10.14, 12.60, 19.86 [08:38:23] PROBLEM - elephants.ggreig.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'elephants.ggreig.com' expires in 15 day(s) (Tue 07 May 2024 08:14:31 AM GMT +0000). [08:38:36] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/359fa619ef6b...21c1cd9f44e4 [08:38:37] [02miraheze/ssl] 07WikiTideSSLBot 0321c1cd9 - Bot: Update SSL cert for elephants.ggreig.com [09:07:16] RECOVERY - elephants.ggreig.com - LetsEncrypt on sslhost is OK: OK - Certificate 'elephants.ggreig.com' will expire on Sat 20 Jul 2024 07:38:29 AM GMT +0000. [09:08:11] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [09:31:17] !log [alex@mwtask171] Upgraded packages bsdutils, libsmartcols1, util-linux-extra, util-linux, mount, apache2, apache2-bin, apache2-data, apache2-utils, uuid-dev, libuuid1, uuid-runtime, libblkid-dev, libblkid1, libmount-dev, libmount1, libfdisk1, fdisk, bsdextrautils, eject, linux-libc-dev, php8.2-zip, php8.2-xml, php8.2-readline, php8.2-opcache, [09:31:18] php8.2-mysql, php8.2-mbstring, php8.2-ldap, php8.2-intl, php8.2-gmp, php8.2-gd, php8.2-cli, php8.2-fpm, php8.2-dba, php8.2-curl, php8.2-bz2, php8.2-bcmath, php8.2-common, util-linux-locales, xserver-common, xserver-xorg-core, and xvfb [09:31:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:31:43] didn't actually upgrade ^ [09:31:55] working on a little something [09:32:04] I'll remove it from SAL soon [09:36:09] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [09:36:14] !log [alex@mwtask171] Upgraded packages bsdutils, libsmartcols1, util-linux-extra, util-linux, mount, apache2, apache2-bin, apache2-data, apache2-utils, uuid-dev, libuuid1, uuid-runtime, libblkid-dev, libblkid1, libmount-dev, libmount1, libfdisk1, fdisk, bsdextrautils, eject, linux-libc-dev, php8.2-zip, php8.2-xml, php8.2-readline, php8.2-opcache, [09:36:15] php8.2-mysql, php8.2-mbstring, php8.2-ldap, php8.2-intl, php8.2-gmp, php8.2-gd, php8.2-cli, php8.2-fpm, php8.2-dba, php8.2-curl, php8.2-bz2, php8.2-bcmath, php8.2-common, util-linux-locales, xserver-common, xserver-xorg-core, and xvfb [09:36:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:37:21] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [10:32:55] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.01, 18.59, 12.53 [10:34:55] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 18.81, 18.57, 13.26 [10:38:14] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:38:55] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.52, 20.58, 15.32 [10:42:55] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.52, 22.63, 17.27 [10:46:55] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.56, 23.68, 18.94 [10:48:55] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.74, 24.20, 19.68 [10:50:55] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.42, 22.59, 19.65 [10:52:10] PROBLEM - wiki.dragonsmp.pro - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.dragonsmp.pro' expires in 15 day(s) (Tue 07 May 2024 10:32:10 AM GMT +0000). [10:52:22] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/21c1cd9f44e4...b31e3c9862fb [10:52:24] [02miraheze/ssl] 07WikiTideSSLBot 03b31e3c9 - Bot: Update SSL cert for wiki.dragonsmp.pro [10:52:55] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.51, 24.03, 20.55 [10:56:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.40, 18.39, 14.07 [10:58:28] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.11, 17.93, 14.46 [11:02:55] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.23, 23.54, 22.25 [11:03:24] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [11:08:55] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.24, 23.41, 22.52 [11:10:55] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.94, 23.84, 22.81 [11:12:34] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [11:14:55] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.36, 23.69, 22.90 [11:21:21] RECOVERY - wiki.dragonsmp.pro - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.dragonsmp.pro' will expire on Sat 20 Jul 2024 09:52:16 AM GMT +0000. [11:31:53] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:33:33] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.58 ms [11:35:34] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.21 ms [11:38:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.94, 22.16, 18.66 [11:40:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.44, 22.28, 19.18 [11:41:57] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.11, 20.81, 17.87 [11:43:43] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.19, 20.71, 17.33 [11:43:57] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 18.16, 19.87, 17.90 [11:44:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.80, 23.62, 20.27 [11:46:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.98, 23.85, 20.81 [11:47:41] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 18.22, 20.30, 18.04 [11:50:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.76, 23.83, 21.43 [11:51:54] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.74, 21.83, 19.60 [11:52:25] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [11:52:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.15, 21.77, 21.01 [11:53:36] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.03, 22.34, 19.59 [11:55:35] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.44, 23.35, 20.26 [11:55:53] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 20.90, 22.86, 20.59 [11:57:34] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.57, 22.83, 20.45 [11:58:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.23, 22.63, 21.49 [11:59:32] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 27.66, 24.35, 21.28 [11:59:53] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 26.18, 23.63, 21.36 [12:01:31] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.44, 23.55, 21.40 [12:02:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.70, 23.55, 22.22 [12:03:52] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.61, 22.92, 21.64 [12:05:51] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.22, 23.38, 21.92 [12:06:10] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [12:06:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.64, 24.54, 22.83 [12:08:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.81, 22.83, 22.46 [12:09:27] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 24.70, 23.42, 22.04 [12:10:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.87, 23.62, 22.78 [12:11:50] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 20.93, 23.70, 22.73 [12:12:35] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.gab.pt.eu.org could not be found [12:13:24] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.44, 23.79, 22.56 [12:15:49] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 27.52, 25.07, 23.47 [12:17:49] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.49, 23.55, 23.09 [12:18:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 15.80, 21.05, 22.27 [12:21:19] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 10.90, 16.30, 19.77 [12:22:55] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 14.01, 19.04, 23.25 [12:26:28] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.75, 17.70, 20.08 [12:26:55] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.11, 20.74, 22.87 [12:28:55] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.36, 22.03, 23.12 [12:31:46] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 15.40, 17.82, 19.76 [12:34:55] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.18, 23.10, 23.10 [12:35:45] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 20.88, 19.72, 20.13 [12:38:11] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:40:05] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 26.81, 23.35, 20.68 [12:42:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.10, 21.14, 20.08 [12:42:34] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [12:44:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.42, 22.94, 20.88 [12:45:36] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 167.36 ms [12:45:43] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 26.99, 22.82, 21.39 [12:46:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.21, 21.69, 20.67 [12:47:43] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 18.83, 21.10, 20.93 [12:51:42] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.87 ms [12:52:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.64, 22.16, 21.03 [12:53:41] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 27.61, 23.64, 21.93 [12:56:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.51, 23.49, 21.84 [12:57:40] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 19.32, 22.37, 21.95 [12:58:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.66, 24.27, 22.35 [13:00:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.80, 22.13, 21.79 [13:05:39] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 15.26, 17.82, 19.92 [13:05:53] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 18.81, 23.56, 23.90 [13:06:12] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:08:28] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 10.97, 16.87, 19.92 [13:10:55] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 14.94, 18.09, 22.85 [13:11:53] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 10.48, 14.96, 19.88 [13:12:34] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.gab.pt.eu.org could not be found [13:18:55] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 14.93, 16.46, 20.04 [13:32:55] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.92, 22.28, 20.66 [13:34:55] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.45, 21.71, 20.70 [13:37:03] [1/2] hey guys, now this is concerning [13:37:03] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1231599585525370930/Captura_de_Pantalla_2024-04-21_a_las_7.36.30_a.m..png?ex=66378b7f&is=6625167f&hm=dcbb9fb708a5c0d1e3b60f33a95d609260b40db366b01b2894f94b610a297563& [13:37:14] https://snowballaspedia.fr.to [13:37:17] Try refreshing [13:38:29] [1/2] yea it works but please call someone to fix it [13:38:29] [2/2] sorry if wrong ping but @Site Reliability Engineers [13:42:34] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [13:44:55] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.22, 21.39, 20.81 [13:45:59] No need to ping SRE over a few seconds [13:46:48] Refresh and make sure it’s not just you first [13:46:55] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.15, 21.69, 21.03 [13:50:55] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.62, 23.57, 21.98 [13:52:55] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.77, 22.55, 21.80 [14:16:55] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 13.27, 18.08, 19.89 [14:19:44] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.54 ms [14:21:45] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.51 ms [14:25:51] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 165.96 ms [14:29:57] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.15 ms [14:34:33] PROBLEM - www.mwcosmos.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.mwcosmos.com All nameservers failed to answer the query. [15:04:18] RECOVERY - www.mwcosmos.com - reverse DNS on sslhost is OK: SSL OK - www.mwcosmos.com reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [15:09:38] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.50 ms [15:12:35] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.gab.pt.eu.org could not be found [15:17:46] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.63 ms [15:23:45] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.57, 20.32, 17.70 [15:27:34] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 20.07, 20.10, 18.21 [15:29:12] PROBLEM - cp41 PowerDNS Recursor on cp41 is CRITICAL: No response from DNS ::1 [15:31:10] RECOVERY - cp41 PowerDNS Recursor on cp41 is OK: DNS OK: 0.727 seconds response time. wikitide.net returns 109.123.230.163,2400:d320:2161:9775::1 [15:31:24] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.19, 20.76, 18.99 [15:39:03] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.82, 23.30, 20.73 [15:42:55] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.23, 23.57, 21.43 [15:48:55] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.20, 24.13, 22.24 [15:50:55] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.87, 23.33, 22.16 [15:52:25] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [15:53:52] uh woops :derp: [15:53:53] ignore that [15:55:43] Lol [15:55:46] Use #verify [15:56:57] oh okay i saw i had to /auth in any channels mb [15:57:35] Yeah but that bot has dementia sometimes [15:57:47] yeap it didnt respond after i authed twice 😭 [15:57:50] So we have two dementia bots and hope one of them works at any given time [15:57:50] im in :salutation: [15:58:00] We ball [16:01:16] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 15.37, 18.65, 20.28 [16:17:09] !log [macfan@test151] starting deploy of {'l10n': True, 'force_upgrade': True, 'versions': '1.42', 'upgrade_extensions': 'MultiPurge'} to test151 [16:17:10] !log [macfan@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [16:17:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:17:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:17:46] !log [macfan@test151] starting deploy of {'l10n': True, 'force': True, 'force_upgrade': True, 'versions': '1.42', 'upgrade_extensions': 'MultiPurge'} to test151 [16:17:51] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.481 second response time [16:17:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:18:40] RECOVERY - test151 HTTPS on test151 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.085 second response time [16:20:55] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.94, 23.03, 19.75 [16:23:14] !log [macfan@test151] finished deploy of {'l10n': True, 'force': True, 'force_upgrade': True, 'versions': '1.42', 'upgrade_extensions': 'MultiPurge'} to test151 - SUCCESS in 328s [16:23:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:23:53] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 29.14, 20.98, 16.37 [16:24:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.95, 20.20, 16.83 [16:26:28] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.66, 19.59, 17.05 [16:27:53] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.22, 22.46, 18.07 [16:28:01] Thanks MacFan4000 [16:29:53] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 27.46, 24.22, 19.25 [16:31:53] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.28, 22.87, 19.38 [16:32:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.64, 20.46, 18.14 [16:35:53] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.21, 24.46, 20.72 [16:36:28] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.51, 19.81, 18.50 [16:40:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.36, 21.57, 19.62 [16:42:07] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.81, 20.03, 16.58 [16:42:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.53, 23.25, 20.42 [16:50:04] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.54 ms [16:52:05] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.70 ms [16:55:20] [02puppet] 07Universal-Omega created branch 03mw - 13https://github.com/miraheze/puppet [16:55:21] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mw [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/8e75afe93779 [16:55:23] [02miraheze/puppet] 07Universal-Omega 038e75afe - mediawiki: add sslredirects and csp [16:55:25] [02puppet] 07Universal-Omega opened pull request 03#3839: mediawiki: add sslredirects and csp - 13https://github.com/miraheze/puppet/pull/3839 [17:01:23] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mw [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8e75afe93779...deccec56028c [17:01:26] [02miraheze/puppet] 07Universal-Omega 03deccec5 - Add [17:01:29] [02puppet] 07Universal-Omega synchronize pull request 03#3839: mediawiki: add sslredirects and csp - 13https://github.com/miraheze/puppet/pull/3839 [17:03:32] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mw [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/deccec56028c...d9f4924fcf6e [17:03:34] [02miraheze/puppet] 07Universal-Omega 03d9f4924 - Add [17:03:37] [02puppet] 07Universal-Omega synchronize pull request 03#3839: mediawiki: add sslredirects and csp - 13https://github.com/miraheze/puppet/pull/3839 [17:14:31] [02puppet] 07Universal-Omega synchronize pull request 03#3839: mediawiki: add sslredirects and csp - 13https://github.com/miraheze/puppet/pull/3839 [17:14:32] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mw [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d9f4924fcf6e...90f7337e8c07 [17:14:34] [02miraheze/puppet] 07Universal-Omega 0390f7337 - -MJ12bot [17:20:22] [02puppet] 07Universal-Omega synchronize pull request 03#3839: mediawiki: add sslredirects and csp - 13https://github.com/miraheze/puppet/pull/3839 [17:20:25] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mw [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/90f7337e8c07...b2dae1fb57cc [17:20:29] [02miraheze/puppet] 07Universal-Omega 03b2dae1f - Commons Redirect [17:47:53] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 5.56, 16.09, 23.62 [17:47:53] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 6.25, 15.17, 21.96 [17:48:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 7.80, 15.99, 22.78 [17:48:55] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 5.48, 14.77, 22.91 [17:49:53] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 7.65, 12.91, 20.34 [17:51:53] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 8.31, 11.39, 19.94 [17:52:28] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 6.91, 11.10, 19.21 [17:52:55] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 7.92, 10.99, 19.48 [18:07:35] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [18:07:46] [02miraheze/puppet] 07Universal-Omega deleted branch 03mw [18:07:48] [02puppet] 07Universal-Omega closed pull request 03#3839: mediawiki: add sslredirects and csp - 13https://github.com/miraheze/puppet/pull/3839 [18:07:50] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±2] 13https://github.com/miraheze/puppet/compare/304fead67afd...63cbfd0b53b6 [18:07:53] [02miraheze/puppet] 07Universal-Omega 0363cbfd0 - mediawiki: add sslredirects and csp (#3839) [18:07:55] [02puppet] 07Universal-Omega deleted branch 03mw - 13https://github.com/miraheze/puppet [18:08:56] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:09:35] RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:09:58] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:12:11] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [18:13:00] RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [18:13:11] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [18:13:18] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:13:34] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:13:47] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:14:32] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:18:00] PROBLEM - puppet181 Current Load on puppet181 is WARNING: LOAD WARNING - total load average: 10.65, 7.89, 4.24 [18:20:01] RECOVERY - puppet181 Current Load on puppet181 is OK: LOAD OK - total load average: 9.79, 8.51, 4.92 [18:30:15] [02puppet] 07redbluegreenhat opened pull request 03#3840: Add secupgrade.sh script - 13https://github.com/miraheze/puppet/pull/3840 [18:32:25] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [18:54:05] !log [@test151] starting deploy of {'config': True} to test151 [18:54:06] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [18:54:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:54:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:56:42] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/63cbfd0b53b6...c3837962b591 [18:56:43] [02miraheze/puppet] 07Universal-Omega 03c383796 - phorge: add cloudflare to firewall [18:57:41] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c3837962b591...3084e858c1df [18:57:43] [02miraheze/puppet] 07Universal-Omega 033084e85 - reports: add cloudflare to firewall [19:00:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [19:08:55] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.64, 20.36, 12.70 [19:10:55] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.73, 21.81, 14.19 [19:12:55] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.25, 23.36, 15.72 [19:38:55]