[00:01:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.01, 24.61, 24.48 [00:11:18] [02mw-config] 07BlankEclair opened pull request 03#5619: T12367: Add steam:// to $wgUrlProtocols for downgradegameswiki - 13https://github.com/miraheze/mw-config/pull/5619 [00:12:08] miraheze/mw-config - BlankEclair the build passed. [00:29:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:31:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.05, 21.70, 23.76 [00:39:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.53, 23.57, 23.59 [00:41:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.74, 22.81, 23.32 [00:43:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.11, 22.96, 23.29 [00:47:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.92, 23.13, 23.28 [00:55:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.55, 23.69, 23.22 [00:57:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.44, 22.57, 22.86 [01:01:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.20, 23.19, 23.02 [01:03:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.06, 21.22, 22.27 [01:13:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.95, 23.39, 22.34 [01:38:01] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [01:39:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 12.65, 19.39, 23.99 [01:40:01] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 222 system event log (SEL) entries present] [01:41:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 22.86, 21.62, 24.28 [01:45:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.88, 22.49, 23.85 [01:49:34] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [01:53:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.90, 22.22, 22.92 [01:55:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.15, 21.24, 22.49 [01:59:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.10, 22.76, 22.75 [02:01:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.32, 20.96, 22.12 [02:03:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.50, 21.88, 22.28 [02:04:32] RECOVERY - 9b9t.wiki - LetsEncrypt on sslhost is OK: OK - Certificate '*.parklogic.com' will expire on Wed 19 Feb 2025 11:59:59 PM GMT +0000. [02:07:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 15.05, 20.20, 21.64 [02:11:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.75, 21.93, 21.91 [02:19:08] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:03:02] PROBLEM - mon181 Backups Grafana on mon181 is WARNING: FILE_AGE WARNING: /var/log/grafana-backup.log is 864163 seconds old and 92 bytes [03:03:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.25, 22.39, 23.98 [03:15:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.51, 23.74, 23.43 [03:17:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:17:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.44, 22.22, 22.90 [03:23:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.21, 24.33, 23.46 [03:29:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.96, 23.81, 23.62 [03:31:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.32, 23.86, 23.63 [03:35:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.73, 23.54, 23.56 [03:37:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 30.37, 25.88, 24.40 [03:47:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.10, 23.19, 23.77 [03:57:04] [02mw-config] 07Universal-Omega closed pull request 03#5619: T12367: Add steam:// to $wgUrlProtocols for downgradegameswiki - 13https://github.com/miraheze/mw-config/pull/5619 [03:57:06] [02mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/15dac3a6d83c...0b2095fbe97f [03:57:08] [02mw-config] 07BlankEclair 030b2095f - T12367: Add steam:// to $wgUrlProtocols for downgradegameswiki (#5619) [03:57:54] miraheze/mw-config - Universal-Omega the build passed. [03:58:11] [02mw-config] 07Universal-Omega reviewed pull request 03#5598 commit - 13https://github.com/miraheze/mw-config/pull/5598#discussion_r1685632301 [04:01:29] [02mw-config] 07BlankEclair synchronize pull request 03#5598: T12214: Avoid redirects if action=raw is set - 13https://github.com/miraheze/mw-config/pull/5598 [04:02:01] [02mw-config] 07BlankEclair reviewed pull request 03#5598 commit - 13https://github.com/miraheze/mw-config/pull/5598#discussion_r1685632552 [04:02:15] miraheze/mw-config - BlankEclair the build passed. [04:02:40] !log [@mwtask181] starting deploy of {'config': True} to all [04:02:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:03:03] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 23s [04:03:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:03:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.85, 23.31, 22.81 [04:05:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.19, 23.04, 22.80 [04:07:29] !log [@mwtask171] starting deploy of {'config': True} to all [04:07:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:07:48] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 18s [04:07:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:11:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.23, 24.77, 23.50 [04:13:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.34, 23.84, 23.38 [04:15:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 16.57, 20.95, 23.33 [04:19:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.47, 22.61, 23.33 [04:19:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.51, 22.38, 22.64 [04:21:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 16.93, 20.64, 22.01 [04:23:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.13, 22.04, 23.08 [04:23:19] !log [@test151] starting deploy of {'config': True} to test151 [04:23:20] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [04:23:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:23:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:29:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.63, 22.78, 22.92 [04:31:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.79, 21.88, 22.59 [04:33:02] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 1.82, 0.39, 0.13 [04:33:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.09, 22.32, 21.76 [04:35:27] !log [void@mw151] reboot mw151 (attempt to reduce high load -- didn't word :( ) [04:35:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:35:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.96, 22.79, 22.00 [04:41:48] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.15, 18.60, 20.29 [04:42:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:45:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.41, 20.80, 20.83 [04:53:48] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.80, 19.79, 20.32 [04:57:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.02, 20.27, 20.44 [05:00:10] !log [void@mwtask181] starting deploy of {'folders': '1.42'} to all [05:00:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:01:03] !log [void@mwtask181] finished deploy of {'folders': '1.42'} to all - SUCCESS in 52s [05:01:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:01:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.19, 22.22, 21.13 [05:03:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.92, 22.66, 21.43 [05:05:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.34, 23.20, 21.78 [05:07:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.94, 22.27, 21.58 [05:15:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.29, 21.86, 21.28 [05:17:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.67, 21.86, 21.41 [05:19:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.97, 23.97, 22.21 [05:33:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.48, 23.06, 23.41 [05:35:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.58, 23.54, 23.54 [05:36:31] PROBLEM - 9b9t.wiki - LetsEncrypt on sslhost is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket [05:37:09] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [05:37:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.44, 21.17, 22.67 [05:39:09] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [225 system event log (SEL) entries present] [05:39:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.14, 22.21, 22.84 [05:41:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.91, 21.60, 22.55 [05:42:37] PROBLEM - cloud18 Puppet on cloud18 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [05:43:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.55, 22.31, 22.65 [05:45:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.33, 22.75, 22.83 [05:52:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:01:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.34, 22.12, 22.15 [06:05:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.27, 23.27, 22.81 [06:07:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.18, 24.39, 23.26 [06:10:50] RECOVERY - cloud18 Puppet on cloud18 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:11:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.85, 23.56, 23.28 [06:12:25] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:17:05] PROBLEM - swiftac171 Current Load on swiftac171 is CRITICAL: LOAD CRITICAL - total load average: 22.49, 11.47, 5.73 [06:17:46] PROBLEM - swiftac171 PowerDNS Recursor on swiftac171 is CRITICAL: CRITICAL - Plugin timed out while executing system call [06:17:48] PROBLEM - swiftac171 SSH on swiftac171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:18:09] PROBLEM - swiftac171 APT on swiftac171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:19:41] RECOVERY - swiftac171 PowerDNS Recursor on swiftac171 is OK: DNS OK: 0.264 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [06:19:44] RECOVERY - swiftac171 SSH on swiftac171 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [06:20:07] RECOVERY - swiftac171 APT on swiftac171 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [06:21:03] PROBLEM - swiftac171 Current Load on swiftac171 is WARNING: LOAD WARNING - total load average: 5.09, 10.97, 7.16 [06:21:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.96, 23.18, 23.02 [06:23:03] RECOVERY - swiftac171 Current Load on swiftac171 is OK: LOAD OK - total load average: 2.57, 8.04, 6.55 [06:27:25] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:32:25] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:33:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.91, 23.21, 23.60 [06:39:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.55, 21.90, 22.67 [06:41:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.82, 20.51, 22.06 [06:46:31] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:55:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.57, 22.68, 22.16 [06:57:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.02, 20.65, 21.49 [07:09:48] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.37, 19.00, 20.20 [07:13:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.56, 20.03, 20.36 [07:15:39] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:17:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.60, 23.53, 21.60 [07:21:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.91, 23.16, 22.04 [07:23:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.35, 23.79, 22.39 [07:27:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:35:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.70, 20.17, 17.68 [07:37:02] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.77, 18.53, 17.37 [07:39:44] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [07:41:44] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [226 system event log (SEL) entries present] [07:54:40] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.09, 20.58, 18.77 [07:56:36] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.67, 19.41, 18.58 [08:05:18] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.77, 20.71, 19.57 [08:07:14] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.94, 23.03, 20.56 [08:09:10] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.28, 22.56, 20.71 [08:09:15] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 9.56, 6.22, 3.44 [08:11:05] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.48, 22.82, 20.99 [08:11:15] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 5.65, 5.89, 3.66 [08:13:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.93, 22.22, 21.00 [08:23:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.55, 22.51, 21.54 [08:27:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.62, 22.40, 21.74 [08:27:42] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.18, 20.62, 18.61 [08:28:43] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:29:40] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.32, 20.40, 18.81 [08:32:43] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001251906157 secs [08:35:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.79, 24.29, 22.57 [08:37:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.10, 22.94, 22.27 [08:45:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.13, 22.83, 22.22 [08:47:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.54, 22.15, 22.02 [08:50:46] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw152 [08:52:45] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [08:55:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.63, 22.99, 22.27 [08:59:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.54, 22.63, 22.34 [09:03:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.42, 23.15, 22.50 [09:07:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.57, 23.69, 23.01 [09:15:02] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 14.35, 17.08, 20.10 [09:20:43] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [09:22:38] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 9.602308273e-05 secs [09:37:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.57, 20.23, 19.83 [09:40:31] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [09:41:02] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 19.45, 19.35, 19.58 [09:42:27] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001022011042 secs [09:55:29] PROBLEM - wiki.secondrenaissance.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.secondrenaissance.net' expires in 15 day(s) (Tue 06 Aug 2024 09:33:59 AM GMT +0000). [09:55:39] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a4a98f6f283f...e62cc2da6731 [09:55:42] [02ssl] 07WikiTideSSLBot 03e62cc2d - Bot: Update SSL cert for wiki.secondrenaissance.net [10:06:11] PROBLEM - wiki.msnld.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.msnld.com' expires in 7 day(s) (Mon 29 Jul 2024 09:45:57 AM GMT +0000). [10:06:14] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.89, 21.05, 19.76 [10:10:05] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.31, 19.43, 19.45 [10:24:58] RECOVERY - wiki.secondrenaissance.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.secondrenaissance.net' will expire on Sat 19 Oct 2024 08:55:34 AM GMT +0000. [10:39:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 10.07, 15.89, 22.99 [10:42:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [10:45:02] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 15.29, 14.04, 19.75 [10:47:18] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:05:20] [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [11:07:35] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 8.99, 5.12, 2.26 [11:09:33] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.85, 3.86, 2.15 [11:10:20] [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [11:11:33] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.25, 2.58, 1.88 [11:12:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [11:15:37] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:46:20] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:52:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:15:30] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:22:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:30:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.07, 21.66, 19.19 [12:36:15] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.38, 19.36, 19.04 [12:47:19] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [12:49:19] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [228 system event log (SEL) entries present] [13:22:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [13:29:35] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.24, 21.36, 19.12 [13:31:33] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.97, 21.21, 19.32 [13:39:28] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.45, 22.86, 20.58 [13:41:26] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.05, 21.53, 20.38 [13:43:25] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.93, 19.27, 19.70 [13:51:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [13:51:18] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.58, 20.58, 19.80 [13:53:17] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.08, 19.12, 19.36 [14:06:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [14:29:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.34, 22.47, 23.86 [14:37:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.55, 23.34, 23.55 [14:45:09] PROBLEM - wiki.cowgl.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.cowgl.xyz' expires in 15 day(s) (Tue 06 Aug 2024 02:42:21 PM GMT +0000). [14:45:20] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e62cc2da6731...c53851a90e1a [14:45:23] [02ssl] 07WikiTideSSLBot 03c53851a - Bot: Update SSL cert for wiki.cowgl.xyz [14:47:06] [02mw-config] 07redbluegreenhat opened pull request 03#5620: Re-enable RatePage - 13https://github.com/miraheze/mw-config/pull/5620 [14:47:16] PROBLEM - dragdown.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dragdown.wiki' expires in 15 day(s) (Tue 06 Aug 2024 02:44:12 PM GMT +0000). [14:47:30] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c53851a90e1a...b032b903fede [14:47:31] [02ssl] 07WikiTideSSLBot 03b032b90 - Bot: Update SSL cert for dragdown.wiki [14:47:54] miraheze/mw-config - redbluegreenhat the build passed. [14:48:58] PROBLEM - wiki.jeremyboyd.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.jeremyboyd.org' expires in 15 day(s) (Tue 06 Aug 2024 02:44:40 PM GMT +0000). [14:49:10] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b032b903fede...f51aa95d36eb [14:49:11] [02ssl] 07WikiTideSSLBot 03f51aa95 - Bot: Update SSL cert for wiki.jeremyboyd.org [14:50:35] [02mw-config] 07redbluegreenhat closed pull request 03#5620: Re-enable RatePage - 13https://github.com/miraheze/mw-config/pull/5620 [14:50:38] [02mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/0b2095fbe97f...679f40ef9813 [14:50:40] [02mw-config] 07redbluegreenhat 03679f40e - Re-enable RatePage (#5620) [14:51:36] miraheze/mw-config - redbluegreenhat the build passed. [14:51:57] !log [alex@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [14:52:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:52:10] !log [alex@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 13s [14:52:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:53:14] !log [alex@mwtask181] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'RatePage'} to all [14:53:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:53:29] !log [alex@mwtask181] finished deploy of {'versions': '1.42', 'upgrade_extensions': 'RatePage'} to all - SUCCESS in 14s [14:53:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:53:36] !log [@test151] starting deploy of {'config': True} to test151 [14:53:37] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [14:53:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:53:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:57:41] PROBLEM - www.pornwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.pornwiki.org' expires in 15 day(s) (Tue 06 Aug 2024 02:53:35 PM GMT +0000). [14:57:58] PROBLEM - lab.wikimimo.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'lab.wikimimo.com' expires in 15 day(s) (Tue 06 Aug 2024 02:54:57 PM GMT +0000). [14:58:10] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f51aa95d36eb...13404c37e1c2 [14:58:13] [02ssl] 07WikiTideSSLBot 0313404c3 - Bot: Update SSL cert for lab.wikimimo.com [14:59:10] PROBLEM - wiki.elgeis.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.elgeis.com' expires in 15 day(s) (Tue 06 Aug 2024 02:45:43 PM GMT +0000). [14:59:21] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/13404c37e1c2...0ef9e75bb148 [14:59:22] [02ssl] 07WikiTideSSLBot 030ef9e75 - Bot: Update SSL cert for wiki.elgeis.com [14:59:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.07, 22.03, 23.69 [15:02:00] PROBLEM - en.pornwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.pornwiki.org' expires in 15 day(s) (Tue 06 Aug 2024 02:53:35 PM GMT +0000). [15:03:45] PROBLEM - with.cpt-ra.bid - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'with.cpt-ra.bid' expires in 15 day(s) (Tue 06 Aug 2024 02:45:02 PM GMT +0000). [15:03:56] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0ef9e75bb148...df339c72e4ea [15:03:58] [02ssl] 07WikiTideSSLBot 03df339c7 - Bot: Update SSL cert for with.cpt-ra.bid [15:06:59] !log [@mwtask171] starting deploy of {'config': True} to all [15:07:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:07:11] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 12s [15:07:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:07:39] PROBLEM - wiki.drycleaninglobby.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.drycleaninglobby.net' expires in 15 day(s) (Tue 06 Aug 2024 02:53:18 PM GMT +0000). [15:07:53] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/df339c72e4ea...5a334434164e [15:07:54] [02ssl] 07WikiTideSSLBot 035a33443 - Bot: Update SSL cert for wiki.drycleaninglobby.net [15:08:57] PROBLEM - puritwiki.p-e.kr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'puritwiki.p-e.kr' expires in 15 day(s) (Tue 06 Aug 2024 02:43:02 PM GMT +0000). [15:09:09] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5a334434164e...9a2ed093d553 [15:09:12] [02ssl] 07WikiTideSSLBot 039a2ed09 - Bot: Update SSL cert for puritwiki.p-e.kr [15:09:49] PROBLEM - pornwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.pornwiki.org' expires in 15 day(s) (Tue 06 Aug 2024 02:53:35 PM GMT +0000). [15:11:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.99, 20.22, 21.44 [15:12:47] PROBLEM - chobots.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'chobots.wiki' expires in 15 day(s) (Tue 06 Aug 2024 02:51:45 PM GMT +0000). [15:12:58] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9a2ed093d553...812ea768ca46 [15:12:59] [02ssl] 07WikiTideSSLBot 03812ea76 - Bot: Update SSL cert for chobots.wiki [15:15:45] PROBLEM - wiki.fbpml.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.fbpml.org' expires in 15 day(s) (Tue 06 Aug 2024 02:45:27 PM GMT +0000). [15:15:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.40, 20.83, 21.57 [15:15:58] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/812ea768ca46...d0849cf4617e [15:15:59] [02ssl] 07WikiTideSSLBot 03d0849cf - Bot: Update SSL cert for wiki.fbpml.org [15:22:15] PROBLEM - wiki.mineplay.nl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mineplay.nl' expires in 15 day(s) (Tue 06 Aug 2024 02:52:48 PM GMT +0000). [15:22:27] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d0849cf4617e...631eb376b786 [15:22:28] [02ssl] 07WikiTideSSLBot 03631eb37 - Bot: Update SSL cert for wiki.mineplay.nl [15:23:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.35, 21.56, 21.27 [15:26:56] RECOVERY - lab.wikimimo.com - LetsEncrypt on sslhost is OK: OK - Certificate 'lab.wikimimo.com' will expire on Sat 19 Oct 2024 01:58:04 PM GMT +0000. [15:27:49] RECOVERY - wiki.elgeis.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.elgeis.com' will expire on Sat 19 Oct 2024 01:59:15 PM GMT +0000. [15:33:42] RECOVERY - with.cpt-ra.bid - LetsEncrypt on sslhost is OK: OK - Certificate 'with.cpt-ra.bid' will expire on Sat 19 Oct 2024 02:03:50 PM GMT +0000. [15:35:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.81, 22.20, 22.44 [15:37:39] RECOVERY - wiki.drycleaninglobby.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.drycleaninglobby.net' will expire on Sat 19 Oct 2024 02:07:45 PM GMT +0000. [15:38:25] RECOVERY - puritwiki.p-e.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'puritwiki.p-e.kr' will expire on Sat 19 Oct 2024 02:09:03 PM GMT +0000. [15:39:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.62, 22.84, 22.49 [15:41:34] RECOVERY - chobots.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'chobots.wiki' will expire on Sat 19 Oct 2024 02:12:52 PM GMT +0000. [15:41:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.57, 22.47, 22.38 [15:44:26] RECOVERY - wiki.cowgl.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cowgl.xyz' will expire on Sat 19 Oct 2024 01:45:14 PM GMT +0000. [15:44:31] RECOVERY - wiki.fbpml.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.fbpml.org' will expire on Sat 19 Oct 2024 02:15:51 PM GMT +0000. [15:45:16] RECOVERY - dragdown.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'dragdown.wiki' will expire on Sat 19 Oct 2024 01:47:25 PM GMT +0000. [15:45:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.27, 22.85, 22.42 [15:46:58] RECOVERY - wiki.jeremyboyd.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.jeremyboyd.org' will expire on Sat 19 Oct 2024 01:49:04 PM GMT +0000. [15:47:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.44, 22.01, 22.14 [15:51:29] RECOVERY - wiki.mineplay.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mineplay.nl' will expire on Sat 19 Oct 2024 02:22:19 PM GMT +0000. [15:55:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.25, 24.11, 22.87 [15:57:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.24, 22.37, 22.40 [16:03:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.45, 22.64, 22.24 [16:17:13] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:26:59] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.41, 19.95, 17.69 [16:28:59] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.16, 21.51, 18.51 [16:30:59] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.22, 20.99, 18.66 [16:32:59] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.76, 20.20, 18.68 [16:36:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [16:36:07] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [16:38:07] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [230 system event log (SEL) entries present] [17:06:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [17:29:50] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:31:45] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.85, 23.08, 20.76 [17:31:51] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.358 second response time [17:33:43] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.30, 21.89, 20.64 [17:34:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [17:37:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.59, 20.67, 15.68 [17:39:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.06, 21.75, 16.72 [17:39:39] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.07, 19.65, 20.17 [17:41:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 29.09, 24.41, 18.30 [17:43:50] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.16, 19.52, 15.59 [17:45:46] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.17, 19.23, 15.98 [17:49:21] [02CreateWiki] 07redbluegreenhat reviewed pull request 03#531 commit - 13https://github.com/miraheze/CreateWiki/pull/531#discussion_r1685785299 [17:49:39] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.75, 23.16, 18.27 [17:51:34] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.61, 21.66, 18.32 [17:53:30] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.37, 23.17, 19.23 [17:57:25] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.26, 21.15, 19.73 [17:59:17] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.74, 23.81, 21.05 [17:59:24] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.59, 19.57, 19.29 [18:01:12] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.66, 24.83, 21.72 [18:09:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.03, 23.29, 22.44 [18:14:25] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw181 [18:14:38] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw181 [18:14:42] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:16:24] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [18:18:32] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [18:21:02] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.35, 18.77, 20.19 [18:24:22] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 2 backends are down. mw152 mw181 [18:25:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.31, 21.59, 20.88 [18:26:21] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [18:31:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.45, 23.09, 22.10 [18:35:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.34, 23.63, 22.53 [18:37:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.74, 22.45, 22.26 [18:42:37] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [18:43:02] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 11.22, 17.15, 20.18 [18:44:38] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [231 system event log (SEL) entries present] [18:47:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.19, 22.41, 21.59 [18:51:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.02, 22.43, 21.87 [18:55:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.31, 22.65, 22.06 [18:59:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 17.07, 21.12, 21.74 [19:03:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.66, 23.59, 22.46 [19:03:38] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw152 [19:05:35] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [19:10:25] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:11:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.89, 23.60, 23.14 [19:11:13] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw152 [19:12:28] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.103 second response time [19:13:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.68, 23.41, 23.08 [19:13:10] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [19:15:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.16, 22.18, 22.69 [19:19:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.49, 24.75, 23.54 [19:21:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.67, 23.73, 23.32 [19:23:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.71, 24.36, 23.61 [19:24:58] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:27:05] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.855 second response time [19:27:52] PROBLEM - dcmultiverse.wikitide.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikitide.org' expires in 15 day(s) (Tue 06 Aug 2024 07:23:30 PM GMT +0000). [19:32:53] PROBLEM - polcompball.wikitide.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikitide.org' expires in 15 day(s) (Tue 06 Aug 2024 07:23:30 PM GMT +0000). [19:35:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.33, 23.82, 24.00 [19:36:05] PROBLEM - issue-tracker.wikitide.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikitide.org' expires in 15 day(s) (Tue 06 Aug 2024 07:23:30 PM GMT +0000). [19:36:32] PROBLEM - dc.wikitide.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikitide.org' expires in 15 day(s) (Tue 06 Aug 2024 07:23:30 PM GMT +0000). [19:37:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.02, 23.97, 23.99 [19:39:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.91, 22.28, 23.36 [19:40:42] PROBLEM - rippaverse.wikitide.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikitide.org' expires in 15 day(s) (Tue 06 Aug 2024 07:23:30 PM GMT +0000). [19:41:45] PROBLEM - wikitide.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikitide.org' expires in 15 day(s) (Tue 06 Aug 2024 07:23:30 PM GMT +0000). [19:45:33] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:49:02] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 15.47, 17.62, 20.37 [20:14:02] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [20:14:40] [02ManageWiki] 07songnguxyz opened pull request 03#475: Create an alias for special pages in Vietnamese - 13https://github.com/miraheze/ManageWiki/pull/475 [20:14:45] [02ManageWiki] 07coderabbitai[bot] commented on pull request 03#475: Create an alias for special pages in Vietnamese - 13https://github.com/miraheze/ManageWiki/pull/475#issuecomment-2241761867 [20:15:04] [02ManageWiki] 07songnguxyz commented on pull request 03#475: Create an alias for special pages in Vietnamese - 13https://github.com/miraheze/ManageWiki/pull/475#issuecomment-2241761945 [20:15:08] [02ManageWiki] 07coderabbitai[bot] commented on pull request 03#475: Create an alias for special pages in Vietnamese - 13https://github.com/miraheze/ManageWiki/pull/475#issuecomment-2241761954 [20:15:10] [02ManageWiki] 07coderabbitai[bot] edited a comment on pull request 03#475: Create an alias for special pages in Vietnamese - 13https://github.com/miraheze/ManageWiki/pull/475#issuecomment-2241761867 [20:16:47] [02ManageWiki] 07coderabbitai[bot] edited pull request 03#475: Create an alias for special pages in Vietnamese - 13https://github.com/miraheze/ManageWiki/pull/475 [20:17:02] [02ManageWiki] 07coderabbitai[bot] edited a comment on pull request 03#475: Create an alias for special pages in Vietnamese - 13https://github.com/miraheze/ManageWiki/pull/475#issuecomment-2241761867 [20:17:54] [02CreateWiki] 07songnguxyz opened pull request 03#544: Translate special page alias to Vietnamese - 13https://github.com/miraheze/CreateWiki/pull/544 [20:18:04] [02CreateWiki] 07coderabbitai[bot] commented on pull request 03#544: Translate special page alias to Vietnamese - 13https://github.com/miraheze/CreateWiki/pull/544#issuecomment-2241762598 [20:20:53] [02RequestSSL] 07songnguxyz opened pull request 03#53: Translate special page alias to Vietnamese - 13https://github.com/miraheze/RequestSSL/pull/53 [20:20:59] [02RequestSSL] 07coderabbitai[bot] commented on pull request 03#53: Translate special page alias to Vietnamese - 13https://github.com/miraheze/RequestSSL/pull/53#issuecomment-2241763228 [20:35:55] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.15, 20.86, 20.03 [20:37:50] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 18.42, 19.51, 19.61 [20:44:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [20:51:14] [02ManageWiki] 07OAuthority closed pull request 03#475: Create an alias for special pages in Vietnamese - 13https://github.com/miraheze/ManageWiki/pull/475 [20:51:15] [02ManageWiki] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/4aafe3fcbc01...da2c38332d5f [20:51:16] [02ManageWiki] 07songnguxyz 03da2c383 - Update ManageWikiAliases.php (#475) [20:51:29] miraheze/ManageWiki - songnguxyz the build passed. [20:54:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [20:56:12] miraheze/ManageWiki - OAuthority the build passed. [21:35:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.24, 19.87, 18.03 [21:37:02] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 19.08, 19.76, 18.23 [22:08:07] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.52, 22.32, 20.44 [22:11:58] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.66, 22.54, 21.13 [22:13:53] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.29, 23.03, 21.45 [22:19:40] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.84, 23.63, 22.43 [22:21:36] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.72, 24.32, 22.82 [22:24:24]