[00:00:09] !log sudo apt upgrade and reboot cp22 [00:00:13] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.84, 3.71, 3.71 [00:00:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:00:17] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:00:45] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:00:57] !log [universalomega@puppet141] Upgraded packages libgnutls30, libssl1.1, bind9-dnsutils, bind9-libs, bind9-host, curl, libcurl4, dnsutils, git-man, libcurl3-gnutls, git, libtiff5, linux-libc-dev, openssl, and sudo on ns1 [00:00:59] !log cp22 upgrades included kernal upgrade [00:01:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:01:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:01:11] RECOVERY - ns2 APT on ns2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [00:01:15] RECOVERY - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is OK: OK - NGINX Error Rate is 5% [00:02:13] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.07, 4.05, 3.82 [00:02:26] RECOVERY - ns1 APT on ns1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [00:03:48] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 16.65, 10.21, 5.65 [00:04:10] !log [universalomega@puppet141] Upgraded packages libgnutls30, libssl1.1, bind9-dnsutils, bind9-libs, bind9-host, curl, libcurl4, git-man, libcurl3-gnutls, git, libtiff5, linux-libc-dev, openssl, and sudo on prometheus131 [00:04:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:05:02] RECOVERY - prometheus131 APT on prometheus131 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [00:05:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.51, 10.79, 8.83 [00:06:17] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.62, 10.64, 8.85 [00:07:46] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.20, 9.53, 8.62 [00:08:12] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.73, 11.08, 9.23 [00:08:13] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.08, 3.68, 3.84 [00:08:26] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.2597161233 secs [00:09:35] RECOVERY - mw121 APT on mw121 is OK: APT OK: 25 packages available for upgrade (0 critical updates). [00:11:52] !log [universalomega@puppet141] Upgraded packages linux-libc-dev, dbus, libdbus-1-3, libexpat1-dev, libexpat1, libtiff5, libxml2, libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, libgnutls30, libk5crypto3, libkrb5support0, libkrb5-3, libgssapi-krb5-2, libssl-dev, libssl1.1, isc-dhcp-client, [00:11:53] isc-dhcp-common, bind9-dnsutils, bind9-libs, bind9-host, krb5-locales, curl, libcurl4, dbus-user-session, ffmpeg, libavdevice58, libavfilter7, libswscale5, libavformat58, libavcodec58, libswresample3, libpostproc55, libavresample4, libavutil56, git-man, libcurl3-gnutls, git, imagemagick-6.q16, imagemagick, libde265-0, libgraphicsmagick-q16-3, [00:11:54] libgraphicsmagick++-q16-12, libksba8, libmagick++-6.q16-8, libopenexr25, libmagickcore-6.q16-6-extra, nginx-core, nginx, libnginx-mod-stream-geoip, libnginx-mod-stream, libnginx-mod-mail, libnginx-mod-http-xslt-filter, libnginx-mod-http-geoip, libnginx-mod-http-image-filter, nginx-common, nodejs, libnode72, libpixman-1-dev, libpixman-1-0, nodejs-doc, openssl, [00:11:55] php7.4-zip, php7.4-xml, php7.4-readline, php7.4-opcache, php7.4-mysql, php7.4-mbstring, php7.4-ldap, php7.4-json, php7.4-intl, php7.4-gmp, php7.4-gd, php7.4-cli, php7.4-fpm, php7.4-dev, php7.4-dba, php7.4-curl, php7.4-bz2, php7.4-bcmath, php7.4-common, sudo, xserver-common, xserver-xorg-core, and xvfb on mw121 [00:11:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:13:37] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 2.64, 7.31, 6.71 [00:14:00] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.34, 10.17, 9.47 [00:14:13] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.38, 2.63, 3.32 [00:14:48] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.66, 3.87, 3.11 [00:15:21] PROBLEM - mw121 Puppet on mw121 is CRITICAL: CRITICAL: Puppet has 21 failures. Last run 4 minutes ago with 21 failures. Failed resources (up to 3 shown): Package[git],Package[openssl],Package[imagemagick],Package[ffmpeg] [00:15:35] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 1.89, 5.54, 6.13 [00:16:48] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.23, 3.94, 3.23 [00:20:48] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.51, 3.40, 3.19 [00:26:13] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.96, 3.60, 3.52 [00:26:26] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset -0.01406806707 secs [00:28:13] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.77, 3.10, 3.36 [00:32:13] PROBLEM - ping6 on cp32 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.04 ms [00:33:12] PROBLEM - matomo131 HTTPS on matomo131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.010 second response time [00:33:53] RECOVERY - cp23 APT on cp23 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:34:49] !log sudo apt upgrade and reboot cp23 (includes kernel upgrades) [00:35:13] RECOVERY - matomo131 HTTPS on matomo131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.000 second response time [00:36:20] RECOVERY - ping6 on cp32 is OK: PING OK - Packet loss = 0%, RTA = 115.99 ms [00:36:48] !log sudo apt upgrade and reboot cp23 (includes kernel upgrades) [00:37:56] PROBLEM - cp23 HTTPS on cp23 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.138 second response time [00:37:58] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is CRITICAL: CRITICAL - NGINX Error Rate is 95% [00:38:02] PROBLEM - cp23 Varnish Backends on cp23 is WARNING: No backends detected. If this is an error, see readme.txt [00:38:16] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 109.228.51.216/cpweb, 2a00:da00:1800:328::1/cpweb [00:38:30] RECOVERY - cp32 APT on cp32 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:38:42] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 109.228.51.216/cpweb, 2a00:da00:1800:328::1/cpweb [00:38:52] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:38:56] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.53, 5.36, 4.16 [00:40:22] !log sudo apt upgrade and reboot cp32 (includes kernel upgrades) [00:40:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:40:53] !log sudo apt upgrade and reboot cp33 (includes kernel upgrades) [00:40:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:41:34] PROBLEM - cp33 HTTPS on cp33 is CRITICAL: connect to address 2607:f1c0:1800:26f::1 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [00:42:43] PROBLEM - cp33 Varnish Backends on cp33 is WARNING: No backends detected. If this is an error, see readme.txt [00:42:44] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.79, 0.37, 0.14 [00:43:33] RECOVERY - mw121 Puppet on mw121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:43:38] PROBLEM - ping6 on cp32 is WARNING: PING WARNING - Packet loss = 0%, RTA = 143.10 ms [00:43:50] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [00:43:53] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 98% [00:43:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 9.46, 7.31, 5.09 [00:43:56] RECOVERY - cp23 HTTPS on cp23 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3626 bytes in 0.129 second response time [00:44:03] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.51, 6.89, 5.19 [00:44:25] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 10.37, 7.99, 5.67 [00:44:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.49, 7.85, 5.54 [00:44:38] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 19.36, 11.00, 6.82 [00:44:40] RECOVERY - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is OK: OK - NGINX Error Rate is 7% [00:47:37] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.28, 3.48, 3.08 [00:48:04] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 12.56, 9.96, 6.83 [00:48:57] PROBLEM - ping6 on cp33 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 115.39 ms [00:49:31] PROBLEM - cp33 Puppet on cp33 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:49:31] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.90, 3.20, 3.02 [00:49:48] RECOVERY - ping6 on cp32 is OK: PING OK - Packet loss = 0%, RTA = 119.05 ms [00:50:06] RECOVERY - ping6 on cp33 is OK: PING OK - Packet loss = 0%, RTA = 115.26 ms [00:50:51] !log [universalomega@puppet141] Upgraded packages libgnutls30 on reports121 [00:50:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:52:26] !log [universalomega@puppet141] Upgraded packages swift-container, swift-account, swift, python3-swift, libgnutls30, libk5crypto3, libkrb5support0, libkrb5-3, libgssapi-krb5-2, libssl1.1, libxml2, bind9-dnsutils, bind9-libs, bind9-host, krb5-locales, curl, libcurl4, git-man, libcurl3-gnutls, libexpat1-dev, libexpat1, git, libksba8, libpixman-1-0, libtiff5, [00:52:27] linux-libc-dev, openssl, and sudo on swiftac111 [00:52:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:52:39] RECOVERY - swiftac111 APT on swiftac111 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [00:52:44] RECOVERY - reports121 APT on reports121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:53:41] RECOVERY - cp33 Puppet on cp33 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [00:53:45] !log [universalomega@puppet141] Upgraded packages linux-libc-dev, swift-proxy, swift, python3-swift, libgnutls30, libk5crypto3, libkrb5support0, libkrb5-3, libgssapi-krb5-2, libssl1.1, libxml2, bind9-dnsutils, bind9-libs, bind9-host, krb5-locales, curl, libcurl4, git-man, libcurl3-gnutls, libexpat1-dev, libexpat1, git, libksba8, nginx-core, nginx, libnginx- [00:53:46] mod-stream-geoip, libnginx-mod-stream, libnginx-mod-mail, libnginx-mod-http-xslt-filter, libnginx-mod-http-geoip, libnginx-mod-http-image-filter, nginx-common, libpixman-1-0, libtiff5, openssl, and sudo on swiftproxy111 [00:53:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:54:10] RECOVERY - swiftproxy111 APT on swiftproxy111 is OK: APT OK: 23 packages available for upgrade (0 critical updates). [00:54:17] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.87, 7.76, 6.85 [00:54:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.50, 7.75, 6.91 [00:55:04] !log [universalomega@puppet141] Upgraded packages libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, libgnutls30, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, and libmagickcore-6.q16-6-extra on test131 [00:55:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:55:42] PROBLEM - swiftproxy111 Swift Proxy on swiftproxy111 is CRITICAL: connect to address 2a10:6740::6:201 and port 80: Connection refused [00:55:44] !log [universalomega@puppet141] Upgraded packages linux-libc-dev, dbus, libdbus-1-3, libexpat1-dev, libexpat1, swift-proxy, swift, python3-swift, libgnutls30, libk5crypto3, libkrb5support0, libkrb5-3, libgssapi-krb5-2, libssl1.1, isc-dhcp-client, isc-dhcp-common, libxml2, bind9-dnsutils, bind9-libs, bind9-host, krb5-locales, curl, libcurl4, git-man, [00:55:45] libcurl3-gnutls, git, libksba8, nginx-core, nginx, libnginx-mod-stream-geoip, libnginx-mod-stream, libnginx-mod-mail, libnginx-mod-http-xslt-filter, libnginx-mod-http-geoip, libnginx-mod-http-image-filter, nginx-common, libpixman-1-0, libtiff5, openssl, and sudo on swiftproxy131 [00:55:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:56:10] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 2.69, 3.55, 3.26 [00:56:14] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 3.06, 6.08, 6.34 [00:56:22] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 3.16, 6.26, 6.48 [00:56:22] RECOVERY - test131 APT on test131 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:56:32] PROBLEM - swiftproxy111 HTTP on swiftproxy111 is CRITICAL: connect to address 2a10:6740::6:201 and port 80: Connection refusedHTTP CRITICAL - Unable to open TCP socket [00:56:32] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 3.03, 6.12, 6.42 [00:56:39] RECOVERY - swiftproxy131 APT on swiftproxy131 is OK: APT OK: 23 packages available for upgrade (0 critical updates). [00:56:43] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [00:57:06] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is WARNING: WARNING - NGINX Error Rate is 57% [00:57:18] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:57:35] RECOVERY - cp33 HTTPS on cp33 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3628 bytes in 0.598 second response time [00:57:36] PROBLEM - swiftproxy111 HTTPS on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 502 Bad Gateway [00:57:36] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.30, 6.43, 7.31 [00:57:37] !log [universalomega@puppet141] Upgraded packages libgnutls30 on puppet141 [00:57:37] PROBLEM - swiftproxy131 HTTP on swiftproxy131 is CRITICAL: connect to address 2a10:6740::6:315 and port 80: Connection refusedHTTP CRITICAL - Unable to open TCP socket [00:57:38] RECOVERY - puppet141 APT on puppet141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:57:41] PROBLEM - swiftproxy131 Swift Proxy on swiftproxy131 is CRITICAL: connect to address 2a10:6740::6:315 and port 80: Connection refused [00:57:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:57:51] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.98, 3.65, 2.43 [00:58:05] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 5.16, 4.15, 3.51 [00:58:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 1.83, 5.91, 7.44 [00:58:16] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:58:34] PROBLEM - swiftproxy131 HTTPS on swiftproxy131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 502 Bad Gateway [00:59:05] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 67% [00:59:34] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 1.99, 4.97, 6.67 [00:59:48] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.75, 2.99, 2.34 [01:00:10] PROBLEM - ping6 on cp32 is WARNING: PING WARNING - Packet loss = 0%, RTA = 122.24 ms [01:00:56] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is WARNING: WARNING - NGINX Error Rate is 51% [01:01:03] RECOVERY - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is OK: OK - NGINX Error Rate is 33% [01:01:12] RECOVERY - swiftproxy111 HTTPS on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.038 second response time [01:01:42] RECOVERY - swiftproxy111 Swift Proxy on swiftproxy111 is OK: TCP OK - 0.001 second response time on 2a10:6740::6:201 port 80 [01:01:44] !log reboot swiftproxy111 [01:01:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:02:03] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.68, 4.96, 6.65 [01:02:13] PROBLEM - ping6 on cp32 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 196.82 ms [01:02:32] RECOVERY - swiftproxy111 HTTP on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.016 second response time [01:02:50] RECOVERY - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is OK: OK - NGINX Error Rate is 32% [01:03:03] PROBLEM - ping6 on cp32 is WARNING: PING WARNING - Packet loss = 0%, RTA = 174.43 ms [01:03:48] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 2.72, 4.00, 3.71 [01:04:17] !log reboot swiftproxy131 [01:04:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:05:23] RECOVERY - swiftproxy131 HTTP on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.256 second response time [01:05:41] RECOVERY - swiftproxy131 Swift Proxy on swiftproxy131 is OK: TCP OK - 0.001 second response time on 2a10:6740::6:315 port 80 [01:05:42] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.56, 4.01, 3.73 [01:06:34] RECOVERY - swiftproxy131 HTTPS on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.025 second response time [01:07:10] PROBLEM - ping6 on cp32 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 181.74 ms [01:07:21] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 10.84, 8.05, 7.19 [01:07:37] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.50, 3.75, 3.66 [01:07:53] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.81, 7.81, 7.35 [01:08:03] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.63, 7.19, 6.26 [01:09:14] PROBLEM - ping6 on cp32 is WARNING: PING WARNING - Packet loss = 0%, RTA = 151.98 ms [01:10:03] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 10.43, 8.39, 6.81 [01:11:17] PROBLEM - ping6 on cp32 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 182.21 ms [01:12:03] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.10, 7.99, 6.87 [01:13:13] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.06, 7.66, 7.36 [01:13:19] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.24, 2.98, 3.37 [01:13:21] PROBLEM - ping6 on cp32 is WARNING: PING WARNING - Packet loss = 0%, RTA = 133.40 ms [01:15:10] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.41, 7.61, 7.36 [01:17:07] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.00, 7.25, 7.28 [01:17:33] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.65, 7.84, 7.69 [01:18:01] [02mediawiki] 07dependabot[bot] opened pull request 03#13305: Bump skins/Citizen from `4b0f0e2` to `33421ea` - 13https://github.com/miraheze/mediawiki/pull/13305 [01:18:03] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_39/skins/Citizen-33421ea [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e738cacfe082 [01:18:06] [02miraheze/mediawiki] 07dependabot[bot] 03e738cac - Bump skins/Citizen from `4b0f0e2` to `33421ea` [01:18:07] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_39/skins/Citizen-33421ea - 13https://github.com/miraheze/mediawiki [01:18:12] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_39/extensions/DiscordNotifications-555a572 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ebe6f12d72f4 [01:18:13] [02miraheze/mediawiki] 07dependabot[bot] 03ebe6f12 - Bump extensions/DiscordNotifications from `6ff8b68` to `555a572` [01:18:14] [02mediawiki] 07dependabot[bot] opened pull request 03#13306: Bump extensions/DiscordNotifications from `6ff8b68` to `555a572` - 13https://github.com/miraheze/mediawiki/pull/13306 [01:18:18] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_39/extensions/DiscordNotifications-555a572 - 13https://github.com/miraheze/mediawiki [01:18:19] [02mediawiki] 07github-actions[bot] labeled pull request 03#13305: Bump skins/Citizen from `4b0f0e2` to `33421ea` - 13https://github.com/miraheze/mediawiki/pull/13305 [01:18:20] [02mediawiki] 07github-actions[bot] labeled pull request 03#13305: Bump skins/Citizen from `4b0f0e2` to `33421ea` - 13https://github.com/miraheze/mediawiki/pull/13305 [01:19:05] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 9.96, 8.19, 7.61 [01:19:29] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 10.51, 8.88, 8.08 [01:19:32] PROBLEM - ping6 on cp32 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 278.85 ms [01:20:04] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.46, 8.31, 7.33 [01:20:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.08, 8.33, 6.84 [01:21:04] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 5.94, 7.61, 7.49 [01:21:35] PROBLEM - ping6 on cp32 is WARNING: PING WARNING - Packet loss = 0%, RTA = 170.60 ms [01:22:04] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.97, 7.95, 7.33 [01:22:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.81, 7.49, 6.70 [01:22:25] [02mediawiki] 07Universal-Omega closed pull request 03#13305: Bump skins/Citizen from `4b0f0e2` to `33421ea` - 13https://github.com/miraheze/mediawiki/pull/13305 [01:22:27] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_39/skins/Citizen-33421ea [01:22:29] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/353c160ba699...8ba4a69f2e61 [01:22:31] [02miraheze/mediawiki] 07dependabot[bot] 038ba4a69 - Bump skins/Citizen from `4b0f0e2` to `33421ea` (#13305) [01:22:33] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_39/skins/Citizen-33421ea - 13https://github.com/miraheze/mediawiki [01:22:36] [02mediawiki] 07Universal-Omega labeled pull request 03#13306: Bump extensions/DiscordNotifications from `6ff8b68` to `555a572` - 13https://github.com/miraheze/mediawiki/pull/13306 [01:22:41] [02mediawiki] 07Universal-Omega closed pull request 03#13306: Bump extensions/DiscordNotifications from `6ff8b68` to `555a572` - 13https://github.com/miraheze/mediawiki/pull/13306 [01:22:42] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/8ba4a69f2e61...ac440a431039 [01:22:44] [02miraheze/mediawiki] 07dependabot[bot] 03ac440a4 - Bump extensions/DiscordNotifications from `6ff8b68` to `555a572` (#13306) [01:22:46] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_39/extensions/DiscordNotifications-555a572 - 13https://github.com/miraheze/mediawiki [01:22:49] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_39/extensions/DiscordNotifications-555a572 [01:23:25] !log [universalomega@mwtask141] starting deploy of {'world': True, 'l10n': True} to all [01:23:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:23:39] PROBLEM - ping6 on cp32 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 186.07 ms [01:23:53] !log [universalomega@test131] starting deploy of {'world': True, 'l10n': True} to all [01:23:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:24:05] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 121.64 ms [01:25:17] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.57, 7.19, 7.58 [01:25:43] PROBLEM - ping6 on cp32 is WARNING: PING WARNING - Packet loss = 0%, RTA = 179.37 ms [01:26:03] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 4.71, 6.31, 6.80 [01:26:08] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 117.36 ms [01:26:22] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 4.98, 6.37, 6.43 [01:28:29] !log [universalomega@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw121.miraheze.org [01:28:33] !log [universalomega@test131] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 279s [01:28:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:28:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:29:14] !log [universalomega@mwtask141] starting deploy of {'world': True} to all [01:29:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:29:51] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.95, 11.59, 9.65 [01:31:05] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.25, 6.93, 7.26 [01:31:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.11, 11.78, 10.07 [01:31:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.63, 11.19, 9.74 [01:31:53] PROBLEM - ping6 on cp32 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 182.72 ms [01:32:03] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.29, 7.62, 7.15 [01:32:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.25, 6.98, 6.68 [01:33:02] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.14, 6.84, 7.18 [01:33:57] PROBLEM - ping6 on cp32 is WARNING: PING WARNING - Packet loss = 0%, RTA = 160.56 ms [01:34:03] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 5.92, 6.95, 6.96 [01:34:22] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.06, 6.23, 6.43 [01:35:04] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.16, 6.19, 6.68 [01:35:23] !log [universalomega@mwtask141] finished deploy of {'world': True} to all - SUCCESS in 368s [01:35:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:37:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.52, 11.18, 10.48 [01:37:44] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.87, 10.04, 9.73 [01:38:03] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.37, 6.23, 6.67 [01:39:02] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.98, 7.51, 7.26 [01:42:11] RECOVERY - ping6 on cp32 is OK: PING OK - Packet loss = 0%, RTA = 115.65 ms [01:43:21] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.67, 8.93, 9.78 [01:44:31] [02puppet] 07Universal-Omega closed pull request 03#3143: csp: remove `www.recaptcha.net` - 13https://github.com/miraheze/puppet/pull/3143 [01:44:34] [02miraheze/puppet] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [01:44:36] [02puppet] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/puppet [01:44:37] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ff4b0a29bedf...65bf9b45a0c7 [01:44:39] [02miraheze/puppet] 07Universal-Omega 0365bf9b4 - csp: remove `www.recaptcha.net` (#3143) [01:44:41] [02puppet] 07Universal-Omega closed pull request 03#3142: mediawiki: remove `$wgReCaptchaSecretKey` from PrivateSettings.php - 13https://github.com/miraheze/puppet/pull/3142 [01:44:44] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/65bf9b45a0c7...181a50e2fc3c [01:44:46] [02miraheze/puppet] 07Universal-Omega 03181a50e - mediawiki: remove `$wgReCaptchaSecretKey` from PrivateSettings.php (#3142) [01:44:47] [02miraheze/puppet] 07Universal-Omega deleted branch 03remove-wgReCaptchaSecretKey [01:44:49] [02puppet] 07Universal-Omega deleted branch 03remove-wgReCaptchaSecretKey - 13https://github.com/miraheze/puppet [01:46:01] [02MirahezeMagic] 07Universal-Omega edited pull request 03#401: Remove ReCaptchaNoCaptcha - 13https://github.com/miraheze/MirahezeMagic/pull/401 [01:46:11] [02MirahezeMagic] 07Universal-Omega edited pull request 03#401: Remove ReCaptchaNoCaptcha - 13https://github.com/miraheze/MirahezeMagic/pull/401 [01:46:24] [02MirahezeMagic] 07Universal-Omega closed pull request 03#401: Remove ReCaptchaNoCaptcha - 13https://github.com/miraheze/MirahezeMagic/pull/401 [01:46:25] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-50/±1] 13https://github.com/miraheze/MirahezeMagic/compare/5cfd3b5c2067...a5f0d2c3a1b1 [01:46:26] [02miraheze/MirahezeMagic] 07Universal-Omega 03a5f0d2c - Remove ReCaptchaNoCaptcha (#401) [01:46:29] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03remove-ReCaptchaNoCaptcha [01:46:31] [02MirahezeMagic] 07Universal-Omega deleted branch 03remove-ReCaptchaNoCaptcha - 13https://github.com/miraheze/MirahezeMagic [01:47:02] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 5.95, 7.54, 7.48 [01:47:46] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 13889 MB (10% inode=98%); [01:50:26] !log [universalomega@puppet141] Remove passwords::recaptcha::secretkey from private puppet [01:50:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:50:33] PROBLEM - ping6 on cp32 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 211.04 ms [01:51:10] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 23.77, 10.78, 5.79 [01:53:03] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 4.36, 5.60, 6.67 [01:54:40] RECOVERY - ping6 on cp32 is OK: PING OK - Packet loss = 0%, RTA = 103.58 ms [01:55:04] miraheze/MirahezeMagic - Universal-Omega the build passed. [01:58:49] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.12, 3.16, 2.89 [02:00:49] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.74, 3.01, 2.87 [02:03:10] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.63, 7.64, 7.11 [02:05:10] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 11.78, 8.65, 7.50 [02:06:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.28, 10.54, 9.90 [02:08:14] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.49, 9.96, 9.77 [02:09:10] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 4.86, 7.09, 7.14 [02:13:10] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 25.97, 12.75, 9.13 [02:13:46] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7744 MB (5% inode=98%); [02:39:10] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.64, 5.29, 7.97 [02:45:10] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 3.67, 4.36, 6.71 [02:53:10] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 6.90, 8.00, 7.48 [02:59:10] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 3.57, 5.28, 6.47 [03:09:46] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 15008 MB (11% inode=98%); [03:13:46] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 13171 MB (9% inode=98%); [03:15:10] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 8.73, 7.58, 6.39 [03:17:10] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 7.93, 7.58, 6.52 [03:19:10] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 16.89, 11.90, 8.30 [03:23:10] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 4.46, 7.54, 7.29 [03:27:10] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 4.39, 5.71, 6.59 [03:31:10] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 6.77, 8.22, 7.65 [03:31:46] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 17413 MB (13% inode=98%); [03:31:48] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [03:33:10] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 4.65, 7.12, 7.32 [03:39:10] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 1.31, 4.46, 6.30 [04:01:37] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:31:25] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [06:25:52] !log DELETED AND DROPPED (yesterday, around 21:00 UTC) https://phabricator.miraheze.org/P488 [06:25:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:44:42] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.2308538854 secs [06:50:42] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.01755464077 secs [07:28:19] !log DELETED and DROPPED https://phabricator.miraheze.org/P489 [07:28:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:50:25] PROBLEM - swiftproxy131 NTP time on swiftproxy131 is WARNING: NTP WARNING: Offset 0.1062767804 secs [09:03:19] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1497635245 secs [09:07:05] RECOVERY - swiftproxy131 NTP time on swiftproxy131 is OK: NTP OK: Offset 0.09809464216 secs [09:33:19] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.0768340528 secs [11:35:19] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1176846027 secs [11:47:19] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.08229714632 secs [12:10:35] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/landing/compare/d294b2b157ee...91e07dacd090 [12:10:38] [02miraheze/landing] 07translatewiki 0391e07da - Localisation updates from https://translatewiki.net. [12:10:40] [02miraheze/ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/1c618f76593f...ddb134085f57 [12:10:42] [02miraheze/ImportDump] 07translatewiki 03ddb1340 - Localisation updates from https://translatewiki.net. [12:10:44] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/4809f34b5f1c...df0c0b69ed0d [12:10:46] [02miraheze/ManageWiki] 07translatewiki 03df0c0b6 - Localisation updates from https://translatewiki.net. [12:10:48] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±10] 13https://github.com/miraheze/CreateWiki/compare/375685c87079...36824cda20d5 [12:10:51] [02miraheze/CreateWiki] 07translatewiki 0336824cd - Localisation updates from https://translatewiki.net. [12:10:53] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±29] 13https://github.com/miraheze/MirahezeMagic/compare/a5f0d2c3a1b1...cce2b7478a1a [12:10:54] [02miraheze/MirahezeMagic] 07translatewiki 03cce2b74 - Localisation updates from https://translatewiki.net. [12:13:09] miraheze/landing - translatewiki the build passed. [12:16:26] miraheze/ManageWiki - translatewiki the build passed. [12:18:46] miraheze/ImportDump - translatewiki the build passed. [12:21:27] miraheze/CreateWiki - translatewiki the build passed. [12:23:59] miraheze/MirahezeMagic - translatewiki the build passed. [13:23:12] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/0c0e2ad5d30d...0cd4f861ce96 [13:23:15] [02miraheze/ssl] 07MirahezeSSLBot 030cd4f86 - Bot: Add SSL cert for nijigen.wiki [13:24:21] miraheze/ssl - MirahezeSSLBot the build has errored. [13:34:50] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.17, 9.61, 8.68 [13:36:50] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.94, 9.41, 8.73 [14:04:37] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.36, 9.29, 8.54 [14:06:37] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.59, 9.57, 8.75 [14:35:12] !log RENAMED nightwindrowiki to nightwindwiki [14:35:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:41:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.83, 9.77, 8.92 [14:43:34] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.89, 9.16, 8.80 [14:55:37] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [14:57:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.15, 10.31, 9.52 [14:59:03] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.34, 11.09, 9.91 [14:59:37] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 7 = Critical] [15:00:59] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.02, 10.91, 9.98 [15:05:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.47, 11.71, 10.27 [15:06:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.87, 9.78, 9.78 [15:07:23] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.13, 11.49, 10.33 [15:09:18] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.25, 10.86, 10.22 [15:09:39] PROBLEM - www.lab612.at - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.lab612.at All nameservers failed to answer the query. [15:15:02] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.38, 10.18, 10.15 [15:28:37] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.89, 10.07, 9.68 [15:31:38] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=akudamadrivewiki --no-updates --username-prefix=wikia:akudamadrive /home/reception/ImportDump/akudamadrivewiki-20230216152255.xml (START) [15:31:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:34:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/1e9ed326302a [15:34:25] [02miraheze/puppet] 07paladox 031e9ed32 - swift: set nodelaycompress instead of delaycompress for logrotate [15:34:26] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/puppet [15:34:28] [02puppet] 07paladox opened pull request 03#3156: swift: set nodelaycompress instead of delaycompress for logrotate - 13https://github.com/miraheze/puppet/pull/3156 [15:34:32] [02puppet] 07paladox closed pull request 03#3156: swift: set nodelaycompress instead of delaycompress for logrotate - 13https://github.com/miraheze/puppet/pull/3156 [15:34:33] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [15:34:34] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/puppet [15:34:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/181a50e2fc3c...7156eb903a50 [15:34:36] [02miraheze/puppet] 07paladox 037156eb9 - swift: set nodelaycompress instead of delaycompress for logrotate (#3156) [15:34:37] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.35, 11.93, 10.51 [15:36:26] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.57, 11.50, 10.19 [15:36:37] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.79, 11.61, 10.56 [15:38:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.11, 10.82, 10.10 [15:39:15] PROBLEM - www.lab612.at - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query lab612.at. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:39:59] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=akudamadrivewiki --no-updates --username-prefix=wikia:akudamadrive /home/reception/ImportDump/akudamadrivewiki-20230216152255.xml (END - exit=0) [15:40:00] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=akudamadrivewiki (START) [15:40:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:40:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:40:18] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.06, 11.80, 10.50 [15:42:16] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=akudamadrivewiki (END - exit=0) [15:42:17] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=akudamadrivewiki --active --update (END - exit=0) [15:42:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:42:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:42:37] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.82, 12.48, 11.16 [15:48:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.82, 11.77, 11.32 [15:48:37] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.05, 11.23, 11.21 [15:49:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.73, 12.34, 11.54 [15:52:37] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.35, 12.27, 11.59 [15:53:57] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.03, 3.10, 2.45 [15:55:56] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 3.38, 3.28, 2.61 [16:03:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.20, 10.97, 11.77 [16:08:51] RECOVERY - www.lab612.at - reverse DNS on sslhost is OK: SSL OK - www.lab612.at reverse DNS resolves to cp22.miraheze.org - CNAME OK [16:12:37] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.80, 10.92, 11.93 [16:18:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.10, 10.21, 10.62 [16:20:21] [02mw-config] 07Reception123 closed pull request 03#5081: add RemoveRedlinks extension per request T10310 - 13https://github.com/miraheze/mw-config/pull/5081 [16:20:22] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-3 [16:20:23] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-3 - 13https://github.com/miraheze/mw-config [16:20:25] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/088eefaff983...e83b85e42d07 [16:20:26] [02miraheze/mw-config] 07Reception123 03e83b85e - add RemoveRedlinks extension per request T10310 (#5081) [16:20:39] !log [reception@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [16:20:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:20:49] !log [reception@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [16:20:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.23, 9.56, 10.33 [16:20:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:21:27] miraheze/mw-config - Reception123 the build passed. [16:24:44] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.23, 9.35, 10.12 [16:27:03] !log [@test131] starting deploy of {'config': True} to all [16:27:04] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [16:27:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:27:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:30:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.37, 10.20, 10.23 [16:34:37] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.32, 11.10, 10.93 [16:36:37] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.97, 10.38, 10.70 [16:36:38] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.77, 10.01, 10.20 [16:40:37] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.06, 8.37, 9.79 [16:49:19] !log reception@mwtask141:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/findBadBlobs.php --wiki=stampencyclopediawiki --revisions=17573,17574,17575,17576,17577,17578,17579,17580,17581,17582,17583 --mark=corrupt [16:49:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:52:38] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.17, 10.66, 9.83 [16:56:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.13, 11.26, 10.29 [17:00:34] [02mw-config] 07Reception123 closed pull request 03#5089: T10323: Add more links to Special:AdminLinks for paneidoversewiki - 13https://github.com/miraheze/mw-config/pull/5089 [17:00:36] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e83b85e42d07...bdca23768978 [17:00:36] !log [reception@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [17:00:37] [02miraheze/mw-config] 07redbluegreenhat 03bdca237 - T10323: Add more links to Special:AdminLinks for paneidoversewiki (#5089) [17:00:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:00:45] !log [reception@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [17:00:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:01:35] miraheze/mw-config - Reception123 the build passed. [17:05:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.88, 11.80, 10.37 [17:07:44]