[00:00:10] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:00:13] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:00:59] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09999719262 secs [00:02:10] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:02:14] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:04:51] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 21427MiB (24% inode=98%); [00:04:59] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1014699042 secs [00:05:36] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 21436MiB (24% inode=98%); [00:38:22] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/191f3f3c6fa9...366a64167acc [00:38:24] [02miraheze/dns] 07Universal-Omega 03366a641 - Update comment [00:48:59] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.0981528163 secs [01:00:00] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/4b54b196ca86...f6a90bb2c802 [01:00:03] [02miraheze/ImportDump] 07Universal-Omega 03f6a90bb - Readd to DefaultUserOptions [01:08:41] miraheze/ImportDump - Universal-Omega the build passed. [01:14:04] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e88f965d7292...631d9b7b32e3 [01:14:06] [02miraheze/ssl] 07Universal-Omega 03631d9b7 - Add path_redirects for wiki.surf [01:15:32] PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 3.35, 1.96, 1.51 [01:19:23] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.13, 1.71, 1.53 [01:21:21] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 0.94, 1.41, 1.44 [01:41:00] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [02:09:54] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.gab.pt.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:38:46] RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:00:32] RECOVERY - db171 Backups SQL on db171 is OK: FILE_AGE OK: /var/log/sql-backup.log is 31 seconds old and 0 bytes [03:03:58] PROBLEM - db162 Backups SQL on db162 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864224 seconds old and 1218 bytes [03:14:28] PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 2.90, 2.38, 1.56 [03:16:26] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 0.80, 1.78, 1.44 [03:18:24] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 1.03, 1.55, 1.39 [05:18:32] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1050085723 secs [06:19:12] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 326.38 ms [06:20:01] PROBLEM - kodiak.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kodiak.wiki' expires in 15 day(s) (Wed 03 Apr 2024 06:00:59 AM GMT +0000). [06:20:16] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/631d9b7b32e3...4745a328528e [06:20:18] [02miraheze/ssl] 07MirahezeSSLBot 034745a32 - Bot: Update SSL cert for kodiak.wiki [06:21:15] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 295.53 ms [06:40:33] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09955686331 secs [07:03:08] PROBLEM - cloud16 Puppet on cloud16 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [07:08:32] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1028844416 secs [07:18:01] RECOVERY - kodiak.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kodiak.wiki' will expire on Sun 16 Jun 2024 05:20:09 AM GMT +0000. [07:31:08] RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:33:52] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [07:56:32] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.0998197794 secs [08:00:32] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1013212204 secs [08:02:46] RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:06:32] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09828060865 secs [08:46:45] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:44:39] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [11:42:32] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1012306511 secs [12:13:19] [02miraheze/ErrorPages] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/5e4b620dcfa2...b60dca5a34f0 [12:13:22] [02miraheze/ErrorPages] 07translatewiki 03b60dca5 - Localisation updates from https://translatewiki.net. [12:13:23] [02miraheze/ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/ImportDump/compare/f6a90bb2c802...b42756516de0 [12:13:25] [02miraheze/ImportDump] 07translatewiki 03b427565 - Localisation updates from https://translatewiki.net. [12:13:28] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/compare/6e5610af7d48...526baf01b0e1 [12:13:29] [02miraheze/CreateWiki] 07translatewiki 03526baf0 - Localisation updates from https://translatewiki.net. [12:13:31] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/landing/compare/0e4ce697cd7a...06e0daa7bd6e [12:13:32] [02miraheze/landing] 07translatewiki 0306e0daa - Localisation updates from https://translatewiki.net. [12:13:35] [02miraheze/DataDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/DataDump/compare/e50065ed7d48...7ff8b9eeae56 [12:13:37] [02miraheze/DataDump] 07translatewiki 037ff8b9e - Localisation updates from https://translatewiki.net. [12:13:38] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±45] 13https://github.com/miraheze/MirahezeMagic/compare/2024db371dfe...0ba4ace854cd [12:13:41] [02miraheze/MirahezeMagic] 07translatewiki 030ba4ace - Localisation updates from https://translatewiki.net. [12:14:09] miraheze/landing - translatewiki the build passed. [12:14:13] miraheze/ErrorPages - translatewiki the build passed. [12:18:00] miraheze/DataDump - translatewiki the build passed. [12:21:07] miraheze/CreateWiki - translatewiki the build passed. [12:21:11] miraheze/ImportDump - translatewiki the build passed. [12:24:02] !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 [12:24:03] !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:11] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [12:24:12] !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:27:41] miraheze/MirahezeMagic - translatewiki the build passed. [12:32:23] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [12:32:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:32:32] !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 9s [12:32:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:00:36] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 19.12, 20.97, 9.65 [13:04:36] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 0.56, 9.52, 7.49 [14:15:12] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:08:32] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09978595376 secs [15:34:32] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1008289754 secs [15:43:46] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [15:46:32] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09763932228 secs [15:53:27] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1088651717 secs [16:10:51] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09220173955 secs [16:13:11] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [16:14:45] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1066860259 secs [16:20:22] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 178.81 ms [16:20:34] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.0919585824 secs [16:22:25] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 182.85 ms [16:24:33] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1058965623 secs [16:31:52] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 182.14 ms [16:33:52] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [16:40:07] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 175.04 ms [16:55:46] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 176.50 ms [16:57:48] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 180.31 ms [17:02:00] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 183.00 ms [17:02:46] RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK [17:04:02] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 176.74 ms [17:10:19] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 179.46 ms [17:11:50] PROBLEM - ru-teirailway.f5.si - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ru-teirailway.f5.si' expires in 15 day(s) (Wed 03 Apr 2024 05:01:28 PM GMT +0000). [17:12:03] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4745a328528e...1aa2871f2b4f [17:12:06] [02miraheze/ssl] 07MirahezeSSLBot 031aa2871 - Bot: Update SSL cert for ru-teirailway.f5.si [17:12:21] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 178.20 ms [17:18:32] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09994378686 secs [17:26:10] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/366a64167acc...4847c967eacc [17:26:13] [02miraheze/dns] 07Universal-Omega 034847c96 - Point phorge to issue-tracker [17:29:57] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on bots171 [17:30:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:30:14] RECOVERY - bots171 APT on bots171 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [17:32:39] PROBLEM - wiki.barengreza.my.id - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.barengreza.my.id' expires in 15 day(s) (Wed 03 Apr 2024 05:11:46 PM GMT +0000). [17:33:01] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1aa2871f2b4f...c92d987d6d6a [17:33:04] [02miraheze/ssl] 07MirahezeSSLBot 03c92d987 - Bot: Update SSL cert for wiki.barengreza.my.id [17:38:25] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 179.02 ms [17:41:10] RECOVERY - ru-teirailway.f5.si - LetsEncrypt on sslhost is OK: OK - Certificate 'ru-teirailway.f5.si' will expire on Sun 16 Jun 2024 04:11:56 PM GMT +0000. [17:42:34] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 177.31 ms [17:47:18] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[gdnsd-syntax] [17:48:47] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 182.99 ms [17:50:50] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 182.53 ms [17:55:29] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/4847c967eacc...f972cb8b5a2e [17:55:30] [02miraheze/dns] 07Universal-Omega 03f972cb8 - Fix [17:56:03] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[gdnsd-syntax] [17:57:05] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 171.20 ms [17:58:31] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 54% [17:59:08] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.19 ms [17:59:10] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:59:54] RECOVERY - phorge.miraheze.org - reverse DNS on sslhost is OK: SSL OK - phorge.miraheze.org reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [18:00:04] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:01:31] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on db171 [18:01:36] RECOVERY - db171 APT on db171 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [18:01:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:02:08] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on db151 [18:02:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:02:17] RECOVERY - wiki.barengreza.my.id - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.barengreza.my.id' will expire on Sun 16 Jun 2024 04:32:45 PM GMT +0000. [18:02:23] RECOVERY - db151 APT on db151 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [18:03:10] !log [universalomega@puppet181] Upgraded packages squid-common, squid, libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on bast161 [18:03:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:03:33] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on graylog161 [18:03:37] RECOVERY - graylog161 APT on graylog161 is OK: APT OK: 31 packages available for upgrade (0 critical updates). [18:03:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:03:49] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on mem161 [18:03:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:04:03] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, composer, libpq5, and pdns-recursor on matomo151 [18:04:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:04:32] RECOVERY - bast161 APT on bast161 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:04:40] RECOVERY - mem161 APT on mem161 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:04:47] RECOVERY - matomo151 APT on matomo151 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:04:52] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on mem151 [18:04:54] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [18:05:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:05:03] !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 9s [18:05:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:05:09] RECOVERY - mem151 APT on mem151 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:05:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:05:26] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, composer, libpq5, and pdns-recursor on jobchron171 [18:05:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:05:48] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on cp36 [18:05:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:06:16] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on db182 [18:06:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:06:34] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on ldap171 [18:06:39] RECOVERY - jobchron171 APT on jobchron171 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:06:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:06:53] RECOVERY - cp36 APT on cp36 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:07:23] !log [universalomega@puppet181] Upgraded packages libunbound8, libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on cloud16 [18:07:23] RECOVERY - db182 APT on db182 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [18:07:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:08:08] RECOVERY - ldap171 APT on ldap171 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:08:24] RECOVERY - cloud16 APT on cloud16 is OK: APT OK: 52 packages available for upgrade (0 critical updates). [18:10:59] !log [universalomega@puppet181] Upgraded packages libunbound8, libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on cloud15 [18:11:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:11:18] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on cp37 [18:11:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:11:34] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on db181 [18:11:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:12:25] RECOVERY - cp37 APT on cp37 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:12:26] !log [universalomega@puppet181] Upgraded packages squid-common, squid, libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on bast181 [18:12:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:12:50] RECOVERY - cloud15 APT on cloud15 is OK: APT OK: 55 packages available for upgrade (0 critical updates). [18:13:00] RECOVERY - db181 APT on db181 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [18:13:22] RECOVERY - bast181 APT on bast181 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:14:11] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on db161 [18:14:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:14:28] !log [universalomega@puppet181] Upgraded packages libunbound8, libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on cloud17 [18:14:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:14:44] !log [universalomega@puppet181] Upgraded packages libunbound8, libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on cloud18 [18:14:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:14:53] RECOVERY - db161 APT on db161 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [18:14:59] RECOVERY - cloud18 APT on cloud18 is OK: APT OK: 55 packages available for upgrade (0 critical updates). [18:15:13] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on cp26 [18:15:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:15:34] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on cp27 [18:15:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:15:41] RECOVERY - cloud17 APT on cloud17 is OK: APT OK: 52 packages available for upgrade (0 critical updates). [18:15:54] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on prometheus151 [18:16:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:16:05] RECOVERY - cp26 APT on cp26 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:16:08] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on os161 [18:16:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:16:22] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on os151 [18:16:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:16:40] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on mon181 [18:16:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:16:52] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 181.60 ms [18:16:53] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on rdb151 [18:17:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:17:03] RECOVERY - os161 APT on os161 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [18:17:08] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on os162 [18:17:10] RECOVERY - cp27 APT on cp27 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:17:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:17:24] RECOVERY - os151 APT on os151 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [18:17:28] RECOVERY - rdb151 APT on rdb151 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:17:34] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:17:45] !log [universalomega@puppet181] Upgraded packages libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, libuv1, bind9-host, bind9-dnsutils, bind9-libs, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, libmagickcore-6.q16-6-extra, libpq5, and pdns-recursor on mw152 [18:17:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:18:17] RECOVERY - mw152 APT on mw152 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:18:20] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, composer, libpq5, and pdns-recursor on reports171 [18:18:24] RECOVERY - mon181 APT on mon181 is OK: APT OK: 32 packages available for upgrade (0 critical updates). [18:18:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:18:28] RECOVERY - os162 APT on os162 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [18:18:29] RECOVERY - reports171 APT on reports171 is OK: APT OK: 30 packages available for upgrade (0 critical updates). [18:18:49] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, postgresql-client-15, and postgresql-15 on puppet181 [18:18:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:19:40] RECOVERY - puppet181 APT on puppet181 is OK: APT OK: 33 packages available for upgrade (0 critical updates). [18:20:50] !log [universalomega@puppet181] Upgraded packages libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, libuv1, bind9-host, bind9-dnsutils, bind9-libs, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, libmagickcore-6.q16-6-extra, libpq5, and pdns-recursor on mw162 [18:20:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:21:19] !log [universalomega@puppet181] Upgraded packages libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, libuv1, bind9-host, bind9-dnsutils, bind9-libs, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, libmagickcore-6.q16-6-extra, libpq5, and pdns-recursor on mw151 [18:21:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:21:37] !log [universalomega@puppet181] Upgraded packages libuv1 on mwtask171 [18:21:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:21:51] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [18:21:58] !log [universalomega@puppet181] Upgraded packages libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, libuv1, bind9-host, bind9-dnsutils, bind9-libs, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, libmagickcore-6.q16-6-extra, libpq5, and pdns-recursor on mw181 [18:22:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:22:19] [02puppet] 07Reception123 closed pull request 03#3624: ssl-certificate use pyyaml and check for duplicate keys - 13https://github.com/miraheze/puppet/pull/3624 [18:22:20] RECOVERY - mw162 APT on mw162 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:22:20] !log [universalomega@puppet181] Upgraded packages libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, libuv1, bind9-host, bind9-dnsutils, bind9-libs, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, libmagickcore-6.q16-6-extra, libpq5, and pdns-recursor on mw161 [18:22:22] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/ca1229e56866...b921eb263d9c [18:22:23] [02miraheze/puppet] 07Universal-Omega 03b921eb2 - ssl-certificate use pyyaml and check for duplicate keys (#3624) [18:22:25] [02miraheze/puppet] 07Reception123 deleted branch 03ssl-yaml-duplicate [18:22:27] [02puppet] 07Reception123 deleted branch 03ssl-yaml-duplicate - 13https://github.com/miraheze/puppet [18:22:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:22:29] RECOVERY - mw151 APT on mw151 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:22:41] !log [universalomega@puppet181] Upgraded packages libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, libuv1, bind9-host, bind9-dnsutils, bind9-libs, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, libmagickcore-6.q16-6-extra, libpq5, and pdns-recursor on mw171 [18:22:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:11] RECOVERY - mw161 APT on mw161 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:23:13] !log [universalomega@puppet181] Upgraded packages libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, libuv1, bind9-host, bind9-dnsutils, bind9-libs, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, libmagickcore-6.q16-6-extra, libpq5, and pdns-recursor on mw182 [18:23:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:29] RECOVERY - mw181 APT on mw181 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:23:39] !log [universalomega@puppet181] Upgraded packages libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, libuv1, bind9-host, bind9-dnsutils, bind9-libs, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, libmagickcore-6.q16-6-extra, libpq5, and pdns-recursor on mw172 [18:23:41] RECOVERY - mw172 APT on mw172 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:23:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:54] miraheze/puppet - Reception123 the build passed. [18:23:59] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, and libpq5 on ns1 [18:24:03] RECOVERY - mw171 APT on mw171 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:24:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:24:17] RECOVERY - mw182 APT on mw182 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:24:23] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on cp41 [18:24:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:24:44] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on phorge171 [18:24:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:25:04] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on swiftac171 [18:25:06] PROBLEM - ping6 on cp51 is WARNING: PING WARNING - Packet loss = 0%, RTA = 348.87 ms [18:25:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:25:12] RECOVERY - cp41 APT on cp41 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:25:19] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on swiftobject151 [18:25:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:25:35] !log [universalomega@puppet181] Upgraded packages libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, libuv1, bind9-host, bind9-dnsutils, bind9-libs, composer, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, libmagickcore-6.q16-6-extra, libpq5, and pdns-recursor on test151 [18:25:38] RECOVERY - ns1 APT on ns1 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:25:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:25:49] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on swiftproxy161 [18:25:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:25:58] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/c92d987d6d6a...f5ebbdeb192d [18:26:01] [02miraheze/ssl] 07MirahezeSSLBot 03f5ebbde - Bot: Add SSL cert for wiki.gowashtenaw.org [18:26:02] RECOVERY - swiftproxy161 APT on swiftproxy161 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:26:04] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on swiftobject161 [18:26:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:26:16] RECOVERY - phorge171 APT on phorge171 is OK: APT OK: 30 packages available for upgrade (0 critical updates). [18:26:20] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on swiftobject171 [18:26:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:26:38] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on swiftobject181 [18:26:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:26:45] RECOVERY - swiftac171 APT on swiftac171 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:26:54] miraheze/ssl - MirahezeSSLBot the build has errored. [18:27:05] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, dnsutils, and libpq5 on ns2 [18:27:09] RECOVERY - test151 APT on test151 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:27:09] RECOVERY - swiftobject151 APT on swiftobject151 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:27:10] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 179.82 ms [18:27:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:27:22] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on swiftproxy171 [18:27:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:27:32] RECOVERY - swiftobject161 APT on swiftobject161 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:27:46] RECOVERY - swiftproxy171 APT on swiftproxy171 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:27:54] !log [universalomega@puppet181] Upgraded packages libuv1, bind9-host, bind9-dnsutils, bind9-libs, libpq5, and pdns-recursor on cp51 [18:28:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:28:09] RECOVERY - swiftobject181 APT on swiftobject181 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:28:17] RECOVERY - swiftobject171 APT on swiftobject171 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:28:26] RECOVERY - cp51 APT on cp51 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:28:51] RECOVERY - ns2 APT on ns2 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [18:29:17] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/f972cb8b5a2e...2f083b958868 [18:29:19] [02miraheze/dns] 07Reception123 032f083b9 - add aryavratpedia.co zone [18:40:53] !log MariaDB [metawiki]> UPDATE import_requests SET request_status = 'pending' WHERE request_id = 699; (per mistake) [18:41:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:43:49] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 178.12 ms [18:45:06] [02CreateWiki] 07Universal-Omega closed pull request 03#483: RemoteWiki: convert to service - 13https://github.com/miraheze/CreateWiki/pull/483 [18:46:39] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03rw-service [18:46:42] [02CreateWiki] 07Universal-Omega deleted branch 03rw-service - 13https://github.com/miraheze/CreateWiki [18:55:16] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 175.81 ms [18:57:17] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 179.57 ms [19:12:47] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 181.24 ms [19:14:48] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 178.22 ms [19:22:05] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 175.15 ms [19:24:06] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 190.71 ms [19:40:30] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [19:59:11] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03motd [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/2a4d9ce9f3b0 [19:59:12] [02miraheze/puppet] 07Universal-Omega 032a4d9ce - Update motd module [19:59:13] [02puppet] 07Universal-Omega created branch 03motd - 13https://github.com/miraheze/puppet [19:59:18] [02puppet] 07Universal-Omega opened pull request 03#3776: Update motd module - 13https://github.com/miraheze/puppet/pull/3776 [20:02:15] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03motd [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/2a4d9ce9f3b0...eaa31a2a9ab6 [20:02:17] [02miraheze/puppet] 07Universal-Omega 03eaa31a2 - Add [20:02:18] [02puppet] 07Universal-Omega synchronize pull request 03#3776: Update motd module - 13https://github.com/miraheze/puppet/pull/3776 [20:03:46] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03motd [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/eaa31a2a9ab6...d09ad234a17d [20:03:49] [02miraheze/puppet] 07Universal-Omega 03d09ad23 - Update [20:03:50] [02puppet] 07Universal-Omega synchronize pull request 03#3776: Update motd module - 13https://github.com/miraheze/puppet/pull/3776 [20:05:18] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03motd [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d09ad234a17d...25ae59f6dbd3 [20:05:21] [02miraheze/puppet] 07Universal-Omega 0325ae59f - Update [20:05:23] [02puppet] 07Universal-Omega synchronize pull request 03#3776: Update motd module - 13https://github.com/miraheze/puppet/pull/3776 [20:08:05] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03motd [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/25ae59f6dbd3...57bfc44d10ca [20:08:06] [02miraheze/puppet] 07Universal-Omega 0357bfc44 - Add [20:08:07] [02puppet] 07Universal-Omega synchronize pull request 03#3776: Update motd module - 13https://github.com/miraheze/puppet/pull/3776 [20:09:24] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03motd [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/57bfc44d10ca...bdc59670a868 [20:09:26] [02miraheze/puppet] 07Universal-Omega 03bdc5967 - Add [20:09:28] [02puppet] 07Universal-Omega synchronize pull request 03#3776: Update motd module - 13https://github.com/miraheze/puppet/pull/3776 [20:15:26] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03motd [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/bdc59670a868...3247bfbd6150 [20:15:27] [02miraheze/puppet] 07Universal-Omega 033247bfb - Add [20:15:33] [02puppet] 07Universal-Omega synchronize pull request 03#3776: Update motd module - 13https://github.com/miraheze/puppet/pull/3776 [20:16:33] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03motd [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/3247bfbd6150...0304c56bbdd8 [20:16:36] [02miraheze/puppet] 07Universal-Omega 030304c56 - Add [20:16:42] [02puppet] 07Universal-Omega synchronize pull request 03#3776: Update motd module - 13https://github.com/miraheze/puppet/pull/3776 [20:17:54] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03motd [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/0304c56bbdd8...b842d5e45b45 [20:17:56] [02miraheze/puppet] 07Universal-Omega 03b842d5e - Add [20:17:58] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 172.54 ms [20:17:59] [02puppet] 07Universal-Omega synchronize pull request 03#3776: Update motd module - 13https://github.com/miraheze/puppet/pull/3776 [20:18:56] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03motd [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/b842d5e45b45...e70a2a026def [20:18:59] [02miraheze/puppet] 07Universal-Omega 03e70a2a0 - Add [20:19:00] [02puppet] 07Universal-Omega synchronize pull request 03#3776: Update motd module - 13https://github.com/miraheze/puppet/pull/3776 [20:19:53] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03motd [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/e70a2a026def...a657a13195b7 [20:19:54] [02miraheze/puppet] 07Universal-Omega 03a657a13 - Add [20:19:57] [02puppet] 07Universal-Omega synchronize pull request 03#3776: Update motd module - 13https://github.com/miraheze/puppet/pull/3776 [20:20:10] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03motd [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a657a13195b7...12c70418bcb7 [20:20:13] [02miraheze/puppet] 07Universal-Omega 0312c7041 - Update [20:20:16] [02puppet] 07Universal-Omega synchronize pull request 03#3776: Update motd module - 13https://github.com/miraheze/puppet/pull/3776 [20:21:27] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03motd [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/12c70418bcb7...b1d983404ab3 [20:21:30] [02miraheze/puppet] 07Universal-Omega 03b1d9834 - Delete [20:21:33] [02puppet] 07Universal-Omega synchronize pull request 03#3776: Update motd module - 13https://github.com/miraheze/puppet/pull/3776 [20:23:56] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03motd [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/b1d983404ab3...fa05d8bab1e8 [20:23:57] [02miraheze/puppet] 07Universal-Omega 03fa05d8b - Add [20:23:58] [02puppet] 07Universal-Omega synchronize pull request 03#3776: Update motd module - 13https://github.com/miraheze/puppet/pull/3776 [20:24:02] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.95 ms [20:27:49] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03motd [+0/-0/±27] 13https://github.com/miraheze/puppet/compare/fa05d8bab1e8...7f31e519075f [20:27:52] [02miraheze/puppet] 07Universal-Omega 037f31e51 - Use system::role [20:27:53] [02puppet] 07Universal-Omega synchronize pull request 03#3776: Update motd module - 13https://github.com/miraheze/puppet/pull/3776 [20:30:20] [02puppet] 07Universal-Omega edited pull request 03#3776: Update motd module, and ansi, and migrate to system::role - 13https://github.com/miraheze/puppet/pull/3776 [20:30:32] [02puppet] 07Universal-Omega edited pull request 03#3776: Update motd module, add ansi, and migrate to system::role - 13https://github.com/miraheze/puppet/pull/3776 [20:34:15] PROBLEM - mwtask181 Puppet on mwtask181 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 1 minute ago with 1 failures [20:35:34] PROBLEM - mwtask171 Puppet on mwtask171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 27 minutes ago with 3 failures [20:35:39] PROBLEM - test151 Puppet on test151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 12 minutes ago with 2 failures [20:35:56] PROBLEM - cloud16 Puppet on cloud16 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 5 minutes ago with 0 failures [20:35:56] PROBLEM - db151 Puppet on db151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 19 minutes ago with 0 failures [20:36:03] PROBLEM - db181 Puppet on db181 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 6 minutes ago with 0 failures [20:36:04] PROBLEM - cloud15 Puppet on cloud15 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 2 minutes ago with 0 failures [20:36:05] PROBLEM - cp27 Puppet on cp27 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 30 minutes ago with 0 failures [20:36:05] PROBLEM - mw162 Puppet on mw162 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 29 minutes ago with 0 failures [20:36:06] PROBLEM - swiftac171 Puppet on swiftac171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 11 minutes ago with 0 failures [20:36:08] PROBLEM - prometheus151 Puppet on prometheus151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 21 minutes ago with 0 failures [20:36:08] PROBLEM - rdb151 Puppet on rdb151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 8 minutes ago with 0 failures [20:36:15] PROBLEM - jobchron171 Puppet on jobchron171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 28 minutes ago with 0 failures [20:36:15] PROBLEM - mw152 Puppet on mw152 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 13 minutes ago with 0 failures [20:36:16] PROBLEM - db182 Puppet on db182 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 5 minutes ago with 0 failures [20:36:16] PROBLEM - mw161 Puppet on mw161 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 27 minutes ago with 0 failures [20:36:17] PROBLEM - swiftobject151 Puppet on swiftobject151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 19 minutes ago with 0 failures [20:36:17] PROBLEM - swiftproxy171 Puppet on swiftproxy171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 4 minutes ago with 0 failures [20:36:21] PROBLEM - mem151 Puppet on mem151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 3 minutes ago with 0 failures [20:36:24] PROBLEM - mw171 Puppet on mw171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 21 minutes ago with 0 failures [20:36:26] PROBLEM - phorge171 Puppet on phorge171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 27 minutes ago with 0 failures [20:36:31] PROBLEM - swiftproxy161 Puppet on swiftproxy161 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 5 minutes ago with 0 failures [20:36:38] PROBLEM - cp41 Puppet on cp41 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 31 minutes ago with 0 failures [20:36:40] PROBLEM - swiftobject181 Puppet on swiftobject181 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 9 minutes ago with 0 failures [20:36:41] PROBLEM - cp51 Puppet on cp51 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 6 minutes ago with 0 failures [20:36:44] PROBLEM - graylog161 Puppet on graylog161 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 7 minutes ago with 0 failures [20:36:47] PROBLEM - os162 Puppet on os162 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 28 minutes ago with 0 failures [20:36:50] PROBLEM - cloud18 Puppet on cloud18 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 26 minutes ago with 0 failures [20:36:54] PROBLEM - os161 Puppet on os161 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 9 minutes ago with 0 failures [20:36:57] PROBLEM - cp36 Puppet on cp36 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 18 minutes ago with 0 failures [20:36:57] PROBLEM - mw151 Puppet on mw151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 2 minutes ago with 0 failures [20:36:57] PROBLEM - matomo151 Puppet on matomo151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 11 minutes ago with 0 failures [20:36:59] PROBLEM - db171 Puppet on db171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 9 minutes ago with 0 failures [20:36:59] PROBLEM - swiftobject161 Puppet on swiftobject161 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 12 minutes ago with 0 failures [20:36:59] PROBLEM - ns2 Puppet on ns2 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 23 minutes ago with 0 failures [20:37:08] PROBLEM - mon181 Puppet on mon181 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 20 minutes ago with 0 failures [20:37:08] PROBLEM - db161 Puppet on db161 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 10 minutes ago with 0 failures [20:37:08] PROBLEM - mw181 Puppet on mw181 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 12 minutes ago with 0 failures [20:37:11] PROBLEM - mem161 Puppet on mem161 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 9 minutes ago with 0 failures [20:37:11] PROBLEM - os151 Puppet on os151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 4 minutes ago with 0 failures [20:37:11] PROBLEM - swiftobject171 Puppet on swiftobject171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 24 minutes ago with 0 failures [20:37:14] PROBLEM - ns1 Puppet on ns1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 15 minutes ago with 0 failures [20:37:16] PROBLEM - cp26 Puppet on cp26 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 15 minutes ago with 0 failures [20:37:29] PROBLEM - cloud17 Puppet on cloud17 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 23 minutes ago with 0 failures [20:37:33] PROBLEM - bast181 Puppet on bast181 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 11 minutes ago with 0 failures [20:37:37] PROBLEM - reports171 Puppet on reports171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 12 minutes ago with 0 failures [20:37:38] PROBLEM - db162 Puppet on db162 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 14 minutes ago with 0 failures [20:37:40] PROBLEM - ldap171 Puppet on ldap171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 32 minutes ago with 0 failures [20:37:43] PROBLEM - mw182 Puppet on mw182 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 10 minutes ago with 0 failures [20:37:44] PROBLEM - mw172 Puppet on mw172 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 21 minutes ago with 0 failures [20:37:44] PROBLEM - bast161 Puppet on bast161 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 9 minutes ago with 0 failures [20:37:44] PROBLEM - cp37 Puppet on cp37 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 8 minutes ago with 0 failures [20:39:44] [02puppet] 07Universal-Omega closed pull request 03#3776: Update motd module, add ansi, and migrate to system::role - 13https://github.com/miraheze/puppet/pull/3776 [20:39:47] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+9/-1/±29] 13https://github.com/miraheze/puppet/compare/b921eb263d9c...352aa380e299 [20:39:48] [02miraheze/puppet] 07Universal-Omega 03352aa38 - Update motd module, add ansi, and migrate to system::role (#3776) [20:39:51] [02miraheze/puppet] 07Universal-Omega deleted branch 03motd [20:39:52] [02puppet] 07Universal-Omega deleted branch 03motd - 13https://github.com/miraheze/puppet [20:40:15] RECOVERY - jobchron171 Puppet on jobchron171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:40:15] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:40:21] RECOVERY - mem151 Puppet on mem151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:40:24] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:40:28] RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:40:31] RECOVERY - swiftproxy161 Puppet on swiftproxy161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:40:39] RECOVERY - swiftobject181 Puppet on swiftobject181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:40:41] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:40:44] RECOVERY - graylog161 Puppet on graylog161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:40:46] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:40:48] RECOVERY - os162 Puppet on os162 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:40:52] RECOVERY - cloud18 Puppet on cloud18 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:40:54] RECOVERY - cp36 Puppet on cp36 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:40:56] RECOVERY - os161 Puppet on os161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:40:57] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:40:57] RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:40:59] RECOVERY - matomo151 Puppet on matomo151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:40:59] RECOVERY - db171 Puppet on db171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:40:59] RECOVERY - swiftobject161 Puppet on swiftobject161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:41:08] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:41:08] RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:41:08] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:41:10] RECOVERY - os151 Puppet on os151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:41:11] RECOVERY - swiftobject171 Puppet on swiftobject171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:41:11] RECOVERY - mem161 Puppet on mem161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:41:13] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:41:18] RECOVERY - cp26 Puppet on cp26 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:41:29] RECOVERY - cloud17 Puppet on cloud17 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [20:41:31] RECOVERY - bast181 Puppet on bast181 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [20:41:34] PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki-REL1_41 MirahezeMagic],Exec[git_checkout_MediaWiki-REL1_41 UnlinkedWikibase],Exec[git_checkout_MediaWiki-REL1_41 WikiSEO] [20:41:38] RECOVERY - reports171 Puppet on reports171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:41:38] RECOVERY - db162 Puppet on db162 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [20:41:41] RECOVERY - ldap171 Puppet on ldap171 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [20:41:41] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:41:42] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_clone_MediaWiki-REL1_41 SemanticMediaWiki],Exec[git_checkout_MediaWiki-REL1_41 UnlinkedWikibase] [20:41:43] RECOVERY - cp37 Puppet on cp37 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [20:41:45] RECOVERY - bast161 Puppet on bast161 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [20:41:46] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:41:56] RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [20:41:56] RECOVERY - db151 Puppet on db151 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [20:42:03] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [20:42:05] RECOVERY - cp27 Puppet on cp27 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [20:42:05] RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [20:42:05] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:42:06] RECOVERY - rdb151 Puppet on rdb151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:42:06] RECOVERY - swiftac171 Puppet on swiftac171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:42:08] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:42:15] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:42:15] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:42:16] RECOVERY - swiftproxy171 Puppet on swiftproxy171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:42:16] RECOVERY - swiftobject151 Puppet on swiftobject151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:42:16] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_clone_MediaWiki-REL1_41 SemanticMediaWiki] [21:15:30] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [21:31:04] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1011439264 secs [21:46:07] [02puppet] 07Universal-Omega created branch 03systemd-logrotate - 13https://github.com/miraheze/puppet [21:46:08] [02miraheze/puppet] 07Universal-Omega created branch 03systemd-logrotate 13https://github.com/miraheze/puppet/commit/352aa380e299b4902b43537c141ad836e79f2b12 [21:51:04] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09921428561 secs [22:15:51] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 25% [22:22:33] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03systemd-logrotate [+11/-5/±9] 13https://github.com/miraheze/puppet/compare/352aa380e299...72822b302aeb [22:22:34] [02miraheze/puppet] 07Universal-Omega 0372822b3 - Update systemd and logrotate modules [22:22:48] [02puppet] 07Universal-Omega opened pull request 03#3777: Update systemd and logrotate modules - 13https://github.com/miraheze/puppet/pull/3777 [22:23:24] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03systemd-logrotate [+0/-0/±9] 13https://github.com/miraheze/puppet/compare/72822b302aeb...d8ad682e65a2 [22:23:26] [02miraheze/puppet] 07github-actions 03d8ad682 - CI: lint puppet code to standards [22:23:27] [02puppet] 07github-actions[bot] synchronize pull request 03#3777: Update systemd and logrotate modules - 13https://github.com/miraheze/puppet/pull/3777 [22:23:34] [02puppet] 07Universal-Omega synchronize pull request 03#3777: Update systemd and logrotate modules - 13https://github.com/miraheze/puppet/pull/3777 [22:23:37] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03systemd-logrotate [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d8ad682e65a2...34609e475894 [22:23:39] [02miraheze/puppet] 07Universal-Omega 0334609e4 - - [22:25:06] miraheze/puppet - Universal-Omega the build passed. [22:25:10] miraheze/puppet - Universal-Omega the build passed. [22:28:31] [02puppet] 07Universal-Omega synchronize pull request 03#3777: Update systemd and logrotate modules - 13https://github.com/miraheze/puppet/pull/3777 [22:28:32] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03systemd-logrotate [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/34609e475894...80c140b44458 [22:28:33] [02miraheze/puppet] 07Universal-Omega 0380c140b - Fix [22:30:30] miraheze/puppet - Universal-Omega the build passed. [22:31:34] PROBLEM - mwtask171 Puppet on mwtask171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 23 minutes ago with 3 failures [22:32:30] PROBLEM - test151 Puppet on test151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 8 minutes ago with 2 failures [22:32:30] PROBLEM - mwtask181 Puppet on mwtask181 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 29 minutes ago with 1 failures [22:33:29] PROBLEM - cloud17 Puppet on cloud17 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 19 minutes ago with 0 failures [22:33:37] PROBLEM - os161 Puppet on os161 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 5 minutes ago with 0 failures [22:33:38] PROBLEM - db162 Puppet on db162 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 10 minutes ago with 0 failures [22:33:42] PROBLEM - cp37 Puppet on cp37 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 4 minutes ago with 0 failures [22:33:46] PROBLEM - ldap171 Puppet on ldap171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 27 minutes ago with 0 failures [22:33:46] PROBLEM - cloud18 Puppet on cloud18 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 23 minutes ago with 0 failures [22:33:47] PROBLEM - cp27 Puppet on cp27 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 27 minutes ago with 0 failures [22:33:50] PROBLEM - mem151 Puppet on mem151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 31 minutes ago with 0 failures [22:33:56] PROBLEM - cloud16 Puppet on cloud16 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 3 minutes ago with 0 failures [22:33:56] PROBLEM - db151 Puppet on db151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 17 minutes ago with 0 failures [22:34:00] PROBLEM - mw161 Puppet on mw161 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 26 minutes ago with 0 failures [22:34:00] PROBLEM - reports171 Puppet on reports171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 8 minutes ago with 0 failures [22:34:01] PROBLEM - cp41 Puppet on cp41 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 28 minutes ago with 0 failures [22:34:03] PROBLEM - db181 Puppet on db181 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 4 minutes ago with 0 failures [22:34:05] PROBLEM - mw162 Puppet on mw162 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 27 minutes ago with 0 failures [22:34:05] PROBLEM - swiftac171 Puppet on swiftac171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 8 minutes ago with 0 failures [22:34:06] PROBLEM - bast161 Puppet on bast161 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 5 minutes ago with 0 failures [22:34:07] PROBLEM - phorge171 Puppet on phorge171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 26 minutes ago with 0 failures [22:34:08] PROBLEM - prometheus151 Puppet on prometheus151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 18 minutes ago with 0 failures [22:34:09] PROBLEM - ns2 Puppet on ns2 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 20 minutes ago with 0 failures [22:34:10] PROBLEM - cp26 Puppet on cp26 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 12 minutes ago with 0 failures [22:34:15] PROBLEM - jobchron171 Puppet on jobchron171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 27 minutes ago with 0 failures [22:34:15] PROBLEM - mw152 Puppet on mw152 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 10 minutes ago with 0 failures [22:34:16] PROBLEM - db182 Puppet on db182 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 3 minutes ago with 0 failures [22:34:17] PROBLEM - swiftproxy171 Puppet on swiftproxy171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 31 minutes ago with 0 failures [22:34:17] PROBLEM - swiftobject151 Puppet on swiftobject151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 17 minutes ago with 0 failures [22:34:17] PROBLEM - cp36 Puppet on cp36 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 15 minutes ago with 0 failures [22:34:24] PROBLEM - mw171 Puppet on mw171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 19 minutes ago with 0 failures [22:34:29] PROBLEM - swiftproxy161 Puppet on swiftproxy161 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 3 minutes ago with 0 failures [22:34:34] PROBLEM - cloud15 Puppet on cloud15 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 30 minutes ago with 0 failures [22:34:38] PROBLEM - cp51 Puppet on cp51 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 4 minutes ago with 0 failures [22:34:40] PROBLEM - mw172 Puppet on mw172 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 18 minutes ago with 0 failures [22:34:40] PROBLEM - swiftobject181 Puppet on swiftobject181 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 7 minutes ago with 0 failures [22:34:44] PROBLEM - graylog161 Puppet on graylog161 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 5 minutes ago with 0 failures [22:34:57] PROBLEM - mw151 Puppet on mw151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 30 minutes ago with 0 failures [22:34:59] PROBLEM - db171 Puppet on db171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 8 minutes ago with 0 failures [22:34:59] PROBLEM - swiftobject161 Puppet on swiftobject161 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 10 minutes ago with 0 failures [22:35:05] PROBLEM - bast181 Puppet on bast181 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 8 minutes ago with 0 failures [22:35:07] PROBLEM - mon181 Puppet on mon181 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 18 minutes ago with 0 failures [22:35:08] PROBLEM - db161 Puppet on db161 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 8 minutes ago with 0 failures [22:35:11] PROBLEM - os162 Puppet on os162 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 25 minutes ago with 0 failures [22:35:11] PROBLEM - swiftobject171 Puppet on swiftobject171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 22 minutes ago with 0 failures [22:35:11] PROBLEM - mem161 Puppet on mem161 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 7 minutes ago with 0 failures [22:35:11] PROBLEM - os151 Puppet on os151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 32 minutes ago with 0 failures [22:35:13] PROBLEM - ns1 Puppet on ns1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 13 minutes ago with 0 failures [22:35:14] PROBLEM - puppet181 Puppet on puppet181 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 6 minutes ago with 0 failures [22:35:24] PROBLEM - mw181 Puppet on mw181 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 10 minutes ago with 0 failures [22:35:29] PROBLEM - mw182 Puppet on mw182 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 8 minutes ago with 0 failures [22:35:29] PROBLEM - rdb151 Puppet on rdb151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 7 minutes ago with 0 failures [22:37:05] [02puppet] 07Universal-Omega closed pull request 03#3777: Update systemd and logrotate modules - 13https://github.com/miraheze/puppet/pull/3777 [22:37:08] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+11/-5/±9] 13https://github.com/miraheze/puppet/compare/352aa380e299...d9b613277051 [22:37:11] [02miraheze/puppet] 07Universal-Omega 03d9b6132 - Update systemd and logrotate modules (#3777) [22:37:14] [02miraheze/puppet] 07Universal-Omega deleted branch 03systemd-logrotate [22:37:16] [02puppet] 07Universal-Omega deleted branch 03systemd-logrotate - 13https://github.com/miraheze/puppet [22:37:46] RECOVERY - ldap171 Puppet on ldap171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:37:47] RECOVERY - cp27 Puppet on cp27 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:37:48] RECOVERY - cloud18 Puppet on cloud18 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:37:49] RECOVERY - mem151 Puppet on mem151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:37:56] RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:37:56] RECOVERY - db151 Puppet on db151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:37:59] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:38:01] RECOVERY - reports171 Puppet on reports171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:38:03] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:38:04] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:38:05] RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:38:05] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:38:05] RECOVERY - swiftac171 Puppet on swiftac171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:38:08] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:38:08] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:38:09] RECOVERY - bast161 Puppet on bast161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:38:12] RECOVERY - cp26 Puppet on cp26 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:38:15] RECOVERY - jobchron171 Puppet on jobchron171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:38:15] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:38:15] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:38:16] RECOVERY - swiftobject151 Puppet on swiftobject151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:38:16] RECOVERY - swiftproxy171 Puppet on swiftproxy171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:38:17] RECOVERY - cp36 Puppet on cp36 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:38:24] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:38:29] RECOVERY - swiftproxy161 Puppet on swiftproxy161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:38:31] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_clone_MediaWiki-REL1_41 SemanticMediaWiki] [22:38:33] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_clone_MediaWiki-REL1_41 SemanticMediaWiki],Exec[git_checkout_MediaWiki-REL1_41 UnlinkedWikibase] [22:38:35] RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:38:38] miraheze/puppet - Universal-Omega the build passed. [22:38:39] RECOVERY - swiftobject181 Puppet on swiftobject181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:38:41] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:38:42] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:38:44] RECOVERY - graylog161 Puppet on graylog161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:38:57] RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:38:59] RECOVERY - db171 Puppet on db171 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [22:38:59] RECOVERY - swiftobject161 Puppet on swiftobject161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:39:05] RECOVERY - bast181 Puppet on bast181 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [22:39:07] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:39:08] RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [22:39:10] RECOVERY - os151 Puppet on os151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:39:11] RECOVERY - swiftobject171 Puppet on swiftobject171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:39:11] RECOVERY - mem161 Puppet on mem161 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [22:39:11] RECOVERY - os162 Puppet on os162 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:39:13] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:39:15] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:39:24] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:39:29] RECOVERY - cloud17 Puppet on cloud17 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [22:39:29] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:39:29] RECOVERY - rdb151 Puppet on rdb151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:39:34] PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:39:38] RECOVERY - db162 Puppet on db162 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [22:39:39] RECOVERY - os161 Puppet on os161 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [22:39:42] RECOVERY - cp37 Puppet on cp37 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [22:42:00] PROBLEM - mw161 Puppet on mw161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:42:05] PROBLEM - mw162 Puppet on mw162 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:42:15] PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:42:24] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:42:45] PROBLEM - mw172 Puppet on mw172 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:42:57] PROBLEM - mw151 Puppet on mw151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:43:25] PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:06:13] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9698MiB (10% inode=98%); [23:08:57] PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9706MiB (10% inode=98%); [23:13:56] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [23:36:57] [02miraheze/puppet] 07Universal-Omega created branch 03graphite 13https://github.com/miraheze/puppet/commit/d9b613277051c5f3ecdfe6c9babe7b41b6e25ed1 [23:37:00] [02puppet] 07Universal-Omega created branch 03graphite - 13https://github.com/miraheze/puppet [23:37:32] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03graphite [+36/-0/±2] 13https://github.com/miraheze/puppet/compare/d9b613277051...02baad430845 [23:37:34] [02miraheze/puppet] 07Universal-Omega 0302baad4 - Add graphite [23:37:46] [02puppet] 07Universal-Omega opened pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [23:38:37] miraheze/puppet - Universal-Omega the build has errored. [23:39:35] miraheze/puppet - Universal-Omega the build passed. [23:43:10] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK