[00:00:39] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:02:39] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:03:51] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:04:13] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 21487MiB (24% inode=98%); [00:04:57] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 21490MiB (24% inode=98%); [00:19:51] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 58% [00:23:51] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [00:25:51] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d9b613277051...67d9361b0493 [00:25:52] [02miraheze/puppet] 07Universal-Omega 0367d9361 - Fix [00:26:57] RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [00:28:05] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/67d9361b0493...b54c8192b924 [00:28:08] [02miraheze/puppet] 07Universal-Omega 03b54c819 - newline [00:29:29] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:38:05] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [00:39:43] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:46:24] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:47:43] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:54:15] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [00:55:42] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [01:09:45] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03graphite [+3/-0/±3] 13https://github.com/miraheze/puppet/compare/02baad430845...b33a0ee409d8 [01:09:48] [02miraheze/puppet] 07Universal-Omega 03b33a0ee - Add role [01:09:51] [02puppet] 07Universal-Omega synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [01:10:51] miraheze/puppet - Universal-Omega the build has errored. [01:11:11] miraheze/puppet - Universal-Omega the build passed. [01:13:17] [02puppet] 07Universal-Omega synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [01:13:18] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03graphite [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b33a0ee409d8...eced76fc90f0 [01:13:21] [02miraheze/puppet] 07Universal-Omega 03eced76f - - [01:14:11] miraheze/puppet - Universal-Omega the build has errored. [01:14:43] miraheze/puppet - Universal-Omega the build passed. [01:27:04] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1034559906 secs [01:46:11] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b54c8192b924...92bbc77af01e [01:46:14] [02miraheze/puppet] 07Universal-Omega 0392bbc77 - systemd-timer-mail-wrapper: fix [01:47:16] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/92bbc77af01e...dfcb6f6c0fe1 [01:47:18] [02miraheze/puppet] 07Universal-Omega 03dfcb6f6 - Remove unused import [01:47:45] miraheze/puppet - Universal-Omega the build passed. [01:49:05] miraheze/puppet - Universal-Omega the build passed. [01:58:02] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/72670a50b7d9...c795d7dabf77 [01:58:04] [02miraheze/mw-config] 07AgentIsai 03c795d7d - T10611: Enable SMW/Maps integration [01:58:43] !log [agent@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [01:58:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:58:54] !log [agent@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [01:58:57] miraheze/mw-config - AgentIsai the build passed. [01:59:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:00:00] [02puppet] 07Universal-Omega synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [02:00:01] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03graphite [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/eced76fc90f0...7ee07b5366ff [02:00:02] [02miraheze/puppet] 07Universal-Omega 037ee07b5 - Fix [02:00:58] miraheze/puppet - Universal-Omega the build has errored. [02:01:14] [02puppet] 07Universal-Omega synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [02:01:16] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03graphite [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7ee07b5366ff...8d805db4f5ed [02:01:18] [02miraheze/puppet] 07Universal-Omega 038d805db - Fix [02:01:38] miraheze/puppet - Universal-Omega the build passed. [02:02:08] miraheze/puppet - Universal-Omega the build has errored. [02:02:40] miraheze/puppet - Universal-Omega the build passed. [02:03:14] [02puppet] 07Universal-Omega synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [02:03:17] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03graphite [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8d805db4f5ed...7d47eb022fd8 [02:03:19] [02miraheze/puppet] 07Universal-Omega 037d47eb0 - Move [02:04:06] miraheze/puppet - Universal-Omega the build has errored. [02:04:37] miraheze/puppet - Universal-Omega the build passed. [02:07:00] !log [@mwtask171] starting deploy of {'config': True} to all [02:07:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:07:09] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [02:07:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:23:42] !log [@test151] starting deploy of {'config': True} to test151 [02:23:43] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [02:23:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:23:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:24:38] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/b42756516de0...ce3669ec1d9d [02:24:40] [02miraheze/ImportDump] 07Universal-Omega 03ce3669e - Allow to set status back to pending even if it is already set as complete [02:26:19] [02ImportDump] 07Universal-Omega closed pull request 03#84: Remove duplicates from ImportDumpJob - 13https://github.com/miraheze/ImportDump/pull/84 [02:26:22] [02miraheze/ImportDump] 07Universal-Omega deleted branch 03remove-duplicates [02:26:24] [02ImportDump] 07Universal-Omega deleted branch 03remove-duplicates - 13https://github.com/miraheze/ImportDump [02:26:46] [02ImportDump] 07Universal-Omega closed pull request 03#85: Renamespace under Miraheze\RequestImport - 13https://github.com/miraheze/ImportDump/pull/85 [02:26:49] [02ImportDump] 07Universal-Omega commented on pull request 03#85: Renamespace under Miraheze\RequestImport - 13https://github.com/miraheze/ImportDump/pull/85#issuecomment-2005632296 [02:26:52] [02miraheze/ImportDump] 07Universal-Omega deleted branch 03renamespace [02:26:54] [02ImportDump] 07Universal-Omega deleted branch 03renamespace - 13https://github.com/miraheze/ImportDump [02:30:32] [02puppet] 07Universal-Omega synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [02:30:33] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03graphite [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7d47eb022fd8...687a36652385 [02:30:35] [02miraheze/puppet] 07Universal-Omega 03687a366 - Fix [02:31:04] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09914430976 secs [02:31:56] [02miraheze/puppet] 07Universal-Omega pushed 037 commits to 03graphite [+39/-0/±10] 13https://github.com/miraheze/puppet/compare/687a36652385...83498e83f92a [02:31:57] miraheze/puppet - Universal-Omega the build passed. [02:31:57] [02miraheze/puppet] 07Universal-Omega 034248b03 - Add graphite [02:31:58] [02miraheze/puppet] 07Universal-Omega 036ecf29c - Add role [02:32:01] [02miraheze/puppet] 07Universal-Omega 03b75c64c - - [02:32:04] [02miraheze/puppet] ... and 4 more commits. [02:32:06] [02puppet] 07Universal-Omega synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [02:32:38] miraheze/ImportDump - Universal-Omega the build passed. [02:32:39] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03graphite [+0/-0/±14] 13https://github.com/miraheze/puppet/compare/83498e83f92a...8c1689ed61ab [02:32:42] [02miraheze/puppet] 07github-actions 038c1689e - CI: lint puppet code to standards [02:32:44] [02puppet] 07github-actions[bot] synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [02:33:30] miraheze/puppet - Universal-Omega the build passed. [02:40:07] [02puppet] 07Universal-Omega synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [02:40:10] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03graphite [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8c1689ed61ab...2bff2d120725 [02:40:13] [02miraheze/puppet] 07Universal-Omega 032bff2d1 - Replace legacy facts [02:41:44] miraheze/puppet - Universal-Omega the build passed. [02:47:57] [02puppet] 07Universal-Omega synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [02:47:59] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03graphite [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2bff2d120725...673f1b7fb6f9 [02:48:01] [02miraheze/puppet] 07Universal-Omega 03673f1b7 - Only use one cache and set default value for admin user [02:49:25] miraheze/puppet - Universal-Omega the build passed. [02:52:09] [02puppet] 07Universal-Omega synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [02:52:10] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03graphite [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/673f1b7fb6f9...ecdae64f338b [02:52:13] [02miraheze/puppet] 07Universal-Omega 03ecdae64 - -var [02:53:32] miraheze/puppet - Universal-Omega the build passed. [03:01:29] RECOVERY - db181 Backups SQL on db181 is OK: FILE_AGE OK: /var/log/sql-backup.log is 87 seconds old and 0 bytes [03:01:40] RECOVERY - db161 Backups SQL on db161 is OK: FILE_AGE OK: /var/log/sql-backup.log is 99 seconds old and 0 bytes [03:09:00] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03graphite [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ecdae64f338b...13a03d551698 [03:09:01] [02miraheze/puppet] 07Universal-Omega 0313a03d5 - Add pushgateway job [03:09:03] [02puppet] 07Universal-Omega synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [03:09:59] miraheze/puppet - Universal-Omega the build has errored. [03:10:05] [02puppet] 07Universal-Omega synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [03:10:07] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03graphite [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/13a03d551698...928e86b055ab [03:10:09] [02miraheze/puppet] 07Universal-Omega 03928e86b - space [03:10:21] miraheze/puppet - Universal-Omega the build passed. [03:10:57] miraheze/puppet - Universal-Omega the build has errored. [03:11:39] miraheze/puppet - Universal-Omega the build passed. [03:14:11] [02puppet] 07Universal-Omega synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [03:14:12] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03graphite [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/928e86b055ab...cdbbd2f2666d [03:14:15] [02miraheze/puppet] 07Universal-Omega 03cdbbd2f - - [03:15:51] miraheze/puppet - Universal-Omega the build passed. [03:23:26] [02puppet] 07Universal-Omega synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [03:23:27] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03graphite [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cdbbd2f2666d...681191ce3068 [03:23:30] [02miraheze/puppet] 07Universal-Omega 03681191c - Remove cassandra [03:24:55] miraheze/puppet - Universal-Omega the build passed. [03:29:32] [02puppet] 07Universal-Omega synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [03:29:35] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03graphite [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/681191ce3068...60e3cbe9ff20 [03:29:37] [02miraheze/puppet] 07Universal-Omega 0360e3cbe - Only one carbon cache for now [03:31:19] miraheze/puppet - Universal-Omega the build passed. [03:51:04] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1003851891 secs [04:05:04] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09952634573 secs [04:06:25] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 172.80 ms [04:08:26] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 178.40 ms [04:12:38] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.36 ms [04:14:39] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 173.03 ms [04:33:22] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 174.30 ms [04:35:22] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 173.54 ms [04:46:42] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 176.98 ms [04:48:42] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 173.19 ms [04:59:09] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 174.04 ms [05:01:09] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 180.62 ms [05:08:27] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.33 ms [05:12:33] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 173.57 ms [05:24:56] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.70 ms [05:31:02] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 175.23 ms [05:39:14] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 177.71 ms [05:41:15] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.16 ms [06:00:55] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 187.25 ms [06:06:59] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 175.11 ms [06:12:24] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:25:04] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1013391316 secs [06:27:05] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09876272082 secs [06:31:04] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1034655571 secs [06:37:46] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 181.22 ms [06:41:49] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 182.01 ms [07:11:28] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 170.10 ms [07:11:38] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [07:25:43] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.65 ms [07:43:30] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.34 ms [07:45:30] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.21 ms [07:49:41] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.49 ms [07:53:43] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.81 ms [08:11:05] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09574016929 secs [08:13:02] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:14:15] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.75 ms [08:18:17] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 175.82 ms [08:24:30] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.68 ms [08:34:38] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.64 ms [08:42:36] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:59:24] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 175.86 ms [09:03:26] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 175.18 ms [09:07:37] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.98 ms [09:09:37] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.26 ms [10:15:04] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1012533903 secs [10:15:51] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 171.62 ms [10:17:04] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09920242429 secs [10:17:55] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.86 ms [10:43:28] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 171.98 ms [10:45:29] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.00 ms [10:51:33] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1000754237 secs [10:53:31] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.0995554626 secs [11:09:01] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.99 ms [11:10:14] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:11:02] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.86 ms [11:39:27] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK [12:30:47] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1002469957 secs [12:34:43] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09948781133 secs [13:16:04] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1019692719 secs [13:41:54] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 175.81 ms [13:43:55] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.28 ms [13:58:08] PROBLEM - ping6 on cp51 is WARNING: PING WARNING - Packet loss = 0%, RTA = 301.95 ms [14:04:11] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 210.49 ms [14:05:22] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09922727942 secs [14:06:12] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 262.15 ms [14:19:09] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1048377156 secs [15:20:23] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 183.67 ms [15:22:23] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 185.79 ms [16:17:31] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 185.97 ms [16:19:32] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 189.04 ms [16:36:37] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/dfcb6f6c0fe1...cb6dfb4abc6b [16:36:40] [02miraheze/puppet] 07Universal-Omega 03cb6dfb4 - matomo: update Ldap name to match hostname [16:37:51] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cb6dfb4abc6b...ebb586c04b1a [16:37:53] [02miraheze/puppet] 07Universal-Omega 03ebb586c - grafana: update org_name to match hostname [16:39:54] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 27.21, 44.08, 20.69 [16:49:24] PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 0.30, 6.73, 11.29 [16:51:24] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.14, 4.55, 9.94 [16:51:59] [02puppet] 07Universal-Omega created branch 03ldap - 13https://github.com/miraheze/puppet [16:52:00] [02miraheze/puppet] 07Universal-Omega created branch 03ldap 13https://github.com/miraheze/puppet/commit/ebb586c04b1a3bfa99b8cf6b252d92f77f48894e [16:55:05] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09346297383 secs [16:57:14] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03ldap [+0/-0/±4] 13https://github.com/miraheze/puppet/compare/ebb586c04b1a...3c7e1d5dd033 [16:57:16] [02miraheze/puppet] 07Universal-Omega 033c7e1d5 - ldap: use ldap.wikitide.net [16:57:33] [02puppet] 07Universal-Omega opened pull request 03#3779: ldap: use ldap.wikitide.net - 13https://github.com/miraheze/puppet/pull/3779 [16:59:04] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1043475568 secs [17:01:04] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09937018156 secs [17:03:56] Hi! Could someone help me change my wiki's URL? [17:05:50] I want https://impostors.miraheze.org to be moved to https://malorittoz.miraheze.org [17:06:50] Please open a task on https://issue-tracker.miraheze.org [17:07:13] Thanks! [17:09:27] How do I do that? [17:11:59] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.106228143 secs [17:12:25] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.401 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [17:13:00] ? [17:15:15] How do I open a task? [17:15:29] @MediaWiki Trusted Support Volunteers ^ [17:15:36] I'm on train wifi [17:15:41] Can someone explain [17:15:54] Netherized: someone will answer shortly [17:15:57] um [17:15:58] ok [17:17:26] [1/2] go here and type that you want to change data base name for your wiki, providing current and desired URL [17:17:26] [2/2] to login press MediaWiki button [17:18:47] thank you [17:19:26] PROBLEM - mw172 Puppet on mw172 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libboost-regex1.74.0-icu72] [17:21:35] Ty, I couldn't get phab to load. WiFi useless [17:25:46] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09648829699 secs [17:29:40] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1099559963 secs [17:38:10] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 188.22 ms [17:40:11] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 176.71 ms [17:47:39] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:50:33] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 198.69 ms [17:51:21] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09977117181 secs [17:55:16] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1005648673 secs [17:59:12] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09922257066 secs [18:06:56] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.34 ms [18:11:40] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [18:18:32] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 179.42 ms [18:20:33] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 176.09 ms [18:23:04] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1016061902 secs [18:24:49] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 177.15 ms [18:35:03] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 211.89 ms [18:41:17] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [18:46:13] !log create graphite151 VM [18:46:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:03:47] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 187.83 ms [19:09:55] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 174.09 ms [19:13:07] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:15:11] [02miraheze/.github] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/actions/cache-4.0.2 [+0/-0/±1] 13https://github.com/miraheze/.github/commit/593e1eee98c2 [19:15:13] [02miraheze/.github] 07dependabot[bot] 03593e1ee - Bump actions/cache from 4.0.1 to 4.0.2 [19:15:14] [02.github] 07dependabot[bot] created branch 03dependabot/github_actions/actions/cache-4.0.2 - 13https://github.com/miraheze/.github [19:15:16] [02.github] 07dependabot[bot] opened pull request 03#47: Bump actions/cache from 4.0.1 to 4.0.2 - 13https://github.com/miraheze/.github/pull/47 [19:15:17] [02.github] 07dependabot[bot] labeled pull request 03#47: Bump actions/cache from 4.0.1 to 4.0.2 - 13https://github.com/miraheze/.github/pull/47 [19:31:42] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 170.30 ms [19:35:04] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09994518757 secs [19:41:55] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.49 ms [19:55:27] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 170.73 ms [20:01:31] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.37 ms [20:07:46] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 173.69 ms [20:12:20] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [20:24:04] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 166.93 ms [20:28:17] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 171.41 ms [20:39:59] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1054384708 secs [20:41:12] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/2f083b958868...975604d9856d [20:41:14] [02miraheze/dns] 07Universal-Omega 03975604d - Add graphite151 to dns [20:48:38] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 182.91 ms [20:48:43] PROBLEM - graphite151 NTP time on graphite151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:48:44] PROBLEM - graphite151 PowerDNS Recursor on graphite151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:48:45] PROBLEM - graphite151 conntrack_table_size on graphite151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:48:45] PROBLEM - graphite151 Disk Space on graphite151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:49:34] PROBLEM - graphite151 Puppet on graphite151 is UNKNOWN: UNKNOWN: Failed to check. Reason is: no_summary_file [20:49:39] PROBLEM - graphite151 ferm_active on graphite151 is UNKNOWN: NRPE: Unable to read output [20:50:41] RECOVERY - graphite151 Disk Space on graphite151 is OK: DISK OK - free space: / 24777MiB (89% inode=95%); [20:50:43] RECOVERY - graphite151 PowerDNS Recursor on graphite151 is OK: DNS OK: 0.032 seconds response time. miraheze.org returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [20:50:44] PROBLEM - graphite151 conntrack_table_size on graphite151 is UNKNOWN: NRPE: Unable to read output [20:56:43] RECOVERY - graphite151 conntrack_table_size on graphite151 is OK: OK: nf_conntrack is 0 % full [20:57:21] RECOVERY - graphite151 Puppet on graphite151 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [20:57:33] RECOVERY - graphite151 ferm_active on graphite151 is OK: OK ferm input default policy is set [21:07:31] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09875375032 secs [21:11:25] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03graphite [+1/-0/±3] 13https://github.com/miraheze/puppet/compare/60e3cbe9ff20...bad5d59ee105 [21:11:28] [02miraheze/puppet] 07Universal-Omega 03bad5d59 - Use graphite151 [21:11:29] [02puppet] 07Universal-Omega synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [21:12:54] miraheze/puppet - Universal-Omega the build passed. [21:14:32] PROBLEM - mwtask171 Puppet on mwtask171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 6 minutes ago with 3 failures [21:14:54] PROBLEM - mwtask181 Puppet on mwtask181 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 12 minutes ago with 1 failures [21:15:33] [02miraheze/puppet] 07Universal-Omega pushed 0317 commits to 03graphite [+40/-0/±35] 13https://github.com/miraheze/puppet/compare/bad5d59ee105...f47173ca82ee [21:15:36] [02miraheze/puppet] 07Universal-Omega 03e24baa8 - Add graphite [21:15:37] [02miraheze/puppet] 07Universal-Omega 03bf8ed28 - Add role [21:15:39] [02miraheze/puppet] 07Universal-Omega 0323e0f78 - - [21:15:41] [02miraheze/puppet] ... and 14 more commits. [21:15:42] [02puppet] 07Universal-Omega synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [21:16:07] PROBLEM - cloud18 Puppet on cloud18 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 6 minutes ago with 0 failures [21:16:09] PROBLEM - test151 Puppet on test151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 22 minutes ago with 2 failures [21:16:10] PROBLEM - cloud17 Puppet on cloud17 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 2 minutes ago with 0 failures [21:16:19] PROBLEM - mem161 Puppet on mem161 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 17 minutes ago with 0 failures [21:16:22] PROBLEM - mw151 Puppet on mw151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 12 minutes ago with 0 failures [21:16:25] PROBLEM - swiftproxy161 Puppet on swiftproxy161 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 15 minutes ago with 0 failures [21:16:25] PROBLEM - cp36 Puppet on cp36 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 27 minutes ago with 0 failures [21:16:26] PROBLEM - mw171 Puppet on mw171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 30 minutes ago with 0 failures [21:16:28] PROBLEM - puppet181 Puppet on puppet181 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 17 minutes ago with 0 failures [21:16:28] PROBLEM - cp41 Puppet on cp41 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 11 minutes ago with 0 failures [21:16:36] PROBLEM - rdb151 Puppet on rdb151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 18 minutes ago with 0 failures [21:16:41] PROBLEM - cp26 Puppet on cp26 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 24 minutes ago with 0 failures [21:16:48] miraheze/puppet - Universal-Omega the build passed. [21:16:49] PROBLEM - mw182 Puppet on mw182 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 19 minutes ago with 0 failures [21:16:50] PROBLEM - swiftobject161 Puppet on swiftobject161 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 21 minutes ago with 0 failures [21:16:50] PROBLEM - mon181 Puppet on mon181 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 30 minutes ago with 0 failures [21:16:50] PROBLEM - bast181 Puppet on bast181 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 20 minutes ago with 0 failures [21:16:50] PROBLEM - mw172 Puppet on mw172 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 30 minutes ago with 0 failures [21:16:51] PROBLEM - db161 Puppet on db161 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 20 minutes ago with 0 failures [21:16:51] PROBLEM - db162 Puppet on db162 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 23 minutes ago with 0 failures [21:16:52] PROBLEM - os151 Puppet on os151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 15 minutes ago with 0 failures [21:16:56] PROBLEM - cloud16 Puppet on cloud16 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 16 minutes ago with 0 failures [21:17:05] PROBLEM - jobchron171 Puppet on jobchron171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 10 minutes ago with 0 failures [21:17:06] PROBLEM - os161 Puppet on os161 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 19 minutes ago with 0 failures [21:17:08] PROBLEM - ns1 Puppet on ns1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 25 minutes ago with 0 failures [21:17:09] PROBLEM - mem151 Puppet on mem151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 14 minutes ago with 0 failures [21:17:21] PROBLEM - graphite151 Puppet on graphite151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:17:21] PROBLEM - cp27 Puppet on cp27 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 11 minutes ago with 0 failures [21:17:22] PROBLEM - mw161 Puppet on mw161 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 9 minutes ago with 0 failures [21:17:25] PROBLEM - mw162 Puppet on mw162 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 10 minutes ago with 0 failures [21:17:26] PROBLEM - os162 Puppet on os162 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 8 minutes ago with 0 failures [21:17:28] PROBLEM - swiftobject171 Puppet on swiftobject171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 4 minutes ago with 0 failures [21:17:28] PROBLEM - cloud15 Puppet on cloud15 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 13 minutes ago with 0 failures [21:17:28] PROBLEM - db171 Puppet on db171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 20 minutes ago with 0 failures [21:17:29] PROBLEM - swiftac171 Puppet on swiftac171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 22 minutes ago with 0 failures [21:17:32] PROBLEM - cp37 Puppet on cp37 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 18 minutes ago with 0 failures [21:17:35] PROBLEM - graylog161 Puppet on graylog161 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 18 minutes ago with 0 failures [21:17:37] PROBLEM - ns2 Puppet on ns2 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 4 minutes ago with 0 failures [21:17:39] PROBLEM - cp51 Puppet on cp51 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 16 minutes ago with 0 failures [21:17:41] PROBLEM - db181 Puppet on db181 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 17 minutes ago with 0 failures [21:17:42] PROBLEM - prometheus151 Puppet on prometheus151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 32 minutes ago with 0 failures [21:17:49] PROBLEM - reports171 Puppet on reports171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 21 minutes ago with 0 failures [21:17:52] PROBLEM - db182 Puppet on db182 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 16 minutes ago with 0 failures [21:17:52] PROBLEM - swiftobject181 Puppet on swiftobject181 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 20 minutes ago with 0 failures [21:17:56] PROBLEM - phorge171 Puppet on phorge171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 9 minutes ago with 0 failures [21:17:57] PROBLEM - ldap171 Puppet on ldap171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 11 minutes ago with 0 failures [21:17:58] PROBLEM - bast161 Puppet on bast161 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 18 minutes ago with 0 failures [21:18:01] PROBLEM - mw152 Puppet on mw152 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 24 minutes ago with 0 failures [21:18:03] PROBLEM - matomo151 Puppet on matomo151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 23 minutes ago with 0 failures [21:18:04] PROBLEM - db151 Puppet on db151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 31 minutes ago with 0 failures [21:18:05] PROBLEM - swiftobject151 Puppet on swiftobject151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 31 minutes ago with 0 failures [21:18:09] PROBLEM - mw181 Puppet on mw181 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 23 minutes ago with 0 failures [21:18:11] PROBLEM - swiftproxy171 Puppet on swiftproxy171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 16 minutes ago with 0 failures [21:19:34] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 186.37 ms [21:20:49] [02puppet] 07Universal-Omega synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [21:20:52] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03graphite [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f47173ca82ee...1b10dd68f4c9 [21:20:55] [02miraheze/puppet] 07Universal-Omega 031b10dd6 - Fix [21:21:23] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03graphite [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1b10dd68f4c9...54a3b51faa06 [21:21:25] [02miraheze/puppet] 07github-actions 0354a3b51 - CI: lint puppet code to standards [21:21:28] [02puppet] 07github-actions[bot] synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [21:22:14] miraheze/puppet - Universal-Omega the build passed. [21:25:29] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/975604d9856d...4f45b128fda8 [21:25:32] [02miraheze/dns] 07Universal-Omega 034f45b12 - Add graphite [21:27:07] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [21:27:43] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [21:32:23] [02puppet] 07Universal-Omega synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [21:32:26] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03graphite [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/54a3b51faa06...7fc07a53fc9c [21:32:27] [02miraheze/puppet] 07Universal-Omega 037fc07a5 - Fix [21:33:44] miraheze/puppet - Universal-Omega the build passed. [21:38:01] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 180.33 ms [21:50:23] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.44 ms [21:56:32] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.30 ms [22:03:58] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03graphite [+30/-1/±2] 13https://github.com/miraheze/puppet/compare/7fc07a53fc9c...73cb3d38bacc [22:03:59] [02miraheze/puppet] 07Universal-Omega 0373cb3d3 - Use apache [22:04:02] [02puppet] 07Universal-Omega synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [22:04:35] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03graphite [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/73cb3d38bacc...e45e3cc8357b [22:04:38] [02miraheze/puppet] 07github-actions 03e45e3cc - CI: lint puppet code to standards [22:04:40] [02puppet] 07github-actions[bot] synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [22:05:21] miraheze/puppet - Universal-Omega the build passed. [22:12:59] [02puppet] 07Universal-Omega synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [22:13:01] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03graphite [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e45e3cc8357b...0aeb3b33a1e5 [22:13:02] [02miraheze/puppet] 07Universal-Omega 030aeb3b3 - Fix [22:14:35] miraheze/puppet - Universal-Omega the build passed. [22:15:05] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 185.20 ms [22:16:56] PROBLEM - graphite151 NTP time on graphite151 is WARNING: NTP WARNING: Offset 0.2894592881 secs [22:21:15] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.27 ms [22:22:06] [02puppet] 07Universal-Omega synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [22:22:09] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03graphite [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0aeb3b33a1e5...cf3629e6a581 [22:22:10] [02miraheze/puppet] 07Universal-Omega 03cf3629e - Fix [22:23:21] RECOVERY - graphite151 Puppet on graphite151 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [22:23:22] miraheze/puppet - Universal-Omega the build passed. [22:25:14] [02puppet] 07Universal-Omega synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [22:25:15] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03graphite [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cf3629e6a581...a2ed1c190192 [22:25:18] [02miraheze/puppet] 07Universal-Omega 03a2ed1c1 - Fix [22:26:32] miraheze/puppet - Universal-Omega the build passed. [22:30:22] PROBLEM - graphite151 ferm_active on graphite151 is CRITICAL: ERROR ferm input drop default policy not set, ferm might not have been started correctly [22:35:40] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:37:04] [02puppet] 07Universal-Omega synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [22:37:05] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03graphite [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a2ed1c190192...7eecc58ad21e [22:37:08] [02miraheze/puppet] 07Universal-Omega 037eecc58 - Fix [22:38:26] miraheze/puppet - Universal-Omega the build passed. [22:40:50] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:43:16] PROBLEM - prometheus151 HTTPS on prometheus151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [22:45:11] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1006676257 secs [22:50:01] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.10 ms [22:51:02] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09963315725 secs [22:54:03] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.43 ms [22:57:13] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/4f45b128fda8...f23bb054c291 [22:57:14] [02miraheze/dns] 07Universal-Omega 03f23bb05 - Add prometheus-pushgateway [22:58:05] PROBLEM - graphite151 NTP time on graphite151 is CRITICAL: NTP CRITICAL: Offset 0.5022411048 secs [23:01:56] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03graphite [+1/-1/±1] 13https://github.com/miraheze/puppet/compare/7eecc58ad21e...e2fa2c9d32bb [23:01:58] [02miraheze/puppet] 07Universal-Omega 03e2fa2c9 - Use apache [23:02:00] [02puppet] 07Universal-Omega synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [23:03:11] miraheze/puppet - Universal-Omega the build passed. [23:04:35] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:08:15] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 175.87 ms [23:09:24] [02puppet] 07Universal-Omega synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [23:09:27] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03graphite [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e2fa2c9d32bb...fd8596723000 [23:09:29] [02miraheze/puppet] 07Universal-Omega 03fd85967 - Fix [23:09:56] [02puppet] 07Universal-Omega synchronize pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [23:09:58] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03graphite [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fd8596723000...a3a32de5682c [23:09:59] [02miraheze/puppet] 07Universal-Omega 03a3a32de - - [23:10:15] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.69 ms [23:10:41] miraheze/puppet - Universal-Omega the build passed. [23:11:45] miraheze/puppet - Universal-Omega the build passed. [23:12:32] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:21:34] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/f23bb054c291...cb9c6b4a2f78 [23:21:37] [02miraheze/dns] 07Universal-Omega 03cb9c6b4 - Use private IP [23:26:39] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.09 ms [23:27:20] PROBLEM - prometheus151 ferm_active on prometheus151 is CRITICAL: ERROR ferm input drop default policy not set, ferm might not have been started correctly [23:28:38] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.96 ms [23:28:38] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9697MiB (10% inode=98%); [23:36:50] [02puppet] 07Universal-Omega closed pull request 03#3778: Add graphite - 13https://github.com/miraheze/puppet/pull/3778 [23:36:53] [02miraheze/puppet] 07Universal-Omega deleted branch 03graphite [23:36:55] [02puppet] 07Universal-Omega deleted branch 03graphite - 13https://github.com/miraheze/puppet [23:36:57] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+69/-0/±4] 13https://github.com/miraheze/puppet/compare/ebb586c04b1a...08ee24791542 [23:36:59] [02miraheze/puppet] 07Universal-Omega 0308ee247 - Add graphite (#3778) [23:37:55] PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9689MiB (10% inode=98%); [23:37:55] RECOVERY - cloud17 Puppet on cloud17 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:37:57] RECOVERY - db162 Puppet on db162 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:37:58] RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:38:01] RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:38:02] RECOVERY - mem161 Puppet on mem161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:38:08] miraheze/puppet - Universal-Omega the build passed. [23:38:08] RECOVERY - swiftobject171 Puppet on swiftobject171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:38:11] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 1 minute ago with 2 failures. Failed resources (up to 3 shown): Exec[git_clone_MediaWiki-REL1_41 SemanticMediaWiki],Exec[git_checkout_MediaWiki-REL1_41 UnlinkedWikibase] [23:38:13] RECOVERY - cloud18 Puppet on cloud18 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:38:14] RECOVERY - os162 Puppet on os162 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:38:18] RECOVERY - db151 Puppet on db151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:38:18] RECOVERY - swiftproxy161 Puppet on swiftproxy161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:38:22] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:38:23] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:38:24] PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 1 minute ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki-REL1_41 MirahezeMagic],Exec[git_checkout_MediaWiki-REL1_41 UnlinkedWikibase],Exec[git_checkout_MediaWiki-REL1_41 WikiSEO] [23:38:31] RECOVERY - cp26 Puppet on cp26 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:38:31] RECOVERY - swiftobject181 Puppet on swiftobject181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:38:31] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:38:32] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:38:32] RECOVERY - swiftobject161 Puppet on swiftobject161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:38:38] RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:38:39] RECOVERY - os151 Puppet on os151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:38:39] RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:38:40] RECOVERY - os161 Puppet on os161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:38:41] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:38:43] RECOVERY - matomo151 Puppet on matomo151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:38:44] RECOVERY - jobchron171 Puppet on jobchron171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:38:47] RECOVERY - graylog161 Puppet on graylog161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:38:49] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:38:50] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:38:50] RECOVERY - bast161 Puppet on bast161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:38:53] RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:38:56] RECOVERY - reports171 Puppet on reports171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:39:07] RECOVERY - cp27 Puppet on cp27 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:39:13] RECOVERY - swiftobject151 Puppet on swiftobject151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:39:14] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:39:14] RECOVERY - mem151 Puppet on mem151 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [23:39:14] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:39:15] RECOVERY - cp37 Puppet on cp37 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [23:39:16] RECOVERY - swiftac171 Puppet on swiftac171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:39:21] RECOVERY - bast181 Puppet on bast181 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [23:39:22] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:39:27] RECOVERY - db171 Puppet on db171 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [23:39:28] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:39:28] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:39:33] RECOVERY - ldap171 Puppet on ldap171 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [23:39:33] RECOVERY - rdb151 Puppet on rdb151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:39:46] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Exec[git_clone_MediaWiki-REL1_41 SemanticMediaWiki] [23:39:53] RECOVERY - cp36 Puppet on cp36 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [23:40:02] RECOVERY - swiftproxy171 Puppet on swiftproxy171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures