[00:01:31] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 50% [00:01:32] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:03:30] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 67% [00:03:32] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:03:55] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 21578MiB (24% inode=98%); [00:05:13] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 21557MiB (24% inode=98%); [00:19:02] [02mw-config] 07AgentIsai closed pull request 03#4971: Add DPE reasons - 13https://github.com/miraheze/mw-config/pull/4971 [00:19:03] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c795d7dabf77...614463605cd6 [00:19:06] [02miraheze/mw-config] 07AgentIsai 036144636 - Add DPE reasons (#4971) [00:19:52] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1005874574 secs [00:20:01] miraheze/mw-config - AgentIsai the build passed. [00:23:29] !log [@test151] starting deploy of {'config': True} to test151 [00:23:30] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [00:23:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:23:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:29:01] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.09 ms [00:30:51] [02miraheze/puppet] 07Universal-Omega created branch 03statsd 13https://github.com/miraheze/puppet/commit/08ee24791542064acde5b7bae11747f28741cc8c [00:30:53] [02puppet] 07Universal-Omega created branch 03statsd - 13https://github.com/miraheze/puppet [00:31:00] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.68 ms [00:31:39] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03statsd [+10/-0/±1] 13https://github.com/miraheze/puppet/compare/08ee24791542...710b13e90e77 [00:31:41] [02miraheze/puppet] 07Universal-Omega 03710b13e - Add statsd to Graphite [00:31:49] !log [@mwtask181] starting deploy of {'config': True} to all [00:31:50] [02puppet] 07Universal-Omega opened pull request 03#3780: Add statsd to Graphite - 13https://github.com/miraheze/puppet/pull/3780 [00:31:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:31:59] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 9s [00:32:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:30] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03statsd [+0/-0/±4] 13https://github.com/miraheze/puppet/compare/710b13e90e77...9115a0fea92e [00:32:31] [02miraheze/puppet] 07github-actions 039115a0f - CI: lint puppet code to standards [00:32:34] [02puppet] 07github-actions[bot] synchronize pull request 03#3780: Add statsd to Graphite - 13https://github.com/miraheze/puppet/pull/3780 [00:33:52] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09975594282 secs [00:37:26] !log [@mwtask171] starting deploy of {'config': True} to all [00:37:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:37:35] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [00:37:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:06:22] RECOVERY - graphite151 ferm_active on graphite151 is OK: OK ferm input default policy is set [01:07:48] [02puppet] 07Universal-Omega synchronize pull request 03#3780: Add statsd to Graphite - 13https://github.com/miraheze/puppet/pull/3780 [01:07:49] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03statsd [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9115a0fea92e...5f8fd39a1445 [01:07:52] [02miraheze/puppet] 07Universal-Omega 035f8fd39 - - [01:08:27] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03statsd [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5f8fd39a1445...aa98a92721f5 [01:08:28] [02miraheze/puppet] 07github-actions 03aa98a92 - CI: lint puppet code to standards [01:08:30] [02puppet] 07github-actions[bot] synchronize pull request 03#3780: Add statsd to Graphite - 13https://github.com/miraheze/puppet/pull/3780 [01:08:34] [02puppet] 07Universal-Omega synchronize pull request 03#3780: Add statsd to Graphite - 13https://github.com/miraheze/puppet/pull/3780 [01:08:35] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03statsd [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/aa98a92721f5...19b2be3b52bf [01:08:36] [02miraheze/puppet] 07Universal-Omega 0319b2be3 - - [01:09:01] PROBLEM - graphite151 Puppet on graphite151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:09:23] [02puppet] 07Universal-Omega synchronize pull request 03#3780: Add statsd to Graphite - 13https://github.com/miraheze/puppet/pull/3780 [01:09:25] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03statsd [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/19b2be3b52bf...25879fbd9ba7 [01:09:27] [02miraheze/puppet] 07Universal-Omega 0325879fb - - [01:17:29] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 173.44 ms [01:18:03] PROBLEM - graphite151 statsd-proxy process on graphite151 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'statsd-proxy' [01:19:27] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.83 ms [01:27:54] [02puppet] 07Universal-Omega synchronize pull request 03#3780: Add statsd to Graphite - 13https://github.com/miraheze/puppet/pull/3780 [01:27:55] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03statsd [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/25879fbd9ba7...278f7cc8ce0b [01:27:57] [02miraheze/puppet] 07Universal-Omega 03278f7cc - Fix [01:29:14] [02puppet] 07Universal-Omega synchronize pull request 03#3780: Add statsd to Graphite - 13https://github.com/miraheze/puppet/pull/3780 [01:29:17] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03statsd [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/278f7cc8ce0b...c955da7fce2e [01:29:19] [02miraheze/puppet] 07Universal-Omega 03c955da7 - Fix [01:30:45] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03statsd [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c955da7fce2e...70edc73de474 [01:30:47] [02miraheze/puppet] 07Universal-Omega 0370edc73 - Fix [01:30:49] [02puppet] 07Universal-Omega synchronize pull request 03#3780: Add statsd to Graphite - 13https://github.com/miraheze/puppet/pull/3780 [01:32:00] RECOVERY - graphite151 statsd-proxy process on graphite151 is OK: PROCS OK: 1 process with command name 'statsd-proxy' [01:44:32] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03statsd [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/70edc73de474...8fccfa6d8418 [01:44:35] [02miraheze/puppet] 07Universal-Omega 038fccfa6 - Add [01:44:37] [02puppet] 07Universal-Omega synchronize pull request 03#3780: Add statsd to Graphite - 13https://github.com/miraheze/puppet/pull/3780 [01:47:12] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03statsd [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8fccfa6d8418...1017005d548a [01:47:15] [02miraheze/puppet] 07Universal-Omega 031017005 - use systemd::service [01:47:18] [02puppet] 07Universal-Omega synchronize pull request 03#3780: Add statsd to Graphite - 13https://github.com/miraheze/puppet/pull/3780 [01:48:59] RECOVERY - graphite151 Puppet on graphite151 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [01:49:21] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 170.32 ms [01:50:53] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03statsd [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1017005d548a...ea3b0842bc72 [01:50:54] [02miraheze/puppet] 07Universal-Omega 03ea3b084 - use www-data [01:50:57] [02puppet] 07Universal-Omega synchronize pull request 03#3780: Add statsd to Graphite - 13https://github.com/miraheze/puppet/pull/3780 [01:51:19] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.74 ms [01:57:46] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/614463605cd6...e11655d347d0 [01:57:49] [02miraheze/mw-config] 07Universal-Omega 03e11655d - Set $wgStatsdServer [01:58:29] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [01:58:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:58:39] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [01:58:45] miraheze/mw-config - Universal-Omega the build passed. [01:58:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:07:13] !log [@mwtask171] starting deploy of {'config': True} to all [02:07:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:07:21] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [02:07:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:08:17] PROBLEM - graphite151 ferm_active on graphite151 is CRITICAL: ERROR ferm input drop default policy not set, ferm might not have been started correctly [02:08:37] [02puppet] 07Universal-Omega synchronize pull request 03#3780: Add statsd to Graphite - 13https://github.com/miraheze/puppet/pull/3780 [02:08:39] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03statsd [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ea3b0842bc72...a9e973179d90 [02:08:41] [02miraheze/puppet] 07Universal-Omega 03a9e9731 - Use localhost [02:10:02] PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 2.77, 2.18, 1.60 [02:13:54] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/08ee24791542...e16a1b042605 [02:13:57] [02miraheze/puppet] 07Universal-Omega 03e16a1b0 - graphite-index: fix python3 path [02:14:44] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e16a1b042605...4c8120a8a070 [02:14:47] [02miraheze/puppet] 07Universal-Omega 034c8120a - Add /usr/bin/python3 [02:14:56] [02miraheze/puppet] 07Universal-Omega pushed 0313 commits to 03statsd [+11/-0/±15] 13https://github.com/miraheze/puppet/compare/a9e973179d90...1defdf74d8e2 [02:14:58] [02miraheze/puppet] 07Universal-Omega 038940272 - Add statsd to Graphite [02:15:01] [02miraheze/puppet] 07github-actions 03e8a5abf - CI: lint puppet code to standards [02:15:03] [02miraheze/puppet] 07Universal-Omega 03d0b68c0 - - [02:15:05] [02miraheze/puppet] ... and 10 more commits. [02:15:08] [02puppet] 07Universal-Omega synchronize pull request 03#3780: Add statsd to Graphite - 13https://github.com/miraheze/puppet/pull/3780 [02:15:35] miraheze/puppet - Universal-Omega the build passed. [02:21:55] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.53, 2.00, 1.89 [02:23:08] !log [@test151] starting deploy of {'config': True} to test151 [02:23:09] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [02:23:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:23:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:26:13] [02puppet] 07Universal-Omega synchronize pull request 03#3780: Add statsd to Graphite - 13https://github.com/miraheze/puppet/pull/3780 [02:26:14] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03statsd [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1defdf74d8e2...30686d5b7f94 [02:26:16] [02miraheze/puppet] 07Universal-Omega 0330686d5 - Add system::role [02:27:50] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 0.87, 1.39, 1.67 [02:28:18] [02puppet] 07Universal-Omega closed pull request 03#3780: Add statsd to Graphite - 13https://github.com/miraheze/puppet/pull/3780 [02:28:20] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+11/-0/±1] 13https://github.com/miraheze/puppet/compare/4c8120a8a070...c9e162e26228 [02:28:21] [02miraheze/puppet] 07Universal-Omega 03c9e162e - Add statsd to Graphite (#3780) [02:28:24] [02miraheze/puppet] 07Universal-Omega deleted branch 03statsd [02:28:26] [02puppet] 07Universal-Omega deleted branch 03statsd - 13https://github.com/miraheze/puppet [02:29:57] RECOVERY - graphite151 ferm_active on graphite151 is OK: OK ferm input default policy is set [02:30:34] RECOVERY - prometheus151 ferm_active on prometheus151 is OK: OK ferm input default policy is set [02:41:37] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.86, 1.92, 1.73 [02:45:04] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 165.67 ms [02:45:55] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1017276049 secs [02:47:32] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 1.18, 1.51, 1.62 [02:49:52] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09958511591 secs [02:54:59] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.95 ms [02:56:44] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1011051536 secs [03:12:57] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 177.84 ms [03:20:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:20:52] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.98 ms [03:26:20] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09866005182 secs [03:33:54] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 171.96 ms [03:34:16] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03statsd-timing [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/commit/5d513e5d77be [03:34:17] [02miraheze/MirahezeMagic] 07Universal-Omega 035d513e5 - Add statsd timing measurements [03:34:18] [02MirahezeMagic] 07Universal-Omega created branch 03statsd-timing - 13https://github.com/miraheze/MirahezeMagic [03:34:21] [02MirahezeMagic] 07Universal-Omega opened pull request 03#482: Add statsd timing measurements - 13https://github.com/miraheze/MirahezeMagic/pull/482 [03:35:03] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03statsd-timing [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/5d513e5d77be...f1058c344906 [03:35:04] [02miraheze/MirahezeMagic] 07Universal-Omega 03f1058c3 - Add hook [03:35:07] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#482: Add statsd timing measurements - 13https://github.com/miraheze/MirahezeMagic/pull/482 [03:38:44] miraheze/MirahezeMagic - Universal-Omega the build has errored. [03:41:29] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#482: Add statsd timing measurements - 13https://github.com/miraheze/MirahezeMagic/pull/482 [03:41:30] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03statsd-timing [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/f1058c344906...a4fa1b2cfe08 [03:41:32] [02miraheze/MirahezeMagic] 07Universal-Omega 03a4fa1b2 - Fix [03:42:32] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03statsd-timing [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/a4fa1b2cfe08...583659628c1c [03:42:34] [02miraheze/MirahezeMagic] 07Universal-Omega 035836596 - Add MobileFrontend to phan_dependencies [03:42:36] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#482: Add statsd timing measurements - 13https://github.com/miraheze/MirahezeMagic/pull/482 [03:43:14] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03statsd-timing [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/583659628c1c...96bdd0736352 [03:43:16] [02miraheze/MirahezeMagic] 07Universal-Omega 0396bdd07 - Add MobileFrontend to phan config [03:43:19] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#482: Add statsd timing measurements - 13https://github.com/miraheze/MirahezeMagic/pull/482 [03:43:51] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 187.36 ms [03:45:33] miraheze/MirahezeMagic - Universal-Omega the build has errored. [03:46:27] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03statsd-timing [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/96bdd0736352...bbc73c63b6ac [03:46:29] [02miraheze/MirahezeMagic] 07Universal-Omega 03bbc73c6 - - [03:46:32] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#482: Add statsd timing measurements - 13https://github.com/miraheze/MirahezeMagic/pull/482 [03:50:05] miraheze/MirahezeMagic - Universal-Omega the build has errored. [03:50:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:51:51] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 172.55 ms [03:53:37] [02MirahezeMagic] 07Universal-Omega commented on pull request 03#482: Add statsd timing measurements - 13https://github.com/miraheze/MirahezeMagic/pull/482#issuecomment-2008623812 [03:53:41] [02MirahezeMagic] 07Universal-Omega closed pull request 03#482: Add statsd timing measurements - 13https://github.com/miraheze/MirahezeMagic/pull/482 [03:53:44] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03statsd-timing [03:53:47] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/MirahezeMagic/compare/0ba4ace854cd...3c38baebed6b [03:53:49] [02miraheze/MirahezeMagic] 07Universal-Omega 033c38bae - Add statsd timing measurements (#482) [03:53:51] [02MirahezeMagic] 07Universal-Omega deleted branch 03statsd-timing - 13https://github.com/miraheze/MirahezeMagic [03:53:57] !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 [03:53:58] !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [03:54:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:54:07] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [03:54:07] !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [03:54:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:54:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:54:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:54:55] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [03:55:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:55:04] !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 9s [03:55:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:55:50] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.51 ms [03:57:09] miraheze/MirahezeMagic - Universal-Omega the build has errored. [03:59:56] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 176.38 ms [04:01:54] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 192.54 ms [04:10:01] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 191.55 ms [04:11:59] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 173.51 ms [04:16:23] PROBLEM - graphite151 ferm_active on graphite151 is CRITICAL: ERROR ferm input drop default policy not set, ferm might not have been started correctly [04:18:21] RECOVERY - graphite151 ferm_active on graphite151 is OK: OK ferm input default policy is set [04:18:44] PROBLEM - graphite151 NTP time on graphite151 is WARNING: NTP WARNING: Offset 0.1482168436 secs [04:26:16] PROBLEM - graphite151 ferm_active on graphite151 is CRITICAL: ERROR ferm input drop default policy not set, ferm might not have been started correctly [04:27:03] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c9e162e26228...802cee6bb066 [04:27:06] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 185.63 ms [04:27:06] [02miraheze/puppet] 07Universal-Omega 03802cee6 - Use www-data [04:30:13] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/802cee6bb066...a746923e77cb [04:30:17] [02miraheze/puppet] 07Universal-Omega 03a746923 - jobrunner-hi: add statsd [04:31:03] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.20 ms [04:36:09] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a746923e77cb...26a0dc842c10 [04:36:11] [02miraheze/puppet] 07Universal-Omega 0326a0dc8 - jobrunner: add statsd [05:05:35] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1010570228 secs [05:27:29] PROBLEM - graphite151 NTP time on graphite151 is CRITICAL: NTP CRITICAL: Offset 0.5051929653 secs [05:28:48] PROBLEM - graphite151 Puppet on graphite151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 4 minutes ago with 0 failures [05:30:48] RECOVERY - graphite151 Puppet on graphite151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [05:43:06] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 234.56 ms [05:49:02] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 218.53 ms [05:53:13] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.78 ms [05:55:11] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 191.78 ms [06:10:20] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 173.40 ms [06:18:23] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.18 ms [06:22:34] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 188.82 ms [06:30:40] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.96 ms [06:33:36] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09181466699 secs [06:37:35] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1283212304 secs [06:57:35] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09723106027 secs [06:57:48] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 181.49 ms [07:05:48] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 199.87 ms [07:09:53] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 186.51 ms [07:27:56] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.09 ms [07:45:22] PROBLEM - airlineinsider.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'airlineinsider.org' expires in 15 day(s) (Fri 05 Apr 2024 07:18:27 AM GMT +0000). [07:45:35] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f5ebbdeb192d...7608845ab680 [07:45:38] [02miraheze/ssl] 07MirahezeSSLBot 037608845 - Bot: Update SSL cert for airlineinsider.org [07:46:30] miraheze/ssl - MirahezeSSLBot the build has errored. [08:05:30] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [08:07:07] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 211.98 ms [08:09:05] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 260.56 ms [08:14:17] RECOVERY - airlineinsider.org - LetsEncrypt on sslhost is OK: OK - Certificate 'airlineinsider.org' will expire on Tue 18 Jun 2024 06:45:29 AM GMT +0000. [08:19:36] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 56% [08:31:46] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:35:58] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [08:47:03] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 169.11 ms [08:49:01] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 174.11 ms [09:02:17] PROBLEM - cloud16 Puppet on cloud16 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [09:05:51] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK [09:13:09] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 203.84 ms [09:15:07] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 184.30 ms [09:30:17] RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [09:34:14] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 205.74 ms [09:40:15] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 173.26 ms [09:44:19] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.59 ms [09:46:17] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.74 ms [09:49:35] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1009804606 secs [09:53:23] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 175.59 ms [10:01:20] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.11 ms [10:12:21] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.57 ms [10:17:34] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09934321046 secs [10:20:18] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 177.08 ms [10:27:24] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.35 ms [10:33:23] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.70 ms [10:39:29] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 171.21 ms [10:47:29] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.16 ms [11:09:20] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1014026105 secs [11:21:10] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.0985904932 secs [13:11:37] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1027615368 secs [13:33:36] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.07461127639 secs [13:37:35] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1021140516 secs [14:29:36] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.08413940668 secs [14:33:35] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1204913557 secs [15:09:45] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [15:10:48] PROBLEM - ping6 on cp27 is WARNING: PING WARNING - Packet loss = 0%, RTA = 208.98 ms [15:13:45] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 59% [15:14:47] RECOVERY - ping6 on cp27 is OK: PING OK - Packet loss = 0%, RTA = 152.35 ms [15:19:45] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [16:21:50] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.46 ms [16:25:47] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.26 ms [16:37:50] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 170.51 ms [16:41:47] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 166.67 ms [16:51:35] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09207180142 secs [17:36:00] @bluemoon0332 does wkmr.liao.media seem pointed to you? [17:36:15] I don't see it pointed but often I seem to be corrected on that so just double checking [17:39:22] it is pointed in the most unbeliavable of ways [17:39:37] [1/4] Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems: [17:39:37] [2/4] Domain: wkmr.liao.media [17:39:37] [3/4] Type: dns [17:39:38] [4/4] Detail: DNS problem: SERVFAIL looking up A for wkmr.liao.media - the domain's nameservers may be malfunctioning; DNS problem: SERVFAIL looking up AAAA for wkmr.liao.media - the domain's nameservers may be malfunctioning [17:39:39] [1/25] ``` [17:39:39] [2/25] localhost:~$ kdig @v2n0.nic.media wkmr.liao.media NS [17:39:39] [3/25] ;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 48078 [17:39:39] [4/25] ;; Flags: qr rd; QUERY: 1; ANSWER: 0; AUTHORITY: 4; ADDITIONAL: 0 [17:39:40] [5/25] ;; QUESTION SECTION: [17:39:40] [6/25] ;; wkmr.liao.media. IN NS [17:39:40] [7/25] ;; AUTHORITY SECTION: [17:39:41] [8/25] liao.media. 3600 IN NS ns1-32.azure-dns.com. [17:39:41] [9/25] liao.media. 3600 IN NS ns2-32.azure-dns.net. [17:39:41] [10/25] liao.media. 3600 IN NS ns3-32.azure-dns.org. [17:39:42] [11/25] liao.media. 3600 IN NS ns4-32.azure-dns.info. [17:39:42] [12/25] ;; Received 170 B [17:39:43] [13/25] ;; Time 2024-03-20 18:38:07 CET [17:39:43] [14/25] ;; From 65.22.23.3@53(UDP) in 48.6 ms [17:39:44] [15/25] localhost:~$ kdig @ns1-32.azure-dns.com wkmr.liao.media NS [17:39:44] [16/25] ;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 12319 [17:39:45] [17/25] ;; Flags: qr rd ad; QUERY: 1; ANSWER: 0; AUTHORITY: 2; ADDITIONAL: 0 [17:39:45] [18/25] ;; QUESTION SECTION: [17:39:46] [19/25] ;; wkmr.liao.media. IN NS [17:39:46] [20/25] ;; AUTHORITY SECTION: [17:39:47] [21/25] wkmr.liao.media. 31449600 IN NS ns1.miraheze.org. [17:39:47] [22/25] wkmr.liao.media. 31449600 IN NS ns2.miraheze.org. [17:39:48] [23/25] ;; Received 81 B [17:39:48] [24/25] ;; Time 2024-03-20 18:39:09 CET [17:39:49] [25/25] ;; From 150.171.10.32@53(UDP) in 17.5 ms``` [17:39:57] [1/6] oh, NS? I guess the WHOIS I used is lying to me: [17:39:57] [2/6] Name Servers: [17:39:58] [3/6] ns1-32.azure-dns.com [17:39:58] [4/6] ns2-32.azure-dns.net [17:39:58] [5/6] ns3-32.azure-dns.org [17:39:59] [6/6] ns4-32.azure-dns.info [17:40:06] it is not [17:40:12] it's saying the truth [17:40:12] oh I see, I didn't know you could do that? [17:40:17] point a subdomain via NS? how did they do that? [17:40:40] honestly I don't think this will even work [17:40:55] why not just a CNAME if it is a subdomain??????? [17:53:19] [02mw-config] 07redbluegreenhat opened pull request 03#5504: grant everyone centralauth-merge - 13https://github.com/miraheze/mw-config/pull/5504 [17:54:24] miraheze/mw-config - redbluegreenhat the build passed. [17:58:44] [02mw-config] 07redbluegreenhat edited pull request 03#5504: DNM: grant everyone centralauth-merge - 13https://github.com/miraheze/mw-config/pull/5504 [18:07:19] yeah, I don't know how they even did that [18:15:42] [02mw-config] 07RhinosF1 commented on pull request 03#5504: DNM: grant everyone centralauth-merge - 13https://github.com/miraheze/mw-config/pull/5504#issuecomment-2010298876 [18:16:49] Yes you can [18:17:39] @rhinosf1 re: https://github.com/miraheze/mw-config/pull/5504#issuecomment-2010298876, yeah, I'm looking into https://github.com/wikimedia/mediawiki-extensions-CentralAuth/blob/master/maintenance/attachAccount.php [18:18:39] Yes [18:19:04] Very rarely things can go ever so slightly wrong [18:19:21] CentralAuth-merge for everyone is a bad idea to fix a once a year bug [18:20:07] What's the account [18:20:18] Thrace, both on-wiki and global [18:21:06] Looking [18:21:54] @bluemoon0332 https://meta.miraheze.org/wiki/Special:CentralAuth?target=Thrace looks attached on every wiki [18:21:58] Which wiki isn't there [18:22:13] kylariswiki [18:22:20] https://discord.com/channels/407504499280707585/1219707363347533905 [18:23:10] @bluemoon0332 probably best to do https://github.com/wikimedia/mediawiki-extensions-CentralAuth/blob/master/maintenance/migrateAccount.php [18:23:33] ohh perfect [18:24:28] run mwscript CentralAuth/migrateLocalAccount.php kylariswiki --username=Thrace --auto --attachmissing --attachbroken --homewiki metawiki [18:24:38] @bluemoon0332 [18:25:45] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/CentralAuth/migrateAccount.php --wiki=kylariswiki --username=Thrace --auto --attachmissing --attachbroken --homewiki=metawiki (END - exit=256) [18:25:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:26:31] I guess that didn't work [18:26:36] What did it say @bluemoon0332 [18:27:15] that the script doesn't exist [18:27:23] Huh [18:27:26] which seems obvious by the path [18:27:42] should be extensions/CentralAuth/migrateAccount.php [18:28:06] Oh ye [18:28:07]