[00:26:30] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [00:58:12] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:10:30] [02mw-config] 07brandon-wm synchronize pull request 03#5559: T11757: Disable Upload Wizard Tutorial and Remove Creative Commons selection on aieseattle.miraheze.org - 13https://github.com/miraheze/mw-config/pull/5559 [02:10:43] [02mw-config] 07brandon-wm synchronize pull request 03#5559: T11757: Disable Upload Wizard Tutorial and Remove Creative Commons selection on aieseattle.miraheze.org - 13https://github.com/miraheze/mw-config/pull/5559 [02:11:26] miraheze/mw-config - brandon-wm the build passed. [02:11:40] miraheze/mw-config - brandon-wm the build passed. [02:37:43] PROBLEM - wiki.pulsus.cc - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.pulsus.cc' expires in 15 day(s) (Fri 31 May 2024 02:07:57 AM GMT +0000). [02:37:55] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/da200871d9db...a736332f3b51 [02:37:57] [02ssl] 07WikiTideSSLBot 03a736332 - Bot: Update SSL cert for wiki.pulsus.cc [02:44:25] [02mediawiki-repos] 07brandon-wm opened pull request 03#16: T11623: Request for PageLanguage and PerPageLanguage extensions for rumtr.miraheze.org - 13https://github.com/miraheze/mediawiki-repos/pull/16 [02:46:06] [02mediawiki-repos] 07brandon-wm synchronize pull request 03#16: T11623: Request for PageLanguage and PerPageLanguage extensions for rumtr.miraheze.org - 13https://github.com/miraheze/mediawiki-repos/pull/16 [02:49:30] [02mediawiki-repos] 07brandon-wm synchronize pull request 03#16: T11623: Request for PageLanguage and PerPageLanguage extensions for rumtr.miraheze.org - 13https://github.com/miraheze/mediawiki-repos/pull/16 [02:51:51] [02mediawiki-repos] 07Universal-Omega edited pull request 03#16: T11623: add PageLanguage and PerPageLanguage - 13https://github.com/miraheze/mediawiki-repos/pull/16 [02:54:49] [02mw-config] 07brandon-wm opened pull request 03#5562: Add PageLanguage and PerPageLanguage to MW extensions - 13https://github.com/miraheze/mw-config/pull/5562 [02:55:42] miraheze/mw-config - brandon-wm the build passed. [02:56:28] [02mw-config] 07brandon-wm synchronize pull request 03#5562: Add PageLanguage and PerPageLanguage to MW extensions - 13https://github.com/miraheze/mw-config/pull/5562 [02:57:06] [02mw-config] 07brandon-wm synchronize pull request 03#5562: Add PageLanguage and PerPageLanguage to MW extensions - 13https://github.com/miraheze/mw-config/pull/5562 [02:57:25] miraheze/mw-config - brandon-wm the build passed. [02:58:01] miraheze/mw-config - brandon-wm the build passed. [02:58:14] [02mw-config] 07brandon-wm synchronize pull request 03#5562: Add PageLanguage and PerPageLanguage to MW extensions - 13https://github.com/miraheze/mw-config/pull/5562 [02:59:04] miraheze/mw-config - brandon-wm the build passed. [03:02:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:07:25] RECOVERY - wiki.pulsus.cc - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.pulsus.cc' will expire on Tue 13 Aug 2024 01:37:49 AM GMT +0000. [03:26:06] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:28:25] [02mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/940a3a48a182...5287a1de8808 [03:28:26] [02mw-config] 07Universal-Omega 035287a1d - Remove Meta Tech NS sitenotice [03:29:23] miraheze/mw-config - Universal-Omega the build passed. [03:32:51] !log [@mwtask181] starting deploy of {'config': True} to all [03:32:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:33:01] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 10s [03:33:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:37:52] !log [@mwtask171] starting deploy of {'config': True} to all [03:37:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:38:01] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [03:38:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:39:41] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 188.53 ms [03:41:39] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 122.00 ms [03:53:07] !log [@test151] starting deploy of {'config': True} to test151 [03:53:08] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [03:53:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:53:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:16:32] [02mw-config] 07clarfonthey commented on pull request 03#5562: Add PageLanguage and PerPageLanguage to MW extensions - 13https://github.com/miraheze/mw-config/pull/5562#issuecomment-2111549280 [04:30:54] [02mw-config] 07clarfonthey edited a comment on pull request 03#5562: Add PageLanguage and PerPageLanguage to MW extensions - 13https://github.com/miraheze/mw-config/pull/5562#issuecomment-2111549280 [05:52:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:08:35] [02mw-config] 07colleirose commented on pull request 03#5521: Add "Special:" in more languages - 13https://github.com/miraheze/mw-config/pull/5521#issuecomment-2111655320 [06:08:46] [02mw-config] 07colleirose synchronize pull request 03#5521: Add "Special:" in more languages - 13https://github.com/miraheze/mw-config/pull/5521 [06:09:50] miraheze/mw-config - colleirose the build passed. [06:55:08] [02mw-config] 07Universal-Omega closed pull request 03#5521: Add "Special:" in more languages - 13https://github.com/miraheze/mw-config/pull/5521 [06:55:09] [02mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5287a1de8808...e269b23dc3d8 [06:55:10] [02mw-config] 07colleirose 03e269b23 - Add "Special:" in more languages (#5521) [06:56:03] miraheze/mw-config - Universal-Omega the build passed. [06:56:12] [02mw-config] 07Universal-Omega edited pull request 03#5561: T12085: About wgTabberNeueParseTabName to ManageWiki - 13https://github.com/miraheze/mw-config/pull/5561 [06:57:07] [02mw-config] 07Universal-Omega edited pull request 03#5561: T12085: Add wgTabberNeueParseTabName to ManageWiki - 13https://github.com/miraheze/mw-config/pull/5561 [06:58:11] [02mw-config] 07Universal-Omega edited pull request 03#5559: T11757: set some UploadWizard config for aieseattlewiki - 13https://github.com/miraheze/mw-config/pull/5559 [06:58:52] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 192.67 ms [06:59:46] [02mediawiki-repos] 07Universal-Omega closed pull request 03#16: T11623: add PageLanguage and PerPageLanguage - 13https://github.com/miraheze/mediawiki-repos/pull/16 [06:59:49] [02mediawiki-repos] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/5c341b55acaf...320fe4664786 [06:59:50] [02mediawiki-repos] 07brandon-wm 03320fe46 - T11623: add PageLanguage and PerPageLanguage (#16) [07:00:50] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 195.65 ms [07:02:24] !log [@mwtask181] starting deploy of {'config': True} to all [07:02:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:02:34] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 10s [07:02:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:07:02] !log [@mwtask171] starting deploy of {'config': True} to all [07:07:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:07:10] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [07:07:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:23:27] !log [@test151] starting deploy of {'config': True} to test151 [07:23:28] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [07:23:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:23:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:42:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:08:48] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 190.69 ms [08:20:50] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 124.33 ms [08:32:53] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 186.44 ms [08:42:51] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 192.01 ms [09:05:01] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 187.28 ms [09:13:00] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 188.30 ms [09:20:08] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 185.75 ms [09:24:08] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 189.81 ms [09:33:17] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 185.99 ms [09:51:16] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 194.16 ms [09:55:25] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 193.31 ms [09:59:25] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 189.05 ms [10:06:36] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 192.44 ms [10:16:36] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 202.90 ms [10:22:42] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 192.25 ms [10:26:26] PROBLEM - db181 Backups SQL on db181 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864236 seconds old and 125241 bytes [10:26:40] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 193.77 ms [10:29:47] PROBLEM - lebork.info.pl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'lebork.info.pl' expires in 15 day(s) (Fri 31 May 2024 10:26:26 AM GMT +0000). [10:30:01] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a736332f3b51...110dbae7e9cf [10:30:04] [02ssl] 07WikiTideSSLBot 03110dbae - Bot: Update SSL cert for lebork.info.pl [10:30:47] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 196.23 ms [10:36:44] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 193.02 ms [10:46:52] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 191.62 ms [10:52:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [10:59:03] RECOVERY - lebork.info.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'lebork.info.pl' will expire on Tue 13 Aug 2024 09:29:55 AM GMT +0000. [11:05:02] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 195.82 ms [11:09:09] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 195.04 ms [11:09:42] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [11:43:28] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 195.57 ms [11:47:42] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 195.76 ms [11:50:04] PROBLEM - db161 Backups SQL on db161 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864209 seconds old and 125354 bytes [11:53:41] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 195.74 ms [12:01:56] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 195.37 ms [12:02:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:03:55] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 195.97 ms [12:07:37] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [12:12:08] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 200.30 ms [12:17:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:42:17] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 197.03 ms [12:46:26] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 199.19 ms [12:48:25] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 196.06 ms [12:55:40] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 198.46 ms [13:09:41] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 198.21 ms [13:13:53] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 197.52 ms [13:17:05] PROBLEM - wiki.sukotuze.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.sukotuze.com' expires in 15 day(s) (Fri 31 May 2024 01:05:39 PM GMT +0000). [13:17:22] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/110dbae7e9cf...2279ee363be0 [13:17:25] [02ssl] 07WikiTideSSLBot 032279ee3 - Bot: Update SSL cert for wiki.sukotuze.com [13:21:55] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 197.47 ms [13:28:26] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:30:13] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 200.21 ms [13:39:42] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [13:45:41] RECOVERY - wiki.sukotuze.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.sukotuze.com' will expire on Tue 13 Aug 2024 12:17:15 PM GMT +0000. [13:52:21] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 213.46 ms [13:56:28] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 197.74 ms [13:58:45] PROBLEM - dlfm-wiki.top - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dlfm-wiki.top' expires in 15 day(s) (Fri 31 May 2024 01:29:00 PM GMT +0000). [13:58:58] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2279ee363be0...9bd983f2c2da [13:58:59] [02ssl] 07WikiTideSSLBot 039bd983f - Bot: Update SSL cert for dlfm-wiki.top [14:02:26] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 198.21 ms [14:06:38] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 197.95 ms [14:24:49] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 198.66 ms [14:26:38] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [14:28:38] RECOVERY - dlfm-wiki.top - LetsEncrypt on sslhost is OK: OK - Certificate 'dlfm-wiki.top' will expire on Tue 13 Aug 2024 12:58:51 PM GMT +0000. [14:29:00] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 197.63 ms [14:35:00] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 206.26 ms [14:35:35] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [14:39:09] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 199.21 ms [14:45:08] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 198.85 ms [14:49:15] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 197.90 ms [14:51:13] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 197.41 ms [14:55:22] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 197.60 ms [14:56:06] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [15:27:59] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 132.30 ms [15:53:32] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 196.27 ms [15:57:35] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 193.52 ms [16:01:46] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 195.74 ms [16:09:48] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 132.51 ms [16:18:59] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 190.40 ms [16:22:58] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 188.23 ms [16:37:10] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 183.91 ms [16:41:08] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 183.00 ms [16:50:48] [02puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+1/-0/±0] 13https://github.com/miraheze/puppet/commit/52b83a6b50b0 [16:50:51] [02puppet] 07Reception123 0352b83a6 - Create partial-reset-wiki.py [16:50:54] [02puppet] 07Reception123 created branch 03Reception123-patch-1 - 13https://github.com/miraheze/puppet [16:51:00] [02puppet] 07Reception123 opened pull request 03#3854: script to facilitate resetting wikis (only partial) - 13https://github.com/miraheze/puppet/pull/3854 [16:52:24] miraheze/puppet - Reception123 the build passed. [16:54:16] [02ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/commit/ecef338300c4 [16:54:17] [02ssl] 07MacFan4000 03ecef338 - Remove vedopedia.witches-empire.com [16:54:20] [02ssl] 07MacFan4000 created branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [16:54:21] [02ssl] 07MacFan4000 opened pull request 03#779: Remove vedopedia.witches-empire.com - 13https://github.com/miraheze/ssl/pull/779 [16:54:27] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 166.20 ms [16:54:45] [02ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ecef338300c4...f313dc3a28b4 [16:54:48] [02ssl] 07MacFan4000 03f313dc3 - Update certs.yaml [16:54:49] [02ssl] 07MacFan4000 synchronize pull request 03#779: Remove vedopedia.witches-empire.com - 13https://github.com/miraheze/ssl/pull/779 [16:55:50] [02ssl] 07MacFan4000 closed pull request 03#779: Remove vedopedia.witches-empire.com - 13https://github.com/miraheze/ssl/pull/779 [16:55:51] [02ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/ssl/compare/9bd983f2c2da...df43d50e3640 [16:55:52] [02ssl] 07MacFan4000 03df43d50 - Remove vedopedia.witches-empire.com (#779) [16:55:53] [02ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 [16:55:56] [02ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [16:57:03] [02dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/7d45e3cbb34f...ee03367cd368 [16:57:05] [02dns] 07Reception123 03ee03367 - Create urbanshade.org [16:58:28] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 135.04 ms [17:00:29] [02puppet] 07Reception123 synchronize pull request 03#3854: script to facilitate resetting wikis (only partial) - 13https://github.com/miraheze/puppet/pull/3854 [17:00:30] [02puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/52b83a6b50b0...0aad97a700d8 [17:00:33] [02puppet] 07Reception123 030aad97a - Update partial-reset-wiki.py [17:02:17] miraheze/puppet - Reception123 the build passed. [17:06:29] PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [17:07:21] [02ImportDump] 07Universal-Omega pushed 031 commit to 03noduplicate [+0/-0/±1] 13https://github.com/miraheze/ImportDump/commit/140908e48456 [17:07:23] [02ImportDump] 07Universal-Omega 03140908e - Prevent duplicate comments [17:07:25] [02ImportDump] 07Universal-Omega created branch 03noduplicate - 13https://github.com/miraheze/ImportDump [17:07:28] [02ImportDump] 07Universal-Omega opened pull request 03#101: Prevent duplicate comments - 13https://github.com/miraheze/ImportDump/pull/101 [17:07:35] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:14:37] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 171.96 ms [17:15:37] miraheze/ImportDump - Universal-Omega the build passed. [17:17:19] [02ImportDump] 07Universal-Omega pushed 031 commit to 03noduplicate [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/140908e48456...a238350abfc2 [17:17:21] [02ImportDump] 07Universal-Omega 03a238350 - Add check [17:17:24] [02ImportDump] 07Universal-Omega synchronize pull request 03#101: Prevent duplicate comments - 13https://github.com/miraheze/ImportDump/pull/101 [17:20:38] miraheze/ImportDump - Universal-Omega the build has errored. [17:22:15] [02ImportDump] 07Universal-Omega synchronize pull request 03#101: Prevent duplicate comments - 13https://github.com/miraheze/ImportDump/pull/101 [17:22:17] [02ImportDump] 07Universal-Omega pushed 031 commit to 03noduplicate [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/a238350abfc2...e8dd6abe8596 [17:22:19] [02ImportDump] 07Universal-Omega 03e8dd6ab - Use Status [17:25:00] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 166.56 ms [17:30:59] miraheze/ImportDump - Universal-Omega the build passed. [17:33:22] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 187.93 ms [17:35:21] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 162.37 ms [17:36:26] PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.yahyabd.xyz All nameservers failed to answer the query. [17:39:36] [02ImportDump] 07Universal-Omega edited pull request 03#101: Prevent duplicate comments - 13https://github.com/miraheze/ImportDump/pull/101 [17:41:26] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 141.65 ms [17:41:59] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/df43d50e3640...9ccd00f89c9a [17:42:00] [02ssl] 07WikiTideSSLBot 039ccd00f - Bot: Add SSL cert for urbanshade.org [17:45:24] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 183.93 ms [17:51:34] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 168.23 ms [17:53:32] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 131.57 ms [17:56:09] [02ImportDump] 07Universal-Omega edited pull request 03#101: Prevent duplicate comments - 13https://github.com/miraheze/ImportDump/pull/101 [17:57:42] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 157.40 ms [17:59:41] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 129.86 ms [18:07:32] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:08:51] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 156.80 ms [18:10:50] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 153.93 ms [18:53:52] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 141.20 ms [19:08:03] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 141.32 ms [19:22:26] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 141.21 ms [19:24:26] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 141.45 ms [19:28:38] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 141.18 ms [19:30:38] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 141.58 ms [19:46:58] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 141.41 ms [19:55:04] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 125.86 ms [20:01:15] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 141.14 ms [20:03:15] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 141.05 ms [20:09:49] PROBLEM - cloud18 APT on cloud18 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [20:10:51] PROBLEM - mw181 APT on mw181 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (4 critical updates). [20:11:32] PROBLEM - mw161 APT on mw161 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (4 critical updates). [20:11:52] PROBLEM - cloud18 APT on cloud18 is CRITICAL: APT CRITICAL: 103 packages available for upgrade (27 critical updates). [20:13:12] PROBLEM - mw182 APT on mw182 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (4 critical updates). [20:15:52] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (4 critical updates). [20:16:26] PROBLEM - mw162 APT on mw162 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (4 critical updates). [20:16:29] PROBLEM - mw171 APT on mw171 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (4 critical updates). [20:16:32] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 148.18 ms [20:17:27] PROBLEM - mw151 APT on mw151 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (4 critical updates). [20:18:33] PROBLEM - jobchron171 APT on jobchron171 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [20:18:34] PROBLEM - mw172 APT on mw172 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (4 critical updates). [20:19:09] PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (4 critical updates). [20:20:34] PROBLEM - jobchron171 APT on jobchron171 is CRITICAL: APT CRITICAL: 72 packages available for upgrade (45 critical updates). [20:23:40] PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (4 critical updates). [20:23:58] PROBLEM - mw152 APT on mw152 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (4 critical updates). [20:24:36] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 141.24 ms [20:28:47] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 141.92 ms [20:32:49] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 142.24 ms [20:43:01] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 141.69 ms [20:45:01] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 140.55 ms [20:52:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [20:54:14] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 140.68 ms [20:58:19] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 145.05 ms [21:17:45] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 141.28 ms [21:19:45] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 140.93 ms [21:22:23] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:49:00] [02ImportDump] 07Universal-Omega synchronize pull request 03#101: Prevent duplicate comments - 13https://github.com/miraheze/ImportDump/pull/101 [21:49:02] [02ImportDump] 07Universal-Omega pushed 031 commit to 03noduplicate [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/e8dd6abe8596...e245cb865019 [21:49:04] [02ImportDump] 07Universal-Omega 03e245cb8 - Try session [21:55:51] [02ImportDump] 07Universal-Omega synchronize pull request 03#101: Prevent duplicate comments - 13https://github.com/miraheze/ImportDump/pull/101 [21:55:52] [02ImportDump] 07Universal-Omega pushed 031 commit to 03noduplicate [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/e245cb865019...cb6ab13f71cc [21:55:54] [02ImportDump] 07Universal-Omega 03cb6ab13 - Check token [21:58:05] [02ImportDump] 07Universal-Omega synchronize pull request 03#101: Prevent duplicate comments - 13https://github.com/miraheze/ImportDump/pull/101 [21:58:08] [02ImportDump] 07Universal-Omega pushed 031 commit to 03noduplicate [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/cb6ab13f71cc...0a8608414966 [21:58:11] [02ImportDump] 07Universal-Omega 030a86084 - Try something else [22:00:41] [02ImportDump] 07Universal-Omega synchronize pull request 03#101: Prevent duplicate comments - 13https://github.com/miraheze/ImportDump/pull/101 [22:00:43] [02ImportDump] 07Universal-Omega pushed 031 commit to 03noduplicate [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/0a8608414966...5b3a2a7c17bb [22:00:44] [02ImportDump] 07Universal-Omega 035b3a2a7 - Move [22:09:07] miraheze/ImportDump - Universal-Omega the build passed. [22:11:38] [02ImportDump] 07Universal-Omega synchronize pull request 03#101: Prevent duplicate comments - 13https://github.com/miraheze/ImportDump/pull/101 [22:11:39] [02ImportDump] 07Universal-Omega pushed 031 commit to 03noduplicate [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/5b3a2a7c17bb...53e9b675a6d0 [22:11:41] [02ImportDump] 07Universal-Omega 0353e9b67 - Try something else [22:18:48] [02ImportDump] 07Universal-Omega synchronize pull request 03#101: Prevent duplicate comments - 13https://github.com/miraheze/ImportDump/pull/101 [22:18:51] [02ImportDump] 07Universal-Omega pushed 031 commit to 03noduplicate [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/53e9b675a6d0...ad34351c8d28 [22:18:53] [02ImportDump] 07Universal-Omega 03ad34351 - Move [22:20:07] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [22:20:34] [02ImportDump] 07Universal-Omega synchronize pull request 03#101: Prevent duplicate comments - 13https://github.com/miraheze/ImportDump/pull/101 [22:20:37] [02ImportDump] 07Universal-Omega pushed 031 commit to 03noduplicate [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/ad34351c8d28...edb856030ec9 [22:20:39] [02ImportDump] 07Universal-Omega 03edb8560 - Move [22:23:29] [02ImportDump] 07Universal-Omega synchronize pull request 03#101: Prevent duplicate comments - 13https://github.com/miraheze/ImportDump/pull/101 [22:23:31] [02ImportDump] 07Universal-Omega pushed 031 commit to 03noduplicate [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/edb856030ec9...2efa20cfd4d4 [22:23:34] [02ImportDump] 07Universal-Omega 032efa20c - Try something else [22:26:27] miraheze/ImportDump - Universal-Omega the build has errored. [22:46:17] [02ImportDump] 07Universal-Omega synchronize pull request 03#101: Prevent duplicate comments - 13https://github.com/miraheze/ImportDump/pull/101 [22:46:58] [02ImportDump] 07Universal-Omega pushed 031 commit to 03noduplicate [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/2efa20cfd4d4...6456039885af [22:47:01] [02ImportDump] 07Universal-Omega 036456039 - Try session again [22:51:09] miraheze/ImportDump - Universal-Omega the build has errored. [22:51:10] [02ImportDump] 07Universal-Omega synchronize pull request 03#101: Prevent duplicate comments - 13https://github.com/miraheze/ImportDump/pull/101 [22:51:11] [02ImportDump] 07Universal-Omega pushed 031 commit to 03noduplicate [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/6456039885af...2b4ae4da7ff9 [22:51:13] [02ImportDump] 07Universal-Omega 032b4ae4d - Try something else [22:52:36] [02ImportDump] 07Universal-Omega synchronize pull request 03#101: Prevent duplicate comments - 13https://github.com/miraheze/ImportDump/pull/101 [22:52:43] [02ImportDump] 07Universal-Omega pushed 031 commit to 03noduplicate [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/2b4ae4da7ff9...a1ec517f52ca [22:52:46] [02ImportDump] 07Universal-Omega 03a1ec517 - - [22:54:19] miraheze/ImportDump - Universal-Omega the build has errored. [22:56:10] [02ImportDump] 07Universal-Omega synchronize pull request 03#101: Prevent duplicate comments - 13https://github.com/miraheze/ImportDump/pull/101 [22:56:12] [02ImportDump] 07Universal-Omega pushed 031 commit to 03noduplicate [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/a1ec517f52ca...266c53deedab [22:56:14] [02ImportDump] 07Universal-Omega 03266c53d - Add message [22:57:33] [02ImportDump] 07Universal-Omega synchronize pull request 03#101: Prevent duplicate comments - 13https://github.com/miraheze/ImportDump/pull/101 [22:57:35] [02ImportDump] 07Universal-Omega pushed 031 commit to 03noduplicate [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/266c53deedab...153a4e7ca162 [22:57:36] [02ImportDump] 07Universal-Omega 03153a4e7 - Remove [22:59:27] [02ImportDump] 07Universal-Omega pushed 031 commit to 03noduplicate [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/153a4e7ca162...5add80080b69 [22:59:28] [02ImportDump] 07Universal-Omega 035add800 - Add message [22:59:31] [02ImportDump] 07Universal-Omega synchronize pull request 03#101: Prevent duplicate comments - 13https://github.com/miraheze/ImportDump/pull/101 [23:00:49] [02ImportDump] 07Universal-Omega pushed 031 commit to 03noduplicate [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/5add80080b69...c784cde60e33 [23:00:50] [02ImportDump] 07Universal-Omega 03c784cde - Add qqq [23:00:53] [02ImportDump] 07Universal-Omega synchronize pull request 03#101: Prevent duplicate comments - 13https://github.com/miraheze/ImportDump/pull/101 [23:01:08] [02ImportDump] 07Universal-Omega pushed 031 commit to 03noduplicate [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/c784cde60e33...7e836f678592 [23:01:09] [02ImportDump] 07Universal-Omega 037e836f6 - Punctuation [23:01:11] [02ImportDump] 07Universal-Omega synchronize pull request 03#101: Prevent duplicate comments - 13https://github.com/miraheze/ImportDump/pull/101 [23:03:10] [02ImportDump] 07Universal-Omega edited pull request 03#101: Prevent duplicate comments - 13https://github.com/miraheze/ImportDump/pull/101 [23:04:36] [02ImportDump] 07Universal-Omega synchronize pull request 03#101: Prevent duplicate comments - 13https://github.com/miraheze/ImportDump/pull/101 [23:04:39] [02ImportDump] 07Universal-Omega pushed 031 commit to 03noduplicate [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/7e836f678592...67f882dc02f9 [23:04:41] [02ImportDump] 07Universal-Omega 0367f882d - Remove if another part of form is submitted [23:12:48] miraheze/ImportDump - Universal-Omega the build passed.