[00:00:08] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:02:08] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:04:33] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 21564MiB (24% inode=98%); [00:05:31] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 21571MiB (24% inode=98%); [00:13:31] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 175.46 ms [00:15:34] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.03 ms [02:09:41] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 169.45 ms [02:11:44] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.05 ms [02:36:35] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 176.82 ms [02:38:38] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.73 ms [02:59:48] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03log-message [+0/-0/±1] 13https://github.com/miraheze/ImportDump/commit/920842f7ba9e [02:59:49] [02miraheze/ImportDump] 07Universal-Omega 03920842f - ImportDumpJob: use simplified log message [02:59:51] [02ImportDump] 07Universal-Omega created branch 03log-message - 13https://github.com/miraheze/ImportDump [02:59:54] [02ImportDump] 07Universal-Omega opened pull request 03#87: ImportDumpJob: use simplified log message - 13https://github.com/miraheze/ImportDump/pull/87 [03:03:19] [02ImportDump] 07Universal-Omega synchronize pull request 03#87: ImportDumpJob: use simplified log message - 13https://github.com/miraheze/ImportDump/pull/87 [03:03:21] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03log-message [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/920842f7ba9e...45419ba7e9b2 [03:03:24] [02miraheze/ImportDump] 07Universal-Omega 0345419ba - Use rollbackPrimaryChangesAndLog [03:03:53] miraheze/ImportDump - Universal-Omega the build has errored. [03:06:26] miraheze/ImportDump - Universal-Omega the build has errored. [03:06:30] [02ImportDump] 07Universal-Omega synchronize pull request 03#87: ImportDumpJob: use simplified log message - 13https://github.com/miraheze/ImportDump/pull/87 [03:06:31] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03log-message [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/45419ba7e9b2...dd2cf571637b [03:06:33] [02miraheze/ImportDump] 07Universal-Omega 03dd2cf57 - Use Telemetry [03:08:16] [02ImportDump] 07Universal-Omega synchronize pull request 03#87: ImportDumpJob: use simplified log message - 13https://github.com/miraheze/ImportDump/pull/87 [03:08:17] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03log-message [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/dd2cf571637b...83d64ff48af3 [03:08:20] [02miraheze/ImportDump] 07Universal-Omega 0383d64ff - Simplify [03:15:58] miraheze/ImportDump - Universal-Omega the build passed. [03:36:16] [02ImportDump] 07Universal-Omega closed pull request 03#87: ImportDumpJob: use simplified log message - 13https://github.com/miraheze/ImportDump/pull/87 [03:36:19] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/398ab2dd107d...fc8cb6e0e70e [03:36:20] [02miraheze/ImportDump] 07Universal-Omega 03fc8cb6e - ImportDumpJob: use simplified log message (#87) [03:36:22] [02miraheze/ImportDump] 07Universal-Omega deleted branch 03log-message [03:36:25] [02ImportDump] 07Universal-Omega deleted branch 03log-message - 13https://github.com/miraheze/ImportDump [03:44:32] miraheze/ImportDump - Universal-Omega the build passed. [03:46:54] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.89 ms [03:51:01] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.11 ms [03:54:44] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=asutorufuwiki --no-updates --username-prefix=wikia:asutorufu /home/universalomega/asutorufuwiki-20240307043422.xml (START) [03:54:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:55:56] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=asutorufuwiki --no-updates --username-prefix=wikia:asutorufu /home/universalomega/asutorufuwiki-20240307043422.xml (END - exit=0) [03:55:57] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=asutorufuwiki (START) [03:56:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:56:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:56:10] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=asutorufuwiki (END - exit=0) [03:56:11] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=asutorufuwiki --active --update (END - exit=0) [03:56:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:56:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:06:16] PROBLEM - wiki.rsf.world - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.rsf.world could not be found [04:33:52] PROBLEM - wiki.rsf.world - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [04:36:58] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [05:00:17] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 177.71 ms [05:04:24] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.86 ms [05:05:46] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [05:08:38] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 172.68 ms [05:10:42] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.08 ms [05:16:58] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.91 ms [05:19:01] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.50 ms [05:46:08] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 178.59 ms [05:50:15] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.07 ms [06:02:40] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.18 ms [06:04:43] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.84 ms [06:06:55] [02MirahezeMagic] 07Universal-Omega closed pull request 03#481: Remove 'forcesafemode' option - 13https://github.com/miraheze/MirahezeMagic/pull/481 [06:06:56] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/MirahezeMagic/compare/e55417181135...6ac5d48defb8 [06:06:57] [02miraheze/MirahezeMagic] 07Dringsim 036ac5d48 - Remove 'forcesafemode' option (#481) [06:10:03] [02MirahezeMagic] 07Universal-Omega commented on pull request 03#481: Remove 'forcesafemode' option - 13https://github.com/miraheze/MirahezeMagic/pull/481#issuecomment-1993639570 [06:10:23] [02MirahezeMagic] 07Universal-Omega edited a comment on pull request 03#481: Remove 'forcesafemode' option - 13https://github.com/miraheze/MirahezeMagic/pull/481#issuecomment-1993639570 [06:16:57] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [06:21:30] miraheze/MirahezeMagic - Universal-Omega the build passed. [06:21:47] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/6ac5d48defb8...3491ada8c440 [06:21:50] [02miraheze/MirahezeMagic] 07Universal-Omega 033491ada - phan: remove PhanParamNameIndicatingUnused from suppress_issue_types [06:23:40] !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 [06:23:41] !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [06:23:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:23:50] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [06:23:51] !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [06:23:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:24:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:24:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:32:54] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [06:32:55] !log [@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw151.wikitide.net [06:33:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:33:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:34:34] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1035584509 secs [06:35:32] miraheze/MirahezeMagic - Universal-Omega the build passed. [06:44:05] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.19 ms [06:46:08] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.54 ms [06:55:33] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.54 ms [06:57:36] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.57 ms [07:06:01] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=woodwikiwiki (END - exit=0) [07:06:08] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=woodwikiwiki (END - exit=0) [07:06:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:06:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:06:48] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=eanawiki (END - exit=0) [07:06:54] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=eanawiki (END - exit=0) [07:06:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:07:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:07:19] !log [reception@mwtask181] starting deploy of {'files': '../mediawiki/1.41/extensions/SemanticMediaWiki/.smw.json'} to all [07:07:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:07:28] !log [reception@mwtask181] finished deploy of {'files': '../mediawiki/1.41/extensions/SemanticMediaWiki/.smw.json'} to all - SUCCESS in 9s [07:07:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:32:49] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [07:53:33] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 171.02 ms [07:55:36] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.53 ms [08:01:13] RECOVERY - wiki.rsf.world - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rsf.world' will expire on Sat 06 Apr 2024 12:31:23 PM GMT +0000. [08:01:17] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:01:52] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.65 ms [08:06:00] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.42 ms [08:12:34] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.0972482264 secs [08:21:38] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 171.75 ms [08:23:41] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.31 ms [08:30:00] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.07 ms [08:32:03] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 276.77 ms [08:32:42] PROBLEM - wiki.rsf.world - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [08:42:27] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 172.21 ms [08:48:34] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1020630002 secs [08:48:38] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.62 ms [09:03:31] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 207.03 ms [09:05:31] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 176.23 ms [09:21:47] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.93 ms [09:25:01] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 183.32 ms [09:26:15] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libboost-regex1.74.0-icu72] [09:26:35] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09961435199 secs [09:27:01] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 176.90 ms [09:27:59] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 166.62 ms [09:38:23] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 167.71 ms [09:44:34] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.33 ms [09:51:34] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 176.38 ms [09:53:35] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 177.28 ms [09:53:59] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 169.07 ms [09:54:15] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [09:56:02] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.03 ms [10:04:27] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 168.93 ms [10:05:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [10:06:30] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.48 ms [10:13:23] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1001306474 secs [10:17:20] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09963914752 secs [10:22:10] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.10 ms [10:24:13] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.85 ms [10:32:38] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.43 ms [10:36:45] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.93 ms [10:55:34] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.09 ms [10:59:40] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1012051404 secs [10:59:42] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.17 ms [11:00:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [11:20:31] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.51 ms [11:24:40] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.30 ms [11:33:06] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 173.19 ms [11:35:09] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.21 ms [11:39:04] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09984961152 secs [11:39:24] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.48 ms [11:44:09] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:45:35] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.16 ms [11:45:55] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1007398069 secs [11:51:50] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09955656528 secs [11:54:05] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 175.00 ms [12:03:37] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1014631093 secs [12:06:28] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.37 ms [12:12:46] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 173.40 ms [12:16:53] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 177.68 ms [12:28:21] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 169.56 ms [12:43:40] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [12:47:02] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 166.04 ms [13:29:36] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.61 ms [13:31:40] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.12 ms [13:50:22] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.22 ms [13:52:25] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.60 ms [13:56:37] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.49 ms [13:58:04] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=ngnlzhwiki (END - exit=0) [13:58:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:58:24] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=ngnlzhwiki --update (END - exit=0) [13:58:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:00:16] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php rebuildall --wiki=ngnlzhwiki (START) [14:00:19] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php rebuildall --wiki=ngnlzhwiki (END - exit=256) [14:00:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:00:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:00:44] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.19 ms [14:13:14] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.79 ms [14:14:40] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildtextindex.php --wiki=ngnlzhwiki (END - exit=256) [14:14:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:15:17] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.52 ms [14:36:50] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [14:42:38] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [14:46:34] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09972158074 secs [14:47:31] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 170.03 ms [14:51:38] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 205.38 ms [14:54:34] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.102719903 secs [15:04:34] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09995347261 secs [15:05:34] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [15:08:34] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1013537049 secs [15:14:26] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.18 ms [15:18:33] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.19 ms [15:22:46] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 168.18 ms [15:24:49] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.65 ms [15:40:40] !log sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=ngnlzhwiki [15:40:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:41:06] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=ngnlzhwiki (START) [15:41:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:41:34] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=ngnlzhwiki (END - exit=0) [15:41:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:43:16] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=ngnlzhwiki (END - exit=0) [15:43:21] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=ngnlzhwiki --update (END - exit=0) [15:43:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:43:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:44:35] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.89 ms [15:50:46] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.45 ms [15:59:15] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 169.12 ms [16:01:18] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.49 ms [16:03:31] PROBLEM - wiki.theluxuryelevator.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.theluxuryelevator.com' expires in 15 day(s) (Fri 29 Mar 2024 03:35:40 PM GMT +0000). [16:03:43] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e82d1c017bba...e7a29bbb8aa6 [16:03:45] [02miraheze/ssl] 07MirahezeSSLBot 03e7a29bb - Bot: Update SSL cert for wiki.theluxuryelevator.com [16:06:58] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:24:35] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09185642004 secs [16:28:34] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1069835722 secs [16:32:01] RECOVERY - wiki.theluxuryelevator.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.theluxuryelevator.com' will expire on Tue 11 Jun 2024 03:03:37 PM GMT +0000. [16:32:02] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [16:32:37] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.92 ms [16:34:40] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 174.37 ms [16:37:26] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [16:38:54] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 170.17 ms [16:40:34] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09964555502 secs [16:40:57] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.20 ms [17:06:57] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [17:19:22] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.15 ms [17:25:36] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.18 ms [17:46:31] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 186.11 ms [17:54:50] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.55 ms [17:59:04] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 165.35 ms [18:01:07] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.03 ms [18:04:46] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [18:11:07] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/478eed30cbad...7da309f0013a [18:11:10] [02miraheze/mw-config] 07Universal-Omega 037da309f - Use namespaced Title class [18:12:01] miraheze/mw-config - Universal-Omega the build passed. [18:14:31] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/7da309f0013a...26da1bf61bb9 [18:14:33] [02miraheze/mw-config] 07Universal-Omega 0326da1bf - Use namespaced SiteConfiguration class [18:15:28] miraheze/mw-config - Universal-Omega the build passed. [18:23:41] !log [@test151] starting deploy of {'config': True} to test151 [18:23:42] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [18:23:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:29:23] [02mw-config] 07Universal-Omega created branch 03localwiki-namespaced - 13https://github.com/miraheze/mw-config [18:29:24] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03localwiki-namespaced [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/50537a87dcd6 [18:29:26] [02miraheze/mw-config] 07Universal-Omega 0350537a8 - LocalWiki: use namespaced classes [18:29:32] [02mw-config] 07Universal-Omega opened pull request 03#5499: LocalWiki: use namespaced classes - 13https://github.com/miraheze/mw-config/pull/5499 [18:30:34] miraheze/mw-config - Universal-Omega the build passed. [18:32:32] !log [@mwtask181] starting deploy of {'config': True} to all [18:32:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:32:42] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 9s [18:32:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:34:47] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [18:37:04] !log [@mwtask171] starting deploy of {'config': True} to all [18:37:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:37:12] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [18:37:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:38:41] [02mw-config] 07Universal-Omega closed pull request 03#5499: LocalWiki: use namespaced classes - 13https://github.com/miraheze/mw-config/pull/5499 [18:38:43] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/26da1bf61bb9...ca51fb674741 [18:38:44] [02miraheze/mw-config] 07Universal-Omega 03ca51fb6 - LocalWiki: use namespaced classes (#5499) [18:38:45] [02miraheze/mw-config] 07Universal-Omega deleted branch 03localwiki-namespaced [18:38:46] [02mw-config] 07Universal-Omega deleted branch 03localwiki-namespaced - 13https://github.com/miraheze/mw-config [18:39:32] miraheze/mw-config - Universal-Omega the build passed. [18:52:14] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03wgMirahezeMagicAccessIdsMap [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/aa20ccea0970 [18:52:15] [02miraheze/mw-config] 07Universal-Omega 03aa20cce - Configure wgMirahezeMagicAccessIdsMap [18:52:17] [02mw-config] 07Universal-Omega created branch 03wgMirahezeMagicAccessIdsMap - 13https://github.com/miraheze/mw-config [18:52:25] [02mw-config] 07Universal-Omega opened pull request 03#5500: Configure wgMirahezeMagicAccessIdsMap - 13https://github.com/miraheze/mw-config/pull/5500 [18:53:15] miraheze/mw-config - Universal-Omega the build passed. [18:53:45] !log [@test151] starting deploy of {'config': True} to test151 [18:53:46] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [18:53:47] [02mw-config] 07Universal-Omega closed pull request 03#5500: Configure wgMirahezeMagicAccessIdsMap - 13https://github.com/miraheze/mw-config/pull/5500 [18:53:49] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ca51fb674741...6a4c1d42d5c4 [18:53:52] [02miraheze/mw-config] 07Universal-Omega 036a4c1d4 - Configure wgMirahezeMagicAccessIdsMap (#5500) [18:53:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:53:55] [02miraheze/mw-config] 07Universal-Omega deleted branch 03wgMirahezeMagicAccessIdsMap [18:53:58] [02mw-config] 07Universal-Omega deleted branch 03wgMirahezeMagicAccessIdsMap - 13https://github.com/miraheze/mw-config [18:54:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:54:44] miraheze/mw-config - Universal-Omega the build passed. [18:56:14] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1044446826 secs [19:02:52] !log [@mwtask181] starting deploy of {'config': True} to all [19:02:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:03:01] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 9s [19:03:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:07:40] !log [@mwtask171] starting deploy of {'config': True} to all [19:07:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:07:48] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [19:07:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:09:05] [02MirahezeMagic] 07Universal-Omega closed pull request 03#478: Support different IDs for different wikis in UserGetRightsRemove - 13https://github.com/miraheze/MirahezeMagic/pull/478 [19:09:06] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MirahezeMagic/compare/3491ada8c440...4f06569463d5 [19:09:08] [02miraheze/MirahezeMagic] 07Universal-Omega 034f06569 - Support different IDs for different wikis in UserGetRightsRemove (#478) [19:09:10] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03rights-access-ids [19:09:12] [02MirahezeMagic] 07Universal-Omega deleted branch 03rights-access-ids - 13https://github.com/miraheze/MirahezeMagic [19:15:47] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.92 ms [19:17:50] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.00 ms [19:22:27] miraheze/MirahezeMagic - Universal-Omega the build passed. [19:23:22] !log [@test151] starting deploy of {'config': True} to test151 [19:23:23] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [19:23:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:23:31] !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 [19:23:32] !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [19:23:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:23:40] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [19:23:41] !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [19:23:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:23:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:24:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:24:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:32:04] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [19:32:07] !log [@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw152.wikitide.net [19:32:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:32:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:10:17] [02CreateWiki] 07redbluegreenhat opened pull request 03#486: Make on hold the first option when handling wiki requests - 13https://github.com/miraheze/CreateWiki/pull/486 [20:10:45] [02CreateWiki] 07redbluegreenhat closed pull request 03#486: Make on hold the first option when handling wiki requests - 13https://github.com/miraheze/CreateWiki/pull/486 [20:10:46] [02miraheze/CreateWiki] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/31edd3cbbc2a...dc9286a2a930 [20:10:47] [02miraheze/CreateWiki] 07redbluegreenhat 03dc9286a - Make on hold the first option when handling wiki requests (#486) [20:12:35] !log [alex@test151] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 [20:12:36] !log [alex@test151] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 - SUCCESS in 1s [20:12:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:12:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:14:51] !log [reception@mwtask181] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to all [20:14:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:15:01] !log [reception@mwtask181] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to all - SUCCESS in 10s [20:15:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:18:24] miraheze/CreateWiki - redbluegreenhat the build passed. [20:18:26] miraheze/CreateWiki - redbluegreenhat the build passed. [20:49:50] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.25 ms [20:53:58] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.41 ms [20:56:53] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/7a006f80a8ab...e90ac9f7a236 [20:56:56] [02miraheze/dns] 07Universal-Omega 03e90ac9f - Add logging to wikitide.net [21:00:36] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [21:00:52] PROBLEM - wiki.minkyu.kim - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.minkyu.kim could not be found [21:36:52] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [21:41:16] [02puppet] 07Universal-Omega closed pull request 03#3763: graylog: use logging.wikitide.net - 13https://github.com/miraheze/puppet/pull/3763 [21:41:17] [02miraheze/puppet] 07Universal-Omega deleted branch 03logging [21:41:20] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+1/-1/±3] 13https://github.com/miraheze/puppet/compare/21ae2d830fb8...5b8d22634951 [21:41:23] [02miraheze/puppet] 07Universal-Omega 035b8d226 - graylog: use logging.wikitide.net (#3763) [21:41:26] [02puppet] 07Universal-Omega deleted branch 03logging - 13https://github.com/miraheze/puppet [21:45:36] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [22:05:36] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [22:06:25] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09974321723 secs [22:10:25] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.104480058 secs [22:14:18] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/e90ac9f7a236...d5273fed869d [22:14:19] [02miraheze/dns] 07Universal-Omega 03d5273fe - Add grafana and monitoring to wikitide.net [22:16:03] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 184.12 ms [22:18:04] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 189.32 ms [22:19:25] [02puppet] 07Universal-Omega closed pull request 03#3765: grafana: use grafana.wikitide.net - 13https://github.com/miraheze/puppet/pull/3765 [22:19:27] [02miraheze/puppet] 07Universal-Omega deleted branch 03grafana [22:19:29] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/puppet/compare/5b8d22634951...e81603e7e562 [22:19:30] [02miraheze/puppet] 07Universal-Omega 03e81603e - grafana: use grafana.wikitide.net (#3765) [22:19:31] [02puppet] 07Universal-Omega deleted branch 03grafana - 13https://github.com/miraheze/puppet [22:20:19] miraheze/puppet - Universal-Omega the build has errored. [22:28:37] [02miraheze/puppet] 07Universal-Omega pushed 0318 commits to 03monitoring [+10/-2/±37] 13https://github.com/miraheze/puppet/compare/2537f7607c7c...752bfa47459e [22:28:40] [02miraheze/puppet] 07Universal-Omega 03752bfa4 - Merge branch 'master' into monitoring [22:28:42] [02puppet] 07Universal-Omega synchronize pull request 03#3764: icinga: use monitoring.wikitide.net - 13https://github.com/miraheze/puppet/pull/3764 [22:29:36] miraheze/puppet - Universal-Omega the build has errored. [22:39:56] [02puppet] 07Universal-Omega closed pull request 03#3764: icinga: use monitoring.wikitide.net - 13https://github.com/miraheze/puppet/pull/3764 [22:39:59] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/e81603e7e562...3552f5bbafb7 [22:40:01] [02miraheze/puppet] 07Universal-Omega 033552f5b - icinga: use monitoring.wikitide.net (#3764) [22:40:02] [02miraheze/puppet] 07Universal-Omega deleted branch 03monitoring [22:40:04] [02puppet] 07Universal-Omega deleted branch 03monitoring - 13https://github.com/miraheze/puppet [22:40:47] miraheze/puppet - Universal-Omega the build has errored. [22:50:35] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9708MiB (10% inode=98%); [22:55:25] PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9698MiB (10% inode=98%); [23:07:11] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.55 ms [23:07:30] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/d5273fed869d...d8db0b4733c9 [23:07:31] [02miraheze/dns] 07Universal-Omega 03d8db0b4 - Remove from miraheze.org [23:09:11] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.15 ms [23:15:33] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [23:45:33] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1