[00:01:24] PROBLEM - eventgate181 NTP time on eventgate181 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [00:03:23] PROBLEM - eventgate181 NTP time on eventgate181 is CRITICAL: NTP CRITICAL: Offset 278.663523 secs [00:45:15] RECOVERY - wiki.lewisgill.com - reverse DNS on sslhost is OK: SSL OK - wiki.lewisgill.com reverse DNS resolves to cp37.wikitide.net - CNAME OK [00:47:13] RECOVERY - wiki.lewisgill.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.lewisgill.com' will expire on Tue 23 Jul 2024 09:32:19 PM GMT +0000. [00:51:38] [02mw-config] 07pixDeVl opened pull request 03#5583: T12178: Added edittrusteduserprotected to protected grant - 13https://github.com/miraheze/mw-config/pull/5583 [00:52:36] miraheze/mw-config - pixDeVl the build passed. [01:12:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.34, 21.68, 18.49 [01:20:11] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.58, 20.11, 19.23 [01:24:38] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw161 [01:28:38] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [02:10:13] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:39:18] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:57:16] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.28, 20.95, 19.16 [03:10:45] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.48, 19.54, 20.03 [03:20:25] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.62, 19.26, 19.21 [03:22:21] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.20, 21.58, 20.05 [03:26:13] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.95, 21.22, 20.29 [03:28:08] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.43, 20.14, 20.01 [03:32:01] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.86, 22.69, 21.04 [03:33:57] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.92, 19.77, 20.15 [03:47:56] PROBLEM - db181 Backups SQL on db181 is CRITICAL: FILE_AGE CRITICAL: /var/log/sql-backup.log is 1209692 seconds old and 16001 bytes [04:47:44] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.32, 21.57, 19.42 [04:49:44] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.06, 20.28, 19.23 [05:12:10] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 2 backends are down. mw152 mw171 [05:14:07] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [05:40:21] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:50:30] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.03, 20.87, 19.12 [05:52:25] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 11.80, 17.35, 18.04 [06:07:53] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.31, 17.55, 16.03 [06:09:27] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [06:09:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.22, 20.78, 17.41 [06:11:44] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.00, 19.13, 17.21 [06:16:32] PROBLEM - swiftac171 APT on swiftac171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:21:26] RECOVERY - swiftac171 APT on swiftac171 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:22:33] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.81, 22.71, 19.76 [06:30:15] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.25, 21.69, 20.62 [06:34:07] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.36, 19.70, 20.02 [06:37:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:39:55] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.35, 22.08, 20.72 [06:45:44] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 15.33, 22.04, 21.49 [06:47:44] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 12.41, 18.53, 20.26 [07:03:47] PROBLEM - airlineinsider.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'airlineinsider.org' expires in 15 day(s) (Tue 18 Jun 2024 06:45:29 AM GMT +0000). [07:04:00] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6b01586c6ff7...790a7b8fadd0 [07:04:01] [02ssl] 07WikiTideSSLBot 03790a7b8 - Bot: Update SSL cert for airlineinsider.org [07:05:15] !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=sagan4alphawiki --procs=12 --memory-limit=max (START) [07:05:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:06:41] !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=sagan4alphawiki --procs=12 --memory-limit=max (END - exit=0) [07:06:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:08:25] !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=regenoswiki --procs=12 --memory-limit=max (START) [07:08:26] !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=regenoswiki --procs=12 --memory-limit=max (END - exit=0) [07:08:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:08:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:09:31] !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=idolish7wiki --procs=12 --memory-limit=max (START) [07:09:34] !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=idolish7wiki --procs=12 --memory-limit=max (END - exit=0) [07:09:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:09:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:12:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:32:34] RECOVERY - airlineinsider.org - LetsEncrypt on sslhost is OK: OK - Certificate 'airlineinsider.org' will expire on Sat 31 Aug 2024 06:03:53 AM GMT +0000. [07:32:56] [1/11] ``` [07:32:56] [2/11] > var_dump( MatomoAnalytics::getSiteID( $config->get( 'DBname' ) ) ); [07:32:57] [3/11] int(1) [07:32:57] [4/11] = null [07:32:57] [5/11] > MatomoAnalyticsHooks::wikiCreation( $config->get( 'DBname' ) ); [07:32:58] [6/11] = null [07:32:58] [7/11] > var_dump( MatomoAnalytics::getSiteID( $config->get( 'DBname' ) ) ); [07:32:58] [8/11] string(5) "12432" [07:32:59] [9/11] = null [07:32:59] [10/11] ``` [07:32:59] [11/11] https://issue-tracker.miraheze.org/T12160 is fixed. On another note, here's yet another situation where we get either an string or an int depending on circumstances, just like wiki request suppression not too long ago: https://github.com/miraheze/CreateWiki/commit/66b4874c4d7bf4fbb8c873083814ee472d045031 [07:33:22] Unlike there, I don't think this creates any bugs, but still, I should fix this so it is always an int [07:33:56] !log yohajiwiki shell.php: MatomoAnalyticsHooks::wikiCreation( $config->get( 'DBname' ) ); [07:34:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:07:52] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/790a7b8fadd0...9f13aca8a058 [08:07:53] [02ssl] 07WikiTideSSLBot 039f13aca - Bot: Add SSL cert for zh.tardis.wiki [08:41:38] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:07:38] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [10:38:23] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:01:08] !log disabled rocket loader on rnswiki to test stuff [11:01:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:03:45] PROBLEM - drachronicles.voxel.gay - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'drachronicles.voxel.gay' expires in 7 day(s) (Mon 10 Jun 2024 10:35:30 AM GMT +0000). [11:07:03] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [11:31:43] Isn't there a script [11:32:05] this is not really meant to be run manually, so no [11:32:23] that function is called via hooks on wiki creation by CreateWiki [11:32:40] https://github.com/miraheze/MatomoAnalytics/blob/master/maintenance/addMissingMatomos.php [11:32:45] @bluemoon0332 [11:33:09] welp [11:33:10] Or https://github.com/miraheze/MatomoAnalytics/blob/master/maintenance/modifyMatomo.php [11:33:14] way too late for that [11:33:40] @bluemoon0332 why [11:33:59] There's also a delete script too for wikis [11:34:09] it was already done, according to MA that is... [11:34:16] PROBLEM - db161 Backups SQL on db161 is CRITICAL: FILE_AGE CRITICAL: /var/log/sql-backup.log is 1209664 seconds old and 128765 bytes [11:34:18] Matomo itself doesn't agree tho [11:34:51] @bluemoon0332 remove the wiki and readd it [11:35:44] hmm, it is worth a shot at least [11:37:12] I'm not convinced returning the global site id is the best fallback if the wiki isn't there [11:37:35] If you ask for wiki stats and they don't exist, we should except because we need to fix that [11:37:38] I've been thinking we could return a random number there [11:37:42] for the lols [11:37:55] No [11:37:59] :/ [11:38:03] I'd say it should be an exception [11:39:07] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/MatomoAnalytics/maintenance/modifyMatomo.php --wiki=yohajiwiki --remove (END - exit=0) [11:39:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:39:15] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/MatomoAnalytics/maintenance/modifyMatomo.php --wiki=yohajiwiki (END - exit=0) [11:39:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:53:53] for the record, while we did get a new siteId from Matomo, it doesn't seem to have actually worked, I see the same results as before (Special:Analytics not working, site doesn't appear in Matomo's web interface) [12:07:17] PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 10.34, 9.95, 4.62 [12:07:40] PROBLEM - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 502 Bad Gateway [12:07:47] PROBLEM - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.015 second response time [15:18:42] [02landing] 07OAuthority opened pull request 03#149: Check if this is a right to left language - 13https://github.com/miraheze/landing/pull/149 [15:18:51] [02landing] 07coderabbitai[bot] commented on pull request 03#149: Check if this is a right to left language - 13https://github.com/miraheze/landing/pull/149#issuecomment-2143897302 [15:19:33] miraheze/landing - OAuthority the build passed. [16:37:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [16:42:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [19:35:48] !log disabled rocketloader across all wikis [19:35:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:06] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw162 [20:05:09] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw162 [20:07:09] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [20:09:04] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [20:43:43] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.19, 20.66, 16.24 [20:45:43] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 20.35, 20.27, 16.63 [20:49:43] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.37, 23.63, 18.77 [20:53:43] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.16, 23.93, 20.12 [20:57:43] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.58, 19.31, 19.13 [21:34:39] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw152 [21:34:53] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw152 [21:34:54] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw152 [21:35:09] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw152 [21:35:15] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [21:35:31] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5627 bytes in 0.013 second response time [21:35:32] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw152 [21:35:50] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw152 [21:39:13] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 286 bytes in 0.063 second response time [21:39:32] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [21:39:50] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [21:40:53] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [21:40:54] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [21:41:09] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [21:41:35] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.534 second response time [21:42:39] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [22:17:20] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [22:52:20] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [23:04:39] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw152 [23:05:32] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw151 [23:06:39] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [23:07:32] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [23:52:20] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [23:57:20] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1