[00:05:11] [02CreateWiki] 07amire80 opened pull request 03#536: Remove an unnecessary space before a line break - 13https://github.com/miraheze/CreateWiki/pull/536 [00:05:26] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.04, 21.26, 17.89 [00:05:43] [02CreateWiki] 07coderabbitai[bot] commented on pull request 03#536: Remove an unnecessary space before a line break - 13https://github.com/miraheze/CreateWiki/pull/536#issuecomment-2156233556 [00:07:24] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.27, 19.92, 17.83 [00:14:50] miraheze/CreateWiki - amire80 the build passed. [00:27:20] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:12:20] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:47:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:07:44] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:06:52] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [05:36:26] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:02:51] [02mw-config] 07songnguxyz opened pull request 03#5584: Update footer icon settings for lhmnwiki - 13https://github.com/miraheze/mw-config/pull/5584 [06:03:48] miraheze/mw-config - songnguxyz the build passed. [06:06:00] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [06:35:34] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:07:23] [02puppet] 07redbluegreenhat closed pull request 03#3866: exempt miraheze.org from RDNS checks; update domains_as_tlds - 13https://github.com/miraheze/puppet/pull/3866 [07:07:26] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/1d346d527cee...4a8c9573b8f1 [07:07:27] [02puppet] 07MacFan4000 034a8c957 - exempt miraheze.org from RDNS checks; update domains_as_tlds (#3866) [07:08:41] miraheze/puppet - redbluegreenhat the build passed. [07:18:06] RECOVERY - fanpedia.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:18:06] RECOVERY - commonswiki.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:18:38] RECOVERY - phabricator.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:19:05] RECOVERY - polcompball.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:19:59] RECOVERY - ajhalili2006.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:20:16] RECOVERY - pandorastale.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:20:33] RECOVERY - recaptime.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:21:17] RECOVERY - wiki.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:21:19] RECOVERY - japanesefeaturephones.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:21:29] RECOVERY - pokemon.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:21:44] RECOVERY - template.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:21:45] RECOVERY - cnt.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:22:14] RECOVERY - yokaiwatch.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:22:48] RECOVERY - status.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:23:41] RECOVERY - schizoidnightmares.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:23:51] RECOVERY - sega.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:24:20] RECOVERY - nintendo.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:24:38] RECOVERY - rare.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:24:42] RECOVERY - croc.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:26:10] RECOVERY - inext.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:27:29] RECOVERY - smashbros.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:27:30] RECOVERY - squareenix.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:27:38] RECOVERY - piwik.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:28:39] RECOVERY - donkeykong.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:28:45] RECOVERY - mario.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:29:12] RECOVERY - kaylaandrachel.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:29:13] RECOVERY - crashspyro.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:30:48] RECOVERY - taswin.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:31:05] RECOVERY - thelangyalist.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:31:41] RECOVERY - dragonquest2.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:31:51] RECOVERY - dcmultiverse.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:31:57] !log [alex@eventgate181] Upgraded packages bsdutils, libc-devtools, libc6-dev, libc-dev-bin, linux-libc-dev, libc6, libsmartcols1, util-linux-extra, util-linux, libc-bin, libruby3.1, ruby3.1, mount, libuuid1, uuid-runtime, libblkid1, libmount1, libfdisk1, fdisk, less, libc-l10n, locales, bsdextrautils, eject, libarchive13, libdav1d6, libglib2.0-0, [07:31:58] libglib2.0-data, pdns-recursor, and util-linux-locales [07:32:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:32:13] RECOVERY - dc.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:32:32] RECOVERY - projectsekai.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:32:51] RECOVERY - eventgate181 APT on eventgate181 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [07:32:51] !log [alex@eventgate181] Upgraded packages linux-image-6.1.0-21-amd64, and linux-image-amd64 [07:32:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:33:10] !log rebooted eventgate181 [07:33:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:33:35] RECOVERY - www.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:33:46] MacFan4000: ^ [07:33:53] your changes just got pulled by puppet [07:33:54] RECOVERY - tintinpedia.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:34:10] RECOVERY - metroid.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:34:44] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [07:36:44] RECOVERY - yoshi.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:39:03] RECOVERY - luxuryelevator.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:39:03] RhinosF1: wiki.themanaworld.org is redirecting to www.polandballwiki.com [07:40:29] well, more specifically https://wiki.themanaworld.org/wiki/ is the one redirecting [07:40:54] RECOVERY - allthetropes.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:40:59] RECOVERY - removededmsongs.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:41:25] RECOVERY - phorge.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:42:02] RECOVERY - uncyclopediakr.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:42:12] RECOVERY - 321nailswiki.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:42:20] RECOVERY - closinglogosgroup.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:42:53] RECOVERY - wkmr.liao.media - reverse DNS on sslhost is OK: SSL OK - wkmr.liao.media reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [07:43:07] Orange_Star: not for me [07:43:24] and whenever we've checked, these have come from the backend [07:43:39] yeah, x-served-by: mw172.wikitide.net [07:44:19] RECOVERY - sims.miraheze.org - reverse DNS on sslhost is OK: SSL OK - domain exempt from check [07:45:04] Orange_Star: go find out why the backend is doing it [07:47:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:50:50] RhinosF1: Can you purge wiki.themanaworld.org/wiki/? [07:53:22] Orange_Star: done [08:08:31] Orange_Star: do you recongise gamebaikeshuwiki.miraheze.org ? [08:09:55] nope [08:10:32] Orange_Star: it's traffic pattern makes no sense [08:10:37] it's super spiky [08:10:45] it's clearly chinese [08:11:02] but it's traffic from China doesn't look legitimate [08:11:23] GFW probes? [08:11:28] good old bots? [08:16:04] Orange_Star: i mean it's not good bots [08:16:09] it's browser-like UAs [08:16:17] but it's not human traffic [08:16:25] wiki seems sparsely edited [08:33:45] huh, now it's working correctly [08:34:09] redirects to /wiki/ which redirects correctly to that wiki's main page [08:34:30] \/wiki redirects to polandballwiki still [08:34:37] (ignore the \) [08:36:33] Orange_Star: purged without the trailing / too [08:37:20] now that works too [08:38:11] for the record, I've been checking the nginx config on mwtask181 and played around with shell.php but there was nothing obviously wrong [08:38:37] quite the mystery [08:38:38] ye i've no idea [08:38:47] but i blame the mediawiki servers [08:52:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [09:17:21] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:18:14] PROBLEM - db151 Backups SQL on db151 is CRITICAL: FILE_AGE CRITICAL: /var/log/sql-backup.log is 1209606 seconds old and 106503 bytes [09:45:10] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:10:09] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o