[00:05:19] !log refreshLinks for 321nailswiki [00:05:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:54:33] PROBLEM - www.dovearchives.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.dovearchives.wiki' expires in 15 day(s) (Fri 22 Mar 2024 12:42:30 AM GMT +0000). [00:54:53] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2252f57ee3a8...53279423f2e9 [00:54:55] [02miraheze/ssl] 07MirahezeSSLBot 035327942 - Bot: Update SSL cert for www.dovearchives.wiki [01:16:12] PROBLEM - wiki.puucraft.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.puucraft.net' expires in 15 day(s) (Fri 22 Mar 2024 12:48:39 AM GMT +0000). [01:16:23] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/53279423f2e9...155412f2ab22 [01:16:25] [02miraheze/ssl] 07MirahezeSSLBot 03155412f - Bot: Update SSL cert for wiki.puucraft.net [01:23:06] RECOVERY - www.dovearchives.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.dovearchives.wiki' will expire on Mon 03 Jun 2024 11:54:48 PM GMT +0000. [01:45:23] RECOVERY - wiki.puucraft.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.puucraft.net' will expire on Tue 04 Jun 2024 12:16:17 AM GMT +0000. [03:03:14] PROBLEM - mon181 Backups Grafana on mon181 is WARNING: FILE_AGE WARNING: /var/log/grafana-backup.log is 864176 seconds old and 92 bytes [03:04:17] PROBLEM - cloud16 Puppet on cloud16 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [03:30:20] RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [06:21:20] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.21 ms [06:23:21] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 140.40 ms [07:54:35] PROBLEM - ping6 on ns2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 281.18 ms [07:58:40] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 172.34 ms [08:27:07] PROBLEM - db162 NTP time on db162 is WARNING: NTP WARNING: Offset 0.1044366956 secs [08:45:45] RECOVERY - db162 NTP time on db162 is OK: NTP OK: Offset 0.08818739653 secs [09:34:55] [02RequestSSL] 07redbluegreenhat synchronize pull request 03#40: T11699: Create a job to automate CNAME and Reverse DNS checks - 13https://github.com/miraheze/RequestSSL/pull/40 [09:37:44] miraheze/RequestSSL - redbluegreenhat the build has errored. [09:40:58] [02RequestSSL] 07redbluegreenhat synchronize pull request 03#40: T11699: Create a job to automate CNAME and Reverse DNS checks - 13https://github.com/miraheze/RequestSSL/pull/40 [09:44:01] miraheze/RequestSSL - redbluegreenhat the build has errored. [09:55:31] [02RequestSSL] 07redbluegreenhat commented on pull request 03#40: T11699: Create a job to automate CNAME and Reverse DNS checks - 13https://github.com/miraheze/RequestSSL/pull/40#issuecomment-1980489476 [09:58:57] [02RequestSSL] 07redbluegreenhat edited pull request 03#40: T11699: Create a job to automate CNAME and Reverse DNS checks - 13https://github.com/miraheze/RequestSSL/pull/40 [12:09:50] [02mw-config] 07colleirose opened pull request 03#5495: Fix multiple issues in robots.txt - 13https://github.com/miraheze/mw-config/pull/5495 [12:10:08] [02mw-config] 07colleirose edited pull request 03#5495: Fix multiple issues in robots.txt - 13https://github.com/miraheze/mw-config/pull/5495 [13:14:37] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:41:43] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.22 ms [13:43:43] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 175.41 ms [14:00:07] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 178.65 ms [14:02:07] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 180.30 ms [14:17:26] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 322.77 ms [14:17:31] PROBLEM - ping6 on cp41 is WARNING: PING WARNING - Packet loss = 0%, RTA = 345.18 ms [14:19:27] PROBLEM - ping6 on cp51 is WARNING: PING WARNING - Packet loss = 0%, RTA = 329.72 ms [14:19:32] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 350.88 ms [14:21:28] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 295.29 ms [14:21:33] PROBLEM - ping6 on cp41 is WARNING: PING WARNING - Packet loss = 0%, RTA = 344.90 ms [14:23:34] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 181.83 ms [14:25:35] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 178.31 ms [14:42:31] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [14:51:09] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 180.95 ms [14:59:15] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 179.94 ms [15:42:38] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [16:44:39] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [16:50:43] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [17:13:42] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [17:20:44] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [17:30:50] miraheze/mw-config - colleirose the build passed. [17:31:18] [02mw-config] 07Universal-Omega closed pull request 03#5495: Fix multiple issues in robots.txt - 13https://github.com/miraheze/mw-config/pull/5495 [17:31:21] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c0bdc9168a6d...f23ba171bd9c [17:31:23] [02miraheze/mw-config] 07colleirose 03f23ba17 - Fix multiple issues in robots.txt (#5495) [17:32:04] !log [@mwtask181] starting deploy of {'config': True} to all [17:32:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:32:14] miraheze/mw-config - Universal-Omega the build passed. [17:32:14] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 10s [17:32:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:37:31] !log [@mwtask171] starting deploy of {'config': True} to all [17:37:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:37:39] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [17:37:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:44:52] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:13:52] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [18:18:21] PROBLEM - cp41 APT on cp41 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [18:20:22] PROBLEM - cp41 APT on cp41 is CRITICAL: APT CRITICAL: 33 packages available for upgrade (5 critical updates). [18:50:43] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [19:20:44] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [20:17:46] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 14 minutes ago with 4 failures. Failed resources (up to 3 shown): Exec[git_checkout_MediaWiki-REL1_41 MessageCachePerformance],Exec[git_pull_MediaWiki-REL1_41 MirahezeMagic],Exec[git_checkout_MediaWiki-master MessageCachePerformance],Exec[git_pull_MediaWiki-master MirahezeMagic] [20:18:40] PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 43 packages available for upgrade (15 critical updates). [20:33:44] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03rename-tables [+1/-1/±0] 13https://github.com/miraheze/ImportDump/compare/df1f53cfcfb4...a400c1da6f66 [20:33:47] [02miraheze/ImportDump] 07Universal-Omega 03a400c1d - Fix [20:33:49] [02ImportDump] 07Universal-Omega synchronize pull request 03#77: Rename importdump_request* tables - 13https://github.com/miraheze/ImportDump/pull/77 [20:36:49] [02miraheze/ImportDump] 07Universal-Omega pushed 038 commits to 03rename-tables [+3/-3/±87] 13https://github.com/miraheze/ImportDump/compare/a400c1da6f66...0d5d40579c03 [20:36:50] [02miraheze/ImportDump] 07Universal-Omega 030d5d405 - Merge branch 'master' into rename-tables [20:36:53] [02ImportDump] 07Universal-Omega synchronize pull request 03#77: Rename importdump_request* tables - 13https://github.com/miraheze/ImportDump/pull/77 [20:44:42] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [20:44:51] miraheze/ImportDump - Universal-Omega the build passed. [20:50:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [21:14:22] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [21:15:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [22:17:10] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03remove-duplicates [+0/-0/±1] 13https://github.com/miraheze/ImportDump/commit/e05e03602c9d [22:17:12] [02miraheze/ImportDump] 07Universal-Omega 03e05e036 - Remove duplicates from ImportDumpJob [22:17:14] [02ImportDump] 07Universal-Omega created branch 03remove-duplicates - 13https://github.com/miraheze/ImportDump [22:17:16] [02ImportDump] 07Universal-Omega opened pull request 03#84: Remove duplicates from ImportDumpJob - 13https://github.com/miraheze/ImportDump/pull/84 [22:24:54] miraheze/ImportDump - Universal-Omega the build passed. [22:37:28] [02mw-config] 07Universal-Omega reviewed pull request 03#5410 commit - 13https://github.com/miraheze/mw-config/pull/5410#discussion_r1515244169 [22:40:55] [02mw-config] 07Tali64 synchronize pull request 03#5410: Improvements to CW canned messages - 13https://github.com/miraheze/mw-config/pull/5410 [22:41:02] [02mw-config] 07Tali64 reviewed pull request 03#5410 commit - 13https://github.com/miraheze/mw-config/pull/5410#discussion_r1515247514 [22:55:03] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03fix-hide-if [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/commit/f0ef993cc63d [22:55:05] [02miraheze/ManageWiki] 07Universal-Omega 03f0ef993 - Fix hide-if [22:55:08] [02ManageWiki] 07Universal-Omega created branch 03fix-hide-if - 13https://github.com/miraheze/ManageWiki [22:55:10] [02ManageWiki] 07Universal-Omega opened pull request 03#454: Fix hide-if - 13https://github.com/miraheze/ManageWiki/pull/454 [22:55:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [22:56:39] [02ManageWiki] 07Universal-Omega closed pull request 03#454: Fix hide-if - 13https://github.com/miraheze/ManageWiki/pull/454 [22:56:41] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/428be82c7ba7...cc77da0b3810 [22:56:44] [02miraheze/ManageWiki] 07Universal-Omega 03cc77da0 - Fix hide-if (#454) [22:56:45] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03fix-hide-if [22:56:46] [02ManageWiki] 07Universal-Omega deleted branch 03fix-hide-if - 13https://github.com/miraheze/ManageWiki [22:56:48] [02ManageWiki] 07Universal-Omega created branch 03fix-hide-if - 13https://github.com/miraheze/ManageWiki [22:56:51] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03fix-hide-if [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/commit/f0ef993cc63d [22:56:54] [02miraheze/ManageWiki] 07Universal-Omega 03f0ef993 - Fix hide-if [22:56:57] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03fix-hide-if [22:57:00] [02ManageWiki] 07Universal-Omega deleted branch 03fix-hide-if - 13https://github.com/miraheze/ManageWiki [22:59:58] miraheze/ManageWiki - Universal-Omega the build passed. [23:01:39] miraheze/ManageWiki - Universal-Omega the build passed. [23:04:17] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [23:40:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1