[00:07:25] PROBLEM - dccomicswiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dccomicswiki.com could not be found [00:23:13] PROBLEM - dccomicswiki.com - LetsEncrypt on sslhost is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket [00:37:20] RECOVERY - dccomicswiki.com - reverse DNS on sslhost is OK: SSL OK - dccomicswiki.com reverse DNS resolves to cp37.wikitide.net - CNAME FLAT [00:55:26] PROBLEM - www.dccomicswiki.com - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [01:09:10] PROBLEM - dccomicswiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dccomicswiki.com could not be found [01:12:32] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.15 ms [01:16:33] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.28 ms [01:18:59] PROBLEM - www.dccomicswiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.dccomicswiki.com could not be found [01:51:21] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.98 ms [01:53:21] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.51 ms [02:26:47] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.55 ms [02:28:47] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.38 ms [02:38:01] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.31 ms [02:40:01] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.46 ms [02:54:14] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 171.19 ms [02:56:14] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.08 ms [04:10:55] [1/22] per https://discord.com/channels/407504499280707585/407537962553966603/1234344424524222494, this should assist with reducing overall work to automate wiki closures [04:10:55] [2/22] * there is a lack of activity with only log actions [04:10:55] [3/22] ```sql [04:10:55] [4/22] SELECT * FROM recentchanges [04:10:56] [5/22] LEFT JOIN logging ON rc_id = log_id [04:10:56] [6/22] WHERE rc_timestamp >= DATE_SUB(NOW(), INTERVAL 30 DAY) [04:10:56] [7/22] AND log_type NOT IN ("renameuser","newusers","massmessage","spamblacklist","abusefilter", "globalauth","usermerge") [04:10:57] [8/22] AND rc_actor != massmessagebot [04:10:57] [9/22] ORDER BY rc_timestamp DESC [04:10:57] [10/22] LIMIT 1000; [04:10:58] [11/22] ``` [04:10:58] [12/22] * no wiki leadership exist either by self demotion [04:10:59] [13/22] ```sql [04:10:59] [14/22] SELECT DISTINCT ug_user [04:11:00] [15/22] FROM user_groups [04:11:00] [16/22] WHERE ug_group IN EXPLICITGROUPS [04:11:01] [17/22] ``` [04:11:01] [18/22] might be good to add a feature to GDPR tool [04:11:02] [19/22] - check if user has any explicit usergroups at any wiki [04:11:02] [20/22] > - automatically demote the user < this is not currently done, no purpose retaining if converted to system account [04:11:03] [21/22] > - generate a report for stewards [04:11:03] [22/22] explicit usergroups: `UserGroupManager::getUserEffectiveGroups()` [04:38:26] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.39 ms [04:40:26] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.23 ms [04:44:36] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.34 ms [04:46:36] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.84 ms [04:53:54] [02mw-config] 07brandon-wm opened pull request 03#5550: T12096: Edit $wgRestrictionLevels for brandonwmwiki - 13https://github.com/miraheze/mw-config/pull/5550 [04:54:49] miraheze/mw-config - brandon-wm the build passed. [04:54:57] oh yay [05:01:41] this won't work out of the box, but its a good start to replacing the non-functional dormancy script [05:04:53] [1/2] looks like from the documentation page, `php sendMessages.php --pagelist --subject --message ` can be used to automate sending a dormancy notice after grabbing users from Special:ActiveUsers [05:04:53] [2/2] just needs the template [05:06:39] I might actually suggest Extension:MassMessageEmail as well, sending notices on wiki and via email [05:09:58] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 138.34 ms [05:11:57] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 135.52 ms [05:14:00] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 178.82 ms [05:16:00] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.36 ms [05:16:24] now one could argue if there is no wiki leadership, perhaps have it open for adoption...maybe that's a better solution but that's not in play [05:29:53] [1/2] https://issue-tracker.miraheze.org/T12095 [05:29:54] [2/2] Why is this problem occurring? [05:34:39] so visual editor broke on 1.42? [05:36:44] I presume visual editor is enabled per managewiki? [05:38:01] 1.42? [05:38:08] We use 1.41 [05:38:20] 1.42 isn't out yet [05:40:29] saw in here 1.42 [05:40:41] I am confused... huh? [05:40:51] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 29.13, 17.64, 9.79 [05:40:53] the visual editor works but only by adding `?veaction=edit` to the end of the url [05:41:02] !log [alex@test151] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'CreateWiki'} to test151 [05:41:06] it doesn't seem to be appearing in the page tabs [05:41:47] Beta used 1.42 but not prod [05:43:17] ok, disregard that [05:45:36] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.14 ms [05:48:51] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 16.66, 19.67, 14.59 [05:52:41] What do I have to do to get him to appear in the options on the page? [05:53:42] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.99 ms [06:06:58] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.30 ms [06:11:00] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.89 ms [06:21:55] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:50:51] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [07:35:54] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 167.99 ms [07:39:41] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 153.30 ms [07:39:55] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.13 ms [07:41:40] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 150.19 ms [07:55:38] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 155.28 ms [07:57:37] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 148.33 ms [09:00:34] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.09 ms [09:10:41] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.59 ms [09:23:04] PROBLEM - db171 Backups SQL on db171 is CRITICAL: FILE_AGE CRITICAL: /var/log/sql-backup.log is 1209635 seconds old and 122126 bytes [09:25:56] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.46 ms [09:27:56] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.23 ms [09:58:32] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 152.97 ms [10:00:31] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 153.61 ms [10:04:26] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.76 ms [10:06:26] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.34 ms [10:34:13] [02CreateWiki] 07redbluegreenhat opened pull request 03#512: Add RestWikiRequestsByUser - 13https://github.com/miraheze/CreateWiki/pull/512 [10:39:35] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 153.31 ms [10:41:34] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 159.01 ms [10:42:03] miraheze/CreateWiki - redbluegreenhat the build passed. [10:54:09] [02CreateWiki] 07redbluegreenhat synchronize pull request 03#512: Add RestWikiRequestsByUser - 13https://github.com/miraheze/CreateWiki/pull/512 [11:02:50] miraheze/CreateWiki - redbluegreenhat the build passed. [11:12:40] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 150.98 ms [11:14:39] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 153.69 ms [11:16:12] [02CreateWiki] 07redbluegreenhat synchronize pull request 03#512: Add RestWikiRequestsByUser - 13https://github.com/miraheze/CreateWiki/pull/512 [11:24:20] miraheze/CreateWiki - redbluegreenhat the build passed. [11:27:32] [02CreateWiki] 07redbluegreenhat closed pull request 03#512: Add RestWikiRequestsByUser - 13https://github.com/miraheze/CreateWiki/pull/512 [11:27:35] [02miraheze/CreateWiki] 07redbluegreenhat pushed 031 commit to 03master [+1/-0/±3] 13https://github.com/miraheze/CreateWiki/compare/b21aa80d873f...e84a5716336d [11:27:37] [02miraheze/CreateWiki] 07redbluegreenhat 03e84a571 - Add RestWikiRequestsByUser (#512) [11:29:27] !log [alex@test151] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'CreateWiki'} to test151 [11:29:28] !log [alex@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [11:29:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:29:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:30:24] !log [alex@test151] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'CreateWiki'} to test151 [11:30:25] !log [alex@test151] finished deploy of {'versions': '1.42', 'upgrade_extensions': 'CreateWiki'} to test151 - SUCCESS in 0s [11:30:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:30:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:32:10] !log [alex@test151] starting deploy of {'folders': 'extensions/CreateWiki'} to test151 [11:32:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:32:24] !log [alex@test151] starting deploy of {'folders': '1.42/extensions/CreateWiki'} to test151 [11:32:25] !log [alex@test151] finished deploy of {'folders': '1.42/extensions/CreateWiki'} to test151 - SUCCESS in 0s [11:32:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:32:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:34:26] miraheze/CreateWiki - redbluegreenhat the build has errored. [11:38:03] [02CreateWiki] 07redbluegreenhat opened pull request 03#513: fix CI - 13https://github.com/miraheze/CreateWiki/pull/513 [11:39:11] [02CreateWiki] 07redbluegreenhat closed pull request 03#513: fix CI - 13https://github.com/miraheze/CreateWiki/pull/513 [11:39:12] [02miraheze/CreateWiki] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/e84a5716336d...ffc5bfeb16ce [11:39:15] [02miraheze/CreateWiki] 07redbluegreenhat 03ffc5bfe - fix CI (#513) [11:39:16] !log [alex@test151] starting deploy of {'folders': 'extensions/CreateWiki'} to test151 [11:39:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:39:21] !log [alex@test151] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'CreateWiki'} to test151 [11:39:22] !log [alex@test151] finished deploy of {'versions': '1.42', 'upgrade_extensions': 'CreateWiki'} to test151 - SUCCESS in 1s [11:39:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:39:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:46:17] miraheze/CreateWiki - redbluegreenhat the build passed. [11:47:28] miraheze/CreateWiki - redbluegreenhat the build passed. [11:54:16] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.74 ms [11:56:16] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.01 ms [12:00:25] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.45 ms [12:02:26] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.69 ms [12:07:16] !log [alex@test151] starting deploy of {'l10n': True, 'versions': '1.42'} to test151 [12:07:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:12:37] !log [alex@test151] finished deploy of {'l10n': True, 'versions': '1.42'} to test151 - SUCCESS in 321s [12:12:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:13:02] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/CreateWiki/compare/ffc5bfeb16ce...a1f9cf6c3bc4 [12:13:04] [02miraheze/CreateWiki] 07translatewiki 03a1f9cf6 - Localisation updates from https://translatewiki.net. [12:13:07] [02miraheze/ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/a285bb6bed65...4f44c2328f4f [12:13:09] [02miraheze/ImportDump] 07translatewiki 034f44c23 - Localisation updates from https://translatewiki.net. [12:13:11] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/8dd7ffcae8a5...1c34044ac66c [12:13:14] [02miraheze/MirahezeMagic] 07translatewiki 031c34044 - Localisation updates from https://translatewiki.net. [12:17:35] miraheze/MirahezeMagic - translatewiki the build has errored. [12:18:49] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 172.51 ms [12:20:49] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.58 ms [12:21:33] miraheze/ImportDump - translatewiki the build passed. [12:22:18] miraheze/CreateWiki - translatewiki the build passed. [12:24:05] !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 [12:24:06] !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:17] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [12:24:18] !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:28] !log [@test151] starting deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 [12:24:29] !log [@test151] finished deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:32:08] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [12:32:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:32:18] !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 10s [12:32:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:32:28] !log [@mwtask181] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [12:32:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:32:38] !log [@mwtask181] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 9s [12:32:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:37:25] !log [@mwtask171] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [12:37:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:37:33] !log [@mwtask171] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 8s [12:37:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:37:43] !log [@mwtask171] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [12:37:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:37:51] !log [@mwtask171] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 7s [12:37:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:07:41] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 156.85 ms [13:08:31] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [13:11:40] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 157.93 ms [13:21:39] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.60 ms [13:25:40] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 173.70 ms [13:33:50] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 156.01 ms [13:37:33] RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [13:45:50] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 160.66 ms [13:56:01] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 155.59 ms [13:57:59] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 160.40 ms [13:59:23] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 170.68 ms [14:01:23] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.06 ms [14:11:57] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.25 ms [14:13:56] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 169.19 ms [14:17:00] [02mw-config] 07Reception123 closed pull request 03#5550: T12096: Edit $wgRestrictionLevels for brandonwmwiki - 13https://github.com/miraheze/mw-config/pull/5550 [14:17:03] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1798e53901a2...36ab4e200aaf [14:17:05] [02miraheze/mw-config] 07brandon-wm 0336ab4e2 - T12096: Edit $wgRestrictionLevels for brandonwmwiki (#5550) [14:17:48] !log [reception@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [14:17:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:17:59] !log [reception@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [14:18:00] miraheze/mw-config - Reception123 the build passed. [14:18:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:23:38] !log [@test151] starting deploy of {'config': True} to test151 [14:23:39] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [14:23:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:23:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:24:06] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 166.27 ms [14:26:04] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 162.57 ms [14:29:42] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.66 ms [14:31:42] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.88 ms [14:32:57] !log [reception@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [14:33:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:33:07] !log [reception@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [14:33:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:37:22] !log [@mwtask171] starting deploy of {'config': True} to all [14:37:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:37:31] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [14:37:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:50:11] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 170.51 ms [14:56:17] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.71 ms [15:06:38] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 170.13 ms [15:06:57] PROBLEM - evilgeniuswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'evilgeniuswiki.com' expires in 15 day(s) (Sat 18 May 2024 02:50:45 PM GMT +0000). [15:07:11] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/cb9295779eb4...e1e92228d8b5 [15:07:12] [02miraheze/ssl] 07WikiTideSSLBot 03e1e9222 - Bot: Update SSL cert for evilgeniuswiki.com [15:08:38] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 173.31 ms [15:10:08] PROBLEM - electowiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'electowiki.org' expires in 15 day(s) (Sat 18 May 2024 03:01:50 PM GMT +0000). [15:10:10] PROBLEM - psycho.engineering - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'psycho.engineering' expires in 15 day(s) (Sat 18 May 2024 02:53:17 PM GMT +0000). [15:10:20] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e1e92228d8b5...221faccd7b02 [15:10:21] [02miraheze/ssl] 07WikiTideSSLBot 03221facc - Bot: Update SSL cert for electowiki.org [15:10:34] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/221faccd7b02...8ec1de8fbd3c [15:10:37] [02miraheze/ssl] 07WikiTideSSLBot 038ec1de8 - Bot: Update SSL cert for psycho.engineering [15:15:04] PROBLEM - wiki.yuanpi.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.yuanpi.eu.org' expires in 15 day(s) (Sat 18 May 2024 02:52:07 PM GMT +0000). [15:15:18] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8ec1de8fbd3c...f4ec827c2ea5 [15:15:20] [02miraheze/ssl] 07WikiTideSSLBot 03f4ec827 - Bot: Update SSL cert for wiki.yuanpi.eu.org [15:16:24] Some new problems have arisen [15:16:27] https://issue-tracker.miraheze.org/T12099 [15:18:07] PROBLEM - storytime.jdstroy.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'storytime.jdstroy.cf' expires in 15 day(s) (Sat 18 May 2024 02:59:15 PM GMT +0000). [15:18:18] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f4ec827c2ea5...80cde1c87d31 [15:18:20] [02miraheze/ssl] 07WikiTideSSLBot 0380cde1c - Bot: Update SSL cert for storytime.jdstroy.cf [15:20:27] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 166.28 ms [15:22:25] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 167.44 ms [15:26:31] PROBLEM - privacywiki.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'privacywiki.xyz' expires in 15 day(s) (Sat 18 May 2024 02:56:14 PM GMT +0000). [15:26:45] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/80cde1c87d31...4e739898fed8 [15:26:48] [02miraheze/ssl] 07WikiTideSSLBot 034e73989 - Bot: Update SSL cert for privacywiki.xyz [15:26:58] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.29 ms [15:28:58] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.10 ms [15:32:25] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [15:35:55] RECOVERY - evilgeniuswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'evilgeniuswiki.com' will expire on Wed 31 Jul 2024 02:07:03 PM GMT +0000. [15:39:39] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [15:46:44] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 166.33 ms [15:55:32] RECOVERY - privacywiki.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'privacywiki.xyz' will expire on Wed 31 Jul 2024 02:26:40 PM GMT +0000. [15:58:47] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 161.24 ms [16:02:55] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 158.32 ms [16:04:54] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 159.28 ms [16:07:39] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:08:25] RECOVERY - electowiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'electowiki.org' will expire on Wed 31 Jul 2024 02:10:14 PM GMT +0000. [16:08:30] RECOVERY - psycho.engineering - LetsEncrypt on sslhost is OK: OK - Certificate 'psycho.engineering' will expire on Wed 31 Jul 2024 02:10:28 PM GMT +0000. [16:09:00] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 161.66 ms [16:10:00] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.12 ms [16:12:00] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.39 ms [16:12:58] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 156.30 ms [16:14:11] RECOVERY - wiki.yuanpi.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.yuanpi.eu.org' will expire on Wed 31 Jul 2024 02:15:12 PM GMT +0000. [16:14:42] [02CreateWiki] 07colleirose opened pull request 03#514: Remove colon from the database name header - 13https://github.com/miraheze/CreateWiki/pull/514 [16:15:43] [02CreateWiki] 07colleirose edited pull request 03#514: Remove colon from the database name header - 13https://github.com/miraheze/CreateWiki/pull/514 [16:17:20] RECOVERY - storytime.jdstroy.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'storytime.jdstroy.cf' will expire on Wed 31 Jul 2024 02:18:12 PM GMT +0000. [16:17:36] [02GlobalNewFiles] 07Universal-Omega synchronize pull request 03#56: Migrate to the usage of central ids - 13https://github.com/miraheze/GlobalNewFiles/pull/56 [16:17:39] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03centralids [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/compare/20dcf8d20f64...7ee08212caa2 [16:17:41] [02miraheze/GlobalNewFiles] 07Universal-Omega 037ee0821 - Keep setting files_user for now [16:20:06] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 159.32 ms [16:21:14] [02GlobalNewFiles] 07Universal-Omega synchronize pull request 03#56: Migrate to the usage of central ids - 13https://github.com/miraheze/GlobalNewFiles/pull/56 [16:21:17] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03centralids [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/compare/7ee08212caa2...14fb5431cc2e [16:21:19] [02miraheze/GlobalNewFiles] 07Universal-Omega 0314fb543 - Fix [16:22:05] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 159.03 ms [16:23:53] [02GlobalNewFiles] 07Universal-Omega synchronize pull request 03#56: Migrate to the usage of central ids - 13https://github.com/miraheze/GlobalNewFiles/pull/56 [16:23:54] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03centralids [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/compare/14fb5431cc2e...834da37e128b [16:23:57] [02miraheze/GlobalNewFiles] 07Universal-Omega 03834da37 - Keep using files_user if uploader is not set [16:24:19] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.72 ms [16:26:11] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 160.77 ms [16:26:19] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.63 ms [16:28:10] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 157.43 ms [16:28:19] miraheze/GlobalNewFiles - Universal-Omega the build passed. [16:34:31] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03centralids [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/compare/834da37e128b...7a52369bc256 [16:34:33] [02miraheze/GlobalNewFiles] 07Universal-Omega 037a52369 - Add [16:34:36] [02GlobalNewFiles] 07Universal-Omega synchronize pull request 03#56: Migrate to the usage of central ids - 13https://github.com/miraheze/GlobalNewFiles/pull/56 [16:35:41] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03centralids [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/compare/7a52369bc256...6d3db88f6193 [16:35:43] [02miraheze/GlobalNewFiles] 07Universal-Omega 036d3db88 - Add [16:35:46] [02GlobalNewFiles] 07Universal-Omega synchronize pull request 03#56: Migrate to the usage of central ids - 13https://github.com/miraheze/GlobalNewFiles/pull/56 [16:40:04] miraheze/GlobalNewFiles - Universal-Omega the build passed. [16:40:40] [02GlobalNewFiles] 07Universal-Omega synchronize pull request 03#56: Migrate to the usage of central ids - 13https://github.com/miraheze/GlobalNewFiles/pull/56 [16:40:41] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03centralids [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/compare/6d3db88f6193...9cdeb8e16cb1 [16:40:42] [02miraheze/GlobalNewFiles] 07Universal-Omega 039cdeb8e - Use getServiceContainer [16:42:25] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [16:45:00] miraheze/GlobalNewFiles - Universal-Omega the build passed. [16:52:00] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [16:56:16] PROBLEM - miraheze.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'miraheze.wiki' expires in 7 day(s) (Fri 10 May 2024 04:35:32 PM GMT +0000). [16:57:12] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 167.36 ms [17:01:12] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 154.48 ms [17:07:38] [02GlobalNewFiles] 07Universal-Omega synchronize pull request 03#56: Migrate to the usage of central ids - 13https://github.com/miraheze/GlobalNewFiles/pull/56 [17:07:39] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03centralids [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/compare/9cdeb8e16cb1...a6144ae15ffb [17:07:40] [02miraheze/GlobalNewFiles] 07Universal-Omega 03a6144ae - Default to NULL in patch for now [17:11:50] [1/2] !log [17:11:50] [2/2] ALTER TABLE gnf_files ADD COLUMN files_uploader INT UNSIGNED DEFAULT NULL AFTER files_name; on mhglobal/testglobal [17:12:23] !log ALTER TABLE gnf_files ADD COLUMN files_uploader INT UNSIGNED DEFAULT NULL AFTER files_name; on mhglobal/testglobal [17:12:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:12:49] miraheze/GlobalNewFiles - Universal-Omega the build passed. [17:15:58] [02GlobalNewFiles] 07Universal-Omega closed pull request 03#56: Migrate to the usage of central ids - 13https://github.com/miraheze/GlobalNewFiles/pull/56 [17:15:59] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03master [+2/-0/±5] 13https://github.com/miraheze/GlobalNewFiles/compare/6b7eaef7eab4...749c91f98173 [17:16:01] [02miraheze/GlobalNewFiles] 07Universal-Omega 03749c91f - Migrate to the usage of central ids (#56) [17:16:03] [02miraheze/GlobalNewFiles] 07Universal-Omega deleted branch 03centralids [17:16:05] [02GlobalNewFiles] 07Universal-Omega deleted branch 03centralids - 13https://github.com/miraheze/GlobalNewFiles [17:20:31] miraheze/GlobalNewFiles - Universal-Omega the build passed. [17:20:55] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [17:26:22] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 158.84 ms [17:28:21] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 156.36 ms [17:46:43] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03batch [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/commit/870ed6ec3926 [17:46:46] [02miraheze/GlobalNewFiles] 07Universal-Omega 03870ed6e - Support batching migration script [17:46:48] [02GlobalNewFiles] 07Universal-Omega created branch 03batch - 13https://github.com/miraheze/GlobalNewFiles [17:46:51] [02GlobalNewFiles] 07Universal-Omega opened pull request 03#57: Support batching migration script - 13https://github.com/miraheze/GlobalNewFiles/pull/57 [17:48:56] [02CreateWiki] 07redbluegreenhat opened pull request 03#515: cast the request ID to int - 13https://github.com/miraheze/CreateWiki/pull/515 [17:50:38] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 157.55 ms [17:50:46] [02CreateWiki] 07redbluegreenhat synchronize pull request 03#515: cast the request ID to int - 13https://github.com/miraheze/CreateWiki/pull/515 [17:51:26] miraheze/GlobalNewFiles - Universal-Omega the build passed. [17:51:26] [02CreateWiki] 07redbluegreenhat edited pull request 03#515: cast the request ID to int and `bio` to bool - 13https://github.com/miraheze/CreateWiki/pull/515 [17:54:37] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 153.25 ms [17:55:43] [02GlobalNewFiles] 07Universal-Omega synchronize pull request 03#57: Support batching migration script - 13https://github.com/miraheze/GlobalNewFiles/pull/57 [17:55:46] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03batch [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/compare/870ed6ec3926...86acd3e92709 [17:55:48] [02miraheze/GlobalNewFiles] 07Universal-Omega 0386acd3e - output [17:57:26] [02CreateWiki] 07redbluegreenhat edited pull request 03#515: REST: Cast some data to appropiate types - 13https://github.com/miraheze/CreateWiki/pull/515 [17:57:32] miraheze/CreateWiki - redbluegreenhat the build passed. [17:58:25] [02CreateWiki] 07redbluegreenhat edited pull request 03#515: REST: Cast some data to appropiate types - 13https://github.com/miraheze/CreateWiki/pull/515 [17:59:17] miraheze/CreateWiki - redbluegreenhat the build passed. [17:59:50] miraheze/GlobalNewFiles - Universal-Omega the build passed. [17:59:52] [02CreateWiki] 07redbluegreenhat closed pull request 03#515: REST: Cast some data to appropiate types - 13https://github.com/miraheze/CreateWiki/pull/515 [17:59:54] [02miraheze/CreateWiki] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/compare/a1f9cf6c3bc4...32b5c7e1a8ad [17:59:55] [02miraheze/CreateWiki] 07redbluegreenhat 0332b5c7e - REST: Cast some data to appropiate types (#515) [18:00:34] !log [alex@test151] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'CreateWiki'} to test151 [18:00:35] !log [alex@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [18:00:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:00:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:00:55] Here we go again [18:00:59] AAAAAAAAAAAAAAAAA [18:01:13] 😭 [18:01:59] Is it the PR again [18:02:04] yes [18:02:55] !log [alex@test151] starting deploy of {'folders': '1.42/extensions/CreateWiki'} to test151 [18:02:56]