[00:02:31] [discord] Can I be un-disabled from phabricator? [00:02:47] [discord] If there’s an email I should appeal please send it [00:03:09] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:04:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.85, 3.68, 3.13 [00:06:30] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.62, 3.35, 3.08 [00:10:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.43, 4.26, 3.51 [00:13:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.83, 3.74, 3.11 [00:13:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.34, 3.97, 3.35 [00:14:36] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:15:11] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.78, 3.38, 3.05 [00:15:20] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.64, 3.34, 3.18 [00:20:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.19, 3.89, 3.79 [00:20:07] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:27:16] PROBLEM - puppet111 APT on puppet111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [00:27:44] PROBLEM - puppet111 PowerDNS Recursor on puppet111 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:28:01] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a3db070ae0f2...6c0f4de96475 [00:28:03] [02miraheze/mw-config] 07Universal-Omega 036c0f4de - Raise restbase timeout to 30 seconds [00:28:59] PROBLEM - puppet111 Current Load on puppet111 is CRITICAL: CRITICAL - load average: 35.71, 20.09, 9.41 [00:29:02] miraheze/mw-config - Universal-Omega the build passed. [00:29:36] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:30:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.97, 3.52, 3.51 [00:30:36] PROBLEM - puppet111 Puppet on puppet111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:31:03] PROBLEM - puppet111 conntrack_table_size on puppet111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:32:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.88, 2.73, 3.22 [00:33:29] PROBLEM - puppet111 SSH on puppet111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:37:04] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:37:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:37:17] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 13s [00:37:20] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all [00:37:21] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all - SUCCESS in 1s [00:37:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:37:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:37:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:39:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.87, 9.88, 8.94 [00:40:19] PROBLEM - puppet111 NTP time on puppet111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:41:18] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.71, 9.76, 9.02 [00:41:47] PROBLEM - puppet111 ferm_active on puppet111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:42:21] paladox: ^ what is up with puppet111? [00:43:03] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:43:09] RECOVERY - puppet111 conntrack_table_size on puppet111 is OK: OK: nf_conntrack is 0 % full [00:43:15] RECOVERY - puppet111 NTP time on puppet111 is OK: NTP OK: Offset -0.0224968493 secs [00:43:27] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:43:44] RECOVERY - puppet111 SSH on puppet111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [00:43:45] @Orduin: ^^ [00:43:46] RECOVERY - puppet111 ferm_active on puppet111 is OK: OK ferm input default policy is set [00:43:51] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:43:58] RECOVERY - puppet111 APT on puppet111 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [00:43:58] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:44:16] PROBLEM - bast121 Puppet on bast121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:44:25] PROBLEM - mw102 Puppet on mw102 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:45:00] RECOVERY - puppet111 PowerDNS Recursor on puppet111 is OK: DNS OK: 0.124 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [00:45:20] PROBLEM - mw121 Puppet on mw121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:45:43] PROBLEM - mw111 Puppet on mw111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:45:45] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:46:02] PROBLEM - gluster111 Puppet on gluster111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:46:12] PROBLEM - db111 Puppet on db111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:46:14] PROBLEM - db112 Puppet on db112 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:46:22] PROBLEM - es121 Puppet on es121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:46:34] PROBLEM - ldap111 Puppet on ldap111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:46:35] PROBLEM - mem121 Puppet on mem121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:46:37] PROBLEM - phab121 Puppet on phab121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:46:40] PROBLEM - mw122 Puppet on mw122 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:46:40] PROBLEM - gluster101 Puppet on gluster101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:46:43] PROBLEM - mon111 Puppet on mon111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:46:53] PROBLEM - bast101 Puppet on bast101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:46:56] PROBLEM - graylog121 Puppet on graylog121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:46:59] PROBLEM - mail121 Puppet on mail121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:47:04] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/commit/14fce82fd7c7 [00:47:05] [02miraheze/ManageWiki] 07Universal-Omega 0314fce82 - pass DBConnRef to CreateWikiJsonBuilder [00:47:07] [02ManageWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ManageWiki [00:47:08] PROBLEM - gluster121 Puppet on gluster121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:47:08] [02ManageWiki] 07Universal-Omega opened pull request 03#354: pass DBConnRef to CreateWikiJsonBuilder - 13https://github.com/miraheze/ManageWiki/pull/354 [00:49:36] ok : [RESOLVED] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:51:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.21, 9.60, 9.03 [00:52:17] miraheze/ManageWiki - Universal-Omega the build passed. [00:53:18] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.98, 8.67, 8.76 [00:54:50] RECOVERY - puppet111 Puppet on puppet111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:55:27] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [00:55:51] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:57:45] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:58:02] RECOVERY - gluster111 Puppet on gluster111 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [00:58:43] RECOVERY - mon111 Puppet on mon111 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [00:58:56] RECOVERY - graylog121 Puppet on graylog121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:59:08] RECOVERY - gluster121 Puppet on gluster121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:02:13] RECOVERY - db112 Puppet on db112 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:02:35] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:02:35] RECOVERY - mem121 Puppet on mem121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:04:40] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [01:04:48] RECOVERY - bast101 Puppet on bast101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:04:58] !log sudo apt upgrade, then sudo reboot on mw101 (upgrades kernel to 5.10.0-13-amd64) [01:05:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:05:43] RECOVERY - mw111 Puppet on mw111 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [01:07:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.12, 10.19, 8.76 [01:08:37] RECOVERY - phab121 Puppet on phab121 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [01:09:04] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [01:09:43] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 10.18, 10.01, 8.86 [01:09:47] RECOVERY - db111 Puppet on db111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:11:42] !log sudo apt upgrade, then sudo reboot on mw102 (upgrades kernel to 5.10.0-13-amd64) [01:12:00] RECOVERY - bast121 Puppet on bast121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:12:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:12:09] PROBLEM - puppet111 Current Load on puppet111 is WARNING: WARNING - load average: 0.94, 1.30, 5.70 [01:12:34] RECOVERY - ldap111 Puppet on ldap111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:12:48] RECOVERY - mail121 Puppet on mail121 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [01:13:20] RECOVERY - mw121 Puppet on mw121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:14:22] RECOVERY - es121 Puppet on es121 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [01:14:41] RECOVERY - gluster101 Puppet on gluster101 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [01:16:01] RECOVERY - puppet111 Current Load on puppet111 is OK: OK - load average: 1.18, 1.21, 4.68 [01:16:25] RECOVERY - mw102 Puppet on mw102 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:16:41] !log sudo apt upgrade, then sudo reboot on mw111 (upgrades kernel to 5.10.0-13-amd64) [01:16:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:18:49] !log sudo apt upgrade, then sudo reboot on mw112 (upgrades kernel to 5.10.0-13-amd64) [01:19:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:19:58] !log sudo apt upgrade, then sudo reboot on mw121 (upgrades kernel to 5.10.0-13-amd64) [01:20:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:21:04] !log sudo apt upgrade, then sudo reboot on mw122 (upgrades kernel to 5.10.0-13-amd64) [01:21:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:23:06] [02ManageWiki] 07Universal-Omega closed pull request 03#354: pass DBConnRef to CreateWikiJsonBuilder - 13https://github.com/miraheze/ManageWiki/pull/354 [01:23:07] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/59cafcbeca90...7c51b12218dc [01:23:08] PROBLEM - mw112 Puppet on mw112 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 3 minutes ago with 6 failures. Failed resources (up to 3 shown): Package[gnupg],Package[dirmngr],Package[openssl],Package[htmldoc] [01:23:09] [02miraheze/ManageWiki] 07Universal-Omega 037c51b12 - pass DBConnRef to CreateWikiJsonBuilder (#354) [01:23:10] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [01:23:12] [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ManageWiki [01:24:40] !log [universalomega@test101] starting deploy of {'folders': 'w/extensions/CreateWiki'} to all [01:24:41] !log [universalomega@test101] finished deploy of {'folders': 'w/extensions/CreateWiki'} to all - SUCCESS in 0s [01:24:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:24:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:25:07] RECOVERY - mw112 Puppet on mw112 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [01:27:29] miraheze/ManageWiki - Universal-Omega the build passed. [01:28:07] miraheze/MirahezeMagic - Universal-Omega the build has errored. [01:28:25] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/37f186087a76...09d50a0bc5f4 [01:28:27] [02miraheze/MirahezeMagic] 07Universal-Omega 0309d50a0 - remove unused `use` [01:32:22] miraheze/MirahezeMagic - Universal-Omega the build has errored. [01:33:05] Now that is the upstream issue... [01:39:23] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [01:41:58] [02CreateWiki] 07Universal-Omega closed pull request 03#312: Create new CreateWikiRegexConstraint class - 13https://github.com/miraheze/CreateWiki/pull/312 [01:41:59] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±3] 13https://github.com/miraheze/CreateWiki/compare/3ac41942d549...53ef09b5526b [01:42:01] [02miraheze/CreateWiki] 07Universal-Omega 0353ef09b - Create new CreateWikiRegexConstraint class (#312) [01:42:07] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/CreateWiki [01:42:08] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [01:46:10] miraheze/CreateWiki - Universal-Omega the build has errored. [01:47:22] dmehus: around? [01:50:01] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/9027ea952b51...4a03f385fd7c [01:50:03] [02miraheze/mediawiki] 07Universal-Omega 034a03f38 - Update CreateWiki [01:50:27] !log [universalomega@test101] starting deploy of {'world': True} to all [01:50:30] !log [universalomega@test101] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [01:50:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:50:35] !log [universalomega@test101] starting deploy of {'world': True, 'l10n': True} to all [01:50:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:50:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:50:50] !log [universalomega@mwtask111] starting deploy of {'world': True} to all [01:51:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:51:37] !log [universalomega@mwtask111] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [01:51:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:52:19] !log [universalomega@mwtask111] starting deploy of {'folders': 'w/extensions/CreateWiki'} to all [01:52:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:52:36] !log [universalomega@mwtask111] finished deploy of {'folders': 'w/extensions/CreateWiki'} to all - SUCCESS in 16s [01:52:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:52:51] !log [universalomega@mwtask111] starting deploy of {'world': True, 'l10n': True} to all [01:52:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:54:26] dmehus: If around, can you please delete the redirect at https://meta.miraheze.org/wiki/MediaWiki:CreateWiki-blacklist? [01:54:27] [url] MediaWiki:CreateWiki-disallowlist - Miraheze Meta | meta.miraheze.org [01:55:38] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/53ef09b5526b...80317730744b [01:55:39] [02miraheze/CreateWiki] 07Universal-Omega 038031773 - Remove MediaWiki:CreateWiki-blacklist support [01:57:18] !log [universalomega@test101] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [01:57:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:59:13] miraheze/CreateWiki - Universal-Omega the build has errored. [02:01:24] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.52, 1.91, 1.37 [02:01:35] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/4a03f385fd7c...b08de3781b59 [02:01:37] [02miraheze/mediawiki] 07Universal-Omega 03b08de37 - Update CreateWiki [02:01:46] !log [universalomega@mwtask111] starting deploy of {'world': True, 'l10n': True} to all [02:01:50] !log [universalomega@test101] starting deploy of {'world': True, 'l10n': True} to all [02:01:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:01:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:02:09] !log [universalomega@test101] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [02:02:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:02:19] !log [universalomega@test101] starting deploy of {'world': True, 'l10n': True} to all [02:02:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:02:26] !log [universalomega@mwtask111] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [02:02:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:02:34] !log [universalomega@mwtask111] starting deploy of {'world': True, 'l10n': True} to all [02:02:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:05:14] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.60, 1.34, 1.26 [02:08:15] !log [universalomega@test101] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 356s [02:08:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:09:34] !log [universalomega@mwtask111] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 419s [02:09:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:34:45] [02mw-config] 07ugochimobi commented on pull request 03#4583: Update Wikibase.php with propertyOrderUrl and dataRightsText - 13https://github.com/miraheze/mw-config/pull/4583#issuecomment-1099796167 [02:38:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:43:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:55:10] [02mw-config] 07Universal-Omega closed pull request 03#4583: Update Wikibase.php with propertyOrderUrl and dataRightsText - 13https://github.com/miraheze/mw-config/pull/4583 [02:55:12] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6c0f4de96475...4c0fa04dee4d [02:55:13] [02miraheze/mw-config] 07ugochimobi 034c0fa04 - Update Wikibase.php with propertyOrderUrl and dataRightsText (#4583) [02:56:23] miraheze/mw-config - Universal-Omega the build passed. [03:04:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.62, 3.86, 3.12 [03:06:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.80, 3.37, 3.03 [03:13:49] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/commit/ed936e015754 [03:13:51] [02miraheze/CreateWiki] 07Universal-Omega 03ed936e0 - Update nl.json [03:13:52] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/CreateWiki [03:14:25] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/ed936e015754...d6759796fa91 [03:14:27] [02miraheze/CreateWiki] 07Universal-Omega 03d675979 - Update pt-br.json [03:14:51] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/d6759796fa91...1146830a94d8 [03:14:53] [02miraheze/CreateWiki] 07Universal-Omega 031146830 - Update es.json [03:15:30] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/1146830a94d8...0492e8ad2d3e [03:15:32] [02miraheze/CreateWiki] 07Universal-Omega 030492e8a - Update fr.json [03:16:19] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/0492e8ad2d3e...7bee72f24271 [03:16:20] [02miraheze/CreateWiki] 07Universal-Omega 037bee72f - Update hi.json [03:16:49] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/7bee72f24271...b6d5f069b660 [03:16:50] [02miraheze/CreateWiki] 07Universal-Omega 03b6d5f06 - Update ia.json [03:17:21] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/b6d5f069b660...3c7d2a20a470 [03:17:23] [02miraheze/CreateWiki] 07Universal-Omega 033c7d2a2 - Update nb.json [03:17:53] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/3c7d2a20a470...18694ea14cc7 [03:17:54] [02miraheze/CreateWiki] 07Universal-Omega 0318694ea - Update sk.json [03:18:18] !log [@test101] starting deploy of {'config': True} to all [03:18:19] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [03:18:20] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/18694ea14cc7...3912d1fff6c8 [03:18:22] [02miraheze/CreateWiki] 07Universal-Omega 033912d1f - Update de.json [03:18:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:18:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:18:52] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/3912d1fff6c8...7e9dd3b9a95a [03:18:54] [02miraheze/CreateWiki] 07Universal-Omega 037e9dd3b - Update pt.json [03:19:31] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/7e9dd3b9a95a...479ce1a261c6 [03:19:32] [02miraheze/CreateWiki] 07Universal-Omega 03479ce1a - Update he.json [03:20:05] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/479ce1a261c6...602e982bccc6 [03:20:07] [02miraheze/CreateWiki] 07Universal-Omega 03602e982 - Update sv.json [03:20:37] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/602e982bccc6...4434b99ac31d [03:20:38] [02miraheze/CreateWiki] 07Universal-Omega 034434b99 - Update uk.json [03:21:06] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/4434b99ac31d...c003fac34733 [03:21:07] [02miraheze/CreateWiki] 07Universal-Omega 03c003fac - Update en.json [03:21:45] !log [@mwtask111] starting deploy of {'config': True} to all [03:21:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:21:54] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 9s [03:22:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:23:39] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/c003fac34733...c231bb32ff7f [03:23:40] [02miraheze/CreateWiki] 07Universal-Omega 03c231bb3 - Update ja.json [03:24:09] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/c231bb32ff7f...1e7694ee3a2f [03:24:10] [02miraheze/CreateWiki] 07Universal-Omega 031e7694e - Update zh-hans.json [03:24:32] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/1e7694ee3a2f...671d3aff1732 [03:24:34] [02miraheze/CreateWiki] 07Universal-Omega 03671d3af - Update zh-hant.json [03:25:02] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/671d3aff1732...bd2e377c3dc8 [03:25:04] [02miraheze/CreateWiki] 07Universal-Omega 03bd2e377 - Update nl-informal.json [03:25:32] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/bd2e377c3dc8...13ea955898a1 [03:25:33] [02miraheze/CreateWiki] 07Universal-Omega 0313ea955 - Update pl.json [03:26:08] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/13ea955898a1...da6d77aad588 [03:26:10] [02miraheze/CreateWiki] 07Universal-Omega 03da6d77a - Update fi.json [03:26:50] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/da6d77aad588...954cfb64a456 [03:26:51] [02miraheze/CreateWiki] 07Universal-Omega 03954cfb6 - Update tr.json [03:27:22] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/954cfb64a456...005393092c94 [03:27:24] [02miraheze/CreateWiki] 07Universal-Omega 030053930 - Update ru.json [03:27:49] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/005393092c94...53a370ff8de7 [03:27:50] [02miraheze/CreateWiki] 07Universal-Omega 0353a370f - Update it.json [03:28:19] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/53a370ff8de7...ad4c72c43939 [03:28:20] [02miraheze/CreateWiki] 07Universal-Omega 03ad4c72c - Update mk.json [03:29:00] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/ad4c72c43939...2f1fce210c5e [03:29:01] [02miraheze/CreateWiki] 07Universal-Omega 032f1fce2 - Update WikiRequest.php [03:31:30] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/2f1fce210c5e...1c6cc73269dd [03:31:31] [02miraheze/CreateWiki] 07Universal-Omega 031c6cc73 - Update en.json [03:32:14] [02CreateWiki] 07Universal-Omega opened pull request 03#316: Rename createwiki-error-blacklisted to createwiki-error-disallowed - 13https://github.com/miraheze/CreateWiki/pull/316 [03:33:00] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [03:35:45] miraheze/CreateWiki - Universal-Omega the build has errored. [03:38:10] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4c0fa04dee4d...a5810663be3b [03:38:11] [02miraheze/mw-config] 07Universal-Omega 03a581066 - Raise parsoid timeout to 45 seconds [03:39:12] miraheze/mw-config - Universal-Omega the build passed. [03:43:48] CosmicAlpha: {{Done}}. I didn't see your ping until now. I just deleted the redirect when I saw your change. No issues with that. TIL [[MediaWiki:CreateWiki-blacklist]] existed :D [03:43:48] https://meta.miraheze.org/wiki/Template:Done https://meta.miraheze.org/wiki/MediaWiki:CreateWiki-blacklist [03:45:51] dmehus: Thanks! And LOL, Agent said the exact same thing earlier, about it existing. But yeah I renamed it to CreateWiki-disallowlist to follow new conventions. But it's used as a blacklist for request descriptions. [03:46:30] Too bad WCs can’t edit it [03:46:54] would be useful to add certain strings to it like against abbw’s requests [03:48:17] !log [@test101] starting deploy of {'config': True} to all [03:48:18] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [03:48:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:48:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:52:16] !log [@mwtask111] starting deploy of {'config': True} to all [03:52:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:52:35] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 18s [03:52:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:54:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.17, 3.25, 3.00 [03:56:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.25, 3.14, 2.98 [03:59:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:01:35] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [04:03:49] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a5810663be3b...655f209291fb [04:03:50] [02miraheze/mw-config] 07Universal-Omega 03655f209 - Raise parsoid timeout to 60 seconds [04:04:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:04:59] miraheze/mw-config - Universal-Omega the build passed. [04:06:01] [02CreateWiki] 07Universal-Omega closed pull request 03#316: Rename createwiki-error-blacklisted to createwiki-error-disallowed - 13https://github.com/miraheze/CreateWiki/pull/316 [04:06:03] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±25] 13https://github.com/miraheze/CreateWiki/compare/80317730744b...68e080f9a477 [04:06:05] [02miraheze/CreateWiki] 07Universal-Omega 0368e080f - Rename createwiki-error-blacklisted to createwiki-error-disallowed (#316) [04:06:06] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/CreateWiki [04:06:08] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [04:09:24] miraheze/CreateWiki - Universal-Omega the build has errored. [04:13:19] CosmicAlpha: Oh, just for the description field, not for the subdomain field? I was thinking it wouldn't be a bad idea to migrate the CreateWiki subdomain blacklist to a MediaWiki interface message. Or if not migrate, rather have the CreateWiki subdomain blacklist array in LocalSettings.php that supercedes any specifications in the MediaWiki interface message, so that the MediaWiki interface message could supplement the LS.php subdomain [04:13:19] blacklist [04:14:39] dmehus: that one is just for the description field. The subdomain one is more sysadmin controlled because it includes so much of only technical subdomain being blacklisted, that moving it on-wiki wouldn't be the best approach. [04:14:55] I do however think maybe canned responses should be moved on-wiki. [04:16:03] And we can't have both at the same time, it'd complicate things to much within CreateWiki I think. [04:18:00] CosmicAlpha, well, we could still move the subdomain blacklist to an interface message, and add a new `editglobalgroupprotected` user right that could then be assigned to each of SRE, Stewards, and Trust and Safety. We would then protect the page as "Allow only users with specified global permissions" protected [04:18:16] and okay re: the latter comment [04:18:28] !log [@test101] starting deploy of {'config': True} to all [04:18:29] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [04:18:47] Moving the canned responses would be fine on-wiki, would that complicate the gradual rollout of the CreateWiki AI auto-decisioning? [04:18:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:18:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:19:31] With your SiteConfiguration changes, I think the jobqueue could potentially handle the CreateWiki AI scoring and auto-decisioning, with strict scoring thresholds for auto-approval [04:19:54] dmehus: on wiki means steward controlled, the blacklist is almost entirely used for technical reasons, with some exceptions, I don't think moving that is necessary. As for AI question, no that wouldn't be an issue. [04:19:59] It might be able to yeah. [04:20:19] Though that is still extremely memory-intensive so not sure. [04:21:12] CosmicAlpha, not necessarily. It would still be possible for SRE to control that interface message. Stewards would be trusted not to remove subdomains required for our infrastructure. They could, however, add and remove other, non-SRE subdomains to the blacklist :) [04:21:28] CosmicAlpha, cool re: the latter [04:21:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.79, 1.70, 1.22 [04:22:25] !log [@mwtask111] starting deploy of {'config': True} to all [04:22:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:22:32] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 7s [04:22:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:23:11] dmehus: I don't think moving it there is worth it is all. It'll complicate things a bit, and I just rewrote the entire regex system of CreateWiki (all parts of it) to add validation, and log if regex is invalid, but still pass the valid parts of it. So doing this now would be a pain tbh. [04:23:49] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.50, 1.28, 1.13 [04:37:01] [02mw-config] 07dmehus opened pull request 03#4588: Remove user rights sysadmin did not have - 13https://github.com/miraheze/mw-config/pull/4588 [04:37:59] miraheze/mw-config - dmehus the build passed. [04:38:10] [02mw-config] 07Universal-Omega closed pull request 03#4588: Remove user rights sysadmin did not have - 13https://github.com/miraheze/mw-config/pull/4588 [04:38:12] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/655f209291fb...46f3b7a438b8 [04:38:13] [02miraheze/mw-config] 07dmehus 0346f3b7a - Remove user rights sysadmin did not have (#4588) [04:38:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/46f3b7a438b8...1ea88cb57a0d [04:38:49] [02miraheze/mw-config] 07Universal-Omega 031ea88cb - Lower parsoid timeout to 30 seconds [04:39:10] miraheze/mw-config - Universal-Omega the build passed. [04:39:48] miraheze/mw-config - Universal-Omega the build passed. [04:47:48] !log [@test101] starting deploy of {'config': True} to all [04:47:49] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [04:47:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:48:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:51:48] !log [@mwtask111] starting deploy of {'config': True} to all [04:51:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:51:56] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 7s [04:52:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:55:55] [02MirahezeMagic] 07dmehus commented on pull request 03#333: use CW config variables for SiteNoticeAfter -- - 13https://github.com/miraheze/MirahezeMagic/pull/333#issuecomment-1099849055 [05:00:12] > dmehus: I don't think moving it there is worth it is all. It'll complicate things a bit, and I just rewrote the entire regex system of CreateWiki (all parts of it) to add validation, and log if regex is invalid, but still pass the valid parts of it. So doing this now would be a pain tbh. [05:00:12] CosmicAlpha: oh okay, that's fine then [05:03:20] [02MirahezeMagic] 07Universal-Omega commented on pull request 03#333: use CW config variables for SiteNoticeAfter -- - 13https://github.com/miraheze/MirahezeMagic/pull/333#issuecomment-1099851306 [05:13:26] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1ea88cb57a0d...d23f8b4c1fe9 [05:13:28] [02miraheze/mw-config] 07Universal-Omega 03d23f8b4 - Remove spf-tmp logging [05:14:29] miraheze/mw-config - Universal-Omega the build passed. [05:15:09] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d23f8b4c1fe9...f8ab21db9668 [05:15:11] [02miraheze/mw-config] 07Universal-Omega 03f8ab21d - Disable HitCounters logging channel [05:15:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.81, 3.96, 3.10 [05:16:18] miraheze/mw-config - Universal-Omega the build passed. [05:17:12] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.58, 3.39, 2.99 [05:18:55] !log [@test101] starting deploy of {'config': True} to all [05:18:56] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [05:18:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:19:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:22:14] !log [@mwtask111] starting deploy of {'config': True} to all [05:22:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:22:23] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 9s [05:22:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:45:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:50:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:10:00] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.24, 1.63, 1.17 [06:42:00] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 2.00, 2.00, 2.00 [06:44:00] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.19, 2.10, 2.03 [07:00:00] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.64, 1.80, 1.97 [07:06:00] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.06, 1.26, 1.67 [07:21:43] PROBLEM - wiki.jkrstudios.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.jkrstudios.org' expires in 15 day(s) (Sun 01 May 2022 07:09:53 GMT +0000). [07:27:44] PROBLEM - microworldwiki.ga - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'microworldwiki.ga' expires in 15 day(s) (Sun 01 May 2022 07:07:47 GMT +0000). [07:28:57] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a239a7c79353...a3ba9c88dc89 [07:28:58] [02miraheze/ssl] 07MirahezeSSLBot 03a3ba9c8 - Bot: Update SSL cert for wiki.jkrstudios.org [07:30:03] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a3ba9c88dc89...1e84050a2097 [07:30:05] [02miraheze/ssl] 07MirahezeSSLBot 031e84050 - Bot: Update SSL cert for microworldwiki.ga [07:35:08] [02CreateWiki] 07dmehus commented on pull request 03#312: Create new CreateWikiRegexConstraint class - 13https://github.com/miraheze/CreateWiki/pull/312#issuecomment-1099930945 [07:36:42] [02MirahezeMagic] 07dmehus commented on pull request 03#333: use CW config variables for SiteNoticeAfter -- - 13https://github.com/miraheze/MirahezeMagic/pull/333#issuecomment-1099931661 [07:52:09] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:52:34] Reception123: are you doing ssl? [07:53:38] Nope [07:53:52] And I doubt MacFan4000 is at this time [07:54:27] Reception123: can you check nginx conftest then [07:54:34] and see why it won't restart [07:56:28] RECOVERY - microworldwiki.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'microworldwiki.ga' will expire on Thu 14 Jul 2022 06:29:57 GMT +0000. [07:58:07] PROBLEM - nottingham.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nottingham.wiki' expires in 15 day(s) (Sun 01 May 2022 07:35:10 GMT +0000). [08:06:15] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1e84050a2097...0562212b7eb4 [08:06:16] [02miraheze/ssl] 07MirahezeSSLBot 030562212 - Bot: Update SSL cert for nottingham.wiki [08:08:04] Will do in a few mins [08:10:31] hmm [08:10:32] Testing nginx configuration:. [08:10:43] nothing other than [08:10:47] https://www.irccloud.com/pastebin/7CXzEYhm/ [08:11:01] so must be a test101 exclusive thing? [08:12:53] should be fixed, /var/log/nginx was missing [08:18:07] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [08:18:48] Reception123: ok [08:18:50] Thanks [08:21:08] RECOVERY - wiki.jkrstudios.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.jkrstudios.org' will expire on Thu 14 Jul 2022 06:28:51 GMT +0000. [08:26:00] PROBLEM - wiki.wilderyogi.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.wilderyogi.eu' expires in 15 day(s) (Sun 01 May 2022 08:14:32 GMT +0000). [08:26:37] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0562212b7eb4...5aa7c81b3c5a [08:26:39] [02miraheze/ssl] 07MirahezeSSLBot 035aa7c81 - Bot: Update SSL cert for wiki.wilderyogi.eu [08:27:13] RECOVERY - nottingham.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nottingham.wiki' will expire on Thu 14 Jul 2022 07:06:09 GMT +0000. [09:24:00] RECOVERY - wiki.wilderyogi.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.wilderyogi.eu' will expire on Thu 14 Jul 2022 07:26:31 GMT +0000. [10:02:05] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [10:02:05] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:00:06] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [11:29:28] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.58, 3.19, 2.67 [11:30:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.15, 3.38, 2.82 [11:31:24]