[00:00:27] [02mediawiki-repos] 07Universal-Omega created 03LdapAuthentication-fork (+1 new commit) 13https://github.com/miraheze/mediawiki-repos/commit/ad4bc00c177a [00:00:29] 02mediawiki-repos/03LdapAuthentication-fork 07CosmicAlpha 03ad4bc00 Switch LdapAuthentication to use our fork [00:00:33] [02mediawiki-repos] 07Universal-Omega opened pull request #105: Switch LdapAuthentication to use our fork (03main...03LdapAuthentication-fork) 13https://github.com/miraheze/mediawiki-repos/pull/105 [00:00:39] [02mediawiki-repos] 07coderabbitai[bot] commented on pull request #105:
[…] 13https://github.com/miraheze/mediawiki-repos/pull/105#issuecomment-3762256588 [00:01:26] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/aab6dd5e1a6b9fbbab7e96be5e73c0c40665ceb6 [00:01:27] 02statichelp/03main 07WikiTideBot 03aab6dd5 Bot: Auto-update Tech namespace pages 2026-01-17 00:01:23 [00:02:08] [02mediawiki-repos] 07Universal-Omega merged pull request #105: Switch LdapAuthentication to use our fork (03main...03LdapAuthentication-fork) 13https://github.com/miraheze/mediawiki-repos/pull/105 [00:02:08] [02mediawiki-repos] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mediawiki-repos/commit/2a3bf0fe6b0502e804aac3298c19b596648aec84 [00:02:08] 02mediawiki-repos/03main 07CosmicAlpha 032a3bf0f Switch LdapAuthentication to use our fork (#105) [00:02:09] [02mediawiki-repos] 07Universal-Omega 04deleted 03LdapAuthentication-fork at 03ad4bc00 13https://api.github.com/repos/miraheze/mediawiki-repos/commit/ad4bc00 [00:03:11] miraheze/ManageWiki - xaoseric the build has errored. [00:09:55] !log [universalomega@mwtask181] starting deploy of {'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'LdapAuthentication'} to all [00:09:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:10:24] !log [universalomega@mwtask181] finished deploy of {'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'LdapAuthentication'} to all - SUCCESS in 29s [00:10:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:13:04] !log [universalomega@test151] starting deploy of {'force_upgrade': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'LdapAuthentication'} to test151 [00:13:05] !log [universalomega@test151] finished deploy of {'force_upgrade': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'LdapAuthentication'} to test151 - SUCCESS in 1s [00:13:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:13:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:13:47] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 66680MiB (14.7% inode=100%); [00:16:11] [02ManageWiki] 07Universal-Omega retitled PR #758: "Postgres support" ➜ "Convert to abstract schema" 13https://github.com/miraheze/ManageWiki/pull/758 [00:17:20] [02CreateWiki] 07Universal-Omega merged 07dependabot[bot]'s pull request #796: Update mediawiki/mediawiki-codesniffer requirement from 48.0.0 to 49.0.0 (03main...03dependabot/composer/mediawiki/mediawiki-codesniffer-49.0.0) 13https://github.com/miraheze/CreateWiki/pull/796 [00:17:20] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/CreateWiki/commit/9c2e89b6382d22a17e41afe22057f6f9aa587677 [00:17:20] 02CreateWiki/03main 07dependabot[bot] 039c2e89b Update mediawiki/mediawiki-codesniffer requirement from 48.0.0 to 49.0.0 (#796)… [00:17:21] [02CreateWiki] 07Universal-Omega 04deleted 03dependabot/composer/mediawiki/mediawiki-codesniffer-49.0.0 at 03ff5e96c 13https://api.github.com/repos/miraheze/CreateWiki/commit/ff5e96c [00:17:33] [02CreateWiki] 07Universal-Omega merged 07dependabot[bot]'s pull request #797: Update mediawiki/minus-x requirement from 1.1.3 to 2.0.0 (03main...03dependabot/composer/mediawiki/minus-x-2.0.0) 13https://github.com/miraheze/CreateWiki/pull/797 [00:17:33] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/CreateWiki/commit/a9dfd093e7b969ad248ce8d2fdad77178090a9aa [00:17:34] 02CreateWiki/03main 07dependabot[bot] 03a9dfd09 Update mediawiki/minus-x requirement from 1.1.3 to 2.0.0 (#797)… [00:17:35] [02CreateWiki] 07Universal-Omega 04deleted 03dependabot/composer/mediawiki/minus-x-2.0.0 at 03be1ca4e 13https://api.github.com/repos/miraheze/CreateWiki/commit/be1ca4e [00:18:42] [02.github] 07Universal-Omega merged 07dependabot[bot]'s pull request #84: Bump actions/cache from 5.0.1 to 5.0.2 (03main...03dependabot/github_actions/actions/cache-5.0.2) 13https://github.com/miraheze/.github/pull/84 [00:18:42] [02.github] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/.github/commit/7b01d92fda252d50d0cca93c3957c22b88843154 [00:18:42] 02.github/03main 07dependabot[bot] 037b01d92 Bump actions/cache from 5.0.1 to 5.0.2 (#84)… [00:18:43] [02.github] 07Universal-Omega 04deleted 03dependabot/github_actions/actions/cache-5.0.2 at 03ae14e92 13https://api.github.com/repos/miraheze/.github/commit/ae14e92 [00:19:04] [02phorge-extensions] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/phorge-extensions/commit/fcabe39b287456792f51365241ee36c3b898fdba [00:19:04] 02phorge-extensions/03main 07dependabot[bot] 03fcabe39 Bump styfle/cancel-workflow-action from 0.12.1 to 0.13.0 (#65)… [00:19:04] [02phorge-extensions] 07Universal-Omega merged 07dependabot[bot]'s pull request #65: Bump styfle/cancel-workflow-action from 0.12.1 to 0.13.0 (03main...03dependabot/github_actions/styfle/cancel-workflow-action-0.13.0) 13https://github.com/miraheze/phorge-extensions/pull/65 [00:19:06] [02phorge-extensions] 07Universal-Omega 04deleted 03dependabot/github_actions/styfle/cancel-workflow-action-0.13.0 at 0364eb928 13https://api.github.com/repos/miraheze/phorge-extensions/commit/64eb928 [00:19:18] [02DiscordNotifications] 07Universal-Omega merged 07dependabot[bot]'s pull request #117: Update mediawiki/minus-x requirement from 1.1.3 to 2.0.0 (03main...03dependabot/composer/mediawiki/minus-x-2.0.0) 13https://github.com/miraheze/DiscordNotifications/pull/117 [00:19:18] [02DiscordNotifications] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/DiscordNotifications/commit/5d7a312066a8b0e08a7e0f2bd7a2a8df1b71c9fe [00:19:18] 02DiscordNotifications/03main 07dependabot[bot] 035d7a312 Update mediawiki/minus-x requirement from 1.1.3 to 2.0.0 (#117)… [00:19:19] [02DiscordNotifications] 07Universal-Omega 04deleted 03dependabot/composer/mediawiki/minus-x-2.0.0 at 03606a4a5 13https://api.github.com/repos/miraheze/DiscordNotifications/commit/606a4a5 [00:19:43] [02DiscordNotifications] 07Universal-Omega merged 07dependabot[bot]'s pull request #115: Update mediawiki/mediawiki-codesniffer requirement from 48.0.0 to 49.0.0 (03main...03dependabot/composer/mediawiki/mediawiki-codesniffer-49.0.0) 13https://github.com/miraheze/DiscordNotifications/pull/115 [00:19:43] [02DiscordNotifications] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/DiscordNotifications/commit/73ad99506ceb36034b425c8194d4cac578a0c46f [00:19:43] 02DiscordNotifications/03main 07dependabot[bot] 0373ad995 Update mediawiki/mediawiki-codesniffer requirement from 48.0.0 to 49.0.0 (#115)… [00:19:44] [02DiscordNotifications] 07Universal-Omega 04deleted 03dependabot/composer/mediawiki/mediawiki-codesniffer-49.0.0 at 039933301 13https://api.github.com/repos/miraheze/DiscordNotifications/commit/9933301 [00:19:47] miraheze/phorge-extensions - Universal-Omega the build passed. [00:20:56] [02IncidentReporting] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/IncidentReporting/commit/cf8d359afe20c971ccb23b6648e8a23e19d57525 [00:20:56] 02IncidentReporting/03main 07dependabot[bot] 03cf8d359 Update mediawiki/mediawiki-codesniffer requirement from 48.0.0 to 49.0.0 (#117)… [00:20:56] [02IncidentReporting] 07Universal-Omega merged 07dependabot[bot]'s pull request #117: Update mediawiki/mediawiki-codesniffer requirement from 48.0.0 to 49.0.0 (03main...03dependabot/composer/mediawiki/mediawiki-codesniffer-49.0.0) 13https://github.com/miraheze/IncidentReporting/pull/117 [00:20:57] [02IncidentReporting] 07Universal-Omega 04deleted 03dependabot/composer/mediawiki/mediawiki-codesniffer-49.0.0 at 03d8a868c 13https://api.github.com/repos/miraheze/IncidentReporting/commit/d8a868c [00:21:08] [02IncidentReporting] 07Universal-Omega merged 07dependabot[bot]'s pull request #116: Update mediawiki/minus-x requirement from 1.1.3 to 2.0.0 (03main...03dependabot/composer/mediawiki/minus-x-2.0.0) 13https://github.com/miraheze/IncidentReporting/pull/116 [00:21:09] [02IncidentReporting] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/IncidentReporting/commit/a19e8ca38da1797dff2e310cbbd46c035fc341ec [00:21:09] 02IncidentReporting/03main 07dependabot[bot] 03a19e8ca Update mediawiki/minus-x requirement from 1.1.3 to 2.0.0 (#116)… [00:21:10] [02IncidentReporting] 07Universal-Omega 04deleted 03dependabot/composer/mediawiki/minus-x-2.0.0 at 03129c83e 13https://api.github.com/repos/miraheze/IncidentReporting/commit/129c83e [00:21:31] [02WikiDiscover] 07Universal-Omega merged 07dependabot[bot]'s pull request #154: Update mediawiki/minus-x requirement from 1.1.3 to 2.0.0 (03main...03dependabot/composer/mediawiki/minus-x-2.0.0) 13https://github.com/miraheze/WikiDiscover/pull/154 [00:21:31] [02WikiDiscover] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/WikiDiscover/commit/86672192af50f786152b8d0a8bbfe7f237a4d493 [00:21:31] 02WikiDiscover/03main 07dependabot[bot] 038667219 Update mediawiki/minus-x requirement from 1.1.3 to 2.0.0 (#154)… [00:21:32] [02WikiDiscover] 07Universal-Omega 04deleted 03dependabot/composer/mediawiki/minus-x-2.0.0 at 0308a5a7f 13https://api.github.com/repos/miraheze/WikiDiscover/commit/08a5a7f [00:22:01] [02WikiDiscover] 07Universal-Omega merged 07dependabot[bot]'s pull request #153: Update mediawiki/mediawiki-codesniffer requirement from 48.0.0 to 49.0.0 (03main...03dependabot/composer/mediawiki/mediawiki-codesniffer-49.0.0) 13https://github.com/miraheze/WikiDiscover/pull/153 [00:22:01] [02WikiDiscover] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/WikiDiscover/commit/fbf29e2a75c6f4400d805a2c12e1e2941c9bf7cc [00:22:01] 02WikiDiscover/03main 07dependabot[bot] 03fbf29e2 Update mediawiki/mediawiki-codesniffer requirement from 48.0.0 to 49.0.0 (#153)… [00:22:03] [02WikiDiscover] 07Universal-Omega 04deleted 03dependabot/composer/mediawiki/mediawiki-codesniffer-49.0.0 at 0392ad1a2 13https://api.github.com/repos/miraheze/WikiDiscover/commit/92ad1a2 [00:22:23] [02DataDump] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/DataDump/commit/d381d33cdffd340e398dcef99f892bbfb9e1eb19 [00:22:23] 02DataDump/03main 07dependabot[bot] 03d381d33 Update mediawiki/mediawiki-codesniffer requirement from 48.0.0 to 49.0.0 (#116)… [00:22:23] [02DataDump] 07Universal-Omega merged 07dependabot[bot]'s pull request #116: Update mediawiki/mediawiki-codesniffer requirement from 48.0.0 to 49.0.0 (03main...03dependabot/composer/mediawiki/mediawiki-codesniffer-49.0.0) 13https://github.com/miraheze/DataDump/pull/116 [00:22:24] [02DataDump] 07Universal-Omega 04deleted 03dependabot/composer/mediawiki/mediawiki-codesniffer-49.0.0 at 034986dac 13https://api.github.com/repos/miraheze/DataDump/commit/4986dac [00:22:37] [02DataDump] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/DataDump/commit/767226f1bc014bded06542647060e0f8c87e38b6 [00:22:37] 02DataDump/03main 07dependabot[bot] 03767226f Update mediawiki/minus-x requirement from 1.1.3 to 2.0.0 (#115)… [00:22:37] [02DataDump] 07Universal-Omega merged 07dependabot[bot]'s pull request #115: Update mediawiki/minus-x requirement from 1.1.3 to 2.0.0 (03main...03dependabot/composer/mediawiki/minus-x-2.0.0) 13https://github.com/miraheze/DataDump/pull/115 [00:22:38] [02DataDump] 07Universal-Omega 04deleted 03dependabot/composer/mediawiki/minus-x-2.0.0 at 03172324e 13https://api.github.com/repos/miraheze/DataDump/commit/172324e [00:23:09] [02MirahezeMagic] 07Universal-Omega merged 07dependabot[bot]'s pull request #617: Update mediawiki/minus-x requirement from 1.1.3 to 2.0.0 (03main...03dependabot/composer/mediawiki/minus-x-2.0.0) 13https://github.com/miraheze/MirahezeMagic/pull/617 [00:23:10] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/77210cf1422b4b499f276977fc7ba854260fe357 [00:23:10] 02MirahezeMagic/03main 07dependabot[bot] 0377210cf Update mediawiki/minus-x requirement from 1.1.3 to 2.0.0 (#617)… [00:23:11] [02MirahezeMagic] 07Universal-Omega 04deleted 03dependabot/composer/mediawiki/minus-x-2.0.0 at 03c4d1239 13https://api.github.com/repos/miraheze/MirahezeMagic/commit/c4d1239 [00:23:22] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/8a55252636a9cc1cd92836f744ab9e07909ccc4d [00:23:22] 02MirahezeMagic/03main 07dependabot[bot] 038a55252 Update mediawiki/mediawiki-codesniffer requirement from 48.0.0 to 49.0.0 (#618)… [00:23:22] [02MirahezeMagic] 07Universal-Omega merged 07dependabot[bot]'s pull request #618: Update mediawiki/mediawiki-codesniffer requirement from 48.0.0 to 49.0.0 (03main...03dependabot/composer/mediawiki/mediawiki-codesniffer-49.0.0) 13https://github.com/miraheze/MirahezeMagic/pull/618 [00:23:22] [02MirahezeMagic] 07Universal-Omega 04deleted 03dependabot/composer/mediawiki/mediawiki-codesniffer-49.0.0 at 03ebc2cdd 13https://api.github.com/repos/miraheze/MirahezeMagic/commit/ebc2cdd [00:23:41] [02RottenLinks] 07Universal-Omega merged 07dependabot[bot]'s pull request #99: Update mediawiki/mediawiki-codesniffer requirement from 48.0.0 to 49.0.0 (03main...03dependabot/composer/mediawiki/mediawiki-codesniffer-49.0.0) 13https://github.com/miraheze/RottenLinks/pull/99 [00:23:41] [02RottenLinks] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/RottenLinks/commit/855d51cc7356c0af3e9df9b5e2ca4165a888fcf1 [00:23:41] 02RottenLinks/03main 07dependabot[bot] 03855d51c Update mediawiki/mediawiki-codesniffer requirement from 48.0.0 to 49.0.0 (#99)… [00:23:45] [02RottenLinks] 07Universal-Omega 04deleted 03dependabot/composer/mediawiki/mediawiki-codesniffer-49.0.0 at 039bee72d 13https://api.github.com/repos/miraheze/RottenLinks/commit/9bee72d [00:23:57] [02RottenLinks] 07Universal-Omega merged 07dependabot[bot]'s pull request #98: Update mediawiki/minus-x requirement from 1.1.3 to 2.0.0 (03main...03dependabot/composer/mediawiki/minus-x-2.0.0) 13https://github.com/miraheze/RottenLinks/pull/98 [00:23:57] [02RottenLinks] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/RottenLinks/commit/a90e7aac2adb2f025c8331c686860b7318aef7dd [00:23:57] 02RottenLinks/03main 07dependabot[bot] 03a90e7aa Update mediawiki/minus-x requirement from 1.1.3 to 2.0.0 (#98)… [00:24:00] [02RottenLinks] 07Universal-Omega 04deleted 03dependabot/composer/mediawiki/minus-x-2.0.0 at 03ccab00e 13https://api.github.com/repos/miraheze/RottenLinks/commit/ccab00e [00:24:15] [02ManageWiki] 07Universal-Omega merged 07dependabot[bot]'s pull request #757: Update mediawiki/minus-x requirement from 1.1.3 to 2.0.0 (03main...03dependabot/composer/mediawiki/minus-x-2.0.0) 13https://github.com/miraheze/ManageWiki/pull/757 [00:24:15] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/ManageWiki/commit/baf9d54fa4ac34c50fc370f9b530c0a68be9acde [00:24:15] 02ManageWiki/03main 07dependabot[bot] 03baf9d54 Update mediawiki/minus-x requirement from 1.1.3 to 2.0.0 (#757)… [00:24:16] [02ManageWiki] 07Universal-Omega 04deleted 03dependabot/composer/mediawiki/minus-x-2.0.0 at 03214f8e8 13https://api.github.com/repos/miraheze/ManageWiki/commit/214f8e8 [00:24:28] [02ManageWiki] 07Universal-Omega merged 07dependabot[bot]'s pull request #756: Update mediawiki/mediawiki-codesniffer requirement from 48.0.0 to 49.0.0 (03main...03dependabot/composer/mediawiki/mediawiki-codesniffer-49.0.0) 13https://github.com/miraheze/ManageWiki/pull/756 [00:24:28] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/ManageWiki/commit/65653d83f1e84fa6c13884b57bd918767ba195df [00:24:28] 02ManageWiki/03main 07dependabot[bot] 0365653d8 Update mediawiki/mediawiki-codesniffer requirement from 48.0.0 to 49.0.0 (#756)… [00:24:30] [02ManageWiki] 07Universal-Omega 04deleted 03dependabot/composer/mediawiki/mediawiki-codesniffer-49.0.0 at 0355ad5b7 13https://api.github.com/repos/miraheze/ManageWiki/commit/55ad5b7 [00:24:47] [02MatomoAnalytics] 07Universal-Omega merged 07dependabot[bot]'s pull request #180: Update mediawiki/minus-x requirement from 1.1.3 to 2.0.0 (03main...03dependabot/composer/mediawiki/minus-x-2.0.0) 13https://github.com/miraheze/MatomoAnalytics/pull/180 [00:24:47] [02MatomoAnalytics] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/MatomoAnalytics/commit/960ce9e3d47e8cc971512dabfd93be3ae117086f [00:24:48] 02MatomoAnalytics/03main 07dependabot[bot] 03960ce9e Update mediawiki/minus-x requirement from 1.1.3 to 2.0.0 (#180)… [00:24:49] [02MatomoAnalytics] 07Universal-Omega 04deleted 03dependabot/composer/mediawiki/minus-x-2.0.0 at 03014a991 13https://api.github.com/repos/miraheze/MatomoAnalytics/commit/014a991 [00:25:02] [02MatomoAnalytics] 07Universal-Omega merged 07dependabot[bot]'s pull request #179: Update mediawiki/mediawiki-codesniffer requirement from 48.0.0 to 49.0.0 (03main...03dependabot/composer/mediawiki/mediawiki-codesniffer-49.0.0) 13https://github.com/miraheze/MatomoAnalytics/pull/179 [00:25:02] [02MatomoAnalytics] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/MatomoAnalytics/commit/acb04ed338fa592537e7741eed82e0f0fb67d0ba [00:25:02] 02MatomoAnalytics/03main 07dependabot[bot] 03acb04ed Update mediawiki/mediawiki-codesniffer requirement from 48.0.0 to 49.0.0 (#179)… [00:25:03] [02MatomoAnalytics] 07Universal-Omega 04deleted 03dependabot/composer/mediawiki/mediawiki-codesniffer-49.0.0 at 03d81ca0a 13https://api.github.com/repos/miraheze/MatomoAnalytics/commit/d81ca0a [00:25:13] [02GlobalNewFiles] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/GlobalNewFiles/commit/1b091293eb9c0b3d29e54882e82d17fe2671b3fb [00:25:13] 02GlobalNewFiles/03main 07dependabot[bot] 031b09129 Update mediawiki/mediawiki-codesniffer requirement from 48.0.0 to 49.0.0 (#108)… [00:25:14] [02GlobalNewFiles] 07Universal-Omega merged 07dependabot[bot]'s pull request #108: Update mediawiki/mediawiki-codesniffer requirement from 48.0.0 to 49.0.0 (03main...03dependabot/composer/mediawiki/mediawiki-codesniffer-49.0.0) 13https://github.com/miraheze/GlobalNewFiles/pull/108 [00:25:15] [02GlobalNewFiles] 07Universal-Omega 04deleted 03dependabot/composer/mediawiki/mediawiki-codesniffer-49.0.0 at 036220bad 13https://api.github.com/repos/miraheze/GlobalNewFiles/commit/6220bad [00:25:26] [02GlobalNewFiles] 07Universal-Omega merged 07dependabot[bot]'s pull request #107: Update mediawiki/minus-x requirement from 1.1.3 to 2.0.0 (03main...03dependabot/composer/mediawiki/minus-x-2.0.0) 13https://github.com/miraheze/GlobalNewFiles/pull/107 [00:25:26] [02GlobalNewFiles] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/GlobalNewFiles/commit/915d4b8cc657499ea91efe86f1fec817e5deae18 [00:25:26] 02GlobalNewFiles/03main 07dependabot[bot] 03915d4b8 Update mediawiki/minus-x requirement from 1.1.3 to 2.0.0 (#107)… [00:25:27] [02GlobalNewFiles] 07Universal-Omega 04deleted 03dependabot/composer/mediawiki/minus-x-2.0.0 at 03961c49f 13https://api.github.com/repos/miraheze/GlobalNewFiles/commit/961c49f [00:25:44] [02PDFEmbed] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/PDFEmbed/commit/42c7c177013e3a16b91b2ac6821af093d9be305d [00:25:44] 02PDFEmbed/03main 07dependabot[bot] 0342c7c17 Update mediawiki/minus-x requirement from 1.1.3 to 2.0.0 (#34)… [00:25:44] [02PDFEmbed] 07Universal-Omega merged 07dependabot[bot]'s pull request #34: Update mediawiki/minus-x requirement from 1.1.3 to 2.0.0 (03main...03dependabot/composer/mediawiki/minus-x-2.0.0) 13https://github.com/miraheze/PDFEmbed/pull/34 [00:25:47] [02PDFEmbed] 07Universal-Omega 04deleted 03dependabot/composer/mediawiki/minus-x-2.0.0 at 03bb542a6 13https://api.github.com/repos/miraheze/PDFEmbed/commit/bb542a6 [00:25:55] [02PDFEmbed] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/PDFEmbed/commit/b56394aff6fbcda39ba21174c1942013de94e125 [00:25:56] 02PDFEmbed/03main 07dependabot[bot] 03b56394a Update mediawiki/mediawiki-codesniffer requirement from 48.0.0 to 49.0.0 (#33)… [00:25:56] miraheze/IncidentReporting - Universal-Omega the build passed. [00:25:57] [02PDFEmbed] 07Universal-Omega merged 07dependabot[bot]'s pull request #33: Update mediawiki/mediawiki-codesniffer requirement from 48.0.0 to 49.0.0 (03main...03dependabot/composer/mediawiki/mediawiki-codesniffer-49.0.0) 13https://github.com/miraheze/PDFEmbed/pull/33 [00:25:57] [02PDFEmbed] 07Universal-Omega 04deleted 03dependabot/composer/mediawiki/mediawiki-codesniffer-49.0.0 at 030775626 13https://api.github.com/repos/miraheze/PDFEmbed/commit/0775626 [00:26:10] [02ImportDump] 07Universal-Omega merged 07dependabot[bot]'s pull request #163: Update mediawiki/minus-x requirement from 1.1.3 to 2.0.0 (03main...03dependabot/composer/mediawiki/minus-x-2.0.0) 13https://github.com/miraheze/ImportDump/pull/163 [00:26:10] [02ImportDump] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/ImportDump/commit/6cfd5a27fc0f9595060a1e2bbec01a4103df2e2e [00:26:10] 02ImportDump/03main 07dependabot[bot] 036cfd5a2 Update mediawiki/minus-x requirement from 1.1.3 to 2.0.0 (#163)… [00:26:11] [02ImportDump] 07Universal-Omega 04deleted 03dependabot/composer/mediawiki/minus-x-2.0.0 at 0358ab0c4 13https://api.github.com/repos/miraheze/ImportDump/commit/58ab0c4 [00:26:22] [02ImportDump] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/ImportDump/commit/35bf63cd0390ecda8a791a2416a4d883e4dfa05d [00:26:22] 02ImportDump/03main 07dependabot[bot] 0335bf63c Update mediawiki/mediawiki-codesniffer requirement from 48.0.0 to 49.0.0 (#162)… [00:26:22] [02ImportDump] 07Universal-Omega merged 07dependabot[bot]'s pull request #162: Update mediawiki/mediawiki-codesniffer requirement from 48.0.0 to 49.0.0 (03main...03dependabot/composer/mediawiki/mediawiki-codesniffer-49.0.0) 13https://github.com/miraheze/ImportDump/pull/162 [00:26:23] [02ImportDump] 07Universal-Omega 04deleted 03dependabot/composer/mediawiki/mediawiki-codesniffer-49.0.0 at 0305b05a8 13https://api.github.com/repos/miraheze/ImportDump/commit/05b05a8 [00:26:26] miraheze/CreateWiki - Universal-Omega the build passed. [00:26:37] [02RemovePII] 07Universal-Omega merged 07dependabot[bot]'s pull request #118: Update mediawiki/mediawiki-codesniffer requirement from 48.0.0 to 49.0.0 (03main...03dependabot/composer/mediawiki/mediawiki-codesniffer-49.0.0) 13https://github.com/miraheze/RemovePII/pull/118 [00:26:38] [02RemovePII] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/RemovePII/commit/a4f4994be7f617ff1464a54e3f2b998525699c38 [00:26:38] 02RemovePII/03main 07dependabot[bot] 03a4f4994 Update mediawiki/mediawiki-codesniffer requirement from 48.0.0 to 49.0.0 (#118)… [00:26:39] [02RemovePII] 07Universal-Omega 04deleted 03dependabot/composer/mediawiki/mediawiki-codesniffer-49.0.0 at 039ea7bab 13https://api.github.com/repos/miraheze/RemovePII/commit/9ea7bab [00:26:52] [02RemovePII] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/RemovePII/commit/430123fe9f73ef33747c919e47d731abde1b82c1 [00:26:53] 02RemovePII/03main 07dependabot[bot] 03430123f Update mediawiki/minus-x requirement from 1.1.3 to 2.0.0 (#117)… [00:26:53] [02RemovePII] 07Universal-Omega merged 07dependabot[bot]'s pull request #117: Update mediawiki/minus-x requirement from 1.1.3 to 2.0.0 (03main...03dependabot/composer/mediawiki/minus-x-2.0.0) 13https://github.com/miraheze/RemovePII/pull/117 [00:26:59] [02RemovePII] 07Universal-Omega 04deleted 03dependabot/composer/mediawiki/minus-x-2.0.0 at 0362491e1 13https://api.github.com/repos/miraheze/RemovePII/commit/62491e1 [00:27:41] miraheze/DiscordNotifications - Universal-Omega the build passed. [00:28:28] miraheze/MirahezeMagic - Universal-Omega the build has errored. [00:29:37] miraheze/WikiDiscover - Universal-Omega the build passed. [00:31:45] miraheze/MirahezeMagic - Universal-Omega the build has errored. [00:31:51] miraheze/DataDump - Universal-Omega the build passed. [00:31:58] miraheze/DiscordNotifications - Universal-Omega the build passed. [00:32:49] miraheze/RottenLinks - Universal-Omega the build passed. [00:35:00] miraheze/MatomoAnalytics - Universal-Omega the build passed. [00:36:28] miraheze/PDFEmbed - Universal-Omega the build passed. [00:37:23] miraheze/GlobalNewFiles - Universal-Omega the build passed. [00:37:33] miraheze/ManageWiki - Universal-Omega the build passed. [00:38:33] miraheze/RemovePII - Universal-Omega the build passed. [00:38:52] miraheze/ImportDump - Universal-Omega the build passed. [00:51:50] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 65584MiB (14.4% inode=100%); [00:54:09] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 72425MiB (16.0% inode=100%); [01:02:18] PROBLEM - matomo151 Check unit status of matomo-archiver-3 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-3 [01:03:27] PROBLEM - matomo151 Check unit status of matomo-archiver-1 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-1 [01:03:35] PROBLEM - matomo151 Check unit status of matomo-archiver-2 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-2 [01:04:24] !log [petramagna@test151] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php updateSpecialPages --wiki=exttestwikibeta --only=UnusedRedirects (END - exit=0) [01:04:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:09:34] [02.github] 07Universal-Omega created 03coverage (+1 new commit) 13https://github.com/miraheze/.github/commit/20ad71d181af [01:09:34] 02.github/03coverage 07CosmicAlpha 0320ad71d Fix coverage [01:09:41] [02.github] 07Universal-Omega opened pull request #85: Fix coverage (03main...03coverage) 13https://github.com/miraheze/.github/pull/85 [01:09:46] [02.github] 07coderabbitai[bot] commented on pull request #85:
[…] 13https://github.com/miraheze/.github/pull/85#issuecomment-3762435065 [01:10:31] [02CreateWiki] 07Universal-Omega created 03coverage (+1 new commit) 13https://github.com/miraheze/CreateWiki/commit/5babab012e06 [01:10:32] 02CreateWiki/03coverage 07CosmicAlpha 035babab0 Coverage test [01:10:37] [02CreateWiki] 07Universal-Omega opened pull request #798: Coverage test (03main...03coverage) 13https://github.com/miraheze/CreateWiki/pull/798 [01:10:42] [02CreateWiki] 07coderabbitai[bot] commented on pull request #798:
[…] 13https://github.com/miraheze/CreateWiki/pull/798#issuecomment-3762436197 [01:16:03] [02.github] 07Universal-Omega pushed 1 new commit to 03coverage 13https://github.com/miraheze/.github/commit/056cfc628cdd1d1adc85c7eab859884ed3a21b7e [01:16:03] 02.github/03coverage 07CosmicAlpha 03056cfc6 Try [01:19:04] miraheze/CreateWiki - Universal-Omega the build passed. [01:21:13] !log [petramagna@test151] starting deploy of {'versions': '1.45', 'upgrade_extensions': 'GlobalUsage'} to test151 [01:21:14] !log [petramagna@test151] finished deploy of {'versions': '1.45', 'upgrade_extensions': 'GlobalUsage'} to test151 - SUCCESS in 1s [01:21:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:21:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:21:35] [02.github] 07Universal-Omega created 03REL1_45-CI (+1 new commit) 13https://github.com/miraheze/.github/commit/967a0e891c0c [01:21:35] 02.github/03REL1_45-CI 07CosmicAlpha 03967a0e8 Add REL1_45 to CI [01:21:39] [02.github] 07Universal-Omega opened pull request #86: Add REL1_45 to CI (03main...03REL1_45-CI) 13https://github.com/miraheze/.github/pull/86 [01:21:45] [02.github] 07coderabbitai[bot] commented on pull request #86:
[…] 13https://github.com/miraheze/.github/pull/86#issuecomment-3762452613 [01:24:21] [02CreateWiki] 07codecov[bot] commented on pull request #798: :white_check_mark: All modified and coverable lines are covered by tests. […] 13https://github.com/miraheze/CreateWiki/pull/798#issuecomment-3762455501 [01:25:37] [02CreateWiki] 07Universal-Omega closed pull request #798: Coverage test (03main...03coverage) 13https://github.com/miraheze/CreateWiki/pull/798 [01:25:41] [02CreateWiki] 07Universal-Omega 04deleted 03coverage at 035babab0 13https://api.github.com/repos/miraheze/CreateWiki/commit/5babab0 [01:25:49] [02.github] 07Universal-Omega merged pull request #85: Fix coverage (03main...03coverage) 13https://github.com/miraheze/.github/pull/85 [01:25:49] [02.github] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/.github/commit/986c95e95f5ba78a601b4236e488e4351c8ae9fe [01:25:49] 02.github/03main 07CosmicAlpha 03986c95e Fix coverage [01:25:50] [02.github] 07Universal-Omega 04deleted 03coverage at 03056cfc6 13https://api.github.com/repos/miraheze/.github/commit/056cfc6 [01:26:46] [02.github] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/.github/commit/2032f9e45f182e062fcc7ada61537a061fa9c8de [01:26:47] 02.github/03main 07CosmicAlpha 032032f9e Add REL1_45 to CI [01:26:47] [02.github] 07Universal-Omega merged pull request #86: Add REL1_45 to CI (03main...03REL1_45-CI) 13https://github.com/miraheze/.github/pull/86 [01:26:48] [02.github] 07Universal-Omega 04deleted 03REL1_45-CI at 03967a0e8 13https://api.github.com/repos/miraheze/.github/commit/967a0e8 [01:26:59] [02GlobalNewFiles] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/GlobalNewFiles/commit/cedc600163411ffb0d184c0d9bf0cb065b93e5d9 [01:26:59] 02GlobalNewFiles/03main 07Universal-Omega 03cedc600 Update workflow: Add REL1_45 to CI… [01:27:00] [02IncidentReporting] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/IncidentReporting/commit/1036c362dd8b7797b3224d3bb1bcc9a4d6781845 [01:27:00] 02IncidentReporting/03main 07Universal-Omega 031036c36 Update workflow: Add REL1_45 to CI… [01:27:02] [02DataDump] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/DataDump/commit/61a56b3cc2df1a3e29e622bcc5e55f13c41d0fa1 [01:27:04] [02MatomoAnalytics] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/MatomoAnalytics/commit/25cce149b38b787382039ae661dd10eb584072b0 [01:27:06] [02RequestCustomDomain] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/RequestCustomDomain/commit/a2d46f68085c3ee2b73c00b5878e2691c59aeb04 [01:27:08] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/CreateWiki/commit/88f49b8b189f8c0e4ede77c978ea98089d6525ec [01:27:10] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/ManageWiki/commit/7dc75f5765503ebd6d79aadee78f78d9d572b1c5 [01:27:12] [02ImportDump] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/ImportDump/commit/c70af4c1c84d8e1fbf3644f8e04559ac99dbf997 [01:27:14] [02DiscordNotifications] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/DiscordNotifications/commit/dbaaabbe7695640092fdc1a3f88350ec52d852d9 [01:27:15] 02DataDump/03main 07Universal-Omega 0361a56b3 Update workflow: Add REL1_45 to CI… [01:27:17] [02RemovePII] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/RemovePII/commit/2558c038278b6a0513a77d5e7e10647f4c7e6832 [01:27:19] [02RottenLinks] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/RottenLinks/commit/6c744232b6a897b87032a707cede845a6c892610 [01:27:20] 02MatomoAnalytics/03main 07Universal-Omega 0325cce14 Update workflow: Add REL1_45 to CI… [01:27:22] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/1c65bc9204830135270531a27629ea5c697f8ffa [01:27:24] [02WikiDiscover] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/WikiDiscover/commit/e5a60314179922ec2231039b65eb0f39d8494ddd [01:27:26] [02PDFEmbed] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/PDFEmbed/commit/ec147e0c6d8784a35402a8c13a69009b69017844 [01:27:28] 02RequestCustomDomain/03main 07Universal-Omega 03a2d46f6 Update workflow: Add REL1_45 to CI… [01:27:29] 02CreateWiki/03main 07Universal-Omega 0388f49b8 Update workflow: Add REL1_45 to CI… [01:27:30] 02ManageWiki/03main 07Universal-Omega 037dc75f5 Update workflow: Add REL1_45 to CI… [01:27:31] 02ImportDump/03main 07Universal-Omega 03c70af4c Update workflow: Add REL1_45 to CI… [01:27:33] 02DiscordNotifications/03main 07Universal-Omega 03dbaaabb Update workflow: Add REL1_45 to CI… [01:27:34] 02RemovePII/03main 07Universal-Omega 032558c03 Update workflow: Add REL1_45 to CI… [01:27:35] 02RottenLinks/03main 07Universal-Omega 036c74423 Update workflow: Add REL1_45 to CI… [01:27:37] 02MirahezeMagic/03main 07Universal-Omega 031c65bc9 Update workflow: Add REL1_45 to CI… [01:27:38] 02WikiDiscover/03main 07Universal-Omega 03e5a6031 Update workflow: Add REL1_45 to CI… [01:27:39] 02PDFEmbed/03main 07Universal-Omega 03ec147e0 Update workflow: Add REL1_45 to CI… [01:31:28] miraheze/CreateWiki - Universal-Omega the build passed. [01:31:57] miraheze/DataDump - Universal-Omega the build passed. [01:32:06] miraheze/GlobalNewFiles - Universal-Omega the build passed. [01:32:37] miraheze/IncidentReporting - Universal-Omega the build passed. [01:34:49] miraheze/MatomoAnalytics - Universal-Omega the build passed. [01:35:08] miraheze/CreateWiki - Universal-Omega the build passed. [01:35:17] miraheze/MirahezeMagic - Universal-Omega the build has errored. [01:35:19] miraheze/DiscordNotifications - Universal-Omega the build passed. [01:35:53] miraheze/ManageWiki - Universal-Omega the build passed. [01:36:39] miraheze/RequestCustomDomain - Universal-Omega the build passed. [01:37:11] miraheze/ImportDump - Universal-Omega the build passed. [01:37:15] miraheze/PDFEmbed - Universal-Omega the build passed. [01:37:22] miraheze/RemovePII - Universal-Omega the build passed. [01:37:32] miraheze/RottenLinks - Universal-Omega the build passed. [01:38:51] miraheze/WikiDiscover - Universal-Omega the build passed. [01:43:50] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/963676f2f0b28065b108328d42c598dd78293cd7 [01:43:50] 02mw-config/03main 07CosmicAlpha 03963676f Set GlobalUsage virtual domain [01:44:40] miraheze/mw-config - Universal-Omega the build passed. [01:45:40] [02mw-config] 07Universal-Omega pushed 1 new commit to 03sul3-support 13https://github.com/miraheze/mw-config/commit/6c1132b643ea2a3e6b030cb9caa9af3c8c00a2a8 [01:45:40] 02mw-config/03sul3-support 07CosmicAlpha 036c1132b Merge branch 'main' into sul3-support [01:45:47] !log [universalomega@test151] starting deploy of {'pull': 'config', 'branch': 'sul3-support', 'config': True} to test151 [01:45:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:45:54] !log [universalomega@test151] finished deploy of {'pull': 'config', 'branch': 'sul3-support', 'config': True} to test151 - SUCCESS in 6s [01:45:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:46:37] miraheze/mw-config - Universal-Omega the build passed. [02:20:34] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.81, 18.32, 11.61 [02:22:34] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 17.50, 18.46, 12.53 [03:00:07] !log [petramagna@test151] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/extensions/GlobalUsage/maintenance/refreshGlobalimagelinks.php --wiki=exttestwikibeta --pages=existing,nonexisting (END - exit=0) [03:00:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:57:07] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 40% [06:59:07] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 33% [08:01:27] RECOVERY - matomo151 Check unit status of matomo-archiver-1 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-1 [08:01:35] RECOVERY - matomo151 Check unit status of matomo-archiver-2 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-2 [08:02:18] RECOVERY - matomo151 Check unit status of matomo-archiver-3 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-3 [09:41:00] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:41:01] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:41:08] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [09:41:10] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:41:12] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:41:12] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:41:12] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:41:13] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:41:14] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [09:41:14] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:41:17] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [09:41:23] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:41:26] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [09:41:26] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 14 backends are down. mw151 mw152 mw161 mw162 mw171 mw182 mw153 mw173 mw183 mw191 mw192 mw193 mw202 mw203 [09:41:27] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:41:28] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 14 backends are down. mw151 mw152 mw161 mw162 mw171 mw182 mw153 mw173 mw183 mw191 mw192 mw193 mw202 mw203 [09:41:31] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [09:41:33] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [09:41:34] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:41:35] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:41:36] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [09:41:37] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:41:39] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [09:41:40] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 146.27, 69.49, 27.80 [09:41:47] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:41:47] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:41:54] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [09:41:58] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [09:41:58] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [09:41:59] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:41:59] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [09:42:03] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 14 backends are down. mw151 mw152 mw161 mw162 mw171 mw182 mw153 mw173 mw183 mw191 mw192 mw193 mw202 mw203 [09:42:08] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:42:13] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:42:16] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [09:42:19] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:42:21] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:42:29] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [09:42:29] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:42:36] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:42:38] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:42:42] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 14 backends are down. mw151 mw152 mw161 mw162 mw171 mw182 mw153 mw173 mw183 mw191 mw192 mw193 mw202 mw203 [09:42:43] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:42:47] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [09:42:49] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:43:07] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 70% [09:45:07] PROBLEM - db161 MariaDB Connections on db161 is UNKNOWN: [09:45:22] PROBLEM - db161 APT on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:45:33] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.701 second response time [09:45:34] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.065 second response time [09:45:34] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 1.702 second response time [09:45:36] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.323 second response time [09:45:40] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.230 second response time [09:45:40] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.230 second response time [09:45:41] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.063 second response time [09:45:41] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/rebuildall.php --wiki=shintowiki (END - exit=256) [09:45:42] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.056 second response time [09:45:43] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/initSiteStats.php --wiki=shintowiki --update (END - exit=256) [09:45:46] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.229 second response time [09:45:49] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4303 bytes in 2.357 second response time [09:45:55] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.063 second response time [09:45:57] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.016 second response time [09:45:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:46:02] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.660 second response time [09:46:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:46:06] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.225 second response time [09:46:06] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.185 second response time [09:46:08] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.069 second response time [09:46:08] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.064 second response time [09:46:14] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.064 second response time [09:46:17] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.207 second response time [09:46:19] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.051 second response time [09:46:27] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.171 second response time [09:46:28] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.053 second response time [09:46:29] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.185 second response time [09:46:37] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.181 second response time [09:46:38] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4357 bytes in 0.069 second response time [09:46:42] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [09:46:45] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.213 second response time [09:46:48] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.057 second response time [09:46:53] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.170 second response time [09:46:55] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4303 bytes in 0.065 second response time [09:47:06] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.059 second response time [09:47:07] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4303 bytes in 0.071 second response time [09:47:07] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 58% [09:47:14] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.177 second response time [09:47:18] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.199 second response time [09:47:18] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.070 second response time [09:47:19] PROBLEM - db161 Puppet on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:47:20] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.056 second response time [09:47:23] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.058 second response time [09:47:26] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.048 second response time [09:47:26] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [09:47:26] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.187 second response time [09:47:28] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [09:47:30] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.067 second response time [09:47:35] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.057 second response time [09:47:59] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [09:48:03] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [09:49:07] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 38% [09:49:18] RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 35 minutes ago with 0 failures [09:49:39] PROBLEM - db161 MariaDB on db161 is CRITICAL: Can't connect to server on 'db161.fsslc.wtnet' (115) [09:49:40] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 0.83, 0.27, 0.10 [09:50:10] RECOVERY - db161 MariaDB Connections on db161 is OK: OK connection usage: 45%Current connections: 450 [09:51:39] RECOVERY - db161 MariaDB on db161 is OK: Uptime: 178 Threads: 457 Questions: 2366 Slow queries: 6 Opens: 483 Open tables: 476 Queries per second avg: 13.292 [09:51:57] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [09:52:03] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:52:04] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [09:52:06] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:52:07] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [09:52:08] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:52:09] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [09:52:10] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:52:18] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [09:52:19] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:52:20] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [09:52:25] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [09:52:26] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [09:52:36] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [09:52:37] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [09:52:37] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:52:38] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:52:42] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 14 backends are down. mw151 mw152 mw161 mw162 mw171 mw182 mw153 mw173 mw183 mw191 mw192 mw193 mw202 mw203 [09:52:51] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [09:52:55] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:53:07] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 70% [09:53:14] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [09:53:16] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [09:53:17] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [09:53:17] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [09:53:21] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [09:53:23] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [09:53:26] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [09:53:26] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 14 backends are down. mw151 mw152 mw161 mw162 mw171 mw182 mw153 mw173 mw183 mw191 mw192 mw193 mw202 mw203 [09:53:26] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [09:53:28] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 14 backends are down. mw151 mw152 mw161 mw162 mw171 mw182 mw153 mw173 mw183 mw191 mw192 mw193 mw202 mw203 [09:53:35] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:53:36] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:53:37] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [09:53:38] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [09:53:39] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [09:53:41] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:53:43] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [09:53:43] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 616.01, 287.67, 114.85 [09:53:44] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:53:46] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:53:47] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:53:47] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [09:53:49] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:53:56] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:54:03] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 14 backends are down. mw151 mw152 mw161 mw162 mw171 mw182 mw153 mw173 mw183 mw191 mw192 mw193 mw202 mw203 [09:54:10] <.seablock.> Wha [09:55:18] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.056 second response time [09:55:20] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.056 second response time [09:55:22] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.059 second response time [09:55:22] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.071 second response time [09:55:32] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.202 second response time [09:55:34] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.068 second response time [09:55:36] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.190 second response time [09:55:38] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.067 second response time [09:55:39] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.061 second response time [09:55:40] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.202 second response time [09:55:41] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.059 second response time [09:55:42] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.049 second response time [09:55:44] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.206 second response time [09:55:46] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.172 second response time [09:55:53] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.190 second response time [09:56:11] PROBLEM - db161 Puppet on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:56:13] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 2.630 second response time [09:56:15] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.272 second response time [09:56:17] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.063 second response time [09:56:20] !log [somerandomdeveloper@mwtask181] starting deploy of {'config': True} to all [09:56:21] !log [somerandomdeveloper@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@localhost [09:56:23] !log [somerandomdeveloper@mwtask181] starting deploy of {'config': True, 'force': True} to all [09:56:27] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.176 second response time [09:56:29] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.043 second response time [09:56:31] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.625 second response time [09:56:32] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.210 second response time [09:56:36] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.438 second response time [09:56:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:56:43] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.050 second response time [09:56:44] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.174 second response time [09:56:55] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4303 bytes in 0.069 second response time [09:56:55] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.241 second response time [09:56:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:57:06] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.057 second response time [09:57:07] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4303 bytes in 0.070 second response time [09:57:07] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 53% [09:57:18] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.839 second response time [09:57:20] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.185 second response time [09:57:23] !log [somerandomdeveloper@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 59s [09:57:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:57:35] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.061 second response time [09:57:50] RECOVERY - db161 APT on db161 is OK: APT OK: 143 packages available for upgrade (0 critical updates). [09:57:55] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4303 bytes in 0.071 second response time [09:57:58] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.210 second response time [09:58:03] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [09:58:10] <.seablock.> What is this 😭 [09:58:14] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.070 second response time [09:58:16] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.209 second response time [09:58:17] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.244 second response time [09:58:18] An outage [09:58:19] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.061 second response time [09:58:19] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.065 second response time [09:58:27] RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 44 minutes ago with 0 failures [09:58:42] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [09:58:45] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4357 bytes in 0.072 second response time [09:58:47] !log depooled c2 [09:58:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:59:07] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 38% [09:59:26] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [09:59:28] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [10:01:31] !log reboot db161 via proxmox due to high load [10:01:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:01:59] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 0.37, 0.08, 0.03 [10:03:07] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 43% [10:03:09] !log [somerandomdeveloper@mwtask181] starting deploy of {'config': True, 'force': True} to all [10:03:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:03:25] !log repool c2 [10:03:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:03:34] !log [somerandomdeveloper@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 24s [10:03:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:05:07] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 31% [10:05:59] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [11:27:25] PROBLEM - mem191 APT on mem191 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:27:45] PROBLEM - cp161 APT on cp161 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:27:50] PROBLEM - mw192 APT on mw192 is CRITICAL: APT CRITICAL: 158 packages available for upgrade (1 critical updates). [11:27:55] PROBLEM - swiftobject171 APT on swiftobject171 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:29:02] PROBLEM - os162 APT on os162 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:29:26] PROBLEM - graylog161 APT on graylog161 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [11:29:33] PROBLEM - mw163 APT on mw163 is CRITICAL: APT CRITICAL: 158 packages available for upgrade (1 critical updates). [11:29:39] PROBLEM - mw151 APT on mw151 is CRITICAL: APT CRITICAL: 159 packages available for upgrade (1 critical updates). [11:30:25] PROBLEM - os202 APT on os202 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:30:46] PROBLEM - db181 APT on db181 is CRITICAL: APT CRITICAL: 137 packages available for upgrade (1 critical updates). [11:31:26] PROBLEM - reports171 APT on reports171 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:31:51] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 40 packages available for upgrade (1 critical updates). [11:32:42] PROBLEM - db151 APT on db151 is CRITICAL: APT CRITICAL: 145 packages available for upgrade (1 critical updates). [11:32:57] PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 153 packages available for upgrade (1 critical updates). [11:32:58] PROBLEM - os161 APT on os161 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:33:23] PROBLEM - mw181 APT on mw181 is CRITICAL: APT CRITICAL: 158 packages available for upgrade (1 critical updates). [11:33:26] PROBLEM - eventgate182 APT on eventgate182 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:33:32] PROBLEM - swiftobject151 APT on swiftobject151 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:33:49] PROBLEM - cloud20 APT on cloud20 is CRITICAL: APT CRITICAL: 207 packages available for upgrade (1 critical updates). [11:33:54] PROBLEM - os191 APT on os191 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:34:14] PROBLEM - swiftobject161 APT on swiftobject161 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:34:29] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: APT CRITICAL: 76 packages available for upgrade (1 critical updates). [11:34:33] Screw you [11:34:43] You don't want to give me any calm today icinga-miraheze [11:34:56] i had a heart attack i thought we were having another outage [11:35:10] PROBLEM - db192 APT on db192 is CRITICAL: APT CRITICAL: 44 packages available for upgrade (1 critical updates). [11:35:29] PROBLEM - mon181 APT on mon181 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:35:39] PROBLEM - bast181 APT on bast181 is CRITICAL: APT CRITICAL: 45 packages available for upgrade (1 critical updates). [11:35:50] PROBLEM - mw183 APT on mw183 is CRITICAL: APT CRITICAL: 158 packages available for upgrade (1 critical updates). [11:35:51] PROBLEM - mw201 APT on mw201 is CRITICAL: APT CRITICAL: 158 packages available for upgrade (1 critical updates). [11:35:51] PROBLEM - ldap171 APT on ldap171 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:36:21] PROBLEM - mw152 APT on mw152 is CRITICAL: APT CRITICAL: 157 packages available for upgrade (1 critical updates). [11:37:24] PROBLEM - mw182 APT on mw182 is CRITICAL: APT CRITICAL: 158 packages available for upgrade (1 critical updates). [11:37:44] !log [rhinos@puppet181] Upgraded packages on cp161: python3-urllib3 [11:37:44] RECOVERY - cp161 APT on cp161 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:37:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:38:03] !log [rhinos@puppet181] Upgraded packages on bast181: python3-urllib3 [11:38:04] PROBLEM - mattermost2 APT on mattermost2 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:38:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:38:07] PROBLEM - matomo151 APT on matomo151 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:38:14] PROBLEM - mem201 APT on mem201 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:38:15] !log [rhinos@puppet181] Upgraded packages on changeprop202: python3-urllib3 [11:38:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:38:44] PROBLEM - cp201 APT on cp201 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:38:45] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:39:04] !log [rhinos@puppet181] Upgraded packages on graylog161: python3-urllib3 [11:39:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:39:17] !log [rhinos@puppet181] Upgraded packages on db151: python3-urllib3 [11:39:18] PROBLEM - swiftproxy161 APT on swiftproxy161 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:39:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:39:26] RECOVERY - graylog161 APT on graylog161 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [11:39:38] RECOVERY - bast181 APT on bast181 is OK: APT OK: 44 packages available for upgrade (0 critical updates). [11:40:06] !log [rhinos@puppet181] Upgraded packages on db192: python3-urllib3 [11:40:07] PROBLEM - mw202 APT on mw202 is CRITICAL: APT CRITICAL: 158 packages available for upgrade (1 critical updates). [11:40:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:40:21] PROBLEM - db172 APT on db172 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:40:27] !log [rhinos@puppet181] Upgraded packages on cloud20: python3-urllib3 [11:40:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:40:40] !log [rhinos@puppet181] Upgraded packages on eventgate182: python3-urllib3 [11:40:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:40:50] RECOVERY - db151 APT on db151 is OK: APT OK: 144 packages available for upgrade (0 critical updates). [11:40:54] !log [rhinos@puppet181] Upgraded packages on db181: python3-urllib3 [11:40:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:40:57] PROBLEM - bast161 APT on bast161 is CRITICAL: APT CRITICAL: 44 packages available for upgrade (1 critical updates). [11:40:58] PROBLEM - mw172 APT on mw172 is CRITICAL: APT CRITICAL: 157 packages available for upgrade (1 critical updates). [11:41:07] !log [rhinos@puppet181] Upgraded packages on cp201: python3-urllib3 [11:41:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:41:12] RECOVERY - db192 APT on db192 is OK: APT OK: 43 packages available for upgrade (0 critical updates). [11:41:18] !log [rhinos@puppet181] Upgraded packages on ldap171: python3-urllib3 [11:41:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:41:25] RECOVERY - eventgate182 APT on eventgate182 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:41:28] !log [rhinos@puppet181] Upgraded packages on mattermost2: python3-urllib3 [11:41:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:41:38] PROBLEM - mw162 APT on mw162 is CRITICAL: APT CRITICAL: 157 packages available for upgrade (1 critical updates). [11:41:40] !log [rhinos@puppet181] Upgraded packages on matomo151: python3-urllib3 [11:41:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:41:42] PROBLEM - llm191 APT on llm191 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:41:47] RECOVERY - cloud20 APT on cloud20 is OK: APT OK: 206 packages available for upgrade (0 critical updates). [11:41:51] RECOVERY - ldap171 APT on ldap171 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:41:53] !log [rhinos@puppet181] Upgraded packages on llm191: python3-urllib3 [11:41:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:41:56] PROBLEM - rdb191 APT on rdb191 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:42:02] PROBLEM - mw203 APT on mw203 is CRITICAL: APT CRITICAL: 158 packages available for upgrade (1 critical updates). [11:42:04] RECOVERY - mattermost2 APT on mattermost2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:42:04] !log [rhinos@puppet181] Upgraded packages on mem161: python3-urllib3 [11:42:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:42:07] RECOVERY - matomo151 APT on matomo151 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:42:10] PROBLEM - swiftproxy171 APT on swiftproxy171 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:42:14] PROBLEM - mw161 APT on mw161 is CRITICAL: APT CRITICAL: 157 packages available for upgrade (1 critical updates). [11:42:22] !log [rhinos@puppet181] Upgraded packages on mem201: python3-urllib3 [11:42:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:42:35] !log [rhinos@puppet181] Upgraded packages on mem191: python3-urllib3 [11:42:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:42:42] RECOVERY - db181 APT on db181 is OK: APT OK: 136 packages available for upgrade (0 critical updates). [11:42:43] RECOVERY - cp201 APT on cp201 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:42:47] !log [rhinos@puppet181] Upgraded packages on mon181: python3-urllib3 [11:42:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:42:57] PROBLEM - mwtask161 APT on mwtask161 is CRITICAL: APT CRITICAL: 162 packages available for upgrade (1 critical updates). [11:43:00] PROBLEM - puppet181 APT on puppet181 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [11:43:02] !log [rhinos@puppet181] Upgraded packages on mw172: python3-urllib3 [11:43:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:43:11] PROBLEM - cp171 APT on cp171 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [11:43:11] PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 168 packages available for upgrade (1 critical updates). [11:43:18] !log [rhinos@puppet181] Upgraded packages on mw151: python3-urllib3 [11:43:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:43:27] RECOVERY - mon181 APT on mon181 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:43:32] !log [rhinos@puppet181] Upgraded packages on mw161: python3-urllib3 [11:43:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:43:35] RECOVERY - mem191 APT on mem191 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:43:35] PROBLEM - swiftobject191 APT on swiftobject191 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:43:41] RECOVERY - llm191 APT on llm191 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:43:47] !log [rhinos@puppet181] Upgraded packages on mw152: python3-urllib3 [11:43:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:43:49] PROBLEM - swiftobject201 APT on swiftobject201 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:43:50] RECOVERY - mw151 APT on mw151 is OK: APT OK: 158 packages available for upgrade (0 critical updates). [11:44:10] !log [rhinos@puppet181] Upgraded packages on mw162: python3-urllib3 [11:44:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:44:12] PROBLEM - mw173 APT on mw173 is CRITICAL: APT CRITICAL: 158 packages available for upgrade (1 critical updates). [11:44:15] RECOVERY - mem201 APT on mem201 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:44:16] RECOVERY - mw161 APT on mw161 is OK: APT OK: 156 packages available for upgrade (0 critical updates). [11:44:31] RECOVERY - mw152 APT on mw152 is OK: APT OK: 156 packages available for upgrade (0 critical updates). [11:44:35] !log [rhinos@puppet181] Upgraded packages on mw163: python3-urllib3 [11:44:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:44:50] !log [rhinos@puppet181] Upgraded packages on mw173: python3-urllib3 [11:44:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:44:56] RECOVERY - mw172 APT on mw172 is OK: APT OK: 156 packages available for upgrade (0 critical updates). [11:45:15] PROBLEM - swiftobject181 APT on swiftobject181 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:45:16] !log [rhinos@puppet181] Upgraded packages on mw183: python3-urllib3 [11:45:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:45:20] PROBLEM - mwtask151 APT on mwtask151 is CRITICAL: APT CRITICAL: 162 packages available for upgrade (1 critical updates). [11:45:27] PROBLEM - phorge171 APT on phorge171 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:45:41] !log [rhinos@puppet181] Upgraded packages on mw192: python3-urllib3 [11:45:42] RECOVERY - mw162 APT on mw162 is OK: APT OK: 156 packages available for upgrade (0 critical updates). [11:45:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:45:46] RECOVERY - mw163 APT on mw163 is OK: APT OK: 157 packages available for upgrade (0 critical updates). [11:45:56] !log [rhinos@puppet181] Upgraded packages on mw181: python3-urllib3 [11:45:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:45:58] RECOVERY - mw183 APT on mw183 is OK: APT OK: 157 packages available for upgrade (0 critical updates). [11:46:00] RECOVERY - mw192 APT on mw192 is OK: APT OK: 157 packages available for upgrade (0 critical updates). [11:46:04] PROBLEM - db161 APT on db161 is CRITICAL: APT CRITICAL: 144 packages available for upgrade (1 critical updates). [11:46:13] !log [rhinos@puppet181] Upgraded packages on mw201: python3-urllib3 [11:46:13] RECOVERY - mw173 APT on mw173 is OK: APT OK: 157 packages available for upgrade (0 critical updates). [11:46:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:46:14] PROBLEM - swiftac171 APT on swiftac171 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:46:29] !log [rhinos@puppet181] Upgraded packages on mw182: python3-urllib3 [11:46:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:46:41] !log [rhinos@puppet181] Upgraded packages on ns1: python3-urllib3 [11:46:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:46:56] !log [rhinos@puppet181] Upgraded packages on mw202: python3-urllib3 [11:46:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:47:08] PROBLEM - cloud16 APT on cloud16 is CRITICAL: APT CRITICAL: 203 packages available for upgrade (1 critical updates). [11:47:08] PROBLEM - os201 APT on os201 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:47:13] !log [rhinos@puppet181] Upgraded packages on mw203: python3-urllib3 [11:47:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:47:28] !log [rhinos@puppet181] Upgraded packages on mwtask161: python3-urllib3 [11:47:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:47:37] RECOVERY - mw181 APT on mw181 is OK: APT OK: 157 packages available for upgrade (0 critical updates). [11:47:43] RECOVERY - mw182 APT on mw182 is OK: APT OK: 157 packages available for upgrade (0 critical updates). [11:47:44] !log [rhinos@puppet181] Upgraded packages on mwtask171: python3-urllib3 [11:47:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:47:48] PROBLEM - mem151 APT on mem151 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:47:57] PROBLEM - cloud18 APT on cloud18 is CRITICAL: APT CRITICAL: 203 packages available for upgrade (1 critical updates). [11:47:59] !log [rhinos@puppet181] Upgraded packages on mwtask151: python3-urllib3 [11:48:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:48:05] PROBLEM - cloud15 APT on cloud15 is CRITICAL: APT CRITICAL: 206 packages available for upgrade (1 critical updates). [11:48:08] RECOVERY - mw203 APT on mw203 is OK: APT OK: 157 packages available for upgrade (0 critical updates). [11:48:09] RECOVERY - mw202 APT on mw202 is OK: APT OK: 157 packages available for upgrade (0 critical updates). [11:48:10] RECOVERY - mw201 APT on mw201 is OK: APT OK: 157 packages available for upgrade (0 critical updates). [11:48:12] !log [rhinos@puppet181] Upgraded packages on os151: python3-urllib3 [11:48:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:48:25] !log [rhinos@puppet181] Upgraded packages on os161: python3-urllib3 [11:48:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:48:37] !log [rhinos@puppet181] Upgraded packages on os162: python3-urllib3 [11:48:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:48:49] !log [rhinos@puppet181] Upgraded packages on phorge171: python3-urllib3 [11:48:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:48:57] RECOVERY - os161 APT on os161 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:48:59] PROBLEM - mw193 APT on mw193 is CRITICAL: APT CRITICAL: 158 packages available for upgrade (1 critical updates). [11:49:02] RECOVERY - os162 APT on os162 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:49:04] !log [rhinos@puppet181] Upgraded packages on mwtask181: python3-urllib3 [11:49:04] RECOVERY - mwtask161 APT on mwtask161 is OK: APT OK: 161 packages available for upgrade (0 critical updates). [11:49:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:49:15] !log [rhinos@puppet181] Upgraded packages on prometheus151: python3-urllib3 [11:49:15] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 167 packages available for upgrade (0 critical updates). [11:49:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:49:23] RECOVERY - mwtask151 APT on mwtask151 is OK: APT OK: 161 packages available for upgrade (0 critical updates). [11:49:28] !log [rhinos@puppet181] Upgraded packages on os191: python3-urllib3 [11:49:28] RECOVERY - phorge171 APT on phorge171 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:49:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:49:40] !log [rhinos@puppet181] Upgraded packages on reports171: python3-urllib3 [11:49:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:49:53] !log [rhinos@puppet181] Upgraded packages on puppet181: python3-urllib3 [11:49:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:50:02] RECOVERY - os191 APT on os191 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:50:12] !log [rhinos@puppet181] Upgraded packages on os202: python3-urllib3 [11:50:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:50:22] RECOVERY - os202 APT on os202 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:50:25] !log [rhinos@puppet181] Upgraded packages on os201: python3-urllib3 [11:50:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:50:33] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 75 packages available for upgrade (0 critical updates). [11:50:41] !log [rhinos@puppet181] Upgraded packages on rdb191: python3-urllib3 [11:50:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:50:44] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:50:58] !log [rhinos@puppet181] Upgraded packages on ns2: python3-urllib3 [11:50:58] RECOVERY - puppet181 APT on puppet181 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [11:50:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:51:04] PROBLEM - db171 APT on db171 is CRITICAL: APT CRITICAL: 136 packages available for upgrade (1 critical updates). [11:51:07] RECOVERY - os201 APT on os201 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:51:11] !log [rhinos@puppet181] Upgraded packages on swiftobject151: python3-urllib3 [11:51:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:51:25] !log [rhinos@puppet181] Upgraded packages on swiftobject161: python3-urllib3 [11:51:26] RECOVERY - reports171 APT on reports171 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:51:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:51:31] RECOVERY - swiftobject151 APT on swiftobject151 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:51:38] !log [rhinos@puppet181] Upgraded packages on swiftac171: python3-urllib3 [11:51:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:51:43] PROBLEM - cloud17 APT on cloud17 is CRITICAL: APT CRITICAL: 203 packages available for upgrade (1 critical updates). [11:51:52] !log [rhinos@puppet181] Upgraded packages on swiftobject171: python3-urllib3 [11:51:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:51:55] RECOVERY - rdb191 APT on rdb191 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:52:03] RECOVERY - swiftobject171 APT on swiftobject171 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:52:05] !log [rhinos@puppet181] Upgraded packages on swiftobject191: python3-urllib3 [11:52:07] RECOVERY - ns2 APT on ns2 is OK: APT OK: 39 packages available for upgrade (0 critical updates). [11:52:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:52:14] RECOVERY - swiftac171 APT on swiftac171 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:52:14] RECOVERY - swiftobject161 APT on swiftobject161 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:52:19] PROBLEM - mw191 APT on mw191 is CRITICAL: APT CRITICAL: 158 packages available for upgrade (1 critical updates). [11:52:21] !log [rhinos@puppet181] Upgraded packages on swiftobject181: python3-urllib3 [11:52:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:52:35] !log [rhinos@puppet181] Upgraded packages on swiftproxy171: python3-urllib3 [11:52:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:52:40] PROBLEM - mw171 APT on mw171 is CRITICAL: APT CRITICAL: 157 packages available for upgrade (1 critical updates). [11:52:49] !log [rhinos@puppet181] Upgraded packages on swiftobject201: python3-urllib3 [11:52:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:52:59] PROBLEM - db201 APT on db201 is CRITICAL: APT CRITICAL: 43 packages available for upgrade (1 critical updates). [11:53:01] PROBLEM - kafka182 APT on kafka182 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:53:02] !log [rhinos@puppet181] Upgraded packages on swiftproxy161: python3-urllib3 [11:53:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:53:04] PROBLEM - mw153 APT on mw153 is CRITICAL: APT CRITICAL: 158 packages available for upgrade (1 critical updates). [11:53:08] PROBLEM - db182 APT on db182 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:53:15] RECOVERY - swiftobject181 APT on swiftobject181 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:53:17] RECOVERY - swiftproxy161 APT on swiftproxy161 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:53:19] !log [rhinos@puppet181] Upgraded packages on test151: python3-urllib3 [11:53:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:53:34] RECOVERY - swiftobject191 APT on swiftobject191 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:53:54] RECOVERY - swiftobject201 APT on swiftobject201 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:54:02] PROBLEM - cloud19 APT on cloud19 is CRITICAL: APT CRITICAL: 65 packages available for upgrade (1 critical updates). [11:54:09] RECOVERY - swiftproxy171 APT on swiftproxy171 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:55:02] RECOVERY - test151 APT on test151 is OK: APT OK: 152 packages available for upgrade (0 critical updates). [11:55:08] !log [rhinos@puppet181] Upgraded packages on bast161: python3-urllib3 [11:55:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:55:26] !log [rhinos@puppet181] Upgraded packages on bots171: python3-urllib3 [11:55:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:55:43] !log [rhinos@puppet181] Upgraded packages on db172: python3-urllib3 [11:55:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:55:56] !log [rhinos@puppet181] Upgraded packages on db161: python3-urllib3 [11:55:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:56:04] RECOVERY - db161 APT on db161 is OK: APT OK: 143 packages available for upgrade (0 critical updates). [11:56:09] !log [rhinos@puppet181] Upgraded packages on db182: python3-urllib3 [11:56:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:56:20] PROBLEM - cp191 APT on cp191 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:56:25] RECOVERY - db172 APT on db172 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:56:30] !log [rhinos@puppet181] Upgraded packages on cp191: python3-urllib3 [11:56:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:56:53] !log [rhinos@puppet181] Upgraded packages on db171: python3-urllib3 [11:56:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:56:57] RECOVERY - bast161 APT on bast161 is OK: APT OK: 43 packages available for upgrade (0 critical updates). [11:57:06] !log [rhinos@puppet181] Upgraded packages on cloud19: python3-urllib3 [11:57:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:57:08] RECOVERY - db182 APT on db182 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:57:15] RECOVERY - db171 APT on db171 is OK: APT OK: 135 packages available for upgrade (0 critical updates). [11:57:20] !log [rhinos@puppet181] Upgraded packages on cloud15: python3-urllib3 [11:57:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:57:35] !log [rhinos@puppet181] Upgraded packages on cloud16: python3-urllib3 [11:57:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:58:02] RECOVERY - cloud19 APT on cloud19 is OK: APT OK: 64 packages available for upgrade (0 critical updates). [11:58:07] RECOVERY - cloud15 APT on cloud15 is OK: APT OK: 205 packages available for upgrade (0 critical updates). [11:58:08] !log [rhinos@puppet181] Upgraded packages on cp171: python3-urllib3 [11:58:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:58:19] RECOVERY - cp191 APT on cp191 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:58:22] !log [rhinos@puppet181] Upgraded packages on cloud17: python3-urllib3 [11:58:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:58:42] !log [rhinos@puppet181] Upgraded packages on cloud18: python3-urllib3 [11:58:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:59:05] RECOVERY - cloud16 APT on cloud16 is OK: APT OK: 202 packages available for upgrade (0 critical updates). [11:59:07] !log [rhinos@puppet181] Upgraded packages on mem151: python3-urllib3 [11:59:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:59:11] RECOVERY - cp171 APT on cp171 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [11:59:40] !log [rhinos@puppet181] Upgraded packages on db201: python3-urllib3 [11:59:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:59:45] RECOVERY - mem151 APT on mem151 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:59:46] RECOVERY - cloud17 APT on cloud17 is OK: APT OK: 202 packages available for upgrade (0 critical updates). [11:59:55] !log [rhinos@puppet181] Upgraded packages on kafka182: python3-urllib3 [11:59:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:00:08] PROBLEM - mwtask171 HTTPS on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [12:00:10] PROBLEM - mwtask171 jobrunner.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [12:00:12] RECOVERY - cloud18 APT on cloud18 is OK: APT OK: 202 packages available for upgrade (0 critical updates). [12:00:36] PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:00:38] PROBLEM - mwtask171 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [12:00:50] !log [rhinos@puppet181] Upgraded packages on mw153: python3-urllib3 [12:00:51] PROBLEM - mwtask171 videoscaler.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [12:00:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:00:58] RECOVERY - db201 APT on db201 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [12:01:01] RECOVERY - kafka182 APT on kafka182 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [12:01:14] !log [rhinos@puppet181] Upgraded packages on mw171: python3-urllib3 [12:01:14] RECOVERY - mw153 APT on mw153 is OK: APT OK: 157 packages available for upgrade (0 critical updates). [12:01:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:01:38] PROBLEM - mwtask161 HTTPS on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [12:02:35] !log [rhinos@puppet181] Upgraded packages on mw191: python3-urllib3 [12:02:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:02:40] RECOVERY - mw171 APT on mw171 is OK: APT OK: 156 packages available for upgrade (0 critical updates). [12:03:01] !log [rhinos@puppet181] Upgraded packages on mw193: python3-urllib3 [12:03:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:03:03] PROBLEM - mwtask161 jobrunner.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [12:03:05] RECOVERY - mw193 APT on mw193 is OK: APT OK: 157 packages available for upgrade (0 critical updates). [12:03:11] Oh great another task dead [12:03:16] Stop dying task [12:03:43] PROBLEM - mwtask161 videoscaler.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [12:03:52] PROBLEM - mwtask161 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [12:04:00] PROBLEM - mwtask161 MediaWiki Rendering on mwtask161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:04:17] RECOVERY - mw191 APT on mw191 is OK: APT OK: 157 packages available for upgrade (0 critical updates). [12:06:18] RECOVERY - mwtask171 HTTPS on mwtask171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4258 bytes in 0.238 second response time [12:06:27] RECOVERY - mwtask171 jobrunner.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 167 bytes in 0.002 second response time [12:06:40] RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.551 second response time [12:06:54] RECOVERY - mwtask171 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.002 second response time [12:07:01] RECOVERY - mwtask171 videoscaler.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 167 bytes in 0.002 second response time [12:07:59] RECOVERY - mwtask161 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 6.472 second response time [12:08:18] RECOVERY - mwtask161 MediaWiki Rendering on mwtask161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.213 second response time [12:09:57] RECOVERY - mwtask161 videoscaler.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 8.438 second response time [12:09:58] RECOVERY - mwtask161 HTTPS on mwtask161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4258 bytes in 3.001 second response time [12:11:29] RECOVERY - mwtask161 jobrunner.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 170 bytes in 0.943 second response time [12:12:01] !log bounce php-fpm on mwtask161 [12:12:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:13:13] PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:13:15] PROBLEM - mwtask151 HTTPS on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [12:13:59] PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:14:35] PROBLEM - mwtask171 HTTPS on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [12:15:11] RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.238 second response time [12:15:12] RECOVERY - mwtask151 HTTPS on mwtask151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4258 bytes in 0.057 second response time [12:15:21] PROBLEM - mwtask171 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [12:15:55] RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.963 second response time [12:16:34] RECOVERY - mwtask171 HTTPS on mwtask171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4258 bytes in 2.386 second response time [12:17:20] RECOVERY - mwtask171 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 169 bytes in 0.039 second response time [12:21:05] PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [12:21:26] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1768648850000&orgId=1&to=1768652486679 [12:21:56] PROBLEM - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [12:22:02] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:22:14] PROBLEM - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [12:22:20] PROBLEM - mwtask181 videoscaler.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [12:25:06] RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4258 bytes in 0.064 second response time [12:25:59] RECOVERY - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.002 second response time [12:26:05] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.578 second response time [12:26:17] RECOVERY - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.002 second response time [12:26:26] [Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https://grafana.wikitide.net/d/GtxbP1Xnk?from=1768648970000&orgId=1&to=1768652570000 [12:26:27] RECOVERY - mwtask181 videoscaler.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.002 second response time [12:28:00] !log increase db182 ram by 5gb [12:28:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:30:18] PROBLEM - matomo151 Check unit status of matomo-archiver-3 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-3 [12:31:00] PROBLEM - matomo151 Check unit status of matomo-queuedtracking-0 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-queuedtracking-0 [12:31:27] PROBLEM - matomo151 Check unit status of matomo-archiver-1 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-1 [12:31:35] PROBLEM - matomo151 Check unit status of matomo-archiver-2 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-2 [12:33:00] RECOVERY - matomo151 Check unit status of matomo-queuedtracking-0 on matomo151 is OK: OK: Status of the systemd unit matomo-queuedtracking-0 [12:39:22] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/c3a05c445265 [12:39:23] 02puppet/03paladox-patch-1 07paladox 03c3a05c4 db182: increase buffer pool size to 18g [12:39:27] [02puppet] 07paladox opened pull request #4693: db182: increase buffer pool size to 18g (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4693 [12:39:32] [02puppet] 07coderabbitai[bot] commented on pull request #4693:
[…] 13https://github.com/miraheze/puppet/pull/4693#issuecomment-3763509019 [12:39:35] [02puppet] 07paladox merged pull request #4693: db182: increase buffer pool size to 18g (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4693 [12:39:36] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/18c07409fdb659ed09ee780f72289a550f03fe58 [12:39:36] 02puppet/03main 07paladox 0318c0740 db182: increase buffer pool size to 18g (#4693) [12:39:37] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 03c3a05c4 13https://api.github.com/repos/miraheze/puppet/commit/c3a05c4 [12:41:27] RECOVERY - matomo151 Check unit status of matomo-archiver-1 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-1 [12:41:36] RECOVERY - matomo151 Check unit status of matomo-archiver-2 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-2 [12:42:18] RECOVERY - matomo151 Check unit status of matomo-archiver-3 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-3 [12:55:26] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1768650890000&orgId=1&to=1768654526681 [13:05:26] [Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https://grafana.wikitide.net/d/GtxbP1Xnk?from=1768651430000&orgId=1&to=1768655030000 [13:06:58] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1768651610000&orgId=1&to=1768655218008[Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https://grafana.wikitide.net/d/GtxbP1Xnk?from=1768651430000&orgId=1&to=1768655030000 [14:19:28] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 7 backends are down. mw152 mw162 mw182 mw183 mw193 mw202 mw203 [14:20:15] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:20:42] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 3 backends are down. mw171 mw173 mw191 [14:21:26] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 4 backends are down. mw152 mw161 mw182 mw191 [14:23:06] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [14:23:49] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [14:23:54] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [14:24:00] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [14:24:03] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 10 backends are down. mw151 mw152 mw162 mw171 mw182 mw173 mw183 mw193 mw202 mw203 [14:25:07] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.769 second response time [14:25:07] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 50% [14:25:26] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [14:25:27] @paladox@rhinosf1 [14:25:28] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [14:25:32] ok wow [14:25:38] a second after I pinged [14:25:48] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.203 second response time [14:25:54] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.185 second response time [14:25:57] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4357 bytes in 0.064 second response time [14:26:03] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [14:26:26] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.577 second response time [14:26:42] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [14:26:45] seems to be QuickInstantCommons ? [14:26:58] i looked at the slow log on test151 [14:27:07] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 27% [14:27:36] wtf? [14:27:59] how can qic take down several servers [14:31:34] well i only looked at test but see mattermost for the backlog [14:41:58] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?from=1768651610000&orgId=1&to=1768660910000 [15:04:44] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/92f23e838961 [15:04:44] 02puppet/03paladox-patch-1 07paladox 0392f23e8 squid: Lower read and request timeout… [15:04:48] [02puppet] 07paladox opened pull request #4694: squid: Lower read and request timeout (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4694 [15:04:54] [02puppet] 07coderabbitai[bot] commented on pull request #4694:
[…] 13https://github.com/miraheze/puppet/pull/4694#issuecomment-3763979082 [15:05:41] [02puppet] 07paladox merged pull request #4694: squid: Lower read and request timeout (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4694 [15:05:42] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/ba8d5944ecfbb0738027d5e05f1921ce7087c6c3 [15:05:42] 02puppet/03main 07paladox 03ba8d594 squid: Lower read and request timeout (#4694)… [15:05:43] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 0392f23e8 13https://api.github.com/repos/miraheze/puppet/commit/92f23e8 [15:40:59] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/74e382362aa8 [15:40:59] 02puppet/03paladox-patch-1 07paladox 0374e3823 squid: Replace hierarchy_stoplist with always_direct… [15:41:03] [02puppet] 07paladox opened pull request #4695: squid: Replace hierarchy_stoplist with always_direct (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4695 [15:41:09] [02puppet] 07coderabbitai[bot] commented on pull request #4695:
[…] 13https://github.com/miraheze/puppet/pull/4695#issuecomment-3764033446 [15:41:13] [02puppet] 07paladox merged pull request #4695: squid: Replace hierarchy_stoplist with always_direct (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4695 [15:41:13] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/ae7ede67c1ce440e5faad639ae52bb924ad3bd5c [15:41:13] 02puppet/03main 07paladox 03ae7ede6 squid: Replace hierarchy_stoplist with always_direct (#4695)… [15:41:15] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 0374e3823 13https://api.github.com/repos/miraheze/puppet/commit/74e3823 [15:44:53] PROBLEM - mwtask161 Puppet on mwtask161 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_3d2png] [15:53:50] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 45339MiB (10.0% inode=100%); [16:12:53] RECOVERY - mwtask161 Puppet on mwtask161 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [16:17:47] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 45217MiB (10.0% inode=100%); [17:14:09] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 45293MiB (10.0% inode=100%); [17:54:33] [02MirahezeMagic] 07Universal-Omega created 03virtual-globalusage (+1 new commit) 13https://github.com/miraheze/MirahezeMagic/commit/39a8c7633085 [17:54:33] 02MirahezeMagic/03virtual-globalusage 07CosmicAlpha 0339a8c76 Use virtual domain for GlobalUsage [17:54:39] [02MirahezeMagic] 07Universal-Omega opened pull request #620: Use virtual domain for GlobalUsage (03main...03virtual-globalusage) 13https://github.com/miraheze/MirahezeMagic/pull/620 [17:54:44] [02MirahezeMagic] 07coderabbitai[bot] commented on pull request #620:
[…] 13https://github.com/miraheze/MirahezeMagic/pull/620#issuecomment-3764159969 [17:58:34] miraheze/MirahezeMagic - Universal-Omega the build has errored. [18:04:15] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:04:42] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 3 backends are down. mw171 mw183 mw191 [18:05:28] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 1 backends are down. mw152 [18:06:54] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03virtual-globalusage 13https://github.com/miraheze/MirahezeMagic/commit/7767d8c909121a8c75c68f8298fabe3c48262607 [18:06:54] 02MirahezeMagic/03virtual-globalusage 07CosmicAlpha 037767d8c Update [18:07:13] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [18:07:51] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03virtual-globalusage 13https://github.com/miraheze/MirahezeMagic/commit/d485e4ca84784d20ee514c49e6b506169e462e97 [18:07:51] 02MirahezeMagic/03virtual-globalusage 07CosmicAlpha 03d485e4c - [18:08:19] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.304 second response time [18:08:54] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [18:09:10] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 2.988 second response time [18:09:26] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 1 backends are down. mw192 [18:09:37] test151 what have you done again [18:10:42] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [18:10:48] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.203 second response time [18:11:26] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [18:11:28] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [18:12:16] miraheze/MirahezeMagic - Universal-Omega the build has errored. [18:15:40] [02ci-scripts] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/ci-scripts/commit/f0fa6ee1c3d8c0d8ef2b507ecfca8c18ab59b717 [18:15:40] 02ci-scripts/03main 07CosmicAlpha 03f0fa6ee Exclude TestKitchen on REL1_44 and REL1_45 [18:23:13] [02MirahezeMagic] 07Universal-Omega created 03python3.13 (+1 new commit) 13https://github.com/miraheze/MirahezeMagic/commit/b3478b6c96e2 [18:23:13] 02MirahezeMagic/03python3.13 07CosmicAlpha 03b3478b6 Use python 3.13 in workflow… [18:23:21] [02MirahezeMagic] 07Universal-Omega opened pull request #621: Use python 3.13 in workflow (03main...03python3.13) 13https://github.com/miraheze/MirahezeMagic/pull/621 [18:23:28] [02MirahezeMagic] 07coderabbitai[bot] commented on pull request #621:
[…] 13https://github.com/miraheze/MirahezeMagic/pull/621#issuecomment-3764183301 [18:25:57] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03python3.13 13https://github.com/miraheze/MirahezeMagic/commit/dfcaa6c0e9a1860caae4131ad3543851065d5651 [18:25:58] 02MirahezeMagic/03python3.13 07CosmicAlpha 03dfcaa6c Update packages [18:28:00] miraheze/MirahezeMagic - Universal-Omega the build passed. [18:37:43] miraheze/MirahezeMagic - Universal-Omega the build passed. [18:42:44] [02ci-scripts] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/ci-scripts/commit/c856da8610d4425c1a5d5630354b3758170b1ccd [18:42:44] 02ci-scripts/03main 07CosmicAlpha 03c856da8 Exclude MobileFrontend on REL1_45 [18:48:04] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/importDump.php --wiki=shintowiki dump.xml --no-updates (START) [18:48:05] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/importDump.php --wiki=shintowiki dump.xml --no-updates (END - exit=2) [18:48:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:48:06] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/rebuildall.php --wiki=shintowiki (START) [18:48:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:48:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:51:20] [02ci-scripts] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/ci-scripts/commit/5934e852d0dc4a6350375d62030f7167134162a6 [18:51:20] 02ci-scripts/03main 07CosmicAlpha 035934e85 Exclude DiscussionTools as well on REL1_45 [18:58:06] [02MirahezeMagic] 07Universal-Omega merged pull request #621: Use python 3.13 in workflow (03main...03python3.13) 13https://github.com/miraheze/MirahezeMagic/pull/621 [18:58:06] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/bfe263bfdf9dc384a16ab98eed80eee4f6031185 [18:58:06] 02MirahezeMagic/03main 07CosmicAlpha 03bfe263b Use python 3.13 in workflow… [18:58:07] [02MirahezeMagic] 07Universal-Omega 04deleted 03python3.13 at 03dfcaa6c 13https://api.github.com/repos/miraheze/MirahezeMagic/commit/dfcaa6c [19:03:46] miraheze/MirahezeMagic - Universal-Omega the build passed. [19:09:45] miraheze/MirahezeMagic - Universal-Omega the build passed. [19:11:11] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03virtual-globalusage 13https://github.com/miraheze/MirahezeMagic/commit/2ccc815966d6e542d81b6a874a49c1d54fb95099 [19:11:11] 02MirahezeMagic/03virtual-globalusage 07CosmicAlpha 032ccc815 Set $wgGlobalUserPageDBname [19:22:02] miraheze/MirahezeMagic - Universal-Omega the build passed. [19:22:29] [02ci-scripts] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/ci-scripts/commit/88c78660e22de1a9a398f0fdecd075bf48fefc54 [19:22:29] 02ci-scripts/03main 07CosmicAlpha 0388c7866 Exclude SecurePoll as well on REL1_45 [19:25:29] !log [macfan@test151] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/extensions/OATHAuth/maintenance/MoveRecoveryCodesFromTOTP.php --wiki=metawikibeta (END - exit=0) [19:25:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:26:09] !log [macfan@test151] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/extensions/OATHAuth/maintenance/UpdateForMultipleDevicesSupport.php --wiki=metawikibeta (END - exit=256) [19:26:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:27:47] !log [macfan@test151] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/extensions/OATHAuth/maintenance/UpdateSecretsToEncryptedFormat.php --wiki=metawikibeta (END - exit=256) [19:27:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:32:18] PROBLEM - matomo151 Check unit status of matomo-archiver-3 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-3 [19:33:52] miraheze/MirahezeMagic - Universal-Omega the build passed. [19:43:24] PROBLEM - llm191 Puppet on llm191 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[openwebui] [20:11:24] RECOVERY - llm191 Puppet on llm191 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:15:38] [02puppet] 07Universal-Omega created 03mariadb-ipv4-bind (+1 new commit) 13https://github.com/miraheze/puppet/commit/3d5701122a5b [20:15:38] 02puppet/03mariadb-ipv4-bind 07CosmicAlpha 033d57011 MariaDB: only bind on IPv4 [20:15:43] [02puppet] 07Universal-Omega opened pull request #4696: MariaDB: only bind on IPv4 (03main...03mariadb-ipv4-bind) 13https://github.com/miraheze/puppet/pull/4696 [20:15:49] [02puppet] 07coderabbitai[bot] commented on pull request #4696:
[…] 13https://github.com/miraheze/puppet/pull/4696#issuecomment-3764292483 [20:49:44] [02puppet] 07paladox left a file comment in pull request #4696 033d57011: You may just want to unset this and use what ever it sets as the default. 13https://github.com/miraheze/puppet/pull/4696#discussion_r2701538703 [20:56:29] PROBLEM - mwtask181 Disk Space on mwtask181 is WARNING: DISK WARNING - free space: / 24311MiB (10% inode=95%); [21:06:29] RECOVERY - mwtask181 Disk Space on mwtask181 is OK: DISK OK - free space: / 34265MiB (15% inode=95%); [22:04:18] RECOVERY - matomo151 Check unit status of matomo-archiver-3 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-3 [23:27:20] !log [universalomega@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'DynamicPageList4'} to all [23:27:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:27:51] !log [universalomega@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'DynamicPageList4'} to all - SUCCESS in 31s [23:27:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:00] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/4d8b0126f38f0fb325ac749bc5fd8e000409a1f6 [23:30:00] 02mw-config/03main 07CosmicAlpha 034d8b012 Enable $wgDplSettings['alwaysCacheResults'] [23:30:39] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [23:30:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:51] miraheze/mw-config - Universal-Omega the build passed. [23:31:12] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 32s [23:31:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:47:27] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/15c53d5b405048e7b1ed29bc4ce6027204223efe [23:47:29] 02mw-config/03main 07CosmicAlpha 0315c53d5 Don't enable $wgDplSettings['alwaysCacheResults'] on dpl4wiki [23:47:34] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [23:47:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:48:02] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 28s [23:48:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:48:22] miraheze/mw-config - Universal-Omega the build passed.