[00:01:00] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.30, 22.58, 20.66 [00:06:43] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.74, 4.89, 4.42 [00:10:40] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.67, 4.58, 4.42 [00:16:59] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.40, 21.58, 23.58 [00:19:00] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.86, 23.58, 24.07 [00:20:59] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.14, 22.53, 23.60 [00:32:58] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.67, 17.60, 20.17 [00:39:00] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.58, 18.87, 19.84 [00:44:24] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [00:46:59] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.37, 23.86, 21.69 [00:54:59] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.62, 23.49, 22.60 [00:56:58] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.13, 23.32, 22.57 [00:58:59] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.68, 21.53, 22.04 [01:00:59] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.77, 22.92, 22.47 [01:02:58] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.30, 21.69, 22.06 [01:03:22] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.80, 5.18, 4.71 [01:04:58] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.40, 23.43, 22.66 [01:05:19] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.33, 4.62, 4.57 [01:06:59] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.12, 21.47, 22.04 [01:07:12] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [01:22:59] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.75, 21.76, 21.28 [01:24:59] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.53, 20.83, 21.02 [01:28:58] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.42, 18.51, 20.12 [01:34:58] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.04, 20.83, 20.65 [01:36:59] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.37, 18.61, 19.86 [01:51:47] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.30, 19.18, 18.69 [01:53:45] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.07, 18.29, 18.43 [02:02:41] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [02:10:18] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.04, 1.67, 1.19 [02:11:40] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [02:12:18] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.99, 1.38, 1.14 [02:25:31] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [02:48:15] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [02:52:18] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.31, 1.86, 1.42 [02:54:18] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.74, 1.45, 1.32 [04:11:25] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [05:15:49] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [05:29:41] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [05:45:33] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [05:59:24] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [06:15:15] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [07:26:39] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [07:40:32] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [07:49:25] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [08:24:04] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [08:32:58] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [08:33:39] PROBLEM - heavyironmodding.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'heavyironmodding.org' expires in 15 day(s) (Mon 28 Jun 2021 08:28:39 GMT +0000). [08:36:03] PROBLEM - parapedia.pl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'parapedia.pl' expires in 15 day(s) (Mon 28 Jun 2021 08:30:51 GMT +0000). [08:41:36] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JZ1uc [08:41:37] [02miraheze/ssl] 07MirahezeSSLBot 03c3a53f6 - Bot: Update SSL cert for heavyironmodding.org [08:44:36] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JZ1zV [08:44:38] [02miraheze/ssl] 07MirahezeSSLBot 037e37826 - Bot: Update SSL cert for parapedia.pl [09:00:50] RECOVERY - heavyironmodding.org - LetsEncrypt on sslhost is OK: OK - Certificate 'heavyironmodding.org' will expire on Fri 10 Sep 2021 07:41:26 GMT +0000. [09:02:51] RECOVERY - parapedia.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'parapedia.pl' will expire on Fri 10 Sep 2021 07:44:29 GMT +0000. [09:16:30] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [09:30:23] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [10:07:01] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [10:13:53] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [10:29:47] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [11:20:14] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [11:34:06] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [12:33:32] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [13:01:13] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [13:04:00] Okay, so I managed to scrape all the images from the FANDOM version of Cancelled Movies Wiki and I want them imported into the Miraheze version: https://phabricator.miraheze.org/T7446 [13:30:56] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [13:37:48] paladox: I just realised I forgot about something on the MW upgrade docs - are the SQL patches ran before the depools/switch to 1.36 branch for after all mw* servers have been switched? [13:38:06] after [13:38:11] that's how i did it [13:38:23] ok, thanks! [13:39:19] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-2 [+0/-0/±1] 13https://git.io/JZDoz [13:39:21] [02miraheze/mw-config] 07Reception123 03ab8e23a - set all wikis to read only for 1.36 [13:39:22] [02mw-config] 07Reception123 created branch 03Reception123-patch-2 - 13https://git.io/vbvb3 [13:39:24] [02mw-config] 07Reception123 opened pull request 03#3952: set all wikis to read only for 1.36 - 13https://git.io/JZDo2 [13:40:18] miraheze/mw-config - Reception123 the build passed. [13:40:45] Reception|away: for most of the patches it don't really matter [13:40:53] Most are adding columns [13:41:02] yeah true, though there are some that change things [13:41:12] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/JZDKL [13:41:14] [02miraheze/puppet] 07Reception123 03ebce3f7 - switch mw8 to REL1_36 [13:41:15] [02puppet] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/vbiAS [13:41:16] So you could do it now and it wouldn't make a blind bat of a difference [13:41:17] [02puppet] 07Reception123 opened pull request 03#1778: switch mw8 to REL1_36 - 13https://git.io/JZDKm [13:41:27] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-1-1 [+0/-0/±1] 13https://git.io/JZDKG [13:41:29] [02miraheze/puppet] 07Reception123 03c693273 - switch mw9 to REL1_36 [13:41:30] [02puppet] 07Reception123 created branch 03Reception123-patch-1-1 - 13https://git.io/vbiAS [13:41:32] [02puppet] 07Reception123 opened pull request 03#1779: switch mw9 to REL1_36 - 13https://git.io/JZDKC [13:41:49] [02puppet] 07Reception123 created branch 03Reception123-patch-1-2 - 13https://git.io/vbiAS [13:41:50] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-1-2 [+0/-0/±1] 13https://git.io/JZDKR [13:41:52] [02miraheze/puppet] 07Reception123 030de368d - switch mw10 to REL1_36 [13:41:53] [02puppet] 07Reception123 opened pull request 03#1780: switch mw10 to REL1_36 - 13https://git.io/JZDKu [13:42:00] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-1-3 [+0/-0/±1] 13https://git.io/JZDK2 [13:42:01] [02miraheze/puppet] 07Reception123 03eba32ac - switch mw11 to REL1_36 [13:42:03] [02puppet] 07Reception123 created branch 03Reception123-patch-1-3 - 13https://git.io/vbiAS [13:42:04] [02puppet] 07Reception123 opened pull request 03#1781: switch mw11 to REL1_36 - 13https://git.io/JZDKV [13:42:09] yeah, though might as well to them after [13:42:39] Reception|away: don't forget to merge https://github.com/miraheze/mw-config/pull/3824 [13:43:01] Whichever way you do it you'll fatal flagged revs until you do [13:43:16] As flagged revs config changes are breaking [13:43:31] So master only works 1.35 / that only works 1.36 [13:44:44] [02miraheze/mw-config] 07RhinosF1 pushed 0315 commits to 03REL1_36 [+0/-0/±24] 13https://git.io/JZD6S [13:44:46] [02miraheze/mw-config] 07RhinosF1 03e354cd1 - Merge branch 'master' into REL1_36 [13:44:47] [02mw-config] 07RhinosF1 synchronize pull request 03#3824: Update config for 1.36 - 13https://git.io/JYjbk [13:45:44] miraheze/mw-config - RhinosF1 the build passed. [13:46:47] RhinosF1: yeah, will do before the the depools then [13:47:00] Reception|away: you can just set the default to 1.36 in that case [13:47:04] You'll break 1.35 [13:47:06] rather then individually per MW [13:47:20] what do you mean by break? [13:47:26] paladox: but during upgrade [13:47:31] paladox: it'll fatal [13:47:40] you can disable puppet [13:47:43] That's why we have 2 config patches [13:47:47] Yeah that'll work [13:48:03] But if that patch hits 1.35 it'll be breaking [13:48:10] s/patches/branches [13:48:10] RhinosF1 meant to say: That's why we have 2 config branches [13:48:14] oh so instead I should keep it on REL1_36 and set the mediawiki_config branch in puppet for REL1_36? [13:48:28] that's probably better yeah and then merge after [13:49:12] paladox: oh also, even though you'll be around just so I know, what was the command that you used when the puppet run failed on test3? [13:49:39] not sure what you mean [13:51:05] Reception|away: just delete w and do composer install after [13:51:25] And I'd do what paladox says and stop puppet everywhere before upgrading [13:51:32] Then only start when ready to switch [13:51:36] The server [13:51:44] yeah, but I thought there was something else as there was a git isue [13:52:30] Not that deleting w won't fix [13:52:43] let's hope so [14:00:34] [02miraheze/landing] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JZDDH [14:00:35] [02miraheze/landing] 07paladox 0353140e6 - Tweek matomo config [14:00:37] [02landing] 07paladox created branch 03paladox-patch-1 - 13https://git.io/fxEQA [14:00:38] [02landing] 07paladox opened pull request 03#64: Tweek matomo config - 13https://git.io/JZDD7 [14:01:35] miraheze/landing - paladox the build passed. [14:05:36] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [14:06:10] [02miraheze/landing] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JZD98 [14:06:11] [02miraheze/landing] 07paladox 030d3c99d - Update donate.php [14:06:13] [02landing] 07paladox synchronize pull request 03#64: Tweek matomo config - 13https://git.io/JZDD7 [14:06:32] [02miraheze/landing] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JZD9a [14:06:34] [02miraheze/landing] 07paladox 032cf980a - Update index.php [14:06:35] [02landing] 07paladox synchronize pull request 03#64: Tweek matomo config - 13https://git.io/JZDD7 [14:06:55] [02landing] 07paladox closed pull request 03#64: Tweek matomo config - 13https://git.io/JZDD7 [14:06:56] [02miraheze/landing] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JZD9D [14:06:58] [02miraheze/landing] 07paladox 0345a5eda - Tweek matomo config (#64) [14:06:59] [02miraheze/landing] 07paladox deleted branch 03paladox-patch-1 [14:07:01] [02landing] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/fxEQA [14:07:07] miraheze/landing - paladox the build passed. [14:07:29] miraheze/landing - paladox the build passed. [14:07:52] miraheze/landing - paladox the build passed. [14:12:04] [02miraheze/landing] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JZDQj [14:12:05] [02miraheze/landing] 07paladox 03e973535 - donate: remove site notice [14:12:56] miraheze/landing - paladox the build passed. [14:14:29] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [14:20:08] [02miraheze/landing] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JZDFT [14:20:08] [02miraheze/landing] 07paladox 038bfbd6e - update bootstrap files [14:20:09] [02landing] 07paladox created branch 03paladox-patch-1 - 13https://git.io/fxEQA [14:20:09] [02landing] 07paladox opened pull request 03#65: update bootstrap files - 13https://git.io/JZDFt [14:23:11] [02miraheze/landing] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JZDbR [14:23:12] [02miraheze/landing] 07paladox 035cdc439 - Update bootstrap.bundle.min.js [14:23:14] [02landing] 07paladox synchronize pull request 03#65: update bootstrap files - 13https://git.io/JZDFt [14:23:35] [02miraheze/landing] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JZDb6 [14:23:36] [02miraheze/landing] 07paladox 0389f9168 - Update bootstrap.bundle.min.js.map [14:23:38] [02landing] 07paladox synchronize pull request 03#65: update bootstrap files - 13https://git.io/JZDFt [14:24:10] miraheze/landing - paladox the build passed. [14:24:27] miraheze/landing - paladox the build passed. [14:28:20] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [14:34:14] [02miraheze/landing] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JZDhx [14:34:16] [02miraheze/landing] 07paladox 037e6eb40 - Fix title for donate page [14:35:07] miraheze/landing - paladox the build passed. [15:02:23] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/glob-parent-5.1.2 [+0/-0/±1] 13https://git.io/JZyq9 [15:02:24] [02miraheze/mediawiki] 07dependabot[bot] 037ea53f5 - Bump glob-parent from 5.1.1 to 5.1.2 [15:02:26] [02mediawiki] 07dependabot[bot] opened pull request 03#2238: Bump glob-parent from 5.1.1 to 5.1.2 - 13https://git.io/JZyqH [15:02:27] [02mediawiki] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/glob-parent-5.1.2 - 13https://git.io/vbL5b [15:02:29] [02mediawiki] 07dependabot[bot] labeled pull request 03#2238: Bump glob-parent from 5.1.1 to 5.1.2 - 13https://git.io/JZyqH [15:02:30] [02mediawiki] 07dependabot[bot] labeled pull request 03#2238: Bump glob-parent from 5.1.1 to 5.1.2 - 13https://git.io/JZyqH [15:14:42] [02miraheze/mediawiki] 07paladox pushed 03111 commits to 03REL1_36 [+2/-2/±134] 13https://git.io/JZyGE [15:14:43] [02miraheze/mediawiki] 07AaronSchulz 03589fa3c - rdbms: Use server time in DatabaseMysqlBase::getLagFromPtHeartbeat() [15:14:45] [02miraheze/mediawiki] 07reedy 038d661ff - apihelp-query+iwlinks-param-prop: s/interlanguage/interwiki/ [15:14:46] [02miraheze/mediawiki] 07reedy 034f76eb3 - Delete maintenance/cleanupAncientTables.php [15:14:48] [02miraheze/mediawiki] ... and 108 more commits. [15:15:13] PROBLEM - wiki.starship.digital - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.starship.digital could not be found [15:15:37] PROBLEM - wiki.fbpml.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.fbpml.org could not be found [15:20:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JZycs [15:20:12] [02miraheze/services] 07MirahezeSSLBot 0346d6df2 - BOT: Updating services config for wikis [15:20:40] have a nice weekend, all! [15:21:08] + good luck with the upgrade, exciting times [15:22:08] RECOVERY - wiki.starship.digital - reverse DNS on sslhost is OK: rDNS OK - wiki.starship.digital reverse DNS resolves to cp13.miraheze.org [15:22:34] RECOVERY - wiki.fbpml.org - reverse DNS on sslhost is OK: rDNS OK - wiki.fbpml.org reverse DNS resolves to cp13.miraheze.org [15:48:48] thanks Southparkfan :) [15:48:51] you too! [15:52:29] [02miraheze/mw-config] 07paladox pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JZyV8 [15:52:31] [02miraheze/mw-config] 07paladox 032bb592e - Remove duplicate UrlShortener [15:52:32] [02mw-config] 07paladox synchronize pull request 03#3824: Update config for 1.36 - 13https://git.io/JYjbk [15:53:29] miraheze/mw-config - paladox the build passed. [16:04:24] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [16:12:40] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.86, 1.53, 1.12 [16:14:35] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.52, 1.17, 1.04 [16:18:15] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [16:41:05] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [16:47:59] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [16:52:57] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.87, 6.34, 5.73 [16:53:26] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.09, 6.67, 6.15 [16:54:56] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.86, 6.15, 5.73 [16:57:25] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.51, 6.70, 6.34 [17:09:27] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-1-3 [+0/-0/±1] 13https://git.io/JZSq8 [17:09:29] [02miraheze/puppet] 07Reception123 030c12b5d - Update mw11.yaml [17:09:30] [02puppet] 07Reception123 synchronize pull request 03#1781: switch mw11 to REL1_36 - 13https://git.io/JZDKV [17:09:32] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-1-2 [+0/-0/±1] 13https://git.io/JZSq0 [17:09:33] [02miraheze/puppet] 07Reception123 031c03d82 - Update mw10.yaml [17:09:35] [02puppet] 07Reception123 synchronize pull request 03#1780: switch mw10 to REL1_36 - 13https://git.io/JZDKu [17:09:36] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-1-1 [+0/-0/±1] 13https://git.io/JZSqz [17:09:38] [02miraheze/puppet] 07Reception123 031862906 - Update mw9.yaml [17:09:39] [02puppet] 07Reception123 synchronize pull request 03#1779: switch mw9 to REL1_36 - 13https://git.io/JZDKC [17:13:24] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/JZSYY [17:13:26] [02miraheze/puppet] 07Reception123 033b7822b - Update mw8.yaml [17:13:27] [02puppet] 07Reception123 synchronize pull request 03#1778: switch mw8 to REL1_36 - 13https://git.io/JZDKm [17:31:26] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.62, 6.25, 5.84 [17:33:29] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.44, 6.19, 5.87 [18:00:23] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JZS2x [18:00:24] [02miraheze/WikiDiscover] 07Universal-Omega 038d91417 - T7444: don't redirect to private wiki with Special:PrivateWiki [18:00:26] [02WikiDiscover] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vhUAp [18:00:33] [02WikiDiscover] 07Universal-Omega opened pull request 03#50: T7444: don't redirect to private wiki with Special:RandomWiki - 13https://git.io/JZSaJ [18:01:30] miraheze/WikiDiscover - Universal-Omega the build passed. [18:03:28] [02WikiDiscover] 07paladox closed pull request 03#50: T7444: don't redirect to private wiki with Special:RandomWiki - 13https://git.io/JZSaJ [18:03:30] [02miraheze/WikiDiscover] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JZSVc [18:03:31] [02miraheze/WikiDiscover] 07Universal-Omega 03f18927c - T7444: don't redirect to private wiki with Special:PrivateWiki (#50) [18:03:33] [02miraheze/WikiDiscover] 07paladox deleted branch 03Universal-Omega-patch-2 [18:03:34] [02WikiDiscover] 07paladox deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vhUAp [18:04:24] miraheze/WikiDiscover - paladox the build passed. [18:05:27] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_36 [+0/-0/±2] 13https://git.io/JZSwE [18:05:28] [02miraheze/mediawiki] 07paladox 038e154c3 - Update Comments [18:05:30] [02miraheze/mediawiki] 07paladox 03ad5195c - Update WikiDiscover [18:11:03] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://git.io/JZSoS [18:11:05] [02miraheze/mw-config] 07Universal-Omega 03a9e76b3 - Remove 1.36 sitenotice [18:11:06] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-4 - 13https://git.io/vbvb3 [18:11:24] [02mw-config] 07Universal-Omega opened pull request 03#3953: Remove 1.36 sitenotice - 13https://git.io/JZSoj [18:12:45] miraheze/mw-config - Universal-Omega the build passed. [18:24:33] Universal_Omega, re: #3953, Reception|away were talking the other day about having the sitenotice stay up for about 12-18 hours after the upgrade is complete, so perhaps we could modify that PR to instead have a sitenotice that says something like, "`Miraheze has upgraded to MediaWiki 1.36. If you encounter any technical issues, please create a Phabricator task`" or something? [18:28:42] dmehus: OK. I will. [18:29:50] Universal_Omega, cool, thanks :) [18:30:12] No problem. [18:30:14] That's fine yeah [18:35:18] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://git.io/JZSHI [18:35:19] [02miraheze/mw-config] 07Universal-Omega 03405201f - Switch to post-update sitenotice. [18:35:21] [02mw-config] 07Universal-Omega synchronize pull request 03#3953: Remove 1.36 sitenotice - 13https://git.io/JZSoj [18:35:46] [02mw-config] 07Universal-Omega edited pull request 03#3953: Switch to post-upgrade sitenotice - 13https://git.io/JZSoj [18:35:51] [02mw-config] 07Universal-Omega edited pull request 03#3953: Switch to post-upgrade sitenotice - 13https://git.io/JZSoj [18:36:01] dmehus: ^ [18:36:17] miraheze/mw-config - Universal-Omega the build passed. [18:53:30] Universal_Omega, LGTM. 👍 [18:58:00] dmehus: great! Thanks! [19:00:35] [02mw-config] 07Reception123 closed pull request 03#3952: set all wikis to read only for 1.36 - 13https://git.io/JZDo2 [19:00:36] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JZSpY [19:00:38] [02miraheze/mw-config] 07Reception123 03390da59 - set all wikis to read only for 1.36 (#3952) [19:00:44] And it begins! [19:01:37] miraheze/mw-config - Reception123 the build passed. [19:03:08] !log disable puppet on mw* [19:03:30] !log depool mw8 [19:04:17] [02puppet] 07Reception123 closed pull request 03#1778: switch mw8 to REL1_36 - 13https://git.io/JZDKm [19:04:19] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JZShX [19:04:20] [02miraheze/puppet] 07Reception123 032fac107 - switch mw8 to REL1_36 (#1778) [19:05:07] !log disable puppet on mw* [19:05:15] !log depool mw8; re-enable puppet on mw8 and run [19:05:23] Receotion|away: it can't log because read-only [19:05:41] PROBLEM - mw9 Puppet on mw9 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 2 minutes ago with 0 failures [19:05:47] MacFan4000: oh... of course. Guess I'll just use the !log here on IRC just so the time is recorded somewhere [19:05:53] PROBLEM - jobrunner4 Puppet on jobrunner4 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 2 minutes ago with 0 failures [19:06:00] PROBLEM - jobrunner3 Puppet on jobrunner3 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 2 minutes ago with 0 failures [19:06:11] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.28, 7.92, 6.29 [19:06:15] PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 1 backends are down. mw8 [19:06:23] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 1 backends are down. mw8 [19:06:35] paladox: for some reason puppet just ran normally on mw8 and it doesn't seem it took the PR into account https://github.com/miraheze/puppet/pull/1778/files [19:06:51] PROBLEM - mw11 Puppet on mw11 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 2 minutes ago with 0 failures [19:06:52] Reception|away: you shouldn't be running puppet. You need to first git clone mediawiki [19:07:10] paladox: oh, so I should just delete the dir and clone it manually? [19:07:14] sudo -u www-data git clone https://github.com/miraheze/mediawiki --recursive w [19:07:14] yeh [19:07:21] ok [19:07:21] PROBLEM - mw10 Puppet on mw10 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 4 minutes ago with 0 failures [19:07:25] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.49, 8.91, 6.91 [19:07:37] Reception|away: yeah I already said reclone w [19:07:52] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw8 [19:08:21] Make sure to merge config changes too [19:08:31] .op [19:08:32] Attempting to OP... [19:08:44] RhinosF1: did that with the puppet commit yeah [19:08:48] will actually merge at the end [19:08:49] Reception|away: ah right [19:10:49] hmm, not sure why rm -rf w is stuck [19:10:52] it never took this long on test3 [19:11:37] seems to have finally worked now, somehow it didn't remove the cache dir and I had to manually [19:11:53] PROBLEM - mw8 MediaWiki Rendering on mw8 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 270 bytes in 0.024 second response time [19:11:54] !log rm -rf /srv/mediawiki/w and sudo -u www-data git clone https://github.com/miraheze/mediawiki --recursive w on mw8 [19:13:55] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki configuration Error - 1297 bytes in 0.010 second response time [19:15:33] Reception|away / paladox: is `--branch=REL1_36` not needed for clone? [19:15:57] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 1258 bytes in 0.009 second response time [19:16:21] oh yeh [19:16:22] oh.. true [19:17:24] Ye [19:19:49] [02mediawiki] 07Universal-Omega closed pull request 03#2238: Bump glob-parent from 5.1.1 to 5.1.2 - 13https://git.io/JZyqH [19:19:50] [02mediawiki] 07dependabot[bot] commented on pull request 03#2238: Bump glob-parent from 5.1.1 to 5.1.2 - 13https://git.io/JZ9UE [19:19:52] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/glob-parent-5.1.2 - 13https://git.io/vbL5b [19:19:53] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/glob-parent-5.1.2 [19:19:57] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki configuration Error - 1297 bytes in 0.008 second response time [19:21:56] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 1258 bytes in 0.011 second response time [19:22:46] Reception|away, you could log your commands to a local text file and log them later :P [19:23:26] yeah though here it's easier so the time is automatically there [19:24:03] true, but then it won't be in the easy-to-search Tech:Server admin log later :) [19:24:15] RECOVERY - cp13 Varnish Backends on cp13 is OK: All 7 backends are healthy [19:24:16] I'd prefer it to be logged to SAL later for easy looking back [19:24:23] RECOVERY - cp12 Varnish Backends on cp12 is OK: All 7 backends are healthy [19:24:23] !log repool mw8; depool mw9; same as above for mw8 [19:24:30] Be nice to have a bot that can "stash" your logged commands and "unstash" them when editing is possible [19:24:46] Voidwalker: 💯 oh that'd be a nice idea :) [19:24:46] Yeah that's an idea [19:24:46] yeah I can move it to SAL from here anyway [19:25:00] I wonder if we could add that to logbot easily [19:25:01] oh good idea Voidwalker :) [19:25:04] Normally we update manually after [19:25:11] dmehus: easily not really [19:25:18] oh [19:25:41] It's possible but compared to what it is now it would be a big chunk of work [19:25:58] true [19:26:18] RECOVERY - mw9 Disk Space on mw9 is OK: DISK OK - free space: / 6241 MB (32% inode=81%); [19:26:51] logbot is really barebones, so I don't know how possible it would be to modify it to do that [19:26:54] PROBLEM - mw9 MediaWiki Rendering on mw9 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 261 bytes in 0.005 second response time [19:26:56] Quick q; why not just run puppet? A full clone shouldn’t timeout anymore and there’s many other post clone tasks that need to run? [19:27:11] JohnLewis: because it didn't work well on test3 [19:27:27] It timed out? [19:27:27] JohnLewis: I did run puppet after, but it didn't work when I tried the first time with the REL1_36 branch set for the host, and also because there was some trouble when we tried on test3 [19:27:31] It ended up getting the submodules in a really inconsistent state [19:27:47] yeah.. it was strange [19:27:50] How would a fresh clone get submodules inconsistent? [19:28:06] well it was after setting https://github.com/miraheze/puppet/pull/1778/files#diff-f8f4f7e475b6648ed7cfa4557fd874f0aaa697e799f1cae098c1dcde9f70a7f5R6 in puppet (for test3) [19:28:13] No the changing branch via puppet did [19:28:15] PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 1 backends are down. mw9 [19:28:18] then we ran puppet and some strange git errors appeared [19:28:23] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 1 backends are down. mw9 [19:28:24] That's why we went for reclone [19:28:28] You shouldn’t change ran that way [19:28:41] oh? [19:28:54] but in essence running puppet would work then [19:28:54] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki configuration Error - 1297 bytes in 0.041 second response time [19:29:17] well this time even more weirdly on mw8 I did initially try to run puppet and it just ran as if the version was 1.35 [19:29:29] Voidwalker, maybe we could have Void-bot "stash" and "unstash" commands, with "unstash" essentially having Void-bot issue the command again to IRC for Logbot to log? [19:29:43] That would mean the branch change didn’t apply [19:29:55] yeah, though I have no idea why that wouldn't be the case [19:30:03] [02puppet] 07Reception123 closed pull request 03#1779: switch mw9 to REL1_36 - 13https://git.io/JZDKC [19:30:05] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JZ9LO [19:30:06] [02miraheze/puppet] 07Reception123 0391a8495 - switch mw9 to REL1_36 (#1779) [19:30:11] Was the change pulled to puppet2? [19:30:25] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.54, 6.79, 4.76 [19:30:32] Reception|away: if you merge a change to the puppet repo you to need to update puppet2 first [19:30:46] ah, that would probably be the case then. Now I remember that running puppet on puppet3 isn't enough and an extra pull somewhere has to be done [19:30:57] Yep [19:31:07] do you remember what dir it was where I'd do that? [19:32:21] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.46, 7.11, 5.13 [19:32:39] Reception|away: I think we need to do a varnish purge after [19:32:50] When loading from mw8 styling looks funny [19:32:54] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 1258 bytes in 0.019 second response time [19:32:58] RhinosF1: if you mean sidebar that was the case before too [19:33:03] when proxying from test3 [19:33:14] Reception|away: no the header at the top [19:33:31] oh yeah, paladox ^ [19:33:40] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:33:43] https://usercontent.irccloud-cdn.com/file/Q5Pso5wO/1623526417.JPG [19:33:51] test3 doesn't hit varnish [19:34:27] anyway funnily enough now for some reason cache worked perfectly fine on mw8 [19:36:14] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.27, 6.75, 5.42 [19:36:26] Edits appear to be going through in some places: https://respublica.miraheze.org/w/index.php?diff=11143&oldid=11142&rcid=10258 [19:36:39] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JZ9mw [19:36:40] [02miraheze/mw-config] 07Reception123 0305e21b8 - Update LocalSettings.php [19:36:42] [02mw-config] 07Reception123 synchronize pull request 03#3824: Update config for 1.36 - 13https://git.io/JYjbk [19:36:46] Reception|away: hold for voids comment pls [19:36:48] just fixed, didn't realise that [19:36:55] Ah ok [19:36:59] Yeah go then [19:36:59] readonly wasn't set on the REL1_36 branch [19:37:05] Yes, that's exactly what I've been seeing from the #wiki-feed. [19:37:08] Maybe rebase [19:37:24] Hold for the second [19:37:38] miraheze/mw-config - Reception123 the build passed. [19:37:50] !log repool mw9; depool mw10; repeat above [19:37:53] Yeah make sure that's deployed then go Reception|away [19:38:54] PROBLEM - mw9 MediaWiki Rendering on mw9 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 2560 bytes in 0.013 second response time [19:39:17] deployed now yeah [19:39:20] Good [19:39:25] [02puppet] 07Reception123 closed pull request 03#1780: switch mw10 to REL1_36 - 13https://git.io/JZDKu [19:39:26] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JZ9Ya [19:39:28] [02miraheze/puppet] 07Reception123 035da27dd - switch mw10 to REL1_36 (#1780) [19:40:05] Universal_Omega: half way through :) [19:40:14] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.58, 8.66, 6.42 [19:40:18] Great! [19:40:41] 21 edits went through wrongly for the record [19:40:55] PROBLEM - mw10 MediaWiki Rendering on mw10 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 261 bytes in 0.007 second response time [19:41:25] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 1.40, 4.94, 7.68 [19:42:55] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki configuration Error - 1297 bytes in 0.006 second response time [19:43:22] RhinosF1: Universal_Omega hmm it seems cache worked on mw8 but not mw9, what do we do? [19:43:48] Reception|away, oh you didn't set wgReadOnly on the 1_36 branch :( [19:43:50] Reception|away: have you tan composer install in w [19:43:57]