[00:03:27] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 3 minutes ago with 6 failures. Failed resources (up to 3 shown) [00:21:31] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.50, 23.06, 19.03 [00:27:21] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.31, 20.30, 19.12 [00:31:21] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [00:37:17] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.80, 20.13, 19.05 [00:39:16] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.24, 23.27, 20.32 [00:41:16] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.04, 23.05, 20.68 [00:43:15] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.56, 19.78, 19.80 [05:20:40] Yeah... I was wondering the same thing. It's definitely a technical issue / bug, but I wondered whether it was a security-related issue that needed a CVE? [05:21:44] :thistbh: [05:23:52] Yeah, I think it was prudent to disable the extension due to the potential system load issues, but I'm also not satisfied a CVE was required in this case [05:27:12] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.96, 1.80, 1.10 [05:28:09] @RhinosF1, for what it's worth, I don't think @Owen is questioning the actions SRE took in disabling GlobalNewFiles. The concern here is the process by which it's determined whether a CVE is required. It seems very hodge podge, essentially. That's what we need to move forward on improving / revising, I think. The extension had clear bugs and performance issues that need to be resolved, so disabling it was prudent; the issue is [05:28:10] with the CVE [05:29:09] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.95, 1.41, 1.04 [05:30:58] I don't even agree with the publishing of that CVE at the time I think it should've waited until a full investigation was done to provide more details but what's done is done and I also can't complain since I didn't reply in time when asked to review it when RhinosF1 pinged all SRE members. I do however also agree in actions taken to disable the extension though. [05:32:39] Yeah, when I saw that, I was surprised a CVE was issued for that. It's a definite bug and the extension was rightly disabled due to the load issues, but I'm struggling to see how this makes it a critical vulnerability [05:34:24] Yeah... happened once in all the time it's been enabled isn't the worst issue we've had and I think that CVE was just published just to publish it. I'm not 100% satisfied with the way it was handled, and definitely agree with disabling it but a CVE being required at all is very questionable. [05:34:36] True [05:50:47] PROBLEM - en.famepedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.famepedia.org' expires in 15 day(s) (Tue 13 Jul 2021 05:45:30 GMT +0000). [05:58:21] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JcJJc [05:58:22] [02miraheze/ssl] 07MirahezeSSLBot 03eff2119 - Bot: Update SSL cert for en.famepedia.org [06:31:27] RECOVERY - en.famepedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.famepedia.org' will expire on Sat 25 Sep 2021 04:58:14 GMT +0000. [07:49:10] [02mw-config] 07R4356th opened pull request 03#3984: Add WMF Commons as an Import Source for snapdatawiki - 13https://git.io/JcJlA [07:49:23] [02mw-config] 07R4356th edited pull request 03#3984: Add WMF Commons as an Import Source for snapdatawiki - 13https://git.io/JcJlA [07:50:01] miraheze/mw-config - R4356th the build passed. [07:51:19] [02mw-config] 07Universal-Omega closed pull request 03#3984: Add WMF Commons as an Import Source for snapdatawiki - 13https://git.io/JcJlA [07:51:21] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JcJ8a [07:51:22] [02miraheze/mw-config] 07R4356th 03df72408 - Add WMF Commons as an import source for snapdatawiki (#3984) [07:52:14] miraheze/mw-config - Universal-Omega the build passed. [12:30:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JcUnz [12:30:11] [02miraheze/services] 07MirahezeSSLBot 039e4224f - BOT: Updating services config for wikis [12:35:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JcUc0 [12:35:12] [02miraheze/services] 07MirahezeSSLBot 0384de341 - BOT: Updating services config for wikis [12:50:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JcUlv [12:50:11] [02miraheze/services] 07MirahezeSSLBot 0387c77ea - BOT: Updating services config for wikis [13:05:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JcU4A [13:05:10] [02miraheze/services] 07MirahezeSSLBot 03d6522eb - BOT: Updating services config for wikis [13:10:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JcUBN [13:10:12] [02miraheze/services] 07MirahezeSSLBot 03fd16079 - BOT: Updating services config for wikis [13:20:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JcU0D [13:20:11] [02miraheze/services] 07MirahezeSSLBot 039ab0e70 - BOT: Updating services config for wikis [13:25:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JcUue [13:25:10] [02miraheze/services] 07MirahezeSSLBot 033a6d160 - BOT: Updating services config for wikis [13:29:17] !log DELETED and DROPPED https://phabricator.miraheze.org/P437 [13:29:18] [url] ✎ P437 Deleted wikis 27/06/2021 | phabricator.miraheze.org [13:29:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:50:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JcU2h [13:50:10] [02miraheze/services] 07MirahezeSSLBot 0363e43ca - BOT: Updating services config for wikis [14:26:02] [02mw-config] 07R4356th opened pull request 03#3985: Configure ``wgSnapwikiskinWvuiSearchOptions`` - 13https://git.io/JcU6I [14:26:53] miraheze/mw-config - R4356th the build passed. [15:34:56] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.37, 6.19, 5.02 [15:36:56] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.58, 5.31, 4.84 [16:05:01] !log GDPR deletion [16:05:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:22:59] [02miraheze/GlobalNewFiles] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JcTv2 [16:23:01] [02miraheze/GlobalNewFiles] 07RhinosF1 030bc6baa - Hooks: don't actually make dbw changes [16:23:02] [02GlobalNewFiles] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/JtShO [16:23:04] [02GlobalNewFiles] 07RhinosF1 opened pull request 03#11: Hooks: don't actually make dbw changes - 13https://git.io/JcTva [16:24:01] miraheze/GlobalNewFiles - RhinosF1 the build passed. [16:26:24] [02GlobalNewFiles] 07OwenBaines commented on pull request 03#11: Hooks: don't actually make dbw changes - 13https://git.io/JcTfW [16:27:19] @Owen: it's not decided whether we will allow showing stale data yet [16:27:56] But if you remove the write aspects just to enable the extension, what value does it add if it does nothing and you remove the stale data? [16:31:44] @Owen: the stale data ain't removed yet. We need to work out how to unstale the data too. [16:32:05] It's already gonna be over a day of a drift if it was enabled again now [16:32:48] But you just said you don't know whether you will allow showing the stale data, and now you're saying you're not removing it. I'm confused. But either way, whether you do or do not remove it, if the write aspect is removed, the extension might as well stay removed. [16:35:18] Oh we don't know whether that PR is being merged and it's probably not likely unless someone in the community sees a use but the tables still exist [16:35:42] We still have that data stored accurate to yesterday [16:36:25] [02mw-config] 07Universal-Omega closed pull request 03#3985: Configure ``wgSnapwikiskinWvuiSearchOptions`` - 13https://git.io/JcU6I [16:36:25] I've just raised a point over why there is no use re-enabling the extension with stale data with no new data being added. So the focus should adjust to solutions rather than getting it enabled at the cost of use. [16:36:26] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JcTUr [16:36:28] [02miraheze/mw-config] 07R4356th 031d54e32 - Configure ``wgSnapwikiskinWvuiSearchOptions`` (#3985) [16:37:17] miraheze/mw-config - Universal-Omega the build passed. [16:37:57] [02GlobalNewFiles] 07RhinosF1 commented on pull request 03#11: Hooks: don't actually make dbw changes - 13https://git.io/JcTUA [16:37:58] [02GlobalNewFiles] 07RhinosF1 closed pull request 03#11: Hooks: don't actually make dbw changes - 13https://git.io/JcTva [16:38:01] [02GlobalNewFiles] 07RhinosF1 deleted branch 03RhinosF1-patch-1 - 13https://git.io/JtShO [16:38:02] [02miraheze/GlobalNewFiles] 07RhinosF1 deleted branch 03RhinosF1-patch-1 [16:44:11] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JcTku [16:44:12] [02miraheze/GlobalNewFiles] 07Universal-Omega 03cc1d77d - Update GlobalNewFilesHooks.php [16:44:13] [02GlobalNewFiles] 07Universal-Omega synchronize pull request 03#10: Use DBLoadBalancerFactory - 13https://git.io/JcvLH [16:45:06] miraheze/GlobalNewFiles - Universal-Omega the build passed. [16:45:16] [02GlobalNewFiles] 07Universal-Omega closed pull request 03#10: Use DBLoadBalancerFactory - 13https://git.io/JcvLH [16:45:18] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JcTki [16:45:19] [02miraheze/GlobalNewFiles] 07Universal-Omega 03a9707df - Use DBLoadBalancerFactory (#10) [16:45:21] [02GlobalNewFiles] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/JtShO [16:45:22] [02miraheze/GlobalNewFiles] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [16:46:08] miraheze/GlobalNewFiles - Universal-Omega the build passed. [17:56:16] [02miraheze/services] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JcTCl [17:56:18] [02miraheze/services] 07Reception123 0397c4b15 - temporarily remove test3 from services (testing) [18:00:15] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JcTWf [18:00:17] [02miraheze/services] 07MirahezeSSLBot 033f5950f - BOT: Updating services config for wikis [18:31:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.96, 19.14, 15.45 [18:32:15] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.62, 5.12, 3.39 [18:34:10] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.32, 4.81, 3.47 [18:35:31] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.99, 17.76, 15.87 [18:48:33] Reception123: what were you testing? [18:54:02] RhinosF1: https://phabricator.miraheze.org/T7416 [18:54:03] [url] ⚓ T7416 Linter doesn’t work | phabricator.miraheze.org [18:54:59] Reception123: did you find anything? [18:55:15] RhinosF1: well Universal_Omega is onto something and thinks it's because we changed to php parsoid [18:55:27] Ah [18:55:36] That's been broken a while then [18:55:41] How do uodtream do it [18:56:15] dmehus: sorry about the custom domains [18:57:18] It's probably wgLinterSubmitterWhitelist RhinosF1 - it's pointed to services servers which Linter doesn't use anymore I don't think, so wouldn't need to be on that. But that's just an idea. Also Re custom domain: jacksonheights was removed 20 days ago by paladox it seems. [18:59:19] Universal_Omega: yeah a batch was removed. We should check everything I guess. [18:59:21] And ah [19:01:55] At least linter is an easy fix [19:02:12] Just add the right IPs and update docs [19:02:42] Yeah. I will do PR to do it in a minute probably and then it can be tested on test3 and deployed and fixed. [19:04:37] Universal_Omega: any reason https://phabricator.miraheze.org/tag/services/ still has #parsoid [19:04:38] [url] Services · Manage | phabricator.miraheze.org [19:08:52] RhinosF1, ack [19:08:55] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JcT2T [19:08:56] [02miraheze/mw-config] 07Universal-Omega 03de47b7f - Update wgLinterSubmitterWhitelist to mw* and test3 [19:08:58] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [19:08:59] [02mw-config] 07Universal-Omega opened pull request 03#3986: Update wgLinterSubmitterWhitelist to mw* and test3 - 13https://git.io/JcT2k [19:09:28] Universal_Omega: what about jobrunner? [19:09:57] miraheze/mw-config - Universal-Omega the build passed. [19:09:59] RhinosF1: does Jobrunner need it? [19:10:17] Universal_Omega: I'm not 100% [19:10:24] RhinosF1: And also #parsoid should be removed from that. [19:10:30] Can we add comments though saying which ip is which [19:10:40] Yeah I wanted to do that also, so will do. [19:11:09] Updated [19:11:22] Yeah saves future selves an extra job [19:13:02] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JcT2X [19:13:04] [02miraheze/mw-config] 07Universal-Omega 03db68280 - Update LocalSettings.php [19:13:05] [02mw-config] 07Universal-Omega synchronize pull request 03#3986: Update wgLinterSubmitterWhitelist to mw* and test3 - 13https://git.io/JcT2k [19:13:11] RhinosF1: ^ [19:13:58] miraheze/mw-config - Universal-Omega the build passed. [19:14:21] Lgtm [19:48:36] Voidwalker: what do you think about https://github.com/miraheze/mw-config/pull/3986/files ? trying to resolve https://phabricator.miraheze.org/T7416 [19:48:37] [url] Update wgLinterSubmitterWhitelist to mw* and test3 by Universal-Omega · Pull Request #3986 · miraheze/mw-config · GitHub | github.com [19:48:37] [url] ⚓ T7416 Linter doesn’t work | phabricator.miraheze.org [19:48:48] and it seems to be because parsoid was switched to php rather than services [19:53:58] Reception123, not sure that's the only problem, lol [19:54:06] [02mw-config] 07The-Voidwalker reviewed pull request 03#3986 commit - 13https://git.io/JcToN [19:54:48] [02mw-config] 07Universal-Omega reviewed pull request 03#3986 commit - 13https://git.io/JcTKJ [19:57:12] Voidwalker: what else do you think there is? it did work at some point [19:58:50] And I don't think it has since we switched to PHP Parsoid actually. It would make sense really. [19:59:08] And that'd definitely be an issue since it doesn't use services anymore. [19:59:33] And when ip doesn't match something in that error LintErrors aren't ever updated. [20:02:32] [02mw-config] 07Universal-Omega edited pull request 03#3986: Update wgLinterSubmitterWhitelist to mw* and test3 - 13https://git.io/JcT2k [20:16:37] [02mw-config] 07The-Voidwalker reviewed pull request 03#3986 commit - 13https://git.io/JcTPD [20:17:31] seems like that mobilefrontend problem is back again [20:20:27] What is that please ? [20:20:39] CSS set on Mobile.css not loading [20:20:57] iirc, this is a cache issue on MH side [20:21:07] or am I misremembering things [20:22:04] upstream task hasn't been resolved: https://phabricator.wikimedia.org/T265566 [20:22:05] [url] ⚓ T265566 useformat=mobile does not make proper CSS/JS changes | phabricator.wikimedia.org [20:22:57] seems like it will take ages to solve [20:23:10] it sucks that apparently mobile.js also doesn't work [20:23:50] correction, is probably: https://phabricator.wikimedia.org/T270603 [20:23:51] [url] ⚓ T270603 Module site.styles generates different output depending on mobile cookie, if $wgMFSiteStylesRenderBlocking = true; | phabricator.wikimedia.org [20:27:28] [02mw-config] 07Universal-Omega reviewed pull request 03#3986 commit - 13https://git.io/JcT1O [20:27:49] [02mw-config] 07Universal-Omega reviewed pull request 03#3986 commit - 13https://git.io/JcT1O [20:28:32] Wow [20:28:48] idk exaclty what can be done in this situation [20:29:42] @Lake: the issue was never resolved. Its been an issue since 1.35.0 upgrade and never was working since then. [20:30:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JcT12 [20:30:11] [02miraheze/services] 07MirahezeSSLBot 03a2b987e - BOT: Updating services config for wikis [20:30:14] I thought it was working again [20:30:35] Not that I ever noticed. [20:30:44] I worked at a time tho [20:30:47] but what I don't understand is why some wikis outside of Miraheze are not suffering from this [20:30:55] (on >1.35) [20:31:26] feels a bit random, lol [20:31:58] @Lake: I'm not sure but some wikis outside of Miraheze do have the issue. @Ugochimobi: yes it was before we upgraded to 1.35 awhile ago. [20:32:18] yeah, when working on WiKirby I noticed that [20:36:19] I see [20:38:05] But I'm not sure all miraheze wikis are affected tho [20:51:15] just one thing: does templatestyles (or extension CSS) work on mobilefrontend? [20:59:02] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.82, 6.32, 5.12 [21:03:02] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.26, 7.18, 5.69 [21:04:56] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.61, 6.36, 5.14 [21:05:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.66, 7.79, 6.13 [21:06:56] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.48, 6.01, 5.16 [21:09:02] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.79, 6.60, 6.05 [22:38:47] [02mw-config] 07Universal-Omega synchronize pull request 03#3986: Update wgLinterSubmitterWhitelist to mw* and test3 - 13https://git.io/JcT2k [22:38:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JcTht [22:38:50] [02miraheze/mw-config] 07Universal-Omega 03166f0ae - Update LocalSettings.php [22:39:40] miraheze/mw-config - Universal-Omega the build passed. [23:27:09] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jckfh [23:27:11] [02miraheze/mw-config] 07Universal-Omega 03311564f - enable logging for 'Linter' [23:27:58] miraheze/mw-config - Universal-Omega the build passed.