[00:00:21] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 11.11, 7.39, 6.39 [00:01:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.85, 3.30, 2.86 [00:03:01] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 4.89, 3.27, 2.22 [00:03:34] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.63, 7.87, 7.57 [00:04:49] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.15, 7.25, 6.15 [00:05:01] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.74, 3.35, 2.37 [00:05:07] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.47, 3.35, 2.98 [00:06:00] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:06:06] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.65, 8.37, 6.62 [00:06:10] PROBLEM - cp15 Stunnel Http for mw9 on cp15 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:06:11] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:07:01] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.41, 3.13, 2.42 [00:07:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.00, 3.23, 2.98 [00:07:24] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.56, 6.93, 5.68 [00:07:25] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::1d3/cpweb [00:07:25] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::58af/cpweb [00:08:06] RECOVERY - cp15 Stunnel Http for mw9 on cp15 is OK: HTTP OK: HTTP/1.1 200 OK - 15860 bytes in 0.326 second response time [00:08:10] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 20009 bytes in 0.771 second response time [00:09:25] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.77, 8.30, 6.34 [00:11:00] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.61, 3.60, 3.21 [00:11:00] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:11:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:11:25] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:12:06] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.79, 7.83, 7.15 [00:12:08] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.46, 6.96, 6.09 [00:12:38] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.71, 6.65, 6.65 [00:12:58] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.94, 4.01, 3.40 [00:14:02] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.61, 6.35, 5.96 [00:14:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.39, 3.46, 3.28 [00:15:24] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 4.70, 7.14, 6.61 [00:15:35] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 3.70, 6.31, 7.53 [00:16:06] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.80, 5.72, 6.45 [00:18:52] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.37, 3.34, 3.27 [00:19:24] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.56, 6.09, 6.32 [00:21:01] PROBLEM - cp15 Current Load on cp15 is WARNING: WARNING - load average: 1.65, 1.71, 1.24 [00:23:01] RECOVERY - cp15 Current Load on cp15 is OK: OK - load average: 0.95, 1.47, 1.21 [00:23:34] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.14, 5.36, 6.60 [00:35:01] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 4.84, 3.69, 3.05 [00:37:01] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.39, 3.50, 3.06 [00:39:01] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 1.46, 2.72, 2.82 [00:45:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.88, 3.40, 3.20 [00:49:40] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.23, 3.50, 3.27 [00:52:58] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.84, 1.51, 1.10 [00:53:40] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.27, 3.27, 3.26 [00:54:58] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.18, 1.31, 1.07 [01:02:21] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.48, 4.01, 5.69 [01:08:01] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 15.09, 18.01, 22.79 [01:08:21] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.14, 3.32, 4.91 [01:11:40] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.28, 3.71, 3.42 [01:12:05] alerting : [FIRING:1] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:13:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.97, 3.55, 3.40 [01:15:40] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.69, 3.29, 3.32 [01:16:01] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.23, 15.89, 19.82 [01:17:05] ok : [RESOLVED] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:21:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.17, 3.51, 3.41 [01:23:40] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.66, 3.00, 3.22 [01:44:07] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.70, 20.36, 18.22 [01:46:02] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.05, 19.01, 18.04 [01:53:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.40, 3.25 [01:55:40] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.25, 3.48, 3.28 [01:57:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.57, 3.50, 3.31 [01:59:40] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.60, 3.27, 3.26 [02:01:53] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.28, 6.96, 5.63 [02:03:40] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.48, 3.71, 3.42 [02:03:51] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.09, 7.58, 6.01 [02:04:01] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.15, 20.28, 18.11 [02:04:07] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 15.35, 8.40, 5.51 [02:06:01] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.05, 19.91, 18.26 [02:07:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.63, 3.57, 3.46 [02:10:06] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.75, 7.72, 6.52 [02:11:38] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.50, 7.59, 6.83 [02:11:40] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.50, 3.22, 3.36 [02:12:06] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.97, 6.79, 6.32 [02:15:32] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.82, 8.74, 7.45 [02:15:48] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.21, 6.56, 5.70 [02:19:36] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.56, 7.18, 6.12 [02:19:40] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.56, 3.88, 3.55 [02:21:30] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.76, 7.32, 6.29 [02:23:19] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.05, 7.02, 7.22 [02:23:24] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.93, 6.55, 6.14 [02:23:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.67, 3.77, 3.58 [02:27:40] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.71, 3.18, 3.38 [02:28:06] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.88, 7.86, 6.54 [02:28:46] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.70, 20.77, 18.49 [02:29:09] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.30, 5.59, 6.59 [02:30:42] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.50, 20.25, 18.58 [02:33:03] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.24, 6.45, 6.71 [02:33:24] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.93, 6.46, 6.13 [02:33:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.84, 3.39, 3.44 [02:35:03] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.12, 5.53, 6.34 [02:35:24] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.20, 6.04, 6.02 [02:36:27] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.44, 22.74, 20.07 [02:37:03] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.69, 6.81, 5.70 [02:37:40] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.05, 2.92, 3.26 [02:38:22] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.46, 21.89, 20.08 [02:39:01] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.47, 6.21, 5.61 [02:40:17] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.14, 19.61, 19.47 [02:42:06] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 3.83, 7.35, 7.77 [02:46:06] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.19, 5.01, 6.70 [03:56:55] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.31, 8.69, 5.69 [03:58:03] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.44, 6.51, 4.65 [04:00:01] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.05, 19.51, 16.18 [04:02:01] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.04, 19.27, 16.50 [04:02:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.07, 6.96, 5.28 [04:03:42] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.13, 5.98, 4.81 [04:04:04] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.49, 7.96, 5.86 [04:05:41] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 16.38, 8.98, 5.99 [04:06:00] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:06:01] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.41, 26.14, 20.01 [04:06:10] PROBLEM - cp15 Stunnel Http for mw9 on cp15 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:08:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.61, 7.75, 6.30 [04:08:05] RECOVERY - cp15 Stunnel Http for mw9 on cp15 is OK: HTTP OK: HTTP/1.1 200 OK - 15874 bytes in 0.367 second response time [04:10:01] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.08, 22.39, 20.02 [04:11:00] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:12:03] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.14, 5.70, 5.83 [04:13:34] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.42, 7.87, 7.00 [04:15:34] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.76, 6.79, 6.71 [04:16:01] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.36, 18.55, 19.11 [04:30:06] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.83, 6.23, 7.87 [04:40:06] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.06, 5.09, 6.53 [04:54:42] SRE, likely transitory, but just got a 503 on `metawiki` when trying to use userrights-interwiki [04:54:42] [tell] dmehus: 2021-10-26 - 22:57:32UTC tell dmehus to look into 엔펄 (the one who confessed to being Angel Pearl, a locked account, based on this confession: https://jayuvandal.miraheze.org/w/index.php?title=%EC%82%AC%EC%9A%A9%EC%9E%90%ED%86%A0%EB%A1%A0:%EC%97%94%ED%8E%84&oldid=25658), Azulizc and Ilovedogs34, based on the diffs from this #cvt channel, and finally, the potential [05:10:17] dmehus: hmm.. seems like cp15 went down. Seems to have self-recovered though. [06:01:29] CosmicAlpha, ah, thanks for looking. Yeah, seems to be a series of recurring, though short-term, issues [06:01:36] though not sure what the cause is [06:01:48] but at least it is very short term [06:03:09] dmehus: yeah... it does seem to keep happening lately. [06:04:39] yeah :( [06:06:58] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.08, 6.58, 4.14 [06:14:58] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.49, 7.35, 5.61 [06:24:58] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.69, 6.80, 6.17 [06:44:34] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/Ji0Ho [06:44:36] [02miraheze/mw-config] 07Universal-Omega 03ffd1e4e - Unrestrict Evelution [06:44:37] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [06:44:55] [02mw-config] 07Universal-Omega opened pull request 03#4182: Unrestrict Evelution - 13https://git.io/Ji0Q1 [06:45:51] miraheze/mw-config - Universal-Omega the build passed. [07:02:25] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.71, 7.41, 6.69 [07:04:23] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.01, 6.95, 6.61 [07:06:21] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.60, 7.69, 6.92 [07:12:14] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.03, 7.20, 6.96 [07:24:01] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.57, 6.64, 6.79 [07:47:30] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.63, 7.24, 6.82 [07:55:22] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.95, 6.57, 6.71 [07:59:15] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.70, 6.84, 6.80 [08:01:13] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.30, 6.73, 6.77 [08:09:08] PROBLEM - db13 Current Load on db13 is CRITICAL: CRITICAL - load average: 9.33, 6.08, 4.49 [08:11:07] RECOVERY - db13 Current Load on db13 is OK: OK - load average: 3.25, 5.12, 4.35 [08:15:43] @CosmicAlpha the otherProjects stuff doesn't work yet [08:36:01] Joseph: I don't think it will work. I'm confused on how it's supposed to but I'm not sure it will work the way you want. But I'll look further. [08:41:21] It is actually supposed to work, I know you don't understand Wikibase perfectly, bur, We'll work it through together. [08:44:04] Joseph: Your PR just set the same default. I know for certain that would have no effect. I'm not certain but what you want might be blocked on https://phabricator.miraheze.org/T6045 / https://github.com/miraheze/MirahezeMagic/pull/274 which I have no idea how to test if I did fully right though I'll try. But yeah, I don't really understand Wikibase to much, so I'm not sure how to continue here. [08:44:04] [url] ⚓ T6045 Maintain default wikimedia sites when running populateWikibaseSitesTables | phabricator.miraheze.org [08:44:06] [url] T6045: Maintain default Wikimedia sites by Universal-Omega · Pull Request #274 · miraheze/MirahezeMagic · GitHub | github.com [08:46:16] [02mw-config] 07Universal-Omega closed pull request 03#4182: Unrestrict Evelution - 13https://git.io/Ji0Q1 [08:46:17] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [08:46:19] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Ji293 [08:46:20] [02miraheze/mw-config] 07Universal-Omega 0358f4d1f - Unrestrict Evelution (#4182) [08:46:22] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [08:47:24] miraheze/mw-config - Universal-Omega the build passed. [08:51:22] Why is it blocked btw? [08:55:25] Joseph: Just meant it might not be able to be done until that patch is working and deployed. Though I really just don't know. I hate Wikibase. It's so complicated to configure. I know how to use it (somewhat) but configuring it is complicated.... [09:03:09] !log [@test3] starting deploy of {'config': True} to skip [09:03:10] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [09:03:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:03:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:03:45] I don't really think that could make it unable to work though [09:04:14] !log [@mw11] starting deploy of {'config': True} to all [09:04:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:04:34] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 19s [09:04:35] But, It isn't an UBN request, so.. we can move step by step, probably when that is deployed or whenever [09:04:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:05:00] Joseph: Well maybe give me examples of how you want it to work, and I'll test the config on test3 and proxy to your wiki to see if it's doing what you want. [09:06:37] DM me on Discord if you need to also. [09:06:45] I'm going for tonight now. [09:11:59] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/Jia1k [09:12:01] [02miraheze/mw-config] 07RhinosF1 03da5b15a - Update ManageWikiSettings.php [09:12:02] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/vbvb3 [09:12:04] [02mw-config] 07RhinosF1 opened pull request 03#4183: Update ManageWikiSettings.php - 13https://git.io/Jia1n [09:12:26] [02mw-config] 07RhinosF1 edited pull request 03#4183: Add Tweeki settings - 13https://git.io/Jia1n [09:13:15] miraheze/mw-config - RhinosF1 the build passed. [09:15:27] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JiaQa [09:15:29] [02miraheze/mw-config] 07RhinosF1 03fa47ddb - Update LocalSettings.php [09:15:30] [02mw-config] 07RhinosF1 synchronize pull request 03#4183: Add Tweeki settings - 13https://git.io/Jia1n [09:15:49] Reception123: review pls ^ [09:16:32] miraheze/mw-config - RhinosF1 the build passed. [09:41:35] Alright [09:49:47] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JiVhb [09:49:48] [02miraheze/mw-config] 07RhinosF1 0304f512b - AbuseFilter: Enable GAF from test3 [09:50:57] !log [@test3] starting deploy of {'config': True} to skip [09:50:58] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [09:50:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:50:59] miraheze/mw-config - RhinosF1 the build passed. [09:51:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:57:27] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JiwOi [09:57:28] [02miraheze/mw-config] 07RhinosF1 03015df16 - test3: use own bot passwords DB [09:58:17] !log [@test3] starting deploy of {'config': True} to skip [09:58:18] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [09:58:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:58:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:58:33] miraheze/mw-config - RhinosF1 the build passed. [10:04:13] !log [@mw11] starting deploy of {'config': True} to all [10:04:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:04:28] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 15s [10:04:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:00:35] [02puppet] 07RhinosF1 opened pull request 03#2096: add grant for test3 global database - 13https://git.io/Ji6Wd [12:00:52] JohnLewis: there's no Reception123 today but i created ^ [12:01:37] paladox: ^^ as I’m not at a computer [12:02:06] there's no rush as i'm off in like 10 minutes for a few hours [12:18:43] PROBLEM - thestarsareright.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'thestarsareright.org' expires in 7 day(s) (Thu 04 Nov 2021 12:12:37 GMT +0000). [12:53:40] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.45, 3.48, 2.79 [12:55:40] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.46, 3.10, 2.74 [13:07:02] [02puppet] 07paladox closed pull request 03#2096: add grant for test3 global database - 13https://git.io/Ji6Wd [13:07:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JiP7b [13:07:05] [02miraheze/puppet] 07RhinosF1 0321bf191 - add grant for test3 global database (#2096) [13:07:48] !log create testglobal on db11 [13:07:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:07:57] !log apply some grants on db11 [13:08:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:08:07] Spookreeeno: done [13:45:00] Error 503 Backend fetch failed, forwarded for (my ip) , 127.0.0.1 (Varnish XID 96637501) via cp15.miraheze.org at Wed, 27 Oct 2021 13:43:53 GMT. [13:46:20] Fine now [13:50:26] Same, I got a 503 from cp15 too [13:50:35] Should be fine now [14:24:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.91, 3.24, 2.81 [14:26:25] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.92, 3.01, 2.77 [14:36:12] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.24, 3.66, 3.20 [14:38:11] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.44, 3.23, 3.10 [14:44:21] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.32, 5.30, 3.86 [14:46:21] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.63, 5.63, 4.15 [14:46:59] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.72, 3.55, 3.34 [14:48:22] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.03, 5.41, 4.25 [14:48:57] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.97, 3.30, 3.26 [14:50:21] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.27, 4.40, 4.00 [14:52:52] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.80, 4.15, 3.62 [14:54:50] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.65, 3.90, 3.59 [14:56:48] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.48, 4.17, 3.73 [15:00:44] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.36, 3.66, 3.61 [15:12:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.17, 2.98, 3.30 [15:16:59] paladox: Unknown database 'testglobal' (db11) [15:17:30] Spookreeeno: try now [15:22:22] !log rhinos@test3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/GlobalBlocking/sql/mysql/tables-generated-globalblocks.sql --wiki=test3wiki --wikidb=testglobal [15:22:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:23:16] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JiyyP [15:23:17] [02miraheze/mw-config] 07RhinosF1 038df9c47 - Update LocalSettings.php [15:24:14] !log [@test3] starting deploy of {'config': True} to skip [15:24:15] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [15:24:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:24:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:24:29] miraheze/mw-config - RhinosF1 the build passed. [15:26:13] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jiydo [15:26:15] [02miraheze/mw-config] 07RhinosF1 035e723df - Test3: use own urlshortcodes [15:27:07] !log [@test3] starting deploy of {'config': True} to skip [15:27:08] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [15:27:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:27:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:27:30] miraheze/mw-config - RhinosF1 the build passed. [15:28:44] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jiyp0 [15:28:46] [02miraheze/mw-config] 07RhinosF1 0361196e0 - OAUTH/test3: use local DB [15:29:34] !log [@test3] starting deploy of {'config': True} to skip [15:29:35] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [15:29:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:29:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:29:57] miraheze/mw-config - RhinosF1 the build passed. [15:30:30] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JiSvV [15:30:32] [02miraheze/mw-config] 07RhinosF1 03775161f - CU/test3: apply globalblocks via test3 [15:31:24] !log [@test3] starting deploy of {'config': True} to skip [15:31:25] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [15:31:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:31:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:31:45] miraheze/mw-config - RhinosF1 the build passed. [15:33:48] What is Connect-SRC in the CSP. [15:33:55] ./? [15:33:57] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://git.io/JiSt0 [15:33:59] [02miraheze/mw-config] 07RhinosF1 03c5b6a26 - Database: introduce db cluster for beta [15:34:00] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-2 - 13https://git.io/vbvb3 [15:34:05] s/./? [15:34:05] Bongo-Cat meant to say: ?/? [15:34:10] Bongo-Cat: websockets i think [15:34:10] oop- [15:34:17] Okay. [15:34:21] [02mw-config] 07RhinosF1 opened pull request 03#4184: Database: introduce db cluster for beta - 13https://git.io/JiSqW [15:34:27] Thanks. [15:34:52] !log [@mw11] starting deploy of {'config': True} to all [15:34:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:35:01] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://git.io/JiSmF [15:35:03] [02miraheze/mw-config] 07RhinosF1 03c2a3a37 - Update Database.php [15:35:04] [02mw-config] 07RhinosF1 synchronize pull request 03#4184: Database: introduce db cluster for beta - 13https://git.io/JiSqW [15:35:05] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 13s [15:35:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:35:31] miraheze/mw-config - RhinosF1 the build has errored. [15:36:06] miraheze/mw-config - RhinosF1 the build passed. [15:38:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.01, 3.52, 3.32 [15:40:03] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.79, 3.30, 3.26 [15:43:17] [02mw-config] 07RhinosF1 closed pull request 03#4184: Database: introduce db cluster for beta - 13https://git.io/JiSqW [15:43:19] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JiSz0 [15:43:20] [02miraheze/mw-config] 07RhinosF1 039e91ade - Database: introduce db cluster for beta (#4184) [15:43:22] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-2 [15:43:23] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-2 - 13https://git.io/vbvb3 [15:44:50] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JiSax [15:44:52] [02miraheze/mw-config] 07RhinosF1 03fdb5f07 - Echo: move test3 to own cluster [15:44:57] miraheze/mw-config - RhinosF1 the build passed. [15:45:42] !log [@test3] starting deploy of {'config': True} to skip [15:45:43] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 1s [15:45:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:45:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:45:58] miraheze/mw-config - RhinosF1 the build passed. [16:04:17] !log [@mw11] starting deploy of {'config': True} to all [16:04:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:04:29] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 12s [16:04:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:18:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.60, 2.85, 2.76 [16:20:19] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.92, 2.94, 2.81 [16:26:36] majavah: CentralAuth is throwing an exception when accounts created by email set their password [16:26:42] It does everything fine though [16:27:00] I will get you a trace in a bit but it was something to do with the global account not existing [16:40:55] Global user 'PowerPunch' does not exist. [16:41:10] https://www.irccloud.com/pastebin/xzBwwAmj/ [16:48:38] hm, seems like test3 is not loading [16:48:55] the main page at least [16:51:17] @Lake: fine to me, any errors? [16:51:43] it shows a MW DBQuery error for me [16:51:48] MediaWiki internal error. Original exception: [0f73332ba9b35dc7773a63ea] 2021-10-27 16:51:28: Fatal exception of type "Wikimedia\Rdbms\DBQueryError" Exception caught inside exception handler. Set $wgShowExceptionDetails = true; at the bottom of LocalSettings.php to show detailed debugging information. [16:51:54] on the main page specifically [16:52:04] other pages load normally [16:52:17] I've seen this error happening on other wikis but I don't remember the cause [16:53:08] paladox: Error 1044 from Wikimedia\Rdbms\DatabaseMysqlBase::doSelectDomain, Access denied for user 'mediawiki'@'%' to database 'globaltest' (db11) USE `globaltest` db11 [16:53:44] oh [16:53:48] that may be me [16:54:21] the grants are applied on db11 [16:54:51] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JiQHL [16:54:52] [02miraheze/mw-config] 07RhinosF1 034cb40d7 - Update LocalSettings.php [16:54:59] paladox: ye my fault [16:56:02] !log [@test3] starting deploy of {'config': True} to skip [16:56:03] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [16:56:05] miraheze/mw-config - RhinosF1 the build passed. [16:56:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:56:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:56:18] @Lake: should be fixed [16:56:57] thanks :) [17:04:14] !log [@mw11] starting deploy of {'config': True} to all [17:04:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:04:25] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 11s [17:04:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:54:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.79, 3.29, 2.89 [17:56:39] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.43, 2.94, 2.80 [17:58:13] [02mw-config] 07Universal-Omega reviewed pull request 03#4183 commit - 13https://git.io/Ji5ba [17:58:14] [02mw-config] 07Universal-Omega reviewed pull request 03#4183 commit - 13https://git.io/Ji5bw [18:06:58] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.74, 4.65, 3.85 [18:08:55] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.92, 3.93, 3.68 [18:14:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.26, 3.53, 3.21 [18:22:11] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.81, 3.35, 3.28 [18:25:35] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JidzZ [18:25:36] [02miraheze/ManageWiki] 07Universal-Omega 03ff442d9 - Update SpecialManageWiki.php [18:25:38] [02ManageWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vpSns [18:26:01] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.70, 21.20, 17.79 [18:26:23] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.50, 6.00, 4.46 [18:28:01] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.48, 20.25, 17.83 [18:28:15] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/Jidar [18:28:16] [02miraheze/ManageWiki] 07Universal-Omega 03c2284a7 - Update ManageWikiFormFactory.php [18:28:21] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.95, 5.94, 4.63 [18:32:01] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.34, 18.62, 17.92 [18:32:21] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.32, 5.01, 4.56 [18:46:36] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.70, 21.85, 19.94 [18:47:23] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.93, 7.71, 5.85 [18:48:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.82, 6.27, 5.03 [18:49:23] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JidN1 [18:49:24] [02miraheze/ManageWiki] 07Universal-Omega 035b18665 - Update ManageWikiFormFactoryBuilder.php [18:50:04] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.92, 8.25, 5.93 [18:50:18] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.24, 7.02, 5.43 [18:53:44] [02ManageWiki] 07Universal-Omega opened pull request 03#308: Support filtering settings by extension - 13https://git.io/Jidjq [18:54:16] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.95, 23.79, 21.78 [18:55:26] [02miraheze/ManageWiki] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JiFvI [18:55:28] [02miraheze/ManageWiki] 07github-actions 039dddcd6 - CI: lint code to MediaWiki standards [18:55:29] [02ManageWiki] 07github-actions[bot] synchronize pull request 03#308: Support filtering settings by extension - 13https://git.io/Jidjq [18:57:06] miraheze/ManageWiki - Universal-Omega the build has errored. [18:58:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.41, 7.92, 6.90 [18:59:52] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JiFTz [18:59:54]