[00:02:34] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:03:00] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.80, 1.67, 1.53 [00:03:36] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.47, 3.14, 2.82 [00:04:59] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.54, 1.66, 1.54 [00:05:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.68, 7.06, 5.53 [00:05:59] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.64, 8.24, 6.06 [00:06:21] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.66, 7.34, 5.70 [00:06:47] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.73, 20.19, 17.29 [00:07:27] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.26, 6.52, 5.53 [00:07:59] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.85, 6.97, 5.86 [00:08:15] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.64, 6.35, 5.53 [00:08:47] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.91, 19.80, 17.48 [00:09:59] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.85, 6.25, 5.73 [00:11:25] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.38, 3.33, 3.03 [00:12:47] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.00, 22.72, 19.34 [00:13:58] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.12, 1.77, 1.63 [00:14:47] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.40, 24.53, 20.35 [00:15:57] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.03, 1.52, 1.56 [00:24:00] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.44, 2.10, 1.78 [00:24:52] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.55, 22.57, 21.97 [00:26:01] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.25, 1.74, 1.69 [00:29:59] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.18, 1.52, 1.61 [00:32:31] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:35:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.69, 3.32 [00:36:38] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.39, 7.71, 6.42 [00:36:40] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.43, 7.23, 6.07 [00:37:00] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.78, 18.98, 20.18 [00:38:33] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.71, 6.67, 6.19 [00:38:39] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.72, 6.52, 5.94 [00:41:12] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.10, 3.64, 3.39 [00:43:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.74, 3.46 [00:45:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.77, 3.31, 3.33 [00:46:34] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.45, 6.88, 6.35 [00:48:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.84, 8.64, 7.12 [00:49:54] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.07, 1.78, 1.57 [00:52:17] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.83, 7.13, 6.57 [00:52:25] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.40, 7.62, 6.41 [00:52:43] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.70, 7.57, 7.05 [00:53:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.70, 3.78, 3.52 [00:54:26] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.47, 6.91, 6.31 [00:54:33] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.74, 6.71, 5.97 [00:56:00] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.92, 1.58, 1.58 [00:56:07] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.32, 6.17, 6.35 [00:56:26] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.95, 6.73, 6.33 [00:56:32] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.08, 6.32, 5.93 [00:58:41] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.32, 6.20, 6.64 [01:03:04] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.55, 4.05, 3.70 [01:04:12] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 12.94, 8.53, 7.02 [01:04:15] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.86, 21.64, 19.84 [01:05:01] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.99, 3.93, 3.70 [01:06:06] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.56, 7.35, 6.78 [01:06:15] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.58, 19.14, 19.15 [01:08:01] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.31, 6.35, 6.48 [01:12:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.33, 3.01, 3.37 [01:18:42] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.10, 3.24 [01:20:42] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.41, 3.44, 3.34 [01:22:24] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.97, 6.74, 6.12 [01:22:38] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.95, 3.20, 3.27 [01:22:50] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.69, 6.58, 6.21 [01:23:05] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.96, 1.85, 1.55 [01:24:27] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.16, 6.14, 5.98 [01:24:46] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.60, 5.83, 5.99 [01:25:02] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.34, 2.00, 1.64 [01:27:01] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.66, 1.92, 1.65 [01:32:58] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.78, 1.36, 1.51 [01:34:59] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.23, 6.68, 5.84 [01:36:58] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.02, 6.64, 5.92 [01:37:03] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.96, 1.79, 1.66 [01:37:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.30, 3.78, 3.45 [01:38:58] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.00, 6.28, 5.86 [01:39:01] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.47, 1.63, 1.61 [01:39:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.81, 3.84, 3.52 [01:43:23] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.24, 3.14, 3.31 [01:44:59] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.78, 2.00, 1.79 [01:46:58] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.92, 1.62, 1.67 [01:50:18] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.55, 3.43 [01:54:13] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.48, 3.66, 3.52 [01:54:55] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.51, 1.77, 1.72 [01:56:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.42, 3.85, 3.59 [01:58:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.05, 3.57, 3.53 [02:00:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.30, 3.14, 3.37 [02:00:52] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.33, 1.55, 1.66 [02:10:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.61, 3.30, 3.27 [02:14:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.05, 3.26, 3.26 [02:22:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.58, 3.66, 3.43 [02:23:48] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.74, 3.28, 3.31 [02:24:52] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.00, 2.05, 1.72 [02:26:52] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.45, 1.73, 1.64 [02:32:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.15, 3.51, 3.42 [02:32:48] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.04, 1.60, 1.65 [02:40:04] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.13, 3.27, 3.35 [02:48:43] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.16, 1.76, 1.61 [02:50:41] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.08, 1.51, 1.54 [03:21:35] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.84, 1.58, 1.41 [03:25:32] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.85, 1.45, 1.42 [03:31:31] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.47, 1.79, 1.60 [03:33:32] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 6.38, 2.78, 1.94 [03:37:31] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.08, 1.92, 1.78 [03:43:29] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.46, 1.59, 1.68 [03:52:28] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.76, 1.78, 1.71 [03:54:28] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.46, 1.67, 1.68 [07:21:30] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.16, 3.40, 2.86 [07:23:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.67, 3.03, 2.79 [11:15:01] JohnLewis: hi. After doing quite a few reviews I realised I'm a bit confused as to how I would go about checking " Can wikis use the external service, even if the visitor wants to deny any cookies or other form of tracking? [11:15:02] Will wikis stay usable, even if the visitor blocks the external resource by using an ad blocker?" [11:15:24] so far It's mostly been my suppositions and I have reflected than in my comments by saying "unsure" or "possibly/likely" [11:15:52] how can that be determined if for example the service is only used to embed images? [11:16:22] If the service is only used to embed images, the answer is yes [11:17:14] It's to do with things like upstream providers insisting cookies are used, which can then be used for tracking. e.g. some providers don't have cookie opt outs, they simply say you can't use the service if you deny cookies [11:17:18] In what circumstance could a wiki become "unusable" if external resources are blocked? I guess I can't think of any [11:17:40] It's more request justification that [11:18:05] Ah, I see. Many of the privacy policies I've read to mention that [11:18:13] for example the one I'm currently reviewing says " You may also configure your web browser to disable cookies when visiting the website." [11:18:29] A wiki would determine how 'necessary' the service is, if the service then won't work without cookies, it would be deemed 'unusable' [11:18:34] so that sounds like a "Yes" answer for the first question [11:19:11] For example, if people block recaptcha.net urls, Miraheze becomes unusable [11:20:01] makes sense then, so the two questions are quite closely related [11:20:17] Yeah [11:20:55] JohnLewis: so to make sure I understand correctly: the first one asks whether the service will work if cookies are denied and the second asks whether the wiki would work if a user chooses to block the external resource [11:22:25] Yeah [11:24:35] JohnLewis: so for example let's take fontawesome.com which is used with an extension to insert FontAwesome icons. If the external resource is blocked the wiki will still be 'usable' albeit with some potential issues since the icons would presumably not display, so the answer would be yes then [11:25:14] Yes [11:25:31] ok, all clear now. Thanks! [11:28:29] triggers are great too :) [11:37:12] JohnLewis: are there still some CSP review tasks to be opened as for example I don't see libera.chat? [11:38:05] No, I might have missed that one then when mass opening [11:40:23] ah, I'll create one then [11:50:51] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/JE9d4 [11:50:52] [02miraheze/puppet] 07Reception123 031ceeaa1 - remove sorcery.net from CSP per T7890 [11:50:54] [02puppet] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/vbiAS [11:50:55] [02puppet] 07Reception123 opened pull request 03#1914: remove sorcery.net from CSP per T7890 - 13https://git.io/JE9dR [11:51:17] [02puppet] 07Reception123 edited pull request 03#1914: remove sorcery.net from CSP per T7890 - 13https://git.io/JE9dR [11:52:05] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/JE9d5 [11:52:06] [02miraheze/puppet] 07Reception123 03739fb85 - fix mistake when inserting numbers [11:52:08] [02puppet] 07Reception123 synchronize pull request 03#1914: remove sorcery.net from CSP per T7890 - 13https://git.io/JE9dR [12:18:04] [02puppet] 07Reception123 closed pull request 03#1914: remove sorcery.net from CSP per T7890 - 13https://git.io/JE9dR [12:18:05] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JE9xF [12:18:07] [02miraheze/puppet] 07Reception123 03cf7dc38 - remove sorcery.net from CSP per T7890 (#1914) [12:39:52] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_36 [+1/-0/±0] 13https://git.io/JEHvl [12:39:54] [02miraheze/mediawiki] 07RhinosF1 03ebedfad - Create non-rel-extensions [12:40:17] Reception123: ^ [12:40:25] I'll use that when branching [12:40:47] But REL1_36 will switch to REL1_37 [12:40:51] Rest remain the same [12:40:59] Hopefully won't be too many odd ones this year [12:45:43] ok [13:11:46] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [13:11:48] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [13:12:11] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [13:12:18] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [13:31:06] Mass opening phab task for the CSP was surprising [13:35:42] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:35:56] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:36:16] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:36:29] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:44:38] It wasn't surprising for any of SRE or T&S [13:45:16] Ok. The surprise was mine [13:45:20] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JEHG0 [13:45:22] [02miraheze/ssl] 07Reception123 03b52643d - add staff.polandballwiki.com cert [13:46:15] @Avengium [tuscriaturas] It's part of a new CSP policy, so we need to review everything [13:49:20] Is fine. Good practice 👍 [14:19:21] @Owen: do we want to move the CSP to media-src then and have a seperate for script-src [14:20:19] My advice would be at the very least to restrict loading on websites where any user can upload any content. Fine grain control for all sites would be good and ideal, but fine grained control on those sites would be the bare minimum to me. [14:22:15] @Owen: so would media-src be acceptable so only media can be loaded? [14:22:55] I do not know the ins and outs of CSP directives, what I have written is what I expect, how it is implemented is up to SRE entirely as long as it matches the requirements set. [14:23:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.46, 3.14, 2.90 [14:25:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.07, 3.10, 2.92 [14:29:10] @Owen: as they only wanted images, that would limit it to media and block JS [14:29:34] In that case though we would need multiple CSPs [14:32:45] It would still be one HTTP header, just rather than allowing everything, it would be restrictive on what is allowed [14:39:28] yes one header [14:39:33] But multiple files for us [14:42:56] Then multiple lists will need to be built and maintained [14:50:06] Reception123: do we want tasks for that? [14:50:26] My only concern is max header length [14:51:54] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.82, 1.73, 1.29 [14:53:52] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.16, 1.52, 1.26 [14:59:00] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.69, 3.55, 3.25 [15:04:52] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.06, 3.28, 3.23 [15:06:29] PROBLEM - test3 APT on test3 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:13:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03revert-1716-paladox-patch-4 [+0/-0/±3] 13https://git.io/JEHoP [15:13:09] [02miraheze/puppet] 07paladox 038779381 - Revert "Remove db12 from bacula (#1716)" [15:13:10] [02puppet] 07paladox created branch 03revert-1716-paladox-patch-4 - 13https://git.io/vbiAS [15:13:12] [02puppet] 07paladox opened pull request 03#1915: Revert "Remove db12 from bacula" - 13https://git.io/JEHo1 [15:22:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.59, 3.51, 3.25 [15:26:20] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.14, 3.40, 3.27 [15:51:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JEHSu [15:51:37] [02miraheze/puppet] 07paladox 036420d8d - mediawiki: Add custom 404 to nginx [15:51:39] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://git.io/vbiAS [15:51:40] [02puppet] 07paladox opened pull request 03#1916: mediawiki: Add custom 404 to nginx - 13https://git.io/JEHSz [15:51:55] [02puppet] 07paladox edited pull request 03#1916: mediawiki: Add custom 404 to nginx - 13https://git.io/JEHSz [15:52:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JEHSQ [15:52:50] [02miraheze/puppet] 07paladox 03caebc41 - Update init.pp [15:52:52] [02puppet] 07paladox synchronize pull request 03#1916: mediawiki: Add custom 404 to nginx - 13https://git.io/JEHSz [16:08:17] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/JEHQp [16:08:18] [02miraheze/mw-config] 07paladox 031589edc - Add custom 404.php [16:08:20] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [16:08:21] [02mw-config] 07paladox opened pull request 03#4064: Add custom 404.php - 13https://git.io/JEHQh [16:09:25] miraheze/mw-config - paladox the build passed. [16:09:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.26, 3.43, 3.29 [16:11:23] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.49, 3.22, 3.23 [16:17:49] [02CreateWiki] 07Pikse commented on commit 032f4df24f5b17216c748ca64af72f4418ceb00a5c - 13https://git.io/JEH5d [16:21:15] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JEHdE [16:21:17] [02miraheze/mediawiki] 07RhinosF1 03c169748 - Dependabot: add support for REL1_37 [16:21:18] [02mediawiki] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/vbL5b [16:21:19] [02mediawiki] 07RhinosF1 opened pull request 03#2711: Dependabot: add support for REL1_37 - 13https://git.io/JEHdu [16:21:55] Universal_Omega: if you want an easy review [16:35:53] RhinosF1: the dependabot one? If that's what you mean, looks good, but do we want to do that yet? [16:48:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.69, 3.07, 2.93 [16:50:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.06, 3.42, 3.07 [16:52:08] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.14, 2.95, 2.95 [16:52:20] Universal_Omega: I'll probably merge it when I actually do the cut [16:52:36] This week is the penultimate train [16:53:04] We need to watch blockers on wmf.1 closely this year [16:53:06] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.71, 1.49, 1.15 [16:54:48] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JEHph [16:54:49] [02miraheze/CreateWiki] 07Universal-Omega 034b12c16 - Fix [16:55:04] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.84, 1.22, 1.09 [16:55:38] Sounds good RhinosF1 [16:55:46] miraheze/CreateWiki - Universal-Omega the build passed. [16:56:48] Universal_Omega: last time we had a few issues with wmf.1 blockers that just missed the last train but got cut into .37 so I want to avoid a repeat and monitor that closely [16:57:04] We should specifically check commits, causes and test for them [17:35:35] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.22, 7.02, 5.35 [17:37:31] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.64, 6.25, 5.27 [17:37:54] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.22, 1.72, 1.30 [17:39:55] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.05, 1.38, 1.22 [17:59:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.08, 3.48, 3.14 [18:01:02] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.06, 3.15, 3.05 [18:05:18] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.83, 6.61, 5.48 [18:07:14] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.52, 5.69, 5.28 [18:16:19] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.06, 19.99, 16.83 [18:18:20] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.21, 18.61, 16.72 [18:20:23] !log uploading images to drawnfeetwiki via a shell script to preserve author details [18:20:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:52:14] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.12, 1.48, 1.17 [18:54:13] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.03, 1.31, 1.15 [18:59:31] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.71, 7.19, 5.16 [19:00:32] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.10, 7.04, 5.18 [19:00:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.52, 6.95, 4.98 [19:01:31] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.36, 7.48, 5.55 [19:02:32] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.43, 6.31, 5.13 [19:02:42] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.32, 6.76, 5.14 [19:04:41] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.43, 6.07, 5.09 [19:05:27] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.77, 5.93, 5.37 [19:22:06] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.62, 1.97, 1.47 [19:26:03] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.13, 1.69, 1.51 [20:05:11] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.63, 5.81, 4.40 [20:07:09] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.30, 5.71, 4.54 [20:08:58] disk space on mwtask1 will go critical, it is me and it is intentional [20:11:58] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.53, 3.52, 2.87 [20:13:48] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 27.23, 27.90, 20.71 [20:16:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.12, 3.73, 3.14 [20:17:54] PROBLEM - mwtask1 Disk Space on mwtask1 is WARNING: DISK WARNING - free space: / 3895 MB (8% inode=84%); [20:18:15] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.44, 3.33, 3.06 [20:21:58] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 16.03, 23.03, 22.64 [20:31:39] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JEQzT [20:31:40] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 13.56, 17.17, 20.09 [20:31:41] [02miraheze/mediawiki] 07paladox 03d6797bf - Add 404.php to .gitignore [20:31:42] [02mediawiki] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbL5b [20:31:51] [02mediawiki] 07paladox opened pull request 03#2712: Add 404.php to .gitignore - 13https://git.io/JEQzL [20:35:07] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.62, 6.65, 5.22 [20:36:02] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.69, 7.14, 5.59 [20:36:22] PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 328.67 ms [20:37:05] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.83, 6.60, 5.37 [20:37:32] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.70, 19.83, 20.43 [20:38:03] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.83, 7.04, 5.74 [20:39:02] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.32, 6.11, 5.34 [20:40:30] RECOVERY - ping6 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 271.38 ms [20:42:02] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.13, 7.00, 5.98 [20:44:03] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.98, 6.79, 6.01 [20:46:01] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.66, 6.49, 6.00 [20:47:16] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 15.72, 18.51, 19.85 [20:54:41] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.42, 6.79, 6.15 [20:55:18] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.33, 21.04, 20.28 [20:56:25] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.24, 7.40, 6.18 [20:56:58] PROBLEM - gluster3 Disk Space on gluster3 is WARNING: DISK WARNING - free space: / 104871 MB (10% inode=81%); [20:57:03] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.59, 8.52, 6.77 [20:57:17] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.77, 20.22, 20.06 [20:57:24] JohnLewis: I saw task but gluster too ^ [20:57:32] You said task was expected [20:58:20] paladox: ^^ for gluster, 100G left isn't a problem but should be addressed in time [20:58:22] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.68, 7.94, 6.55 [20:58:37] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.43, 7.60, 6.63 [20:58:57] ok [21:00:07] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.78, 8.88, 7.06 [21:00:15] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.46, 9.06, 7.25 [21:00:15] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 7.92, 8.24, 7.09 [21:00:29] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.15, 8.21, 6.84 [21:00:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 11.30, 8.97, 7.25 [21:01:08] [02mw-config] 07paladox synchronize pull request 03#4064: Add custom 404.php - 13https://git.io/JEHQh [21:01:09] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JEQa9 [21:01:11] [02miraheze/mw-config] 07paladox 033109459 - Update 404.php [21:01:33] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 24.98, 22.80, 21.14 [21:02:06] miraheze/mw-config - paladox the build passed. [21:02:30] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.28, 7.96, 6.92 [21:03:01] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.29, 2.97 [21:03:34] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.55, 22.95, 21.39 [21:03:50] [02mw-config] 07paladox closed pull request 03#4064: Add custom 404.php - 13https://git.io/JEHQh [21:03:51] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JEQap [21:03:53] [02miraheze/mw-config] 07paladox 03cbe7257 - Add custom 404.php (#4064) [21:03:55] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [21:03:56] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [21:04:00] [02puppet] 07paladox closed pull request 03#1916: mediawiki: Add custom 404 to nginx - 13https://git.io/JEHSz [21:04:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JEQah [21:04:03] [02miraheze/puppet] 07paladox 034fc7fab - mediawiki: Add custom 404 to nginx (#1916) [21:04:04] [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://git.io/vbiAS [21:04:06] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6 [21:04:06] !log [@test3] starting deploy of {'config': True} to skip [21:04:07] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [21:04:07] [02mediawiki] 07paladox closed pull request 03#2712: Add 404.php to .gitignore - 13https://git.io/JEQzL [21:04:09] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_36 [+0/-0/±2] 13https://git.io/JEQaj [21:04:10] [02miraheze/mediawiki] 07paladox 03a4b5920 - Merge pull request #2712 from miraheze/paladox-patch-1 [21:04:12] [02mediawiki] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbL5b [21:04:13] [02miraheze/mediawiki] 07paladox deleted branch 03paladox-patch-1 [21:04:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:04:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:04:31] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 6.56, 8.01, 7.10 [21:04:47] miraheze/mw-config - paladox the build passed. [21:04:58] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.33, 3.48, 3.07 [21:05:42] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:06:15] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.63, 23.16, 19.69 [21:07:03] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.25, 3.31, 3.06 [21:08:02] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.75, 4.91, 4.06 [21:12:18] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.23, 5.47, 4.50 [21:12:41] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:13:09] PROBLEM - mw13 Puppet on mw13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:13:40] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:14:05] PROBLEM - mw12 Puppet on mw12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:14:24] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 3.83, 4.83, 4.39 [21:14:27] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:14:34] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:14:58] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [21:15:24] RECOVERY - mw13 Puppet on mw13 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [21:15:50] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:16:13] RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [21:16:31] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:16:42] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:17:07] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.37, 23.61, 22.41 [21:19:36] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.46, 7.64, 7.84 [21:21:03] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.46, 6.78, 7.75 [21:22:07] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 3.43, 5.93, 7.55 [21:22:16] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 15.35, 19.51, 22.94 [21:22:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.51, 6.70, 7.81 [21:26:19] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 29.68, 23.01, 23.38 [21:27:01] paladox: test3 doesn't work but meta looks cool [21:27:15] oh, wait let me fix test3 [21:27:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.45, 5.86, 7.60 [21:27:20] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.44, 22.33, 21.70 [21:27:24] i had a custom change so i guess that's why [21:27:36] paladox: also we just ran out of php workers again [21:28:15] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.87, 23.28, 23.45 [21:28:52] RhinosF1: that email was me testing changes to Grafana alert (obviously it didn't work). Unless there was an overlap where they actually ran out when I was testing. [21:28:56] !log [paladox@test3] starting deploy of {'world': True, 'gitinfo': True} to skip [21:28:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:29:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.30, 20.75, 21.18 [21:29:24] RhinosF1: i think high load is correlating to the php workers running out. [21:29:25] Universal_Omega: well I saw a bunch of depools when making sure puppet had ran [21:29:41] the lower the load is the quicker the childs can popup. [21:29:42] paladox: yes that's the most obvious part [21:30:07] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.06, 6.30, 7.03 [21:30:15] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 32.57, 25.18, 24.02 [21:30:43] RhinosF1: Yeah it was an overlap. But yeah some of the emails (like yesterday) where me testing unsuccessfully to get it to properly show the instance. [21:30:44] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.49, 6.39, 7.75 [21:31:01] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.16, 5.93, 6.80 [21:31:02] Universal_Omega: okay [21:32:08] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.64, 6.02, 6.84 [21:32:20] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 7.75, 6.68, 5.25 [21:32:41] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.91, 5.29, 6.53 [21:32:49] !log [paladox@test3] finished deploy of {'world': True, 'gitinfo': True} to skip - SUCCESS in 232s [21:32:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:33:25] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.03, 18.82, 20.32 [21:34:06] God that's slow [21:34:09] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.13, 5.91, 6.68 [21:34:18] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 3.41, 5.37, 4.94 [21:34:46] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.24, 7.47, 7.81 [21:36:04] hmm https://meta.miraheze.org/wiki/Miraheze?forceprofile=1 throws 500 on test3 [21:36:05] [url] Meta | meta.miraheze.org [21:36:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.32, 3.47, 3.23 [21:36:15] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 3.81, 4.95, 4.84 [21:36:37] paladox: anything in logs [21:37:41] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.11, 6.81, 7.52 [21:37:42] mediawiki_host:test3.miraheze.org brings up nothing [21:38:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.28, 3.39, 3.23 [21:38:15] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 18.99, 22.38, 23.49 [21:39:22] paladox: don't say graylog is broke on test3 again [21:39:31] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.14, 7.28, 7.08 [21:40:45] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.15, 7.66, 7.70 [21:41:07] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.03, 1.71, 1.42 [21:41:20] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.92, 8.35, 7.54 [21:41:30] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.95, 20.49, 20.39 [21:41:37] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.96, 8.57, 7.32 [21:43:10] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.85, 1.38, 1.34 [21:43:57] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.79, 8.51, 7.40 [21:44:20]