[00:05:30] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.50, 6.81, 5.17 [00:05:46] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 11.14, 7.03, 5.18 [00:06:00] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.98, 6.67, 5.02 [00:06:03] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 32.77, 22.06, 16.92 [00:07:30] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.62, 6.44, 5.24 [00:07:58] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.61, 6.18, 5.03 [00:08:05] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.81, 19.69, 16.69 [00:09:41] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.80, 6.65, 5.49 [00:21:57] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.71, 20.94, 19.28 [00:23:57] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.80, 19.64, 19.04 [00:28:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.63, 3.40, 3.07 [00:30:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.87, 3.34, 3.10 [00:47:34] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.60, 3.24, 3.15 [00:49:34] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.70, 2.96, 3.05 [01:35:53] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.59, 3.52, 3.26 [01:37:53] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.86, 2.92, 3.07 [01:41:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.18, 3.78, 3.40 [01:45:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.27, 3.75, 3.50 [01:47:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.75, 3.40, 3.40 [01:51:59] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.25, 3.72, 3.52 [01:53:58] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.57, 3.33, 3.40 [01:57:58] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.51, 3.66, 3.49 [01:59:58] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.03, 3.53, 3.47 [02:18:08] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.15, 3.25, 3.39 [02:24:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.67, 3.72, 3.49 [02:26:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.64, 3.32, 3.38 [02:31:41] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.76, 1.66, 1.35 [02:33:39] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.30, 1.48, 1.31 [02:49:15] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.09, 3.55, 3.32 [02:51:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.77, 3.41, 3.28 [02:53:20] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.94, 1.53, 1.27 [02:53:21] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.31, 3.28, 3.25 [02:55:20] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.61, 1.50, 1.29 [03:41:31] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.71, 6.13, 4.56 [03:43:31] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.43, 5.06, 4.36 [03:52:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.27, 3.34, 3.05 [03:54:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.37, 2.90, 2.92 [05:05:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.55, 3.21, 3.06 [05:07:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.82, 2.73, 2.90 [05:51:14] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.38, 3.56, 3.14 [05:53:14] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.87, 3.32, 3.10 [06:33:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.67, 3.24, 2.93 [06:35:21] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.55, 2.98, 2.87 [09:45:31] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.37, 5.83, 4.76 [09:47:31] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.45, 5.15, 4.63 [09:54:23] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.94, 6.34, 5.54 [09:56:19] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.91, 5.99, 5.52 [10:21:37] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.57, 6.26, 5.69 [10:23:39] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.56, 6.30, 5.78 [10:35:59] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.00, 7.04, 5.98 [10:36:28] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.87, 7.32, 6.67 [10:37:57] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.70, 6.64, 5.97 [10:38:26] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.00, 6.20, 6.34 [10:41:40] HolidayRhino: I’m putting quite a bit of time today aside for Miraheze. If anything in the MW-SRE queue needs looking at/unsticking, feel free to shoot it my way and I’ll give them a look over today [10:42:47] JohnLewis: the ssh key from a mw server to the rest (not test3, there's a remote_sync heira variable) would be cool [10:43:09] Task says mw11 but any with enough disk for staging would be fine [10:43:24] I'm currently reading my many emails [10:44:16] You’re better off specifying the ssh key on each command if that’s possible, ssh-agent with a key armed was a pain which is why we moved to puppetserver [10:45:51] Are you using rsync for it? [10:46:46] JohnLewis: yeah, do we need password on the key though? [10:47:26] -e “ssh -i $ssh-key-file” [10:47:47] Added to the rsync command handles ssh auth [10:48:40] And then deployer puts password in each time or? [10:48:49] That's gonna be a pain if they got to [10:49:10] If you use a passphrase, yes [10:50:17] Is it acceptable to not? [10:50:24] You can get away without using one, because if they have the private key, it’s already game over even with a passphrase [10:50:32] Okay [10:50:39] As the key shouldn’t leave the cluster at all [10:50:44] That task can be declined then and just specify the key [10:51:09] It makes it less complicated as well [11:02:23] Much [11:05:37] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.41, 6.31, 5.41 [11:07:38] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.02, 5.32, 5.16 [11:51:58] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.95, 6.75, 5.10 [11:53:58] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.95, 5.57, 4.86 [12:19:11] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.59, 6.28, 4.66 [12:21:09] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.31, 5.56, 4.60 [12:35:32] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.19, 6.30, 4.83 [12:37:31] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.34, 5.42, 4.68 [12:45:00] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.59, 6.45, 4.97 [12:47:00] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.81, 6.84, 5.30 [12:47:17] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.73, 6.96, 5.18 [12:47:30] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.12, 7.10, 5.38 [12:49:01] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.08, 6.29, 5.29 [12:49:13] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.35, 6.34, 5.18 [12:49:31] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.63, 7.38, 5.74 [12:51:30] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.79, 6.46, 5.60 [12:56:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.20, 6.08, 4.79 [12:57:30] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.57, 7.02, 6.13 [12:58:42] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.73, 5.56, 4.77 [12:59:30] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.33, 6.20, 5.92 [13:04:29] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.62, 8.41, 6.13 [13:06:27] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.28, 7.53, 6.09 [13:08:23] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.00, 6.56, 5.90 [13:30:03] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.12, 2.83 [13:32:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.85, 3.13, 2.88 [13:34:38] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.09, 6.36, 5.32 [13:36:34] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.54, 5.80, 5.25 [13:53:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.44, 3.39, 3.18 [13:55:15] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.20, 3.35, 3.19 [13:59:52] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.24, 6.25, 5.29 [14:01:49] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.06, 5.69, 5.20 [14:05:30] JohnLewis: how do we plan to start with bullseye? [14:06:42] No plans currently [14:10:32] JohnLewis: could we make plans [14:11:08] They can be, but aren’t a priority currently I’d imagine [14:14:18] We have a year of oldstable [14:33:20] Can anyone remeber what https://phabricator.wikimedia.org/T264505 was about [14:33:21] [url] ⚓ T264505 MediaWiki:Sidebar and MediaWiki:Sitenotice only having effect intermittently | phabricator.wikimedia.org [14:54:47] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J0lsy [14:54:48] [02miraheze/mw-config] 07JohnFLewis 03f0e6d5d - add rebuildTitleKeys to titlekey install [14:55:47] miraheze/mw-config - JohnFLewis the build passed. [15:25:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.52, 3.32, 3.12 [15:27:29] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.11, 3.20, 3.10 [15:35:40] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.17, 6.07, 4.92 [15:37:38] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.16, 5.83, 4.96 [15:43:30] [02puppet] 07RhinosF1 opened pull request 03#1884: mwdeploy: add ssh keys - 13https://git.io/J08VJ [15:43:55] JohnLewis: ^ [15:44:32] Also https://github.com/miraheze/puppet/pull/1880 [15:44:34] [url] deploy-mediawiki: Fix folders argument by Universal-Omega · Pull Request #1880 · miraheze/puppet · GitHub | github.com [15:45:18] RhinosF1: tabbing and syntax is off ({ where it should be :) [15:45:37] [02puppet] 07JohnFLewis closed pull request 03#1880: deploy-mediawiki: Fix folders argument - 13https://git.io/JRH4b [15:45:39] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J08oG [15:45:40] [02miraheze/puppet] 07Universal-Omega 035be09bd - deploy-mediawiki: Fix folders argument (#1880) [15:50:56] JohnLewis: was mobile [15:51:19] Will sort I can get laptop to charge tonight if not I'll order a cable [15:52:40] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.10, 3.50, 3.20 [15:54:39] no worrie [15:56:14] I think UO was still seeing composer issues too [15:56:19] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.06, 6.93, 5.68 [15:56:19] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.50, 7.10, 5.85 [15:56:31] But I'm not sure if that's just awkward process and I need to automate that [15:56:41] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.27, 3.10, 3.12 [15:58:16] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.76, 6.29, 5.59 [15:59:59] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-3 [+0/-0/±1] 13https://git.io/J08x9 [16:00:00] [02miraheze/mw-config] 07RhinosF1 03901fc3f - Sitenotice: Per Infra [16:00:02] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-3 - 13https://git.io/vbvb3 [16:00:13] [02mw-config] 07RhinosF1 assigned pull request 03#4052: Sitenotice: Per Infra - 13https://git.io/J08pG [16:00:15] [02mw-config] 07RhinosF1 opened pull request 03#4052: Sitenotice: Per Infra - 13https://git.io/J08pG [16:00:16] [02mw-config] 07RhinosF1 assigned pull request 03#4052: Sitenotice: Per Infra - 13https://git.io/J08pG [16:00:17] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.33, 6.38, 5.85 [16:01:08] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-4 [+0/-0/±1] 13https://git.io/J08jF [16:01:09] [02miraheze/mw-config] 07RhinosF1 0356a841a - Database: All wikis RO for DB restart [16:01:11] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-4 - 13https://git.io/vbvb3 [16:01:11] miraheze/mw-config - RhinosF1 the build passed. [16:01:16] [02mw-config] 07RhinosF1 opened pull request 03#4053: Database: All wikis RO for DB restart - 13https://git.io/J04eR [16:01:18] [02mw-config] 07RhinosF1 assigned pull request 03#4053: Database: All wikis RO for DB restart - 13https://git.io/J04eR [16:01:19] [02mw-config] 07RhinosF1 assigned pull request 03#4053: Database: All wikis RO for DB restart - 13https://git.io/J04eR [16:02:13] miraheze/mw-config - RhinosF1 the build passed. [16:07:04] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J04ny [16:07:06] [02miraheze/mw-config] 07paladox 03bd5f26b - Add database maintenance site notice [16:07:08] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [16:07:09] [02mw-config] 07paladox opened pull request 03#4054: Add database maintenance site notice - 13https://git.io/J04np [16:07:27] RhinosF1: ^ [16:08:10] miraheze/mw-config - paladox the build passed. [16:08:11] [02mw-config] 07RhinosF1 commented on pull request 03#4054: Add database maintenance site notice - 13https://git.io/J04WH [16:08:17] [02mw-config] 07RhinosF1 closed pull request 03#4054: Add database maintenance site notice - 13https://git.io/J04np [16:08:18] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J04l3 [16:08:20] [02miraheze/mw-config] 07paladox 0374664b4 - Add database maintenance site notice (#4054) [16:08:21] [02miraheze/mw-config] 07RhinosF1 deleted branch 03paladox-patch-1 [16:08:23] [02mw-config] 07RhinosF1 deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [16:08:31] [02mw-config] 07RhinosF1 closed pull request 03#4052: Sitenotice: Per Infra - 13https://git.io/J08pG [16:08:34] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-3 - 13https://git.io/vbvb3 [16:08:35] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-3 [16:09:14] miraheze/mw-config - RhinosF1 the build passed. [16:18:05] RhinosF1: https://phabricator.miraheze.org/T7753 has this been reviewed as no update in over a week? [16:18:06] [url] ⚓ T7753 Global Abuse Filter #18 has been emergency throttled | phabricator.miraheze.org [16:18:25] JohnLewis: last time I checked it was still high [16:19:28] But from a technical stand point, what else needs to be done on the task? It seems like everything is done [16:19:48] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.96, 3.89, 3.41 [16:20:41] JohnLewis: revert the change when it's back lower [16:20:48] It's not really acceptable that high [16:21:22] Okay, as far as I am concerned, it's been sufficient time for the community to resolve the matter - if this isn't an acceptable value, it needs to be lowered and the task closed. [16:21:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.76, 3.42 [16:23:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.10, 3.88, 3.50 [16:25:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.13, 3.91, 3.58 [16:26:33] https://phabricator.miraheze.org/T7753#157863 [16:26:34] [url] ⚓ T7753 Global Abuse Filter #18 has been emergency throttled | phabricator.miraheze.org [16:29:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.67, 3.21, 3.37 [16:33:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.40, 3.44, 3.45 [16:35:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.80, 3.19, 3.36 [16:37:32] JohnLewis: https://phabricator.miraheze.org/T7753#157865 [16:37:33] [url] ⚓ T7753 Global Abuse Filter #18 has been emergency throttled | phabricator.miraheze.org [16:49:30] [02mw-config] 07AgentIsai opened pull request 03#4055: Fixing typos in Sitenotice - 13https://git.io/J0B6n [16:49:56] [02mw-config] 07RhinosF1 closed pull request 03#4055: Fixing typos in Sitenotice - 13https://git.io/J0B6n [16:49:57] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J0Biv [16:49:59] [02miraheze/mw-config] 07AgentIsai 038d5c992 - Fixing typos in Sitenotice (#4055) [16:50:13] Ty @Agent [16:50:24] np [16:50:44] miraheze/mw-config - RhinosF1 the build passed. [16:50:45] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.43, 1.88, 1.31 [16:50:51] miraheze/mw-config - RhinosF1 the build passed. [16:51:30] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.56, 6.44, 5.87 [16:52:07] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.70, 3.30, 3.17 [16:52:28] JohnLewis: what's the best way to do a list of all mediawiki servers? [16:52:42] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.65, 1.79, 1.34 [16:52:42] Just manually have it in the Python script or can we query something [16:53:28] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.53, 5.88, 5.75 [16:54:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.35, 3.35, 3.20 [16:54:41] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.70, 1.39, 1.25 [17:02:03] RhinosF1: manually unless you can access/use facter but I'm unsure if that is possible [17:02:11] Ok [17:05:15] !log running two imports on mwtask1 [17:05:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:08:26] JohnLewis: thanks for helping out with MW tasks! [17:11:11] I’m just wanting to get the queue down and I know most are away/busy/focused elsewhere currently so no worries [17:15:35] [02puppet] 07RhinosF1 opened pull request 03#1885: mwdeploy: add rsync stuff - 13https://git.io/J0Rrq [17:16:25] not complete but ^ is the start of the work [17:16:38] In current state that won't work because I've seen the ' issue [17:17:04] JohnLewis: what do you suggest for the canary check? Just X-Debug a main page? [17:42:19] PROBLEM - cp13 Current Load on cp13 is WARNING: WARNING - load average: 1.55, 1.73, 0.89 [17:44:16] RECOVERY - cp13 Current Load on cp13 is OK: OK - load average: 0.78, 1.35, 0.85 [17:45:29] Hi Voidwalker [17:46:59] Yeah, that’ll probably be the best check for it [17:52:31] Cool JohnLewis [17:52:56] Do we check each deployed too or just the 1st? [17:54:11] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.83, 1.47, 1.21 [17:56:10] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.79, 1.23, 1.15 [18:03:28] I need to add -delete to rsync too [18:23:04] hi [18:23:24] Voidwalker: sounds like family emergency so hold if you need anything [18:50:19] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.34, 7.42, 5.75 [18:51:30] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.49, 6.49, 5.19 [18:52:16] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.92, 6.99, 5.82 [18:52:48] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.96, 1.81, 1.42 [18:53:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.47, 7.24, 5.85 [18:53:32] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.89, 5.51, 4.99 [18:54:12] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.40, 6.02, 5.60 [18:54:45] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.83, 1.47, 1.35 [18:54:58] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.49, 6.10, 5.59 [19:01:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.81, 3.43, 3.18 [19:03:50] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.40, 3.14, 3.11 [19:04:51] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.05, 6.30, 5.75 [19:06:47] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.29, 5.59, 5.56 [19:46:16] !log importing two more wiki imports [19:46:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:03:21] JohnLewis: feel free to use mwscript [20:12:13] PROBLEM - wiki.rpointer.ml - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.rpointer.ml could not be found [20:37:13] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.10, 3.85, 3.16 [20:39:13] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.59, 3.73, 3.21 [20:41:13] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.75, 3.27 [20:43:12] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.85, 3.40, 3.20 [20:58:20] [02puppet] 07RhinosF1 synchronize pull request 03#1885: mwdeploy: add rsync stuff - 13https://git.io/J0Rrq [20:58:36] [02puppet] 07RhinosF1 synchronize pull request 03#1844: mediawiki: [DNM] Only clone mediawiki to staging servers - 13https://git.io/JB556 [21:02:48] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.48, 6.94, 5.49 [21:03:37] I want to move the main copy jobs to rsync too [21:03:46] Because it's better than cp [21:04:44] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.95, 6.71, 5.59 [21:04:59] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.72, 7.17, 5.74 [21:05:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.52, 3.37, 3.16 [21:06:56] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.36, 6.06, 5.50 [21:09:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.28, 3.08, 3.11 [21:13:22] JohnLewis: where has hit data gone [21:13:47] It used to say up until days ago of the last x actions, y were matched [21:14:15] I see that on all filters, where are you not seeing it? [21:17:04] JohnLewis: on any [21:17:25] I just see 129 hits on https://meta.miraheze.org/wiki/Special:AbuseFilter/3 [21:17:27] [url] Editing abuse filter - Miraheze Meta | meta.miraheze.org [21:17:30] I tried multiple [21:17:37] From both staff & personal account [21:18:03] heh, I can't see it now [21:18:10] I saw it earlier on Filter 18 [21:18:29] I've seen it before [21:18:35] Not sure where it's gone [21:18:49] But it makes it impossible to tell which filters are hitting too much [21:20:43] https://meta.miraheze.org/wiki/Stewards%27_noticeboard#Abuse_Filter_Throttle [21:20:45] [url] Stewards' noticeboard - Miraheze Meta | meta.miraheze.org [21:32:49] Reception|away: advice for future, run imports with --no-updates. They'll be faster and you'll not have a lot of jobqueue events - meaning no redis connection errors. I've done about 6/7 imports today and not had one redis issue [21:55:25] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.94, 7.06, 6.03 [21:57:28] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.53, 6.75, 6.04 [22:08:46] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.59, 3.23, 2.91 [22:10:51] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.96, 3.32, 3.00 [22:15:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.24, 6.59, 6.07 [22:17:32] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.91, 6.25, 6.00 [23:30:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.46, 2.96 [23:32:12] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.81, 2.94, 2.84