[00:04:40] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.76, 6.21, 4.70 [00:06:38] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.89, 20.07, 15.84 [00:07:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.11, 4.08, 3.45 [00:08:37] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.06, 6.56, 5.27 [00:09:27] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.96, 4.37, 2.89 [00:10:34] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.61, 19.59, 16.73 [00:11:25] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.13, 4.08, 2.95 [00:11:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.30, 3.52, 3.40 [00:15:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.79, 3.52 [00:17:57] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.77, 3.32, 3.38 [00:21:34] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.72, 22.72, 19.59 [00:23:33] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.80, 20.27, 19.07 [00:55:23] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.41, 3.55, 3.23 [00:59:24] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.78, 3.42, 3.28 [01:03:24] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.36, 3.01, 3.14 [01:16:42] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.67, 3.32 [01:18:25] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.69, 6.69, 5.28 [01:19:28] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.66, 6.77, 5.31 [01:19:33] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.04, 7.77, 6.05 [01:20:08] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.77, 7.29, 5.45 [01:21:35] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.41, 6.28, 5.32 [01:22:04] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.58, 7.12, 5.59 [01:22:18] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.21, 7.15, 5.64 [01:24:47] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.74, 7.58, 6.25 [01:25:19] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.56, 4.11, 3.64 [01:25:57] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.08, 7.78, 6.80 [01:26:19] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.35, 6.39, 5.73 [01:26:29] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.28, 6.75, 5.89 [01:26:51] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.59, 6.55, 6.04 [01:29:24] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.21, 3.61, 3.55 [01:31:57] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.54, 5.84, 6.22 [01:33:28] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.58, 3.15, 3.38 [01:37:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.97, 3.57, 3.49 [01:47:39] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.26, 3.20, 3.39 [01:53:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.16, 3.41, 3.42 [01:55:51] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.01, 3.05, 3.30 [02:05:57] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.35, 3.31 [02:07:58] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.99, 3.20, 3.26 [02:17:18] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://git.io/JEh51 [02:17:20] [02miraheze/RottenLinks] 07Universal-Omega 03eb504c1 - Create .gitignore [02:17:22] [02RottenLinks] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/fxBrr [02:17:35] [02RottenLinks] 07Universal-Omega opened pull request 03#41: CI: Add PHPCS and Phan - 13https://git.io/JEh5S [02:18:21] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://git.io/JEh5N [02:18:22] [02miraheze/RottenLinks] 07Universal-Omega 03d2bbb99 - Create .phpcs.xml [02:18:24] [02RottenLinks] 07Universal-Omega synchronize pull request 03#41: CI: Add PHPCS and Phan - 13https://git.io/JEh5S [02:18:34] miraheze/RottenLinks - Universal-Omega the build passed. [02:19:18] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JEhdY [02:19:20] [02miraheze/RottenLinks] 07Universal-Omega 038f5da5c - Update continuousIntegration.yml [02:19:21] [02RottenLinks] 07Universal-Omega synchronize pull request 03#41: CI: Add PHPCS and Phan - 13https://git.io/JEh5S [02:19:22] miraheze/RottenLinks - Universal-Omega the build passed. [02:19:29] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://git.io/JEhd3 [02:19:30] [02miraheze/RottenLinks] 07Universal-Omega 036d69943 - Delete phplint.sh [02:19:32] [02RottenLinks] 07Universal-Omega synchronize pull request 03#41: CI: Add PHPCS and Phan - 13https://git.io/JEh5S [02:20:12] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://git.io/JEhdl [02:20:14] [02miraheze/RottenLinks] 07Universal-Omega 03b1f5b43 - Create phpcbf.sh [02:20:15] [02RottenLinks] 07Universal-Omega synchronize pull request 03#41: CI: Add PHPCS and Phan - 13https://git.io/JEh5S [02:20:17] miraheze/RottenLinks - Universal-Omega the build has errored. [02:20:25] miraheze/RottenLinks - Universal-Omega the build has errored. [02:21:04] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://git.io/JEhdu [02:21:06] [02miraheze/RottenLinks] 07Universal-Omega 03d3f2786 - Create composer.json [02:21:07] miraheze/RottenLinks - Universal-Omega the build has errored. [02:21:07] [02RottenLinks] 07Universal-Omega synchronize pull request 03#41: CI: Add PHPCS and Phan - 13https://git.io/JEh5S [02:21:43] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://git.io/JEhda [02:21:44] [02miraheze/RottenLinks] 07Universal-Omega 0386cba62 - Create config.php [02:21:46] [02RottenLinks] 07Universal-Omega synchronize pull request 03#41: CI: Add PHPCS and Phan - 13https://git.io/JEh5S [02:22:12] miraheze/RottenLinks - Universal-Omega the build has errored. [02:22:27] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://git.io/JEhdi [02:22:29] [02miraheze/RottenLinks] 07Universal-Omega 0336734ec - Create package.json [02:22:30] [02RottenLinks] 07Universal-Omega synchronize pull request 03#41: CI: Add PHPCS and Phan - 13https://git.io/JEh5S [02:22:53] miraheze/RottenLinks - Universal-Omega the build has errored. [02:23:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.52, 3.76, 3.48 [02:23:46] miraheze/RottenLinks - Universal-Omega the build has errored. [02:25:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.87, 3.55 [02:25:58] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JEhFa [02:26:00] [02miraheze/RottenLinks] 07Universal-Omega 03efa82ff - Update RottenLinksPager.php [02:26:01] [02RottenLinks] 07Universal-Omega synchronize pull request 03#41: CI: Add PHPCS and Phan - 13https://git.io/JEh5S [02:26:38] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JEhFD [02:26:40] [02miraheze/RottenLinks] 07Universal-Omega 037aec868 - Update RottenLinks.php [02:26:41] [02RottenLinks] 07Universal-Omega synchronize pull request 03#41: CI: Add PHPCS and Phan - 13https://git.io/JEh5S [02:27:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.03, 3.51, 3.46 [02:27:36] [02miraheze/RottenLinks] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±4] 13https://git.io/JEhFA [02:27:37] [02miraheze/RottenLinks] 07github-actions 03f5fb534 - CI: lint code to MediaWiki standards [02:27:39] [02RottenLinks] 07github-actions[bot] synchronize pull request 03#41: CI: Add PHPCS and Phan - 13https://git.io/JEh5S [02:28:00] miraheze/RottenLinks - Universal-Omega the build passed. [02:28:46] miraheze/RottenLinks - Universal-Omega the build passed. [02:31:07] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.19, 2.99, 3.27 [02:37:11] [02RottenLinks] 07Universal-Omega edited pull request 03#41: CI: Add PHPCS and Phan - 13https://git.io/JEh5S [02:39:12] [02RottenLinks] 07Universal-Omega closed pull request 03#41: CI: Add PHPCS and Phan - 13https://git.io/JEh5S [02:39:13] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03master [+7/-1/±5] 13https://git.io/JEhAa [02:39:15] [02miraheze/RottenLinks] 07Universal-Omega 03535ab5f - CI: Add PHPCS and Phan (#41) [02:39:16] [02RottenLinks] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/fxBrr [02:39:18] [02miraheze/RottenLinks] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [02:41:01] miraheze/RottenLinks - Universal-Omega the build passed. [02:43:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.36, 3.54, 3.34 [02:47:18] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.31, 3.50, 3.38 [02:49:20] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.43, 3.71, 3.46 [02:53:23] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.10, 3.82, 3.61 [03:05:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.76, 3.04, 3.33 [06:35:28] PROBLEM - www.winenjoy.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.winenjoy.net' expires in 15 day(s) (Sun 19 Sep 2021 06:31:12 GMT +0000). [06:38:42] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JEji1 [06:38:43] [02miraheze/ssl] 07MirahezeSSLBot 03dcfce14 - Bot: Update SSL cert for www.winenjoy.net [07:09:28] RECOVERY - www.winenjoy.net - LetsEncrypt on sslhost is OK: OK - Certificate 'www.winenjoy.net' will expire on Thu 02 Dec 2021 05:38:37 GMT +0000. [08:43:13] JohnLewis: I can't get the status colours to work on the mediawiki dashboard [08:45:26] Hm? [08:45:49] JohnLewis: look on the new mediawiki dashboard [08:45:57] Yeah [08:45:58] For the fact they repeat based on instance [08:46:06] There's 3 things showing on my screen [08:46:21] Organe Up, Green Up, one with numbers [08:46:54] I'm trying to replicate the old one [08:48:03] The colours will be because of the Threshold, where you’ve set 1 = orange [08:48:49] JohnLewis: I copied the nginx dashboard [08:48:56] It has 1 for both orange and green [08:50:26] Yes, but you have orange on top [08:50:40] Which means it checks green and then orange [08:50:53] JohnLewis: It didn't seem to drag and drop [08:50:55] Since orange is the last one to match, it returns orange [08:51:03] It doesn’t [08:51:46] JohnLewis: how do you move it then [08:51:58] You can’t [08:53:09] JohnLewis: you change the colours [08:53:17] That's one less issue [08:53:40] Yes, although why do you even need an orange if it’s 1/0 [08:54:24] JohnLewis: that's what the current dashboard does [08:54:48] Which makes no sense [08:54:50] I added an alerts box the php-fpm alert but I have no clue where the dashboard for that is set [08:55:10] JohnLewis: then why is that set [08:55:23] I don’t know, I didn’t make the dashboard [09:12:06] JohnLewis: php-fpm says no data now [09:19:32] Prometheus can’t find the data using the query given then likely. If you’re straight copying them, they should work. Best advice is look at the docs [09:22:08] JohnLewis: it's a straight copy [09:22:45] Something must be different or missing [09:23:04] Neither of the php fpm metrics work [09:23:48] That would imply something is missing if both metrics have no data [09:24:20] what though [09:24:29] I have zero knowledge of how grafana works [09:25:12] That’s why I recommended checking out the docs, otherwise building dashboards and metrics is going to extremely difficult [09:25:53] It already is extremely difficult [09:26:30] Because you said you have zero knowledge of how it works [09:29:04] Apart from allowing an all option [09:29:10] There is no difference I can see [09:33:12] phpfpm_up{instance="mw12\\.miraheze\\.org:9253"} [09:33:21] Is what is being executed [09:34:04] what's happening on the old php-fpm dashboard JohnLewis [09:34:58] phpfpm_up{instance="mw12.miraheze.org:9253"} [09:35:31] So it's escaping the text for some reason [09:37:08] JohnLewis: I don't see where that's coming from [09:37:41] I don’t either easily, but something must be different [09:38:27] JohnLewis: okay it's the all option [09:38:33] Which is a tad annoying [09:38:47] But why does nginx_up work [09:39:51] One has instance=, the other has instance~= [09:43:15] JohnLewis: fixed [09:54:33] Now we have https://phabricator.miraheze.org/T7963 [09:54:33] [url] ⚓ T7963 Move MediaWiki metrics to single dashboard | phabricator.miraheze.org [09:54:54] JohnLewis: how can we get grafana alerts to irc [09:55:08] Now we have a message on grafana for all alerts in appservers [09:55:16] https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1&viewPanel=21 [09:55:17] [url] Grafana | grafana.miraheze.org [09:56:39] Grafana doesn’t have built in support for IRC notifications, so either need to find/build an extension that allows it, or work around it with HTTP web requests etc [09:58:01] JohnLewis: there's webhook support https://grafana.com/docs/grafana/next/alerting/old-alerting/notifications/#webhook [09:58:02] [url] Alert notifications | Grafana Labs | grafana.com [09:58:46] I know there is; that is why I said otherwise use HTTP web requests [09:59:04] JohnLewis: how hard would it be for infra to set that up [09:59:13] Then we can move more important alerts [09:59:56] I’ve not even looked into it because it’s not currently a priority [10:00:53] If it’s important to MW-SRE, you can ask Reception as he’s the team’s SRE member [10:02:31] JohnLewis: well if we can do that then we can move alerts to grafana which you wanted and I'll look at replacing icinga alerts we have [10:03:30] I said that was the long term plan, but also stated it’s not a priority to add IRC notifications currently alongside other Infra work [10:03:56] But MW-SRE has an SRE member for the purpose of ensuring their work isn’t blocked on Infra [10:11:20] JohnLewis: if we could setup a flask app that listens for webhook events and stores it in an SQLite file and then a irc Python bot scraping that every 10s to SELECT latest data & delete read data, would that work? [10:11:55] Yes… but that sounds immensely more complicated than it needs to be [10:12:36] All you need is a webhook that appends the alert to the icinga monitoring file that icinga uses [10:13:48] Alternatively you could probably just create a new notification type that writes to a file on the server [10:14:59] JohnLewis: oh yeah icinga is just a file in reality [10:15:31] What format does the line being added need to be? [10:21:41] There’s no defined format [10:21:50] Whatever is put in, get ours out [10:41:54] JohnLewis: how do you trigger test alerts [10:41:57] On grafana [11:23:50] !log testing grafana alerts [11:23:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:06:48] JohnLewis, Reception123: pastebin.com/p3S3Sxcf [12:07:38] That should work for accepting a grafana webhook and writing to a file [12:19:58] https://phabricator.miraheze.org/T7964 [12:19:59] [url] ⚓ T7964 Setup icinga webhook listener for grafana | phabricator.miraheze.org [13:18:14] ok [13:23:42] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.00, 3.67, 3.12 [13:27:41] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.22, 3.36, 3.12 [13:28:14] Reception123: it will still need setting up but it works [13:28:39] ok, sounds good then [13:31:05] Reception123: the message links for mediawiki alerts to the mediawiki dashboard [13:31:54] great [13:32:26] Doesn't that affect all MediaWiki projects? [13:33:18] darkmatterman450: what do you mean [13:33:54] Reception123: I've created a task for copying panels across [13:34:05] We can have a nicely sorted graph [13:34:15] I meant the message links for MediaWiki alerts. [13:34:31] darkmatterman450: but what do you mean by all mediawiki projects [13:35:35] By that, I mean all MediaWiki projects like Wikimedia, Wikipedia, (aside from Miraheze), etc. [13:35:37] Ok [13:36:02] darkmatterman450: why would our grafana alerts affect anything other than Miraheze [13:36:31] I don't know. [13:37:13] darkmatterman450: neither do I [13:37:18] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.85, 5.97, 5.00 [13:37:40] Reception123: do we want to delete copied across alerts or just wait until the dashboards are gone? [13:38:16] I guess the latter is fine [13:38:38] Cool [13:38:43] I imagine it won't be long [13:38:48] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.97, 3.50, 3.23 [13:38:49] Job queue is nearly done [13:39:01] PROBLEM - pt.graalmilitary.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pt.graalmilitary.com' expires in 15 day(s) (Sun 19 Sep 2021 13:32:27 GMT +0000). [13:39:05] It's just the job specific graphs [13:39:16] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.41, 5.93, 5.09 [13:40:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.46, 3.16, 3.14 [13:40:53] PROBLEM - egoish.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'egoish.org' expires in 15 day(s) (Sun 19 Sep 2021 13:38:07 GMT +0000). [13:41:12] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JuvmZ [13:41:14] [02miraheze/ssl] 07MirahezeSSLBot 03ec72ce6 - Bot: Update SSL cert for pt.graalmilitary.com [13:44:19] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JuvYk [13:44:21] [02miraheze/ssl] 07MirahezeSSLBot 03d6bf70b - Bot: Update SSL cert for egoish.org [13:44:51] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.32, 3.59, 3.30 [13:46:36] Reception123: JobQueue dashboard done [13:46:41] Should I kill the old one [13:46:51] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.95, 3.58, 3.33 [13:48:51] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.60, 3.35 [13:50:35] Great and yeah if you're confident everything works [13:50:57] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.60, 3.38 [13:51:19] Reception123: double check while I add the NGINX ones [13:51:47] Currently mobile for a bit [13:52:35] Reception123: graphs should just be a visual check [13:52:57] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.01, 2.96, 3.17 [13:53:53] Seems fine [13:57:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.58, 3.97, 3.55 [13:57:35] !log delete mediawiki job queue dashboard [13:57:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:58:41] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.49, 6.38, 5.29 [13:59:33] Reception123: nginx also done [14:00:36] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.25, 5.51, 5.10 [14:01:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.38, 3.50, 3.49 [14:03:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.85, 2.98, 3.30 [14:03:55] JohnLewis: what does queued requests/max children reached/scrape failures mean [14:04:48] I see the purpose of uptime & accepted connections and I know what to do with the graphs to make them support all [14:05:00] Queued requests has no data whatsoever [14:05:54] Reception123: if nginx looks good too then shall I delete that [14:08:44] RECOVERY - egoish.org - LetsEncrypt on sslhost is OK: OK - Certificate 'egoish.org' will expire on Thu 02 Dec 2021 12:44:14 GMT +0000. [14:09:14] Queued requests is as it states in the name, max children reason is just a binary check for 100% capacity usage [14:11:30] JohnLewis: if we have the php worker usage do we need max children [14:11:45] And is queued requests needed given we're not ever queuing anything [14:13:22] RECOVERY - pt.graalmilitary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pt.graalmilitary.com' will expire on Thu 02 Dec 2021 12:41:06 GMT +0000. [14:23:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.48, 3.40, 3.33 [14:25:17] Data monitoring should be plentiful and contain the whole picture, if you don’t think data is relevant, Sod’s law will say it’ll be useful during the next incident [14:25:24] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.18, 2.93, 3.17 [14:28:20] JohnLewis: for the first one, I'm not seeing the difference between it & php worker usage [14:29:42] For queued requests we have the number & graph, one of them would tell us the same data [14:29:50] Graph is probably more useful if it is needed [14:40:54] !log delete nginx appservers from grafana [14:40:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:48:47] Cc Reception123 [14:49:42] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.53, 3.36, 3.20 [14:51:41] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.49, 3.71, 3.35 [14:53:41] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.53, 3.25, 3.23 [15:17:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.29, 3.19, 3.08 [15:19:53] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.81, 2.89, 2.98 [15:23:00] ack [15:24:01] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.96, 3.48, 3.25 [15:24:17] hi [15:25:15] Reception123: it's a question [15:25:34] Do we want both the stat & graph for queued requests if any [15:25:53] oh I didn't see that, I thought you just wanted to let me know that you deleted nginx [15:26:01] And what's the difference between worker usage and max children [15:26:19] as for the question I guess we could have both [15:26:19] Reception123: no I wanted help on these questions [15:26:28] Reception123: is there a benefit [15:26:37] and the second I'm not fully sure [15:26:49] I mean both show nothing whatsoever anyway but John points out Sod's law [15:26:50] maybe paladox ^ can help [15:27:19] Reception123: so copy across both graph and stat for queued requests? [15:27:35] I mean we could but as you say not sure if there is a benefit [15:28:00] I'll copy both [15:30:07] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.01, 3.76, 3.43 [15:32:18] Reception123: just max children reached left [15:32:28] As we want to know difference between that and usage [15:32:42] Then we have a single dashboard for mediawiki [15:32:56] oh [15:33:23] Reception123: oh? [15:33:43] I didn't realise that would lave us with a single dashboard [15:34:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.93, 3.81, 3.56 [15:34:25] Reception123: that was the entire point [15:34:54] Then we have the alerting at the top [15:35:15] Which will be new grafana based alerts rather than icinga [15:35:27] Ah, I see [15:35:31] So we have a single source of truth in an outage [15:35:43] We will still get irc alerts from icinga-miraheze [15:36:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.41, 3.27, 3.39 [16:18:58] JohnLewis: icinga web is down [16:21:03] https://phabricator.miraheze.org/T7967 [16:21:04] [url] ⚓ T7967 Icingaweb2 is down | phabricator.miraheze.org [16:54:07] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/Juvpu [16:54:08] [02miraheze/mw-config] 07Universal-Omega 03c9c0a0e - Update continuousIntegration.yml [16:54:10] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [16:54:15] [02mw-config] 07Universal-Omega opened pull request 03#4065: CI: Add PHPCS - 13https://git.io/Juvpg [16:55:01] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/Juvpi [16:55:03] [02miraheze/mw-config] 07Universal-Omega 03956c7fd - Update .gitignore [16:55:05] [02mw-config] 07Universal-Omega synchronize pull request 03#4065: CI: Add PHPCS - 13https://git.io/Juvpg [16:55:06] miraheze/mw-config - Universal-Omega the build has errored. [16:55:16] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-1/±0] 13https://git.io/JuvpD [16:55:17] [02miraheze/mw-config] 07Universal-Omega 03979ee03 - Delete phplint.sh [16:55:19] [02mw-config] 07Universal-Omega synchronize pull request 03#4065: CI: Add PHPCS - 13https://git.io/Juvpg [16:55:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+1/-0/±0] 13https://git.io/JuvpF [16:55:49] [02miraheze/mw-config] 07Universal-Omega 03f9b004a - Create phpcbf.sh [16:55:51] [02mw-config] 07Universal-Omega synchronize pull request 03#4065: CI: Add PHPCS - 13https://git.io/Juvpg [16:55:58] miraheze/mw-config - Universal-Omega the build has errored. [16:56:09] miraheze/mw-config - Universal-Omega the build has errored. [16:56:39] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+1/-0/±0] 13https://git.io/JuvhU [16:56:41] [02miraheze/mw-config] 07Universal-Omega 03bd37d44 - Create .phpcs.xml [16:56:42] [02mw-config] 07Universal-Omega synchronize pull request 03#4065: CI: Add PHPCS - 13https://git.io/Juvpg [16:56:46] miraheze/mw-config - Universal-Omega the build has errored. [16:57:37] miraheze/mw-config - Universal-Omega the build has errored. [16:57:52] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+1/-0/±0] 13https://git.io/JuvhB [16:57:53] [02miraheze/mw-config] 07Universal-Omega 03bae06fe - Create composer.json [16:57:55] [02mw-config] 07Universal-Omega synchronize pull request 03#4065: CI: Add PHPCS - 13https://git.io/Juvpg [16:59:01] miraheze/mw-config - Universal-Omega the build has errored. [17:01:07] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/Juvjs [17:01:08] [02miraheze/mw-config] 07Universal-Omega 0307b3ec3 - Update .phpcs.xml [17:01:10] [02mw-config] 07Universal-Omega synchronize pull request 03#4065: CI: Add PHPCS - 13https://git.io/Juvpg [17:02:15] miraheze/mw-config - Universal-Omega the build has errored. [17:02:50] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/Juvjo [17:02:52] [02miraheze/mw-config] 07Universal-Omega 0354392a0 - Update .phpcs.xml [17:02:53] [02mw-config] 07Universal-Omega synchronize pull request 03#4065: CI: Add PHPCS - 13https://git.io/Juvpg [17:03:57] miraheze/mw-config - Universal-Omega the build has errored. [17:04:33] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/Juvjh [17:04:34] [02miraheze/mw-config] 07Universal-Omega 035248fd0 - Update .phpcs.xml [17:04:36] [02mw-config] 07Universal-Omega synchronize pull request 03#4065: CI: Add PHPCS - 13https://git.io/Juvpg [17:05:03] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±9] 13https://git.io/Jufef [17:05:05] [02miraheze/mw-config] 07github-actions 030a33a4f - CI: lint code to MediaWiki standards [17:05:06] [02mw-config] 07github-actions[bot] synchronize pull request 03#4065: CI: Add PHPCS - 13https://git.io/Juvpg [17:05:40] miraheze/mw-config - Universal-Omega the build passed. [17:08:35] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/Jufea [17:08:36] [02miraheze/mw-config] 07Universal-Omega 0373d3b69 - Update LocalExtensions.php [17:08:38] [02mw-config] 07Universal-Omega synchronize pull request 03#4065: CI: Add PHPCS - 13https://git.io/Juvpg [17:09:38] miraheze/mw-config - Universal-Omega the build passed. [17:11:12] [02mw-config] 07Universal-Omega synchronize pull request 03#4065: CI: Add PHPCS - 13https://git.io/Juvpg [17:11:14] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JufeA [17:11:16] [02miraheze/mw-config] 07Universal-Omega 031739832 - Update ManageWikiExtensions.php [17:12:20] miraheze/mw-config - Universal-Omega the build passed. [17:14:58] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JufvK [17:15:00] [02miraheze/mw-config] 07Universal-Omega 03d10bc37 - Update LocalSettings.php [17:15:01] [02mw-config] 07Universal-Omega synchronize pull request 03#4065: CI: Add PHPCS - 13https://git.io/Juvpg [17:16:14] miraheze/mw-config - Universal-Omega the build passed. [17:22:23] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JuffP [17:22:24] [02miraheze/mw-config] 07Universal-Omega 039f6957a - Update LocalSettings.php [17:22:25] [02mw-config] 07Universal-Omega synchronize pull request 03#4065: CI: Add PHPCS - 13https://git.io/Juvpg [17:23:25] miraheze/mw-config - Universal-Omega the build passed. [17:31:30] [02mw-config] 07Universal-Omega edited pull request 03#4065: CI: Add PHPCS - 13https://git.io/Juvpg [18:10:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JuftC [18:10:27] [02miraheze/puppet] 07paladox 03a749274 - icingaweb2: Set ca path for the db [18:10:29] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [18:10:31] [02puppet] 07paladox opened pull request 03#1930: icingaweb2: Set ca path for the db - 13https://git.io/JuftW [18:11:21] [02puppet] 07paladox closed pull request 03#1930: icingaweb2: Set ca path for the db - 13https://git.io/JuftW [18:11:22] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/Juftr [18:11:24] [02miraheze/puppet] 07paladox 03a749274 - icingaweb2: Set ca path for the db [18:11:25] [02miraheze/puppet] 07paladox 037503ac2 - Merge pull request #1930 from miraheze/paladox-patch-2 [18:11:27] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [18:11:28] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [18:11:30] [02miraheze/puppet] 07paladox created branch 03paladox-patch-2 13https://git.io/Jufto [18:11:31] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [18:11:33] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [18:11:34] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [18:14:01] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.53, 3.21, 2.86 [18:16:01] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.92, 2.79, 2.76 [18:16:03] Reception123: around? [18:24:28] Ty paladox [18:41:42] [02phabricator-extensions] 07Universal-Omega opened pull request 03#15: CI: Add PHPCS - 13https://git.io/JufOg [18:42:47] miraheze/phabricator-extensions - Universal-Omega the build has errored. [18:43:54] [02phabricator-extensions] 07Universal-Omega synchronize pull request 03#15: CI: Add PHPCS - 13https://git.io/JufOg [18:44:49] miraheze/phabricator-extensions - Universal-Omega the build has errored. [18:45:55] [02phabricator-extensions] 07Universal-Omega synchronize pull request 03#15: CI: Add PHPCS - 13https://git.io/JufOg [18:46:51] miraheze/phabricator-extensions - Universal-Omega the build passed. [18:50:01] [02phabricator-extensions] 07Universal-Omega synchronize pull request 03#15: CI: Add PHPCS - 13https://git.io/JufOg [18:50:58] miraheze/phabricator-extensions - Universal-Omega the build has errored. [18:55:00] [02phabricator-extensions] 07Universal-Omega synchronize pull request 03#15: CI: Add PHPCS - 13https://git.io/JufOg [18:55:55] miraheze/phabricator-extensions - Universal-Omega the build passed. [18:57:10] paladox: I'll be more around tomorrow, is there anything you needed? [18:58:42] [02phabricator-extensions] 07Universal-Omega synchronize pull request 03#15: CI: Add PHPCS - 13https://git.io/JufOg [18:59:40] miraheze/phabricator-extensions - Universal-Omega the build passed. [19:03:16] [02phabricator-extensions] 07Universal-Omega synchronize pull request 03#15: CI: Add PHPCS - 13https://git.io/JufOg [19:03:35] [02phabricator-extensions] 07github-actions[bot] synchronize pull request 03#15: CI: Add PHPCS - 13https://git.io/JufOg [19:04:17] miraheze/phabricator-extensions - Universal-Omega the build passed. [19:05:31] [02phabricator-extensions] 07Universal-Omega synchronize pull request 03#15: CI: Add PHPCS - 13https://git.io/JufOg [19:06:31] miraheze/phabricator-extensions - Universal-Omega the build passed. [19:08:28] [02phabricator-extensions] 07Universal-Omega edited pull request 03#15: CI: Add PHPCS - 13https://git.io/JufOg [19:38:04] Reception123: just wanted to see if you could run puppet on mon2 but nvm as it's ran [19:38:34] Ok, sorry I couldn't then :( [19:38:43] But good thing it ran [19:40:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-0/±0] 13https://git.io/JufCy [19:40:15] [02miraheze/puppet] 07paladox 039046e16 - Set config_resource to icinga_ido [19:40:16] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [19:40:18] [02puppet] 07paladox opened pull request 03#1931: Set config_resource to icinga_ido - 13https://git.io/JufC9 [19:40:22] [02puppet] 07paladox edited pull request 03#1931: icingaweb2: Set config_resource to icinga_ido - 13https://git.io/JufC9 [19:41:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JufCF [19:41:40]