[00:05:56] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 33.51, 23.70, 18.39 [00:06:18] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.74, 6.09, 5.28 [00:06:58] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 6 minutes ago with 8 failures. Failed resources (up to 3 shown): File[/etc/sysctl.d],File[/etc/logrotate.d/nginx],File[/home/johnflewis],File[/home/paladox] [00:33:54] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:44:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.59, 3.05 [00:47:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.04, 3.01, 2.92 [00:57:29] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 14.40, 17.86, 23.42 [01:08:41] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.58, 16.75, 19.97 [01:09:55] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.20, 4.58, 5.96 [01:15:20] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.48, 5.63, 5.92 [01:18:05] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.31, 5.19, 5.73 [01:31:41] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.10, 6.67, 6.09 [01:34:31] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.62, 5.97, 5.94 [01:37:17] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.32, 6.04, 5.95 [01:42:42] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.91, 5.24, 5.71 [01:50:56] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.35, 5.63, 5.58 [01:53:40] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.70, 5.02, 5.35 [01:56:25] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.91, 4.23, 4.99 [02:20:44] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.97, 5.35, 4.88 [02:23:25] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.02, 5.49, 5.00 [02:26:15] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.34, 4.97, 4.85 [02:28:59] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.25, 4.44, 4.65 [02:38:25] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.32, 6.94, 5.74 [02:46:32] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.19, 5.77, 5.67 [02:48:12] [02mw-config] 07Universal-Omega commented on pull request 03#4155: Remove Collection & Citoid - 13https://git.io/J6Tl3 [02:48:14] [02mw-config] 07Universal-Omega closed pull request 03#4155: Remove Collection & Citoid - 13https://git.io/JK0ZF [02:48:51] [02MirahezeMagic] 07Universal-Omega commented on pull request 03#294: Services: delete - 13https://git.io/J6T8n [02:48:52] [02MirahezeMagic] 07Universal-Omega closed pull request 03#294: Services: delete - 13https://git.io/JK01O [02:48:56] [02MirahezeMagic] 07Universal-Omega deleted branch 03RhinosF1-patch-1 - 13https://git.io/fQRGX [02:48:57] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03RhinosF1-patch-1 [02:49:04] [02miraheze/mw-config] 07Universal-Omega deleted branch 03RhinosF1-patch-1 [02:49:06] [02mw-config] 07Universal-Omega deleted branch 03RhinosF1-patch-1 - 13https://git.io/vbvb3 [02:49:42] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-1/±0] 13https://git.io/J6T4K [02:49:43] [02miraheze/MirahezeMagic] 07Universal-Omega 03ab8cb28 - Delete addWikiToServices.php [02:49:45] [02MirahezeMagic] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/fQRGX [02:49:46] [02MirahezeMagic] 07Universal-Omega opened pull request 03#297: Delete addWikiToServices.php - 13https://git.io/J6T4M [02:50:41] miraheze/MirahezeMagic - Universal-Omega the build passed. [02:51:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/J6T0V [02:51:42] [02miraheze/mw-config] 07Universal-Omega 039c83d8e - Update ManageWikiExtensions.php [02:51:44] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [02:52:11] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/J6TE3 [02:52:13] [02miraheze/mw-config] 07Universal-Omega 03748970c - Update extension-list [02:52:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/J6TuO [02:52:49] [02miraheze/mw-config] 07Universal-Omega 03ab417c3 - Update LocalSettings.php [02:53:56] [02mw-config] 07Universal-Omega opened pull request 03#4161: Remove Citoid - 13https://git.io/J6Tzy [02:55:04] miraheze/mw-config - Universal-Omega the build passed. [02:57:17] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.90, 4.32, 4.88 [03:51:07] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://git.io/J6k5R [03:51:08] [02miraheze/mw-config] 07Universal-Omega 03e933640 - Add Citoid sitenotice [03:51:10] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-5 - 13https://git.io/vbvb3 [03:51:24] [02mw-config] 07Universal-Omega opened pull request 03#4162: Add Citoid sitenotice - 13https://git.io/J6k5A [03:52:27] miraheze/mw-config - Universal-Omega the build passed. [04:16:21] PROBLEM - index.pdcommunity.ir - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - index.pdcommunity.ir All nameservers failed to answer the query. [04:20:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J6Ian [04:20:14] [02miraheze/services] 07MirahezeSSLBot 0353b40b2 - BOT: Updating services config for wikis [04:23:56] RECOVERY - index.pdcommunity.ir - reverse DNS on sslhost is OK: SSL OK - index.pdcommunity.ir reverse DNS resolves to cp14.miraheze.org - CNAME OK [04:27:57] [02mw-config] 07dmehus commented on pull request 03#4162: Add Citoid sitenotice - 13https://git.io/J6IDe [04:28:24] [02mw-config] 07dmehus edited a comment on pull request 03#4162: Add Citoid sitenotice - 13https://git.io/J6IDe [04:29:16] [02mw-config] 07Universal-Omega commented on pull request 03#4162: Add Citoid sitenotice - 13https://git.io/J6ISt [04:30:16] [02mw-config] 07dmehus edited a comment on pull request 03#4162: Add Citoid sitenotice - 13https://git.io/J6IDe [04:32:10] [02mw-config] 07dmehus commented on pull request 03#4162: Add Citoid sitenotice - 13https://git.io/J6I7N [04:32:41] [02mw-config] 07dmehus edited a comment on pull request 03#4162: Add Citoid sitenotice - 13https://git.io/J6I7N [04:32:44] [02mw-config] 07Universal-Omega commented on pull request 03#4162: Add Citoid sitenotice - 13https://git.io/J6I5d [04:33:31] [02mw-config] 07dmehus edited a comment on pull request 03#4162: Add Citoid sitenotice - 13https://git.io/J6I7N [04:34:25] [02mw-config] 07dmehus commented on pull request 03#4162: Add Citoid sitenotice - 13https://git.io/J6Ibt [04:37:29] [02mw-config] 07Universal-Omega commented on pull request 03#4162: Add Citoid sitenotice - 13https://git.io/J6IpM [04:38:19] [02mw-config] 07Universal-Omega edited a comment on pull request 03#4162: Add Citoid sitenotice - 13https://git.io/J6IpM [04:38:52] [02mw-config] 07dmehus commented on pull request 03#4162: Add Citoid sitenotice - 13https://git.io/J6Ijp [04:38:57] dmehus: that discussion is getting a bit off-topic for a PR discussion. [04:39:29] [02mw-config] 07dmehus edited a comment on pull request 03#4162: Add Citoid sitenotice - 13https://git.io/J6Ijp [04:41:33] Agent: thoughts on comments above (about the informational page) [04:41:39] [02mw-config] 07dmehus edited a comment on pull request 03#4162: Add Citoid sitenotice - 13https://git.io/J6Ijp [04:52:55] * dmehus hopes Agent doesn't mind he drafted him :D [04:54:32] doesn't have to be too long, just a paragraph or a number of bullet points for each section... maybe like about two-thirds as long as the [[ManageWiki]] page [04:54:48] just similar organizationally, but shorter [05:44:59] [02puppet] 07Reception123 closed pull request 03#2057: prometheus-es-exporter: query last one minute - 13https://git.io/J6eJE [05:45:01] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/J6tKC [05:45:02] [02miraheze/puppet] 07Universal-Omega 03883fc23 - prometheus-es-exporter: query last one minute (#2057) [05:50:16] [02puppet] 07Universal-Omega synchronize pull request 03#2039: Absent services crons - 13https://git.io/JK0M1 [09:24:27] ack, you all always catch me when I'm sleeping [09:24:33] Drafting a page would be a good idea imo [09:40:11] [02mw-config] 07JohnFLewis commented on pull request 03#4162: Add Citoid sitenotice - 13https://git.io/J6sfX [09:54:15] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03revert-4160-Universal-Omega-patch-2 [+0/-0/±5] 13https://git.io/J6sW1 [09:54:16] [02miraheze/mw-config] 07Reception123 0363a3d02 - Revert "Remove Collection + ElectronPdfService (#4160)" [09:54:18] [02mw-config] 07Reception123 created branch 03revert-4160-Universal-Omega-patch-2 - 13https://git.io/vbvb3 [09:54:19] [02mw-config] 07Reception123 opened pull request 03#4163: Revert "Remove Collection + ElectronPdfService" - 13https://git.io/J6sWH [09:54:39] [02mw-config] 07Reception123 commented on pull request 03#4163: Revert "Remove Collection + ElectronPdfService" - 13https://git.io/J6sl8 [09:55:15] miraheze/mw-config - Reception123 the build passed. [09:55:49] [02mw-config] 07Reception123 closed pull request 03#4163: Revert "Remove Collection + ElectronPdfService" - 13https://git.io/J6sWH [09:55:50] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±5] 13https://git.io/J6s4f [09:55:52] [02miraheze/mw-config] 07Reception123 03d87f6d0 - Revert "Remove Collection + ElectronPdfService (#4160)" (#4163) [09:55:53] [02miraheze/mw-config] 07Reception123 deleted branch 03revert-4160-Universal-Omega-patch-2 [09:55:55] [02mw-config] 07Reception123 deleted branch 03revert-4160-Universal-Omega-patch-2 - 13https://git.io/vbvb3 [09:57:05] miraheze/mw-config - Reception123 the build passed. [09:57:58] !log [reception@mw11] starting deploy of {'config': True, 'world': True, 'l10n': True, 'gitinfo': True} to all [09:58:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:03:12] !log [@test3] starting deploy of {'config': True} to skip [10:03:13] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [10:03:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:03:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:06:22] !log [@mw11] starting deploy of {'config': True} to all [10:06:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:06:34] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 11s [10:06:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:17:06] !log [reception@mw11] finished deploy of {'config': True, 'world': True, 'l10n': True, 'gitinfo': True} to all - SUCCESS in 1148s [10:17:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:21:17] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J6syz [10:21:18] [02miraheze/puppet] 07JohnFLewis 033bfd5ad - bacula: increase concurrent jobs to 2 [10:40:45] PROBLEM - inourownwords.online - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'inourownwords.online' expires in 7 day(s) (Wed 27 Oct 2021 10:35:52 GMT +0000). [12:18:28] PROBLEM - thestarsareright.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'thestarsareright.org' expires in 15 day(s) (Thu 04 Nov 2021 12:12:37 GMT +0000). [12:44:32] !log disabled a spammer account on phab [12:44:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:48:37] @Reception123 Why not also lock the human spammer too as well? Or will that not be necessary for now? [12:49:38] done [12:49:49] Okay, thanks. [14:44:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03migrate-to-ferm [+19/-1/±2] 13https://git.io/J6CKZ [14:44:52] [02miraheze/puppet] 07paladox 03a11fd8d - Migrate to ferm [14:44:54] [02puppet] 07paladox created branch 03migrate-to-ferm - 13https://git.io/vbiAS [14:45:07] [02puppet] 07paladox opened pull request 03#2058: Migrate to ferm - 13https://git.io/J6CK6 [16:09:37] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.33, 21.44, 18.29 [16:10:17] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 5.19, 8.19, 6.80 [16:12:34] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.73, 18.45, 17.70 [16:13:05] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.35, 6.73, 6.45 [16:57:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03migrate-to-ferm [+19/-1/±39] 13https://git.io/J68QM [16:57:38] [02miraheze/puppet] 07paladox 03ba0f26f - Migrate to ferm [16:57:40] [02puppet] 07paladox synchronize pull request 03#2058: Migrate to ferm - 13https://git.io/J6CK6 [17:00:03] large change :P [17:00:13] haven't tested though i think i've converted everything [17:08:02] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 5.84, 8.22, 6.61 [17:08:29] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.55, 7.28, 6.20 [17:10:57] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.10, 6.61, 6.25 [17:11:26] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.77, 6.12, 5.93 [17:20:16] PROBLEM - wiki.astralprojections.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.astralprojections.org' expires in 7 day(s) (Wed 27 Oct 2021 17:16:54 GMT +0000). [17:31:21] [02puppet] 07Universal-Omega opened pull request 03#2059: grafana-webhook: remove ?orgId=1 from URL - 13https://git.io/J64ww [17:32:28] JohnLewis: by the way, just wanted to confirm with you, but does https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki provide all the information you had in mind when creating https://phabricator.miraheze.org/T6979? Just want to make sure there wasn't anything else you wanted. [17:32:30] [url] ⚓ T6979 Collect Statistics for API Requests (Including Module Type) | phabricator.miraheze.org [17:34:11] CosmicAlpha: looks pretty good [17:40:00] Great! [18:05:51] [02puppet] 07paladox closed pull request 03#2059: grafana-webhook: remove ?orgId=1 from URL - 13https://git.io/J64ww [18:05:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J6BnX [18:05:54] [02miraheze/puppet] 07Universal-Omega 032350675 - grafana-webhook: remove ?orgId=1 from URL (#2059) [18:37:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:39:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.17, 7.75, 6.72 [18:40:50] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.20, 7.99, 7.04 [18:42:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:42:34] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.24, 6.59, 6.45 [18:43:46] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.89, 6.61, 6.67 [19:01:09] PROBLEM - cloud5 APT on cloud5 is CRITICAL: APT CRITICAL: 85 packages available for upgrade (7 critical updates). [19:06:41] PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 26 packages available for upgrade (10 critical updates). [19:07:15] PROBLEM - mw8 APT on mw8 is CRITICAL: APT CRITICAL: 26 packages available for upgrade (10 critical updates). [19:07:26] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.01, 7.36, 6.62 [19:07:40] PROBLEM - mw11 APT on mw11 is CRITICAL: APT CRITICAL: 26 packages available for upgrade (10 critical updates). [19:09:47] PROBLEM - mw13 APT on mw13 is CRITICAL: APT CRITICAL: 26 packages available for upgrade (10 critical updates). [19:10:17] PROBLEM - mw10 APT on mw10 is CRITICAL: APT CRITICAL: 26 packages available for upgrade (10 critical updates). [19:10:27] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.80, 6.76, 6.49 [19:11:14] PROBLEM - mw12 APT on mw12 is CRITICAL: APT CRITICAL: 26 packages available for upgrade (10 critical updates). [19:13:39] PROBLEM - services4 APT on services4 is CRITICAL: APT CRITICAL: 21 packages available for upgrade (4 critical updates). [19:16:43] PROBLEM - services3 APT on services3 is CRITICAL: APT CRITICAL: 21 packages available for upgrade (4 critical updates). [19:17:17] PROBLEM - cloud4 APT on cloud4 is CRITICAL: APT CRITICAL: 85 packages available for upgrade (7 critical updates). [19:22:14] PROBLEM - cloud3 APT on cloud3 is CRITICAL: APT CRITICAL: 119 packages available for upgrade (7 critical updates). [19:40:51] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.87, 7.58, 6.52 [19:43:43] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.41, 6.17, 6.16 [19:55:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J60ME [19:55:07] [02miraheze/puppet] 07paladox 0352e1aa6 - gluster: Upgrade to version 9 [19:55:09] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [19:55:10] [02puppet] 07paladox opened pull request 03#2060: gluster: Upgrade to version 9 - 13https://git.io/J60M2 [19:55:35] [02puppet] 07paladox closed pull request 03#2060: gluster: Upgrade to version 9 - 13https://git.io/J60M2 [19:55:36] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [19:55:38] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [19:55:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J60DU [19:55:41] [02miraheze/puppet] 07paladox 03f94c8ae - gluster: Upgrade to version 9 (#2060) [19:55:56] waiting for 5 mins to apply... just getting things ready [19:59:54] Would updating the sitenotice to remind users of on-going maintenance be good? [20:00:42] I'm not too sure, possibly, though the current site notice includes the date and time. [20:01:05] @Agent do you think that'll be beneficial? [20:01:21] Reminder that maintenance is starting [20:01:34] !log make file storage read-only on gluster (gluster volume set mvol read-only enable) [20:01:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:01:43] oh i forgot we renamed, fixing [20:02:07] paladox: I believe so as I'm sure someones going to be wondering why they can't upload images. Perhaps update it to say something like "Maintenance on our file storage system is currently under way. You may not be able to upload images right now until maintenance finishes at 21:00UTC. We apologize for the inconvenience" [20:03:22] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J60Fw [20:03:23] [02miraheze/mw-config] 07paladox 031a9bbc3 - Update site notice [20:03:25] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [20:03:26] Agent: ^ [20:03:26] [02mw-config] 07paladox opened pull request 03#4164: Update site notice - 13https://git.io/J60FK [20:04:02] Checking... [20:04:17] Thanks! [20:04:25] miraheze/mw-config - paladox the build passed. [20:04:27] paladox: Looks good to me :) [20:04:31] thanks! [20:04:38] [02mw-config] 07paladox closed pull request 03#4164: Update site notice - 13https://git.io/J60FK [20:04:40] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J60Nc [20:04:41] [02miraheze/mw-config] 07paladox 03c8c4db9 - Update site notice (#4164) [20:04:43] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [20:04:44] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [20:05:07] !log [paladox@mw11] starting deploy of {'config': True, 'gitinfo': True} to all [20:05:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:50] miraheze/mw-config - paladox the build passed. [20:06:22] !log [paladox@mw11] finished deploy of {'config': True, 'gitinfo': True} to all - SUCCESS in 75s [20:06:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:06:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:06:33] !log upgrade gluster 3 & 4 to version 9 [20:06:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:08:42] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.44, 9.26, 6.99 [20:09:00] !log remove read-only from gluster [20:09:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:10:29] !log upgrade mediawiki application servers to gluster 9 [20:10:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:11:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:11:40] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.18, 7.59, 6.75 [20:14:35] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.64, 6.56, 6.48 [20:21:57] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6EZI [20:21:59] [02miraheze/mw-config] 07paladox 036fefa04 - Maintenance is over [20:22:00] !log maintenance over [20:22:00] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [20:22:02] [02mw-config] 07paladox opened pull request 03#4165: Maintenance is over - 13https://git.io/J6EZt [20:22:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:22:07] [02mw-config] 07paladox closed pull request 03#4165: Maintenance is over - 13https://git.io/J6EZt [20:22:09] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [20:22:10] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [20:22:12] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J6EZC [20:22:13] [02miraheze/mw-config] 07paladox 03fedbe8c - Maintenance is over (#4165) [20:22:21] !log [paladox@mw11] starting deploy of {'config': True, 'gitinfo': True} to all [20:22:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:22:38] !log [paladox@mw11] finished deploy of {'config': True, 'gitinfo': True} to all - SUCCESS in 17s [20:22:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:23:02] miraheze/mw-config - paladox the build passed. [20:23:13] miraheze/mw-config - paladox the build passed. [20:33:12] !log [@test3] starting deploy of {'config': True} to skip [20:33:13] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [20:33:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:33:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:32:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03migrate-to-ferm [+114/-61/±195] 13https://git.io/J6uH4 [21:32:03] [02miraheze/puppet] 07paladox 03e7321f9 - Add support for ferm to base [21:32:04] [02puppet] 07paladox synchronize pull request 03#2058: Migrate to ferm - 13https://git.io/J6CK6 [21:32:32] [02puppet] 07paladox edited pull request 03#2058: Migrate to ferm - 13https://git.io/J6CK6 [21:32:34] [02puppet] 07paladox edited pull request 03#2058: Add support for ferm to base - 13https://git.io/J6CK6 [21:46:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03migrate-to-ferm [+19/-1/±2] 13https://git.io/J6zTt [21:46:10] [02miraheze/puppet] 07paladox 03faf4264 - Add support for ferm to base [21:46:12] [02puppet] 07paladox synchronize pull request 03#2058: Add support for ferm to base - 13https://git.io/J6CK6 [21:55:18] [02puppet] 07paladox closed pull request 03#2058: Add support for ferm to base - 13https://git.io/J6CK6 [21:55:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+19/-1/±2] 13https://git.io/J6zsA [21:55:21] [02miraheze/puppet] 07paladox 0399afd4f - Add support for ferm to base (#2058) [21:55:22] [02puppet] 07paladox deleted branch 03migrate-to-ferm - 13https://git.io/vbiAS [21:55:24] [02miraheze/puppet] 07paladox deleted branch 03migrate-to-ferm [22:02:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6z45 [22:02:18] [02miraheze/puppet] 07paladox 03d0736d0 - mediawiki: Migrate to ferm [22:02:19] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [22:02:21] [02puppet] 07paladox opened pull request 03#2061: mediawiki: Migrate to ferm - 13https://git.io/J6z4N [22:04:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6z0y [22:04:23] [02miraheze/puppet] 07paladox 03aef7c73 - Update mediawiki.pp [22:04:25] [02puppet] 07paladox synchronize pull request 03#2061: mediawiki: Migrate to ferm - 13https://git.io/J6z4N [22:08:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+6/-0/±0] 13https://git.io/J6zaL [22:08:53] [02miraheze/puppet] 07paladox 030eafc26 - Add missing ulogd module for ferm [22:10:21] PROBLEM - mw13 Puppet on mw13 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [22:10:37] PROBLEM - mw12 Puppet on mw12 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [22:10:56] PROBLEM - mw10 Puppet on mw10 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 0 failures [22:11:02] PROBLEM - mw8 Puppet on mw8 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 0 failures [22:11:07] PROBLEM - mwtask1 Puppet on mwtask1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 30 minutes ago with 0 failures [22:11:10] PROBLEM - mw11 Puppet on mw11 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [22:11:37] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:11:38] PROBLEM - mw9 Puppet on mw9 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 0 failures [22:16:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6z1J [22:16:06] [02miraheze/puppet] 07paladox 0376a527f - Update php_fpm.pp [22:16:08] [02puppet] 07paladox synchronize pull request 03#2061: mediawiki: Migrate to ferm - 13https://git.io/J6z4N [22:19:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6z98 [22:20:00] [02miraheze/puppet] 07paladox 037edccae - Update nginx.pp [22:20:02] [02puppet] 07paladox synchronize pull request 03#2061: mediawiki: Migrate to ferm - 13https://git.io/J6z4N [22:24:54] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+1/-0/±1] 13https://git.io/J6zFK [22:24:56] [02miraheze/mediawiki] 07Universal-Omega 031d8215e - Install RatePage [22:26:03] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/J6zbh [22:26:05] [02miraheze/mediawiki] 07Universal-Omega 03c754b68 - Update non-rel-extensions [22:26:51] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+1/-0/±1] 13https://git.io/J6zNh [22:26:53] [02miraheze/mediawiki] 07Universal-Omega 0347c2b22 - Install RatePage [22:27:04] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/J6zA3 [22:27:05] [02miraheze/mediawiki] 07Universal-Omega 03c7a8334 - Update non-rel-extensions [22:31:10] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+1/-0/±1] 13https://git.io/J6geH [22:31:12] [02miraheze/mediawiki] 07Universal-Omega 03f91fd8b - Install Flatbox [22:32:33] CosmicAlpha: ty for updating non-rel, I'm going to move it a new format soon because I want a script for the branch cut and etc [22:32:54] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+1/-0/±1] 13https://git.io/J6gJI [22:32:55] [02miraheze/mediawiki] 07Universal-Omega 03c4127e3 - Install Flatbox [22:33:49] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/J6gU8 [22:33:50] [02miraheze/mediawiki] 07Universal-Omega 03548d7c7 - Update non-rel-extensions [22:34:04] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/J6gU7 [22:34:06] [02miraheze/mediawiki] 07Universal-Omega 03c948591 - Update non-rel-extensions [22:34:32] Spookreeeno: No problem [22:35:05] CosmicAlpha: I might if I'm bored enough allow automating adding submodule a [22:35:15] Depends how much time I get next week [22:37:04] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+1/-0/±1] 13https://git.io/J6gLb [22:37:06] [02miraheze/mediawiki] 07Universal-Omega 03d5895d9 - Install ArticleCreationWorkflow [22:38:44] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+1/-0/±1] 13https://git.io/J6gm3 [22:38:46] [02miraheze/mediawiki] 07Universal-Omega 0362118ce - Install ArticleCreationWorkflow [22:39:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6gYg [22:39:40] [02miraheze/puppet] 07paladox 0392a5241 - Update firewall.pp [22:39:41] [02puppet] 07paladox synchronize pull request 03#2061: mediawiki: Migrate to ferm - 13https://git.io/J6z4N [22:41:04] Spookreeeno: Sounds good! [22:42:02] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+1/-0/±1] 13https://git.io/J6gsK [22:42:03] [02miraheze/mediawiki] 07Universal-Omega 039d758a1 - Install Mirage [22:43:56] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+1/-0/±1] 13https://git.io/J6gnk [22:43:58] [02miraheze/mediawiki] 07Universal-Omega 032393be4 - Install Mirage [22:45:31] paladox: Am I ok to run deploy-mediawiki or do I need to wait? [22:45:44] You can proceed [22:51:52] Thanks! [22:52:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6guC [22:52:40] [02miraheze/puppet] 07paladox 03e28ccd6 - Update mediawiki.pp [22:52:41] [02puppet] 07paladox synchronize pull request 03#2061: mediawiki: Migrate to ferm - 13https://git.io/J6z4N [22:53:13] !log [universalomega@mw11] starting deploy of {'world': True, 'gitinfo': True} to all [22:53:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:53:28] !log [universalomega@test3] starting deploy of {'world': True, 'gitinfo': True} to skip [22:53:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:57:49] !log [universalomega@test3] finished deploy of {'world': True, 'gitinfo': True} to skip - SUCCESS in 260s [22:57:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:59:24] !log [universalomega@mw11] finished deploy of {'world': True, 'gitinfo': True} to all - SUCCESS in 370s [22:59:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:05:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6gyv [23:05:41] [02miraheze/puppet] 07paladox 0335d387b - Update node_exporter.pp [23:05:42] [02puppet] 07paladox synchronize pull request 03#2061: mediawiki: Migrate to ferm - 13https://git.io/J6z4N [23:11:05] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+1/-0/±0] 13https://git.io/J6gde [23:11:07] [02miraheze/CreateWiki] 07Universal-Omega 03bf8b533 - Create RebuildExtensionListCache [23:11:08] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-4 - 13https://git.io/vpJTL [23:11:14] [02CreateWiki] 07Universal-Omega opened pull request 03#251: Create RebuildExtensionListCache - 13https://git.io/J6gdO [23:11:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6gdn [23:11:19] [02miraheze/puppet] 07paladox 034e4dbf1 - Update firewall.pp [23:11:20] [02puppet] 07paladox synchronize pull request 03#2061: mediawiki: Migrate to ferm - 13https://git.io/J6z4N [23:11:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6gFU [23:11:50] [02miraheze/puppet] 07paladox 03458d638 - Update mediawiki.pp [23:11:51] [02puppet] 07paladox synchronize pull request 03#2061: mediawiki: Migrate to ferm - 13https://git.io/J6z4N [23:12:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6gFz [23:12:11] [02miraheze/puppet] 07paladox 03ea3a87f - Update node_exporter.pp [23:12:12] [02puppet] 07paladox synchronize pull request 03#2061: mediawiki: Migrate to ferm - 13https://git.io/J6z4N [23:12:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6gFQ [23:12:26] [02miraheze/puppet] 07paladox 03700152f - Update firewall.pp [23:12:28] [02puppet] 07paladox synchronize pull request 03#2061: mediawiki: Migrate to ferm - 13https://git.io/J6z4N [23:12:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6gbO [23:12:45] [02miraheze/puppet] 07paladox 03fb84712 - Update nginx.pp [23:12:47] [02puppet] 07paladox synchronize pull request 03#2061: mediawiki: Migrate to ferm - 13https://git.io/J6z4N [23:15:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6gxk [23:15:08] [02miraheze/puppet] 07paladox 030a86cd3 - Update php_fpm.pp [23:15:10] [02puppet] 07paladox synchronize pull request 03#2061: mediawiki: Migrate to ferm - 13https://git.io/J6z4N [23:16:16] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://git.io/J6gpc [23:16:17] [02miraheze/CreateWiki] 07Universal-Omega 03294b26f - Update WikiInitialise.php [23:16:19] [02CreateWiki] 07Universal-Omega synchronize pull request 03#251: Create RebuildExtensionListCache - 13https://git.io/J6gdO [23:16:38] [02CreateWiki] 07Universal-Omega edited pull request 03#251: Create RebuildExtensionListCache maintenance script - 13https://git.io/J6gdO [23:17:26] Icinga guest/guest doesn't work [23:18:20] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [23:21:04] [02puppet] 07Universal-Omega opened pull request 03#2062: deploy-mediawiki: add --extensionlist argument - 13https://git.io/J62fQ [23:21:40] Bukkit: That login was disabled I think. [23:22:17] Oh, okay. Can I remove it from [[Tech:Home]] then? [23:22:17]

https://meta.miraheze.org/wiki/Tech:Home [23:22:19] [url] Tech:Home - Miraheze Meta | meta.miraheze.org [23:23:00] :DoneMH: Done [23:23:02] miraheze/CreateWiki - Universal-Omega the build passed. [23:23:11] Thanks, Agent. [23:23:14] I believe grafana is the replacement for it [23:23:57] [02puppet] 07Universal-Omega synchronize pull request 03#2062: deploy-mediawiki: add --extensionlist argument - 13https://git.io/J62fQ [23:24:01] For community eyes, I believe so [23:24:06] [02puppet] 07Universal-Omega edited pull request 03#2062: deploy-mediawiki: add --extension-list argument - 13https://git.io/J62fQ [23:24:15] I know SRE still uses it [23:24:56] Any luck on FAMEPedia's outage? [23:26:06] [02CreateWiki] 07Universal-Omega closed pull request 03#251: Create RebuildExtensionListCache maintenance script - 13https://git.io/J6gdO [23:26:08] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/J62tk [23:26:09] [02miraheze/CreateWiki] 07Universal-Omega 03d77d5ce - Create RebuildExtensionListCache maintenance script (#251) [23:26:11] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 - 13https://git.io/vpJTL [23:26:12] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 [23:26:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+1/-0/±0] 13https://git.io/J62tW [23:26:18] PROBLEM - test3 Check Gluster Clients on test3 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [23:26:18] [02miraheze/puppet] 07paladox 0305f6e51 - Set base::firewall::mode to ferm for all mediawiki application servers [23:26:20] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [23:26:21] [02puppet] 07paladox opened pull request 03#2063: Set base::firewall::mode to ferm for all mediawiki application servers - 13https://git.io/J62tR [23:26:28] [02puppet] 07paladox closed pull request 03#2063: Set base::firewall::mode to ferm for all mediawiki application servers - 13https://git.io/J62tR [23:26:30] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [23:26:31] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [23:26:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/J62to [23:26:34] [02miraheze/puppet] 07paladox 0363abc8f - Set base::firewall::mode to ferm for all mediawiki application servers (#2063) [23:26:36] [02puppet] 07paladox synchronize pull request 03#2061: mediawiki: Migrate to ferm - 13https://git.io/J6z4N [23:26:56] hmm @ that test3 check [23:28:15] RECOVERY - test3 Check Gluster Clients on test3 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [23:28:54] ah [23:29:13] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/J62Oa [23:29:14] [02miraheze/mediawiki] 07Universal-Omega 03f416a6a - Update CreateWiki [23:30:52] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/J62GI [23:30:54] [02miraheze/mediawiki] 07Universal-Omega 03d7bc57c - Update CreateWiki [23:32:13] !log [universalomega@test3] starting deploy of {'world': True, 'gitinfo': True} to skip [23:32:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:33:04]