[00:03:05] [02TSPortal] 07dependabot[bot] created branch 03dependabot/composer/laravel/framework-8.83.28 - 13https://github.com/miraheze/TSPortal [00:03:06] [02TSPortal] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/laravel/framework-8.83.28 [+0/-0/±2] 13https://github.com/miraheze/TSPortal/commit/2a092897554b [00:03:08] [02TSPortal] 07dependabot[bot] 032a09289 - Bump laravel/framework from 8.83.27 to 8.83.28 [00:03:09] [02TSPortal] 07dependabot[bot] opened pull request 03#24: Bump laravel/framework from 8.83.27 to 8.83.28 - 13https://github.com/miraheze/TSPortal/pull/24 [00:03:12] [02TSPortal] 07dependabot[bot] labeled pull request 03#24: Bump laravel/framework from 8.83.27 to 8.83.28 - 13https://github.com/miraheze/TSPortal/pull/24 [00:03:15] [02TSPortal] 07coderabbitai[bot] commented on pull request 03#24: Bump laravel/framework from 8.83.27 to 8.83.28 - 13https://github.com/miraheze/TSPortal/pull/24#issuecomment-2471921127 [00:22:29] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#530: Add CheckWikiDatabases script - 13https://github.com/miraheze/MirahezeMagic/pull/530 [00:22:30] [02MirahezeMagic] 07Universal-Omega pushed 031 commit to 03CheckWikiDatabases [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/462a01a9b223...efd20c500dab [00:22:32] [02MirahezeMagic] 07Universal-Omega 03efd20c5 - Add more table support [00:25:57] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#530: Add CheckWikiDatabases script - 13https://github.com/miraheze/MirahezeMagic/pull/530 [00:25:58] [02MirahezeMagic] 07Universal-Omega pushed 031 commit to 03CheckWikiDatabases [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/efd20c500dab...1532bb63d29d [00:26:00] [02MirahezeMagic] 07Universal-Omega 031532bb6 - order [00:35:44] miraheze/MirahezeMagic - Universal-Omega the build passed. [00:56:39] [02MirahezeMagic] 07Universal-Omega pushed 031 commit to 03CheckWikiDatabases [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/1532bb63d29d...d0ec6088d7e5 [00:56:40] [02MirahezeMagic] 07Universal-Omega 03d0ec608 - Support cargo databases [00:56:43] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#530: Add CheckWikiDatabases script - 13https://github.com/miraheze/MirahezeMagic/pull/530 [01:07:20] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:08:26] miraheze/MirahezeMagic - Universal-Omega the build passed. [01:17:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:37:12] [02MirahezeMagic] 07Universal-Omega pushed 031 commit to 03CheckWikiDatabases [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/d0ec6088d7e5...abebb954eee0 [01:37:14] [02MirahezeMagic] 07Universal-Omega 03abebb95 - Sorting [01:37:16] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#530: Add CheckWikiDatabases script - 13https://github.com/miraheze/MirahezeMagic/pull/530 [01:47:32] miraheze/MirahezeMagic - Universal-Omega the build passed. [01:57:38] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/deleteBatch.php --wiki=metawiki /home/macfan/delete.txt -u=MacFan4000 -r=Removing translation unit pages for pages that shouldn't be translated (START) [01:57:39] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/deleteBatch.php --wiki=metawiki /home/macfan/delete.txt -u=MacFan4000 -r=Removing translation unit pages for pages that shouldn't be translated (END - exit=512) [01:57:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:57:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:58:33] RECOVERY - mwtask181 Disk Space on mwtask181 is OK: DISK OK - free space: / 39401MiB (17% inode=93%); [02:30:05] [02MirahezeMagic] 07Universal-Omega pushed 031 commit to 03CheckWikiDatabases [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/abebb954eee0...589d6fdf92d5 [02:30:07] [02MirahezeMagic] 07Universal-Omega 03589d6fd - Fix dropping [02:30:08] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#530: Add CheckWikiDatabases script - 13https://github.com/miraheze/MirahezeMagic/pull/530 [02:35:47] [02MirahezeMagic] 07Universal-Omega pushed 031 commit to 03CheckWikiDatabases [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/589d6fdf92d5...84ea0b7cc659 [02:35:49] [02MirahezeMagic] 07Universal-Omega 0384ea0b7 - Fix order [02:35:50] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#530: Add CheckWikiDatabases script - 13https://github.com/miraheze/MirahezeMagic/pull/530 [02:36:44] !log drop all *wikibeta databases from production database servers [02:36:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:44:49] oh dear [02:45:38] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#530: Add CheckWikiDatabases script - 13https://github.com/miraheze/MirahezeMagic/pull/530 [02:45:40] [02MirahezeMagic] 07Universal-Omega pushed 031 commit to 03CheckWikiDatabases [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/84ea0b7cc659...eef0515c9991 [02:45:42] [02MirahezeMagic] 07Universal-Omega 03eef0515 - Rename option to tables [02:46:19] miraheze/MirahezeMagic - Universal-Omega the build passed. [02:46:48] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#530: Add CheckWikiDatabases script - 13https://github.com/miraheze/MirahezeMagic/pull/530 [02:46:50] [02MirahezeMagic] 07Universal-Omega pushed 031 commit to 03CheckWikiDatabases [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/eef0515c9991...640c320bc6ec [02:46:52] [02MirahezeMagic] 07Universal-Omega 03640c320 - Update descriptions [02:54:05] [02MirahezeMagic] 07Universal-Omega closed pull request 03#530: Add CheckWikiDatabases script - 13https://github.com/miraheze/MirahezeMagic/pull/530 [02:54:07] [02MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/MirahezeMagic/compare/58561cd25d63...1f9c746a92bc [02:54:09] [02MirahezeMagic] 07Universal-Omega 031f9c746 - Add CheckWikiDatabases script (#530) [02:54:12] [02MirahezeMagic] 07Universal-Omega deleted branch 03CheckWikiDatabases - 13https://github.com/miraheze/MirahezeMagic [02:54:13] [02MirahezeMagic] 07Universal-Omega deleted branch 03CheckWikiDatabases [02:54:51] !log run CheckWikiDatabases, dropping all databases that had no corresponding cw_wikis entry [02:54:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:56:37] !log run CheckWikiDatabases with --tables, deleting entries in all global tables pointing to databases that do not exist [02:56:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:57:24] miraheze/MirahezeMagic - Universal-Omega the build passed. [02:58:57] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.43, 17.15, 9.94 [03:02:55] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#531: Add CheckSwiftContainers script - 13https://github.com/miraheze/MirahezeMagic/pull/531 [03:02:57] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.81, 21.20, 13.25 [03:02:57] [02MirahezeMagic] 07Universal-Omega pushed 031 commit to 03CheckSwiftContainers [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/bd55069f420a...867327391b29 [03:02:58] [02MirahezeMagic] 07Universal-Omega 038673273 - No \n for fatalError [03:04:10] miraheze/MirahezeMagic - Universal-Omega the build passed. [03:06:57] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.17, 22.71, 15.75 [03:08:57] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.88, 24.32, 17.18 [03:12:51] miraheze/MirahezeMagic - Universal-Omega the build passed. [03:14:08] [02MirahezeMagic] 07Universal-Omega closed pull request 03#531: Add CheckSwiftContainers script - 13https://github.com/miraheze/MirahezeMagic/pull/531 [03:14:11] [02MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/MirahezeMagic/compare/1f9c746a92bc...c53c72558f3b [03:14:14] [02MirahezeMagic] 07Universal-Omega 03c53c725 - Add CheckSwiftContainers script (#531) [03:14:17] [02MirahezeMagic] 07Universal-Omega deleted branch 03CheckSwiftContainers [03:14:20] [02MirahezeMagic] 07Universal-Omega deleted branch 03CheckSwiftContainers - 13https://github.com/miraheze/MirahezeMagic [03:20:57] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 6.55, 19.02, 19.50 [03:24:43] miraheze/MirahezeMagic - Universal-Omega the build passed. [03:29:36] [02puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a0af749880f6...a907c1f569e3 [03:29:39] [02puppet] 07Universal-Omega 03a907c1f - Remove IgnoreUsers [03:29:50] @pixldev ^ [03:46:11] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 139.02 ms [03:48:13] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 139.03 ms [04:00:05] uhh [04:00:10] we got k-lined [04:00:39] might need to bug libera staff about that one [04:01:09] looks like the relay kept quitting and rejoining [04:54:15] [02puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a907c1f569e3...30c84141952e [04:54:18] [02puppet] 07Universal-Omega 0330c8414 - Fix [05:12:12] test [05:24:32] test [05:24:41] is it back? [05:24:50] yay [05:24:55] MirahezeRelay: u good? [05:25:02] yippee [05:25:08] !log DROP DATABASE mhglobal; CREATE DATABASE mhglobal; USE mhglobal; SOURCE /home/universalomega/backup/2024-11-10.sql [05:25:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:25:59] Not a log you'd see very often... [05:26:48] heh [05:26:56] I am now restoring the last two days using binlogs [05:31:36] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/invalidateUserSessions.php --wiki=svanweywiki --user=Rskhm2 (END - exit=0) [05:31:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:35:33] !log [universalomega@db171:~]$ sudo -i mysql mhglobal < binlogs_restore.sql --force [05:35:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:36:18] MacFan4000: were they having login problems? [05:36:32] Is it related to this incident? [05:36:46] Yes and I think so [05:37:46] I don't know if the binlogs restored 100% but I did confirm data from yesterday (which is after the backups) exists in gnf_files using a timestamp query so I know some worked. I hope most is restored. I don't know how to confirm for certain though... [06:07:46] !log reboot db171 [06:07:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:12:56] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/populateArcgiveRevId.php --wiki=merawiki (END - exit=256) [06:13:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:13:07] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/populateArcgiveRevId.php --wiki=metawiki (END - exit=256) [06:13:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:13:39] what is that script? [06:13:46] I have never heard of it lol [06:13:55] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/populateArchiveRevId.php --wiki=metawiki (END - exit=256) [06:13:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:14:10] you populate [[mh:rainverse:arc]], and she'll give you the rev ids [06:14:23] oh lol [06:14:54] Does that one even exist? [06:15:29] wdym? [06:22:38] CosmicAlpha: I had a deletebatch running on meta [06:22:54] This left many pages in a partially deleted state [06:23:23] They exist on wiki, but can’t be redeleted because they are already in the archive table [06:23:25] Can't you just restart it? [06:23:29] oh [06:23:34] uh [06:23:40] well [06:23:52] Way too many pages to fix by hand though [06:24:00] I didn't have much of a choice [06:24:42] Wikis were going down entirely once CreateWiki cache was expiring so I had to do it. [06:25:00] That seems like an upstream issue also... [06:25:39] Why would a failing script that just looses database connection (which can happen anyway) leave things in a partial mess [06:26:18] I will see what I can do about it later [06:26:27] Not tonight [06:26:41] lol, i once fixed a minor bug in mw maint script [06:40:19] [02puppet] 07Universal-Omega commented on pull request 03#4075: T12829: Use a different user for pywikibot - 13https://github.com/miraheze/puppet/pull/4075#issuecomment-2472583573 [06:46:50] fixed [06:46:57] !log (metawiki) DELETE FROM archive WHERE ar_rev_id=441793; [06:47:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:47:25] turns out only one page was affected, not sure why [07:23:09] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/invalidateUserSessions.php --wiki=svanweywiki --user=Rskhm2 (END - exit=0) [07:23:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:23:47] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/invalidateUserSessions.php --wiki=loginwiki --user=Rskhm2 (END - exit=0) [07:23:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:28:05] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/invalidateUserSessions.php --wiki=loginwiki --user=LUNARCRAWLER (END - exit=0) [07:28:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:22:20] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [14:25:03] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/invalidateUserSessions.php --wiki=vyletponywiki --user=PineYard (END - exit=0) [14:25:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:25:14] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/invalidateUserSessions.php --wiki=loginwiki --user=PineYard (END - exit=0) [14:25:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:36:04] Morning [14:45:42] PROBLEM - mem161 APT on mem161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:46:31] Hey pix [14:52:19] PROBLEM - mem161 Current Load on mem161 is CRITICAL: LOAD CRITICAL - total load average: 6.58, 3.54, 1.96 [14:54:24] PROBLEM - mem161 Puppet on mem161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:08:21] RECOVERY - mem161 Puppet on mem161 is OK: OK: Puppet is currently enabled, last run 37 minutes ago with 0 failures [15:08:25] RECOVERY - mem161 APT on mem161 is OK: APT OK: 50 packages available for upgrade (0 critical updates). [15:09:02] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 138.99 ms [15:11:04] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 139.01 ms [15:14:19] PROBLEM - mem161 Current Load on mem161 is WARNING: LOAD WARNING - total load average: 0.10, 2.35, 3.87 [15:18:19] RECOVERY - mem161 Current Load on mem161 is OK: LOAD OK - total load average: 0.05, 1.10, 3.00 [15:18:33] PROBLEM - mwtask181 Disk Space on mwtask181 is WARNING: DISK WARNING - free space: / 22617MiB (10% inode=93%); [15:21:27] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/invalidateUserSessions.php --wiki=loginwiki --user= idrinklisterine (END - exit=65280) [15:21:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:21:56] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/invalidateUserSessions.php --wiki=loginwiki --user=idrinklisterine (END - exit=0) [15:22:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:30:48] [02puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/30c84141952e...012f56330d17 [15:30:51] [02puppet] 07paladox 03012f563 - db: fix log used in monitoring [15:31:42] RECOVERY - db161 Backups SQL on db161 is OK: FILE_AGE OK: /var/log/db-backups/db-backups/db-backups.log is 712397 seconds old and 911544 bytes [15:32:07] RECOVERY - db181 Backups SQL on db181 is OK: FILE_AGE OK: /var/log/db-backups/db-backups/db-backups.log is 716289 seconds old and 929259 bytes [15:32:19] RECOVERY - db172 Backups SQL on db172 is OK: FILE_AGE OK: /var/log/db-backups/db-backups/db-backups.log is 649935 seconds old and 355 bytes [15:32:19] PROBLEM - mem161 Current Load on mem161 is CRITICAL: LOAD CRITICAL - total load average: 7.99, 4.66, 3.16 [15:32:38] RECOVERY - db171 Backups SQL on db171 is OK: FILE_AGE OK: /var/log/db-backups/db-backups/db-backups.log is 818395 seconds old and 820789 bytes [15:33:11] RECOVERY - db151 Backups SQL on db151 is OK: FILE_AGE OK: /var/log/db-backups/db-backups/db-backups.log is 112070 seconds old and 906310 bytes [15:33:16] PROBLEM - mem161 APT on mem161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:33:21] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/517c71e0460f...9fb4008762ce [15:33:23] [02ssl] 07WikiTideSSLBot 039fb4008 - Bot: Update SSL cert for geeu.org [15:34:52] PROBLEM - mem151 Current Load on mem151 is CRITICAL: LOAD CRITICAL - total load average: 6.95, 3.30, 1.40 [15:34:55] PROBLEM - mem161 Puppet on mem161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:35:06] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9fb4008762ce...7fcc63093c2e [15:35:07] PROBLEM - mem151 APT on mem151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:35:08] [02ssl] 07WikiTideSSLBot 037fcc630 - Bot: Update SSL cert for geeu.org [15:38:06] PROBLEM - comprehensibleinputwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'comprehensibleinputwiki.org' expires in 14 day(s) (Thu 28 Nov 2024 03:15:37 PM GMT +0000). [15:38:26] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7fcc63093c2e...1ae2993ef7cb [15:38:29] [02ssl] 07WikiTideSSLBot 031ae2993 - Bot: Update SSL cert for comprehensibleinputwiki.org [15:38:38] PROBLEM - mem151 Puppet on mem151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:39:42] PROBLEM - mem161 PowerDNS Recursor on mem161 is CRITICAL: CRITICAL - Plugin timed out while executing system call [15:40:19] RECOVERY - mem161 Current Load on mem161 is OK: LOAD OK - total load average: 0.88, 0.18, 0.06 [15:40:36] RECOVERY - mem151 Puppet on mem151 is OK: OK: Puppet is currently enabled, last run 37 minutes ago with 0 failures [15:40:51] RECOVERY - mem151 Current Load on mem151 is OK: LOAD OK - total load average: 0.74, 0.23, 0.08 [15:41:11] RECOVERY - mem161 APT on mem161 is OK: APT OK: 50 packages available for upgrade (0 critical updates). [15:41:37] RECOVERY - mem161 PowerDNS Recursor on mem161 is OK: DNS OK: 0.059 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [15:41:44] RECOVERY - mem161 Puppet on mem161 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [15:42:17] RECOVERY - mem151 APT on mem151 is OK: APT OK: 50 packages available for upgrade (0 critical updates). [15:44:50] [02puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/2a96f10d9715 [15:44:52] [02puppet] 07paladox 032a96f10 - mem161/mem151: lower memcache size [15:44:54] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [15:44:57] [02puppet] 07paladox opened pull request 03#4083: mem161/mem151: lower memcache size - 13https://github.com/miraheze/puppet/pull/4083 [15:45:00] [02puppet] 07coderabbitai[bot] commented on pull request 03#4083: mem161/mem151: lower memcache size - 13https://github.com/miraheze/puppet/pull/4083#issuecomment-2473994381 [15:46:24] [02puppet] 07paladox synchronize pull request 03#4083: mem161/mem151: lower memcache size - 13https://github.com/miraheze/puppet/pull/4083 [15:46:27] [02puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2a96f10d9715...df33f071b995 [15:46:29] [02puppet] 07paladox 03df33f07 - Update mem161.yaml [15:46:38] [02puppet] 07paladox synchronize pull request 03#4083: mem161/mem151: lower memcache size - 13https://github.com/miraheze/puppet/pull/4083 [15:46:40] [02puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/df33f071b995...28a0e44c1457 [15:46:42] [02puppet] 07paladox 0328a0e44 - Update mem151.yaml [15:47:00] [02puppet] 07paladox edited pull request 03#4083: mem161/mem151: lower memcache size - 13https://github.com/miraheze/puppet/pull/4083 [15:48:55] [02puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/28a0e44c1457...8770ebb1213c [15:48:56] [02puppet] 07paladox 038770ebb - Update mem151.yaml [15:48:58] [02puppet] 07paladox synchronize pull request 03#4083: mem161/mem151: lower memcache size - 13https://github.com/miraheze/puppet/pull/4083 [15:49:05] [02puppet] 07paladox synchronize pull request 03#4083: mem161/mem151: lower memcache size - 13https://github.com/miraheze/puppet/pull/4083 [15:49:08] [02puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8770ebb1213c...7021118bae2b [15:49:09] [02puppet] 07paladox 037021118 - Update mem161.yaml [15:49:46] [02puppet] 07paladox synchronize pull request 03#4083: mem161/mem151: lower memcache size - 13https://github.com/miraheze/puppet/pull/4083 [15:49:49] [02puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7021118bae2b...506289456882 [15:49:50] [02puppet] 07paladox 035062894 - Update mem151.yaml [15:49:56] [02puppet] 07paladox synchronize pull request 03#4083: mem161/mem151: lower memcache size - 13https://github.com/miraheze/puppet/pull/4083 [15:49:58] [02puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/506289456882...e6ddc7955ac5 [15:49:59] [02puppet] 07paladox 03e6ddc79 - Update mem161.yaml [15:50:18] [02puppet] 07paladox edited pull request 03#4083: mem161/mem151: lower memcache size - 13https://github.com/miraheze/puppet/pull/4083 [15:50:23] [02puppet] 07paladox closed pull request 03#4083: mem161/mem151: lower memcache size - 13https://github.com/miraheze/puppet/pull/4083 [15:50:24] [02puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/012f56330d17...ce37875078cf [15:50:26] [02puppet] 07paladox 03ce37875 - mem161/mem151: lower memcache size (#4083) [15:50:29] [02puppet] 07paladox deleted branch 03paladox-patch-1 [15:50:30] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [15:53:35] [02puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/7c4821b5b283 [15:53:36] [02puppet] 07paladox 037c4821b - mem151/mem161: lower memcached size [15:53:37] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [15:53:41] [02puppet] 07paladox opened pull request 03#4084: mem151/mem161: lower memcached size - 13https://github.com/miraheze/puppet/pull/4084 [15:53:48] [02puppet] 07coderabbitai[bot] commented on pull request 03#4084: mem151/mem161: lower memcached size - 13https://github.com/miraheze/puppet/pull/4084#issuecomment-2474017221 [15:54:02] [02puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7c4821b5b283...06ba943e780a [15:54:04] [02puppet] 07paladox 0306ba943 - Update mem161.yaml [15:54:05] [02puppet] 07paladox synchronize pull request 03#4084: mem151/mem161: lower memcached size - 13https://github.com/miraheze/puppet/pull/4084 [15:54:10] [02puppet] 07paladox closed pull request 03#4084: mem151/mem161: lower memcached size - 13https://github.com/miraheze/puppet/pull/4084 [15:54:11] [02puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/ce37875078cf...c1290073eeee [15:54:12] [02puppet] 07paladox 03c129007 - mem151/mem161: lower memcached size (#4084) [15:54:13] [02puppet] 07paladox deleted branch 03paladox-patch-1 [15:54:15] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [16:06:42] RECOVERY - comprehensibleinputwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'comprehensibleinputwiki.org' will expire on Tue 11 Feb 2025 02:39:50 PM GMT +0000. [16:09:06] [02puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/31fb90db4bff [16:09:09] [02puppet] 07paladox 0331fb90d - mem151/mem161: increase memcache to 67gb [16:09:10] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [16:09:11] [02puppet] 07paladox opened pull request 03#4085: mem151/mem161: increase memcache to 67gb - 13https://github.com/miraheze/puppet/pull/4085 [16:09:17] [02puppet] 07coderabbitai[bot] commented on pull request 03#4085: mem151/mem161: increase memcache to 67gb - 13https://github.com/miraheze/puppet/pull/4085#issuecomment-2474058132 [16:09:47] [02puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/31fb90db4bff...9a3ddcf49b30 [16:09:48] [02puppet] 07paladox 039a3ddcf - Update mem161.yaml [16:09:51] [02puppet] 07paladox synchronize pull request 03#4085: mem151/mem161: increase memcache to 67gb - 13https://github.com/miraheze/puppet/pull/4085 [16:10:03] [02puppet] 07paladox closed pull request 03#4085: mem151/mem161: increase memcache to 67gb - 13https://github.com/miraheze/puppet/pull/4085 [16:10:06] [02puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/c1290073eeee...9188ca6d9487 [16:10:08] [02puppet] 07paladox 039188ca6 - mem151/mem161: increase memcache to 67gb (#4085) [16:10:11] [02puppet] 07paladox deleted branch 03paladox-patch-1 [16:10:12] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [16:13:06] RECOVERY - geeu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'geeu.org' will expire on Tue 11 Feb 2025 02:36:30 PM GMT +0000. [16:22:20] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [16:27:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [16:36:33] RECOVERY - mwtask181 Disk Space on mwtask181 is OK: DISK OK - free space: / 39078MiB (17% inode=93%); [16:56:35] [02puppet] 07redbluegreenhat pushed 0320 commits to 03T12829 [+2/-0/±30] 13https://github.com/miraheze/puppet/compare/5ddfdf7c52d5...c45c68182d70 [16:56:38] [02puppet] 07redbluegreenhat 03c45c681 - Solve merge conflict [16:56:39] [02puppet] 07redbluegreenhat synchronize pull request 03#4075: T12829: Use a different user for pywikibot - 13https://github.com/miraheze/puppet/pull/4075 [16:56:41] [02puppet] 07coderabbitai[bot] edited a comment on pull request 03#4075: T12829: Use a different user for pywikibot - 13https://github.com/miraheze/puppet/pull/4075#issuecomment-2470268825 [16:57:26] [02puppet] 07github-actions[bot] pushed 031 commit to 03T12829 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c45c68182d70...36392750e7f7 [16:57:28] [02puppet] 07github-actions 033639275 - CI: lint puppet code to standards [16:57:29] [02puppet] 07github-actions[bot] synchronize pull request 03#4075: T12829: Use a different user for pywikibot - 13https://github.com/miraheze/puppet/pull/4075 [16:58:19] [02puppet] 07redbluegreenhat closed pull request 03#4075: T12829: Use a different user for pywikibot - 13https://github.com/miraheze/puppet/pull/4075 [16:58:22] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/9188ca6d9487...6cd9d6825d68 [16:58:24] [02puppet] 07redbluegreenhat 036cd9d68 - T12829: Use a different user for pywikibot (#4075) [16:58:26] [02puppet] 07redbluegreenhat deleted branch 03T12829 [16:58:28] [02puppet] 07redbluegreenhat deleted branch 03T12829 - 13https://github.com/miraheze/puppet [16:59:25] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6cd9d6825d68...4dfb1ce4ed7c [16:59:27] [02puppet] 07redbluegreenhat 034dfb1ce - Set group to pywikibot [17:08:18] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4dfb1ce4ed7c...6129ba02defc [17:08:21] [02puppet] 07redbluegreenhat 036129ba0 - PyWikiBot -> Pywikibot-stable [17:09:17] PROBLEM - bots171 Puppet on bots171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:15:17] RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [17:15:58] don't you sometimes wish we would do away with all this shit and just run physical servers-only, no puppet or configuration management of any kind [17:16:33] #mainframesrule [17:17:57] like think about it, there's no need for puppet if you run the entirety of miraheze in a single physical server [17:18:34] #BlueMoonforDTech2025 [17:20:48] !log [alex@bots171] cleaned irc's crontab (everything was moved to pywikibot's) [17:20:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:25:20] time for a new userpage for BeeBot [17:38:38] Monoheze [17:52:00] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [18:24:31] [02puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6129ba02defc...e716a0b65b21 [18:24:32] [02puppet] 07Universal-Omega 03e716a0b - Update repo [18:26:43] [02puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e716a0b65b21...093b9fc52bc8 [18:26:45] [02puppet] 07Universal-Omega 03093b9fc - Update repo [18:34:47] [02CVTBot] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/miraheze/CVTBot/compare/8667650099c3...44b93877e87a [18:34:50] [02CVTBot] 07Universal-Omega 0344b9387 - Update README [18:39:25] [02IRC-Discord-Relay] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/IRC-Discord-Relay/compare/da8b64f6832c...c350ed0c52a1 [18:39:26] [02IRC-Discord-Relay] 07Universal-Omega 03c350ed0 - Update Discord.Net to 3.16.0 [19:49:38] [02MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/c53c72558f3b...195b30cf0c94 [19:49:39] [02MirahezeMagic] 07Universal-Omega 03195b30c - Use array_key_exists [19:51:57] miraheze/MirahezeMagic - Universal-Omega the build has errored. [20:12:33] PROBLEM - mwtask181 Disk Space on mwtask181 is CRITICAL: DISK CRITICAL - free space: / 7681MiB (3% inode=93%); [20:15:55] [02MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/195b30cf0c94...c9b2c7b2401b [20:15:56] [02MirahezeMagic] 07Universal-Omega 03c9b2c7b - Add type hints [20:17:51] miraheze/MirahezeMagic - Universal-Omega the build has errored. [21:19:38] !log (metawiki) DELETE FROM archive WHERE ar_title LIKE "Meta:Autopatrolled_users/__/ur"; [21:19:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:20:33] RECOVERY - mwtask181 Disk Space on mwtask181 is OK: DISK OK - free space: / 84000MiB (37% inode=95%); [21:20:57] !log (metawiki) DELETE FROM archive WHERE ar_title="Meta:Autopatrolled_users/9/ur"; [21:21:00] !log (metawiki) DELETE FROM archive WHERE ar_title="Meta:Autopatrolled_users/8/ur"; [21:21:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:21:04] !log (metawiki) DELETE FROM archive WHERE ar_title="Meta:Autopatrolled_users/7/ur"; [21:21:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:21:08] !log (metawiki) DELETE FROM archive WHERE ar_title="Meta:Autopatrolled_users/6/ur"; [21:21:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:21:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:27:12] !log [www-data@mwtask181] Began automatic backing up [21:27:14] !log [www-data@mwtask181] Began automatic backing up [21:27:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:27:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:36:37] !log shell.php: $dbw = wfGetDB( DB_PRIMARY, [], $wgCreateWikiDatabase ); $hookRunner = MediaWiki\MediaWikiServices::getInstance()->get( 'CreateWikiHookRunner' ); $hookRunner->onCreateWikiRename( $dbw, 'basiliagovwiki', 'alderlandwiki' ); [21:36:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:38:38] [1/2] @agentisai this is one rename you did when renames were broken so swift wasn't renamed, did you happen to do anymore? [21:38:39] [2/2] Also @reception123 was there anymore other than the one I fixed already yesterday that you did during that period also? [21:50:18] [02MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/c9b2c7b2401b...c3ffd9359b92 [21:50:19] [02MirahezeMagic] 07Universal-Omega 03c3ffd93 - Fix uniqueWikiCounts [21:52:21] miraheze/MirahezeMagic - Universal-Omega the build has errored. [21:53:59] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/MirahezeMagic/maintenance/checkSwiftContainers.php --wiki=metawiki --container=createwiki-persistent-model --delete (END - exit=0) [21:54:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:14:19] oh dear [22:14:32] I wasn't aware that Swift was broken [22:14:36] afaik no [22:15:07] Yeah it was the period in September where renames broke so swift wasn't renamed on ones done then. [22:19:17] [02puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/093b9fc52bc8...1998223522b8 [22:19:18] [02puppet] 07AgentIsai 031998223 - foreachwikiindblist: Support specifiying where to start or end [22:22:53] @agentisai wont that break run.php being used with foreachwikiindblist? [22:24:43] Hmm maybe not actually I think I was getting my logic confused [22:35:12] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/MirahezeMagic/maintenance/checkSwiftContainers.php --wiki=metawiki --container=timeline-render --delete (END - exit=0) [22:35:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:45:31] [02mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e3c2766d58f7...dc4662b4fcc6 [22:45:33] [02mw-config] 07paladox 03dc4662b - Fix comment [22:46:34] miraheze/mw-config - paladox the build passed. [22:53:10] !log [@test151] starting deploy of {'config': True} to test151 [22:53:11] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [22:53:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:53:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:02:28] !log [@mwtask181] starting deploy of {'config': True} to all [23:02:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:02:52] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 23s [23:02:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:04:40] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/MirahezeMagic/maintenance/checkSwiftContainers.php --wiki=metawiki --container=local-transcoded --delete (END - exit=0) [23:04:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:13:38] !log [www-data@mwtask181] Began automatic backing up [23:13:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:17:12] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/MirahezeMagic/maintenance/checkSwiftContainers.php --wiki=metawiki --container=dumps-backup --delete (END - exit=0) [23:17:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:28:42] [02python-functions] 07dependabot[bot] created branch 03dependabot/pip/dot-github/setuptools-75.5.0 - 13https://github.com/miraheze/python-functions [23:28:43] [02python-functions] 07dependabot[bot] pushed 031 commit to 03dependabot/pip/dot-github/setuptools-75.5.0 [+0/-0/±1] 13https://github.com/miraheze/python-functions/commit/a7e8f75a7388 [23:28:46] [02python-functions] 07dependabot[bot] 03a7e8f75 - Bump setuptools from 75.3.0 to 75.5.0 in /.github [23:28:47] [02python-functions] 07dependabot[bot] opened pull request 03#97: Bump setuptools from 75.3.0 to 75.5.0 in /.github - 13https://github.com/miraheze/python-functions/pull/97 [23:28:49] [02python-functions] 07dependabot[bot] labeled pull request 03#97: Bump setuptools from 75.3.0 to 75.5.0 in /.github - 13https://github.com/miraheze/python-functions/pull/97 [23:28:52] [02python-functions] 07dependabot[bot] labeled pull request 03#97: Bump setuptools from 75.3.0 to 75.5.0 in /.github - 13https://github.com/miraheze/python-functions/pull/97 [23:28:55] [02python-functions] 07dependabot[bot] closed pull request 03#96: Bump setuptools from 75.3.0 to 75.4.0 in /.github - 13https://github.com/miraheze/python-functions/pull/96 [23:28:57] [02python-functions] 07dependabot[bot] commented on pull request 03#96: Bump setuptools from 75.3.0 to 75.4.0 in /.github - 13https://github.com/miraheze/python-functions/pull/96#issuecomment-2475031923 [23:28:59] [02python-functions] 07dependabot[bot] deleted branch 03dependabot/pip/dot-github/setuptools-75.4.0 - 13https://github.com/miraheze/python-functions [23:29:01] [02python-functions] 07dependabot[bot] deleted branch 03dependabot/pip/dot-github/setuptools-75.4.0 [23:29:02] [02python-functions] 07coderabbitai[bot] commented on pull request 03#97: Bump setuptools from 75.3.0 to 75.5.0 in /.github - 13https://github.com/miraheze/python-functions/pull/97#issuecomment-2475032015 [23:30:22] miraheze/python-functions - dependabot[bot] the build has errored. [23:31:29] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [23:31:32] [02puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/28eefee722d9 [23:31:34] [02puppet] 07paladox 0328eefee - prometheus: increase retention time to 120d and size to 100GB [23:31:37] [02puppet] 07paladox opened pull request 03#4086: prometheus: increase retention time to 120d and size to 100GB - 13https://github.com/miraheze/puppet/pull/4086 [23:31:40] [02puppet] 07coderabbitai[bot] commented on pull request 03#4086: prometheus: increase retention time to 120d and size to 100GB - 13https://github.com/miraheze/puppet/pull/4086#issuecomment-2475035155 [23:35:13] [02puppet] 07paladox closed pull request 03#4086: prometheus: increase retention time to 120d and size to 100GB - 13https://github.com/miraheze/puppet/pull/4086 [23:35:16] [02puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1998223522b8...924802ec3457 [23:35:18] [02puppet] 07paladox 03924802e - prometheus: increase retention time to 120d and size to 100GB (#4086) [23:35:20] [02puppet] 07paladox deleted branch 03paladox-patch-1 [23:35:21] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [23:35:50]