[00:08:55] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 2 system event log (SEL) entries present] [01:26:31] [02RequestSSL] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.17.1 [+0/-0/±2] 13https://github.com/miraheze/RequestSSL/commit/daf1c369eb26 [01:26:33] [02RequestSSL] 07dependabot[bot] 03daf1c36 - Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 [01:26:36] [02RequestSSL] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.17.1 - 13https://github.com/miraheze/RequestSSL [01:26:39] [02RequestSSL] 07dependabot[bot] labeled pull request 03#49: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 - 13https://github.com/miraheze/RequestSSL/pull/49 [01:26:42] [02RequestSSL] 07dependabot[bot] labeled pull request 03#49: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 - 13https://github.com/miraheze/RequestSSL/pull/49 [01:26:43] [02RequestSSL] 07dependabot[bot] opened pull request 03#49: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 - 13https://github.com/miraheze/RequestSSL/pull/49 [01:26:45] [02RequestSSL] 07dependabot[bot] closed pull request 03#48: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.0 - 13https://github.com/miraheze/RequestSSL/pull/48 [01:26:48] [02RequestSSL] 07dependabot[bot] commented on pull request 03#48: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.0 - 13https://github.com/miraheze/RequestSSL/pull/48#issuecomment-2151236463 [01:26:49] [02RequestSSL] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.17.0 [01:26:51] [02RequestSSL] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.17.0 - 13https://github.com/miraheze/RequestSSL [01:27:17] [02RequestSSL] 07coderabbitai[bot] commented on pull request 03#49: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 - 13https://github.com/miraheze/RequestSSL/pull/49#issuecomment-2151238337 [01:35:13] miraheze/RequestSSL - dependabot[bot] the build passed. [02:32:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:04:41] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [03:05:41] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [03:06:51] PROBLEM - www.streamerlore.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.808B0D4A5F140000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:../ssl/record/rec_layer_s3.c:303 [03:22:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:34:06] RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Tue 27 Aug 2024 03:54:13 PM GMT +0000. [04:34:51] RECOVERY - www.streamerlore.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.streamerlore.com' will expire on Tue 03 Sep 2024 12:50:20 PM GMT +0000. [05:01:35] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [05:56:08] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [06:24:45] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [06:57:28] a daemon that asks other servers what the current time is [06:58:29] I was trying to fix some of the servers having the incorrect time and ended up discovering we had a bit of a mess when it came to NTP daemons [07:00:27] tho even my current solution still needs some extra work, the ntp package in Debian is not the classic ntpd that's been used for decades, but rather ntpsec, which in debian doesn't use /etc/ntp.conf for config... [07:00:45] so we don't actually have centralized configurations just yet [07:02:09] the correct solution would be to add support for this in the ntp puppetlabs module, but I want to try something first to see if I can avoid doing that [08:03:42] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:32:16] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [09:01:38] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [09:31:38] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [11:04:23] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:28:42] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 37.64, 16.34, 6.50 [11:28:49] PROBLEM - db161 PowerDNS Recursor on db161 is CRITICAL: CRITICAL - Plugin timed out while executing system call [11:32:49] PROBLEM - db161 Puppet on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:32:56] RECOVERY - db161 PowerDNS Recursor on db161 is OK: DNS OK: 0.231 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [11:32:56] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [11:33:12] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [11:33:34] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [11:33:41] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw172 [11:33:42] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw181 [11:34:50] RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:35:11] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3763 bytes in 0.065 second response time [11:35:34] RECOVERY - cp26 HTTPS on cp26 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3760 bytes in 0.882 second response time [11:35:41] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [11:35:42] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [11:48:45] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 0.24, 3.56, 11.71 [11:52:43] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 0.14, 1.74, 9.13 [12:13:32] [02SpriteSheet] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/SpriteSheet/compare/60ee82d9a687...1e73245fde5c [12:13:33] [02SpriteSheet] 07translatewiki 031e73245 - Localisation updates from https://translatewiki.net. [12:13:36] [02MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/MirahezeMagic/compare/e0b02b668c77...6f350919aa08 [12:13:37] [02MirahezeMagic] 07translatewiki 036f35091 - Localisation updates from https://translatewiki.net. [12:13:38] [02DataDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/590f1133ec23...3347ac687aab [12:13:39] [02DataDump] 07translatewiki 033347ac6 - Localisation updates from https://translatewiki.net. [12:13:42] [02WikiDiscover] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/compare/33c35533d8b8...7cc6b158d76c [12:13:43] [02WikiDiscover] 07translatewiki 037cc6b15 - Localisation updates from https://translatewiki.net. [12:17:10] miraheze/MirahezeMagic - translatewiki the build has errored. [12:18:15] miraheze/SpriteSheet - translatewiki the build has errored. [12:18:24] miraheze/WikiDiscover - translatewiki the build passed. [12:18:45] miraheze/DataDump - translatewiki the build passed. [12:23:44] !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 [12:23:45] !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:23:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:23:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:23:56] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [12:23:57] !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:08] !log [@test151] starting deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 [12:24:09] !log [@test151] finished deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:32:28] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [12:32:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:32:39] !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 11s [12:32:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:32:49] !log [@mwtask181] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [12:32:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:32:59] !log [@mwtask181] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 9s [12:33:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:33:41] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:37:44] !log [@mwtask171] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [12:37:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:37:52] !log [@mwtask171] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 8s [12:37:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:38:03] !log [@mwtask171] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [12:38:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:38:12] !log [@mwtask171] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 8s [12:38:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:02:16] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [13:04:30] [02ImportDump] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.17.1 [+0/-0/±2] 13https://github.com/miraheze/ImportDump/commit/cf78f88f0b1c [13:04:33] [02ImportDump] 07dependabot[bot] 03cf78f88 - Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 [13:04:35] [02ImportDump] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.17.1 - 13https://github.com/miraheze/ImportDump [13:04:36] [02ImportDump] 07dependabot[bot] labeled pull request 03#112: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 - 13https://github.com/miraheze/ImportDump/pull/112 [13:04:37] [02ImportDump] 07dependabot[bot] labeled pull request 03#112: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 - 13https://github.com/miraheze/ImportDump/pull/112 [13:04:38] [02ImportDump] 07dependabot[bot] opened pull request 03#112: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 - 13https://github.com/miraheze/ImportDump/pull/112 [13:04:39] [02ImportDump] 07dependabot[bot] closed pull request 03#110: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.0 - 13https://github.com/miraheze/ImportDump/pull/110 [13:04:41] [02ImportDump] 07dependabot[bot] commented on pull request 03#110: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.0 - 13https://github.com/miraheze/ImportDump/pull/110#issuecomment-2152397099 [13:04:42] [02ImportDump] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.17.0 [13:04:45] [02ImportDump] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.17.0 - 13https://github.com/miraheze/ImportDump [13:04:47] [02ImportDump] 07coderabbitai[bot] commented on pull request 03#112: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 - 13https://github.com/miraheze/ImportDump/pull/112#issuecomment-2152397358 [13:15:10] miraheze/ImportDump - dependabot[bot] the build passed. [13:31:37] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [15:27:32] PROBLEM - miraheze.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'miraheze.com' expires in 7 day(s) (Fri 14 Jun 2024 02:58:23 PM GMT +0000). [15:56:42] [02puppet] 07redbluegreenhat commented on pull request 03#3857: Mention python-functions repository on relevant scripts - 13https://github.com/miraheze/puppet/pull/3857#issuecomment-2152877459 [15:56:57] [02puppet] 07redbluegreenhat closed pull request 03#3857: Mention python-functions repository on relevant scripts - 13https://github.com/miraheze/puppet/pull/3857 [15:56:57] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/f58ed81fc3d7...6d80188b5847 [15:56:59] [02puppet] 07clarfonthey 036d80188 - Mention python-functions repository on relevant scripts (#3857) [15:58:24] miraheze/puppet - redbluegreenhat the build has errored. [16:01:31] !log disable puppet on test151 to test ntpsec [16:01:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:03:28] PROBLEM - test151 Puppet on test151 is WARNING: WARNING: Puppet is currently disabled, message: testing Puppet's generated config with ntpsec, need to shoo Puppet away for a while, last run 9 minutes ago with 0 failures [16:24:16] [02MatomoAnalytics] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.17.1 [+0/-0/±2] 13https://github.com/miraheze/MatomoAnalytics/commit/780cae0f78f1 [16:24:18] [02MatomoAnalytics] 07dependabot[bot] 03780cae0 - Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 [16:24:19] [02MatomoAnalytics] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.17.1 - 13https://github.com/miraheze/MatomoAnalytics [16:24:21] [02MatomoAnalytics] 07dependabot[bot] opened pull request 03#131: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 - 13https://github.com/miraheze/MatomoAnalytics/pull/131 [16:24:22] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#131: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 - 13https://github.com/miraheze/MatomoAnalytics/pull/131 [16:24:25] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#131: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 - 13https://github.com/miraheze/MatomoAnalytics/pull/131 [16:24:28] [02MatomoAnalytics] 07dependabot[bot] commented on pull request 03#129: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/129#issuecomment-2152932613 [16:24:29] [02MatomoAnalytics] 07dependabot[bot] closed pull request 03#129: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/129 [16:24:32] [02MatomoAnalytics] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.17.0 [16:24:33] [02MatomoAnalytics] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.17.0 - 13https://github.com/miraheze/MatomoAnalytics [16:24:36] [02MatomoAnalytics] 07coderabbitai[bot] commented on pull request 03#131: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 - 13https://github.com/miraheze/MatomoAnalytics/pull/131#issuecomment-2152932770 [16:28:49] miraheze/MatomoAnalytics - dependabot[bot] the build passed. [16:42:56] [02puppet] 07redbluegreenhat pushed 031 commit to 03ntpsec-support [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/87cce7b8b6e1 [16:42:58] [02puppet] 07redbluegreenhat 0387cce7b - hack the ntp class for ntpsec compatibility [16:43:00] [02puppet] 07redbluegreenhat created branch 03ntpsec-support - 13https://github.com/miraheze/puppet [16:43:18] [02puppet] 07redbluegreenhat opened pull request 03#3865: hack the ntp class for ntpsec compatibility - 13https://github.com/miraheze/puppet/pull/3865 [16:43:24] [02puppet] 07coderabbitai[bot] commented on pull request 03#3865: hack the ntp class for ntpsec compatibility - 13https://github.com/miraheze/puppet/pull/3865#issuecomment-2152968392 [16:43:55] [02puppet] 07github-actions[bot] pushed 031 commit to 03ntpsec-support [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/87cce7b8b6e1...0550bbb8a366 [16:43:58] [02puppet] 07github-actions 030550bbb - CI: lint puppet code to standards [16:43:59] [02puppet] 07github-actions[bot] synchronize pull request 03#3865: hack the ntp class for ntpsec compatibility - 13https://github.com/miraheze/puppet/pull/3865 [17:20:21] [02mw-config] 07redbluegreenhat commented on pull request 03#5570: T12146: Add CustomSubtitle - 13https://github.com/miraheze/mw-config/pull/5570#issuecomment-2153038575 [17:33:28] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:35:26] [02puppet] 07redbluegreenhat closed pull request 03#3865: hack the ntp class for ntpsec compatibility - 13https://github.com/miraheze/puppet/pull/3865 [17:35:29] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6d80188b5847...1d346d527cee [17:35:31] [02puppet] 07redbluegreenhat 031d346d5 - hack the ntp class for ntpsec compatibility (#3865) [17:35:32] [02puppet] 07redbluegreenhat deleted branch 03ntpsec-support [17:35:34] [02puppet] 07redbluegreenhat deleted branch 03ntpsec-support - 13https://github.com/miraheze/puppet [20:22:53] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.47, 17.41, 14.84 [20:24:53] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.85, 17.20, 15.11 [21:04:29] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:33:07] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [21:46:53] !log [void@puppet181] Upgraded packages libarchive13 on changeprop151 [21:46:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:46:59] RECOVERY - changeprop151 APT on changeprop151 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:47:05] !log [void@puppet181] Upgraded packages libarchive13, and python3-pil on bots171 [21:47:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:47:15] RECOVERY - bots171 APT on bots171 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [21:47:17] !log [void@puppet181] Upgraded packages bsdutils, libc-devtools, libc6-dev, libc-dev-bin, linux-libc-dev, libc6, libsmartcols1, util-linux-extra, util-linux, libc-bin, libruby3.1, ruby3.1, mount, libuuid1, uuid-runtime, libblkid1, libmount1, libfdisk1, fdisk, less, libc-l10n, locales, bsdextrautils, eject, libarchive13, libdav1d6, libglib2.0-0, [21:47:18] libglib2.0-data, pdns-recursor, and util-linux-locales on eventgate181 [21:47:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:47:28] !log [void@puppet181] Upgraded packages libarchive13 on db151 [21:47:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:47:39] !log [void@puppet181] Upgraded packages libarchive13 on cp37 [21:47:41] RECOVERY - cp37 APT on cp37 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:47:41] RECOVERY - db151 APT on db151 is OK: APT OK: 33 packages available for upgrade (0 critical updates). [21:47:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:47:50] !log [void@puppet181] Upgraded packages libarchive13, and libgstreamer-plugins-base1.0-0 on cloud15 [21:47:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:48:01] !log [void@puppet181] Upgraded packages libarchive13 on db161 [21:48:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:48:12] !log [void@puppet181] Upgraded packages libarchive13 on cp36 [21:48:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:48:23] !log [void@puppet181] Upgraded packages libarchive13, and libgstreamer-plugins-base1.0-0 on cloud16 [21:48:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:48:34] !log [void@puppet181] Upgraded packages libarchive13 on jobchron171 [21:48:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:48:40] RECOVERY - jobchron171 APT on jobchron171 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:48:45] !log [void@puppet181] Upgraded packages libarchive13 on graphite151 [21:48:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:48:56] !log [void@puppet181] Upgraded packages libarchive13 on graylog161 [21:49:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:49:07] !log [void@puppet181] Upgraded packages libarchive13, and libgstreamer-plugins-base1.0-0 on cloud17 [21:49:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:49:19] !log [void@puppet181] Upgraded packages libarchive13 on db171 [21:49:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:49:30] RECOVERY - db161 APT on db161 is OK: APT OK: 33 packages available for upgrade (0 critical updates). [21:49:30] !log [void@puppet181] Upgraded packages libarchive13 on bast161 [21:49:35] RECOVERY - graylog161 APT on graylog161 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [21:49:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:49:43] !log [void@puppet181] Upgraded packages libarchive13 on db181 [21:49:45] RECOVERY - cloud15 APT on cloud15 is OK: APT OK: 82 packages available for upgrade (0 critical updates). [21:49:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:49:51] RECOVERY - cloud17 APT on cloud17 is OK: APT OK: 76 packages available for upgrade (0 critical updates). [21:49:54] !log [void@puppet181] Upgraded packages libarchive13 on ldap171 [21:49:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:49:59] RECOVERY - cp36 APT on cp36 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:50:05] !log [void@puppet181] Upgraded packages libarchive13 on bast181 [21:50:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:50:10] RECOVERY - db181 APT on db181 is OK: APT OK: 33 packages available for upgrade (0 critical updates). [21:50:17] !log [void@puppet181] Upgraded packages libarchive13, and libgstreamer-plugins-base1.0-0 on cloud18 [21:50:19] RECOVERY - cloud16 APT on cloud16 is OK: APT OK: 76 packages available for upgrade (0 critical updates). [21:50:21] RECOVERY - bast181 APT on bast181 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:50:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:50:29] !log [void@puppet181] Upgraded packages libarchive13 on db182 [21:50:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:50:37] RECOVERY - graphite151 APT on graphite151 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:50:40] RECOVERY - bast161 APT on bast161 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:50:41] !log [void@puppet181] Upgraded packages libarchive13 on kafka181 [21:50:42] RECOVERY - ldap171 APT on ldap171 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:50:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:50:53] RECOVERY - cloud18 APT on cloud18 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [21:50:55] !log [void@puppet181] Upgraded packages libarchive13 on cp26 [21:50:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:51:10] !log [void@puppet181] Upgraded packages libarchive13 on cp27 [21:51:10] RECOVERY - db171 APT on db171 is OK: APT OK: 33 packages available for upgrade (0 critical updates). [21:51:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:51:18] RECOVERY - db182 APT on db182 is OK: APT OK: 33 packages available for upgrade (0 critical updates). [21:51:20] RECOVERY - kafka181 APT on kafka181 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:51:20] !log [void@puppet181] Upgraded packages libarchive13 on mem151 [21:51:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:51:31] !log [void@puppet181] Upgraded packages libarchive13 on matomo151 [21:51:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:51:43] !log [void@puppet181] Upgraded packages libarchive13 on mem161 [21:51:45] RECOVERY - mem151 APT on mem151 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:51:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:51:56] !log [void@puppet181] Upgraded packages libarchive13 on cp41 [21:51:59] RECOVERY - cp27 APT on cp27 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:52:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:52:04] RECOVERY - cp26 APT on cp26 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:52:08] !log [void@puppet181] Upgraded packages libarchive13 on os151 [21:52:09] RECOVERY - mem161 APT on mem161 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:52:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:52:16] RECOVERY - matomo151 APT on matomo151 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:52:19] !log [void@puppet181] Upgraded packages libarchive13 on prometheus151 [21:52:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:52:30] !log [void@puppet181] Upgraded packages libarchive13 on os162 [21:52:33] RECOVERY - cp41 APT on cp41 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:52:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:52:42] !log [void@puppet181] Upgraded packages libarchive13 on phorge171 [21:52:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:52:52] RECOVERY - os151 APT on os151 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [21:52:55] !log [void@puppet181] Upgraded packages libarchive13 on mon181 [21:52:57] RECOVERY - os162 APT on os162 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [21:53:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:53:05] RECOVERY - phorge171 APT on phorge171 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [21:53:08] !log [void@puppet181] Upgraded packages libarchive13, and python3-pil on mw152 [21:53:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:53:20] !log [void@puppet181] Upgraded packages libarchive13, and python3-pil on mw162 [21:53:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:53:33] !log [void@puppet181] Upgraded packages libarchive13, and python3-pil on mw172 [21:53:34] RECOVERY - mon181 APT on mon181 is OK: APT OK: 31 packages available for upgrade (0 critical updates). [21:53:34] RECOVERY - mw152 APT on mw152 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:53:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:53:46] !log [void@puppet181] Upgraded packages libarchive13, and python3-pil on mw171 [21:53:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:53:52] RECOVERY - mw162 APT on mw162 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:53:57] !log [void@puppet181] Upgraded packages libarchive13 on os161 [21:53:59] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:54:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:54:06] RECOVERY - mw172 APT on mw172 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:54:10] !log [void@puppet181] Upgraded packages libarchive13, and python3-pil on mwtask171 [21:54:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:54:22] !log [void@puppet181] Upgraded packages libarchive13 on ns1 [21:54:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:54:34] !log [void@puppet181] Upgraded packages libarchive13, and python3-pil on mw151 [21:54:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:54:47] !log [void@puppet181] Upgraded packages libarchive13, and python3-pil on mwtask181 [21:54:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:54:59] RECOVERY - mw151 APT on mw151 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:55:01] !log [void@puppet181] Upgraded packages libarchive13 on cp51 [21:55:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:55:14] !log [void@puppet181] Upgraded packages libarchive13, and python3-pil on mw161 [21:55:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:55:23] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:55:29] !log [void@puppet181] Upgraded packages libarchive13, and python3-pil on mw182 [21:55:30] RECOVERY - ns1 APT on ns1 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:55:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:55:34] RECOVERY - mw171 APT on mw171 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:55:44] !log [void@puppet181] Upgraded packages libarchive13, and python3-pil on mw181 [21:55:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:55:50] RECOVERY - os161 APT on os161 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [21:55:56] !log [void@puppet181] Upgraded packages libarchive13 on puppet181 [21:55:59] RECOVERY - puppet181 APT on puppet181 is OK: APT OK: 31 packages available for upgrade (0 critical updates). [21:56:00] RECOVERY - mw161 APT on mw161 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:56:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:56:07] !log [void@puppet181] Upgraded packages libarchive13 on rdb151 [21:56:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:56:18] !log [void@puppet181] Upgraded packages libarchive13 on reports171 [21:56:20] RECOVERY - rdb151 APT on rdb151 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:56:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:56:28] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:56:29] RECOVERY - cp51 APT on cp51 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:56:37] !log [void@puppet181] Upgraded packages libarchive13 on ns2 [21:56:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:56:44] RECOVERY - mw181 APT on mw181 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:56:45] RECOVERY - mw182 APT on mw182 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:56:50] !log [void@puppet181] Upgraded packages libarchive13, and python3-pil on swiftobject151 [21:56:52] RECOVERY - ns2 APT on ns2 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [21:56:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:57:03] !log [void@puppet181] Upgraded packages libarchive13, and python3-pil on test151 [21:57:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:57:17] !log [void@puppet181] Upgraded packages libarchive13, and python3-pil on swiftac171 [21:57:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:57:31] !log [void@puppet181] Upgraded packages libarchive13, and python3-pil on swiftobject161 [21:57:32] RECOVERY - swiftobject161 APT on swiftobject161 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:57:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:57:44] !log [void@puppet181] Upgraded packages libarchive13, and python3-pil on swiftproxy161 [21:57:47] RECOVERY - reports171 APT on reports171 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [21:57:47] RECOVERY - swiftobject151 APT on swiftobject151 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:57:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:57:58] !log [void@puppet181] Upgraded packages libarchive13, and python3-pil on swiftobject171 [21:58:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:58:09] RECOVERY - swiftobject171 APT on swiftobject171 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:58:12] !log [void@puppet181] Upgraded packages libarchive13, and python3-pil on swiftobject181 [21:58:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:58:26] !log [void@puppet181] Upgraded packages libarchive13, and python3-pil on swiftproxy171 [21:58:29] RECOVERY - swiftac171 APT on swiftac171 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:58:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:58:38] RECOVERY - test151 APT on test151 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:59:06] RECOVERY - swiftobject181 APT on swiftobject181 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [21:59:39] RECOVERY - swiftproxy161 APT on swiftproxy161 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:00:00] RECOVERY - swiftproxy171 APT on swiftproxy171 is OK: APT OK: 26 packages available for upgrade (0 critical updates).