[00:47:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:39:28] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [01:45:12] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 29.45, 20.25, 13.39 [01:53:12] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 13.50, 21.60, 17.67 [01:55:12] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 9.15, 17.61, 16.71 [02:48:10] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:00:10] RECOVERY - db181 Backups SQL on db181 is OK: FILE_AGE OK: /var/log/sql-backup.log is 9 seconds old and 0 bytes [03:00:53] RECOVERY - db161 Backups SQL on db161 is OK: FILE_AGE OK: /var/log/sql-backup.log is 52 seconds old and 0 bytes [03:07:53] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:18:08] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:39:27] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:02:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:08:54] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [05:07:48] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK [06:08:45] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [07:07:41] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [07:51:10] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [08:16:38] PROBLEM - cloud17 Puppet on cloud17 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [08:21:13] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [08:44:38] RECOVERY - cloud17 Puppet on cloud17 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [08:51:13] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [09:21:34] PROBLEM - db151 Backups SQL on db151 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864206 seconds old and 106503 bytes [11:06:30] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [11:07:03] [02puppet] 07redbluegreenhat pushed 031 commit to 03install-ntp [+86/-0/±1] 13https://github.com/miraheze/puppet/commit/92650629787d [11:07:04] [02puppet] 07redbluegreenhat 039265062 - install ntp module [11:07:07] [02puppet] 07redbluegreenhat created branch 03install-ntp - 13https://github.com/miraheze/puppet [11:07:21] [02puppet] 07redbluegreenhat opened pull request 03#3859: Install ntp module - 13https://github.com/miraheze/puppet/pull/3859 [11:07:25] [02puppet] 07coderabbitai[bot] commented on pull request 03#3859: Install ntp module - 13https://github.com/miraheze/puppet/pull/3859#issuecomment-2149533045 [11:07:57] [02puppet] 07github-actions[bot] pushed 031 commit to 03install-ntp [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/92650629787d...0f6dd6bbf141 [11:07:59] [02puppet] 07github-actions 030f6dd6b - CI: lint puppet code to standards [11:08:01] [02puppet] 07github-actions[bot] synchronize pull request 03#3859: Install ntp module - 13https://github.com/miraheze/puppet/pull/3859 [11:23:23] miraheze/puppet - clarfonthey the build has errored. [11:32:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [13:25:24] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [13:37:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [13:50:26] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/52bda0d76844...33b061ba4f19 [13:50:29] [02ssl] 07WikiTideSSLBot 0333b061b - Bot: Add SSL cert for www.streamerlore.com [13:51:22] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/33b061ba4f19...509c424647c7 [13:51:24] [02ssl] 07WikiTideSSLBot 03509c424 - Bot: Add SSL cert for cpmusicwiki.com [13:52:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [13:54:08] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/509c424647c7...679859bad8c1 [13:54:10] [02ssl] 07WikiTideSSLBot 03679859b - Bot: Add SSL cert for wiki.routedoodle.com [14:03:59] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK [14:45:45] [02puppet] 07redbluegreenhat pushed 031 commit to 03install-ntp [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0f6dd6bbf141...fab743337191 [14:45:46] [02puppet] 07redbluegreenhat 03fab7433 - fix [14:45:47] [02puppet] 07redbluegreenhat synchronize pull request 03#3859: Install ntp module - 13https://github.com/miraheze/puppet/pull/3859 [14:46:44] miraheze/puppet - redbluegreenhat the build has errored. [14:46:51] [02puppet] 07redbluegreenhat pushed 031 commit to 03install-ntp [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fab743337191...c98840fb93d6 [14:46:52] [02puppet] 07redbluegreenhat 03c98840f - oops [14:46:53] [02puppet] 07redbluegreenhat synchronize pull request 03#3859: Install ntp module - 13https://github.com/miraheze/puppet/pull/3859 [15:18:20] [02puppet] 07redbluegreenhat edited pull request 03#3859: Install ntp module - 13https://github.com/miraheze/puppet/pull/3859 [15:33:32] @reception123 please always try and renember to check DNS PRs before pushing anything [15:33:35] :) [15:33:59] [02dns] 07MacFan4000 closed pull request 03#521: add cpmusicwiki.com and streamerlore.com zones - 13https://github.com/miraheze/dns/pull/521 [16:33:00] Hello! I currently have a public wiki on Miraheze - pastport.org. Because auto correct frequently "corrects" to passport.org, I have purchased a new domain - annapolishistorywiki.org - though SquareSpace. Can Is it possible to change the domain of the public wiki from pastport.org to annapolishistorywiki.org? [16:34:07] yes, just make another request via Special:RequestSSL [16:36:13] That simple! Since I own both domains, do you know if it is possible to forward pastport.org to annapolishistorywiki.org once the request has been processed? [16:36:37] Yes that can be done [16:37:15] Great! I will try to arrange that with Squarespace. Thanks! [16:39:21] [02puppet] 07redbluegreenhat closed pull request 03#3859: Install ntp module - 13https://github.com/miraheze/puppet/pull/3859 [16:39:24] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+86/-0/±1] 13https://github.com/miraheze/puppet/compare/170534af0273...7f1835e80498 [16:39:25] [02puppet] 07redbluegreenhat 037f1835e - Install ntp module (#3859) [16:39:28] [02puppet] 07redbluegreenhat deleted branch 03install-ntp [16:39:29] [02puppet] 07redbluegreenhat deleted branch 03install-ntp - 13https://github.com/miraheze/puppet [16:51:39] PROBLEM - cp36 Puppet on cp36 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:54:25] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:54:38] PROBLEM - cp26 Puppet on cp26 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:55:01] PROBLEM - changeprop151 Puppet on changeprop151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:55:28] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:55:47] PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:56:14] PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:56:23] PROBLEM - bots171 Puppet on bots171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:56:38] Orange_Star: please stop puppet everywhere [16:56:48] And work out what icinga is moaning at [16:57:02] I know what it's moaning at [16:57:21] PROBLEM - swiftobject161 Puppet on swiftobject161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:57:54] PROBLEM - bast181 Puppet on bast181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:58:20] PROBLEM - swiftac171 Puppet on swiftac171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:58:59] PROBLEM - matomo151 Puppet on matomo151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:59:06] PROBLEM - reports171 Puppet on reports171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:59:19] PROBLEM - graphite151 Puppet on graphite151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:00:13] PROBLEM - rdb151 Puppet on rdb151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:00:14] PROBLEM - swiftobject181 Puppet on swiftobject181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:00:25] PROBLEM - db161 Puppet on db161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:00:32] PROBLEM - db171 Puppet on db171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:00:38] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:00:46] PROBLEM - mem161 Puppet on mem161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:00:52] PROBLEM - cp37 Puppet on cp37 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:00:59] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7f1835e80498...89cd9bbded51 [17:01:00] [02puppet] 07redbluegreenhat 0389cd9bb - fix [17:01:38] PROBLEM - bast161 Puppet on bast161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:01:40] PROBLEM - os161 Puppet on os161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:02:34] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:02:35] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:02:38] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:02:38] PROBLEM - graylog161 Puppet on graylog161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:02:44] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:02:55] [02puppet] 07redbluegreenhat pushed 031 commit to 03revert-3859-install-ntp [+0/-86/±1] 13https://github.com/miraheze/puppet/commit/a779ffa9df3d [17:02:56] [02puppet] 07redbluegreenhat 03a779ffa - Revert "Install ntp module (#3859)" [17:02:58] [02puppet] 07redbluegreenhat created branch 03revert-3859-install-ntp - 13https://github.com/miraheze/puppet [17:03:04] [02puppet] 07redbluegreenhat opened pull request 03#3860: Revert "Install ntp module" - 13https://github.com/miraheze/puppet/pull/3860 [17:03:17] PROBLEM - cloud16 Puppet on cloud16 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:03:33] [02puppet] 07redbluegreenhat closed pull request 03#3860: Revert "Install ntp module" - 13https://github.com/miraheze/puppet/pull/3860 [17:03:34] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-86/±1] 13https://github.com/miraheze/puppet/compare/89cd9bbded51...90c3ecdd7cfe [17:03:35] [02puppet] 07redbluegreenhat 0390c3ecd - Revert "Install ntp module" (#3860) [17:03:36] [02puppet] 07redbluegreenhat deleted branch 03revert-3859-install-ntp [17:03:39] [02puppet] 07redbluegreenhat deleted branch 03revert-3859-install-ntp - 13https://github.com/miraheze/puppet [17:03:40] PROBLEM - swiftproxy161 Puppet on swiftproxy161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:04:16] PROBLEM - os151 Puppet on os151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:04:19] PROBLEM - swiftproxy171 Puppet on swiftproxy171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:04:42] [02ManageWiki] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.17.1 [+0/-0/±2] 13https://github.com/miraheze/ManageWiki/commit/ceea6087e2d2 [17:04:43] [02ManageWiki] 07dependabot[bot] 03ceea608 - Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 [17:04:46] [02ManageWiki] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.17.1 - 13https://github.com/miraheze/ManageWiki [17:04:48] [02ManageWiki] 07dependabot[bot] opened pull request 03#465: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 - 13https://github.com/miraheze/ManageWiki/pull/465 [17:04:50] [02ManageWiki] 07dependabot[bot] labeled pull request 03#465: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 - 13https://github.com/miraheze/ManageWiki/pull/465 [17:04:53] [02ManageWiki] 07dependabot[bot] labeled pull request 03#465: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 - 13https://github.com/miraheze/ManageWiki/pull/465 [17:04:53] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:04:56] [02ManageWiki] 07dependabot[bot] commented on pull request 03#464: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.0 - 13https://github.com/miraheze/ManageWiki/pull/464#issuecomment-2150548439 [17:04:58] [02ManageWiki] 07dependabot[bot] closed pull request 03#464: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.0 - 13https://github.com/miraheze/ManageWiki/pull/464 [17:05:01] [02ManageWiki] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.17.0 - 13https://github.com/miraheze/ManageWiki [17:05:04] [02ManageWiki] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.17.0 [17:05:07] [02ManageWiki] 07coderabbitai[bot] commented on pull request 03#465: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 - 13https://github.com/miraheze/ManageWiki/pull/465#issuecomment-2150548510 [17:05:37] PROBLEM - mem151 Puppet on mem151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:06:23] PROBLEM - eventgate181 Puppet on eventgate181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:06:25] PROBLEM - cloud15 Puppet on cloud15 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:07:11] PROBLEM - mw151 Puppet on mw151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:07:13] [02puppet] 07redbluegreenhat pushed 031 commit to 03revert-3860-revert-3859-install-ntp [+86/-0/±1] 13https://github.com/miraheze/puppet/commit/952fac29d90f [17:07:16] [02puppet] 07redbluegreenhat 03952fac2 - Revert "Revert "Install ntp module" (#3860)" [17:07:17] [02puppet] 07redbluegreenhat created branch 03revert-3860-revert-3859-install-ntp - 13https://github.com/miraheze/puppet [17:08:31] PROBLEM - ldap171 Puppet on ldap171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:08:58] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:09:20] PROBLEM - mw162 Puppet on mw162 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:09:40] PROBLEM - jobchron171 Puppet on jobchron171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:09:44] PROBLEM - cp27 Puppet on cp27 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:09:44] PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:09:45] miraheze/ManageWiki - dependabot[bot] the build passed. [17:09:52] PROBLEM - phorge171 Puppet on phorge171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:10:17] [02puppet] 07coderabbitai[bot] commented on pull request 03#3860: Revert "Install ntp module" - 13https://github.com/miraheze/puppet/pull/3860#issuecomment-2150558668 [17:10:49] RhinosF1: Fixed, icinga will report that the issue has been fixed for the agents in a while [17:11:11] PROBLEM - mw161 Puppet on mw161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:11:18] seems I made a mistake on my PR [17:11:21] oopsies [17:11:31] Ok [17:12:44] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:15:28] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [17:19:39] RECOVERY - cp36 Puppet on cp36 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [17:21:04] RECOVERY - changeprop151 Puppet on changeprop151 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [17:22:25] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [17:22:44] RECOVERY - cp26 Puppet on cp26 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:23:10] [02puppet] 07redbluegreenhat opened pull request 03#3861: Install ntp module - 13https://github.com/miraheze/puppet/pull/3861 [17:23:15] [02puppet] 07coderabbitai[bot] commented on pull request 03#3861: Install ntp module - 13https://github.com/miraheze/puppet/pull/3861#issuecomment-2150580857 [17:24:14] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [17:24:22] RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [17:25:21] RECOVERY - swiftobject161 Puppet on swiftobject161 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [17:25:47] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:26:20] RECOVERY - swiftac171 Puppet on swiftac171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:26:32] RECOVERY - db171 Puppet on db171 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [17:26:34] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:26:43] RECOVERY - matomo151 Puppet on matomo151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:27:05] RECOVERY - reports171 Puppet on reports171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:27:19] RECOVERY - graphite151 Puppet on graphite151 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [17:27:20] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [17:27:52] RECOVERY - bast181 Puppet on bast181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:28:08] RECOVERY - rdb151 Puppet on rdb151 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [17:28:14] RECOVERY - swiftobject181 Puppet on swiftobject181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:28:25] RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:28:38] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:28:52] RECOVERY - mem161 Puppet on mem161 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [17:29:40] RECOVERY - os161 Puppet on os161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:29:42] RECOVERY - bast161 Puppet on bast161 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [17:30:33] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [17:30:35] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [17:30:39] RECOVERY - graylog161 Puppet on graylog161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:30:51] RECOVERY - cp37 Puppet on cp37 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:31:17] RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:31:40] RECOVERY - swiftproxy161 Puppet on swiftproxy161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:32:12] RECOVERY - os151 Puppet on os151 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [17:32:38] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:33:37] RECOVERY - mem151 Puppet on mem151 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [17:34:19] RECOVERY - swiftproxy171 Puppet on swiftproxy171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:34:22] RECOVERY - eventgate181 Puppet on eventgate181 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [17:34:36] RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [17:34:52] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:35:11] RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:36:31] RECOVERY - ldap171 Puppet on ldap171 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [17:36:57] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:37:28] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [17:37:39] RECOVERY - jobchron171 Puppet on jobchron171 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [17:37:46] RECOVERY - cp27 Puppet on cp27 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:37:52] RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [17:39:11] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [17:39:54] RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:49:34] [02puppet] 07redbluegreenhat closed pull request 03#3861: Install ntp module - 13https://github.com/miraheze/puppet/pull/3861 [17:49:37] [02puppet] 07redbluegreenhat deleted branch 03revert-3860-revert-3859-install-ntp [17:49:39] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+86/-0/±1] 13https://github.com/miraheze/puppet/compare/90c3ecdd7cfe...865ca95be687 [17:49:40] [02puppet] 07redbluegreenhat 03865ca95 - Install ntp module (#3861) [17:49:41] [02puppet] 07redbluegreenhat deleted branch 03revert-3860-revert-3859-install-ntp - 13https://github.com/miraheze/puppet [18:01:48] PROBLEM - bast161 Puppet on bast161 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ntp] [18:01:58] welp [18:02:07] I'll try again tomorrow [18:02:17] [02puppet] 07redbluegreenhat pushed 031 commit to 03revert-3861-revert-3860-revert-3859-install-ntp [+0/-86/±1] 13https://github.com/miraheze/puppet/commit/37baf8bfd468 [18:02:18] [02puppet] 07redbluegreenhat 0337baf8b - Revert "Install ntp module (#3861)" [18:02:19] [02puppet] 07redbluegreenhat created branch 03revert-3861-revert-3860-revert-3859-install-ntp - 13https://github.com/miraheze/puppet [18:02:21] [02puppet] 07redbluegreenhat opened pull request 03#3862: Revert "Install ntp module" - 13https://github.com/miraheze/puppet/pull/3862 [18:02:26] [02puppet] 07coderabbitai[bot] commented on pull request 03#3862: Revert "Install ntp module" - 13https://github.com/miraheze/puppet/pull/3862#issuecomment-2150647353 [18:02:29] [02puppet] 07redbluegreenhat closed pull request 03#3862: Revert "Install ntp module" - 13https://github.com/miraheze/puppet/pull/3862 [18:02:32] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-86/±1] 13https://github.com/miraheze/puppet/compare/865ca95be687...2e5fced6e487 [18:02:34] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ntp] [18:02:35] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ntp] [18:02:35] [02puppet] 07redbluegreenhat 032e5fced - Revert "Install ntp module" (#3862) [18:02:37] [02puppet] 07redbluegreenhat deleted branch 03revert-3861-revert-3860-revert-3859-install-ntp [18:02:40] [02puppet] 07redbluegreenhat deleted branch 03revert-3861-revert-3860-revert-3859-install-ntp - 13https://github.com/miraheze/puppet [18:04:08] PROBLEM - os151 Puppet on os151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ntp] [18:04:19] PROBLEM - swiftproxy171 Puppet on swiftproxy171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ntp] [18:04:38] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ntp] [18:05:28] RECOVERY - eventgate181 NTP time on eventgate181 is OK: NTP OK: Offset 0.006197273731 secs [18:05:28] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ntp] [18:05:37] PROBLEM - mem151 Puppet on mem151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ntp] [18:05:46] at least we fixed time on eventgate181 [18:06:10] problem this time was that some servers are running timesyncd while some, like eventgate and changeprop, don't [18:06:39] this leads to a conflict when puppet goes to install the ntp package [18:06:58] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ntp] [18:07:11] PROBLEM - mw151 Puppet on mw151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ntp] [18:08:19] the fact that it is installed on the "older" servers while it is not installed on the "newer" ones tells me this is not being managed via puppet [18:08:31] PROBLEM - ldap171 Puppet on ldap171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ntp] [18:09:37] PROBLEM - mw162 Puppet on mw162 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ntp] [18:09:40] PROBLEM - jobchron171 Puppet on jobchron171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ntp] [18:09:50] PROBLEM - cp27 Puppet on cp27 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ntp] [18:11:11] PROBLEM - mw161 Puppet on mw161 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ntp] [18:11:39] PROBLEM - os162 Puppet on os162 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ntp] [18:11:52] PROBLEM - phorge171 Puppet on phorge171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[ntp] [18:25:28] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:29:52] RECOVERY - bast161 Puppet on bast161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:30:33] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [18:30:35] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [18:32:04] RECOVERY - os151 Puppet on os151 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [18:32:19] RECOVERY - swiftproxy171 Puppet on swiftproxy171 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [18:32:38] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:33:37] RECOVERY - mem151 Puppet on mem151 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [18:35:11] RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [18:36:31] RECOVERY - ldap171 Puppet on ldap171 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [18:36:57] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:37:39] RECOVERY - jobchron171 Puppet on jobchron171 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [18:37:45] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [18:37:53] RECOVERY - cp27 Puppet on cp27 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:39:11] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:39:43] RECOVERY - os162 Puppet on os162 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [18:39:52] RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:42:20] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [18:52:29] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [19:02:30] PROBLEM - swiftobject151 APT on swiftobject151 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [19:03:08] PROBLEM - bots171 APT on bots171 is CRITICAL: APT CRITICAL: 37 packages available for upgrade (1 critical updates). [19:03:30] PROBLEM - swiftobject181 APT on swiftobject181 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [19:03:38] PROBLEM - swiftproxy161 APT on swiftproxy161 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [19:03:53] PROBLEM - swiftproxy171 APT on swiftproxy171 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [19:04:29] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [19:04:58] PROBLEM - mw161 APT on mw161 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [19:05:32] PROBLEM - swiftobject161 APT on swiftobject161 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [19:05:54] PROBLEM - mw162 APT on mw162 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [19:06:08] PROBLEM - swiftobject171 APT on swiftobject171 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [19:06:24] PROBLEM - mw151 APT on mw151 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [19:06:44] PROBLEM - mw181 APT on mw181 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [19:08:45] PROBLEM - mw182 APT on mw182 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [19:09:23] PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [19:10:23] PROBLEM - mw172 APT on mw172 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [19:11:34] PROBLEM - mw171 APT on mw171 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [19:12:29] PROBLEM - swiftac171 APT on swiftac171 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [19:12:38] PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [19:15:36] PROBLEM - mw152 APT on mw152 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [19:17:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [19:51:51] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:02:41] !log [alex@puppet181] uninstalled systemd-timesyncd [20:02:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:06:15] !log [alex@puppet181] uninstalled systemd-timesyncd on all servers using salt (sudo salt-ssh '*' cmd.run 'sudo apt-get remove systemd-timesyncd -y') [20:06:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:10:29] [02puppet] 07redbluegreenhat pushed 031 commit to 03purge-timesyncd [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/3e74a174413d [20:10:32] [02puppet] 07redbluegreenhat 033e74a17 - Ensure systemd-timesyncd is not installed [20:10:33] [02puppet] 07redbluegreenhat created branch 03purge-timesyncd - 13https://github.com/miraheze/puppet [20:11:09] [02puppet] 07redbluegreenhat opened pull request 03#3863: Ensure systemd-timesyncd is not installed - 13https://github.com/miraheze/puppet/pull/3863 [20:11:13] [02puppet] 07coderabbitai[bot] commented on pull request 03#3863: Ensure systemd-timesyncd is not installed - 13https://github.com/miraheze/puppet/pull/3863#issuecomment-2150877655 [20:15:21] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:19:21] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [20:22:11] swiftproxy161 also didn't have any kind of ntp daemon [20:22:57] as well as mwtask* [20:25:35] [02puppet] 07redbluegreenhat closed pull request 03#3863: Ensure systemd-timesyncd is not installed - 13https://github.com/miraheze/puppet/pull/3863 [20:25:36] [02puppet] 07redbluegreenhat deleted branch 03purge-timesyncd [20:25:39] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2e5fced6e487...e5669d127a63 [20:25:40] [02puppet] 07redbluegreenhat 03e5669d1 - Ensure systemd-timesyncd is not installed (#3863) [20:25:41] [02puppet] 07redbluegreenhat deleted branch 03purge-timesyncd - 13https://github.com/miraheze/puppet [20:34:34] PROBLEM - cp41 APT on cp41 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (1 critical updates). [20:35:02] !log exempt phab server from cloudflare WAF [20:35:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:35:41] PROBLEM - cp37 APT on cp37 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (1 critical updates). [20:35:45] PROBLEM - mem151 APT on mem151 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (1 critical updates). [20:35:58] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (1 critical updates). [20:36:41] PROBLEM - bast161 APT on bast161 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (1 critical updates). [20:36:41] PROBLEM - jobchron171 APT on jobchron171 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (1 critical updates). [20:37:45] PROBLEM - ldap171 APT on ldap171 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (1 critical updates). [20:37:59] PROBLEM - cp36 APT on cp36 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (1 critical updates). [20:37:59] PROBLEM - cp27 APT on cp27 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (1 critical updates). [20:38:11] PROBLEM - db181 APT on db181 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (1 critical updates). [20:38:21] PROBLEM - bast181 APT on bast181 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (1 critical updates). [20:38:59] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (1 critical updates). [20:39:36] [02puppet] 07redbluegreenhat pushed 031 commit to 03revert-3862-revert-3861-revert-3860-revert-3859-install-ntp [+86/-0/±1] 13https://github.com/miraheze/puppet/commit/1268a64f544c [20:39:39] [02puppet] 07redbluegreenhat 031268a64 - Revert "Revert "Install ntp module" (#3862)" [20:39:40] [02puppet] 07redbluegreenhat created branch 03revert-3862-revert-3861-revert-3860-revert-3859-install-ntp - 13https://github.com/miraheze/puppet [20:39:47] PROBLEM - reports171 APT on reports171 is CRITICAL: APT CRITICAL: 29 packages available for upgrade (1 critical updates). [20:39:50] PROBLEM - os161 APT on os161 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [20:39:50] PROBLEM - db161 APT on db161 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (1 critical updates). [20:40:19] PROBLEM - rdb151 APT on rdb151 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (1 critical updates). [20:40:26] [02puppet] 07redbluegreenhat opened pull request 03#3864: Install ntp module - 13https://github.com/miraheze/puppet/pull/3864 [20:40:31] [02puppet] 07coderabbitai[bot] commented on pull request 03#3864: Install ntp module - 13https://github.com/miraheze/puppet/pull/3864#issuecomment-2150927086 [20:40:41] PROBLEM - phorge171 APT on phorge171 is CRITICAL: APT CRITICAL: 29 packages available for upgrade (1 critical updates). [20:40:54] PROBLEM - os151 APT on os151 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [20:40:57] PROBLEM - os162 APT on os162 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [20:42:09] PROBLEM - mem161 APT on mem161 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (1 critical updates). [20:42:37] PROBLEM - graphite151 APT on graphite151 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [20:43:12] PROBLEM - db171 APT on db171 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (1 critical updates). [20:43:17] PROBLEM - db182 APT on db182 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (1 critical updates). [20:43:20] PROBLEM - kafka181 APT on kafka181 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [20:44:18] PROBLEM - matomo151 APT on matomo151 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (1 critical updates). [20:44:48] PROBLEM - db151 APT on db151 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (1 critical updates). [20:45:35] PROBLEM - mon181 APT on mon181 is CRITICAL: APT CRITICAL: 32 packages available for upgrade (1 critical updates). [20:45:56] PROBLEM - changeprop151 APT on changeprop151 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [20:46:07] PROBLEM - cp26 APT on cp26 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (1 critical updates). [20:46:24] PROBLEM - graylog161 APT on graylog161 is CRITICAL: APT CRITICAL: 30 packages available for upgrade (1 critical updates). [20:46:24] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [20:46:32] PROBLEM - cp51 APT on cp51 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (1 critical updates). [20:48:02] PROBLEM - puppet181 APT on puppet181 is CRITICAL: APT CRITICAL: 32 packages available for upgrade (1 critical updates). [20:49:04] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [21:04:59] [02puppet] 07redbluegreenhat closed pull request 03#3864: Install ntp module - 13https://github.com/miraheze/puppet/pull/3864 [21:05:02] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+86/-0/±1] 13https://github.com/miraheze/puppet/compare/e5669d127a63...f58ed81fc3d7 [21:05:03] [02puppet] 07redbluegreenhat 03f58ed81 - Install ntp module (#3864) [21:05:05] [02puppet] 07redbluegreenhat deleted branch 03revert-3862-revert-3861-revert-3860-revert-3859-install-ntp - 13https://github.com/miraheze/puppet [21:05:06] [02puppet] 07redbluegreenhat deleted branch 03revert-3862-revert-3861-revert-3860-revert-3859-install-ntp [21:16:13] RECOVERY - kafka181 NTP time on kafka181 is OK: NTP OK: Offset -0.000577300787 secs [21:19:51] [02ssl] 07redbluegreenhat opened pull request 03#780: drop pokemonwiki.info - 13https://github.com/miraheze/ssl/pull/780 [21:19:57] [02ssl] 07coderabbitai[bot] commented on pull request 03#780: drop pokemonwiki.info - 13https://github.com/miraheze/ssl/pull/780#issuecomment-2150981101 [21:20:14] [02ssl] 07redbluegreenhat pushed 031 commit to 03drop-pokemonwiki.info [+0/-1/±1] 13https://github.com/miraheze/ssl/commit/482152164c90 [21:20:16] [02ssl] 07redbluegreenhat 034821521 - drop pokemonwiki.info [21:20:24] [02ssl] 07redbluegreenhat created branch 03drop-pokemonwiki.info - 13https://github.com/miraheze/ssl [21:21:32] [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#780: drop pokemonwiki.info - 13https://github.com/miraheze/ssl/pull/780#issuecomment-2150981101 [21:22:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [21:23:25] RECOVERY - changeprop151 NTP time on changeprop151 is OK: NTP OK: Offset 0.006107062101 secs [21:23:53] ntpd is now running accross all of our infra! [21:24:16] and is centrally configured via puppet! [21:27:43] RECOVERY - graphite151 NTP time on graphite151 is OK: NTP OK: Offset 0.005606561899 secs [21:45:54] ntpd? [22:02:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [22:03:33] [02IncidentReporting] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.17.1 [+0/-0/±2] 13https://github.com/miraheze/IncidentReporting/commit/8cbacd7f0893 [22:03:36] [02IncidentReporting] 07dependabot[bot] 038cbacd7 - Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 [22:03:37] [02IncidentReporting] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.17.1 - 13https://github.com/miraheze/IncidentReporting [22:03:39] [02IncidentReporting] 07dependabot[bot] labeled pull request 03#85: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 - 13https://github.com/miraheze/IncidentReporting/pull/85 [22:03:41] [02IncidentReporting] 07dependabot[bot] opened pull request 03#85: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 - 13https://github.com/miraheze/IncidentReporting/pull/85 [22:03:43] [02IncidentReporting] 07dependabot[bot] labeled pull request 03#85: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 - 13https://github.com/miraheze/IncidentReporting/pull/85 [22:03:45] [02IncidentReporting] 07dependabot[bot] closed pull request 03#84: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.0 - 13https://github.com/miraheze/IncidentReporting/pull/84 [22:03:48] [02IncidentReporting] 07dependabot[bot] commented on pull request 03#84: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.0 - 13https://github.com/miraheze/IncidentReporting/pull/84#issuecomment-2151034342 [22:03:50] [02IncidentReporting] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.17.0 [22:03:52] [02IncidentReporting] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.17.0 - 13https://github.com/miraheze/IncidentReporting [22:03:54] [02IncidentReporting] 07coderabbitai[bot] commented on pull request 03#85: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 - 13https://github.com/miraheze/IncidentReporting/pull/85#issuecomment-2151034663 [22:07:49] miraheze/IncidentReporting - dependabot[bot] the build passed. [22:17:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [22:48:24] [02CreateWiki] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.17.1 [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/commit/f4367497f3b7 [22:48:25] [02CreateWiki] 07dependabot[bot] 03f436749 - Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 [22:48:26] [02CreateWiki] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.17.1 - 13https://github.com/miraheze/CreateWiki [22:48:29] [02CreateWiki] 07dependabot[bot] labeled pull request 03#535: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 - 13https://github.com/miraheze/CreateWiki/pull/535 [22:48:31] [02CreateWiki] 07dependabot[bot] labeled pull request 03#535: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 - 13https://github.com/miraheze/CreateWiki/pull/535 [22:48:34] [02CreateWiki] 07dependabot[bot] opened pull request 03#535: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 - 13https://github.com/miraheze/CreateWiki/pull/535 [22:48:35] [02CreateWiki] 07dependabot[bot] commented on pull request 03#533: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.0 - 13https://github.com/miraheze/CreateWiki/pull/533#issuecomment-2151085041 [22:48:36] [02CreateWiki] 07dependabot[bot] closed pull request 03#533: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.0 - 13https://github.com/miraheze/CreateWiki/pull/533 [22:48:38] [02CreateWiki] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.17.0 [22:48:39] [02CreateWiki] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.17.0 - 13https://github.com/miraheze/CreateWiki [22:49:54] [02CreateWiki] 07coderabbitai[bot] commented on pull request 03#535: Bump stylelint-config-wikimedia from 0.16.1 to 0.17.1 - 13https://github.com/miraheze/CreateWiki/pull/535#issuecomment-2151086472 [22:57:16] miraheze/CreateWiki - dependabot[bot] the build passed. [23:40:55] PROBLEM - cloud15 IPMI Sensors on cloud15 is WARNING: IPMI Status: Warning [1 system event log (SEL) entry present]