[00:03:20] RECOVERY - SSH on contint2001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [05:44:01] !log restart trafficserver-tls.service on deployment-cache-upload06, was using an expired cert [05:44:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:09:58] PROBLEM - SSH on contint2001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [08:14:57] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: puppet CI task failing with: Cannot allocate memory - https://phabricator.wikimedia.org/T284998 (10hashar) From contint2001: ` $ grep -l 'Cannot allocate' /srv/jenkins/builds/*puppet*/*/log operations-puppet-tests-buster-docker/27249/log ope... [08:20:47] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Pywikibot, 10Pywikibot-tests: Move pywikibot CI from travis-ci.org to travis-ci.com - https://phabricator.wikimedia.org/T285032 (10Xqt) [08:21:00] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Pywikibot, 10Pywikibot-tests: Move pywikibot CI from travis-ci.org to travis-ci.com - https://phabricator.wikimedia.org/T285032 (10Xqt) p:05Triage→03High [08:21:18] Project mediawiki-core-doxygen-docker build #25700: 04FAILURE in 16 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/25700/ [08:23:01] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Pywikibot, 10Pywikibot-tests: Move pywikibot CI from travis-ci.org to travis-ci.com - https://phabricator.wikimedia.org/T285032 (10Xqt) [08:31:14] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: puppet CI task failing with: Cannot allocate memory - https://phabricator.wikimedia.org/T284998 (10hashar) We had an issue in the paste with a big process (like mediawiki test) forking an external command that is itself lightweight. But Linux... [08:46:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: puppet CI task failing with: Cannot allocate memory - https://phabricator.wikimedia.org/T284998 (10hashar) My suspicion is that it is due to multiple concurrent builds of the job AND us running tests in parallel. If the changes triggers run... [09:10:42] RECOVERY - SSH on contint2001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [09:17:59] Yippee, build fixed! [09:18:00] Project mediawiki-core-doxygen-docker build #25701: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/25701/ [11:40:18] https://gerrit.wikimedia.org/r/700000 round numbers! \o/ [12:01:05] majavah: Well done! you can claim your price now ;D [12:12:32] 100k was l10n update as well https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ProofreadPage/+/100000/ [12:12:46] 200k was i18n related as well https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Flow/+/200000/ ;D [12:12:58] PROBLEM - SSH on contint2001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [12:13:02] so I guess most of our changes are generated by robots [13:23:06] 10Continuous-Integration-Infrastructure, 10Pywikibot, 10Pywikibot-tests: Move pywikibot CI from travis-ci.org to travis-ci.com - https://phabricator.wikimedia.org/T285032 (10thcipriani) [14:14:07] RECOVERY - SSH on contint2001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [14:22:47] Project mediawiki-core-doxygen-docker build #25706: 04FAILURE in 18 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/25706/ [15:22:11] Yippee, build fixed! [15:22:11] Project mediawiki-core-doxygen-docker build #25707: 09FIXED in 17 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/25707/ [15:58:14] (03CR) 10Hashar: "> I was thinking of maybe making this three different scripts - one to build the whole set locally, one to update a regex, and one to dele" [integration/config] - 10https://gerrit.wikimedia.org/r/693431 (owner: 10Hashar) [16:04:16] (03CR) 10Jforrester: [C: 03+1] "> Patch Set 2:" [integration/config] - 10https://gerrit.wikimedia.org/r/693431 (owner: 10Hashar) [17:16:06] (03PS1) 10Lars Wirzenius: fix: scap without arguments should not give a traceback [tools/scap] - 10https://gerrit.wikimedia.org/r/700088 [17:16:36] (03CR) 10Lars Wirzenius: "A bare scap without arguments should now give an error message instead of a traceback." [tools/scap] - 10https://gerrit.wikimedia.org/r/700088 (owner: 10Lars Wirzenius) [17:31:30] (03CR) 10Ahmon Dancy: [C: 03+2] fix: scap without arguments should not give a traceback (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/700088 (owner: 10Lars Wirzenius) [17:32:13] (03Merged) 10jenkins-bot: fix: scap without arguments should not give a traceback [tools/scap] - 10https://gerrit.wikimedia.org/r/700088 (owner: 10Lars Wirzenius) [18:20:29] (03PS1) 10Gergő Tisza: Add NormalizedException PHP library to CI [integration/config] - 10https://gerrit.wikimedia.org/r/700092 (https://phabricator.wikimedia.org/T284732) [19:18:05] PROBLEM - SSH on contint2001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [20:18:49] RECOVERY - SSH on contint2001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [21:45:34] (03PS2) 10Jforrester: Zuul: Install CI for mediawiki/libs/NormalizedException [integration/config] - 10https://gerrit.wikimedia.org/r/700092 (https://phabricator.wikimedia.org/T284732) (owner: 10Gergő Tisza) [21:45:40] (03CR) 10Jforrester: [C: 03+2] Zuul: Install CI for mediawiki/libs/NormalizedException [integration/config] - 10https://gerrit.wikimedia.org/r/700092 (https://phabricator.wikimedia.org/T284732) (owner: 10Gergő Tisza) [21:46:52] (03Merged) 10jenkins-bot: Zuul: Install CI for mediawiki/libs/NormalizedException [integration/config] - 10https://gerrit.wikimedia.org/r/700092 (https://phabricator.wikimedia.org/T284732) (owner: 10Gergő Tisza) [21:47:00] !log Zuul: Install CI for mediawiki/libs/NormalizedException T284732 [21:47:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:47:02] T284732: Librarize NormalizedException patch - https://phabricator.wikimedia.org/T284732 [22:19:58] (03PS1) 10Ahmon Dancy: Another Python 3 str/bytes fix [tools/scap] - 10https://gerrit.wikimedia.org/r/700108 [22:20:07] (03PS1) 10Ahmon Dancy: Mention user and hostname when an ssh error occurs [tools/scap] - 10https://gerrit.wikimedia.org/r/700109 [22:20:17] (03PS1) 10Ahmon Dancy: Better error message if a dsh file cannot be found [tools/scap] - 10https://gerrit.wikimedia.org/r/700110 [22:20:25] (03PS1) 10Ahmon Dancy: scap deploy: Better error message if scap.cfg is underconfigured [tools/scap] - 10https://gerrit.wikimedia.org/r/700111 [22:20:35] (03CR) 10jerkins-bot: [V: 04-1] Mention user and hostname when an ssh error occurs [tools/scap] - 10https://gerrit.wikimedia.org/r/700109 (owner: 10Ahmon Dancy) [22:22:16] (03CR) 10Ahmon Dancy: "recheck" [tools/scap] - 10https://gerrit.wikimedia.org/r/700109 (owner: 10Ahmon Dancy) [22:22:49] (03CR) 10Ahmon Dancy: [C: 03+2] Another Python 3 str/bytes fix [tools/scap] - 10https://gerrit.wikimedia.org/r/700108 (owner: 10Ahmon Dancy) [22:23:42] (03Merged) 10jenkins-bot: Another Python 3 str/bytes fix [tools/scap] - 10https://gerrit.wikimedia.org/r/700108 (owner: 10Ahmon Dancy) [22:26:52] (03PS2) 10Ahmon Dancy: Mention user and hostname when an ssh error occurs [tools/scap] - 10https://gerrit.wikimedia.org/r/700109 [22:32:51] (03CR) 10Thcipriani: [C: 03+2] scap deploy: Better error message if scap.cfg is underconfigured [tools/scap] - 10https://gerrit.wikimedia.org/r/700111 (owner: 10Ahmon Dancy) [22:33:30] (03Merged) 10jenkins-bot: scap deploy: Better error message if scap.cfg is underconfigured [tools/scap] - 10https://gerrit.wikimedia.org/r/700111 (owner: 10Ahmon Dancy) [22:36:08] (03CR) 10Thcipriani: [C: 03+1] "Inline suggestion, I don't feel strongly about it, feel free to ignore and self +2." (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/700110 (owner: 10Ahmon Dancy) [22:39:45] (03PS2) 10Ahmon Dancy: Better error message if a dsh file cannot be found [tools/scap] - 10https://gerrit.wikimedia.org/r/700110 [22:39:55] (03CR) 10Ahmon Dancy: Better error message if a dsh file cannot be found (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/700110 (owner: 10Ahmon Dancy) [22:40:19] (03CR) 10Thcipriani: [C: 03+2] Better error message if a dsh file cannot be found [tools/scap] - 10https://gerrit.wikimedia.org/r/700110 (owner: 10Ahmon Dancy) [22:40:59] (03Merged) 10jenkins-bot: Better error message if a dsh file cannot be found [tools/scap] - 10https://gerrit.wikimedia.org/r/700110 (owner: 10Ahmon Dancy) [22:42:18] (03CR) 10Ahmon Dancy: [C: 03+2] Mention user and hostname when an ssh error occurs [tools/scap] - 10https://gerrit.wikimedia.org/r/700109 (owner: 10Ahmon Dancy) [22:42:58] (03Merged) 10jenkins-bot: Mention user and hostname when an ssh error occurs [tools/scap] - 10https://gerrit.wikimedia.org/r/700109 (owner: 10Ahmon Dancy)