[08:25:54] (03PS1) 10Kosta Harlan: zuul: Add dependency on CirrusSearch for GrowthExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/738838 [08:27:29] kostajh: :) [08:32:59] (03CR) 10Hashar: [C: 03+2] zuul: Add dependency on CirrusSearch for GrowthExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/738838 (owner: 10Kosta Harlan) [08:34:12] (03CR) 10Hashar: [C: 03+2] utils/jjb-diff: support job filtering [integration/config] - 10https://gerrit.wikimedia.org/r/738478 (owner: 10Hashar) [08:35:35] (03Merged) 10jenkins-bot: zuul: Add dependency on CirrusSearch for GrowthExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/738838 (owner: 10Kosta Harlan) [08:36:01] (03Merged) 10jenkins-bot: utils/jjb-diff: support job filtering [integration/config] - 10https://gerrit.wikimedia.org/r/738478 (owner: 10Hashar) [08:38:22] (03CR) 10Hashar: "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/738838 (owner: 10Kosta Harlan) [08:39:53] (03CR) 10Hashar: [C: 03+2] jjb: use job template for mw phan jobs [integration/config] - 10https://gerrit.wikimedia.org/r/738479 (owner: 10Hashar) [08:40:20] (03CR) 10Kosta Harlan: zuul: Add dependency on CirrusSearch for GrowthExperiments (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/738838 (owner: 10Kosta Harlan) [08:41:54] (03Merged) 10jenkins-bot: jjb: use job template for mw phan jobs [integration/config] - 10https://gerrit.wikimedia.org/r/738479 (owner: 10Hashar) [08:48:06] (03PS3) 10Hashar: jjb: parameterize the docker-setup-mwext-for-phan macro [integration/config] - 10https://gerrit.wikimedia.org/r/738481 [08:59:11] (03CR) 10Hashar: [C: 03+2] "No change introduced in jjb as expected" [integration/config] - 10https://gerrit.wikimedia.org/r/738481 (owner: 10Hashar) [09:00:26] (03CR) 10Dvogel hallowelt: "This change is ready for review." [integration/config] - 10https://gerrit.wikimedia.org/r/738634 (owner: 10Dvogel hallowelt) [09:01:35] (03Merged) 10jenkins-bot: jjb: parameterize the docker-setup-mwext-for-phan macro [integration/config] - 10https://gerrit.wikimedia.org/r/738481 (owner: 10Hashar) [09:52:41] (03PS1) 10Hashar: jjb: drop duplicates docker-cache-dir creation [integration/config] - 10https://gerrit.wikimedia.org/r/738858 [09:54:21] (03PS1) 10Hashar: jjb: reorder builders in macro-docker [integration/config] - 10https://gerrit.wikimedia.org/r/738859 [09:59:05] 10Continuous-Integration-Infrastructure, 10Quibble, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), and 2 others: Terminating MySQL takes several minutes in (Wikibase?) CI jobs - https://phabricator.wikimedia.org/T265615 (10awight) This i... [12:57:05] (03CR) 10Hashar: [C: 03+2] jjb: drop duplicates docker-cache-dir creation [integration/config] - 10https://gerrit.wikimedia.org/r/738858 (owner: 10Hashar) [12:57:23] (03CR) 10Hashar: [C: 03+2] jjb: reorder builders in macro-docker [integration/config] - 10https://gerrit.wikimedia.org/r/738859 (owner: 10Hashar) [12:57:50] (03CR) 10Hashar: [C: 03+2] Add skin BlueSpiceDiscovery [integration/config] - 10https://gerrit.wikimedia.org/r/738634 (owner: 10Dvogel hallowelt) [12:59:17] (03Merged) 10jenkins-bot: jjb: drop duplicates docker-cache-dir creation [integration/config] - 10https://gerrit.wikimedia.org/r/738858 (owner: 10Hashar) [13:00:18] (03Merged) 10jenkins-bot: jjb: reorder builders in macro-docker [integration/config] - 10https://gerrit.wikimedia.org/r/738859 (owner: 10Hashar) [13:00:38] (03Merged) 10jenkins-bot: Add skin BlueSpiceDiscovery [integration/config] - 10https://gerrit.wikimedia.org/r/738634 (owner: 10Dvogel hallowelt) [13:01:22] (03CR) 10Hashar: "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/738634 (owner: 10Dvogel hallowelt) [13:30:24] PROBLEM - SSH on contint1001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [14:08:37] (03PS1) 10Hashar: Upgrade jjb to 3.10.0 [integration/config] - 10https://gerrit.wikimedia.org/r/738911 [14:37:43] (03PS5) 10Thcipriani: Avoid copying L10N files from/to /tmp [tools/scap] - 10https://gerrit.wikimedia.org/r/738453 (https://phabricator.wikimedia.org/T295304) (owner: 10Ahmon Dancy) [14:42:11] (03PS1) 10Hashar: jjb: remove intermediate docker-ci-src-setup macro [integration/config] - 10https://gerrit.wikimedia.org/r/738917 [14:46:31] (03CR) 10Kosta Harlan: Support for ~/.config/quibble.ini (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/735414 (https://phabricator.wikimedia.org/T238225) (owner: 10Hashar) [14:50:29] (03PS1) 10Hashar: Upgrade jjb to 3.6.0 [integration/config] - 10https://gerrit.wikimedia.org/r/738920 [14:51:22] (03CR) 10Thcipriani: [C: 03+1] "Looks good in isolation." [tools/scap] - 10https://gerrit.wikimedia.org/r/738453 (https://phabricator.wikimedia.org/T295304) (owner: 10Ahmon Dancy) [14:57:01] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Seen): Switch back to upstream jenkins xunit plugin after PHPUnit fix is released - https://phabricator.wikimedia.org/T194096 (10hashar) Quibble based jobs no more generate PHPUnit Junit reports after T256402 [14:57:18] 10Project-Admins: Create a few tag projects for task tracking - https://phabricator.wikimedia.org/T295692 (10dcaro) [14:57:39] (03PS2) 10Hashar: Upgrade jjb to 3.10.0 [integration/config] - 10https://gerrit.wikimedia.org/r/738911 (https://phabricator.wikimedia.org/T194096) [14:57:41] (03PS2) 10Hashar: jjb: remove intermediate docker-ci-src-setup macro [integration/config] - 10https://gerrit.wikimedia.org/r/738917 [14:57:43] (03PS1) 10Hashar: Upgrade jjb to 3.11.0 [integration/config] - 10https://gerrit.wikimedia.org/r/738921 [14:59:00] !log CI Jenkins: upgrading the Xunit plugin from 1.103 hack to 3.0.5 # T194096 [14:59:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:59:03] T194096: Switch back to upstream jenkins xunit plugin after PHPUnit fix is released - https://phabricator.wikimedia.org/T194096 [14:59:16] 10Project-Admins, 10User-dcaro: Create a few tag projects for task tracking - https://phabricator.wikimedia.org/T295692 (10dcaro) [15:05:42] !log Restarting CI Jenkins for plugin update [15:05:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:06:28] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Seen), 10Patch-For-Review: Switch back to upstream jenkins xunit plugin after PHPUnit fix is released - https://phabricator.wikimedia.org/T194096 (10hashar) 05Open→03Resolved a:03hashar [15:11:10] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Done by Wed 24 Nov 🔥): Remove Performance plugin from CI Jenkins - https://phabricator.wikimedia.org/T295577 (10hashar) [15:11:45] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Done by Wed 24 Nov 🔥): Switch back to upstream jenkins xunit plugin after PHPUnit fix is released - https://phabricator.wikimedia.org/T194096 (10hashar) [15:11:58] 10Project-Admins, 10User-dcaro: Create a few tag projects for task tracking - https://phabricator.wikimedia.org/T295692 (10Aklapper) #Worktype-Maintenance and #WorkType-NewFunctionality and #unplanned-sprint-work exist. It's unclear to me what "project related tasks" or "alert created tasks" or "team created t... [15:27:38] 10Project-Admins, 10User-dcaro: Create a few tag projects for task tracking - https://phabricator.wikimedia.org/T295692 (10dcaro) > Is there broader interest in this that could be pointed to? No, it's for personal work management. > WorkType-Maintenance and WorkType-NewFunctionality and Unplanned-Sprint-Work... [15:31:32] RECOVERY - SSH on contint1001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [15:56:22] (03PS6) 10Ahmon Dancy: Avoid copying L10N files from/to /tmp on deploy server [tools/scap] - 10https://gerrit.wikimedia.org/r/738453 (https://phabricator.wikimedia.org/T295304) [16:01:34] (03PS1) 10Ahmon Dancy: deploy: Ensure mwdeploy user is a member of the www-data group [tools/train-dev] - 10https://gerrit.wikimedia.org/r/738954 (https://phabricator.wikimedia.org/T295304) [16:09:41] hi folks, as FYI I managed to break puppet in deployment-prep, working on a fix [16:10:58] Good luck! [16:19:05] (03CR) 10Ahmon Dancy: Avoid copying L10N files from/to /tmp on deploy server (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/738453 (https://phabricator.wikimedia.org/T295304) (owner: 10Ahmon Dancy) [16:20:10] (03PS7) 10Ahmon Dancy: Avoid copying L10N files from/to /tmp on deploy server [tools/scap] - 10https://gerrit.wikimedia.org/r/738453 (https://phabricator.wikimedia.org/T295304) [16:23:47] (03PS8) 10Ahmon Dancy: Avoid copying L10N files to/from /tmp on deploy server [tools/scap] - 10https://gerrit.wikimedia.org/r/738453 (https://phabricator.wikimedia.org/T295304) [16:24:02] (03CR) 10Ahmon Dancy: [C: 03+1] "Ready" [tools/scap] - 10https://gerrit.wikimedia.org/r/738453 (https://phabricator.wikimedia.org/T295304) (owner: 10Ahmon Dancy) [16:33:48] (03CR) 10Hashar: [C: 03+2] "Jobs updated!" [integration/config] - 10https://gerrit.wikimedia.org/r/738920 (owner: 10Hashar) [16:35:07] (03CR) 10Hashar: [C: 03+2] "I got the Jenkins plugin updated via T194096. Jobs updated!" [integration/config] - 10https://gerrit.wikimedia.org/r/738911 (https://phabricator.wikimedia.org/T194096) (owner: 10Hashar) [16:36:39] 10Release-Engineering-Team (Radar), 10GitLab, 10Security-Team, 10serviceops, 10SecTeam-Processed: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10sbassett) [16:36:41] (03Merged) 10jenkins-bot: Upgrade jjb to 3.6.0 [integration/config] - 10https://gerrit.wikimedia.org/r/738920 (owner: 10Hashar) [16:37:16] !log Upgraded Jenkins Job Builder to 3.6.0 with update for the ext-mail plugin [16:37:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:37:59] !log Upgraded Jenkins Job Builder to 3.10.0 which update support for the xunit plugin (got it updated to 3.x) # T194096 [16:38:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:38:01] T194096: Switch back to upstream jenkins xunit plugin after PHPUnit fix is released - https://phabricator.wikimedia.org/T194096 [16:39:05] (03Merged) 10jenkins-bot: Upgrade jjb to 3.10.0 [integration/config] - 10https://gerrit.wikimedia.org/r/738911 (https://phabricator.wikimedia.org/T194096) (owner: 10Hashar) [16:53:22] 10Release-Engineering-Team (Done by Wed 24 Nov 🔥): docker-gc: A tool for partially pruning docker resources - https://phabricator.wikimedia.org/T294034 (10dancy) 05Open→03Resolved a:03dancy [16:53:24] 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10GitLab (CI & Job Runners), 10User-brennen: runner-1002 is out of space - https://phabricator.wikimedia.org/T291221 (10dancy) [16:58:09] 10Project-Admins, 10User-dcaro: Create a few tag projects for task tracking - https://phabricator.wikimedia.org/T295692 (10Aklapper) Which problems are solved by categorizing the task author, in contrast to e.g. setting Priority? I know that the Security team does so via a custom "Affiliation" task field (but... [17:09:41] greg-g: no backups :( [17:10:01] RhinosF1: ? [17:10:06] But I'm assuming it got deleted when he was logged out [17:10:10] greg-g: your Twitter [17:10:35] I wonder why it didn't sync to his account [17:12:05] yeah, he's been logged out before but the local files haven't been deleted like this due to it [17:14:39] I wonder why it wasn't backing up to his account [17:14:43] I'd try emailing them [17:14:52] Dear Microsoft, [17:14:58] Mojang used to be good when it was them [17:15:08] Microsoft maybe less so [17:15:29] But always keep 3 copies of anything important [17:16:50] Reedy: I'm old, I forgot it was now microsoft [17:16:59] Who have to win the worst job site award [17:17:28] I never knew/thought the local files would backup to his MS account? [17:17:58] (this is totally off-topic for here, my son's tablet lost his minecraft worlds) [17:19:10] 10Release-Engineering-Team, 10GitLab: Run docker-gc resource monitor on gitlab runners - https://phabricator.wikimedia.org/T295707 (10dancy) [17:19:15] I thought you could save to Minecraft accounts before [17:19:32] Or maybe it was iCloud automatically [17:19:38] 10Release-Engineering-Team, 10GitLab: Run docker-gc resource monitor on gitlab runners - https://phabricator.wikimedia.org/T295707 (10dancy) [17:19:40] 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10GitLab (CI & Job Runners), 10User-brennen: runner-1002 is out of space - https://phabricator.wikimedia.org/T291221 (10dancy) [17:19:49] It definitely shouldn't just wipe [17:20:20] 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10GitLab: Run docker-gc resource monitor on gitlab runners - https://phabricator.wikimedia.org/T295707 (10dancy) [17:21:25] 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10GitLab (CI & Job Runners): Run docker-gc resource monitor on gitlab runners - https://phabricator.wikimedia.org/T295707 (10dancy) [17:22:15] 10Release-Engineering-Team (Doing), 10Security Team AppSec, 10Security-Team, 10GitLab (CI & Job Runners), and 3 others: Migrate existing proof-of-concept node ci templates to slim node wm node docker images - https://phabricator.wikimedia.org/T294306 (10sbassett) >>! In T294306#7496841, @thcipriani wrote:... [17:25:40] 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10GitLab (CI & Job Runners): Periodically run docker-gc on gitlab runners - https://phabricator.wikimedia.org/T295709 (10dancy) [17:25:57] 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10GitLab (CI & Job Runners): Periodically run docker-gc on gitlab runners - https://phabricator.wikimedia.org/T295709 (10dancy) [17:26:11] 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10GitLab (CI & Job Runners): Periodically run docker-gc on gitlab runners - https://phabricator.wikimedia.org/T295709 (10dancy) [17:32:35] 10Project-Admins, 10User-dcaro: Create a few tag projects for task tracking - https://phabricator.wikimedia.org/T295692 (10dcaro) > Which problems are solved by categorizing the task author, in contrast to e.g. setting Priority? It's not so much the author as it's the origin, if there's a rise in tasks opened... [17:48:07] 10Project-Admins, 10User-dcaro: Create a few tag projects for task tracking - https://phabricator.wikimedia.org/T295692 (10bd808) >>! In T295692#7504462, @dcaro wrote: >> Generally speaking, I'd like to avoid creating global projects tags for individual personal use, so I wonder if there are better approaches.... [17:53:55] 10Project-Admins, 10User-dcaro: Create a few tag projects for task tracking - https://phabricator.wikimedia.org/T295692 (10dcaro) > https://phabricator.wikimedia.org/flag/ would be the closest thing I know of to personal tags. I do not use flags much currently, but in the past I have used them to track groups... [17:55:43] 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10Release, 10Train Deployments: 1.38.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T293950 (10RhinosF1) [17:56:49] First blocker and it's only Monday :( [18:00:07] honestly i'd often rather get a blocker before anything's deployed than have to dance with it afterwards. [18:00:15] RhinosF1: a Monday find is sooo much better than a Wednesday find [18:00:28] or a Friday afternoon find [18:00:33] bd808: no blockers would be better [18:00:34] 'course, sometimes the monday find presages a whole series of finds later in the week [18:01:28] RhinosF1: that's easy! just stop looking for them [18:02:00] bd808: then a user does [18:02:16] no more code, done [18:02:23] if you mean that code with zero bugs is the goal, I would counter that that is mathematically unprovable and unrealistic. All code has bugs unless it is trivial code. [18:03:04] Finding bugs before they affect the wikis is a sign of a healthy process [18:05:48] bd808: <3 [18:06:28] (not meaning to say that process cannot be improved) [18:16:49] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T293952 (10thcipriani) a:05dancy→03brennen [18:17:21] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10thcipriani) p:05Triage→03Medium a:03dancy [18:28:56] (03CR) 10Dduvall: [C: 03+2] Hide ssh-keyscan/ssh-keygen noise [tools/train-dev] - 10https://gerrit.wikimedia.org/r/737779 (owner: 10Ahmon Dancy) [18:29:26] (03Merged) 10jenkins-bot: Hide ssh-keyscan/ssh-keygen noise [tools/train-dev] - 10https://gerrit.wikimedia.org/r/737779 (owner: 10Ahmon Dancy) [18:38:04] (03CR) 10Dduvall: lint.py: Supply useful output on lint fail (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/738024 (https://phabricator.wikimedia.org/T272760) (owner: 10Ahmon Dancy) [18:42:10] (03CR) 10Thcipriani: [C: 03+2] Avoid copying L10N files to/from /tmp on deploy server (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/738453 (https://phabricator.wikimedia.org/T295304) (owner: 10Ahmon Dancy) [18:43:10] (03Merged) 10jenkins-bot: Avoid copying L10N files to/from /tmp on deploy server [tools/scap] - 10https://gerrit.wikimedia.org/r/738453 (https://phabricator.wikimedia.org/T295304) (owner: 10Ahmon Dancy) [18:53:25] (03PS3) 10Ahmon Dancy: lint.py: Supply useful output on lint fail [tools/scap] - 10https://gerrit.wikimedia.org/r/738024 (https://phabricator.wikimedia.org/T272760) [18:53:47] (03CR) 10Ahmon Dancy: lint.py: Supply useful output on lint fail (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/738024 (https://phabricator.wikimedia.org/T272760) (owner: 10Ahmon Dancy) [18:55:50] (03CR) 10Dduvall: [C: 03+2] lint.py: Supply useful output on lint fail [tools/scap] - 10https://gerrit.wikimedia.org/r/738024 (https://phabricator.wikimedia.org/T272760) (owner: 10Ahmon Dancy) [18:57:00] (03Merged) 10jenkins-bot: lint.py: Supply useful output on lint fail [tools/scap] - 10https://gerrit.wikimedia.org/r/738024 (https://phabricator.wikimedia.org/T272760) (owner: 10Ahmon Dancy) [19:07:54] 10Release-Engineering-Team (Radar), 10observability, 10Developer Productivity: Add mwversion to php7-fatal-error.php logstash message - https://phabricator.wikimedia.org/T253781 (10Krinkle) 05Open→03Resolved [19:11:05] 10Project-Admins, 10User-dcaro: Create a few tag projects for task tracking - https://phabricator.wikimedia.org/T295692 (10Aklapper) >>! In T295692#7504560, @dcaro wrote: > I see no flags there, is that expected? (maybe I need some permissions or something?) If nothing is flagged: Yes. See https://www.mediawi... [19:11:50] 10Project-Admins, 10User-dcaro: Create tag projects worktype-project, origin-user, origin-alert, origin-team - https://phabricator.wikimedia.org/T295692 (10Aklapper) [19:20:34] 10Project-Admins, 10User-dcaro: Create tag projects worktype-project, origin-user, origin-alert, origin-team - https://phabricator.wikimedia.org/T295692 (10Aklapper) >>! In T295692#7504462, @dcaro wrote: >> Which problems are solved by categorizing the task author, in contrast to e.g. setting Priority? > > It... [19:33:39] 10Release-Engineering-Team: beta-build-scap-deb failing - https://phabricator.wikimedia.org/T295719 (10dancy) [19:41:01] (03PS1) 10Ahmon Dancy: beta-build-scap-deb: Temporarily disable Junit data collection [integration/config] - 10https://gerrit.wikimedia.org/r/738991 (https://phabricator.wikimedia.org/T295719) [19:41:23] (03CR) 10Ahmon Dancy: [C: 03+2] "Already deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/738991 (https://phabricator.wikimedia.org/T295719) (owner: 10Ahmon Dancy) [19:43:10] (03Merged) 10jenkins-bot: beta-build-scap-deb: Temporarily disable Junit data collection [integration/config] - 10https://gerrit.wikimedia.org/r/738991 (https://phabricator.wikimedia.org/T295719) (owner: 10Ahmon Dancy) [19:55:35] !log Updated scap in beta [19:55:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:58:13] Project beta-scap-sync-world build #27478: 04FAILURE in 1 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/27478/ [19:58:50] ^ That's me. [20:02:53] Yippee, build fixed! [20:02:54] Project beta-scap-sync-world build #27479: 09FIXED in 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/27479/ [20:09:12] !log changed mode of deployment-deploy01:/srv/mediawiki-staging/php-master/cache from 2775 to 0777 to match production. [20:09:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:13:42] 10Release-Engineering-Team: beta-build-scap-deb failing - https://phabricator.wikimedia.org/T295719 (10dancy) Possibly related to {T194096} [20:23:44] Project beta-scap-sync-world build #27481: 04FAILURE in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/27481/ [20:24:08] ^ Still me. [20:53:35] !log reverted scap in beta cluster for now. [20:53:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:00:42] 10Phabricator, 10WMF-Legal: Phabricator Macros and other media are to have clear copyright status - https://phabricator.wikimedia.org/T128771 (10Aklapper) 05Open→03Declined Makes sense. Boldly declining. [21:03:55] Yippee, build fixed! [21:03:56] Project beta-scap-sync-world build #27482: 09FIXED in 8 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/27482/ [21:06:40] !log Add dpifke to 'integration' sudo via Horizon (and prune inactive accounts from the same list) [21:06:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:27:57] !log upgrading PHP 7.4 on deployment-mediawiki12 [21:27:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:29:25] Nov 15 21:28:06 deployment-mediawiki12 php[15529]: PHP Warning: PHP Startup: Unable to load dynamic library 'wddx.so' (tried: /usr/lib/php/20190902/wddx.so (/usr/lib/php/20190902/wddx.so: cannot open shared object file: No such file or d [21:29:26] hm [21:30:10] srs bizness api formats [21:30:37] yeah, but it's all gone, guess we're just loading the PHP extension as a historical accident [21:30:43] s/accident/leftover/ [21:34:02] T295725 [21:34:03] T295725: Stop loading wddx PHP extension - https://phabricator.wikimedia.org/T295725 [21:35:36] heh [22:13:58] !log gitlab-test: upgrading to 14.4.2 [22:13:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:25:45] !log gitlab1001: upgrading to 14.4.2 [22:25:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:26:17] I'm setting up a new thumbor server [22:26:20] currently failing at Error: Execution of '/usr/bin/scap deploy-local --repo 3d2png/deploy -D log_json:False' returned 70: [22:26:47] if someone knows off the top of their head what status code 70 is [22:27:09] Just means unhandled error. [22:27:16] no other output? [22:31:19] no, I think puppet masks it, let me run it manually [22:32:20] scap.runcmd.FailedCommand: Command 'git clone --jobs 46 http://deploy1001.eqiad.wmnet/3d2png/deploy/.git /srv/deployment/3d2png/deploy-cache/cache' failed with exit code 128; stderr: [22:32:20] b"Cloning into '/srv/deployment/3d2png/deploy-cache/cache'...\nfatal: unable to access 'http://deploy1001.eqiad.wmnet/3d2png/deploy/.git/': Could not resolve host: deploy1001.eqiad.wmnet\n" [22:32:28] so something needs to be changed to point to deploy1002? [22:32:37] Seems so. [22:33:43] /srv/deployment/3d2png/deploy-cache/.config has 'git_server: deploy1001.eqiad.wmnet' [22:35:35] /etc/scap.cfg has the correct deploy1002 server [22:36:58] I can't find that file on deploy1002. [22:37:32] the deploy-cache file is on thumbor1005, which is where the deploy-local command run by puppet is failing [22:37:50] I see. [22:38:19] https://gerrit.wikimedia.org/g/mediawiki/tools/scap/+/10ae5db4e1911208ed01e6ca2841a71fc6e892e9/scap/config.py#70 [22:38:28] I'm not familiar with the inner works of deploy-local yet but I wonder what would happen if you moved the deploy-cache dir out of the way.' [22:38:54] Does thumbor1005 have an /etc/scap.cfg ? [22:39:24] yep, that's what I looked at when I said /etc/scap.cfg has the correct deploy1002 server [22:39:31] gotcha. [22:39:55] I edited the .config file right before you suggested moving the entire directory out of the way and it seems to have worked [22:40:09] legoktm: known issue :( but it's in the cache [22:40:18] replacing the hostname works too [22:40:29] yeah, that's what I ended up doing [22:40:37] Is there a scap bug filed for this? [22:41:29] https://phabricator.wikimedia.org/T197470 [22:41:48] thx [22:43:25] the only place I can find deploy1001 still referenced is in scap itself [22:44:09] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T293954 (10thcipriani) p:05Triage→03Medium a:03hashar [22:44:29] but I see that /etc/scap.cfg was definitely created before the deploy-local command ran [22:44:32] it happened with tin->deploy1001 and then deploy1001 to deploy1002 [22:44:38] so it's not a new thing [22:45:25] also see https://gerrit.wikimedia.org/r/c/operations/puppet/+/670784 maybe [22:46:01] dancy: ^ that was once an idea to fix it but did not work apparently [22:46:39] Alright. I'll bring this up w/ the team. Bad and confusing behavior. [22:46:39] 10Release-Engineering-Team (Seen), 10Scap, 10SRE: find a way to systematically update the deployment server name across all repos - https://phabricator.wikimedia.org/T197470 (10Legoktm) Hit this today when setting up new thumbor servers. What I don't really understand is where it's getting deploy1001 these d... [22:46:49] on the ticket I suggested using `deployment.eqiad.wmnet` [22:46:53] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T293957 (10thcipriani) p:05Triage→03Medium a:03mmodell [22:47:19] Using a logical hostname is a very good idea. [22:48:07] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T293958 (10thcipriani) p:05Triage→03Medium a:03dduvall [22:48:51] worst kind of notification to get :( [22:49:00] Sorry man [22:49:43] :) [22:52:44] 10Release-Engineering-Team (Seen), 10Scap, 10SRE: find a way to systematically update the deployment server name across all repos - https://phabricator.wikimedia.org/T197470 (10bd808) >>! In T197470#7505335, @Legoktm wrote: > Hit this today when setting up new thumbor servers. What I don't really understand... [23:17:59] 10Release-Engineering-Team (Radar), 10GitLab, 10Observability-Logging, 10Patch-For-Review: Gitlab Sidekiq mapper parsing exceptions since 2021-11-15@1825 - https://phabricator.wikimedia.org/T295731 (10brennen) [23:21:14] 10Release-Engineering-Team (Radar), 10Observability-Logging, 10GitLab (Infrastructure), 10Patch-For-Review: Gitlab Sidekiq mapper parsing exceptions since 2021-11-15@1825 - https://phabricator.wikimedia.org/T295731 (10brennen)