[04:47:28] 10Phabricator Antivandalism Extension, 10DBA, 10Patch-For-Review, 10User-notice: Switchover m3 master (db1183 -> db1159) - https://phabricator.wikimedia.org/T316506 (10Marostegui) [04:49:02] 10Phabricator Antivandalism Extension, 10DBA, 10Patch-For-Review, 10User-notice: Switchover m3 master (db1183 -> db1159) - https://phabricator.wikimedia.org/T316506 (10Marostegui) [05:01:47] 10Phabricator Antivandalism Extension, 10DBA, 10Patch-For-Review, 10User-notice: Switchover m3 master (db1183 -> db1159) - https://phabricator.wikimedia.org/T316506 (10Marostegui) [05:02:43] 10Phabricator Antivandalism Extension, 10DBA, 10Patch-For-Review, 10User-notice: Switchover m3 master (db1183 -> db1159) - https://phabricator.wikimedia.org/T316506 (10Marostegui) [05:02:53] 10Phabricator Antivandalism Extension, 10DBA, 10Patch-For-Review, 10User-notice: Switchover m3 master (db1183 -> db1159) - https://phabricator.wikimedia.org/T316506 (10Marostegui) This was all done fine [05:03:04] 10Phabricator Antivandalism Extension, 10DBA, 10Patch-For-Review, 10User-notice: Switchover m3 master (db1183 -> db1159) - https://phabricator.wikimedia.org/T316506 (10Marostegui) [05:06:43] 10Phabricator Antivandalism Extension, 10DBA, 10Patch-For-Review, 10User-notice: Switchover m3 master (db1183 -> db1159) - https://phabricator.wikimedia.org/T316506 (10Marostegui) 05Open→03Resolved [08:56:20] 10Continuous-Integration-Config, 10Wiki-Loves-Monuments: Don’t test heritage under PHP 8 - https://phabricator.wikimedia.org/T316462 (10Lokal_Profil) 05Open→03Resolved Thanks! [09:01:32] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Ladsgroup) >>! In T706#8195278, @KHernandez-WMF wrote: > Hi, @Ladsgroup I'd like to join the admin group. I am working with the Cloud Services on the [[ https://phabricator.wikimedia... [09:04:34] 10Release-Engineering-Team, 10Scap: Strictly lock scap requirements dependencies - https://phabricator.wikimedia.org/T316755 (10hashar) [09:31:54] 10Release-Engineering-Team, 10Infrastructure-Foundations, 10SRE-tools: Investigate sharing releng common python code to pywmflib - https://phabricator.wikimedia.org/T316757 (10hashar) [09:37:03] 10Phabricator: Merge multiple Phab accounts made in error into single ITS account - https://phabricator.wikimedia.org/T316721 (10Peachey88) @VirginiaPoundstone It's not possible to merge multiple Phabricator accounts, The options are delete or disable so they can't be used. Phabricator Accounts: @Vpoundstone ha... [09:47:17] 10Phabricator: Merge multiple Phab accounts made in error into single ITS account - https://phabricator.wikimedia.org/T316721 (10Peachey88) I've also looked at the relevant WikiMedia SUL accounts: https://meta.wikimedia.org/wiki/Special:CentralAuth/VPoundstone-WMF https://meta.wikimedia.org/wiki/Special:Central... [11:04:11] GitLab needs a short break at 11:30 UTC (in 30 minutes). Should be back after around 5 minute [11:21:51] (03CR) 10Jakob: "This change is ready for review." [integration/config] - 10https://gerrit.wikimedia.org/r/828501 (https://phabricator.wikimedia.org/T316245) (owner: 10Jakob) [11:44:32] (03CR) 10Ladsgroup: [C: 03+1] "LGTM. I'll deploy it if hashar doesn't object to it in the next day or two." [integration/config] - 10https://gerrit.wikimedia.org/r/828501 (https://phabricator.wikimedia.org/T316245) (owner: 10Jakob) [11:46:58] 10Phabricator: Merge multiple Phab accounts made in error into single ITS account - https://phabricator.wikimedia.org/T316721 (10Aklapper) Hej hej @VirginiaPoundstone! No problem, let's sort this out together. For context, there are three different account types involved in the next lines: * [SUL accounts](http... [12:23:34] Hi all! I need to apply 2 schema changes on beta for https://phabricator.wikimedia.org/T308738 and https://phabricator.wikimedia.org/T312870 but I'm not sure what the process looks like. Not at the DB level, but whether I should wait for a deployment window, log anything in any channel, pair with someone else, etc. Could someone please help me? [12:25:51] Daimona_: beta runs mostly automated via update.php [12:26:02] So if it's not in update.php, just do it and log in here [12:26:13] Not in this case, those tables are in wikishared DB and not modified by update.php. [12:26:16] That's all that's normally done [12:26:24] Daimona_: just do it and log them [12:26:38] That's all that's been done for the CA updates [12:26:53] Got it, thanks. And I should only log them in this channel, right? [12:27:08] Daimona_: here is normally used SAL [12:27:33] For actual stuff [12:27:50] There is a deployment prep SAL but that's mainly VM creating and adding members [12:28:19] Ok... Is there another channel, then? [12:28:49] Also, while we're at it, could you or someone else confirm the host that I should SSH to? I haven't done it in a while and my config may be outdated [12:31:27] Daimona: #wikimedia-cloud has the project SAL. And deployment-mediawiki12 is the active mediawiki host. I'm going to poke zabe or TheresNoTime though to check my sanity [12:32:25] Thank you! I'll wait for someone else to confirm then, I certainly don't want to break beta (not on purpose, at least...) [12:34:15] Daimona: beta is always broken anyway [12:34:39] And TheresNoTime gets an buzz anyway as the only person that monitors it if you do! [12:35:04] > beta is always broken anyway [12:35:30] I guess I woke you up [12:36:00] was just having lunch :p [12:36:34] Poor beta [12:38:38] 10Release-Engineering-Team (Seen), 10MediaWiki-extensions-CodeReview: CodeReview extension archive scope definition - https://phabricator.wikimedia.org/T216219 (10Aklapper) 05Open→03Declined T116948 happened. Declining. [12:40:00] for SAL, tends to get logged here [12:42:59] 10Release-Engineering-Team, 10Scap: Strictly lock scap requirements dependencies - https://phabricator.wikimedia.org/T316755 (10Lucas_Werkmeister_WMDE) > We would probably want to add the packages checksum as well, though there is no easy way to maintain them short of manually generating the checksum and addin... [12:44:19] Daimona: as for when, I'd suggest *not* doing it during the upcoming backport deployment window if that's okay [12:45:37] deployment-db07 is the primary db host [12:46:04] just apply the db change there and let it propagate through replication [12:47:56] and yeah just log it here [12:51:09] Alright, so let me quickly sum up, just to make sure that I got it right: the host is deployment-db07. I should NOT do it during a b/c window, and I should log it here. Did I get it right? :) [12:52:05] sounds good to me :) [12:53:55] Noice, thank you! [12:54:59] Thanks everyone! [13:23:08] (03CR) 10Hashar: [C: 03+2] "Nice! I am happy to see API testing being adopted ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/828501 (https://phabricator.wikimedia.org/T316245) (owner: 10Jakob) [13:26:13] (03Merged) 10jenkins-bot: Add experimental composer apitests job to Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/828501 (https://phabricator.wikimedia.org/T316245) (owner: 10Jakob) [13:31:39] (03CR) 10Hashar: [C: 03+2] "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/828501 (https://phabricator.wikimedia.org/T316245) (owner: 10Jakob) [13:33:25] (03CR) 10Hashar: [C: 03+2] "Thank you, I have updated all the affected Jenkins jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/828026 (owner: 10Gehel) [13:34:34] (03CR) 10Hashar: "This change is ready for review." [tools/scap] - 10https://gerrit.wikimedia.org/r/805909 (owner: 10Hashar) [13:36:10] (03Merged) 10jenkins-bot: jjb: Drop verbose flag to Sonar for all Java jobs [integration/config] - 10https://gerrit.wikimedia.org/r/828026 (owner: 10Gehel) [13:45:50] 10Project-Admins: Create component for product infrastructure orphaned project tickets - https://phabricator.wikimedia.org/T311246 (10Aklapper) ping? anyone? [14:09:52] 10Project-Admins: Create project tag for Apple Silicon support: "ARM support" - https://phabricator.wikimedia.org/T315424 (10hashar) Thanks @Jdforrester-WMF ! [14:12:17] 10Project-Admins, 10ARM support: Create project tag for Apple Silicon support: "ARM support" - https://phabricator.wikimedia.org/T315424 (10hashar) [14:16:58] (03PS1) 10Jforrester: dockerfiles: [quibble-fresnel] Switch from php72 to php74 base image [integration/config] - 10https://gerrit.wikimedia.org/r/828541 (https://phabricator.wikimedia.org/T316525) [14:17:00] (03PS1) 10Jforrester: jjb: [mediawiki-fresnel-patch-docker] Switch from php72 to php74 base image [integration/config] - 10https://gerrit.wikimedia.org/r/828542 (https://phabricator.wikimedia.org/T316525) [14:41:07] 10Project-Admins: Create project tag for MediaWiki-extensions-Shibboleth - https://phabricator.wikimedia.org/T309719 (10Physikerwelt) @CCicalese_WMF (CC @Stegmujo) I am wondering if https://www.mediawiki.org/wiki/Extension:Shibboleth is actively maintained. I was unable to install it since it seems to be incomp... [14:41:20] 10Project-Admins, 10NFDI: Create project tag for MediaWiki-extensions-Shibboleth - https://phabricator.wikimedia.org/T309719 (10Physikerwelt) [14:51:30] MichaelG_WMDE & Lucas_WMDE: I will merge the Quibble patch for running addSite.php ( https://gerrit.wikimedia.org/r/c/integration/quibble/+/820744/ ) [14:51:43] but I then need to cut a new release of Quibble, rebuild all CI images and switch the jobs [14:51:58] I will most probably do that tomorrow [14:55:55] sounds good, thanks a lot! [14:56:05] * Lucas_WMDE will not be here tomorrow or friday though [14:57:11] @hashar Thank you! This is currently "only" for tests that are being added in their own dedicated patch, after the (unit-tested) functionality. So while it is good to not delay that for too long, it is not actively blocking us and so that urgent either. Feel free to fit it into your schedule as it works best for you. [14:57:27] re the comment on the change, is there a standard way to have a “bootstrap for CI / api-testing” script? [14:57:41] the only thing I know in Wikibase is a ci.php file, which would still run on each request [14:58:02] and making wiki changes in a “config” file felt evil to me [14:58:17] * MichaelG_WMDE will be arround tomorrow, though will have to spend time with our new hire on their first day [14:58:29] hm, I suppose we could’ve added something to the api-testing npm script 🤔 [14:59:50] MichaelG_WMDE: Lucas_WMDE I don't think there is any bootstrap feature [14:59:55] well maybe there is one [15:00:03] regardless you wrote all the code in Quibble and that will work :] [15:00:30] ok :) [15:00:31] I imagine the api testing code could potentially shell out to a maintenance script when setting up the test suite [15:00:36] but to be fair, I ahven't looked at it [15:00:51] and I don't even know how complicately broken running the script would be [15:01:01] I am writing the release notes ;) [15:11:05] Lucas_WMDE: MichaelG_WMDE: I crafted the release notes at https://gerrit.wikimedia.org/r/828561 [15:17:18] @hashar looks good to me, thanks :) [15:23:37] hashar: OK for me to sling out the fresnel upgrade to php74 first? [15:23:50] (03CR) 10Jforrester: [C: 03+2] dockerfiles: [quibble-fresnel] Switch from php72 to php74 base image [integration/config] - 10https://gerrit.wikimedia.org/r/828541 (https://phabricator.wikimedia.org/T316525) (owner: 10Jforrester) [15:24:31] James_F: for sure! it only runs on mediawiki/core and is non voting [15:24:37] thanks for all theupdates [15:26:11] (03Merged) 10jenkins-bot: dockerfiles: [quibble-fresnel] Switch from php72 to php74 base image [integration/config] - 10https://gerrit.wikimedia.org/r/828541 (https://phabricator.wikimedia.org/T316525) (owner: 10Jforrester) [15:26:55] Always. [15:27:43] !log Docker: Building and publishing quibble-fresnel image based on php74 for T316525 [15:27:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:27:48] T316525: Move Fresnel from PHP 7.2.x to 7.4.x - https://phabricator.wikimedia.org/T316525 [15:30:36] (03PS2) 10Jforrester: jjb: [mediawiki-fresnel-patch-docker] Switch from php72 to php74 base image [integration/config] - 10https://gerrit.wikimedia.org/r/828542 (https://phabricator.wikimedia.org/T316525) [15:37:17] (03CR) 10Jforrester: [C: 03+2] "Deployed, tested, all looks good." [integration/config] - 10https://gerrit.wikimedia.org/r/828542 (https://phabricator.wikimedia.org/T316525) (owner: 10Jforrester) [15:39:42] (03Merged) 10jenkins-bot: jjb: [mediawiki-fresnel-patch-docker] Switch from php72 to php74 base image [integration/config] - 10https://gerrit.wikimedia.org/r/828542 (https://phabricator.wikimedia.org/T316525) (owner: 10Jforrester) [15:40:17] 10Continuous-Integration-Config, 10Fresnel, 10Performance-Team, 10Patch-For-Review: Move Fresnel from PHP 7.2.x to 7.4.x - https://phabricator.wikimedia.org/T316525 (10Jdforrester-WMF) 05Open→03Resolved a:05Krinkle→03Jdforrester-WMF Deployed and tested; looks good ([[https://integration.wikimedia.o... [15:41:24] 10Continuous-Integration-Config, 10Fresnel, 10Performance-Team, 10Patch-For-Review: Move Fresnel from PHP 7.2.x to 7.4.x - https://phabricator.wikimedia.org/T316525 (10Krinkle) Thanks @Jdforrester-WMF! [15:59:14] (03PS1) 10Hashar: dockerfiles: update to Quibble 1.4.6 [integration/config] - 10https://gerrit.wikimedia.org/r/828574 (https://phabricator.wikimedia.org/T305525) [15:59:20] Woo. [15:59:36] (03CR) 10Hashar: "I haven't tagged Quibble 1.4.6 yet" [integration/config] - 10https://gerrit.wikimedia.org/r/828574 (https://phabricator.wikimedia.org/T305525) (owner: 10Hashar) [16:00:01] James_F: and I did a slight change which is to introduce `ARG QUIBBLE_VERSION=1.4.6` [16:00:10] will make it easier to find the version when we have to bump it [16:00:10] (03CR) 10Jforrester: [C: 03+1] "<3" [integration/config] - 10https://gerrit.wikimedia.org/r/828574 (https://phabricator.wikimedia.org/T305525) (owner: 10Hashar) [16:03:12] * hashar waits for CI [16:05:03] 10Release-Engineering-Team (Bonus Level 🕹️), 10Scap, 10Patch-For-Review: Refactor Scap to use TimeoutLock as the sole locking mechanism - https://phabricator.wikimedia.org/T315531 (10jeena) 05Open→03In progress a:03jeena [16:09:52] FYI: I'm gonna apply the schema changes on beta for https://phabricator.wikimedia.org/T312870 and https://phabricator.wikimedia.org/T308738 shortly. [16:10:08] (I will also !log them later) [16:11:31] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team, 10Upstream: Jobs started failing on https://releases-jenkins.wikimedia.org on 2022-08-21 - https://phabricator.wikimedia.org/T315897 (10hashar) a:03hashar [16:18:34] !log Tag Quibble 1.4.6 @ 8828487d0 # T305525 T314586 [16:18:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:18:38] T314586: Enable Wikibase CI wikis to link to themselves - https://phabricator.wikimedia.org/T314586 [16:18:38] T305525: Experiment with using pnpm in CI to speed up npm install process - https://phabricator.wikimedia.org/T305525 [16:21:18] !log Applying schema change to the wikishared DB on beta for the CampaignEvents extension (1/2) # T312870 [16:21:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:21:22] T312870: Add timestamps to ce_organizers table [V1] - https://phabricator.wikimedia.org/T312870 [16:22:54] James_F: may I build the quibble 1.4.6 images ? https://gerrit.wikimedia.org/r/c/integration/config/+/828574/ [16:23:47] Daimona: thank you for the beta cluster maintenance [16:24:23] (03CR) 10Hashar: [C: 03+2] dockerfiles: update to Quibble 1.4.6 [integration/config] - 10https://gerrit.wikimedia.org/r/828574 (https://phabricator.wikimedia.org/T305525) (owner: 10Hashar) [16:24:26] doing [16:24:51] hashar: Oh, don't thank me, it's not really maintenance :) Just applying some work-related schema changes manually. It's certainly not thanks to me if beta still hasn't caught fire [16:25:33] :]] [16:26:24] (03Merged) 10jenkins-bot: dockerfiles: update to Quibble 1.4.6 [integration/config] - 10https://gerrit.wikimedia.org/r/828574 (https://phabricator.wikimedia.org/T305525) (owner: 10Hashar) [16:27:35] Can someone please link me the status page for the job that updates beta to latest master? I can never find it... [16:28:02] https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/ [16:28:16] Thank you! And let me bookmark it :) [16:34:16] Daimona: see also https://integration.wikimedia.org/ci/view/Beta/ which is a view to all the beta jobs [16:34:37] ...And let me bookmark that one as well :P [16:35:15] it also filter the build executors on the left and only shows the deployment-deploy Jenkins agent so immediately see whether a job is running or blocked [16:35:17] ;) [16:36:48] Yeah, it's definitely useful. Certainly better than my previous approach of just waiting unless someone posted the link :D [16:37:04] !log Applying schema change to the wikishared DB on beta for the CampaignEvents extension (2/2) # T312870 [16:37:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:37:06] T312870: Add timestamps to ce_organizers table [V1] - https://phabricator.wikimedia.org/T312870 [16:39:06] !log Applying schema change to the wikishared DB on beta for the CampaignEvents extension # T308738 [16:39:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:39:08] T308738: Ensure that all tables have all the necessary indexes - https://phabricator.wikimedia.org/T308738 [16:42:16] And now I'm done. [16:46:06] (03CR) 10Hashar: [C: 03+2] "I think that will also have the custom php7.4 packages we have made for T316601#8201982" [integration/config] - 10https://gerrit.wikimedia.org/r/828574 (https://phabricator.wikimedia.org/T305525) (owner: 10Hashar) [16:54:11] hashar: Sorry, yes, go for it. [16:54:18] Was in meetings. [16:57:42] hashar: quibble-buster-php74:1.4.5-s3 has 'PHP 7.4.30 (cli) (built: Aug 31 2022 09:45:57)' which is our special version already (-s2 has 'PHP 7.4.30 (cli) (built: Jul 5 2022 07:16:18)'). [16:57:52] I think? [17:22:53] (03CR) 10Hashar: [C: 03+2] "Successfully published image docker-registry.discovery.wmnet/releng/quibble-buster-php81:1.4.6" [integration/config] - 10https://gerrit.wikimedia.org/r/828574 (https://phabricator.wikimedia.org/T305525) (owner: 10Hashar) [17:23:17] James_F no idea, I am having dinner. I had docker-pkg running in the background [17:23:27] I will check tomorrow! [17:30:06] (03PS1) 10Jforrester: jjb: Switch quibble jobs to 1.4.6 [integration/config] - 10https://gerrit.wikimedia.org/r/828611 (https://phabricator.wikimedia.org/T305525) [17:36:24] 10GitLab (Project Migration), 10Release-Engineering-Team (The Decommission Mission 💀), 10Striker, 10Tools: Figure out workflow for programatically adding GitLab users - https://phabricator.wikimedia.org/T313366 (10bd808) See https://gitlab.wikimedia.org/bd808/pre-account-creation-invite-test for another wa... [18:12:34] (03PS1) 10Ahmon Dancy: automatic-branch-cut: Set PYTHONUNBUFFERED=1 when running branch.py [tools/release] - 10https://gerrit.wikimedia.org/r/828617 [18:15:07] (03PS1) 10Dduvall: checks: Define environment variables for current/done/revs dirs [tools/scap] - 10https://gerrit.wikimedia.org/r/828619 (https://phabricator.wikimedia.org/T313259) [18:33:15] (03CR) 10Ahmon Dancy: [C: 03+1] "Looks reasonable to me. Ready for +2?" [tools/scap] - 10https://gerrit.wikimedia.org/r/828619 (https://phabricator.wikimedia.org/T313259) (owner: 10Dduvall) [18:39:02] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10SRE-OnFire, 10Wikimedia-Incident: Beta cluster Error: 502, Next Hop Connection Failed - https://phabricator.wikimedia.org/T315350 (10Zabe) [18:47:28] (03CR) 10Thcipriani: [C: 03+2] "TIL" [tools/release] - 10https://gerrit.wikimedia.org/r/828617 (owner: 10Ahmon Dancy) [18:48:14] (03Merged) 10jenkins-bot: automatic-branch-cut: Set PYTHONUNBUFFERED=1 when running branch.py [tools/release] - 10https://gerrit.wikimedia.org/r/828617 (owner: 10Ahmon Dancy) [18:51:10] (03CR) 10Thcipriani: [C: 03+2] scap prep: Detect missing .gitmodules in train branches [tools/scap] - 10https://gerrit.wikimedia.org/r/828095 (owner: 10Ahmon Dancy) [18:55:35] (03Merged) 10jenkins-bot: scap prep: Detect missing .gitmodules in train branches [tools/scap] - 10https://gerrit.wikimedia.org/r/828095 (owner: 10Ahmon Dancy) [18:57:09] (03CR) 10Hashar: [C: 03+1] jjb: Switch quibble jobs to 1.4.6 [integration/config] - 10https://gerrit.wikimedia.org/r/828611 (https://phabricator.wikimedia.org/T305525) (owner: 10Jforrester) [19:00:38] (03CR) 10Dduvall: checks: Define environment variables for current/done/revs dirs (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/828619 (https://phabricator.wikimedia.org/T313259) (owner: 10Dduvall) [19:12:14] 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10serviceops, 10serviceops-collab, and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Dzahn) gitlab2003 before reverting the partman change: ` Filesystem Size Used Avail Use% Mounted o... [19:16:36] 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10serviceops, 10serviceops-collab, and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin2002 for host gitlab2003.wikimedia.org with OS bullseye [19:16:54] reimaging gitlab2003 [19:18:25] 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10serviceops, 10serviceops-collab, and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin2002 for host gitlab2003.wikimedia.org with OS bullseye exe... [19:19:10] reimaging failed again [19:21:40] 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10serviceops, 10serviceops-collab, and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin2002 for host gitlab2003.wikimedia.org with OS bullseye [19:22:09] trying again, maybe it was me.. now needed to pretend it's a --new host [19:24:44] (03CR) 10Ahmon Dancy: [C: 03+2] checks: Define environment variables for current/done/revs dirs [tools/scap] - 10https://gerrit.wikimedia.org/r/828619 (https://phabricator.wikimedia.org/T313259) (owner: 10Dduvall) [19:28:08] dancy: thanks! [19:28:23] np [19:29:09] (03Merged) 10jenkins-bot: checks: Define environment variables for current/done/revs dirs [tools/scap] - 10https://gerrit.wikimedia.org/r/828619 (https://phabricator.wikimedia.org/T313259) (owner: 10Dduvall) [19:35:19] James_F: I have checked the php versions in quibble-buster-php74 [19:35:38] 1.4.5-s3 has PHP 7.4.30 (cli) (built: Jul 5 2022 07:16:18) ( NTS ) [19:35:52] which comes from the package 1:7.4.30-3+0~20220627.69+debian10~1.gbpf2b381+wmf1+buster3 [19:36:16] the newish 1.4.6 has PHP 7.4.30 (cli) (built: Aug 31 2022 09:45:57) ( NTS ) [19:36:25] Hmm. Not locally. I guess it rebuilt them here. [19:36:27] which comes from the package 1:7.4.30-3+0~20220627.69+debian10~1.gbpf2b381+wmf1+buster3.1 [19:36:34] Yeah. [19:36:44] success [19:37:05] in theory `docker-pkg` should try to pull an image from the registry instead of rebuilding it locally [19:37:09] but who knows ;) [19:38:38] will switch them tomorrow [19:56:49] 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10serviceops, 10serviceops-collab, and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin2002 for host gitlab2003.wikimedia.org with OS bullseye com... [20:00:53] 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10serviceops, 10serviceops-collab, and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Dzahn) gitlab2003 has been reimaged. ECDSA key fingerprint is SHA256:hdh4RRTigMf9Q0+fqMKk8e35g0wVUhceRYYlEPLmgf8. now it looks like this... [20:23:36] 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10serviceops, 10serviceops-collab, and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin2002 for host gitlab2003.wikimedia.org with OS bullseye [20:58:02] 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10serviceops, 10serviceops-collab, and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin2002 for host gitlab2003.wikimedia.org with OS bullseye com... [21:01:35] 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10serviceops, 10serviceops-collab, and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Dzahn) puppet had not run on apt* yet. I had to repeat the reimaging. But now it looks as intended: ` [gitlab2003:~] $ df -h Filesystem... [21:05:54] 10Phabricator: Merge multiple Phab accounts made in error into single ITS account - https://phabricator.wikimedia.org/T316721 (10VirginiaPoundstone) @Aklapper, thank you. Your help is greatly appreciated. The onboarding instructions were borrowed from multiple other teams and there was duplication, which migh... [21:33:12] hey guys, so the new train-presync.service appears to have an issue yet to be fixed [21:33:15] CRITICAL - degraded: The following units failed: train-presync.service [21:33:26] I'll make a ticket and paste errors [21:40:22] (03CR) 10Jeena Huneidi: "Any ideas on the test failure?" [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi) [21:40:55] 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10Scap, 10Patch-For-Review: Automated Tuesday Train via a timer - https://phabricator.wikimedia.org/T310395 (10Dzahn) The timer/service added to `deploy1002` recently failed and alerted in Icinga as "`CRITICAL - degraded: The following units failed: train... [21:40:57] added https://phabricator.wikimedia.org/T310395#8204411 [21:42:08] 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10Scap, 10Patch-For-Review: Automated Tuesday Train via a timer - https://phabricator.wikimedia.org/T310395 (10Dzahn) Should email notifications be added (via the systemd timer itself if not icinga?) for releng-team? [21:45:41] (03CR) 10CI reject: [V: 04-1] Replace Lock with TimeoutLock [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi) [21:47:43] the branch commit did not merge due to a CI issue: https://phabricator.wikimedia.org/T314188#8197017, thus the timer failed [21:51:53] zabe: thanks, let's add that link [21:52:46] done [21:53:32] 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10Scap, 10Patch-For-Review: Automated Tuesday Train via a timer - https://phabricator.wikimedia.org/T310395 (10Dzahn) as @Zabe points out this is T314188#8197017 [22:45:58] 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10Scap, 10Patch-For-Review: Automated Tuesday Train via a timer - https://phabricator.wikimedia.org/T310395 (10dancy) Hi @Dzahn Thanks for checking in. The service is configured for email: ` systemd::timer::job { 'train-presync': ensur... [22:53:18] (03CR) 10Ahmon Dancy: Replace Lock with TimeoutLock (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi) [22:55:07] 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10Scap, 10Patch-For-Review: Automated Tuesday Train via a timer - https://phabricator.wikimedia.org/T310395 (10Dzahn) @dancy Ah, you are right, it's already configured. Thanks for the extra details. Sounds good :) [23:29:35] (03CR) 10Ahmon Dancy: Replace Lock with TimeoutLock (037 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi) [23:41:23] !log deleted shutoff deployment-restbase03 [23:41:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:57:44] 10Beta-Cluster-Infrastructure, 10SRE-swift-storage: Upgrade deployment-prep Swift cluster to Debian Buster or newer - https://phabricator.wikimedia.org/T298253 (10Zabe)