[00:13:32] (03PS1) 10Jeena Huneidi: scap backport: Add commit subject to confirmation [tools/scap] - 10https://gerrit.wikimedia.org/r/822190 (https://phabricator.wikimedia.org/T314612) [00:13:54] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review: Scap backport: Display change title when asking for confirmation to backport - https://phabricator.wikimedia.org/T314612 (10jeena) 05Open→03In progress a:03jeena [06:44:09] 10Scap, 10MediaWiki-ResourceLoader, 10MediaWiki-extensions-WikimediaMaintenance, 10Performance-Team, 10Technical-Debt: Remove old refreshMessageBlobs.php script from WikimediaMaintenance - https://phabricator.wikimedia.org/T314947 (10Reedy) [08:43:16] (03CR) 10Jaime Nuche: [C: 03+2] Add --skip-config-validation to create-databases [tools/train-dev] - 10https://gerrit.wikimedia.org/r/822102 (owner: 10Ahmon Dancy) [08:43:42] (03Merged) 10jenkins-bot: Add --skip-config-validation to create-databases [tools/train-dev] - 10https://gerrit.wikimedia.org/r/822102 (owner: 10Ahmon Dancy) [09:16:22] (03PS1) 10Jaime Nuche: serial lock: move check for lock dir to `TimeoutLock` class [tools/scap] - 10https://gerrit.wikimedia.org/r/822330 [09:20:36] (03CR) 10Jaime Nuche: "Tested in train-dev" [tools/scap] - 10https://gerrit.wikimedia.org/r/822330 (owner: 10Jaime Nuche) [09:45:41] Project beta-scap-sync-world build #63614: 04FAILURE in 3 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/63614/ [09:49:40] Hi folks! Not sure if zuul or the CI docker hosts are a little under pressure, but https://integration.wikimedia.org/ci/job/helm-lint/lastBuild/ [09:49:54] I have been waiting ~20 mins for the CI's +2 :( [09:58:42] Project beta-scap-sync-world build #63615: 04STILL FAILING in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/63615/ [10:01:54] jnuche: o/ around by any chance? [10:02:39] Project beta-scap-sync-world build #63616: 04STILL FAILING in 1 min 48 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/63616/ [10:03:41] 10Beta-Cluster-Infrastructure: Beta cluster 503, scap failures - https://phabricator.wikimedia.org/T314988 (10TheresNoTime) [10:03:51] 10Beta-Cluster-Infrastructure: Beta cluster 503, scap failures - https://phabricator.wikimedia.org/T314988 (10TheresNoTime) p:05Triage→03Unbreak! [10:05:54] 10Beta-Cluster-Infrastructure: Beta cluster 503, scap failures - https://phabricator.wikimedia.org/T314988 (10TheresNoTime) [10:05:58] TheresNoTime: is db08 online? [10:06:23] Project beta-code-update-eqiad build #404008: 15ABORTED in 3 min 22 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/404008/ [10:07:21] hm I can't SSH to [10:07:38] TheresNoTime: maybe try a hard reboot then [10:07:52] Although if it's crashed, there's a chance it's doomed anyway [10:09:04] I can [10:09:22] (yeah I can now) [10:09:26] There's db07 if it needs failing over, but I don't want to do anything without more eyes (well tell you too) [10:09:43] TheresNoTime: maybe a crash then, does mysql start up fine [10:10:30] seems to be recover(ing|ed) [10:10:59] TheresNoTime: beta is super slow [10:11:44] TheresNoTime: see if you can spot why it might have crashed [10:12:00] I don't trust a database that randomly crashes to not again [10:12:26] ack, yeah, it's taking *time* to SSH to any hosts [10:12:59] TheresNoTime: that's not good [10:13:02] Maybe cloud wide [10:13:21] 10Beta-Cluster-Infrastructure: Beta cluster 503, scap failures - https://phabricator.wikimedia.org/T314988 (10TheresNoTime) SSHing to any host is //slow//, as is loading https://meta.wikimedia.beta.wmflabs.org/wiki/Main_Page - the 503 has resolved though [10:14:08] TheresNoTime: I got a timeout instead of 503 now [10:14:14] Tools checker alerted too [10:14:20] I'm seeing if can find cloud people [10:14:40] RhinosF1: you think it might be cloud-wide? [10:14:59] TheresNoTime: yes [10:15:53] TheresNoTime: well more WMCS-Alerts tells a story [10:15:59] and stashbot [10:16:11] Yippee, build fixed! [10:16:11] Project beta-scap-sync-world build #63617: 09FIXED in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/63617/ [10:16:41] TheresNoTime: DB is in read only still [10:16:53] which is expected I believe after a crash [10:17:22] 10Beta-Cluster-Infrastructure: Beta cluster 503, scap failures - https://phabricator.wikimedia.org/T314988 (10RhinosF1) Database has gone into read only. [10:17:31] it's easy to take off, but i don't know if a data check is needed first. [10:18:08] 10Beta-Cluster-Infrastructure: Beta cluster 503, scap failures - https://phabricator.wikimedia.org/T314988 (10RhinosF1) Not sure if data check or anything needed, co-ordinate in -releng. Looks like a cloud-wide issue though. [10:18:29] TheresNoTime, zabe: should we ask someone if there's anything to check before someone un reads only it. [10:18:45] ack [10:18:53] * TheresNoTime is scared enough of the DBAs already [10:19:12] i'll ask [10:19:14] they nice [10:20:34] 10Beta-Cluster-Infrastructure: Beta cluster 503, scap failures - https://phabricator.wikimedia.org/T314988 (10TheresNoTime) p:05Unbreak!→03High scap has passed (https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-sync-world/63617/console) and page loads feel quicker. 503s are resolved, SSHing still... [10:21:03] left them a message [10:21:42] Project beta-update-databases-eqiad build #60661: 04FAILURE in 1 min 41 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/60661/ [10:24:28] TheresNoTime: just but it back in read write [10:24:40] and blame Amir1 if replication breaks [10:25:17] SET GLOBAL read_only = 0; UNLOCK TABLES; should do it [10:25:44] !log take deployment-prep out of read-only mode [10:25:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:26:15] (I assume you've done it then?) [10:26:25] TheresNoTime: it works [10:26:38] TheresNoTime: now someone should do SHOW SLAVE STATUS\G on db07 [10:26:43] as i did an edit [10:30:09] lgtm, I guess [10:30:34] zabe: as long as it says no errors and it's running, that's fine [10:31:16] yep [10:31:50] \o/ [10:32:13] 10Beta-Cluster-Infrastructure: Beta cluster 503, scap failures - https://phabricator.wikimedia.org/T314988 (10RhinosF1) and a test edit works without either DB exploding. [10:33:48] * TheresNoTime goes back to work [10:34:22] TheresNoTime: surely beta cluster being alive is a part of your job [10:34:30] because otherwise no testing [10:35:04] nahh that's RelEngs job ^^ [10:35:24] TheresNoTime: also WMCS confirmed it was them. i think we can close the task. [10:35:32] but your job is blocked [10:37:09] 10Beta-Cluster-Infrastructure: Beta cluster 503, scap failures - https://phabricator.wikimedia.org/T314988 (10TheresNoTime) 05Open→03Resolved a:03TheresNoTime Per @dcaro: //"a ceph intervention that went the non-expected way"// — no further work for this task [10:37:41] TheresNoTime: maint still ongoing so maybe we watch for today a bit closer [10:39:08] I have pushover alerts set up for a lot of beta so it'll yell if something breaks again ^^ [10:40:19] Yippee, build fixed! [10:40:20] Project beta-update-databases-eqiad build #60662: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/60662/ [10:40:49] TheresNoTime: nice! [10:49:57] elukey: just back for lunch, I'll take a look, maybe related to the db issues? [10:50:22] RhinosF1 TheresNoTime: thanks for looking into that! [10:50:42] No problem jnuche [10:51:17] jnuche: might have been related. Everything cloud wide got a bit slow. [10:51:44] Same root cause but different issue I'd say [10:53:25] looks like it, elukey's job seems to be back to normal now [10:55:10] I'm around another 4 hours in case anything happens again [10:55:24] Apart from whenever I eat [11:07:27] jobs are actually failing to complete [11:13:15] Project mediawiki-core-doxygen-docker build #36439: 04FAILURE in 1 min 40 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/36439/ [11:17:02] Project beta-scap-sync-world build #63622: 04FAILURE in 3 min 46 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/63622/ [11:17:51] jnuche: things don't seem happy again [11:19:09] jnuche: looks like maint went south again [11:19:13] yeah, jobs are piling up, they just hang in there, trying to see if I can see something wrong somwhere [11:19:33] jnuche: very little you can do. it's WMCS maintenance. [11:20:12] Yippee, build fixed! [11:20:12] Project beta-scap-sync-world build #63623: 09FIXED in 1 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/63623/ [11:21:36] RhinosF1: ah, thx for the update, I see a lot of chatter in other channels [11:22:34] RhinosF1: is there a calendar somewhere with maintenance windows? [11:26:36] jnuche: this caught me by surprise, i don't believe it was announced [11:26:36] ack [11:26:36] jnuche: a quick smoke test shows my edits are vanishing on beta [11:26:36] like they save fine and just then nothing shows in history [11:26:36] jnuche: try a test edit on beta pls [11:26:53] RhinosF1: will try, one sec [11:27:50] jnuche: it's behaving again [11:28:39] worked for me too [11:28:46] just very slow in loading the pages [11:29:30] jnuche: my edits disappeared for a few minutes. i've never seen that before but they come back now. maybe replica lagged and beta doesn't depool like mw? [11:29:37] prod [11:31:00] I know very little about the prod db setup, but some latency in replication from write nodes sounds like a reasonable explanation ^^ [11:32:36] jnuche: in production, if a replica (db07 for beta) has fallen behind the primary (db08 on beta) then it will automatically be depooled by MediaWiki and also very noisly alert SRE [11:32:59] it works by writing the time to the db every second on the primary [11:33:20] and a script on the replica checking that the time on the replica matches what the db says [11:34:43] be nice if we could pause the beta deploy timer in jenkins tbf, no point adding to it if things are a bit wonky [11:34:45] that's good info, appreciated 👍 [11:34:56] I have no idea who would know about the beta setup though [11:35:23] TheresNoTime: sure, can do that [11:36:33] (can you pause a timer job in jenkins?) [11:38:52] TheresNoTime: I don't think you can do it from the UI with a vanilla Jenkins, I wouldn't be surprised if there is a plugin for that though [11:39:18] in any case, I've stopped the cron trigger until things stabilize [11:40:57] Probably worth making that known somewhere, before people wonder why their code isn't appearing on beta :p [11:42:41] jnuche: that would be a Q for Amir.1 [11:42:53] TheresNoTime: silly question then, what's the best channel for that? :) [11:42:53] About DB [11:43:28] TheresNoTime: (if you happen to know) [11:43:36] RhinosF1: will do that, thanks [11:43:46] jnuche: possibly -operations, if only because that's where most people will complain :) [11:44:07] thx! [11:44:49] (and log it in SAL if you've not already?) [11:48:03] !log Temporarily disabled CI beta sync jobs until issue in cluster is resolved [11:48:09] also a good point [11:48:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:51:53] jnuche: thanks! [11:54:19] elukey: please note there are still ongoing issues affecting CI, your jobs may fail/block again [11:56:03] jnuche: I retried a build of helm-lint, let's see how it works :) [11:58:46] yep looks not ok again [12:00:55] 10Project-Admins, 10Discovery-Search, 10User-AKlapper: Clarify / clean up Discovery/Search related team project tags in Phabricator - https://phabricator.wikimedia.org/T314431 (10MPhamWMF) 05Open→03Resolved a:03MPhamWMF >>! In T314431#8141162, @Aklapper wrote: > I'll reopen per T314431#8126147 and open... [12:12:01] Yippee, build fixed! [12:12:01] Project mediawiki-core-doxygen-docker build #36440: 09FIXED in 7 min 33 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/36440/ [12:17:55] elukey: I think we're back to normal :) [12:18:02] super! [12:26:32] !log Reenabled CI beta sync jobs after cluster incident [12:26:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:27:37] 10Release-Engineering-Team, 10CampaignEvents, 10Campaign-Registration, 10Campaign-Tools (Campaign-Tools-Sprint-17), 10MW-1.39-notes (1.39.0-wmf.19; 2022-07-04): Release V0 of the CampaignEvents extension to the Beta Cluster - https://phabricator.wikimedia.org/T311752 (10ifried) 05Open→03Resolved a:0... [15:23:33] (03CR) 10Ahmon Dancy: [C: 03+2] "Verified. Thanks Jaime!" [tools/scap] - 10https://gerrit.wikimedia.org/r/822330 (owner: 10Jaime Nuche) [15:27:48] (03Merged) 10jenkins-bot: serial lock: move check for lock dir to `TimeoutLock` class [tools/scap] - 10https://gerrit.wikimedia.org/r/822330 (owner: 10Jaime Nuche) [15:28:38] jnuche: I think we're fine now but I'm disappearing soonish for the rest of the evening [15:29:51] Rhinos1: sure, thank you very much again [15:29:58] enjoy your evening :) [15:30:20] RhinosF1: ^ [15:33:54] Same to you! [15:43:20] (03CR) 10Ahmon Dancy: scap backport: Add commit subject to confirmation (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/822190 (https://phabricator.wikimedia.org/T314612) (owner: 10Jeena Huneidi) [15:48:02] (03CR) 10Ahmon Dancy: Scap backport --list: Add mergeable column (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/810078 (https://phabricator.wikimedia.org/T303967) (owner: 10Jeena Huneidi) [17:32:21] (03PS2) 10Jeena Huneidi: scap backport: Add commit subject to confirmation [tools/scap] - 10https://gerrit.wikimedia.org/r/822190 (https://phabricator.wikimedia.org/T314612) [17:42:04] (03CR) 10Jeena Huneidi: scap backport: Add commit subject to confirmation (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/822190 (https://phabricator.wikimedia.org/T314612) (owner: 10Jeena Huneidi) [18:16:44] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission 💀), 10decommission-hardware, 10serviceops-radar: decommission gerrit2001.wikimedia.org - https://phabricator.wikimedia.org/T315040 (10Dzahn) [18:19:05] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission 💀), 10decommission-hardware, 10serviceops-radar: decommission gerrit2001.wikimedia.org (dcops) - https://phabricator.wikimedia.org/T315040 (10Dzahn) [18:19:42] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission 💀), 10decommission-hardware, 10serviceops-radar: decommission gerrit2001.wikimedia.org (dcops) - https://phabricator.wikimedia.org/T315040 (10Dzahn) [18:20:26] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission 💀), 10decommission-hardware, 10ops-codfw, 10serviceops-radar: decommission gerrit2001.wikimedia.org (dcops) - https://phabricator.wikimedia.org/T315040 (10Dzahn) [18:20:50] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission 💀), 10decommission-hardware, 10ops-codfw, 10serviceops-radar: decommission gerrit2001.wikimedia.org (dcops) - https://phabricator.wikimedia.org/T315040 (10Dzahn) [18:25:13] James_F: https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/16 needs published, yeah? [18:25:26] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission 💀), 10decommission-hardware, 10ops-codfw, 10serviceops-radar: decommission gerrit2001.wikimedia.org (dcops) - https://phabricator.wikimedia.org/T315040 (10Dzahn) a:03Papaul @Papaul This is WMF6408 in rack D5 U11. The decom cookbook has fi... [18:27:20] (testing a local build for paranoia's sake, then publishing.) [18:27:53] brennen: Yes, please! [18:28:01] cool, thanks for that. [18:28:15] (for patch that is.) [18:28:49] Happy to help. [18:30:08] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission 💀), 10SRE, 10serviceops, and 2 others: replacement for gerrit2001, decom gerrit2001 - https://phabricator.wikimedia.org/T243027 (10Dzahn) This is now handed over to dcops for physical decom steps and continues still at T315040. [18:42:28] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission 💀), 10Patch-For-Review: Configure keyholder on devtools deploy host for phabricator deployment - https://phabricator.wikimedia.org/T314195 (10Dzahn) @dduvall In the merged change above I amended to adjust the key comment (from root@puppetmas... [19:12:34] !log Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/16 [19:12:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:04:47] hello releng. so. I merged the first of the 2 large "scap support for phab" patches. [20:05:02] but that includes changes to ALL scap::target hosts [20:05:11] that isn't actually something limited to phab hosts [20:05:17] and that made me concerned a little bit [20:05:37] also was trying to break that into multiple smaller patches to make it easier to merge [20:06:04] but for example what happened now is...on host webperf1003.. totally unrelated [20:06:09] Notice: /Stage[main]/Webperf::Statsv/Scap::Target[statsv/statsv]/User[deploy-service]/groups: groups changed to ['deploy-service'] [20:06:24] because it uses scap for deployment [20:06:57] it comes from https://gerrit.wikimedia.org/r/c/operations/puppet/+/818227/7/modules/scap/manifests/target.pp [20:07:54] if we could break this stuff into even smaller changes ..like just the scap target change by itself [20:09:35] I had checked on gerrit/mwdebug before where it was noop. but that isn't the case for webperf and unknown others [20:10:01] also noop on mw appservers [20:11:50] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission 💀), 10Patch-For-Review: Move Phabricator configuration into deployment repo - https://phabricator.wikimedia.org/T313950 (10Dzahn) 20:04 < mutante> hello releng. so. I merged the first of the 2 large "scap support for phab" patches. 20:05 < m... [20:28:36] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission 💀), 10Patch-For-Review: Move Phabricator configuration into deployment repo - https://phabricator.wikimedia.org/T313950 (10Dzahn) @dduvall part 1 is deployed in production. (phab2001 first, then phab1001). ` Notice: /Stage[main]/Phabricato... [20:55:22] Project beta-scap-sync-world build #63677: 04FAILURE in 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/63677/ [20:58:03] ^ is a known bad deploy. Revert ongoing. [20:58:25] thx [20:58:37] Np mutante [21:03:08] hey mutante - i'll have to look again, i think the group change could probably be broken out - or i guess maybe limited to phab hosts, but i feel like that being global may be intentional [21:03:17] dancy, jeena - do you remember that discussion? [21:03:44] so.. I merged it anyways.. part 1 that is [21:03:48] including the group change [21:03:51] kk [21:03:53] * dancy reads [21:03:59] I saw it change on webperf1003 but NOT on webperf1004 [21:04:04] i'm trying to think what, if any, the undesired knock-on effects might be there [21:04:05] and not on mwdebug or gerrit .. [21:04:18] it seemed like it was weird that the deploy user didn't have its own group [21:04:22] I don't recall any discussion on it [21:04:38] so here is one question for you guys [21:04:44] will phabricator look at /etc/phabricator by default [21:04:46] or ignore it [21:05:11] like.. if we restart the service now and there is now the new config [21:05:12] phabricator itself shouldn't be aware of that at all; it's just values for the scap config templates to use [21:05:16] will that influence anything [21:05:17] or not yet [21:05:23] that's what I was hoping [21:05:25] brennen: I have no useful information. :-( [21:05:27] Project beta-scap-sync-world build #63678: 04STILL FAILING in 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/63678/ [21:05:29] but it could be a location it checks [21:05:30] dancy: yeah fair [21:05:33] and uses if it finds it [21:05:47] mutante: i don't think that it is, but let me do a bit of grepping [21:06:19] what we have now is a new file /etc/phabricator/config.yaml that you can take a look at [21:06:23] if you have phab-root that is [21:06:33] I see it has secrets and configs [21:06:44] but is it all etc [21:07:28] AFAIK all of phabricator's application-level config is stored under the phab directory itself. [21:08:07] we could restart service on phab2001 and strace it I guess [21:08:23] also there is the change to put the role on phab2002 [21:08:32] and of course part 2 of the scap change [21:08:50] but that one looked more like it needs the right timing together [21:10:06] (03CR) 10Ahmon Dancy: [C: 03+1] "It's so pretty!" [tools/scap] - 10https://gerrit.wikimedia.org/r/822190 (https://phabricator.wikimedia.org/T314612) (owner: 10Jeena Huneidi) [21:11:35] (03CR) 10Ahmon Dancy: [V: 03+1 C: 03+2] "Tested in train-dev w/ two test commits. Thanks Jeena!" [tools/scap] - 10https://gerrit.wikimedia.org/r/822190 (https://phabricator.wikimedia.org/T314612) (owner: 10Jeena Huneidi) [21:11:54] !log restarted phd service on phab2001 [21:11:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:14:35] so..it fails to restart on phab2001 because of the DB connection [21:15:10] but that is probably previously known issue [21:15:36] (03Merged) 10jenkins-bot: scap backport: Add commit subject to confirmation [tools/scap] - 10https://gerrit.wikimedia.org/r/822190 (https://phabricator.wikimedia.org/T314612) (owner: 10Jeena Huneidi) [21:16:01] Yippee, build fixed! [21:16:02] Project beta-scap-sync-world build #63679: 09FIXED in 1 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/63679/ [21:19:04] confirmed as not new: [21:19:08] grep "Failed to start" /var/log/syslog* | wc -l [21:19:08] 15658 [21:21:29] makes sense [21:25:54] we are told we can have a DB in codfw now though [21:26:35] I think next I will look at applying the puppet role on phab2002 [21:30:05] mutante: we also will need to redeploy to those hosts via scap to get the app working, but that is not _quite_ working yet [21:37:43] for part2 of the scap deploy change I should pair [21:50:06] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission 💀), 10Patch-For-Review: Move Phabricator configuration into deployment repo - https://phabricator.wikimedia.org/T313950 (10brennen) > This is the change that happened. I see in the new /etc/phabricator/config.yaml we have a bunch of config v... [21:56:18] I deployed part 2 of the scap/phab changes as well.. just now. [21:56:43] this added a line for "www" to the /etc/phabricator/config.yaml and otherwise noop in prod [21:56:56] all changes for https://gerrit.wikimedia.org/r/q/topic:phabricator%252Fdeployment are now merged [21:57:21] brennen: maybe scap deploy would work now [22:00:29] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission 💀), 10Patch-For-Review: Move Phabricator configuration into deployment repo - https://phabricator.wikimedia.org/T313950 (10Dzahn) Thanks for confirming that @brennen ! I also just deployed https://gerrit.wikimedia.org/r/c/operations/puppet/... [22:02:42] mutante: unfortunately not - i'm still sorting out the scap side of things, should have a patch here shortly [22:03:48] brennen: ACK, no rush [23:32:01] (03PS2) 10Jeena Huneidi: Scap backport --list: Add mergeable column [tools/scap] - 10https://gerrit.wikimedia.org/r/810078 (https://phabricator.wikimedia.org/T303967) [23:32:44] (03CR) 10Jeena Huneidi: Scap backport --list: Add mergeable column (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/810078 (https://phabricator.wikimedia.org/T303967) (owner: 10Jeena Huneidi) [23:35:46] (03PS1) 10Jeena Huneidi: Update prettytable version for scap [tools/train-dev] - 10https://gerrit.wikimedia.org/r/822463 (https://phabricator.wikimedia.org/T303967) [23:35:56] (03CR) 10CI reject: [V: 04-1] Scap backport --list: Add mergeable column [tools/scap] - 10https://gerrit.wikimedia.org/r/810078 (https://phabricator.wikimedia.org/T303967) (owner: 10Jeena Huneidi)