[00:00:41] 10Gerrit, 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10User-brennen: Script closing/archiving of migrated repositories on Gerrit - https://phabricator.wikimedia.org/T330345 (10Dzahn) I wonder if we could make migrated Gerrit repos read-only and show that fact in t... [00:15:17] 10GitLab (Auth & Access), 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Infrastructure-Foundations, 10CAS-SSO, 10User-brennen: GitLab sessions expire frequently - https://phabricator.wikimedia.org/T330359 (10brennen) [00:55:17] 10GitLab (Auth & Access), 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Infrastructure-Foundations, 10CAS-SSO, 10User-brennen: GitLab sessions expire frequently - https://phabricator.wikimedia.org/T330359 (10bd808) [04:24:26] 10Phabricator: Phabricator catch-all task should be at bottom - https://phabricator.wikimedia.org/T330262 (10Jidanni) All I know is: {F36865995} [06:49:43] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10User-brennen: Comment on Phabricator tasks for new, merged, and abandoned changes on GitLab - https://phabricator.wikimedia.org/T324150 (10hashar) >>! In T324150#8639102, @brennen wrote: >> I am reopening the task si... [07:00:47] 10Phabricator, 10Trust-and-Safety, 10cloud-services-team, 10wikitech.wikimedia.org: Reset 2FA for Developer account 'Rosalie Perside (WMDE)' and Phabricator account @Rosalie_WMDE - https://phabricator.wikimedia.org/T329179 (10Rosalie_WMDE) @bd808, you are right. This is the correct string `bastion-eqiad1-... [07:01:34] 10Continuous-Integration-Infrastructure, 10OOUI, 10Regression: OOUI PHP demos page is broken (again) - https://phabricator.wikimedia.org/T322357 (10hashar) >>! In T322357#8468528, @Krinkle wrote: > @hashar Do we have enabled and access to some kind of error_log for the doc1002 php-fpm instance where we might... [07:07:51] 10Continuous-Integration-Infrastructure, 10OOUI, 10Regression: OOUI PHP demos page is broken (again) - https://phabricator.wikimedia.org/T322357 (10hashar) >>! In T322357#8603557, @OriginalAuthority wrote: > Is there any update on when this is going to be fixed? Not having this documentation severly hinders... [07:13:42] 10Continuous-Integration-Infrastructure, 10serviceops-collab: migrate doc hosts to bullseye - https://phabricator.wikimedia.org/T319477 (10hashar) May #serviceops-collab create the Bullseye Ganeti VM to replace the hosts? I am guessing the hostname will be: | doc1003.eqiad.wmnet | doc2002.codfw.wmnet They... [07:44:35] (03CR) 10Hashar: [C: 03+2] [Zuul] Make tox-docker a voting job for operations/mediawiki-config [integration/config] - 10https://gerrit.wikimedia.org/r/891317 (https://phabricator.wikimedia.org/T329231) (owner: 10Urbanecm) [07:46:30] (03Merged) 10jenkins-bot: [Zuul] Make tox-docker a voting job for operations/mediawiki-config [integration/config] - 10https://gerrit.wikimedia.org/r/891317 (https://phabricator.wikimedia.org/T329231) (owner: 10Urbanecm) [07:49:20] !log Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/891317 Make tox-docker a voting job for operations/mediawiki-config [07:49:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:57:00] 10Continuous-Integration-Infrastructure, 10Wikimedia-Portals: wikimedia-portals-build permission error: "dubious ownership in repository" - https://phabricator.wikimedia.org/T330194 (10hashar) >>! In T330194#8637200, @Ladsgroup wrote: > Maybe this didn't get deployed? https://gerrit.wikimedia.org/r/c/integrati... [07:59:23] (03PS1) 10Hashar: jjb: drop git safe.directory for wikimedia-portals-build [integration/config] - 10https://gerrit.wikimedia.org/r/891485 (https://phabricator.wikimedia.org/T328924) [08:01:46] (03CR) 10Hashar: [C: 03+2] jjb: drop git safe.directory for wikimedia-portals-build [integration/config] - 10https://gerrit.wikimedia.org/r/891485 (https://phabricator.wikimedia.org/T328924) (owner: 10Hashar) [08:03:00] (03Merged) 10jenkins-bot: jjb: drop git safe.directory for wikimedia-portals-build [integration/config] - 10https://gerrit.wikimedia.org/r/891485 (https://phabricator.wikimedia.org/T328924) (owner: 10Hashar) [08:04:26] 10Continuous-Integration-Infrastructure, 10Wikimedia-Portals, 10Patch-For-Review: wikimedia-portals-build permission error: "dubious ownership in repository" - https://phabricator.wikimedia.org/T330194 (10hashar) [08:05:18] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Wikimedia-Portals, 10Patch-For-Review: wikimedia-portals-build permission error: "dubious ownership in repository" - https://phabricator.wikimedia.org/T328924 (10hashar) [08:05:56] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Wikimedia-Portals, 10Patch-For-Review: wikimedia-portals-build permission error: "dubious ownership in repository" - https://phabricator.wikimedia.org/T328924 (10hashar) A follow up task got filed T330194 cause we do a git operation on `./src/prod` but... [08:07:44] 10Continuous-Integration-Infrastructure, 10Wikimedia-Portals, 10Patch-For-Review: wikimedia-portals-build permission error: "dubious ownership in repository" - https://phabricator.wikimedia.org/T330194 (10hashar) **Solved** by update the git setting to `safe.directory='*'`. I have marked this as a dupe of th... [08:19:56] 10Continuous-Integration-Infrastructure, 10SRE, 10Traffic-Icebox: Make CI run Varnish VCL tests - https://phabricator.wikimedia.org/T128188 (10hashar) The task is from 2016, and I most probably filed it as a placeholder to track effort to enhance test coverage on various repositories. We were on a rampage to... [08:50:16] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - https://phabricator.wikimedia.org/T329930 (10Jelto) >>! In T329930#8637828, @Dzahn wrote: > > I think T296944#8496308 is related here. Thanks for the link! We made sure we use the... [08:55:45] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10Jelto) [09:06:13] 10Phabricator, 10Developer-Advocacy (Jan-Mar 2023): Set up Herald rule to automatically answer some common newcomer questions - https://phabricator.wikimedia.org/T330379 (10Aklapper) p:05Triage→03Low [09:09:22] thcipriani: ack, thanks [09:26:44] James_F: https://phabricator.wikimedia.org/T330380#8640101 [09:27:45] James_F: It's nice to have fresh riding ahead of the curve this time. Makes me feel a lot better prepped this time around as I've had the chance to opt-in to node16 for a while and then opt-out of the default fresh-node (16 since last year) and use fresh-node14 (to match WMF CI) for repos that don't pass and then incentice to make them pass on the default fresh-node. [09:27:54] Perhaps a good strategy going forward [09:28:48] incentivized*, where default fresh-node = near/next WMF CI. [09:34:00] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10Jelto) [09:36:03] 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Scap: Scap fails on debian bullseye targets - https://phabricator.wikimedia.org/T326668 (10jnuche) @jbond, that seems a different issue while installing Scap3 services. This ticket's issue is about a problem installing Scap itself via the self-installer... [09:39:54] 10Phabricator, 10Developer-Advocacy (Jan-Mar 2023): Set up Herald rule to automatically answer some common newcomer questions - https://phabricator.wikimedia.org/T330379 (10Aklapper) 05Open→03Resolved Done in {H414} [09:44:42] 10Phabricator: Phabricator catch-all task should be at bottom - https://phabricator.wikimedia.org/T330262 (10Aklapper) See https://www.mediawiki.org/wiki/Phabricator/Help/Forms#Adding_forms_to_your_Favorites [09:51:28] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10Jelto) [09:51:56] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10Jelto) [09:59:18] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10Jelto) [10:37:30] 10Continuous-Integration-Config, 10Wikimedia-Site-requests, 10Patch-For-Review, 10User-Urbanecm: CI should ensure that wmf-config/logos.php matches logos/config.yaml - https://phabricator.wikimedia.org/T329231 (10Urbanecm) 05Open→03Resolved Did a final test with [r891505](https://gerrit.wikimedia.org/r... [10:48:06] 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Scap: Scap fails on debian bullseye targets - https://phabricator.wikimedia.org/T326668 (10MoritzMuehlenhoff) Didn't have a closer look at the failure, but noticed a correlation: logstash, netmon and stat all have the security update of git, which added... [11:26:35] 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Scap, 10SRE: Scap fails on debian bullseye targets - https://phabricator.wikimedia.org/T326668 (10fgiunchedi) Adding #sre for visibility [11:29:52] 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Scap, 10SRE: Scap fails on debian bullseye targets - https://phabricator.wikimedia.org/T326668 (10fgiunchedi) For logstash and arclamp hosts the `scap` module doesn't seem to be found/loadable: ` deploy-service@logstash1032:~$ scap deploy-local --re... [11:31:09] (03PS1) 10Kosta Harlan: jjb: Enable reporting-url for quibble-{packages-source}-{database}-{php}-docker [integration/config] - 10https://gerrit.wikimedia.org/r/891514 (https://phabricator.wikimedia.org/T323750) [12:20:56] 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Scap, 10SRE: Scap fails on debian bullseye targets - https://phabricator.wikimedia.org/T326668 (10jnuche) @fgiunchedi as a quickfix I manually installed scap's latest version on those hosts (arclamp and logstash): ` scap@logstash1032:~$ scap version... [12:34:53] 10Phabricator, 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Scap: puppet/scap: failing on aphlict2001 - https://phabricator.wikimedia.org/T330393 (10jbond) [12:36:28] 10Scap, 10Analytics-Clusters: Scap issues with stat hosts - https://phabricator.wikimedia.org/T330394 (10jbond) [12:37:40] 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Scap, 10SRE: Scap fails on debian bullseye targets - https://phabricator.wikimedia.org/T326668 (10jbond) >>! In T326668#8640118, @jnuche wrote: > @jbond, that seems a different issue while installing Scap3 services. This ticket's issue is about a prob... [13:26:08] (03CR) 10Hashar: [C: 03+2] "It is being deployed ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/891514 (https://phabricator.wikimedia.org/T323750) (owner: 10Kosta Harlan) [13:27:26] (03Merged) 10jenkins-bot: jjb: Enable reporting-url for quibble-{packages-source}-{database}-{php}-docker [integration/config] - 10https://gerrit.wikimedia.org/r/891514 (https://phabricator.wikimedia.org/T323750) (owner: 10Kosta Harlan) [13:34:19] 10Scap, 10Analytics-Clusters: Scap issues with stat hosts - https://phabricator.wikimedia.org/T330394 (10nfraison) Updating .config file in /srv/deployment/analytics/hdfs-tools/deploy-cache in order to have git_server set to deploy1002.eqiad.wmnet instead of deploy1001.eqiad.wmnet This make the scap command r... [13:35:48] 10Scap, 10Analytics-Clusters: Scap issues with stat hosts - https://phabricator.wikimedia.org/T330394 (10nfraison) Here is the result of the debug puppet logs for one of those scap target ` Debug: Executing: '/usr/bin/git -C /srv/deployment/analytics/hdfs-tools/deploy tag --points-at HEAD' Debug: scap pkg [ana... [13:38:32] (03PS1) 10Kosta Harlan: Revert "jjb: Enable reporting-url for quibble-{packages-source}-{database}-{php}-docker" [integration/config] - 10https://gerrit.wikimedia.org/r/891303 [13:39:00] 10Scap, 10Analytics-Clusters: Scap issues with stat hosts - https://phabricator.wikimedia.org/T330394 (10nfraison) Running the `/usr/bin/git -C /srv/deployment/analytics/hdfs-tools/deploy tag --points-at HEAD` manually as the user owning the folder (analytics-deploy) works: `scap/sync/2020-02-28/0001` While r... [13:39:13] 10Scap, 10Analytics-Clusters: Scap issues with stat hosts - https://phabricator.wikimedia.org/T330394 (10nfraison) a:03nfraison [13:43:28] 10Scap, 10Analytics-Clusters: Scap issues with stat hosts - https://phabricator.wikimedia.org/T330394 (10hashar) >>! In T330394#8640720, @nfraison wrote: > Running the `/usr/bin/git -C /srv/deployment/analytics/hdfs-tools/deploy tag --points-at HEAD` manually as the user owning the folder (analytics-deploy) wo... [13:45:08] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10SRE, 10Patch-For-Review: git: detected dubious ownership in repository at '/srv/mediawiki-staging' - https://phabricator.wikimedia.org/T325128 (10hashar) The same issue occurs on the deployment target as reported by T330394: ` nfraison@st... [13:45:30] 10Scap, 10Analytics-Clusters: Scap issues with stat hosts - https://phabricator.wikimedia.org/T330394 (10nfraison) Or we could ensure that the first call to get state is also run as the user owning the folder? [13:51:24] 10Scap, 10Analytics-Clusters: Scap issues with stat hosts - https://phabricator.wikimedia.org/T330394 (10hashar) The git security update for `safe.directory` is intended exactly for that use case. A deployer could inject in the git repository some hook (as the deployment user), then when Puppet runs git as roo... [13:54:37] What's the deal with https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php74-docker/12871/console "quibble: error: unrecognized arguments: --reporting-url=https://earlywarningbot.toolforge.org" [13:55:03] bawolff: https://gerrit.wikimedia.org/r/c/integration/config/+/891303 [13:55:14] we need hashar or someone who can deploy jenkins jobs to push that revert [13:55:32] it enabled an experimental feature in quibble 1.5.0, except Quibble 1.4.7 is set for all jobs, not quibble 1.5.0 [13:55:47] Thanks :) [13:55:59] let's see if I still remember how to deploy jjb changes [13:56:05] (sorry for the inconvenience!) [14:00:09] !log update quibble* jenkins jobs for 891303 [14:00:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:00:19] (03CR) 10Majavah: [C: 03+2] "deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/891303 (owner: 10Kosta Harlan) [14:00:25] bawolff: kostajh: try now? [14:00:55] trying [14:00:57] thanks taavi [14:01:32] (03Merged) 10jenkins-bot: Revert "jjb: Enable reporting-url for quibble-{packages-source}-{database}-{php}-docker" [integration/config] - 10https://gerrit.wikimedia.org/r/891303 (owner: 10Kosta Harlan) [14:02:55] kostajh: I am there [14:03:10] so revert the change that adds the earlywarningbot parameter right? [14:03:28] oh thank you taavi :-] [14:03:52] Can confirm working now. Thanks taavi :) [14:04:04] I thought I had all jobs update to Quibble 1.5 [14:04:26] oh NO [14:04:41] only the integration-quibble-* ones which are solely for the `integration/quibble` repo. My bad [14:05:23] yeah, sorry I forgot about that as well hashar [14:05:33] I will craft the patch [14:05:40] and cherry pick your commit from this morning on top of it [14:05:45] danke [14:09:30] (03PS1) 10Hashar: jjb: switch jobs to Quibble 1.5.0 [integration/config] - 10https://gerrit.wikimedia.org/r/891548 (https://phabricator.wikimedia.org/T323750) [14:09:35] (03PS1) 10Hashar: jjb: Enable reporting-url for quibble-{packages-source}-{database}-{php}-docker [integration/config] - 10https://gerrit.wikimedia.org/r/891549 (https://phabricator.wikimedia.org/T323750) [14:09:38] kostajh: ^ :) [14:10:07] I guess I can do a single pass to switch to 1.5.0 AND add the reporting-url parameter [14:10:17] (03CR) 10Kosta Harlan: [C: 03+1] jjb: switch jobs to Quibble 1.5.0 [integration/config] - 10https://gerrit.wikimedia.org/r/891548 (https://phabricator.wikimedia.org/T323750) (owner: 10Hashar) [14:10:30] (03CR) 10Kosta Harlan: [C: 03+1] jjb: Enable reporting-url for quibble-{packages-source}-{database}-{php}-docker [integration/config] - 10https://gerrit.wikimedia.org/r/891549 (https://phabricator.wikimedia.org/T323750) (owner: 10Hashar) [14:11:28] !log Update Jenkins jobs to use Quibble 1.5.0 and have some to use the new reporting-url parameter [14:11:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:14:13] 10Scap, 10Analytics-Clusters: Scap issues with stat hosts - https://phabricator.wikimedia.org/T330394 (10MoritzMuehlenhoff) >>! In T330394#8640754, @nfraison wrote: > Or we could ensure that the first call to get state is also run as the user owning the folder? Agreed, the best way to fix that for such a depl... [14:14:56] taavi: bawolff: I am reenabling the reporting-url feature of Quibble, but this time upgrading to Quibble 1.5.0 [14:17:25] (03CR) 10Hashar: "It is being deployed, together with https://gerrit.wikimedia.org/r/c/integration/config/+/891548/ for the reporting url" [integration/config] - 10https://gerrit.wikimedia.org/r/891549 (https://phabricator.wikimedia.org/T323750) (owner: 10Hashar) [14:17:30] (03CR) 10Hashar: [C: 03+2] jjb: Enable reporting-url for quibble-{packages-source}-{database}-{php}-docker [integration/config] - 10https://gerrit.wikimedia.org/r/891549 (https://phabricator.wikimedia.org/T323750) (owner: 10Hashar) [14:17:53] (03CR) 10Hashar: [C: 03+2] "It is being deployed, this time with https://gerrit.wikimedia.org/r/c/integration/config/+/891549/ which switch the jobs to Quibble 1.5. w" [integration/config] - 10https://gerrit.wikimedia.org/r/891548 (https://phabricator.wikimedia.org/T323750) (owner: 10Hashar) [14:18:40] kostajh: we should bless you with the privileges to adjust Jenkins jobs config [14:19:16] (03Merged) 10jenkins-bot: jjb: switch jobs to Quibble 1.5.0 [integration/config] - 10https://gerrit.wikimedia.org/r/891548 (https://phabricator.wikimedia.org/T323750) (owner: 10Hashar) [14:19:20] (03Merged) 10jenkins-bot: jjb: Enable reporting-url for quibble-{packages-source}-{database}-{php}-docker [integration/config] - 10https://gerrit.wikimedia.org/r/891549 (https://phabricator.wikimedia.org/T323750) (owner: 10Hashar) [14:30:54] 10Scap, 10Analytics-Clusters, 10Patch-For-Review: Scap issues with stat hosts - https://phabricator.wikimedia.org/T330394 (10jbond) >>! In T330394#8640720, @nfraison wrote: > Running the `/usr/bin/git -C /srv/deployment/analytics/hdfs-tools/deploy tag --points-at HEAD` manually as the user owning the folder... [14:32:51] (03PS1) 10Kosta Harlan: jjb: Enable --reporting-url for more Quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/891559 (https://phabricator.wikimedia.org/T323750) [14:33:52] 10Scap, 10Analytics-Clusters, 10Patch-For-Review: Scap issues with stat hosts - https://phabricator.wikimedia.org/T330394 (10nfraison) seems we mostly end up with same patch @jbond https://gerrit.wikimedia.org/r/891555 / https://gerrit.wikimedia.org/r/891557 I would be interested in reading/seeing how you d... [14:34:39] 10Project-Admins, 10Data-Engineering, 10PM: Archive Analytics tag - https://phabricator.wikimedia.org/T298671 (10Aklapper) Could someone please answer the previous comment, so I could finish this task? Thanks a lot in advance! [14:40:17] 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Scap, 10SRE: Scap fails on debian bullseye targets - https://phabricator.wikimedia.org/T326668 (10fgiunchedi) >>! In T326668#8640506, @jnuche wrote: > @fgiunchedi as a quickfix I manually installed scap's latest version on those hosts (arclamp and log... [14:42:28] (03CR) 10Hashar: [C: 03+2] "Tadam!" [integration/config] - 10https://gerrit.wikimedia.org/r/891559 (https://phabricator.wikimedia.org/T323750) (owner: 10Kosta Harlan) [14:44:14] (03Merged) 10jenkins-bot: jjb: Enable --reporting-url for more Quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/891559 (https://phabricator.wikimedia.org/T323750) (owner: 10Kosta Harlan) [14:49:03] 10Scap, 10Analytics-Clusters, 10Patch-For-Review: Scap issues with stat hosts - https://phabricator.wikimedia.org/T330394 (10jbond) >>! In T330394#8640954, @nfraison wrote: > seems we mostly end up with same patch @jbond https://gerrit.wikimedia.org/r/891555 / https://gerrit.wikimedia.org/r/891557 > > I wou... [14:57:57] 10Scap, 10Analytics-Clusters, 10Patch-For-Review: Scap issues with stat hosts - https://phabricator.wikimedia.org/T330394 (10nfraison) Seems to work fine Before trying to redeploy analytics/hdfs-tools/deploy ` Info: Unable to serialize catalog to json, retrying with pson Info: Applying configuration version... [15:30:59] 10Scap, 10Analytics-Clusters: Scap issues with stat hosts - https://phabricator.wikimedia.org/T330394 (10nfraison) 05Open→03Resolved [15:31:01] 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Scap, 10SRE: Scap fails on debian bullseye targets - https://phabricator.wikimedia.org/T326668 (10nfraison) [15:54:39] 10Continuous-Integration-Config, 10Release-Engineering-Team (Seen), 10Scap, 10MW-on-K8s, and 6 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Ladsgroup) [15:55:40] 10GitLab (Infrastructure), 10serviceops-collab: Investigate gitlab maintenance/read-only mode - https://phabricator.wikimedia.org/T330277 (10eoghan) a:03eoghan We have a few options here to show varying levels of user experience. The deployment page is nice, but it alone doesn't block API calls or ssh writ... [16:17:07] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10SRE, 10Patch-For-Review: git: detected dubious ownership in repository at '/srv/mediawiki-staging' - https://phabricator.wikimedia.org/T325128 (10MoritzMuehlenhoff) >>! In T325128#8640737, @hashar wrote: > The same issue occurs on the dep... [16:26:46] 10Phabricator, 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Scap: puppet/scap: failing on aphlict2001 - https://phabricator.wikimedia.org/T330393 (10MoritzMuehlenhoff) @Dzahn /etc/phabricator and /etc/phabricator/config.yaml are only defined in the main class phabricator which is not present for... [16:27:45] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - https://phabricator.wikimedia.org/T329930 (10eoghan) 05Open→03Resolved This was completed successfully yesterday and everything remains stable 24 hours lat... [17:15:45] 10Scap, 10Analytics-Clusters: Scap issues with stat hosts - https://phabricator.wikimedia.org/T330394 (10hashar) Very nice fix @nfraison thank you! [17:29:52] 10Phabricator, 10Trust-and-Safety, 10cloud-services-team, 10wikitech.wikimedia.org, 10User-bd808: Reset 2FA for Developer account 'Rosalie Perside (WMDE)' and Phabricator account @Rosalie_WMDE - https://phabricator.wikimedia.org/T329179 (10bd808) 05Open→03Resolved a:03bd808 ` $ ssh root@bastion-eqi... [18:08:48] Krinkle: Clearly we should add node18 to fresh very soon. Was thinking of adding to CI anyway this weekend. [18:09:21] oh I had a question for node16, does that mean npm get upgraded as well? [18:09:32] hashar: No, still pinned to 7.21.0 [18:09:38] hashar: Probably do that as a different task. [18:09:41] It's bundled by nodejs now in the post 16 builds right? [18:09:45] There was an issue with the packagin. [18:10:32] we will get it with the next Debian iirc (node 18) [18:11:59] We re-started use of our pin to avoid accidental downgrade during node 14 [18:13:45] great [18:14:04] Huh, James_F tried this with 16 as planned but found they were non operable? [18:29:04] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10Jelto) I scheduled a [new broadcast message](https://gitlab.wikimedia.org/admin/broadcast_messages/7/edit) on GitLab for t... [18:41:49] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Radar), 10serviceops-collab: Cleanup and attach volumes for gitlab-runners WMCS project - https://phabricator.wikimedia.org/T328283 (10Jelto) a:03Jelto I deleted all unused volumes in the `gitlab-runners` project. This freed some storage/volume quota... [19:07:40] 10Phabricator, 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Scap: puppet/scap: failing on aphlict2001 - https://phabricator.wikimedia.org/T330393 (10brennen) > @Dzahn /etc/phabricator and /etc/phabricator/config.yaml are only defined in the main class phabricator which is not present for the aph... [19:08:55] 10Phabricator, 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Scap: puppet/scap: failing on aphlict2001 - https://phabricator.wikimedia.org/T330393 (10Dzahn) @MoritzMuehlenhoff Thanks for reporting! We are aware of this and it's because aphlict2001 is still being setup (T322369). We just recently... [19:09:33] 10Phabricator, 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Scap: puppet/scap: failing on aphlict2001 - https://phabricator.wikimedia.org/T330393 (10Dzahn) [19:09:36] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab: Deploy phabricator to aphlict2002.codfw.wmnet - https://phabricator.wikimedia.org/T329908 (10Dzahn) [19:09:39] 10Phabricator, 10serviceops-collab: create aphlict2001 (Phabricator realtime notifications codfw) - https://phabricator.wikimedia.org/T322369 (10Dzahn) [19:11:01] 10Phabricator, 10serviceops-collab: create aphlict2001 (Phabricator realtime notifications codfw) - https://phabricator.wikimedia.org/T322369 (10Dzahn) T330393 was created because puppet fails on this machine but we know that and the scap deploy at T329908 should fix that. [19:11:50] 10Phabricator, 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Scap, 10Patch-For-Review: puppet/scap: failing on aphlict2001 - https://phabricator.wikimedia.org/T330393 (10Dzahn) p:05Triage→03Medium [19:12:27] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: Deploy phabricator to aphlict2002.codfw.wmnet - https://phabricator.wikimedia.org/T329908 (10Dzahn) This should also fix T330393. [19:13:18] 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Scap, 10SRE: Scap fails on debian bullseye targets - https://phabricator.wikimedia.org/T326668 (10Dzahn) [19:14:45] 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Scap, 10SRE: Scap fails on debian bullseye targets - https://phabricator.wikimedia.org/T326668 (10Dzahn) [19:15:23] 10Scap, 10Analytics-Clusters: Scap issues with stat hosts - https://phabricator.wikimedia.org/T330394 (10Dzahn) Is T330360 a duplicate of this? [19:18:53] 10Scap, 10Analytics-Clusters: Scap issues with stat hosts - https://phabricator.wikimedia.org/T330394 (10Dzahn) Looks like it is. merging in as duplicate. also see T326668 [19:19:20] 10Scap, 10Analytics-Clusters: Scap issues with stat hosts - https://phabricator.wikimedia.org/T330394 (10Dzahn) [19:24:18] 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Scap, 10SRE: Scap fails on debian bullseye targets - https://phabricator.wikimedia.org/T326668 (10Dzahn) [19:41:56] scap has defaults to use gerrit as origin: modules/scap/lib/puppet/provider/scap_source/default.rb: "https://gerrit.wikimedia.org/r/#{repo_name}.git" [19:42:09] so you cant use gitlab repos as origin for something deployed with scap [19:42:26] this currently is a problem for phabricator repos which got moved from phabricator itself to gitlab [20:21:55] 10GitLab, 10Release-Engineering-Team (GitLab V: Event Horizon 🌄): Isito interferes with HTTP traffic from buildkitd build containers - https://phabricator.wikimedia.org/T330433 (10dancy) [20:30:42] 10GitLab, 10Release-Engineering-Team (GitLab V: Event Horizon 🌄): Isito interferes with HTTP traffic from buildkitd build containers - https://phabricator.wikimedia.org/T330433 (10dancy) [21:00:34] 10Continuous-Integration-Infrastructure, 10serviceops-collab: migrate doc hosts to bullseye - https://phabricator.wikimedia.org/T319477 (10Dzahn) Yes, the hostnames will be doc1003 and doc2002, confirmed. And yes, serviceops-collab will create them. We will soon assign it to a person. [21:02:26] 10Continuous-Integration-Infrastructure, 10serviceops-collab: migrate doc hosts to bullseye - https://phabricator.wikimedia.org/T319477 (10Dzahn) contint2001 migration is higher prio though for sure, btw. [21:06:34] 10Phabricator, 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Scap: puppet/scap: failing on aphlict2001 - https://phabricator.wikimedia.org/T330393 (10Dzahn) 19:41 < mutante> scap has defaults to use gerrit as origin: modules/scap/lib/puppet/provider/scap_source/default.rb: "https://gerrit.wi... [21:06:54] 10Phabricator, 10serviceops-collab: create aphlict2001 (Phabricator realtime notifications codfw) - https://phabricator.wikimedia.org/T322369 (10Dzahn) 19:41 < mutante> scap has defaults to use gerrit as origin: modules/scap/lib/puppet/provider/scap_source/default.rb: "https://gerrit.wikimedia.org/r/#{rep... [21:07:09] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab: Deploy phabricator to aphlict2002.codfw.wmnet - https://phabricator.wikimedia.org/T329908 (10Dzahn) 19:41 < mutante> scap has defaults to use gerrit as origin: modules/scap/lib/puppet/provider/scap_source/default.rb: "https://gerrit.wikimed... [21:10:19] (03PS1) 10Jforrester: Docker: [node16] Upgrade node from 16.16.0 to 16.19.1 [integration/config] - 10https://gerrit.wikimedia.org/r/891648 [21:10:21] (03PS1) 10Jforrester: Docker: [node16] Switch from pinned npm (7.21.0) to upstream's version (currently, 8.19.3) [integration/config] - 10https://gerrit.wikimedia.org/r/891649 [21:11:29] (03CR) 10CI reject: [V: 04-1] Docker: [node16] Switch from pinned npm (7.21.0) to upstream's version (currently, 8.19.3) [integration/config] - 10https://gerrit.wikimedia.org/r/891649 (owner: 10Jforrester) [21:12:15] (03CR) 10Jforrester: [C: 03+2] Docker: [node16] Upgrade node from 16.16.0 to 16.19.1 [integration/config] - 10https://gerrit.wikimedia.org/r/891648 (owner: 10Jforrester) [21:13:23] (03Merged) 10jenkins-bot: Docker: [node16] Upgrade node from 16.16.0 to 16.19.1 [integration/config] - 10https://gerrit.wikimedia.org/r/891648 (owner: 10Jforrester) [21:13:53] !log Docker: [node16] Upgrade node from 16.16.0 to 16.19.1 [21:13:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:22:43] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab: Deploy phabricator to aphlict2001.codfw.wmnet - https://phabricator.wikimedia.org/T329908 (10brennen) [21:41:02] 10GitLab, 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Patch-For-Review: Isito interferes with HTTP traffic from buildkitd build containers - https://phabricator.wikimedia.org/T330433 (10dancy) Workaround deployed and operational. [21:47:10] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - https://phabricator.wikimedia.org/T329930 (10Dzahn) Ah, cool, this is a nice explanation and outcome. Thanks! [21:57:27] 10GitLab, 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Patch-For-Review: Isito interferes with HTTP traffic from buildkitd build containers - https://phabricator.wikimedia.org/T330433 (10dduvall) Awesome! [22:03:09] (03PS1) 10Jforrester: jjb: Switch node jobs' images from node 16.16.0 to 16.19.1 [integration/config] - 10https://gerrit.wikimedia.org/r/891658 [22:03:11] (03PS1) 10Jforrester: Docker: [quibble-buster] Re-use node from the node image rather than our own [integration/config] - 10https://gerrit.wikimedia.org/r/891659 [22:09:48] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/891658 (owner: 10Jforrester) [22:10:56] (03Merged) 10jenkins-bot: jjb: Switch node jobs' images from node 16.16.0 to 16.19.1 [integration/config] - 10https://gerrit.wikimedia.org/r/891658 (owner: 10Jforrester) [22:12:52] (03CR) 10Jforrester: [C: 04-1] Docker: [node16] Switch from pinned npm (7.21.0) to upstream's version (currently, 8.19.3) [integration/config] - 10https://gerrit.wikimedia.org/r/891649 (owner: 10Jforrester) [22:12:57] (03CR) 10Jforrester: [C: 03+2] Docker: [quibble-buster] Re-use node from the node image rather than our own [integration/config] - 10https://gerrit.wikimedia.org/r/891659 (owner: 10Jforrester) [22:14:06] (03Merged) 10jenkins-bot: Docker: [quibble-buster] Re-use node from the node image rather than our own [integration/config] - 10https://gerrit.wikimedia.org/r/891659 (owner: 10Jforrester) [22:15:13] !log Docker: [quibble-buster] Re-use node from the node image rather than our own [22:15:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:30:55] 10Phabricator, 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Scap: puppet/scap: failing on aphlict2001 - https://phabricator.wikimedia.org/T330393 (10Dzahn) >>! In T330393#8641405, @MoritzMuehlenhoff wrote: > @Dzahn /etc/phabricator and /etc/phabricator/config.yaml are only defined in the main cl... [23:36:23] (03PS1) 10Jforrester: jjb: Switch quibble jobs to images with node16 [integration/config] - 10https://gerrit.wikimedia.org/r/891723 (https://phabricator.wikimedia.org/T314470) [23:36:25] (03PS1) 10Jforrester: jenkins-jobs: Specify python rather than assume tox is mapped in the path [integration/config] - 10https://gerrit.wikimedia.org/r/891724 [23:36:51] (03CR) 10Jforrester: [C: 03+2] jjb: Switch quibble jobs to images with node16 [integration/config] - 10https://gerrit.wikimedia.org/r/891723 (https://phabricator.wikimedia.org/T314470) (owner: 10Jforrester) [23:37:07] (03PS2) 10Jforrester: jenkins-jobs: Specify python rather than assume tox is mapped in the path [integration/config] - 10https://gerrit.wikimedia.org/r/891724 [23:38:07] (03Merged) 10jenkins-bot: jjb: Switch quibble jobs to images with node16 [integration/config] - 10https://gerrit.wikimedia.org/r/891723 (https://phabricator.wikimedia.org/T314470) (owner: 10Jforrester) [23:38:15] >23:33:54 node: ../src/coroutine.cc:134: void* find_thread_id_key(void*): Assertion `thread_id_key != 0x7777' failed. [23:38:15] nice [23:38:27] Oh dear. [23:38:29] Where's that from? [23:39:04] The DiscussionTools CI [23:39:09] Meh. [23:39:11] https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php74-selenium-docker/25909/console [23:39:30] This might be from my upgrade. :-( [23:39:37] https://github.com/laverdet/node-fibers/issues/451 ? [23:39:57] Yeah, damn thing will need re-building in core. [23:45:10] Oh, or worse, in all the different repos with this dependency in the gate. [23:45:29] I'm going to have to revert. :-( [23:45:57] (03PS1) 10Jforrester: Revert "jjb: Switch quibble jobs to images with node16" [integration/config] - 10https://gerrit.wikimedia.org/r/891629 [23:46:02] (03CR) 10Jforrester: [C: 03+2] Revert "jjb: Switch quibble jobs to images with node16" [integration/config] - 10https://gerrit.wikimedia.org/r/891629 (owner: 10Jforrester) [23:47:10] (03Merged) 10jenkins-bot: Revert "jjb: Switch quibble jobs to images with node16" [integration/config] - 10https://gerrit.wikimedia.org/r/891629 (owner: 10Jforrester) [23:47:23] 10Continuous-Integration-Config, 10Patch-For-Review: Upgrade all CI jobs for WMF-deployed projects from Node 14 to Node 16 - https://phabricator.wikimedia.org/T314470 (10Jdforrester-WMF) [23:48:03] 10Continuous-Integration-Config, 10Patch-For-Review: Upgrade all CI jobs for WMF-deployed projects from Node 14 to Node 16 - https://phabricator.wikimedia.org/T314470 (10Jdforrester-WMF) [23:50:24] Everything about people being 'clever' and writing platform-specific code on a platform-agnostic runtime makes me sad. [23:52:55] :empathy: from a former professional Java developer :) [23:53:06] Indeed. [23:53:14] OK, everything's now revertef.