[00:13:41] 10Beta-Cluster-Infrastructure: beta cluster down - https://phabricator.wikimedia.org/T329592 (10Zabe) [00:13:51] 10Beta-Cluster-Infrastructure: beta cluster down - https://phabricator.wikimedia.org/T329592 (10Zabe) [00:16:47] 10Beta-Cluster-Infrastructure: beta cluster down - https://phabricator.wikimedia.org/T329592 (10Zabe) [00:22:32] !log install mariadb 10.6 via role::mariadb::beta on deployment-db11 # T329577 [00:22:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:22:38] T329577: deployment-db10 databases are broken - https://phabricator.wikimedia.org/T329577 [00:27:52] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10User-brennen: Build a widget to display GitLab changes on related Phabricator tasks - https://phabricator.wikimedia.org/T324149 (10brennen) Comment bot was likely down earlier, but: [[https://gitlab.wikimedia.... [00:28:08] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10User-brennen: Build a widget to display GitLab changes on related Phabricator tasks - https://phabricator.wikimedia.org/T324149 (10brennen) 05Open→03In progress [00:28:20] 10GitLab (Integrations), 10GitLab-Test, 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), and 2 others: Experiment with GitLab-Phabricator integration - https://phabricator.wikimedia.org/T265617 (10brennen) [01:06:47] !log rsync dump from deployment-db09 to deployment-db11 # T329577 [01:06:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:06:49] T329577: deployment-db10 databases are broken - https://phabricator.wikimedia.org/T329577 [01:14:45] !log deployment-db11: import dump into mariadb # T329577 [01:14:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:14:48] T329577: deployment-db10 databases are broken - https://phabricator.wikimedia.org/T329577 [01:28:42] 10Beta-Cluster-Infrastructure: beta cluster down - https://phabricator.wikimedia.org/T329592 (10Zabe) [01:38:40] !log create deployment-db12 as g3.cores8.ram16.disk20 # T329577 [01:38:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:38:42] T329577: deployment-db10 databases are broken - https://phabricator.wikimedia.org/T329577 [01:50:28] !log create volume db12 and attach to deployment-db12 # T329577 [01:50:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:50:30] T329577: deployment-db10 databases are broken - https://phabricator.wikimedia.org/T329577 [02:09:54] !log install mariadb 10.6 via role::mariadb::beta on deployment-db12 # T329577 [02:09:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:09:56] T329577: deployment-db10 databases are broken - https://phabricator.wikimedia.org/T329577 [02:16:18] !log rsync dump from deployment-db09 to deployment-db12 # T329577 [02:16:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:16:20] T329577: deployment-db10 databases are broken - https://phabricator.wikimedia.org/T329577 [02:19:45] !log deployment-db12: import dump into mariadb # T329577 [02:19:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:57:50] 10Release-Engineering-Team, 10Security-Team: deploy_security.py hangs after scap - https://phabricator.wikimedia.org/T329602 (10Tgr) [02:58:33] 10Release-Engineering-Team, 10Security-Team, 10SecTeam-Processed: Add --pause-after-testserver-sync option to deploy_security.py - https://phabricator.wikimedia.org/T328667 (10Tgr) Seems like this happened and broke the script: {T329602} [03:19:44] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10mwbot-rs, 10mwcli, 10Patch-For-Review: GitLab CI jobs failing with "You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-... - https://phabricator.wikimedia.org/T329216 [07:25:12] 10Gerrit, 10serviceops-collab, 10Patch-For-Review: Issues with Gerrit test instance - https://phabricator.wikimedia.org/T329444 (10Ameisenigel) Well, you have written that it should be possible to use the regular account. I just wanted to try out if it works. [07:30:01] https://integration.wikimedia.org/ci/job/quibble-composer-mysql-php73-noselenium-docker/38814/console - well that's a CI failure I have never seen before "remote: Support for password authentication was removed" [07:30:16] (its on a 1.35 branch, so presumably its due to being old) [08:04:34] Oh, not just on CI [08:04:46] Looks like i cannot composer update locally either on REL1_35 [08:25:35] Oh, i guess this is T278569 [08:25:36] T278569: Archive the wikimedia/avro library - https://phabricator.wikimedia.org/T278569 [08:29:48] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10MW-1.35-release: Wikimedia CI is failing for REL1_35 (both core and extension repos) due to missing wikimedia/avro package - https://phabricator.wikimedia.org/T329608 (10Bawolff) [08:39:44] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10Data Pipelines, 10Data-Engineering-Planning, 10serviceops-collab: Experiencing pipeline failure due to disk-space issues - https://phabricator.wikimedia.org/T310593 (10JAllemandou) Thank you ! [11:27:34] 10Gerrit, 10Pywikibot: 500 server error when pulling Pywikibot i18n - https://phabricator.wikimedia.org/T329452 (10Samwalton9) Have you tried running `--verbose` to see if this gives you more specific information? [11:35:58] 10Gerrit, 10Pywikibot: 500 server error when pulling Pywikibot i18n - https://phabricator.wikimedia.org/T329452 (10binbot) >>! In T329452#8613943, @Samwalton9 wrote: > Have you tried running `--verbose` to see if this gives you more specific information? Whereever I write `--verbose`,in `"git submodule update... [11:37:34] 10Gerrit, 10Pywikibot: 500 server error when pulling Pywikibot i18n - https://phabricator.wikimedia.org/T329452 (10Aklapper) @binbot Please don't paraphrase but post exact commands and exact outputs, to avoid any misunderstandings. Thanks. [11:38:02] !log deployment-db11: start replicating from deployment-db09 # T329577 [11:38:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:38:05] T329577: deployment-db10 databases are broken - https://phabricator.wikimedia.org/T329577 [11:39:43] (good work zabe, thank you) [11:40:55] beta is still completly down :p [11:41:24] *good work so far, keep it up :D [13:16:56] 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Scap: Avoid local venv when distributing Scap - https://phabricator.wikimedia.org/T329622 (10jnuche) [13:17:44] 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Scap: Avoid local venv when distributing Scap - https://phabricator.wikimedia.org/T329622 (10jnuche) 05Open→03In progress [13:59:10] dancy, I think you can reenable the beta cluster jenkins jobs, they should work again [14:04:26] !log delete deployment-db10 and volume db10 # T329577 [14:04:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:04:28] T329577: deployment-db10 databases are broken - https://phabricator.wikimedia.org/T329577 [14:17:29] 10Gerrit, 10Pywikibot: 500 server error when pulling Pywikibot i18n - https://phabricator.wikimedia.org/T329452 (10Dzahn) Can't immediately reproduce the problem from over here: works for me in under 5 seconds. ` time git clone --recursive --branch master --depth 3 https://gerrit.wikimedia.org/r/pywikibot/c... [14:23:22] 10Continuous-Integration-Config, 10Wikidata, 10Wikidata Dev Team, 10Wikidata Query Builder, and 4 others: Move build scripts from CI to the repository - https://phabricator.wikimedia.org/T328543 (10ItamarWMDE) **Task Review Notes:** - This ticket is a result of a workaround applied while addressing an inc... [14:24:11] 10Continuous-Integration-Config, 10Wikidata, 10Wikidata Dev Team, 10Wikidata Query Builder, and 4 others: Move build scripts from CI to the repository - https://phabricator.wikimedia.org/T328543 (10ItamarWMDE) [14:52:12] (03PS1) 10Subramanya Sastry: Use double-colon for psuedo-selectors [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/889147 [14:52:14] (03PS1) 10Subramanya Sastry: Rmeove support for the error-test refgroup [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/889148 [14:54:49] (03CR) 10Subramanya Sastry: [C: 03+2] Rmeove support for the error-test refgroup [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/889148 (owner: 10Subramanya Sastry) [14:55:09] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10mwbot-rs, 10mwcli, 10Patch-For-Review: GitLab CI jobs failing with "You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-... - https://phabricator.wikimedia.org/T329216 [14:56:40] (03PS2) 10Subramanya Sastry: Use double-colon for psuedo-selectors [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/889147 [14:56:42] (03PS2) 10Subramanya Sastry: Rmeove support for the error-test refgroup [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/889148 [15:02:47] !log deployment-db12: start replicating from deployment-db09 # T329577 [15:02:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:02:49] T329577: deployment-db10 databases are broken - https://phabricator.wikimedia.org/T329577 [15:14:51] (03CR) 10Subramanya Sastry: [C: 03+2] Use double-colon for psuedo-selectors [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/889147 (owner: 10Subramanya Sastry) [15:15:28] (03Merged) 10jenkins-bot: Use double-colon for psuedo-selectors [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/889147 (owner: 10Subramanya Sastry) [15:15:34] (03Merged) 10jenkins-bot: Rmeove support for the error-test refgroup [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/889148 (owner: 10Subramanya Sastry) [15:17:13] 10Beta-Cluster-Infrastructure: beta cluster down - https://phabricator.wikimedia.org/T329592 (10Zabe) [15:17:33] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: deployment-db10 databases are broken - https://phabricator.wikimedia.org/T329577 (10Zabe) 05Open→03Resolved a:03Zabe deployment-db10 has been replaced [15:32:45] zabe: beta still completely down, though I note the db has been recreated, what are we waiting on? :) [15:34:10] someone getting haproxy on deployment-cache-text07 running again [15:39:12] 10Beta-Cluster-Infrastructure: beta cluster down - https://phabricator.wikimedia.org/T329592 (10TheresNoTime) `lang=sh samtar@deployment-cache-text07:~$ sudo systemctl restart haproxy Job for haproxy.service failed because the control process exited with error code. See "systemctl status haproxy.service" and "jo... [15:39:41] zabe: I'm online now. Still ready for me to reenable those jobs? [15:39:51] dancy, yep [15:40:02] Done [15:40:13] thx [15:43:45] 10Release-Engineering-Team, 10Security-Team: deploy_security.py hangs after scap - https://phabricator.wikimedia.org/T329602 (10sbassett) As fraught as the manual security deployment process is, most of us on the #security-team still use it, as security deployments in general are far more prone to errors and s... [15:45:58] Yippee, build fixed! [15:45:58] Project beta-scap-sync-world build #90348: 09FIXED in 3 min 36 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/90348/ [15:48:11] Yippee, build fixed! [15:48:12] Project beta-update-databases-eqiad build #65135: 09FIXED in 1 min 48 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65135/ [15:48:22] any ideas on `OCSP update failed for /etc/update-ocsp.d/digicert-2021-rsa-unified.conf` for `deployment-cache-text07` then? :/ [15:49:33] why is it trying to do that? we only have LE certs on deployment-prep [15:52:05] (this is beta, sorry, T329592) [15:52:06] T329592: beta cluster down - https://phabricator.wikimedia.org/T329592 [15:56:15] digicert. huh. strange. [15:57:36] 10Beta-Cluster-Infrastructure: beta cluster down - https://phabricator.wikimedia.org/T329592 (10TheresNoTime) [16:02:20] anyone know why mediawiki-config diffConfig job isn't showing any diffs? [16:02:25] e.g. this change: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/889162 [16:02:28] def has a diff [16:02:28] but [16:02:40] operations-mw-config-php74-composer-diffConfig-docker/1686/console : FAILURE No change detected [16:02:43] https://integration.wikimedia.org/ci/job/operations-mw-config-php74-composer-diffConfig-docker/1686/console [16:03:18] ottomata: T329518 [16:03:18] T329518: diffConfig no longer detecs any changes in operations/mediawiki-config.git - https://phabricator.wikimedia.org/T329518 [16:03:53] ah k, known bug. thanks Lucas_WMDE [16:04:08] I’m glad I’m not the only one who’s bothered by it ^^ [16:04:58] 10Continuous-Integration-Config, 10MediaWiki-Configuration: diffConfig no longer detecs any changes in operations/mediawiki-config.git - https://phabricator.wikimedia.org/T329518 (10Ottomata) I've also noticed this a few times this week. Def affects my config deployment confidence :) [16:23:41] !log make Dreamy Jazz a mediawiki/* +2'er T329000 [16:23:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:23:43] T329000: +2 in mediawiki/ for Dreamy Jazz - https://phabricator.wikimedia.org/T329000 [16:24:56] \o/ [16:27:45] 10Gerrit, 10Pywikibot: 500 server error when pulling Pywikibot i18n - https://phabricator.wikimedia.org/T329452 (10BCornwall) Same here: ` real 0m3.225s user 0m1.370s sys 0m0.242s ` But looking at the Gerrit logs, I see a number of the following since around the 6th: ` Internal error during upload-pack fro... [16:34:47] Congrats Dreamy_Jazz [16:34:59] Thanks :) [16:53:31] 10Gerrit, 10Pywikibot: 500 server error when pulling Pywikibot i18n - https://phabricator.wikimedia.org/T329452 (10binbot) >>! In T329452#8614863, @BCornwall wrote: > Seeing as @binbot appears to be in central Europe, I figured it might be a location thing (since @dzahn and I are in the west coast US) but I w... [17:18:24] 10Gerrit, 10Pywikibot: 500 server error when pulling Pywikibot i18n - https://phabricator.wikimedia.org/T329452 (10Dzahn) I am wondering if you would mind deleting the local repo dir and git clone from scratch like we did, then run the "git submodule update' and see if the probem goes away. [17:21:38] 10Gerrit, 10Pywikibot: 500 server error when pulling Pywikibot i18n - https://phabricator.wikimedia.org/T329452 (10binbot) >>! In T329452#8615143, @Dzahn wrote: > I am wondering if you would mind deleting the local repo dir and git clone from scratch like we did, then run the "git submodule update' and see if... [17:34:20] (03PS1) 10Pwangai: Zuul: [mediawiki/extensions/CheckUser] Enable Sonar Codehealth Checks [integration/config] - 10https://gerrit.wikimedia.org/r/889182 (https://phabricator.wikimedia.org/T321837) [17:42:03] 10Beta-Cluster-Infrastructure: beta cluster down - https://phabricator.wikimedia.org/T329592 (10Jdforrester-WMF) This doesn't block T329535, it was triggered by T329535. [17:42:05] 10Beta-Cluster-Infrastructure: beta cluster down - https://phabricator.wikimedia.org/T329592 (10Jdforrester-WMF) [17:42:12] 10Beta-Cluster-Infrastructure: beta cluster down - https://phabricator.wikimedia.org/T329592 (10Jdforrester-WMF) [17:50:02] 10Beta-Cluster-Infrastructure: beta cluster down - https://phabricator.wikimedia.org/T329592 (10thcipriani) hrm, so what's changed about `deployment-cache-text07`? The haproxy systemd unit file definition contains an `ExecStartPre` that's failing: `ExecStartPre=/usr/local/sbin/update-ocsp-all` And that file w... [18:04:03] 10Beta-Cluster-Infrastructure: beta cluster down - https://phabricator.wikimedia.org/T329592 (10Jdforrester-WMF) >>! In T329592#8615340, @thcipriani wrote: > hrm, so what's changed about `deployment-cache-text07`? Was this part of the T293585 amelioration (aka T320930)? I think that (Oct 2022) was the last time... [18:04:13] (03PS1) 10Pwangai: Zuul: [mediawiki/extensions/DiscussionTools] Enable Sonar Codehealth Checks [integration/config] - 10https://gerrit.wikimedia.org/r/889187 (https://phabricator.wikimedia.org/T321837) [18:27:03] (03PS1) 10Pwangai: Zuul: [mediawiki/extensions/DismissableSiteNotice] Enable Sonar Codehealth Checks [integration/config] - 10https://gerrit.wikimedia.org/r/889197 (https://phabricator.wikimedia.org/T321837) [18:28:15] (03CR) 10CI reject: [V: 04-1] Zuul: [mediawiki/extensions/DismissableSiteNotice] Enable Sonar Codehealth Checks [integration/config] - 10https://gerrit.wikimedia.org/r/889197 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [18:37:51] (03PS2) 10Pwangai: Zuul [mediawiki/extensions/DismissableSiteNotice] Enable Sonar Codehealth Checks [integration/config] - 10https://gerrit.wikimedia.org/r/889197 (https://phabricator.wikimedia.org/T321837) [18:43:31] 10Phabricator, 10Developer Productivity: Disable parent task metadata by default for new sub tasks - https://phabricator.wikimedia.org/T239378 (10Clement_Goubert) +1 for disabling too, it makes having a meta-task communicated to a lot of different teams a hindrance, because you then have to remove most of thes... [18:51:20] 10Phabricator, 10Release-Engineering-Team, 10Developer Productivity, 10User-brennen: Disable parent task metadata by default for new sub tasks - https://phabricator.wikimedia.org/T239378 (10brennen) [18:59:43] (03PS1) 10Pwangai: Zuul: [mediawiki/extensions/ContentTranslation] Enable Sonar Codehealth Checks [integration/config] - 10https://gerrit.wikimedia.org/r/889201 (https://phabricator.wikimedia.org/T321837) [19:19:51] 10Gerrit, 10Pywikibot: 500 server error when pulling Pywikibot i18n - https://phabricator.wikimedia.org/T329452 (10Dzahn) T287459 T246763 are tickets from the past where a similar error appeared but I am not sure if they tell us a fix [19:20:57] 10Gerrit, 10Pywikibot: 500 server error when pulling Pywikibot i18n - https://phabricator.wikimedia.org/T329452 (10Dzahn) >>! In T329452#8614955, @binbot wrote: > Please help me, how to set a different network for git. I can easily choose proxies from my block list. :-) It should work with settings like this... [19:23:42] 10Gerrit, 10serviceops-collab, 10Patch-For-Review: Issues with Gerrit test instance - https://phabricator.wikimedia.org/T329444 (10Dzahn) @Ameisenigel Yes, I am sorry about that. I was informed myself after the fact that we can't do that. Would the gerrit test instance still be helpful for you if it's up b... [19:24:58] 10Gerrit, 10serviceops-collab, 10Patch-For-Review: Issues with Gerrit test instance - https://phabricator.wikimedia.org/T329444 (10Dzahn) Also we should replace "wmflabs.org" with "wmcloud.org". I already expanded the SSL cert to contain both and created a new proxy in Horizon. [19:29:24] 10Phabricator, 10serviceops-collab: create aphlict2001 (Phabricator realtime notifications codfw) - https://phabricator.wikimedia.org/T322369 (10Dzahn) This VM would not boot. After debugging we found out it's because it's not in DHCP server config. Until recently we had to add the MAC address of a freshly cr... [19:35:20] 10Deployments, 10Release-Engineering-Team (Doing), 10SRE, 10bacula, 10Parsoid (Tracking): Accidental removal of some files under /srv/deployment on deploy1002 - https://phabricator.wikimedia.org/T307349 (10Dzahn) There is a directory "deployment.T307349" under /srv/ on deploy1002 that uses 47GB. And the... [19:37:45] 10Deployments, 10Release-Engineering-Team (Doing), 10SRE, 10bacula, 10Parsoid (Tracking): Accidental removal of some files under /srv/deployment on deploy1002 - https://phabricator.wikimedia.org/T307349 (10Dzahn) I deleted it. 19:36 < mutante> !log root@deploy1002:/srv# rm -rf deployment.T307349/ 19:36... [19:42:17] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥): Periodically run docker-gc on gitlab runners - https://phabricator.wikimedia.org/T295709 (10dancy) 05In progress→03Resolved [19:42:20] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10User-brennen: runner-1002 is out of space - https://phabricator.wikimedia.org/T291221 (10dancy) [19:42:48] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10User-brennen: runner-1002 is out of space - https://phabricator.wikimedia.org/T291221 (10dancy) 05Open→03Resolved [20:02:35] 10Gerrit, 10Pywikibot: 500 server error when pulling Pywikibot i18n - https://phabricator.wikimedia.org/T329452 (10BCornwall) Well, I was going for a more basic approach; My thought was that perhaps there was something odd with the client network (firewall, etc.) So I was thinking of testing other networks lik... [20:19:22] 10Release-Engineering-Team (GitLab V: Event Horizon 🌄): Refactor buildkitd deployment for better build container isolation - https://phabricator.wikimedia.org/T329213 (10dduvall) [20:19:39] 10Beta-Cluster-Infrastructure: beta cluster down - https://phabricator.wikimedia.org/T329592 (10Vgutierrez) The culprit was actually the following hiera config for the `deployment-cache` puppet prefix: ` profile::cache::haproxy::unified_certs: - digicert-2021-ecdsa-unified - digicert-2021-rsa-unified ` I got ri... [20:20:27] 10Beta-Cluster-Infrastructure: beta cluster down - https://phabricator.wikimedia.org/T329592 (10Vgutierrez) 05Open→03Resolved [20:58:53] 10Release-Engineering-Team (GitLab V: Event Horizon 🌄): Run docker-gc on deploy servers - https://phabricator.wikimedia.org/T329678 (10dancy) [21:10:16] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab V: Event Horizon 🌄): Disallow direct access to the docker.io registry from gitlab runners - https://phabricator.wikimedia.org/T329679 (10dancy) [21:11:05] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10mwbot-rs, 10mwcli, 10Patch-For-Review: GitLab CI jobs failing with "You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-... - https://phabricator.wikimedia.org/T329216 [21:11:11] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab V: Event Horizon 🌄): Disallow direct access to the docker.io registry from gitlab runners - https://phabricator.wikimedia.org/T329679 (10dancy) [21:15:38] 10Gerrit, 10Pywikibot: 500 server error when pulling Pywikibot i18n - https://phabricator.wikimedia.org/T329452 (10binbot) >>! In T329452#8615931, @BCornwall wrote: > Well, I was going for a more basic approach; My thought was that perhaps there was something odd with the client network (firewall, etc.) So I w... [21:54:55] 10Gerrit, 10Pywikibot: 500 server error when pulling Pywikibot i18n - https://phabricator.wikimedia.org/T329452 (10binbot) I installed TortoiseGit, the same result. [22:04:21] 10Gerrit, 10serviceops-collab, 10Patch-For-Review: Issues with Gerrit test instance - https://phabricator.wikimedia.org/T329444 (10Ameisenigel) It would be great if one would be able to perform actions at the test instance, but if this is not possible I will look out for another solution. [22:09:59] 10Gerrit, 10serviceops-collab, 10Patch-For-Review: Issues with Gerrit test instance - https://phabricator.wikimedia.org/T329444 (10Dzahn) @Ameisenigel If you can give us another week or so that would be great. I have brought up the topic today but it needs a little time to discuss with others/other team. The... [22:10:48] 10Phabricator, 10serviceops-collab: create aphlict2001 (Phabricator realtime notifications codfw) - https://phabricator.wikimedia.org/T322369 (10ops-monitoring-bot) Cookbook cookbooks.sre.ganeti.reimage was started by eoghan@cumin2002 for host aphlict2001.codfw.wmnet with OS bullseye [22:11:15] 10Gerrit, 10serviceops-collab, 10Patch-For-Review: Issues with Gerrit test instance - https://phabricator.wikimedia.org/T329444 (10Dzahn) The issue is that we can't use Wikitech for auth, so either we'd have to setup a separate LDAP server for this or just disable auth or use another auth method. [22:13:46] 10Gerrit, 10Pywikibot: 500 server error when pulling Pywikibot i18n - https://phabricator.wikimedia.org/T329452 (10BCornwall) @binbot, can you give the output of the following three commands? 1. `git clone --verbose https://gerrit.wikimedia.org/r/pywikibot/i18n.git` 2. `git clone --verbose --recursive https:/... [22:21:19] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T325586 (10cscott) The #content-transform-team has continued to work on the Table of Contents. Potentially risky changes: * https://ger... [22:35:44] 10Gerrit, 10Pywikibot: 500 server error when pulling Pywikibot i18n - https://phabricator.wikimedia.org/T329452 (10binbot) 1. This is very strange, it installed the Pywikibot itself to a directory calles i18n rather than core, with two differences: there is no .git in scripts\i18n subdirectory and it did not w... [22:39:54] 10Phabricator, 10serviceops-collab: create aphlict2001 (Phabricator realtime notifications codfw) - https://phabricator.wikimedia.org/T322369 (10ops-monitoring-bot) Cookbook cookbooks.sre.ganeti.reimage started by eoghan@cumin2002 for host aphlict2001.codfw.wmnet with OS bullseye completed: - aphlict2001 (**PA... [22:52:35] (03PS1) 10Cwhite: Update deploy/phatality-2.4.1.zip for deployment [releng/phatality] - 10https://gerrit.wikimedia.org/r/888737 (https://phabricator.wikimedia.org/T314098) [22:54:43] (03CR) 10Cwhite: "Currently deployed on beta." [releng/phatality] - 10https://gerrit.wikimedia.org/r/888737 (https://phabricator.wikimedia.org/T314098) (owner: 10Cwhite) [22:55:30] (03CR) 10Ahmon Dancy: [C: 03+2] "Let 'er rip!" [releng/phatality] - 10https://gerrit.wikimedia.org/r/888737 (https://phabricator.wikimedia.org/T314098) (owner: 10Cwhite) [22:55:54] (03Merged) 10jenkins-bot: Update deploy/phatality-2.4.1.zip for deployment [releng/phatality] - 10https://gerrit.wikimedia.org/r/888737 (https://phabricator.wikimedia.org/T314098) (owner: 10Cwhite) [23:25:42] (03PS1) 10Cwhite: bugfix: fix typo in legacy logs url reconstruction [releng/phatality] - 10https://gerrit.wikimedia.org/r/888739 (https://phabricator.wikimedia.org/T314098) [23:36:55] (03CR) 10Ahmon Dancy: [C: 03+2] bugfix: fix typo in legacy logs url reconstruction [releng/phatality] - 10https://gerrit.wikimedia.org/r/888739 (https://phabricator.wikimedia.org/T314098) (owner: 10Cwhite) [23:37:19] (03Merged) 10jenkins-bot: bugfix: fix typo in legacy logs url reconstruction [releng/phatality] - 10https://gerrit.wikimedia.org/r/888739 (https://phabricator.wikimedia.org/T314098) (owner: 10Cwhite) [23:45:04] (03PS1) 10Cwhite: Update deploy/phatality-2.4.1.zip for deployment [releng/phatality] - 10https://gerrit.wikimedia.org/r/888741 (https://phabricator.wikimedia.org/T314098) [23:45:47] (03CR) 10Cwhite: "Running on beta." [releng/phatality] - 10https://gerrit.wikimedia.org/r/888741 (https://phabricator.wikimedia.org/T314098) (owner: 10Cwhite) [23:50:41] (03CR) 10Ahmon Dancy: [C: 03+2] Update deploy/phatality-2.4.1.zip for deployment [releng/phatality] - 10https://gerrit.wikimedia.org/r/888741 (https://phabricator.wikimedia.org/T314098) (owner: 10Cwhite) [23:51:05] (03Merged) 10jenkins-bot: Update deploy/phatality-2.4.1.zip for deployment [releng/phatality] - 10https://gerrit.wikimedia.org/r/888741 (https://phabricator.wikimedia.org/T314098) (owner: 10Cwhite)