[00:00:34] (DatasourceError) firing: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:10:34] (DatasourceError) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:13:30] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.42.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T350088 (10Jdlrobson) [00:23:00] 10Continuous-Integration-Infrastructure, 10Jenkins: mwext-phpunit-coverage-patch-docker fails to report increase in coverage when base does not have tests/phpunit folder - https://phabricator.wikimedia.org/T353851 (10Dreamy_Jazz) [00:24:01] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: mwext-phpunit-coverage-patch-docker fails to report increase in coverage when base does not have tests/phpunit folder - https://phabricator.wikimedia.org/T353851 (10Dreamy_Jazz) [00:25:14] !log (DiscussionTools:persistRevisionThreadItems done, output: https://phabricator.wikimedia.org/P54503) [00:25:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:31:14] (03PS1) 10Zoranzoki21: Zuul: Add basic quibble CI for the SubpageNavigation extension [integration/config] - 10https://gerrit.wikimedia.org/r/984655 [00:50:24] 10GitLab (Account Approval), 10Release-Engineering-Team, 10User-bd808: Requesting GitLab account activation for russblau - https://phabricator.wikimedia.org/T353585 (10bd808) 05Open→03Resolved a:03bd808 Not sure who did the needful, but https://gitlab.wikimedia.org/russblau is active [00:58:03] 10GitLab (Account Approval), 10Release-Engineering-Team, 10User-bd808: Requesting GitLab account activation for hunsvotti - https://phabricator.wikimedia.org/T353431 (10bd808) 05Open→03Resolved a:03bd808 https://gitlab.wikimedia.org/hunsvotti should be working now. The issue here appeared to be that wh... [01:09:49] 10GitLab (Account Approval), 10Release-Engineering-Team: Requesting GitLab account activation for LD - https://phabricator.wikimedia.org/T353581 (10bd808) @LD: your Developer account record in the backing LDAP directory is missing an email address. This is likely fallout from the {T339917} bug. Because your LD... [01:15:52] 10GitLab (Account Approval): Correct and then approve https://gitlab.wikimedia.org/dnshitobu - https://phabricator.wikimedia.org/T353856 (10bd808) [01:19:59] 10GitLab (Account Approval): Correct and then approve https://gitlab.wikimedia.org/sethabathaba - https://phabricator.wikimedia.org/T353857 (10bd808) [01:21:22] 10GitLab (Account Approval): Correct and then approve https://gitlab.wikimedia.org/sethabathaba - https://phabricator.wikimedia.org/T353857 (10bd808) [01:21:46] 10GitLab (Account Approval): Correct and then approve https://gitlab.wikimedia.org/dnshitobu - https://phabricator.wikimedia.org/T353856 (10bd808) [10:06:32] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.42.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T350086 (10matmarex) [10:50:10] when running a maintenance script on the beta cluster, is it possible to increase the memory limit? [10:50:23] i would like to run this: mwscript DiscussionTools:persistRevisionThreadItems --wiki=enwiktionary --all --current [10:50:37] but it crashes with: mmap() failed: [12] Cannot allocate memory [10:50:37] Fatal error: Out of memory (allocated 3140497408) (tried to allocate 20480 bytes) in /srv/mediawiki/php-master/vendor/wikimedia/parsoid/src/Wt2Html/Grammar.php on line 4804 [10:51:19] that's about 3 GB, which doesn't seem like that much. surely the machine has more than that to give [10:52:34] `free -h` says 3,9Gi on deployment-mwmaint02 [10:52:59] and the type is g3.cores2.ram4.disk20 according to https://openstack-browser.toolforge.org/project/deployment-prep [10:53:28] yeah, i just figured out `free`. so i guess it really is that small [10:53:34] tbh if the script needs that much memory I’d also be worried about running it in production… production mwmaint probably has some more memory, but production databases are also much larger than beta databases [10:53:42] apparently deployment-deploy03 is bigger. would anyone be mad i i ran the script there? [10:54:31] it's probably a bug that it takes so much memory, there are some tasks about Parsoid out-of-memory errors. but it's not infinite memory, and i don't very much want to investigate today :) [10:56:19] I’d prefer to fix the bug, but that’s easy for me to say of course ^^ [10:56:44] but can you resume the script from where it crashed? using the --start option that it seems to print? [10:57:21] it crashes right at the beginning [10:57:29] o_O [10:57:32] ok that’s weird [10:58:01] there's probably some huge page or something that's causing the high memory usage [10:58:17] i ran that script on most of the other beta wikis yesterday successfully [10:59:21] i would ordinarily be able to tell you which page, but the script just exits after printing the error, instead of doing all of the normal error handling that would report the problematic page :D [10:59:52] does it at least print the first --start line? [11:00:11] (since it looks like that should happen before it starts processing result rows) [11:02:20] hmm, only prints "Processing... (estimated 2112 rows)". i guess that's a bit weird [11:02:41] unless it's getting overprinted in the terminal or something [11:03:04] o_O [11:04:01] (i retried and redirected to a file now, nope, no other output) [11:05:24] hm, it's probably buffering output, and can't print it before crashing? [11:05:31] possibly [11:05:46] though at least `Maintenance` just `print`s directly afaict, it doesn’t buffer internally [11:05:53] maybe PHP buffers it [11:06:26] I think it would be fine for you to add debug statements to the code (as long as you can live with them being overwritten every 10 minutes) [11:06:31] if you want to investigate it [11:06:35] it probably does, i never encountered this with PHP before, but i did in othe rlanguages [11:06:50] maybe adding some flush() https://www.php.net/manual/en/function.flush.php calls would make it output things [11:06:55] touch('/tmp/some-path') to indicate that the code was reached ^^ [11:07:14] alright, i'll fiddle with that for a few minutes and see if i can get output [11:09:17] maybe print the $qb->getSQL() so you can run the query in mwscript mysql and see which titles it’s choking on [11:12:28] Lucas_WMDE: oh, the script isn't supposed to print the --start line on the first loop :D duh. maybe we changed that at some point [11:12:32] anyway, i added some echos [11:12:36] here's the bad page: https://en.wiktionary.beta.wmflabs.org/wiki/Wiktionary:Why_create_an_account%3F lmao [11:12:49] MatmaRex: ah, derp :D [11:12:54] ok then it makes a bit more sense [11:13:03] wait, enwiktionary? [11:13:15] yeah. it's 2 megabytes of exclamation marks [11:13:20] oh, so you wrote [11:13:24] I just missed that part [11:13:28] pffffft [11:13:38] let’s delete the page then? ^^ [11:13:39] (which maybe shouldn't consume 3 gigabytes of memory. but still, it's just a weird page) [11:13:42] * Lucas_WMDE checks what links there [11:14:39] MatmaRex: honey I shrunk the page [11:14:55] :D [11:16:48] https://en.wiktionary.beta.wmflabs.org/wiki/Template:editnotice_load was another one with tons of exclamation marks [11:17:51] and another one seen in https://en.wiktionary.beta.wmflabs.org/wiki/Special:LongPages [11:17:58] (also updated) [11:18:03] it seems like there are more pages, since it's still crashing. it seems a bit silly trying to fix them [11:18:13] oh, i guess you got them all now [11:18:24] i didn't tihnk of LongPages [11:18:46] `SELECT page_namespace, page_title, page_len FROM page ORDER BY page_len DESC LIMIT 25;` finds a couple more [11:18:54] (I guess LongPages is filtered by namespace?) [11:20:40] i'll file a parsoid bug about this btw [11:22:10] ok thanks [11:22:17] and i think i will try running the script on deployment-deploy03, to see if it works. it should be quick when it's not crashing [11:27:25] tbh I’m not sure how much that is parsoid’s fault. if we can’t even hold two copies of the raw page contents in memory at the same time, how can we reasonably parse it [11:27:55] that feels like mwmaint02 might be underprovisioned for our current $wgMaxArticleSize [11:28:38] filed https://phabricator.wikimedia.org/T353874 [11:28:48] apparently deployment-mediawiki11 and deployment-mediawiki12 have 8g of ram each, I guess that’s how the page could be saved at all [11:29:02] no, i blame parsoid, it takes more than 1000x the article size. surely that's too much [11:29:17] *counts* [11:29:20] oh yeah you’re right [11:29:23] I skipped an SI level [11:29:27] the page isn’t two gigs long [11:29:28] the page content is in MB, memory use is in GB [11:29:30] thanks [11:29:31] yeah [11:35:42] Project beta-code-update-eqiad build #475442: 04FAILURE in 2 min 41 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/475442/ [11:35:42] Project beta-update-databases-eqiad build #72582: 04FAILURE in 15 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/72582/ [11:35:51] started it on deployment-deploy03, it hasn't crashed yet, but doesn't seem to be going very fast either [11:36:11] i hope that alert isn't my fault [11:37:24] (i stopped it) [11:38:12] according to the CI logs deploy03 stopped responding in some way 😬 [11:38:33] let’s see if the next run in ten minutes works better? [11:39:02] i actually couldn't ssh into it from another terminal [11:39:07] so it probably was my faukt [11:39:20] i think i'll leave this here :D [11:40:22] maybe i'll try again after the holidays, hopefully the parsoid bug is fixed [11:45:47] I can ssh to it now [11:45:50] so hopefully it recovered [11:46:43] Yippee, build fixed! [11:46:44] Project beta-code-update-eqiad build #475443: 09FIXED in 3 min 43 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/475443/ [11:55:59] 10Release-Engineering-Team, 10Traffic, 10collaboration-services, 10Patch-For-Review: CI on gitlab for eBPF / networking heavy projects - https://phabricator.wikimedia.org/T353279 (10CodeReviewBot) vgutierrez opened https://gitlab.wikimedia.org/repos/sre/tcp-mss-clamper/-/merge_requests/11 clamper: Provide... [12:30:31] Yippee, build fixed! [12:30:31] Project beta-update-databases-eqiad build #72583: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/72583/ [13:42:51] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10Patch-For-Review, 10SecTeam-Processed, and 2 others: Create rule-filtering functionality for semgrep-merge-tool rule repos - https://phabricator.wikimedia.org/T353536 (10CodeReviewBot) mmartorana merged https://gitlab.wikimedia.org/repos/security/s... [15:33:09] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.42.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T350086 (10matmarex) [15:46:07] 10Release-Engineering-Team (Radar), 10DBA, 10MediaWiki-libs-Rdbms, 10Observability-Logging, and 2 others: Transaction profiler logs full query which is truncated by logstash - https://phabricator.wikimedia.org/T349140 (10Ladsgroup) In the last day we had only 14 cases of json truncated and mostly they are... [16:06:46] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10SecTeam-Processed, 10Security, 10user-sbassett: Create rule-filtering functionality for semgrep-merge-tool rule repos - https://phabricator.wikimedia.org/T353536 (10sbassett) **New tag:** https://gitlab.wikimedia.org/repos/security/semgrep-merge-... [16:21:32] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10SecTeam-Processed, 10Security, 10user-sbassett: Create rule-filtering functionality for semgrep-merge-tool rule repos - https://phabricator.wikimedia.org/T353536 (10sbassett) So, after testing a bit more, it looks like the affected upstream rules... [17:01:02] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.42-release: Expand the set of bundled extensions and skins in MediaWiki 1.42 - https://phabricator.wikimedia.org/T333405 (10Reedy) [17:01:14] 10MediaWiki-Releasing, 10MW-1.41-release: Write release announcement for 1.41.0-rc.1 - https://phabricator.wikimedia.org/T346932 (10Reedy) 05Open→03Declined [17:01:20] 10MediaWiki-Releasing, 10MW-1.41-release, 10Release: Release MW 1.41.0-rc.1 (if needed) - https://phabricator.wikimedia.org/T346931 (10Reedy) [17:01:26] 10MediaWiki-Releasing, 10MW-1.41-release, 10Release: Release MW 1.41.0-rc.1 (if needed) - https://phabricator.wikimedia.org/T346931 (10Reedy) 05Open→03Declined [17:01:32] 10MediaWiki-Releasing, 10MW-1.41-release, 10Release: Release MediaWiki 1.41.0 - https://phabricator.wikimedia.org/T346919 (10Reedy) [17:01:40] (03PS1) 10Ahmon Dancy: mediawiki-phan-php*: update run.sh and run-core.sh [integration/config] - 10https://gerrit.wikimedia.org/r/984867 [17:01:46] 10MediaWiki-Releasing, 10Documentation, 10MW-1.41-release: Finish up RELEASE-NOTES for 1.41.0 - https://phabricator.wikimedia.org/T346933 (10Reedy) 05Open→03Resolved [17:01:48] 10MediaWiki-Releasing, 10MW-1.41-release, 10Release: Release MediaWiki 1.41.0 - https://phabricator.wikimedia.org/T346919 (10Reedy) [19:06:33] (03PS1) 10Reedy: Drop REL1_35 support in CI [integration/config] - 10https://gerrit.wikimedia.org/r/984986 (https://phabricator.wikimedia.org/T340396) [19:11:06] (03CR) 10Reedy: Drop REL1_35 support in CI (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/984986 (https://phabricator.wikimedia.org/T340396) (owner: 10Reedy) [19:12:32] (03CR) 10Reedy: "Do we just want to remove REL1_35 from those two comments?" [integration/config] - 10https://gerrit.wikimedia.org/r/984986 (https://phabricator.wikimedia.org/T340396) (owner: 10Reedy) [19:15:20] 10MediaWiki-Releasing, 10MW-1.41-release, 10Patch-For-Review, 10Release: Release MediaWiki 1.41.0 - https://phabricator.wikimedia.org/T346919 (10Reedy) [19:15:26] 10MediaWiki-Releasing, 10MW-1.41-release, 10Patch-For-Review, 10Release: Release MediaWiki 1.41.0 - https://phabricator.wikimedia.org/T346919 (10Reedy) [19:20:10] (03PS2) 10Reedy: Drop REL1_35 support in CI [integration/config] - 10https://gerrit.wikimedia.org/r/984986 (https://phabricator.wikimedia.org/T340396) [19:21:16] Thanks thcipriani! [19:51:26] 10GitLab (Account Approval): Correct and then approve https://gitlab.wikimedia.org/dnshitobu - https://phabricator.wikimedia.org/T353856 (10Dnshitobu) @bd808 I have updated my email now and you could check back again. [20:29:49] 10Release-Engineering-Team (Radar), 10MediaWiki-Release-Tools: mediawiki/extensions and mediawiki/skins missing the REL1_XX branches - https://phabricator.wikimedia.org/T264365 (10Umherirrender) This would conflict with {T351341} [20:49:43] 10GitLab (Account Approval), 10User-bd808: Correct and then approve https://gitlab.wikimedia.org/dnshitobu - https://phabricator.wikimedia.org/T353856 (10bd808) 05Open→03Resolved a:03bd808 @Dnshitobu Your https://gitlab.wikimedia.org/dnshitobu GitLab account should be ready to use now. It turns out that... [21:01:11] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.42.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T350086 (10dancy) 05Open→03Resolved [21:52:16] 10Phabricator, 10Legalpad, 10WMF-Legal, 10WMF-NDA-Requests, 10User-AKlapper: Clarify if NDAs (to access #WMF-NDA protected Phab tasks) are on paper or in Legalpad's L2 or both - https://phabricator.wikimedia.org/T349595 (10KFrancis) Question... Historically, do you have a policy or guideline on your end... [22:00:43] 10GitLab (Auth & Access), 10Release-Engineering-Team (Priority Backlog 📥), 10gitlab-settings, 10User-brennen: Create an approved-accounts project in GitLab for distributing account approval process to non-administrators - https://phabricator.wikimedia.org/T350832 (10bd808) With https://wikitech.wikimedia.o... [22:00:55] 10GitLab (Auth & Access), 10Release-Engineering-Team (Priority Backlog 📥), 10gitlab-settings, 10Tool-gitlab-account-approval, 10User-brennen: Create an approved-accounts project in GitLab for distributing account approval process to non-administrators - https://phabricator.wikimedia.org/T350832 (10bd808) [22:20:01] Project beta-update-databases-eqiad build #72593: 04FAILURE in 0.71 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/72593/ [22:21:29] what did I break [22:21:46] oh, I'm guessing it didn't update everything it needed [22:22:16] heh, ^ Amir1 needs https://gerrit.wikimedia.org/r/c/mediawiki/core/+/985006 [22:22:47] Done! [23:12:26] Yippee, build fixed! [23:12:26] Project beta-update-databases-eqiad build #72594: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/72594/ [23:54:08] 10Release-Engineering-Team, 10Diffusion-Repository-Administrators, 10Projects-Cleanup: Archive Gerrit repositories "operations/software/hhvm-dev*" (20141017) - https://phabricator.wikimedia.org/T351600 (10thcipriani) [23:55:14] 10Release-Engineering-Team, 10Diffusion-Repository-Administrators, 10Projects-Cleanup: Archive Gerrit repositories "operations/software/hhvm-dev*" (20141017) - https://phabricator.wikimedia.org/T351600 (10thcipriani) 05Open→03Resolved a:03thcipriani Did this today based on SRE feedback in {T343707} (sp...