[03:23:10] (03CR) 10Abijeet Patro: Add banana-i18n library (031 comment) [integration/docroot] - 10https://gerrit.wikimedia.org/r/812953 (owner: 10Abijeet Patro) [10:24:21] (03PS3) 10Hashar: Send events to Wikimedia EventGate [software/gerrit/plugins/events-wikimedia] - 10https://gerrit.wikimedia.org/r/814807 [10:30:45] (03CR) 10Hashar: Send events to Wikimedia EventGate (032 comments) [software/gerrit/plugins/events-wikimedia] - 10https://gerrit.wikimedia.org/r/814807 (owner: 10Hashar) [10:32:12] (03CR) 10Hashar: Send events to Wikimedia EventGate (031 comment) [software/gerrit/plugins/events-wikimedia] - 10https://gerrit.wikimedia.org/r/814807 (owner: 10Hashar) [10:39:11] (03CR) 10Hashar: Send events to Wikimedia EventGate (031 comment) [software/gerrit/plugins/events-wikimedia] - 10https://gerrit.wikimedia.org/r/814807 (owner: 10Hashar) [10:45:46] (03PS4) 10Hashar: Send events to Wikimedia EventGate [software/gerrit/plugins/events-wikimedia] - 10https://gerrit.wikimedia.org/r/814807 [10:48:06] (03CR) 10Hashar: "On commenting with a +1 I have managed to get a ref-updated and comment-added json events dumped to stderr:" [software/gerrit/plugins/events-wikimedia] - 10https://gerrit.wikimedia.org/r/814807 (owner: 10Hashar) [10:51:20] Hi. https://www.mediawiki.org/w/index.php?title=MediaWiki_1.39/wmf.20/Changelog should probably be merged into https://www.mediawiki.org/w/index.php?title=MediaWiki_1.39/wmf.21/Changelog as wmf.20 doesn't exist and the wmf.21 changelog doesn't include patches that would have been in wmf.20 (so unless people know where to look for the wmf.20 changelog, looking for a change is harder). [11:00:27] (03PS5) 10Hashar: Send events to Wikimedia EventGate [software/gerrit/plugins/events-wikimedia] - 10https://gerrit.wikimedia.org/r/814807 [11:01:55] (03CR) 10Hashar: Send events to Wikimedia EventGate (031 comment) [software/gerrit/plugins/events-wikimedia] - 10https://gerrit.wikimedia.org/r/814807 (owner: 10Hashar) [17:00:45] !log deployment-prep: urbanecm@deployment-mwmaint02:~$ mwscript extensions/GrowthExperiments/maintenance/migrateWikitextMentorList.php --wiki=arwiki # T310905 [17:00:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:00:47] T310905: Deploy structured wikitext mentor list to Wikimedia wikis - https://phabricator.wikimedia.org/T310905 [17:15:42] (03PS1) 10Ahmon Dancy: php_fpm.py Remove an unnecessary check from get_batch_size() [tools/scap] - 10https://gerrit.wikimedia.org/r/815333 [17:17:21] (03CR) 10Jeena Huneidi: [C: 03+2] "LGTM" [tools/scap] - 10https://gerrit.wikimedia.org/r/815333 (owner: 10Ahmon Dancy) [17:21:05] (03PS2) 10Ahmon Dancy: php_fpm.py Remove an unnecessary check from get_batch_size() [tools/scap] - 10https://gerrit.wikimedia.org/r/815333 [17:21:07] (03CR) 10CI reject: [V: 04-1] php_fpm.py Remove an unnecessary check from get_batch_size() [tools/scap] - 10https://gerrit.wikimedia.org/r/815333 (owner: 10Ahmon Dancy) [17:21:54] (03CR) 10Jeena Huneidi: [C: 03+2] php_fpm.py Remove an unnecessary check from get_batch_size() [tools/scap] - 10https://gerrit.wikimedia.org/r/815333 (owner: 10Ahmon Dancy) [17:24:53] (03CR) 10CI reject: [V: 04-1] php_fpm.py Remove an unnecessary check from get_batch_size() [tools/scap] - 10https://gerrit.wikimedia.org/r/815333 (owner: 10Ahmon Dancy) [17:27:47] (03PS3) 10Ahmon Dancy: php_fpm.py Remove an unnecessary check from get_batch_size() [tools/scap] - 10https://gerrit.wikimedia.org/r/815333 [17:28:10] (03CR) 10Jeena Huneidi: [C: 03+2] php_fpm.py Remove an unnecessary check from get_batch_size() [tools/scap] - 10https://gerrit.wikimedia.org/r/815333 (owner: 10Ahmon Dancy) [17:36:46] (03Merged) 10jenkins-bot: php_fpm.py Remove an unnecessary check from get_batch_size() [tools/scap] - 10https://gerrit.wikimedia.org/r/815333 (owner: 10Ahmon Dancy) [17:37:58] (03PS1) 10Ahmon Dancy: Release 4.11.1-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/815339 [17:38:00] (03CR) 10Ahmon Dancy: [C: 03+2] Release 4.11.1-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/815339 (owner: 10Ahmon Dancy) [17:40:01] !log Upgrading scap to 4.11.0-1+0~20220719173732.346~1.gbpe07bc9 in beta cluster [17:40:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:43:40] (03Merged) 10jenkins-bot: Release 4.11.1-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/815339 (owner: 10Ahmon Dancy) [18:27:52] (03CR) 10Ebernhardson: [C: 04-2] "not ready to deploy" [tools/release] - 10https://gerrit.wikimedia.org/r/814872 (https://phabricator.wikimedia.org/T313248) (owner: 10Ebernhardson) [19:36:09] jeena: I think we should rollback to wmf.21 in prod until the cache poisoning bug is resolved. any wmf.21 version will keep poisoning enwiki and all other group2/1 wikis despite being only on group0. [19:36:15] to no* [19:36:22] I'm working on a fix meanwhile [19:37:20] okay, roll back to testwikis or roll back all wikis? [19:37:58] all, unless we can ensure there is no traffic to testwikis during the next hour [19:38:17] okay, I will roll back all wikis [19:44:09] (03PS6) 10Hashar: Send events to Wikimedia EventGate [software/gerrit/plugins/events-wikimedia] - 10https://gerrit.wikimedia.org/r/814807 [19:50:22] (03CR) 10Hashar: "I got the dev server from mediawiki/extensions/EventLogging to accept the events as defined from the parent change and proposed at https:/" [software/gerrit/plugins/events-wikimedia] - 10https://gerrit.wikimedia.org/r/814807 (owner: 10Hashar) [20:18:15] jeena: Have cherry-picked Krinkle's wmf.21 fix; should I merge and deploy it? [20:19:11] James_F: that would be great :D I think some backports are underway at the moment [20:19:40] Ack, will make sure I don't scap-lock. ;-) [20:53:35] nice debugging [20:55:23] I hadn't realized we don't segment ACPU by MW version [20:55:50] I'm surprised bugs like this don't occur more often [22:15:58] ori: most keys are segmented by wiki ID, and thus one version (apart from during backport or branch deploys) as makeKey does by default. For makeGlobalKey to be able to purge or set correctly between wikis it needs to cover all. Since multiversion we usually follow the cache version model both to cover this for global keys as well as for the short time when an in-branch deploy is not yet applied to all servers. Which was actually a [22:15:58] fairly common source of bugs during deployments before we adopted that model. Eg values flipping and sometimes ending up for hours/days with the wrong version [22:16:54] But now that we restart fpm after every sync and drop all apcu data, at least that is less likely to happen outside of memc. [22:17:15] Well scratch that, it should be impossible for apcu now. [22:17:16] (03CR) 10Eric Gardner: [C: 03+1] "I am fine to go ahead with this." [integration/config] - 10https://gerrit.wikimedia.org/r/811368 (owner: 10Kosta Harlan) [22:18:01] Though I think we may get that problem back as we move more data to potentially an on-host memc that survives that restart and is shared between several mw pods [23:54:02] Krinkle: oh yes, I remember the ResourceLoader cache issues of yore!