[00:40:04] Jdlrobson: Never seen that before, sorry. [00:56:56] thcipriani: for next week, can we make sure the Tuesday train happens in the "secondary timeslot" so it happens after the switchover? https://wikitech.wikimedia.org/wiki/Deployments#Tuesday,_September_14 [00:58:36] It's assigned to h.ashar with t.wentyafterfour as backup, so that's do-able. [01:00:05] ^ [01:01:05] let me write that down so I make sure it happens. secondary == 19:00UTC? [01:01:20] And I guess we should delete it from the calendar? [01:01:32] the EU one? yeah. [01:01:39] Put in a DO NOT DEPLOY shouty window for two hours before the DC switch-over. [01:01:51] that's not a bad idea [01:02:14] ^ useful legoktm ? [01:02:37] could extend the 12UTC pre-train break for a bit... [01:02:42] yes please [01:02:55] Done. [01:03:03] https://wikitech.wikimedia.org/w/index.php?title=Deployments&diff=1924675&oldid=1924674 [01:03:29] <3 [01:03:47] ty :)) [01:03:52] Doing the REL1_37 branch cut at that point would probably be fine, though. So if Antoine is bored… ;-) [03:09:09] PROBLEM - SSH on contint2001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [04:10:05] RECOVERY - SSH on contint2001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [08:08:24] (03CR) 10Hashar: WIP add patch author as reviewer to promote patch (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719319 (https://phabricator.wikimedia.org/T281392) (owner: 10Jeena Huneidi) [08:21:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing), 10Release Pipeline, 10bacula: CI backups on contint1001 generating 6GB of file metadata- not happening before- potentially slowing down or making impossible a recovery - https://phabricator.wikimedia.org/T290437 (10hashar) 05Open... [09:01:02] (03CR) 10Elukey: [C: 03+1] helm-linter: Add a helmfile_log_sal stub [integration/config] - 10https://gerrit.wikimedia.org/r/719548 (owner: 10JMeybohm) [09:01:37] (03CR) 10Elukey: [C: 03+1] jjb: update helm-linter job to releng/helm-linter:0.2.17 [integration/config] - 10https://gerrit.wikimedia.org/r/719549 (owner: 10JMeybohm) [09:56:25] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Undefined variable: wgFileBlacklist - https://phabricator.wikimedia.org/T290640 (10dom_walden) [09:59:11] 10Beta-Cluster-Infrastructure, 10MediaWiki-Uploading, 10Beta-Cluster-reproducible: Undefined variable: wgFileBlacklist - https://phabricator.wikimedia.org/T290640 (10dom_walden) [10:00:05] (03CR) 10Elukey: [C: 03+2] "Seems easy enough to submit without the explicit Releng's +2 (and also very easy to revert in case)." [integration/config] - 10https://gerrit.wikimedia.org/r/719548 (owner: 10JMeybohm) [10:02:22] (03Merged) 10jenkins-bot: helm-linter: Add a helmfile_log_sal stub [integration/config] - 10https://gerrit.wikimedia.org/r/719548 (owner: 10JMeybohm) [10:12:25] (03CR) 10Elukey: "Nevermind, we need to wait for the image to be rebuilt (IIUC it is not done automatically)." [integration/config] - 10https://gerrit.wikimedia.org/r/719548 (owner: 10JMeybohm) [10:26:13] (03CR) 10Hashar: [C: 03+2] "Job updated" [integration/config] - 10https://gerrit.wikimedia.org/r/719549 (owner: 10JMeybohm) [10:28:00] (03Merged) 10jenkins-bot: jjb: update helm-linter job to releng/helm-linter:0.2.17 [integration/config] - 10https://gerrit.wikimedia.org/r/719549 (owner: 10JMeybohm) [10:28:13] (03CR) 10Hashar: "I apologize, that is 100% my fault I should have tested it live after deployment." [integration/config] - 10https://gerrit.wikimedia.org/r/719493 (https://phabricator.wikimedia.org/T290584) (owner: 10Urbanecm) [10:30:24] (03PS3) 10Hashar: Add extension-gate to ProofreadPage again [integration/config] - 10https://gerrit.wikimedia.org/r/719496 (https://phabricator.wikimedia.org/T289465) (owner: 10Tpt) [10:30:50] (03CR) 10Hashar: [C: 03+2] "I have amended the commit message to ping T289465 and T290584" [integration/config] - 10https://gerrit.wikimedia.org/r/719496 (https://phabricator.wikimedia.org/T289465) (owner: 10Tpt) [10:32:12] (03Merged) 10jenkins-bot: Add extension-gate to ProofreadPage again [integration/config] - 10https://gerrit.wikimedia.org/r/719496 (https://phabricator.wikimedia.org/T289465) (owner: 10Tpt) [10:33:59] * urbanecm hopes in no new CI failures 🙂 [10:37:21] !log Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/719496 [10:37:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:37:26] urbanecm: will test it this time :) [10:37:34] thanks hashar [10:37:50] with a dummy change https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ProofreadPage/+/442028 [10:38:08] yesterday I did a recheck but forgot to watch the jobs results or monitor what was going on :\ [10:38:43] :/ [10:41:25] hashar: I don't think anyone built the docker image for https://gerrit.wikimedia.org/r/c/integration/config/+/719549/ [10:41:42] yeah effie told me so :( [10:41:56] thank you! [10:42:33] hi folks! Do you know how to trigger a rebuild of the releng/helm-linter docker image? [10:42:44] context is https://gerrit.wikimedia.org/r/c/integration/config/+/719548 [10:43:55] (03CR) 10Hashar: "The image should be build immediately after the change got merged ( ./fab deploy_docker ). I am triggering the build right now ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/719548 (owner: 10JMeybohm) [10:45:10] 10Beta-Cluster-Infrastructure, 10MediaWiki-Uploading, 10Beta-Cluster-reproducible: Undefined variable: wgFileBlacklist - https://phabricator.wikimedia.org/T290640 (10Zabe) Caused by {rMW4dae3b1a060acaf473394ca85a4a324933a7004c}. [10:56:01] !log Successfully published image docker-registry.discovery.wmnet/releng/helm-linter:0.2.17 [10:56:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:06:26] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: CI Docker images failling to build - https://phabricator.wikimedia.org/T290651 (10hashar) [11:12:23] (03CR) 10Hashar: "I have triggered a dummy build via https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ProofreadPage/+/442028 and it passed this time!" [integration/config] - 10https://gerrit.wikimedia.org/r/719496 (https://phabricator.wikimedia.org/T289465) (owner: 10Tpt) [11:12:43] urbanecm: gate passed with ProofreadPage !! https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ProofreadPage/+/442028 :] [11:12:50] good news! [12:00:23] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: CI Docker images failling to build - https://phabricator.wikimedia.org/T290651 (10hashar) I have looked a bit at node12-test-browser-php80-composer, versioned dependencies are missing on bullseye: | Package | Dependency | Bullseye |--|--|--... [12:03:19] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: CI Docker images failling to build - https://phabricator.wikimedia.org/T290651 (10hashar) ` dockerfiles/node10-test-browser-php80-composer/Dockerfile.template:25: echo "deb https://packages.sury.org/php/ stretch main" > /etc/apt/sources.list.... [13:25:50] Daimona feel free to add me as a reviewer to the phan patches that you change, reviewer bot should be adding me to the libup patches that change composer.json but it isn't [13:26:18] Thanks, I think most patches are out [13:27:07] See https://gerrit.wikimedia.org/r/q/hashtag:%2522c%253Bmediawiki/mediawiki-phan-config%253D0.11.0%2522+status:open [13:41:12] everything that has V+2 should be merging now [13:44:17] 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T281164 (10DannyS712) [13:51:18] (03PS1) 10Hashar: dockerfiles: fix sury component [integration/config] - 10https://gerrit.wikimedia.org/r/720020 (https://phabricator.wikimedia.org/T290651) [13:54:20] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: CI Docker images failling to build - https://phabricator.wikimedia.org/T290651 (10hashar) That should fix it for the php80 based images. For the php72 ones, I guess they got moved to Buster or Bullseye but since they: `... [14:04:37] (03CR) 10Hashar: [C: 03+1] "I got both images built locally :]" [integration/config] - 10https://gerrit.wikimedia.org/r/720020 (https://phabricator.wikimedia.org/T290651) (owner: 10Hashar) [14:15:58] (03PS2) 10Hashar: dockerfiles: fix sury component [integration/config] - 10https://gerrit.wikimedia.org/r/720020 (https://phabricator.wikimedia.org/T290651) [14:16:49] (03CR) 10Hashar: [C: 03+1] "Further enhanced to have the distribution dynamically extracted from /etc/os-release. This way we are covered for the next distro upgrade." [integration/config] - 10https://gerrit.wikimedia.org/r/720020 (https://phabricator.wikimedia.org/T290651) (owner: 10Hashar) [14:27:53] Daimona: seems there's a bug in the new Phan classifying code after trigger_error as unreachable. [14:28:01] https://integration.wikimedia.org/ci/job/mwext-php72-phan-docker/134552/console [14:28:13] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaEvents/+/720002/1..2 [14:29:18] DannyS712: ^ this was not unreachable and needs to be reverted, it's changing the behaviour and causing an undefined variable. I'm guessing it's not tested because in phpunit we stop on warnings, but not in production [14:29:26] Krinkle: yes, but it should be the case, no? [14:29:30] Oh wait [14:29:40] set_error_handler I guess? [14:30:20] Sure you could make it fatal but that's not normal behaviour. There is no reason to associate trigger_error as never returning. That would be custom [14:30:55] Under default and prod conditions that just logs to the error log [14:31:10] Eg deprecation warnings [14:33:21] Shouldn't E_USER_ERROR cause a fatal by default? [14:33:28] No [14:33:40] The constants are different string prefixes [14:33:46] Notice info warning error [14:34:00] Are all just warning messsyes to stderr [14:35:24] If it something wants to not return, it would throw. This fn exists only to emit logged errors [14:38:36] Am I getting anything wrong? https://3v4l.org/feg3O#v7.3.30 I was remembering this behaviour [14:49:55] Daimona: ah, 'E_USER_ERROR' specifically seems to behave differently. [14:50:10] Yes [14:50:11] I mean [14:50:20] It can be ignored via set_error_handler [14:50:44] I think phan makes the assumption that this is not the case [14:50:48] sure [14:51:49] but that makes the situation worse - I don't think we should ever use E_USER_ERROR because it presumably bypasses most error handlers and such since it's not a throwable. And most cerainly, it was not intended in this extension. IT should probalby use E_USER_WARNING instead. [14:52:04] And maybe sniff against E_USER_ERROR. Anyone using that probably intended E_USER_WARNING. or should throw instead. [14:52:08] (or exit) [14:59:33] > Woe! This request had its journey cut short by unexpected circumstances (Can Not Connect to MySQL). [14:59:35] Phab [14:59:51] looks like the task made it after all [14:59:54] https://phabricator.wikimedia.org/T290669 [15:00:37] Daimona: so, I take it Phan understands the different ways to use this and defaults to considering it as not fatal? [15:01:42] I agree that this usage of trigger_error is problematic [15:02:16] I'm not sure what phan does, but I guess it assumes that if you use E_USER_ERROR, then trigger_error is kinda equivalent to exit() [15:03:10] Yup https://phan.github.io/demo/?code=%3C%3Fphp%0A%0Afunction+notice%28%29%7B%0A++++trigger_error%28%27foo%27%2CE_USER_NOTICE%29%3B%0A++++echo+%27notice%27%3B%0A%7D%0Anotice%28%29%3B%0A%0Afunction+error%28%29%7B%0A++++trigger_error%28%27foo%27%2CE_USER_ERROR%29%3B%0A++++echo+%27error%27%3B%0A%7D%0Aerror%28%29%3B [15:08:55] who needs server-side rendering, if you can have client-side PHP in WASM. [15:09:16] That's a pretty cool demo, I keep forgetting this exists. [15:14:05] Yeah, it helps a lot [15:14:31] We also use a fork for taint-check https://doc.wikimedia.org/mediawiki-tools-phan-SecurityCheckPlugin/master/demos/ [15:17:56] Daimona, Krinkle: see also https://gerrit.wikimedia.org/r/c/oojs/ui/+/718461 for a legit use of trigger_error( ..., E_USER_ERROR ) [15:18:55] legoktm: I'm not sure it is. I believe that's the same mistake I made. [15:19:08] and untested for the same reason [15:19:45] hmm [15:20:23] I'm not sure what the alternative is though, __toString() can't throw until we're at PHP 7.4 [15:20:47] the fix is to use _WARNING instead [15:20:49] it's not meant to be fatal [15:21:05] im surprised php allowed/allows trigger_error to fatal [15:22:24] but maybe yeah, we can make an exclusion locally if it's really intentiona [15:28:35] 10Beta-Cluster-Infrastructure, 10DiscussionTools: The default mode for Reply tool is not consistent for logged out users on beta ko.wp - https://phabricator.wikimedia.org/T290112 (10ppelberg) [15:36:54] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Add Python 3.9 to Wikimedia CI - https://phabricator.wikimedia.org/T289222 (10hashar) The CI images have `http://apt.wikimedia.org/wikimedia buster-wikimedia thirdparty/pyall` and Python 3.8 got added to it via T241195. Adding python 3.9 to it seem... [15:43:19] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10Scap: Scap should be clearer about the need for a revert after a failed canary check - https://phabricator.wikimedia.org/T290037 (10dancy) @Tgr How about this? ` 18:48:22 Scap failed!: 5/6 canaries failed their endpoint checks(https://en.wikipedia.org). WARNING:... [16:25:49] yak yak yak yak yak shavveeee [16:54:24] hi releng! Is the dev-images gerrit repo set to readonly on purpose? https://gerrit.wikimedia.org/r/admin/repos/releng/dev-images [16:54:46] it moved to gitlab I believe [16:54:49] I'd like to add a new image for the fundraising vagrant environment [16:54:53] oho! [16:54:56] I [16:55:06] 'll update my remotes and learn the new review process then [16:55:10] thanks legoktm [16:55:11] https://gitlab.wikimedia.org/releng/dev-images [16:55:17] :) [16:57:49] (03PS3) 10Dduvall: Perform validation using JSON schema and ajv-cli [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719382 (https://phabricator.wikimedia.org/T225335) [16:59:43] (03CR) 10jerkins-bot: [V: 04-1] Perform validation using JSON schema and ajv-cli [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719382 (https://phabricator.wikimedia.org/T225335) (owner: 10Dduvall) [17:03:41] 10Release-Engineering-Team (Radar), 10Quality-and-Test-Engineering-Team (QTE), 10serviceops-radar, 10CommRel-Specialists-Support (Jul-Sep-2021), and 2 others: Expand the list of group 1 wikis to contain at least one (preferably 2) smaller "top ten size" wikis - https://phabricator.wikimedia.org/T286664 (10J... [17:07:53] legoktm: so I'm trying to figure out how to create a merge request there - Do I have to fork the repo within GitLab and push to my fork there, or should I ask for push rights to the main repo and just push to a new branch? [17:08:08] I have no clue about that. [17:08:18] k, lemme look for more docs [17:12:04] why do we have stuff on gitlab if it's still restricted to wmf/nda [17:20:37] ok, I requested access and also made a fork for this change: https://gitlab.wikimedia.org/releng/dev-images/-/merge_requests/4 [17:20:56] looks like wikibugs doesn't announce gitlabs merge requests yet? [17:22:12] no, see https://phabricator.wikimedia.org/T288381 [17:22:38] AntiComposite: I thought it had been opened up already? if that didn't happen yet, yeah, that's pretty problematic [17:22:48] nope, just tried it. [17:22:56] insufficient permissions. [17:23:31] https://phabricator.wikimedia.org/T288162 [17:23:45] yeah, just found https://gerrit.wikimedia.org/r/c/operations/puppet/+/710083 [17:24:43] * majavah loves the "soonish" from over a month ago [17:34:51] it's not like "public access" is a core feature for a public code review system or anything [17:42:01] I bumped the Gerrit patch [17:45:14] (03PS7) 10Michael Große: Add and use job to deploy branch in node project [integration/config] - 10https://gerrit.wikimedia.org/r/683589 (https://phabricator.wikimedia.org/T278706) [17:46:05] brennen: A comment of some kind instead of a -1 would be helpful/appreciated [17:46:41] thanks [17:48:32] 10Release-Engineering-Team (Doing), 10GitLab, 10Infrastructure-Foundations, 10CAS-SSO, and 2 others: Open gitlab.wikimedia.org to all users with Wikimedia developer accounts - https://phabricator.wikimedia.org/T288162 (10Legoktm) Per Gerrit, this is blocked on {T288392}. [17:48:38] 10Release-Engineering-Team (Doing), 10GitLab, 10Infrastructure-Foundations, 10CAS-SSO, and 2 others: Open gitlab.wikimedia.org to all users with Wikimedia developer accounts - https://phabricator.wikimedia.org/T288162 (10Legoktm) [17:48:44] 10Release-Engineering-Team (Doing), 10GitLab, 10Patch-For-Review, 10Privacy, 10User-brennen: GitLab uses 'real name' as username (rather than 'shell name' or an user-specified name) - https://phabricator.wikimedia.org/T288392 (10Legoktm) [17:49:02] 10Release-Engineering-Team (Doing), 10GitLab, 10Infrastructure-Foundations, 10CAS-SSO, and 2 others: Open gitlab.wikimedia.org to all users with Wikimedia developer accounts - https://phabricator.wikimedia.org/T288162 (10Legoktm) 05Open→03Stalled [17:51:23] tbh my impulse would be to decline T288392 except for what t.gr raises here: https://phabricator.wikimedia.org/T288392#7315915 [17:51:24] T288392: GitLab uses 'real name' as username (rather than 'shell name' or an user-specified name) - https://phabricator.wikimedia.org/T288392 [17:53:05] 10Release-Engineering-Team (Doing), 10GitLab, 10Infrastructure-Foundations, 10CAS-SSO, and 2 others: Open gitlab.wikimedia.org to all users with Wikimedia developer accounts - https://phabricator.wikimedia.org/T288162 (10brennen) p:05Triage→03High [17:54:12] yeah, that seems pretty problematic [17:54:55] but the way I interpreted your comment, opening gitlab up is blocked on a resolution of that task, regardless of what that resolution is [17:55:17] that is correct. [17:55:27] apologies for not having that modeled correctly in phabricator. [18:04:51] :) no worries [18:05:30] ejegg|food: for when you're back - i added a people/wmf-team-fundraising-tech group, that group should now have maintainer access to releng/dev-images [18:06:20] thanks! [19:39:07] (03CR) 10Jforrester: [C: 03+2] dockerfiles: fix sury component [integration/config] - 10https://gerrit.wikimedia.org/r/720020 (https://phabricator.wikimedia.org/T290651) (owner: 10Hashar) [19:39:09] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: CI Docker images failling to build - https://phabricator.wikimedia.org/T290651 (10Jdforrester-WMF) Aha, thanks. Was meaning to get to this this weekend. [19:40:25] (03Merged) 10jenkins-bot: dockerfiles: fix sury component [integration/config] - 10https://gerrit.wikimedia.org/r/720020 (https://phabricator.wikimedia.org/T290651) (owner: 10Hashar) [19:42:11] !log Docker: Building node{10,12}-test-browser-php80-composer for T290651 [19:42:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:42:14] T290651: CI Docker images failling to build - https://phabricator.wikimedia.org/T290651 [19:52:26] Daimona: curious if maybe you know how to tame Phan to accept this patch :D https://gerrit.wikimedia.org/r/c/mediawiki/core/+/719146 [19:52:41] TLDR: There's a bad phpdoc comment in vendor/statsd-client [19:52:45] Let me see [19:52:46] it says @return array where it shouldn't [19:53:20] without @method, it says I declare it in an incompatible way, as if the @return upstream was a real return type [19:53:25] with the @method, it says I declare it twice [19:53:51] I guess @method is mainly for magic __call so probably not the right tag, but couldn't find a better one [19:54:35] I don't think you can have @method together with a concrete definition [19:54:42] Although @method would be suitable for this use case [19:54:53] Could you just suppress the mismatching signature issue? [19:55:28] hi folks, does releng manage doc.wikimedia.org ? [19:55:48] It seems to be unable to load documentation pages [19:56:00] oh oops, might just be for me [19:56:16] seeing some 404s on bg requests and the gears just keep spinning [19:56:23] but cstone says she can load it [19:56:30] e.g. https://doc.wikimedia.org/mediawiki-core/master/js/ [19:58:58] huh, even after force-reload I still see bg requests failing for e.g. https://doc.wikimedia.org/mediawiki-core/master/js/data-e6972899f60561a3f0a43cf6ce5211a1.js [20:01:52] ejegg, the js doc for master occasionally does that, not sure why or if there's a task for it, I usually just use https://doc.wikimedia.org/mediawiki-core/REL1_36/js/ [20:02:04] (when master breaks) [20:02:06] ah, thanks AntiComposite ! [20:05:13] Daimona: maybe. [20:05:32] I'm not sure where though.. woudl that be a // @phan comment between the docblock and the method? [20:05:40] That feels wrong [20:15:29] ejegg, and it's back up again. I did find https://phabricator.wikimedia.org/T257188 which was closed for lack of reproducibility. [20:18:59] the fix is probably going to be "just wait for the JSDuck -> JSDoc migration to finish" as JSDuck is unmaintained. https://phabricator.wikimedia.org/T138401 [20:19:31] oh ok, thanks for the context! [20:25:19] You can use a @suppress annotation inside the docblock [20:25:28] That's usually the best choice for signature issues [20:30:24] 10phabricator maintenance bot, 10SRE Observability (FY2021/2022-Q1): update legacy project tags - https://phabricator.wikimedia.org/T288756 (10lmata) Hola @Ladsgroup! After some deliberation, I have come up with https://github.com/Ladsgroup/Phabricator-maintenance-bot/pull/39 . If you have time, I would app... [20:33:20] 10phabricator maintenance bot, 10SRE Observability (FY2021/2022-Q1): update legacy project tags - https://phabricator.wikimedia.org/T288756 (10Ladsgroup) 💃 Let me know if I can help in any more case. [20:44:19] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: CI Docker images failing to build - https://phabricator.wikimedia.org/T290651 (10Aklapper) [22:00:32] 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T281164 (10Zabe) [22:20:07] !log gitlab-ansible-test: resetting instance data [22:20:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:35:56] AntiComposite: ejegg: the problem is not with jsduck, it's a problem with how doc.wm.o is cached. When the directory is updated, some of the requests remain cached (e.g. the index.html file) and then the other files have already fallen out of cache and thus no longer exist. [22:36:00] a force refresh usually fixes it [22:36:43] cached in Varnish that is [22:39:05] (03PS3) 10Jeena Huneidi: Add patch author/reviewer to promote patch [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719319 (https://phabricator.wikimedia.org/T281392) [23:13:34] (03PS4) 10Dduvall: Provide JSON schema for use in config validation [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719332 (https://phabricator.wikimedia.org/T225335) [23:54:18] 10phabricator maintenance bot, 10SRE Observability (FY2021/2022-Q1): update legacy project tags - https://phabricator.wikimedia.org/T288756 (10lmata) 05Open→03Resolved You’re the best \o/ thanks!