[05:33:29] 10Beta-Cluster-Infrastructure, 10Inuka-Team (Kanban), 10MW-1.39-notes (1.39.0-wmf.18; 2022-06-27), 10Wikistories (MVP): Call to undefined method ForeignDBFile::getExtendedMetadata() - https://phabricator.wikimedia.org/T309668 (10SGautam_WMF) 05Open→03Resolved [09:02:52] Hi folks [09:03:10] I am trying to check if github -> gerrit mirroring is broken for [09:03:11] https://gerrit.wikimedia.org/g/scoring/ores/ores/+/refs/heads/master [09:03:28] in theory it should contain the last commits from https://github.com/wikimedia/ores/commits/master [09:03:38] but I can't recall exactly where the mirroring is configure [09:03:42] *configured [09:07:03] Diffusion shows all the commits https://phabricator.wikimedia.org/source/ores/history/master/ [09:11:12] ah I see, from phab1001's daemon.log there seem to be this [09:11:18] PHLOG: 'Unexpected output while updating repository "ores": [2022-06-27 00:23:20] EXCEPTION: (PhutilProxyException) Error while pushing "rORES" repository to mirrors. {>} (PhutilAggregateException) Exceptions occurred while mirroring the "ores" repository. [09:11:22] phabricator@gerrit.wikimedia.org: Permission denied (publickey). [09:11:23] fatal: Could not read from remote repository. [09:12:47] it happened in the past, forgot about it https://phabricator.wikimedia.org/T224996#7088785 [09:16:30] 10Release-Engineering-Team, 10ML-Governance: Github's wikimedia/ores not mirroring to Gerrit's scoring /ores /ores - https://phabricator.wikimedia.org/T311390 (10elukey) [09:16:35] opened https://phabricator.wikimedia.org/T311390 [09:16:37] :) [09:17:02] 10Release-Engineering-Team, 10Machine-Learning-Team: Github's wikimedia/ores not mirroring to Gerrit's scoring /ores /ores - https://phabricator.wikimedia.org/T311390 (10elukey) [09:28:06] 10Continuous-Integration-Config, 10Shellbox, 10ci-test-error: ERROR: Phan 5.x requires php-ast 1.0.7+ because it depends on AST version 80. php-ast '0.1.6' is installed. - https://phabricator.wikimedia.org/T311392 (10Reedy) [09:29:11] 10Continuous-Integration-Config, 10Shellbox, 10ci-test-error: ERROR: Phan 5.x requires php-ast 1.0.7+ because it depends on AST version 80. php-ast '0.1.6' is installed. - https://phabricator.wikimedia.org/T311392 (10Reedy) 05Open→03Invalid {b9ec6b1ad6bebda1b26d8e45caead8bc02b68184} needs cherry picking. [10:20:01] Project beta-update-databases-eqiad build #59590: 04FAILURE in 0.85 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/59590/ [10:43:01] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Upstream: Jenkins search results missing /ci/ URL component - https://phabricator.wikimedia.org/T294424 (10hashar) Jenkins code is at https://github.com/jenkinsci/jenkins There is a Jelly template: ` lang=xml,name=core/src/main/resources/hudson/search/Se... [10:44:01] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Upstream: Jenkins search results missing /ci/ URL component - https://phabricator.wikimedia.org/T294424 (10hashar) [10:47:31] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Upstream: Jenkins search results missing /ci/ URL component - https://phabricator.wikimedia.org/T294424 (10hashar) 05Open→03Resolved a:03hashar After the Nested View plugin upgrade to 1.26, searching for {nav wikistats} yields the proper URL. The HT... [10:52:25] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Upstream: Jenkins search results missing /ci/ URL component - https://phabricator.wikimedia.org/T294424 (10hashar) Thank you @Lucas_Werkmeister_WMDE for the task filing and @dancy to have poked me about it last week :] [11:10:08] 10Beta-Cluster-Infrastructure, 10CX-cxserver: Upgrade deployment-docker-cxserver01 host to Buster - https://phabricator.wikimedia.org/T306050 (10Nikerabbit) p:05Triage→03Lowest Not currently using the beta cluster to test cxserver. [11:20:04] Project beta-update-databases-eqiad build #59591: 04STILL FAILING in 2.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/59591/ [11:29:37] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [extensions/Share] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/808868 [11:29:39] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [extensions/Share] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/808868 (owner: 10QChris) [11:29:41] (03PS1) 10QChris: Import done. Revoke import grants [extensions/Share] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/808869 [11:29:43] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [extensions/Share] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/808869 (owner: 10QChris) [12:25:58] Yippee, build fixed! [12:25:58] Project beta-update-databases-eqiad build #59592: 09FIXED in 5 min 57 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/59592/ [12:47:21] 10Beta-Cluster-Infrastructure: Create deployment-deploy04 as future secondary/upgrade - https://phabricator.wikimedia.org/T309437 (10TheresNoTime) 05In progress→03Declined This is fairly pointless the more I think about it 🤷‍♀️ will delete the VM and save some quota :) [13:26:11] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10SRE: deployment-deploy03 ran out of memory twice while trying to perform a WikiLambda db migration - https://phabricator.wikimedia.org/T309413 (10TheresNoTime) 05Open→03Resolved Going to mark this resolved (//as it kinda/mostly/sorta is//) [13:55:08] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Next), 10Patch-For-Review: Use Jenkins upstream systemd unit instead of our own - https://phabricator.wikimedia.org/T308637 (10hashar) a:03hashar [15:18:47] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Upstream: Jenkins search results missing /ci/ URL component - https://phabricator.wikimedia.org/T294424 (10dancy) This is fantastic. Thanks @hashar! [15:31:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10CirrusSearch, 10Discovery-Search, 10Gerrit-Privilege-Requests: Positive test reports by Cindy-the-browser-test-bot should not result in setting Attention to the patch owner - https://phabricator.wikimedia.org/T311370 (10Gehel) p:05Tr... [15:31:37] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10CirrusSearch, 10Gerrit-Privilege-Requests, 10Discovery-Search (Current work): Positive test reports by Cindy-the-browser-test-bot should not result in setting Attention to the patch owner - https://phabricator.wikimedia.org/T311370 (10... [15:42:06] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10CirrusSearch, 10Gerrit-Privilege-Requests, 10Discovery-Search (Current work): Positive test reports by Cindy-the-browser-test-bot should not result in setting Attention to the patch owner - https://phabricator.wikimedia.org/T311370 (10... [15:50:11] 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10serviceops, 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10LSobanski) @jelto Is there anything else that needs to happen here? [16:46:38] (03CR) 10Ahmon Dancy: scap backport: add revert option (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/806352 (https://phabricator.wikimedia.org/T287046) (owner: 10Jeena Huneidi) [16:58:42] (03PS26) 10Jeena Huneidi: scap backport: add revert option [tools/scap] - 10https://gerrit.wikimedia.org/r/806352 (https://phabricator.wikimedia.org/T287046) [16:59:26] (03PS27) 10Jeena Huneidi: scap backport: add revert option [tools/scap] - 10https://gerrit.wikimedia.org/r/806352 (https://phabricator.wikimedia.org/T287046) [17:06:15] 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10MW-on-K8s, 10Release Pipeline, 10Patch-For-Review, 10User-brennen: Scap rollback command - https://phabricator.wikimedia.org/T287045 (10dduvall) [17:08:53] 10Release-Engineering-Team (Doing), 10MW-on-K8s, 10Release Pipeline, 10Patch-For-Review, 10User-brennen: Design m8s deployment workflows and tooling - https://phabricator.wikimedia.org/T279322 (10dduvall) [17:09:21] 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10MW-on-K8s, 10Release Pipeline, 10Patch-For-Review, 10User-brennen: Scap rollback command - https://phabricator.wikimedia.org/T287045 (10dduvall) 05Open→03Resolved a:03dduvall Note that after a discussion with @jeena and @dancy, we decided that... [17:11:03] 10GitLab (CI & Job Runners): DNS/networking not working on Trusted Runners - https://phabricator.wikimedia.org/T311241 (10dduvall) Oy. I'll have a look. @Dzahn do you know if this has anything to do with the new firm rules? [17:16:11] 10GitLab (CI & Job Runners): DNS/networking not working on Trusted Runners - https://phabricator.wikimedia.org/T311241 (10dduvall) I don't have access, so someone else will have to take a closer look. [17:18:06] brennen: dduvall: wmf.17 from last week was all quiet on friday and today as well :] [17:18:10] oh and good morning [17:18:24] excellent [17:18:31] and good evening. :) [17:18:37] I have filed one extra bug which I firmly believe to be a pre condition https://phabricator.wikimedia.org/T311389 `LogicException: This ParserOutput contains no text!` [17:18:53] aka it is not a blocker [17:24:37] (03CR) 10Ahmon Dancy: scap backport: add revert option (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/806352 (https://phabricator.wikimedia.org/T287046) (owner: 10Jeena Huneidi) [17:25:21] (03CR) 10Ahmon Dancy: scap backport: add revert option (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/806352 (https://phabricator.wikimedia.org/T287046) (owner: 10Jeena Huneidi) [17:50:03] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10Data-Engineering, 10Performance Issue: Improve speed of Gitlab CI - https://phabricator.wikimedia.org/T311111 (10brennen) [18:01:07] Hello, my name is Mary and I am working with the abstract wikipedia team. Currently we are trying to set up a health check script for our instance on the beta cluster. Looking at some examples of the Icinga custom checks (https://wikitech.wikimedia.org/wiki/Icinga#Configuration), IIRC we can submit a custom script that sends HTTP requests to beta cluster. Is that a correct understanding? Tyvm. [18:05:34] maryyang: I'm afraid there is no more icinga in beta/cloudvps anymore. in production Icinga still exists but is in the process of being replaced with prometheus based monitoring. if you want to check HTTPS URLs then there is a newer and easier way to do that. depends a lot on what you actually want to check, where it runs, what you want to happen when it alerts etc. there are also [18:05:40] #wikimedia-observability and #wikimedia-cloud that might be a better match for it [18:06:56] I think the most effective way forward is to create Phabricator ticket describing what you want to achieve. it will get tagged accordingly and get responses also from people in other timezones who are not working right now [18:09:37] (03PS28) 10Jeena Huneidi: scap backport: add revert option [tools/scap] - 10https://gerrit.wikimedia.org/r/806352 (https://phabricator.wikimedia.org/T287046) [18:20:13] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by aokoth@cumin1001 for hosts: `gitlab2001.wikimedia.org` - gitlab2001.wikimedia.org (**F... [18:25:46] mutante: Thanks for the info! I'll create the phabricator ticket right now. Technically we wouldn't need the check deployed on beta, as long as it is able to ping beta (the endpoint is public). Unless this is more of a matter of principle, where prod Icinga is only for prod services? [18:31:32] maryyang: yea, so the principle is more like..that beta is not made for production services.but if something is not "production" then usually it doesn't need monitoring. or ..the other way around.. if something is in use and needs monitoring.. it should probably move to production. that being said, I'd still ask the observability team about "prometheus blackbox monitoring". looks like this: [18:31:38] https://gerrit.wikimedia.org/r/c/operations/puppet/+/806476/4/modules/profile/manifests/gitlab.pp [18:32:07] you can define custom actions.. do you want email, IRC notification, automatic ticket creation... [18:32:41] and yea,if the endpoint is public it should be usable [18:35:21] mutane: that makes sense. Great tips! Ty!! [18:39:26] yw, feel free to PM if you get stuck, laters [19:23:42] (03PS1) 10Reedy: Skip PHP 8.1 on RemexHtml [integration/config] - 10https://gerrit.wikimedia.org/r/809000 (https://phabricator.wikimedia.org/T308406) [19:24:16] (03CR) 10Reedy: [V: 03+2 C: 03+2] Skip PHP 8.1 on RemexHtml [integration/config] - 10https://gerrit.wikimedia.org/r/809000 (https://phabricator.wikimedia.org/T308406) (owner: 10Reedy) [19:24:38] (03PS2) 10Reedy: Skip PHP 8.1 on RemexHtml [integration/config] - 10https://gerrit.wikimedia.org/r/809000 (https://phabricator.wikimedia.org/T308406) [19:24:52] (03CR) 10Reedy: [C: 03+2] Skip PHP 8.1 on RemexHtml [integration/config] - 10https://gerrit.wikimedia.org/r/809000 (https://phabricator.wikimedia.org/T308406) (owner: 10Reedy) [19:27:21] (03Merged) 10jenkins-bot: Skip PHP 8.1 on RemexHtml [integration/config] - 10https://gerrit.wikimedia.org/r/809000 (https://phabricator.wikimedia.org/T308406) (owner: 10Reedy) [19:28:03] !log Reloading Zuul to deploy https://phabricator.wikimedia.org/T308406 [19:28:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:28:11] (03CR) 10Jeena Huneidi: scap backport: add revert option (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/806352 (https://phabricator.wikimedia.org/T287046) (owner: 10Jeena Huneidi) [19:28:32] (03PS1) 10Ahmon Dancy: Remove ssh-agent startup/kill from train and test-scap-backport scripts [tools/train-dev] - 10https://gerrit.wikimedia.org/r/809001 [19:33:26] 10Continuous-Integration-Config: Re-enable PHP 8.1 on RemexHTML - https://phabricator.wikimedia.org/T311450 (10Reedy) [19:33:40] 10Continuous-Integration-Config: Re-enable PHP 8.1 on RemexHTML - https://phabricator.wikimedia.org/T311450 (10Reedy) [19:33:49] 10Continuous-Integration-Config: Re-enable PHP 8.1 on RemexHTML - https://phabricator.wikimedia.org/T311450 (10Reedy) 05Open→03Stalled [19:34:39] 10Continuous-Integration-Config: Re-enable PHP 8.1 CI on RemexHTML - https://phabricator.wikimedia.org/T311450 (10Reedy) [19:35:56] 10Continuous-Integration-Config: Re-enable PHP 8.1 CI on css-sanitizr - https://phabricator.wikimedia.org/T311451 (10Reedy) [19:36:34] 10Continuous-Integration-Config: Re-enable PHP 8.1 CI on css-sanitizr - https://phabricator.wikimedia.org/T311451 (10Reedy) 05Open→03Stalled [19:36:53] 10Continuous-Integration-Config: Re-enable PHP 8.1 CI on css-sanitizer - https://phabricator.wikimedia.org/T311451 (10Reedy) [19:38:25] 10Continuous-Integration-Config: Re-enable PHP 8.1 CI on css-sanitizer - https://phabricator.wikimedia.org/T311451 (10Reedy) [19:38:36] 10Continuous-Integration-Config: Re-enable PHP 8.1 CI on css-sanitizer - https://phabricator.wikimedia.org/T311451 (10Reedy) [19:38:59] 10Continuous-Integration-Config: Re-enable PHP 8.1 CI on css-sanitizer - https://phabricator.wikimedia.org/T311451 (10Reedy) [19:45:48] (03PS1) 10Ahmon Dancy: test-scap-backport: Use GrowthExperiments instead of Echo for submodule test [tools/train-dev] - 10https://gerrit.wikimedia.org/r/809003 [19:46:54] (03PS2) 10Ahmon Dancy: test-scap-backport: Use GrowthExperiments instead of Echo for submodule test [tools/train-dev] - 10https://gerrit.wikimedia.org/r/809003 [19:57:50] (03CR) 10Jeena Huneidi: [C: 03+2] "LGTM" [tools/train-dev] - 10https://gerrit.wikimedia.org/r/809003 (owner: 10Ahmon Dancy) [19:58:41] (03Merged) 10jenkins-bot: test-scap-backport: Use GrowthExperiments instead of Echo for submodule test [tools/train-dev] - 10https://gerrit.wikimedia.org/r/809003 (owner: 10Ahmon Dancy) [20:39:23] 10GitLab (CI & Job Runners): DNS/networking not working on Trusted Runners - https://phabricator.wikimedia.org/T311241 (10brennen) > I don't have access, so someone else will have to take a closer look. See {T308350}. Based on that, it seems like members of `contint-roots` should be in `gitlab-roots`, and that... [20:40:44] (03CR) 10Ahmon Dancy: scap backport: add revert option (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/806352 (https://phabricator.wikimedia.org/T287046) (owner: 10Jeena Huneidi) [20:43:44] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10Infrastructure-Foundations, 10SRE, and 3 others: Access to trusted gitlab runners for gitlab-roots (or appropriate similar group) - https://phabricator.wikimedia.org/T308350 (10brennen) 05Resolved→03Open While `contint-roots`... [21:10:14] (03PS1) 10Reedy: Skip PHP 8.1 on Dodo and Zest [integration/config] - 10https://gerrit.wikimedia.org/r/809022 (https://phabricator.wikimedia.org/T307506) [21:10:38] 10Continuous-Integration-Config: Re-enable PHP 8.1 CI on Dodo - https://phabricator.wikimedia.org/T311462 (10Reedy) [21:10:44] 10Continuous-Integration-Config: Re-enable PHP 8.1 CI on Dodo - https://phabricator.wikimedia.org/T311462 (10Reedy) 05Open→03Stalled [21:10:56] 10Continuous-Integration-Config: Re-enable PHP 8.1 CI on Dodo - https://phabricator.wikimedia.org/T311462 (10Reedy) [21:11:15] 10Continuous-Integration-Config: Re-enable PHP 8.1 CI on Zest - https://phabricator.wikimedia.org/T311463 (10Reedy) [21:11:34] 10Continuous-Integration-Config: Re-enable PHP 8.1 CI on Zest - https://phabricator.wikimedia.org/T311463 (10Reedy) [21:11:42] 10Continuous-Integration-Config: Re-enable PHP 8.1 CI on Zest - https://phabricator.wikimedia.org/T311463 (10Reedy) 05Open→03Stalled [21:11:44] (03CR) 10Reedy: [C: 03+2] Skip PHP 8.1 on Dodo and Zest [integration/config] - 10https://gerrit.wikimedia.org/r/809022 (https://phabricator.wikimedia.org/T307506) (owner: 10Reedy) [21:14:37] 10GitLab (CI & Job Runners): DNS/networking not working on Trusted Runners - https://phabricator.wikimedia.org/T311241 (10Dzahn) It's not a side-effect of the issue, it's that we applied gitlab-roots to gitlab servers but not so far to the gitlab_runner role. https://gerrit.wikimedia.org/r/c/operations/puppet/+... [21:17:14] (03Merged) 10jenkins-bot: Skip PHP 8.1 on Dodo and Zest [integration/config] - 10https://gerrit.wikimedia.org/r/809022 (https://phabricator.wikimedia.org/T307506) (owner: 10Reedy) [21:18:39] 10GitLab (CI & Job Runners): DNS/networking not working on Trusted Runners - https://phabricator.wikimedia.org/T311241 (10Dzahn) What's interesting here is the error is "Could not resolve host: gitlab.wikimedia.org" and this was very close to T311290 "DNS CI is broken". [21:19:27] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/809022 [21:19:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:20:59] 10GitLab (CI & Job Runners): DNS/networking not working on Trusted Runners - https://phabricator.wikimedia.org/T311241 (10Dzahn) Yea, I could still reproduce the error though. Just clicked on rebuild and it still fails like before. [21:26:03] 10Phabricator: Phab doesn't know about RemexHtml repo? - https://phabricator.wikimedia.org/T311465 (10Reedy) [21:28:09] 10Phabricator: Phab doesn't know about RemexHtml repo? - https://phabricator.wikimedia.org/T311465 (10Reedy) [21:34:35] 10Phabricator: Phab doesn't know about RemexHtml repo? - https://phabricator.wikimedia.org/T311465 (10Dzahn) Not sure if helpful but you can link to this: https://gerrit.wikimedia.org/r/q/5500b2f4f59b0212d1cb000325f2d4966a6a7e51 and it will redirect to https://gerrit.wikimedia.org/r/c/mediawiki/libs/RemexHtml/... [21:51:20] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10Infrastructure-Foundations, 10SRE, and 5 others: Access to trusted gitlab runners for gitlab-roots (or appropriate similar group) - https://phabricator.wikimedia.org/T308350 (10Dzahn) [21:56:09] 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10serviceops, 10serviceops-collab, and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Dzahn) [21:56:47] 10Phabricator, 10Release-Engineering-Team (Next), 10serviceops, 10serviceops-collab: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Dzahn) [21:56:59] 10Phabricator, 10serviceops, 10serviceops-collab, 10Patch-For-Review: move phabricator to new hardware generation - https://phabricator.wikimedia.org/T280597 (10Dzahn) [21:57:24] 10Gerrit, 10SRE, 10serviceops, 10serviceops-collab: replacement for gerrit2001 - https://phabricator.wikimedia.org/T243027 (10Dzahn) [22:31:30] 10GitLab (CI & Job Runners): DNS/networking not working on Trusted Runners - https://phabricator.wikimedia.org/T311241 (10Dzahn) "When you deploy a container on your network, if it cannot find a DNS server defined in /etc/resolv.conf, by default it will take on the DNS configured for the host machine. " This is... [22:33:31] dduvall: soo.. re: buildkit. TLDR: if I get root inside a running container and then do something like: [22:33:47] sed -i 's/10.3.0.1/1.1.1.1/g' /etc/resolv.conf [22:34:01] to replace the DNS server it sets for some reason ..with cloudflare [22:34:32] then I can change the whole problem from a simple "Could not resolve host: gitlab.wikimedia.org" to something different and new [22:35:48] well.let me be more precise. if I use cloudflare then it's just a timeout to get to the DNS server [22:36:05] if I use our own DNS server it's "lookup docker-registry.wikimedia.org on 208.80.154.238:53: no such host " [22:37:06] it's still category DNS problems but .. we do want something different in resolv.conf inside the container and I feel this hits close: https://github.com/moby/moby/issues/28188#issuecomment-266150831 [22:37:22] gotta keep debugging tomorrow