[05:55:16] 10Phabricator, 10wikimedia.biterg.io: Closed tickets in Bugzilla migrated without closing event? - https://phabricator.wikimedia.org/T107254 (10valerio.bozzolan) Wow! nice dashboard! In this case it would be useful to understand what Kibana is reading now. But probably we can start inspecting what we have. `... [07:01:16] 10Phabricator, 10DBA, 10Data-Persistence-Backup, 10serviceops-collab, 10Patch-For-Review: phabricator->phorge migration - database handling - https://phabricator.wikimedia.org/T335080 (10Marostegui) [07:21:23] 10Project-Admins: Add valerio.bozzolan in acl*Project-Admins to create cute Milestones in WMIT-stuff - https://phabricator.wikimedia.org/T340411 (10valerio.bozzolan) p:05Triage→03Low [07:22:33] 10Project-Admins: Add valerio.bozzolan in acl*Project-Admins to create cute Milestones in WMIT-stuff - https://phabricator.wikimedia.org/T340411 (10valerio.bozzolan) [07:22:47] 10Phabricator, 10wikimedia.biterg.io: Closed tickets in Bugzilla migrated without closing event? - https://phabricator.wikimedia.org/T107254 (10Aklapper) >>! In T107254#8962942, @valerio.bozzolan wrote: > Wow! nice dashboard! In this case it would be useful to understand what Kibana is reading now. It does no... [07:24:50] 10Project-Admins: Add valerio.bozzolan in acl*Project-Admins to create cute Milestones in WMIT-stuff - https://phabricator.wikimedia.org/T340411 (10Aklapper) 05Open→03Resolved a:03Aklapper Done! [07:38:43] 10Phabricator (Upstream), 10Upstream: Save a draft of the new task, and restore it - https://phabricator.wikimedia.org/T113930 (10Aklapper) 05Open→03Resolved I cannot reproduce this anymore [07:41:06] 10Phabricator (Upstream), 10Upstream: Separate Space description/name visibility from content visibility - https://phabricator.wikimedia.org/T128121 (10Aklapper) 05Open→03Declined I don't think this information should be exposed by default (or us maintaining a downstream patch for this). If folks want to e... [07:42:59] 10Project-Admins: Add valerio.bozzolan in acl*Project-Admins to create cute Milestones in WMIT-stuff - https://phabricator.wikimedia.org/T340411 (10valerio.bozzolan) Impressing :D thanks [07:45:50] 10Phabricator (Upstream), 10Upstream: maniphest 'project' and 'no icon' icons are confusing/too much alike - https://phabricator.wikimedia.org/T128019 (10Aklapper) 05Open→03Declined Project icons are completely random so I'm afraid there is no differentiation. There is a default image, and there is the opt... [07:47:44] 10Phabricator (Upstream), 10Browser-Support-Apple-Safari, 10Upstream: Phabricator fails to save the new task in Safari browser, resulting in data loss - https://phabricator.wikimedia.org/T132850 (10Aklapper) 05Open→03Declined >>! In T132850#2226120, @mmodell wrote: > Almost certainly not a universal phab... [07:49:33] 10Diffusion, 10Phabricator (Upstream), 10Upstream: Allow us to prefix repo's in search - https://phabricator.wikimedia.org/T139238 (10Aklapper) 05Open→03Declined Not planning to investigate this; see last comment for a workaround [07:51:28] 10Phabricator (Upstream), 10Upstream: Allow seeing Maniphest user contributions per project in Advanced Search - https://phabricator.wikimedia.org/T155215 (10Aklapper) 05Open→03Declined [07:53:28] 10Phabricator (Upstream), 10Upstream: No "find project" list when writing inside square brackets - https://phabricator.wikimedia.org/T164933 (10Aklapper) 05Open→03Declined I don't think this is achievable or worth to investigate further - avoid using square brackets as they have a specific meaning, I'd say. [07:56:31] 10Phabricator, 10wikimedia.biterg.io: Closed tickets in Bugzilla migrated without a closing date - https://phabricator.wikimedia.org/T107254 (10valerio.bozzolan) [07:59:55] 10Phabricator, 10wikimedia.biterg.io: Closed tickets in Bugzilla migrated without a closing date - https://phabricator.wikimedia.org/T107254 (10valerio.bozzolan) Ah I see the point! Thanks I can try to scrape Bugzilla to generate this export: ` BugZillaTaskID1,DateClosed1 BugZillaTaskID2,DateClosed2 BugZilla... [08:41:21] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops, 10Patch-For-Review: Provide ability to tag GitLab CI built images with a datetime format, set as default in pipeline-to-gitlab conversion - https://phabricator.wikimedia.org/T338224 (10CodeReviewBot) kharlan reopened htt... [09:55:28] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Migrate group0 to Kubernetes - https://phabricator.wikimedia.org/T337490 (10Clement_Goubert) [10:07:50] (03CR) 10AikoChou: "Hi @Antoine, could you take a look at this when you have a moment? Thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/931994 (https://phabricator.wikimedia.org/T334182) (owner: 10AikoChou) [10:19:12] 10Phabricator (Upstream), 10Mobile, 10Upstream: Can't see and choose from all of the visibility options on smaller screens - https://phabricator.wikimedia.org/T154333 (10Aklapper) 05Open→03Resolved Cannot reproduce anymore in upstream using Firefox 114.0.2 on an Android phone device. When a menu is expan... [10:26:47] 10GitLab (Infrastructure), 10serviceops-collab, 10Patch-For-Review: gitlab.wikimedia.org ssh host key should appear in wmf-known-host - https://phabricator.wikimedia.org/T337107 (10LSobanski) p:05Triage→03Low [10:27:08] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops-collab, 10Patch-For-Review: Provide mechanism to publish to doc.wikimedia.org from GitLab CI - https://phabricator.wikimedia.org/T336168 (10LSobanski) p:05Triage→03Medium [10:29:52] jnuche,hashar: I'm about to get started on the releases host switchover, FYI [10:29:57] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Migrate group0 to Kubernetes - https://phabricator.wikimedia.org/T337490 (10Clement_Goubert) [10:32:04] 10Phabricator (Upstream), 10Mobile, 10Upstream: On iOS, the screen “Change Details” cannot be scrolled or viewed entirely, so the user is trapped - https://phabricator.wikimedia.org/T95244 (10Aklapper) 05Stalled→03Declined Cannot reproduce anymore in upstream using Firefox 114.0.2 on an Android phone dev... [10:33:02] eoghan: I'm making lunch, can it wait ~40m? [10:33:10] I'd like to be around just in case [10:33:12] Sure [10:33:20] thx 👍 [10:33:30] Let's wait 1h, I'll need to get the kid from school at 12:15, be back by 12:30 [10:34:21] okie dokes [10:37:11] 10Phabricator (Upstream), 10Upstream: autocomplete fields don't work in Firefox 49 on Android - https://phabricator.wikimedia.org/T455 (10Aklapper) 05Open→03Resolved Cannot reproduce in Firefox 114.0.2 on an Android mobile on upstream https://we.phorge.it/search/query/advanced/?query=all Steps above show a... [10:45:46] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab: upgrade phab (phorge) hosts to bullseye - https://phabricator.wikimedia.org/T334519 (10LSobanski) p:05Medium→03High [10:48:39] 10GitLab (Infrastructure), 10serviceops-collab: automatically check for new gitlab releases and send notifications - https://phabricator.wikimedia.org/T323932 (10LSobanski) p:05Medium→03Low We are now subscribed to the GitLab releases mailing list so this is slightly lower priority (but would still be usef... [10:50:46] maintenance-disconnect-full-disks build 503411 integration-agent-docker-1025 (/: 29%, /srv: 100%, /var/lib/docker: 64%): OFFLINE due to disk space [10:55:55] maintenance-disconnect-full-disks build 503412 integration-agent-docker-1025 (/: 29%, /srv: 9%, /var/lib/docker: 64%): RECOVERY disk space OK [11:06:30] maintenance-disconnect-full-disks build 503414 integration-agent-docker-1029 (/: 29%, /srv: 24%, /var/lib/docker: 100%): OFFLINE due to disk space [11:09:44] eoghan: at my computer again [11:10:32] maintenance-disconnect-full-disks build 503415 integration-agent-docker-1029 (/: 29%, /srv: 25%, /var/lib/docker: 100%): still OFFLINE due to disk space [11:12:30] jnuche: Ok, cool. I'm good to go now, don't actually need to do the school run today, so I'll kick off. [11:16:56] jnuche: Starting with disabling jenkins on the old host (releases1002) and starting it on the new one (releases1003) [11:18:07] o/ [11:20:52] maintenance-disconnect-full-disks build 503417 integration-agent-docker-1029 (/: 29%, /srv: 9%, /var/lib/docker: 100%): RECOVERY disk space OK [11:26:00] maintenance-disconnect-full-disks build 503418 integration-agent-docker-1029 (/: 29%, /srv: 30%, /var/lib/docker: 100%): OFFLINE due to disk space [11:27:19] PROBLEM - jenkins_service_running on releases1002 is CRITICAL: PROCS CRITICAL: 0 processes with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [11:30:33] maintenance-disconnect-full-disks build 503419 integration-agent-docker-1029 (/: 29%, /srv: 9%, /var/lib/docker: 100%): RECOVERY disk space OK [11:34:24] Puppet on releases2002 throws an error about git pull on `repos/releng/release`. I'll fix after, this doesn't look like it's impacting anything. [11:35:18] ack [11:36:34] jnuche: Are you able to +1 this for me? https://gerrit.wikimedia.org/r/c/operations/dns/+/932230 [11:36:50] Actually, no need. Was just CI missing. [11:37:16] oh, still did :) [11:37:39] (I mean, did it already) [11:38:23] Cheers [11:42:04] jnuche: Ok, the DNS change looks to have gone through and I can load the jenkins UI (albeit behind a login screen). Can you see if everything looks correct from that point of view? [11:42:20] checking [11:47:24] eoghan: everything looks good, also you can see the landing page now without login (should keep the probe from firing) [11:47:32] thanks a lot :) [11:47:40] Great. [11:48:02] If you want to switch the landing page back to the login page, we can do that either. Whatever works best for you folks. [11:48:44] nop, that is the desired end state, I just haven't merged/deployed the required change yet [11:48:57] so I need to re-add every time the service is restarted [11:56:04] Grand [12:10:27] (03CR) 10Hashar: [C: 03+2] "I am not sure from where the error comes. Sorry for the typo! :)" [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/932640 (owner: 10Paladox) [12:11:03] (03Merged) 10jenkins-bot: Change attribution name [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/932640 (owner: 10Paladox) [12:14:03] (03PS1) 10Ilias Sarantopoulos: inference-services: add langid CI pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/933092 (https://phabricator.wikimedia.org/T99666) [12:16:12] 10Phabricator, 10wikimedia.biterg.io: Closed tickets in Bugzilla migrated without a closing date - https://phabricator.wikimedia.org/T107254 (10valerio.bozzolan) a:03valerio.bozzolan Since "I have a dream" on this I will kindly set myself as Assigner and then remove myself when nothing more can be done from... [12:26:16] (beta was timing out for about 10 minutes, back now, anything recently change or just "beta being beta"?) [12:27:29] (self-answer, probably `12:25 reverted floating IP switch` per https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL) [12:30:11] TheresNoTime: see -cloud [12:30:35] Yup just saw, no panic ^^ [12:31:20] TheresNoTime: you raise the interesting where to !log beta stuff [12:31:50] hi, sorry for the downtime on deployment-prep, I'm switching to a new (upgraded) cache server but something went wrong [12:31:54] so I had to rollback [12:32:39] fabfur: no need to apologise [12:32:55] TNT made isbetabroken.com so knows everything about how broken it is [12:32:55] No worries! (https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Deployment-prep#Release_Engineering_Server_Admin_Log suggests here but no drama) [12:34:32] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops, 10Patch-For-Review: Provide ability to tag GitLab CI built images with a datetime format, set as default in pipeline-to-gitlab conversion - https://phabricator.wikimedia.org/T338224 (10CodeReviewBot) tchanders merged htt... [12:41:55] probably found the issue, I'll try to re-switch the public ip now [12:43:49] switched again to the new instance, seems to work [12:45:36] (03CR) 10Hashar: [C: 03+2] "I have deployed the jobs" [integration/config] - 10https://gerrit.wikimedia.org/r/933092 (https://phabricator.wikimedia.org/T99666) (owner: 10Ilias Sarantopoulos) [12:46:54] (03Merged) 10jenkins-bot: inference-services: add langid CI pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/933092 (https://phabricator.wikimedia.org/T99666) (owner: 10Ilias Sarantopoulos) [12:50:09] (03CR) 10Hashar: [C: 03+2] "Deployed :)" [integration/config] - 10https://gerrit.wikimedia.org/r/933092 (https://phabricator.wikimedia.org/T99666) (owner: 10Ilias Sarantopoulos) [12:50:43] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Buster Deprecation): Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742 (10Fabfur) [12:50:58] (03CR) 10Hashar: [C: 03+2] Add Szymon Świergosz to the trusted users list [integration/config] - 10https://gerrit.wikimedia.org/r/932457 (owner: 10Catrope) [12:51:00] (03CR) 10Hashar: [C: 03+2] Add Nux to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/932672 (owner: 10Bartosz Dziewoński) [12:52:36] (03Merged) 10jenkins-bot: Add Szymon Świergosz to the trusted users list [integration/config] - 10https://gerrit.wikimedia.org/r/932457 (owner: 10Catrope) [12:52:38] (03Merged) 10jenkins-bot: Add Nux to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/932672 (owner: 10Bartosz Dziewoński) [12:53:05] I need to update https://isbetabroken.com/ to switch from deployment-cache-text07 -> deployment-cache-text08 (the new upgraded host) [12:53:08] how can I do? [12:53:52] (btw the cache-upload08 instead is correct so someone already updated it) [12:55:52] lmao [12:55:55] where is that from [12:56:09] it's behind CF? [12:56:50] sorry, CF? [12:57:08] cloudflare [12:57:14] (03CR) 10Hashar: [C: 03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/932457 (owner: 10Catrope) [12:57:18] (03CR) 10Hashar: [C: 03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/932672 (owner: 10Bartosz Dziewoński) [12:57:40] think TheresNoTime is the owner [12:58:32] * Reedy eyes TheresNoTime [13:01:01] Done :p [13:01:35] maintenance-disconnect-full-disks build 503437 integration-agent-docker-1029 (/: 29%, /srv: 19%, /var/lib/docker: 100%): OFFLINE due to disk space [13:05:30] maintenance-disconnect-full-disks build 503438 integration-agent-docker-1029 (/: 29%, /srv: 9%, /var/lib/docker: 74%): RECOVERY disk space OK [13:11:05] !log Updated wmf-quibble* jobs to run `npm cache verify` in order to garbage collect the npm cache before saving it. https://gerrit.wikimedia.org/r/c/integration/config/+/932195 | T340092 [13:11:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:11:08] T340092: Figure out how to garbage collect the npm cache - https://phabricator.wikimedia.org/T340092 [13:14:34] (03CR) 10Hashar: "I have updated the wmf-quibble jobs:" [integration/config] - 10https://gerrit.wikimedia.org/r/932195 (https://phabricator.wikimedia.org/T340092) (owner: 10Hashar) [13:15:47] (03CR) 10Ilias Sarantopoulos: [C: 03+1] "LGTM! You just need to rebase because I merged the langid pipelines a bit earlier." [integration/config] - 10https://gerrit.wikimedia.org/r/931994 (https://phabricator.wikimedia.org/T334182) (owner: 10AikoChou) [13:21:02] maintenance-disconnect-full-disks build 503441 integration-agent-docker-1029 (/: 29%, /srv: 9%, /var/lib/docker: 100%): OFFLINE due to disk space [13:36:09] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Quibble, 10Patch-For-Review, 10castor: Figure out how to garbage collect the npm cache - https://phabricator.wikimedia.org/T340092 (10hashar) I have manually triggered builds for the two largest jobs using `ZUUL_PIPELINE= postmerge`:... [13:40:33] maintenance-disconnect-full-disks build 503445 integration-agent-docker-1029 (/: 29%, /srv: 9%, /var/lib/docker: 100%): still OFFLINE due to disk space [13:46:23] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Quibble, 10Patch-For-Review, 10castor: Figure out how to garbage collect the npm cache - https://phabricator.wikimedia.org/T340092 (10hashar) For the php74 job: ` Cache verified and compressed (/cache/npm/_cacache) Content verified: 9... [13:54:39] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Quibble, 10Patch-For-Review, 10castor: Figure out how to garbage collect the npm cache - https://phabricator.wikimedia.org/T340092 (10hashar) Building against mediawiki/extensions/WikimediaMessages: For the php74: ` Cache verified an... [13:55:19] (03CR) 10Hashar: "Confirmed to be working ( T340092#8963977 ) I am updating all jobs" [integration/config] - 10https://gerrit.wikimedia.org/r/932195 (https://phabricator.wikimedia.org/T340092) (owner: 10Hashar) [13:56:05] (03CR) 10Hashar: [C: 03+2] jjb: gc quibble jobs npm cache before saving it [integration/config] - 10https://gerrit.wikimedia.org/r/932195 (https://phabricator.wikimedia.org/T340092) (owner: 10Hashar) [13:57:49] (03Merged) 10jenkins-bot: jjb: gc quibble jobs npm cache before saving it [integration/config] - 10https://gerrit.wikimedia.org/r/932195 (https://phabricator.wikimedia.org/T340092) (owner: 10Hashar) [14:00:21] fabfur: LOL, AFAIK isbetabroken.com isn't owned by us [14:00:51] now I know, but it's nice! [14:01:19] I'd like it more if it would be called "isbetastillworkingassmoothlyasusual.org" [14:02:09] mmm nobody registered that domain (how curious! ) [14:05:31] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Quibble, 10Patch-For-Review, 10castor: Figure out how to garbage collect the npm cache - https://phabricator.wikimedia.org/T340092 (10hashar) 05Open→03Resolved a:03hashar [14:05:35] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Cloud-VPS (Quota-requests): Rebuild WMCS integration instances to larger flavor - https://phabricator.wikimedia.org/T340070 (10hashar) [14:05:51] maintenance-disconnect-full-disks build 503450 integration-agent-docker-1029 (/: 29%, /srv: 9%, /var/lib/docker: 100%): still OFFLINE due to disk space [14:11:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Quibble, 10castor: Figure out how to garbage collect the npm cache - https://phabricator.wikimedia.org/T340092 (10hashar) Some other large caches: ` ... 1563 castor-mw-ext-and-skins/master/mediawiki-quibble-apitests-composer-php74-docke... [14:11:42] (03PS2) 10AikoChou: inference-services: add readability pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/931994 (https://phabricator.wikimedia.org/T334182) [14:17:13] (03CR) 10Hashar: [C: 03+2] "Thanks for the rebase, and it looks like I missed that one last week sorry." [integration/config] - 10https://gerrit.wikimedia.org/r/931994 (https://phabricator.wikimedia.org/T334182) (owner: 10AikoChou) [14:19:01] (03Merged) 10jenkins-bot: inference-services: add readability pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/931994 (https://phabricator.wikimedia.org/T334182) (owner: 10AikoChou) [14:19:43] (03CR) 10Hashar: [C: 03+2] "Jobs created:" [integration/config] - 10https://gerrit.wikimedia.org/r/931994 (https://phabricator.wikimedia.org/T334182) (owner: 10AikoChou) [14:21:03] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Cloud-VPS (Quota-requests): Rebuild WMCS integration instances to larger flavor - https://phabricator.wikimedia.org/T340070 (10hashar) a:03hashar [14:21:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Cloud-VPS (Quota-requests): Rebuild WMCS integration instances to larger flavor - https://phabricator.wikimedia.org/T340070 (10hashar) 05Stalled→03Declined I think the issue got solved by shrinking the giant npm cache (T340092) saving... [14:30:35] maintenance-disconnect-full-disks build 503455 integration-agent-docker-1029 (/: 29%, /srv: 9%, /var/lib/docker: 100%): still OFFLINE due to disk space [14:33:53] 10GitLab (Auth & Access), 10Release-Engineering-Team (Radar), 10CAS-SSO, 10Infrastructure-Foundations: Attempting to login to gitlab.wikimedia.org sometimes results in CAS 500 Internal Server Error - https://phabricator.wikimedia.org/T291964 (10jbond) 05Open→03Resolved a:03jbond Closing due to no res... [14:36:02] maintenance-disconnect-full-disks build 503456 integration-agent-docker-1029 (/: 29%, /srv: 9%, /var/lib/docker: 3%): RECOVERY disk space OK [14:37:58] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.41.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T340244 (10hashar) Ack! 🚋 🚋 🚋 [14:41:40] 10Phabricator: Please delete user KindRowboat and rename this user to KindRowboat - https://phabricator.wikimedia.org/T340224 (10Motevets) I cannot post from the KindRowboat account on phabricator because I get an error when trying to verify the email. (Probably because it's already associated with this account.... [14:42:50] 10GitLab (Auth & Access), 10Release-Engineering-Team (Priority Backlog 📥), 10CAS-SSO, 10Infrastructure-Foundations, 10User-brennen: GitLab sessions expire frequently - https://phabricator.wikimedia.org/T330359 (10jbond) @thcipriani I think that's sounds about right to me, gitlab is the only service using... [14:44:33] 10Phabricator: Please delete user KindRowboat and rename this user to KindRowboat - https://phabricator.wikimedia.org/T340224 (10SDunlap) For what it's worth, KindRowboat is my volunteer account. (Get it kindrobot is my LDAP username, kindrowboat, eh, eh?) [14:46:04] (03CR) 10Hashar: "Oops. I guess the invoked methods are not the proper one or the registered component should be a bit more than just an element. I am also " [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/932641 (https://phabricator.wikimedia.org/T340372) (owner: 10Paladox) [14:58:35] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥), 10API Platform, 10Anti-Harassment, and 19 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10dancy) [14:58:49] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops, 10Patch-For-Review: Provide ability to tag GitLab CI built images with a datetime format, set as default in pipeline-to-gitlab conversion - https://phabricator.wikimedia.org/T338224 (10dancy) 05In progress→03Resolved [15:05:09] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops, 10Patch-For-Review: Provide ability to tag GitLab CI built images with a datetime format, set as default in pipeline-to-gitlab conversion - https://phabricator.wikimedia.org/T338224 (10CodeReviewBot) kharlan opened https... [15:34:06] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops-collab, 10Patch-For-Review: Provide mechanism to publish to doc.wikimedia.org from GitLab CI - https://phabricator.wikimedia.org/T336168 (10CodeReviewBot) jnuche merged https://gitlab.wikimedia.org/repos/releng/scap3-dev... [15:35:34] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops-collab, 10Patch-For-Review: Provide mechanism to publish to doc.wikimedia.org from GitLab CI - https://phabricator.wikimedia.org/T336168 (10CodeReviewBot) jnuche merged https://gitlab.wikimedia.org/repos/releng/jenkins-d... [16:03:23] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops, 10Patch-For-Review: Provide ability to tag GitLab CI built images with a datetime format, set as default in pipeline-to-gitlab conversion - https://phabricator.wikimedia.org/T338224 (10CodeReviewBot) dancy updated https:... [16:03:33] RECOVERY - jenkins_service_running on releases1002 is OK: PROCS OK: 1 process with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [16:06:44] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops, 10Patch-For-Review: Provide ability to tag GitLab CI built images with a datetime format, set as default in pipeline-to-gitlab conversion - https://phabricator.wikimedia.org/T338224 (10CodeReviewBot) dancy merged https:/... [16:07:04] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops, 10Patch-For-Review: Provide ability to tag GitLab CI built images with a datetime format, set as default in pipeline-to-gitlab conversion - https://phabricator.wikimedia.org/T338224 (10CodeReviewBot) dancy closed https:/... [16:08:07] PROBLEM - jenkins_service_running on releases1002 is CRITICAL: PROCS CRITICAL: 0 processes with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [16:17:05] 10Phabricator: Please delete user KindRowboat and rename this user to KindRowboat - https://phabricator.wikimedia.org/T340224 (10Aklapper) 05Open→03Resolved a:03Aklapper Thanks for the additional information! I deleted the unused account and renamed this one. [16:17:28] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟): Make a tool to convert .pipeline/config.yaml to .gitlab-ci.yaml - https://phabricator.wikimedia.org/T327332 (10dancy) 05Open→03Resolved >>! In T327332#8928687, @thcipriani wrote: > Is this still a good idea? Still in progress? Yes,... [16:17:39] 10Phabricator: Please delete user KindRowboat and rename this user to KindRowboat - https://phabricator.wikimedia.org/T340224 (10Aklapper) [16:28:02] 10Deployments, 10GitLab (Integrations), 10Release-Engineering-Team, 10Release Pipeline, 10Epic: Provide a deployment integration with GitLab so that users can make the currently-manual steps with a click - https://phabricator.wikimedia.org/T340188 (10brennen) [16:29:08] 10GitLab (CI & Job Runners), 10commit-message-validator, 10Documentation: Document how to run commit-message-validator as part of GitLab CI - https://phabricator.wikimedia.org/T339309 (10brennen) [16:29:17] 10GitLab (CI & Job Runners), 10commit-message-validator: Update commit-message-validator to work nicely with GitLab repos - https://phabricator.wikimedia.org/T339307 (10brennen) [16:51:48] 10Beta-Cluster-Infrastructure, 10DBA, 10MediaWiki-libs-Rdbms, 10SRE-Sprint-Week-Sustainability-March2023, and 2 others: Enable MariaDB/MySQL's Strict Mode - https://phabricator.wikimedia.org/T108255 (10Reedy) [16:53:09] 10Continuous-Integration-Config, 10Growth-Team, 10GrowthExperiments, 10ci-test-error: phpbench job for GrowthExperiments fails due to missing dependency - https://phabricator.wikimedia.org/T329280 (10DMburugu) [16:53:39] 10GitLab (Infrastructure), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10serviceops, 10Patch-For-Review, 10cloud-services-team (Kanban): Assess GitLab Container Registry as a default for container build processes - https://phabricator.wikimedia.org/T307537 (10CodeReviewBot) brennen merged https://gitla... [17:45:14] (03PS1) 10Pwangai: Zuul: [mediawiki/extensions/GeoData] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/933144 (https://phabricator.wikimedia.org/T321837) [17:47:41] (03PS1) 10Pwangai: Zuul: [mediawiki/extensions/GlobalUserPage] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/933145 (https://phabricator.wikimedia.org/T321837) [17:49:39] (03PS1) 10Pwangai: Zuul: [mediawiki/extensions/ArticlePlaceholder] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/933167 (https://phabricator.wikimedia.org/T321837) [17:51:35] (03PS1) 10Pwangai: Zuul: [mediawiki/extensions/CentralAuth] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/933168 (https://phabricator.wikimedia.org/T321837) [17:51:45] 10Release-Engineering-Team (Seen): Runner runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T326558 (10bking) Just a heads-up, we experienced [[ https://gitlab.wikimedia.org/repos/search-platform/discolytics/-/jobs/112089 | a disk space failure ]] again today. Add... [18:07:50] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Seen): Runner runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T326558 (10bd808) [18:28:44] brennen: Hej. I'd love to get non-public Phab patch https://phabricator.wikimedia.org/T338611#8965363 merged and deployed (I guess having it tested by a second person is a bit too idealistic?). Help would be appreciated. [18:29:22] andre: having a look. [18:29:26] thank you [18:40:12] makeStuffWork.diff :) genius [19:15:17] brennen: we now have: confirmed working access to db1118 from phab-test1001.. with all the phab databases.. AND we already have a dump of them all without the data :) [19:15:35] the "phuser" works with the prod credentials [19:15:55] just like the user/pass that is in prod phab config file [19:16:19] except the host is db1118 vs m3-slave and the port is default 3306 instead of 3323 [19:16:46] basically you can already run the phorge upgrade .. then we simply repeat the dump command [19:17:11] <3 [19:17:18] nice work mutante [19:18:22] :) [19:18:54] [phab-test1001:~] $ mysql -u phuser -h db1118.eqiad.wmnet -p phabricator_project [19:19:38] [phab1004:~] $ sudo grep phuser -A1 /etc/phabricator/config.yaml [19:20:44] maintenance-disconnect-full-disks build 503513 integration-agent-docker-1030 (/: 29%, /srv: 61%, /var/lib/docker: 96%): OFFLINE due to disk space [19:25:16] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review, 10Wikimedia-Incident: Warn against potentially deploying a config patch NOT specified - https://phabricator.wikimedia.org/T338168 (10CodeReviewBot) dancy opened https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/156 backport: add tests f... [19:25:32] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review, 10Wikimedia-Incident: Warn against potentially deploying a config patch NOT specified - https://phabricator.wikimedia.org/T338168 (10CodeReviewBot) [19:25:40] maintenance-disconnect-full-disks build 503514 integration-agent-docker-1030 (/: 29%, /srv: 51%, /var/lib/docker: 97%): RECOVERY disk space OK [19:35:46] 10Project-Admins: Create project tag for "IP blocking" - https://phabricator.wikimedia.org/T340487 (10kostajh) [19:36:16] 10Phabricator, 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review: Upgrade content license of Phabricator from 3.0 to CC BY-SA 4.0 - https://phabricator.wikimedia.org/T338440 (10CodeReviewBot) brennen merged https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/15 Update con... [19:38:07] 10Phabricator, 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review: Remove "Prototype" suffix from "Reports" menu item on Project pages - https://phabricator.wikimedia.org/T337876 (10CodeReviewBot) brennen merged https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/14 Remove... [19:40:48] 10Release-Engineering-Team (They Live 🕶️🧟), 10Anti-Harassment, 10Security-Team, 10iPoid-Service: Use Gitlab Security Pipeline for ipoid - https://phabricator.wikimedia.org/T338238 (10kostajh) 05Open→03Stalled >>! In T338238#8929361, @Mstyles wrote: > The preferred route would be to update the project s... [19:40:51] 10Release-Engineering-Team (They Live 🕶️🧟), 10Anti-Harassment, 10Projects-Cleanup, 10Security-Team, 10iPoid-Service: Migrate mediawiki/services/ipoid to GitLab - https://phabricator.wikimedia.org/T337714 (10kostajh) [19:41:28] 10Release-Engineering-Team (They Live 🕶️🧟), 10Anti-Harassment, 10Security-Team, 10iPoid-Service: Use Gitlab Security Pipeline for ipoid - https://phabricator.wikimedia.org/T338238 (10CodeReviewBot) kharlan closed https://gitlab.wikimedia.org/repos/mediawiki/services/ipoid/-/merge_requests/8 Draft: Update... [19:51:53] 10GitLab (CI & Job Runners), 10commit-message-validator, 10User-bd808: Update commit-message-validator to work nicely with GitLab repos - https://phabricator.wikimedia.org/T339307 (10bd808) 05Open→03In progress a:03bd808 [20:03:46] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Migrate group0 to Kubernetes - https://phabricator.wikimedia.org/T337490 (10Reedy) >>! In T337490#8963478, @gerritbot wrote: > Change 923385 **merged** by Clément Goubert: > %%%[operations/puppet@production] mw-on-k8s: Redirect... [20:07:52] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Migrate group0 to Kubernetes - https://phabricator.wikimedia.org/T337490 (10Reedy) [21:43:01] 10Project-Admins: Create project tag for "IP blocking" - https://phabricator.wikimedia.org/T340487 (10Peachey88) Could the name be reconsidered?, At the moment it reads as anything to do with IP blocks, which would lead to confusion with #mediawiki-blocks [21:51:13] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Radar), 10Research: GitLab Issues experiment with Research - https://phabricator.wikimedia.org/T304614 (10leila) @thcipriani (context: I'm reviewing tasks in the Research Phabricator board): - FYI: I'm going to move this to Research -... [22:56:43] 10Phabricator, 10DBA, 10Data-Persistence-Backup, 10serviceops-collab, 10Patch-For-Review: phabricator->phorge migration - database handling - https://phabricator.wikimedia.org/T335080 (10Dzahn) We are now able to use phab-test1001 VM to connect to db1118, thanks to Manuel. Also, we already have dumps of... [22:59:07] 10Phabricator, 10DBA, 10Data-Persistence-Backup, 10serviceops-collab, 10Patch-For-Review: phabricator->phorge migration - database handling - https://phabricator.wikimedia.org/T335080 (10Dzahn) ` 19:15 < mutante> .. we now have: confirmed working access to db1118 from phab-test1001.. with all the phab da... [23:41:24] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.41.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T340243 (10Jdlrobson) ##### Risky Patch! 🚂🔥 * **Change**: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/929795 * **Summary**:There is a risk of an in...