[09:10:15] 06serviceops, 10MW-on-K8s, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Create ValidatingAdmissionPolicies to replace mediawiki PSP - https://phabricator.wikimedia.org/T368251#9975998 (10JMeybohm) [10:18:08] 06serviceops, 06SRE: VRT wiki fails to create account - https://phabricator.wikimedia.org/T359901#9976121 (10Krd) The problem occurs just now, created one account, cannot create another one. [10:27:03] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Allow to address Kubernetes API servers from NetworkPolicy - https://phabricator.wikimedia.org/T287491#9976136 (10jijiki) 05Open→03Resolved [12:17:24] 06serviceops, 10conftool, 13Patch-For-Review: Move conftool to gitlab, turn on deb package auto-generation - https://phabricator.wikimedia.org/T369594#9976318 (10Joe) a:03Joe [12:24:06] 06serviceops, 10ChangeProp, 10MediaWiki-Core-HTTP-Cache, 06MediaWiki-Engineering, 07Performance Issue: Redunce the number of resource_change and resource_purge events emitted due to template changes - https://phabricator.wikimedia.org/T369898 (10daniel) 03NEW [15:06:13] 06serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9976878 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host mw1349.eqiad.wmnet with OS buster [15:06:48] 06serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9976881 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host mw1350.eqiad.wmnet with OS buster [15:07:07] 06serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9976882 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host mw1351.eqiad.wmnet with OS buster [15:44:27] swfrench-wmf: sukhe: https://gitlab.wikimedia.org/repos/sre/conftool/-/commit/2c9e3611af31fec0f75cd5a1a090b0bb95d90084 :) [15:44:50] <_joe_> yep pretty amazing tbh, thanks cdanis [15:45:10] <_joe_> now we need to make sure the process collab sets up to collect deps will collect all these [15:45:25] <_joe_> and well, first check it works as expected I guess :) [15:45:40] yeah, it's only a merge into main that triggers deb builds, correct? [15:45:46] <_joe_> yes [15:45:55] <_joe_> a merge in main where debian/changelog changes [15:45:57] <_joe_> to be precise [15:46:18] <_joe_> I thought it was the most natural way to trigger a debian build [15:46:30] <_joe_> it also ensures we don't rebuild the same version [15:46:57] sure [15:47:26] I'm in over my head here tbh, this is the first time I've touched a .gitlab-ci.yml file, and I have no idea about the artifacts collection process afterwards [15:47:39] nice! [15:47:58] cdanis: nice indeed and can be used elsewhere [15:48:27] sukhe: arguably there should be some support for this in debci, but we can burn that bridge when we get to it [15:48:49] hmmm ... `DEB_VERSION_NUMBER: "11"` [15:48:52] 12? [15:48:57] uh is it really [15:49:04] lol [15:49:23] look, I got all the tricky parts working, ok [15:49:26] lol [15:50:03] to be clear, this is awesome - thank you very much! [15:51:20] ok I pushed a fix [15:51:26] thank you! [15:51:42] <_joe_> cdanis: I hope you just pushed to main :) [15:51:46] I did [15:59:01] 06serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9977120 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host mw1349.eqiad.wmnet with OS buster completed: - mw1349 (**WARN**) - Remove... [16:00:05] 06serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9977125 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host mw1350.eqiad.wmnet with OS buster completed: - mw1350 (**WARN**) - Remove... [16:01:48] 06serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9977137 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host mw1351.eqiad.wmnet with OS buster completed: - mw1351 (**PASS**) - Remove... [16:09:15] https://gitlab.wikimedia.org/repos/sre/conftool/-/pipelines/64080 [16:09:19] well, let's see [16:09:20] 06serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9977150 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host mw1349.eqiad.wmnet with OS buster [16:10:01] 06serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9977157 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host mw1350.eqiad.wmnet with OS buster [16:10:38] 06serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9977164 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host mw1351.eqiad.wmnet with OS buster [16:11:53] Uploading artifacts as "archive" to coordinator... 201 Created id=310992 responseStatus=201 Created token=glcbt-64 [16:11:58] I see this from the bookworm build job [16:12:24] https://gitlab.wikimedia.org/repos/sre/conftool/-/artifacts [16:16:55] does anyone know more about the "This might change soon as we're in the process of automating deb package uploads" note on the conftool release instructions? [16:20:07] 06serviceops, 10conftool, 13Patch-For-Review: Move conftool to gitlab, turn on deb package auto-generation - https://phabricator.wikimedia.org/T369594#9977244 (10CDanis) Today we merged https://gitlab.wikimedia.org/repos/sre/conftool/-/commit/2c9e3611af31fec0f75cd5a1a090b0bb95d90084 which automated the 'back... [16:20:20] I see T304491 but that doesn't discuss automated uploading [16:26:06] 06serviceops, 06Infrastructure-Foundations, 13Patch-For-Review: Cleanup old Docker images running Debian Stretch/Jessie - https://phabricator.wikimedia.org/T367427#9977284 (10elukey) Next steps: - Merge and package https://gerrit.wikimedia.org/r/c/operations/docker-images/docker-report/+/966200 - Run `docker... [16:36:34] 06serviceops: Support warmup for local caches in mw-on-k8s - https://phabricator.wikimedia.org/T369921 (10Scott_French) 03NEW [17:00:53] 06serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9977438 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host mw1349.eqiad.wmnet with OS buster completed: - mw1349 (**WARN**) - Downti... [17:03:12] 06serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9977472 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host mw1350.eqiad.wmnet with OS buster completed: - mw1350 (**WARN**) - Downti... [17:06:27] 06serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9977476 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host mw1351.eqiad.wmnet with OS buster completed: - mw1351 (**WARN**) - Downti... [17:13:09] 06serviceops, 13Patch-For-Review: Support warmup for local caches in mw-on-k8s - https://phabricator.wikimedia.org/T369921#9977485 (10Scott_French) p:05Triage→03Low [17:19:20] 06serviceops, 10ChangeProp, 10MediaWiki-Core-HTTP-Cache, 06MediaWiki-Engineering, and 2 others: Reduce the number of resource_change and resource_purge events emitted due to template changes - https://phabricator.wikimedia.org/T369898#9977505 (10Umherirrender) [17:40:48] cdanis: I guess I'd assumed that was a reference to the manual rebuild commits immediately above that note. [17:41:14] but yeah, _j.oe_ would be able to say authoritatively :) [17:46:08] I thought "uploads" meant specifically to the apt repo, but idk :) [17:56:16] ah, I see - yeah, I guess I could see it applying to the section immediately below (indeed, it does specify "uploads"). [17:56:16] I'd still defer to _j.oe_ though :) [18:53:36] 06serviceops: sextant: support module garbage collection - https://phabricator.wikimedia.org/T369932 (10Scott_French) 03NEW [18:53:50] 06serviceops: sextant: support module garbage collection - https://phabricator.wikimedia.org/T369932#9977802 (10Scott_French) p:05Triage→03Low [21:00:28] 06serviceops, 06SRE: deployment_server bullseye - mw-cgroup.service: Failed - https://phabricator.wikimedia.org/T363957#9978281 (10thcipriani) Thanks for documenting this, ran into the same thing in deployment prep (T327742), reboot also fixed it there. [22:38:27] 06serviceops, 06Content-Transform-Team-WIP, 10iOS-app-feature-Performance, 10RESTBase, and 6 others: PCS caching and pregeneration when restbase is decommissioned - https://phabricator.wikimedia.org/T319365#9978558 (10JTannerWMF)