[01:08:01] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12048730 (10Papaul)
[01:10:15] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12048732 (10Papaul)
[01:13:19] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: rack B2 maintenance 2026-07-01 11:00 am CT - https://phabricator.wikimedia.org/T429861#12048738 (10Papaul)
[05:13:43] <wikibugs>	 07Puppet, 06Release-Engineering-Team: registry-homepage-builder.py doesn't sort images as expected - https://phabricator.wikimedia.org/T388287#12048852 (10hashar) The `build-homepage` service is indeed failing https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status?orgId=1&from=now-5m&to=now&timezone=utc&var-...
[05:14:33] <hashar>	 elukey: good morning, the docker registry homepage builder script seems to fail after you have deployed my patch at https://gerrit.wikimedia.org/r/1304677
[05:14:48] <hashar>	 the pages are stuck to `Last updated at: 2026-06-22 08:09`
[05:14:53] <hashar>	 https://phabricator.wikimedia.org/T388287#12046149
[05:15:37] <hashar>	 I suspect there are some tags that are not recognized by debian.debian_support.NativeVersion :\
[05:16:12] <hashar>	 we can surely get the error from  `journalctl -u build-homepage`
[05:44:55] <wikibugs>	 10netops, 06Data-Persistence, 06Data-Platform-SRE, 06DC-Ops, and 5 others: codfw: rack B2 maintenance 2026-07-01 11:00 am CT - https://phabricator.wikimedia.org/T429861#12048903 (10ayounsi)
[05:57:33] <wikibugs>	 10netops, 06Data-Persistence, 06Data-Platform-SRE, 06DC-Ops, and 5 others: codfw: rack B2 maintenance 2026-07-01 11:00 am CT - https://phabricator.wikimedia.org/T429861#12048932 (10ayounsi)
[06:59:04] <wikibugs>	 07Puppet, 06Release-Engineering-Team: registry-homepage-builder.py doesn't sort images as expected - https://phabricator.wikimedia.org/T388287#12049032 (10elukey) ` Jun 24 06:24:07 registry2004 registry-homepage-builder[3522966]: INFO:root:Fetching the image catalog for localhost:5004 Jun 24 06:24:07 registry2...
[07:06:30] <elukey>	 hashar: o/ my proposal is https://gerrit.wikimedia.org/r/c/operations/puppet/+/1305330, lemme know what you think 
[07:11:17] <hashar>	 elukey: good morning, I did suspect some tags would not be valid Debian versions, I should have handled it -:)
[07:12:08] <elukey>	 hashar: I think it is fine! I had some suspicion but it was safe to test, your change is a good improvement :)
[07:13:18] <hashar>	 may you amend the commit message to mention the tag that failed?  `ValueError: Invalid version string 'k8s_116'`
[07:13:30] <elukey>	 yeah I added the str(e) error
[07:13:30] <hashar>	 that saves the future selves from having to look at the task and have it directly when doiing a git log
[07:13:38] <hashar>	 but yeah the graceful fallback is great
[07:14:10] <hashar>	 and maybe later we could replace underscores with dashes
[07:14:42] <hashar>	     re_valid_version = re.compile(
[07:14:42] <hashar>	         r"^((?P<epoch>\d+):)?"
[07:14:42] <hashar>	         "(?P<upstream_version>[A-Za-z0-9.+:~-]+?)"
[07:14:42] <hashar>	         "(-(?P<debian_revision>[A-Za-z0-9+.~]+))?$")
[07:14:43] <hashar>	 fun
[07:14:47] <hashar>	 I should have read that :]
[07:15:57] <hashar>	 I am amending the commit message
[07:16:52] <hashar>	 elukey: amended and +1ed :]
[07:17:54] <hashar>	 that will teach me to be careless, I was really hoping all tags had valid version based on the wrongful assumption that all images are generated by docker-pkg (which relies on Debian changelog version to forge the tag and thus is guaranteed to be valid)
[07:18:14] <hashar>	 hope engineering failed again :b
[07:22:12] <elukey>	 we were not changing the nginx config of the registry, the change was low impact and the homepage builder has a graceful error handling. It was a very safe experiment to run :)
[07:31:31] <hashar>	 indeed
[07:32:46] <hashar>	 elukey: may you manually run it on the registry hosts so we can verify?  Ithink the command would be `sudo systemctl start build-homepage.service`
[07:38:53] <elukey>	 hashar: puppet will take care of that
[07:38:59] * hashar nods
[07:46:00] <elukey>	 hashar: it is running on registry2005, I'll let you know the end result
[07:47:44] <hashar>	 cool! and to verify my fix we can refresh https://docker-registry.wikimedia.org/releng/node22-test-browser/tags/
[07:47:44] <hashar>	 - "Last updated" should have been bumped
[07:47:44] <hashar>	 - 22.6.0 should be the 4th item (after the 22.0.0{,-s1,-s2} tags).  It is currently the last just before "latest"
[09:36:36] <elukey>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1305357 /me hides in shame
[09:36:44] <elukey>	 let's wait another bit :D
[09:40:17] <wikibugs>	 10netops, 06Data-Persistence, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: codfw: rack A6 maintenance - https://phabricator.wikimedia.org/T429812#12049442 (10jcrespo) I've stopped ms-backups2003 network operations for now, codfw media backups will continue to flow temporarily only through ms-backu...
[10:33:03] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Kubernetes: Calico IPv4/IPv6 block exhaustion on dse-k8s cluster, blocking new node provisioning - https://phabricator.wikimedia.org/T429773#12049644 (10BTullis) I think that what I'm going to do in the short term is t...
[12:02:01] <wikibugs>	 10netops, 06Data-Persistence, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: codfw: rack A6 maintenance - https://phabricator.wikimedia.org/T429812#12049887 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=b3768ce5-4982-4cdb-ac8d-3735e9e5290b) set by ayounsi@cumin1003 for 2:00:00...
[12:02:58] <wikibugs>	 10netops, 06Data-Persistence, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: codfw: rack A6 maintenance - https://phabricator.wikimedia.org/T429812#12049888 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=ac4b8e18-e54c-4708-804e-e3c84d435ded) set by ayounsi@cumin1003 for 2:00:00...
[12:18:45] <elukey>	 hashar: the builder now works, but the page is still cached by the CDN, so it may take a bit before we see the new ordering
[12:42:09] <wikibugs>	 10netops, 06Data-Persistence, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: codfw: rack A6 maintenance - https://phabricator.wikimedia.org/T429812#12049997 (10ayounsi) 05Open→03Resolved All done, and all services re-pooled.
[13:23:53] <hashar>	 elukey: confirmed :]
[13:24:04] <elukey>	 nice!
[13:24:05] <hashar>	 I have purged https://docker-registry.wikimedia.org/releng/node22-test-browser/tags/  and it shows up with the properly sorted tags
[13:26:47] <wikibugs>	 10netops, 06Data-Persistence, 06Data-Platform-SRE, 06DC-Ops, and 5 others: codfw: rack B2 maintenance 2026-07-01 11:00 am CT - https://phabricator.wikimedia.org/T429861#12050252 (10ayounsi)
[13:30:46] <wikibugs>	 07Puppet, 06Release-Engineering-Team: registry-homepage-builder.py doesn't sort images as expected - https://phabricator.wikimedia.org/T388287#12050271 (10hashar) The pages got generated. I went to purge the example page: ` $ mwscript purgeList --wiki=aawiki https://docker-registry.wikimedia.org/releng/node22-...
[13:30:57] <wikibugs>	 07Puppet, 06Release-Engineering-Team: registry-homepage-builder.py doesn't sort images as expected - https://phabricator.wikimedia.org/T388287#12050274 (10hashar) 05Open→03Resolved
[13:31:05] <hashar>	 elukey: also don't you have a Phabricator tag for the Docker registry?
[13:41:23] <elukey>	 hashar: not that I know of. The I/F team is going to manage it more and more in the future IIUC, but so far it has been in a limbo. 
[13:43:31] <hashar>	 I have tagged it with #infrastructure-foundations :]
[13:43:33] <hashar>	 thx!
[17:53:07] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Kubernetes: Calico IPv4/IPv6 block exhaustion on dse-k8s cluster, blocking new node provisioning - https://phabricator.wikimedia.org/T429773#12051595 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was star...
[17:53:09] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Kubernetes: Calico IPv4/IPv6 block exhaustion on dse-k8s cluster, blocking new node provisioning - https://phabricator.wikimedia.org/T429773#12051596 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was star...
[17:53:10] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Kubernetes: Calico IPv4/IPv6 block exhaustion on dse-k8s cluster, blocking new node provisioning - https://phabricator.wikimedia.org/T429773#12051597 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was star...
[18:31:45] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Kubernetes: Calico IPv4/IPv6 block exhaustion on dse-k8s cluster, blocking new node provisioning - https://phabricator.wikimedia.org/T429773#12051757 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started...
[18:35:10] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Kubernetes: Calico IPv4/IPv6 block exhaustion on dse-k8s cluster, blocking new node provisioning - https://phabricator.wikimedia.org/T429773#12051761 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started...
[18:39:43] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Kubernetes: Calico IPv4/IPv6 block exhaustion on dse-k8s cluster, blocking new node provisioning - https://phabricator.wikimedia.org/T429773#12051786 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started...