[01:30:04] FIRING: [2x] PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [03:33:31] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [05:30:04] FIRING: [2x] PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [06:37:25] FIRING: SystemdUnitFailed: prometheus-ganeti-exporter.service on ganeti2049:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:33:32] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [07:56:07] 10netbox, 06Infrastructure-Foundations: Upgrade Netbox to 4.3.x - https://phabricator.wikimedia.org/T371889#10957331 (10ayounsi) [09:30:04] FIRING: [2x] PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [11:33:32] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [12:40:57] 10Mail, 06Infrastructure-Foundations, 10MediaWiki-Email, 10MediaWiki-extensions-EmailAuth, and 3 others: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#10958413 (10Tgr) I see the patch got merged upstream, I assume that meets the "look... [13:30:04] FIRING: [2x] PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [14:09:05] 10CAS-SSO, 10Gerrit, 06Infrastructure-Foundations: gerrit.wikimedia.org should use IDP for login - https://phabricator.wikimedia.org/T147864#10958761 (10Arendpieter) [15:28:39] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: network and DNS configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10959219 (10Jgreen) [15:33:32] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [15:34:43] volans: should we abandon https://gerrit.wikimedia.org/r/c/operations/software/cumin/+/514840 ? do we still need https://phabricator.wikimedia.org/T205900 ? we have lived without it until now [15:35:52] XioNoX: good question, I think l.uca was interested, and in some occasional cases it would be nice to use but not 100% if worth the time [15:36:11] what worries me the most is to make it compatible with multiple netbox versions without becoming crazy [15:36:23] also for anyone, any ideas for column categories for https://phabricator.wikimedia.org/tag/netbox/ ? [15:36:29] ideally we should ask around the SREs to see if they could find it useful or not [15:36:46] volans: if you ask you will get "yes" from people who won't write it :) [15:36:56] lol [15:37:05] I'd say we close if and reopen if someone ask for it [15:39:58] fair enough for me [15:40:50] topranks: https://phabricator.wikimedia.org/T378730 is that for you ? [15:41:58] topranks: I guess you can close https://phabricator.wikimedia.org/T271577 as duplicate of your hackathon project [15:42:40] 10netbox, 06Infrastructure-Foundations: Make DNS generation faster - https://phabricator.wikimedia.org/T271577#10959332 (10cmooney) [15:42:58] XioNoX: thanks, I'll make it a child for now [15:43:13] as regards the cassandra one yeah let me take a look at what's going wrong [15:44:46] 10netbox, 06Infrastructure-Foundations, 06SRE: Netbox script for adding secondary IPs - https://phabricator.wikimedia.org/T378730#10959364 (10cmooney) a:03cmooney @Eevans sorry this one escaped me somehow let me take a look, agreed it seems there is something wrong here. [15:47:19] topranks: is that in the scope of your hackathon too https://phabricator.wikimedia.org/T276403 ? [15:48:00] XioNoX: no, but that's very easy to fix [15:48:03] lol @ the subtle plug [15:48:08] we can just add it periodically [15:48:24] I never went back to it [15:48:39] 10netbox, 06Infrastructure-Foundations: https://netbox-exports.wikimedia.org/dns.git is very slow to clone (using dumb HTTP) - https://phabricator.wikimedia.org/T276403#10959402 (10ayounsi) a:03Volans [15:48:56] https://usercontent.irccloud-cdn.com/file/wOUeC0Ep/Screenshot%202025-06-30%20at%2017-48-46%20%E2%9A%93%20T276403%20https%20__netbox-exports.wikimedia.org_dns.git%20is%20very%20slow%20to%20clone%20(using%20dumb%20HTTP).png [15:49:05] XioNoX: why assigned to me? [15:49:16] not planning to work on that anytime soon [15:49:20] "that's very easy to fix" :) [15:49:31] didn't we have that issue with another repo? we ran some git command to roll-up the previous commits or something which speeded things up? [15:49:54] 10netbox, 06Infrastructure-Foundations: https://netbox-exports.wikimedia.org/dns.git is very slow to clone (using dumb HTTP) - https://phabricator.wikimedia.org/T276403#10959405 (10ayounsi) a:05Volans→03None [15:49:57] fwiw we started doing that on the DNS hosts recently, as a systemd-timer that runs on the first Monday of a month [15:50:02] either way the path forward is probably the hackathon new repo? though it's not gonna be any faster I think [15:50:03] I usually run git gc [15:50:05] brought down authdns-update by a full minute [15:50:13] yep that was it [15:50:22] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1143593 [15:50:23] ok so it's already being done? perhaps we can just close that task then? [15:50:30] we could add it to the cookbook [15:50:39] before the sync run git gc [15:50:53] iirc correctly it was quite slow from remote pops, eqsin etc., due to latency combined with the number of HTTP requests [15:51:08] XioNoX: assign it to sukhe clearly has already solved the problem [15:51:22] haha well [15:51:24] for the more generic slowness I suggested some approaches in the task [15:51:26] where do we need to do the git gc? [15:51:34] topranks: all checkouts [15:51:38] so cumin hosts, netbox, authdns [15:52:05] ok, so yes adding it to the cookbook makes sense, one assumes if done on the local cumin box that will also sync with reposync? [15:52:35] 10netbox, 10netops, 06cloud-services-team, 10DNS, and 2 others: Cloud: define relationship between wikimediacloud.org domain, CIDR prefixes and netbox automation - https://phabricator.wikimedia.org/T266331#10959423 (10ayounsi) 05Open→03Declined Closing for now, please reopen if needed. [15:53:10] the git gc? [15:53:17] no it's local to the checkout [15:53:24] per-repo [15:55:04] I am happy to take this fwiw [15:55:08] assign it to me [15:57:31] 10netbox, 06Infrastructure-Foundations: https://netbox-exports.wikimedia.org/dns.git is very slow to clone (using dumb HTTP) - https://phabricator.wikimedia.org/T276403#10959468 (10ayounsi) a:03ssingh [15:57:33] sukhe: <3 done [15:57:56] cool :) I am out tomorrow so will pick up on Wed [15:58:42] sukhe: that task is from Mar 4 2021, it can wait a bit longer :) [15:58:58] XioNoX: I owe topranks one so [15:59:40] topranks: I hope I didn't blow that one favor :) [16:01:26] hahaha sabotaging me :) [16:01:43] fyi, I sorted all the netbox backlog https://phabricator.wikimedia.org/project/board/4031/ [16:02:55] feel free to edit the categories, etc etc [16:07:22] thx [17:01:42] 10Mail, 06Infrastructure-Foundations, 10MediaWiki-Email, 10MediaWiki-extensions-EmailAuth, and 3 others: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#10959874 (10jhathaway) >>! In T383047#10958413, @Tgr wrote: > I see the patch got m... [17:30:04] FIRING: [2x] PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [19:33:32] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [20:17:25] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [20:32:25] RESOLVED: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [21:30:04] FIRING: [2x] PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [23:33:32] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts