[02:19:31] (SystemdUnitFailed) firing: generate_os_reports.service Failed on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:19:31] (SystemdUnitFailed) firing: generate_os_reports.service Failed on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:40:06] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [10:07:55] Hey :) the debmonitor-client packae is built by us right? I'm trying to find where so I can add some missing dependencies (python3-apt and python3-setuptools) [10:08:18] It's making docker-report fail and I could manually install these packages through docker-report but I don't think that's the best solution [10:09:15] claime: which version do you have? we recently migrated it to its standalone repo [10:10:24] mmmh interesting] [10:10:49] slyngs: something didn't work as expected with the debmonitor-client deps [10:10:53] $ apt-cache depends debmonitor-client [10:10:53] debmonitor-client [10:10:53] Depends: [10:10:53] python3 [10:10:53] Depends: apt [10:11:53] We apparently install 3 different versions I guess depending on debian version? [10:12:25] while ${python3:Depends} should have brought in python3-apt and python3-requests [10:12:39] claime: the last deb* part is for the 3 different OSes [10:12:42] yes [10:12:52] That was my impression as will, ${python3:Depends} [10:12:57] 0.3.4 is the latest [10:13:03] So 0.3.4-1 [10:14:26] mmmh something didn't work as expected with dh_python/pybuild? [10:17:56] Rebuilding the package yields: Depends: python3:any, apt [10:17:59] slyngs: ok I found why dh_python fails [10:18:10] Cool.... why :-) [10:18:22] because debmonitor_client.egg-info/requires.txt doesn't have any deps for the package, just for testing, looking why [10:19:15] slyngs: doh install_requires is not used in setup.py [10:19:21] install_requires is defined but never used [10:19:27] Yup [10:19:31] sending a patch [10:20:46] (SystemdUnitFailed) firing: generate_os_reports.service Failed on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:22:01] claime: are you sure python3-setuptools is required? [10:22:33] ah yes, for pkg_resources [10:22:39] volans: it fails importing pkg_resources on buster [10:24:19] slyngs: https://gerrit.wikimedia.org/r/c/operations/software/debmonitor-client/+/998326 [10:25:18] claime: once merged I'll make a release shortly [10:25:23] awesome thanks [10:25:42] sorry for the trouble [10:25:49] no worries :) [10:29:08] volans: your updated patch still says install_required :-) [10:29:33] rotfl, git add didn't work because of a typo [10:29:38] done now [10:30:28] what did I do now [10:31:48] Isn't there something weird about apt not actually being available in pypi [10:32:18] I think it's by choice [10:32:36] https://pypi.org/project/python-apt/ [10:33:17] Their is no active plans to distribute PyPi at this time. [10:33:21] from https://answers.launchpad.net/ubuntu/+source/python-apt/+question/707491 [10:34:42] Okay, so we yank it from setup.py and have it in control [10:38:07] let's see if the third time is a charm [10:39:00] (but we're already at PS4) [10:39:11] given my empty change PS [10:39:34] Those are rookie numbers, I've hit 20+ on some patches [11:14:24] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [11:42:22] slyngs: when you have time for a sanity check :) https://gerrit.wikimedia.org/r/c/operations/software/debmonitor-client/+/998355 [11:44:35] open question: should we archive the operations/software/debmonitor/deploy repository at this point? [11:49:54] I don't think so. On the contrary, let's all pretend it never existed in the first place :-) [11:51:07] lol [11:51:18] You simply refuse acknowledge that any deployment strategy beyond apt may exist :-) [11:51:43] volans: Done [11:51:47] thx [11:52:29] btw if you're interested I added the repo to https://www.mediawiki.org/wiki/Git/Reviewers#operations/software/debmonitor-client [11:58:48] slyngs: dpkg -I now gives [11:58:48] Depends: python3-pkg-resources, python3-requests, python3:any, apt, python3-apt [11:59:44] Perfect, python3-pkg-resources should pull in setuptools if it needs it [12:00:06] it pulls pkg_resources that is what is used [12:00:09] so should be good [12:02:38] 10netbox, 10Data-Engineering, 10Data-Persistence-Backup, 10Infrastructure-Foundations, and 2 others: Convert Netbox data (PostgresQL) longterm storage backups (bacula) into full backups rather than incrementals - https://phabricator.wikimedia.org/T316655 (10jcrespo) This is looking good- I will check in th... [12:02:38] claime: debmonitor-client_0.3.5 released to apt.w.o, if you update apt you should get the latest and it should solve the problem [12:02:42] let us know if that works [12:03:11] volans: I'll relaunch the docker-report script and report back :) [12:03:22] thx [12:55:20] 10netbox, 10Data-Engineering, 10Data-Persistence-Backup, 10Infrastructure-Foundations, 10bacula: Convert Netbox data (PostgresQL) longterm storage backups (bacula) into full backups rather than incrementals - https://phabricator.wikimedia.org/T316655 (10Volans) 05Open→03Resolved @BTullis thanks for t... [14:18:00] volans: looks good, we're having another error but that's on our side, docker-registry proxy timeouts are a bit short [14:18:37] claime: glad to hear that solved it, less so about your new issue :) [14:18:50] and thanks for reporting back [14:19:00] I just doubled the nginx timeouts that should take care of it x) [14:19:18] moritzm: are you doing any fleet-wide upgrade? I might as well deploy debmonitor-client_0.3.5 at this point [14:20:47] (SystemdUnitFailed) firing: generate_os_reports.service Failed on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:28:55] volans: go ahead, not currently rolling out anything fleetwide [14:29:04] ack thx [14:39:27] {done} [14:41:14] cheers [14:43:22] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack A2 from asw-a2-codfw to lsw1-a2-codfw - https://phabricator.wikimedia.org/T355861 (10klausman) [14:45:14] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack A2 from asw-a2-codfw to lsw1-a2-codfw - https://phabricator.wikimedia.org/T355861 (10klausman) [15:23:36] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A3 from asw-a3-codfw to lsw1-a3-codfw - https://phabricator.wikimedia.org/T355862 (10Jhancock.wm) This rack is physically ready [15:38:05] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack A7 from asw-a7-codfw to lsw1-a7-codfw - https://phabricator.wikimedia.org/T355867 (10Andrew) There's no need to coordinate with us for cloudbackup2001, it might cause us to get a transient alert... [18:20:47] (SystemdUnitFailed) firing: generate_os_reports.service Failed on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:47:06] 10Mail, 10Infrastructure-Foundations, 10SRE: Access to DMARCIAN - https://phabricator.wikimedia.org/T356920 (10Dzahn) [21:48:40] 10Mail, 10Infrastructure-Foundations, 10SRE: Access to DMARCIAN - https://phabricator.wikimedia.org/T356920 (10Dzahn) Looks like dmarcian has been replaced by dmarcdigests.com. (details in T330944). Adding some tags for visibility. [22:20:47] (SystemdUnitFailed) firing: generate_os_reports.service Failed on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed