[00:04:13] (DiskSpace) resolved: Disk space puppetmaster1001:9100:/ 5.761% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=puppetmaster1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [00:05:13] (SystemdUnitFailed) firing: (8) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:15:13] (SystemdUnitFailed) firing: (8) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:20:13] (SystemdUnitFailed) firing: (8) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:05:13] (SystemdUnitFailed) firing: (8) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:10:13] (SystemdUnitFailed) firing: (8) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:40:28] (SystemdUnitFailed) firing: (8) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:45:13] (SystemdUnitFailed) firing: (8) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:45:13] (SystemdUnitFailed) firing: (8) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:40:13] (SystemdUnitFailed) firing: (9) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:22:41] 10netops, 10Infrastructure-Foundations, 10SRE: Plan codfw row A/B top-of-rack switch refresh - https://phabricator.wikimedia.org/T327938 (10RobH) [12:40:13] (SystemdUnitFailed) firing: (9) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:29:24] are any folks around for our monday morning meeting? [13:36:37] I am pretty everyone else but the americas is off for iternational workers day, so I am going to skip joining, unless someone hollers! [14:04:39] jhathaway: I think it's just you and me :-) [14:05:23] slyngs: surprising that denmark doesn't honor the day [14:07:08] It was half the day of until 1974, then it became something the employers and unions had to negotiate. That's pretty much how we prefer to deal with these things, that's also why there's no minimum wage. [14:08:14] I'm a member of the Engineers Union, and the foundation isn't of a member of any company group, so no May 1st for me :-) [14:11:58] ah! [14:12:48] If I had been in construction, or worked in a slaughterhouse, then I'd have half a day off [14:13:20] On the other hand I just called a contractor and he seems to be working. [14:14:36] often a necessity of the trade [14:16:25] Yeah, I called to ask if he forgot that he was to fix some bathroom tiles.. he's just insanely busy, there's weeks if not months of waiting period for any type of construction. [14:27:00] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 9 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10ssingh) [15:25:00] 10Puppet, 10Wikidata, 10Wikidata-Query-Service, 10wdwb-tech, 10Discovery-Search (Current work): Migrate WDQS to profile::java - https://phabricator.wikimedia.org/T264181 (10Gehel) [15:43:11] 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10Spicerack, 10Discovery-Search (Current work): Create cookbook to reindex into elasticsearch / cirrus - https://phabricator.wikimedia.org/T219507 (10MPhamWMF) [16:40:13] (SystemdUnitFailed) firing: (8) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:42:36] 10netbox, 10Infrastructure-Foundations, 10Traffic, 10Patch-For-Review: Issues converting services from active/passive to active/active - https://phabricator.wikimedia.org/T330084 (10BCornwall) This ticket could do with a little more clarity: I'm going to Boldly assume this ticket is for identifying/fixing... [16:45:48] 10netbox, 10Infrastructure-Foundations, 10Traffic, 10Patch-For-Review: gdnsd failures when converting services from active/passive to active/active - https://phabricator.wikimedia.org/T330084 (10BCornwall) [16:54:20] 10SRE-tools, 10Infrastructure-Foundations, 10Traffic: Cookbook to depool a site in AuthDNS - https://phabricator.wikimedia.org/T334048 (10BCornwall) I'm hesitant to the idea of creating an abstraction over an abstraction - I may be an outlier but my experience with depooling has been with confctl rather than... [17:03:22] 10Mail, 10DNS, 10Fundraising-Backlog, 10Infrastructure-Foundations, and 2 others: Add support for Brand Indicators for Message Identification (BIMI) for wiki mail - https://phabricator.wikimedia.org/T311685 (10BCornwall) [17:31:27] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10Jclark-ctr) @Papaul Cables where connected to correct ports. i did swap cables while verifying Replaced Cable new cableid23030450029... [17:40:13] (SystemdUnitFailed) firing: (8) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:45:13] (SystemdUnitFailed) firing: (8) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:10:05] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10Traffic, 10ops-knams: Q4/Q1:knams racking elevations & planning - https://phabricator.wikimedia.org/T331886 (10BCornwall) [21:18:50] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 10 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10bking) [21:45:13] (SystemdUnitFailed) firing: (8) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:50:13] (SystemdUnitFailed) firing: (8) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:39:13] (DiskSpace) firing: Disk space puppetmaster1001:9100:/ 5.937% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=puppetmaster1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [23:45:13] (SystemdUnitFailed) firing: (8) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:50:13] (SystemdUnitFailed) firing: (8) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed