[00:06:13] (DiskSpace) resolved: Disk space puppetmaster1001:9100:/ 4.835% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=puppetmaster1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [02:03:42] (SystemdUnitFailed) firing: docker-reporter-k8s-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:03:42] (SystemdUnitFailed) firing: docker-reporter-k8s-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:39:37] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Add per-output queue monitoring for Juniper network devices - https://phabricator.wikimedia.org/T326322 (10ayounsi) Update on the JTAC case, they're still working on it, they tested it without SSL and it worked fine, but: > Hope you are doi... [07:46:30] jbond, slyngs, juniper finally finished their side of the configuration for their staging API, they emailed us two docs with instruction on how to use it. I'll need your help to actually integrate it with our OIDC, maybe as a small POC python script to fetch data from them? [07:55:25] at least to make 1 call to check that it works so we can ask them to give us prod access [08:10:33] I'm so far only getting a 401 following their instructions [08:35:23] 10netbox, 10netops, 10Infrastructure-Foundations, 10SRE: Netbox Juniper report - https://phabricator.wikimedia.org/T306238 (10ayounsi) They replied back saying that the staging env is ready. Following their instructions I get a 401, to be investigated. [09:26:15] 10netops, 10Infrastructure-Foundations, 10SRE: Put Dell SONiC switches in production - https://phabricator.wikimedia.org/T335028 (10ayounsi) [10:04:31] (SystemdUnitFailed) firing: docker-reporter-k8s-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:31:44] XioNoX: we need to know what they have ultimatly used for the client id and client secret. in the document they say N/A OpenId based. however OpenID connect sits on top of Oauth2.0 so i dont think that really make senses [10:32:32] i tried a bunch of combinations using the appid, customerSourceID, andthe secret we have in the private repo and also only got 401 [11:47:53] we asked them to use "juniper" as client ID and Audience ID [11:48:11] not sure what's appid or customerSourceID but it doesn't seem required from their examples [11:49:04] jbond: does their server queries our server when we make a curl request? [11:49:13] do we have any logs on our side? [12:03:30] 10Packaging, 10Infrastructure-Foundations, 10Thumbor, 10Wikimedia-SVG-rendering: Update librsvg to > 2.44.10 - https://phabricator.wikimedia.org/T265549 (10hnowlan) The upgrade to bullseye is ready for [[ https://gerrit.wikimedia.org/r/c/operations/software/thumbor-plugins/+/920760 | review ]] (as part of... [12:06:24] XioNoX: there shgouldn;t be any interaction with our serveres see https://auth0.com/docs/get-started/authentication-and-authorization-flow/client-credentials-flow for an example of the flow. in this instance we (curl) are the M2M app [12:17:57] jbond: just replied to them, does my reply make sens? [12:19:18] XioNoX: yes that make senses however i would say to use printf instead of echo when yuopipe to base64 as the later will add `\n` [12:19:27] however i tried using printf and itstill failed [12:19:40] yeah I tried base64 -i and the output was the same [12:20:23] interesting, -i doesn't remove the \n [12:20:43] yes i guess the consider that an alphabet character :/ [12:25:06] XioNoX: -i is for decoding [12:33:03] ah right [12:34:52] sent a rectification email jsut to be sure [14:08:42] (SystemdUnitFailed) firing: docker-reporter-k8s-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:13:21] 10netops, 10Infrastructure-Foundations, 10SRE, 10observability, 10SRE Observability (FY2023/2024-Q2): librenms.syslog table size - https://phabricator.wikimedia.org/T349362 (10lmata) [16:21:02] Quick question, does our CAS/SSO system support OAuth2? I know that we can use it for OIDC now, but I'm wondering if this OAuth2 config is likely to work for Superset. https://superset.apache.org/docs/installation/configuring-superset/#custom-oauth2-configuration [17:23:42] (SystemdUnitFailed) firing: (2) docker-reporter-k8s-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:58:42] (SystemdUnitFailed) firing: (3) docker-reporter-k8s-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:23:42] (SystemdUnitFailed) firing: (2) docker-reporter-k8s-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:13:13] (DiskSpace) firing: Disk space puppetmaster1001:9100:/ 5.946% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=puppetmaster1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [22:23:42] (SystemdUnitFailed) firing: docker-reporter-k8s-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed