[09:12:50] so we map countries to DCs via DNS [09:14:10] mutante: which cert is triggering the error? [09:15:07] shop.wikimedia.org in Europe is the Digicert version of the unified cert [09:15:18] but that's redirected to store.wikimedia.org [09:15:28] and that's handled by shopify [09:15:37] and it's a Let's Encrypt issued cert [09:59:34] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE: Reverse DNS zones includes for drmrs - https://phabricator.wikimedia.org/T301447 (10Volans) [09:59:39] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE: Reverse DNS zones includes for drmrs - https://phabricator.wikimedia.org/T301447 (10Volans) p:05Triage→03Medium [13:55:25] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, and 2 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10dcausse) For some production jobs we still use the proxy to access: - MW APIs (all our sites) - ores.wikimedia.org For... [14:05:42] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, and 2 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10Ottomata) > MW APIs (all our sites) BTW, the proper way to access MW APIs from within our networks is to use e.g. https... [14:39:00] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, and 2 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10akosiaris) >>! In T300977#7700725, @Ottomata wrote: >> MW APIs (all our sites) > BTW, the proper way to access MW APIs... [14:43:57] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, and 2 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10Ottomata) > avoiding a SPOF (there aren't that many web proxies nor is it a highly available setup cause there isn't an... [14:47:02] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, and 2 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10jbond) > This has bitten me before when I used to use the webproxy internally. Don't do it! :) Its worth mentioning th... [15:19:25] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, and 2 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10Ottomata) Hahah, maybe what we should do is excludelist the internal domains in the webproxy! [15:44:03] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, and 2 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10mpopov) A couple of questions/comments: >>! In T300977#7700842, @jbond wrote: > Its worth mentioning that when i took... [16:01:04] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, and 2 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10EBernhardson) > Oh and ORES is also available under https://ores.discovery.wmnet (and it's the exact same service!) Do... [16:01:26] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, and 2 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10mpopov) > **First**: How difficult & how much overhead would it be to make the proxy redirect requests made to internal... [16:05:01] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, and 2 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10Ottomata) > Is the intention to allow us to talk to prod in a more general fashion then? I think so, see the parent tic... [16:29:43] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, and 2 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10jbond) > First: How difficult & how much overhead would it be to make the proxy redirect requests made to internal doma... [16:35:01] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE: Reverse DNS zones includes for drmrs - https://phabricator.wikimedia.org/T301447 (10cmooney) Added to patch for new Eqiad includes: https://gerrit.wikimedia.org/r/c/operations/dns/+/761473 [16:37:37] vgutierrez: it was shop.wikimedia.org on Watchmouse monitoring, only when coming from Warsaw, Poland [16:50:50] so thats the digicert certificate unless they are doing some weird stuff on their DNS clients [16:59:57] either that, or watchmouse is following the redirect first and then reporting on the shopify cert [17:00:57] does it actually show a hash or signature or anything of the cert it didn't like? [17:03:04] bblack: check in your inbox "ALERT! shop.wikimedia.org: SSL certificate problem" [17:03:19] it has a link to the report, but just says expired, I didn't find any useful info there [17:47:31] I went to the watchmouse UI. you can find the credentials in pwstore [17:47:42] there I could see it only ever failed from Warsaw but never any other [17:48:00] I even clicked temp. to make it only check from 3 specific countries and every minute instead of every 15 [17:48:15] only failed from the Poland location.. then I switched it back to normal (random, worldwide, 15 min) [17:48:53] the error sounded like it could also be CA expired or missing CA certs I think [18:23:47] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE: Reverse DNS zones includes for drmrs - https://phabricator.wikimedia.org/T301447 (10cmooney) 05Open→03Resolved a:03cmooney Working ok after merge: ` cmooney@wikilap:~/repos/random_wmf/netbox_scripts$ dig +noall +answer -x 2620:0:860:fe0a::1 @n... [18:23:51] 10Traffic, 10SRE, 10Patch-For-Review: Configure dns and puppet repositories for new drmrs datacenter - https://phabricator.wikimedia.org/T282787 (10cmooney) [19:39:56] (EdgeTrafficDrop) firing: 49% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org [19:44:56] (EdgeTrafficDrop) resolved: 49% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org [20:01:57] (EdgeTrafficDrop) firing: 41% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org [20:11:57] (EdgeTrafficDrop) resolved: 38% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org