[06:47:56] (EdgeTrafficDrop) firing: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [06:52:56] (EdgeTrafficDrop) resolved: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [07:00:26] (EdgeTrafficDrop) firing: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [07:05:26] (EdgeTrafficDrop) resolved: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [08:56:56] 10Traffic: Frequent backend server errors (503), happened several times in the last 2 days - https://phabricator.wikimedia.org/T297544 (10ema) p:05Triage→03High Hello @Ade56facc and @Yann, thanks for the bug reports. We may be looking at two different problems here, at least judging from the fact that some... [09:03:06] 10Traffic: Frequent server errors (503 and 502), happened several times in the last 2 days - https://phabricator.wikimedia.org/T297544 (10ema) [09:33:15] 10Traffic, 10SRE, 10Patch-For-Review: Test envoyproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T271421 (10Vgutierrez) [09:42:56] (EdgeTrafficDrop) firing: 64% request drop in text@ulsfo during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=ulsfo&var-cache_type=text - https://alerts.wikimedia.org [09:57:56] (EdgeTrafficDrop) resolved: 69% request drop in text@ulsfo during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=ulsfo&var-cache_type=text - https://alerts.wikimedia.org [12:39:56] (EdgeTrafficDrop) firing: 66% request drop in text@ulsfo during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=ulsfo&var-cache_type=text - https://alerts.wikimedia.org [12:49:56] (EdgeTrafficDrop) resolved: 69% request drop in text@ulsfo during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=ulsfo&var-cache_type=text - https://alerts.wikimedia.org [13:02:05] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10SRE, 10Traffic-Icebox: Collect netflow data for internal traffic - https://phabricator.wikimedia.org/T263277 (10ayounsi) >>! In T263277#7570288, @JAllemandou wrote: > Am I right in assuming that this data has the same schema as the original `n... [13:11:48] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10SRE, 10Traffic-Icebox: Collect netflow data for internal traffic - https://phabricator.wikimedia.org/T263277 (10JAllemandou) No need to detail the fields and schema :) About data augmentation, [[ https://github.com/wikimedia/analytics-refiner... [13:27:55] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10SRE, 10Traffic-Icebox: Collect netflow data for internal traffic - https://phabricator.wikimedia.org/T263277 (10ayounsi) Cool, only `ip_version` and `region` are useful here. [13:31:07] 10Traffic: Frequent server errors (503 and 502), happened several times in the last 2 days - https://phabricator.wikimedia.org/T297544 (10Ade56facc) >>! In T297544#7571621, @ema wrote: > > @Ade56facc: To further diagnose the problem and figure out whether or not we are looking at two distinct issues it would be... [14:24:10] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10SRE, 10Traffic-Icebox: Collect netflow data for internal traffic - https://phabricator.wikimedia.org/T263277 (10Ottomata) Would it hurt to keep the same augmentations? If the schema is the sameish (it sounds like it is), we can just apply the... [14:30:52] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10SRE, 10Traffic-Icebox: Collect netflow data for internal traffic - https://phabricator.wikimedia.org/T263277 (10JAllemandou) >>! In T263277#7572471, @Ottomata wrote: > Would it hurt to keep the same augmentations? If the schema is the sameish... [14:34:45] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10SRE, 10Traffic-Icebox: Collect netflow data for internal traffic - https://phabricator.wikimedia.org/T263277 (10Ottomata) I'd prefer to avoid scheduling another special job for this if we can. Can we make the NetflowTransform functions smart... [14:37:33] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10SRE, 10Traffic-Icebox: Collect netflow data for internal traffic - https://phabricator.wikimedia.org/T263277 (10JAllemandou) I have the opposite view: I'd rather have another job instead of custom logic to prevent doing something :) [14:43:38] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10SRE, 10Traffic-Icebox: Collect netflow data for internal traffic - https://phabricator.wikimedia.org/T263277 (10Ottomata) Yeahh...but then we have to manage and maintain another custom ingestion job. We're trying to reduce the number of those... [15:30:25] 10Traffic: Frequent server errors (503 and 502), happened several times in the last 2 days - https://phabricator.wikimedia.org/T297544 (10Ade56facc) Talking about changes to above mentioned wiki article (web server) it usually takes around 1.5 .. 2 seconds to send (POST) data to server and to receive the updated... [18:05:22] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10SRE, 10Traffic-Icebox: Collect netflow data for internal traffic - https://phabricator.wikimedia.org/T263277 (10JAllemandou) >>! In T263277#7572522, @Ottomata wrote: > The custom logic could even just be varied on the hardcoded stream / tablen... [19:58:39] 10Traffic, 10Application Security Reviews, 10SRE, 10Security Awareness, and 3 others: Wikipedia Birthday 2022 - https://phabricator.wikimedia.org/T297816 (10Dzahn) [20:12:47] 10Traffic, 10Application Security Reviews, 10SRE, 10Security Awareness, and 3 others: Wikipedia Birthday 2022 - https://phabricator.wikimedia.org/T297816 (10MSRamos) Hi. I'm already in contact with the security team and have already received a Privacy Engineering review because I was told that this would b... [20:14:24] 10Traffic, 10Application Security Reviews, 10SRE, 10Security Awareness, and 3 others: Wikipedia Birthday 2022 - https://phabricator.wikimedia.org/T297816 (10RhinosF1) > So, no, the assessment made before, when this tasked was closed, that this wasn't submitted to any team is incorrect How is anyone on Phab... [20:18:10] 10Traffic, 10Application Security Reviews, 10SRE, 10Security Awareness, and 3 others: Wikipedia Birthday 2022 - https://phabricator.wikimedia.org/T297816 (10MSRamos) I was under the impression that the Application Security team had received a heads-up regarding the project. As for tags, this being my first... [20:27:07] 10Traffic, 10Application Security Reviews, 10SRE, 10Security-Team, and 2 others: Wikipedia Birthday 2022 - https://phabricator.wikimedia.org/T297816 (10sbassett) Hey @MSRamos - Yes, the #security-team did get a heads up about this. Typically we follow [[ https://www.mediawiki.org/wiki/Security/SOP/Applic... [20:28:29] 10Traffic, 10Application Security Reviews, 10SRE, 10Security-Team, and 2 others: Application Security Review Request : Wikipedia Birthday 2022 - https://phabricator.wikimedia.org/T297816 (10sbassett) [20:32:48] 10Traffic, 10Application Security Reviews, 10SRE, 10Security-Team, and 2 others: Application Security Review Request : Wikipedia Birthday 2022 - https://phabricator.wikimedia.org/T297816 (10Dzahn) Hi @MSRamos, so there really is more to this than just the security review. One thing for example is the fol... [20:35:17] 10Traffic, 10Application Security Reviews, 10SRE, 10Security-Team, and 2 others: Application Security Review Request : Wikipedia Birthday 2022 - https://phabricator.wikimedia.org/T297816 (10sbassett) >>! In T297816#7573659, @Dzahn wrote: > wikimediafoundation.org is not running on WIkimedia's servers. That... [20:41:05] 10Traffic, 10Application Security Reviews, 10SRE, 10Security-Team, and 2 others: Application Security Review Request : Wikipedia Birthday 2022 - https://phabricator.wikimedia.org/T297816 (10Dzahn) Ok, thank you @sbassett. (This is a good example why this kind of thing needs clarification and planning ear... [20:43:24] 10Traffic, 10Application Security Reviews, 10SRE, 10Security-Team, and 2 others: Application Security Review Request : Wikipedia Birthday 2022 - https://phabricator.wikimedia.org/T297816 (10sbassett) [20:43:47] 10Traffic, 10Application Security Reviews, 10SRE, 10Security-Team, and 2 others: Application Security Review Request : Wikipedia Birthday 2022 - https://phabricator.wikimedia.org/T297816 (10sbassett) [20:44:28] 10Traffic, 10Application Security Reviews, 10SRE, 10Security-Team, and 2 others: Application Security Review Request : Wikipedia Birthday 2022 - https://phabricator.wikimedia.org/T297816 (10MSRamos) [20:50:17] 10Traffic, 10Application Security Reviews, 10SRE, 10Security-Team, and 2 others: Application Security Review Request : Wikipedia Birthday 2022 - https://phabricator.wikimedia.org/T297816 (10MSRamos) @sbassett @Dzahn The landing page will live on wikimediafoundation.org, like our 20th Birthday Landing Page... [20:51:18] 10Traffic, 10Application Security Reviews, 10SRE, 10Security-Team, and 2 others: Application Security Review Request : Wikipedia Birthday 2022 - https://phabricator.wikimedia.org/T297816 (10sbassett) [20:52:30] 10Traffic, 10Application Security Reviews, 10SRE, 10Security-Team, and 2 others: Application Security Review Request : Wikipedia Birthday 2022 - https://phabricator.wikimedia.org/T297816 (10sbassett) >>! In T297816#7573717, @MSRamos wrote: > @sbassett @Dzahn The landing page will live on wikimediafoundatio... [20:58:21] 10Traffic, 10Application Security Reviews, 10SRE, 10Security-Team, and 2 others: Application Security Review Request : Wikipedia Birthday 2022 - https://phabricator.wikimedia.org/T297816 (10MSRamos) @sbassett Correct! It lives under our Corp website umbrella. Happy to clarify if any other questions arise!... [22:53:45] 10Traffic, 10Application Security Reviews, 10SRE, 10Security-Team, and 2 others: Application Security Review Request : Wikipedia Birthday 2022 - https://phabricator.wikimedia.org/T297816 (10Dzahn) @MSRamos Thank you for the clarification! Yes, as Scott said this makes this a bit less of a concern. You can...