[04:43:25] 10Traffic, 10SRE: oom killed varnish on cp4047 - https://phabricator.wikimedia.org/T322903 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez THP has been disabled globally as a result of this task with https://gerrit.wikimedia.org/r/857686. A rolling restart has been performed to applied this change: ` vgu... [09:33:35] 10Traffic, 10Data Pipelines, 10Data-Engineering-Planning, 10Foundational Technology Requests, and 2 others: Add a webrequest sampled topic and ingest into druid/turnilo - https://phabricator.wikimedia.org/T314981 (10elukey) Status: We deployed benthos on two centrallog nodes, and we are now evaluating its... [11:48:22] 10Traffic: Wikipedia on flow with no http request, still responds with a Bad Request 400 - https://phabricator.wikimedia.org/T323263 (10taavi) [12:07:37] 10Traffic: Wikipedia on flow with no http request, still responds with a Bad Request 400 - https://phabricator.wikimedia.org/T323263 (10Vgutierrez) p:05Triage→03Lowest I'm curious about those firewalls considering that we are talking about TLSv1.3 traffic that shouldn't be able to be inspected by them. We de... [14:03:13] 10Traffic, 10SRE: Wikipedia on flow with no http request, still responds with a Bad Request 400 - https://phabricator.wikimedia.org/T323263 (10Mohawkdavitty) TLSv1.3 just prevents RSA handshake decryption using the website cert/key, TLSv1.3 uses forward perfect secrecy connections that prevents this, but the... [15:35:11] 10Traffic, 10SRE: Wikipedia on flow with no http request, still responds with a Bad Request 400 - https://phabricator.wikimedia.org/T323263 (10Vgutierrez) this seems to be triggered by HAProxy, I just logged the H1 trace on a cloud test instance using: ` echo "trace h1 event +any; trace h1 level developer; tra... [16:04:59] 10Traffic, 10Commons, 10MediaWiki-Uploading, 10SRE, and 2 others: 502 Server Hangup Error on esams for "Upload a new version of this file" on Special:Upload on Commons - https://phabricator.wikimedia.org/T247454 (10jijiki) [16:11:51] 10Traffic, 10SRE, 10Upstream: Wikipedia on flow with no http request, still responds with a Bad Request 400 - https://phabricator.wikimedia.org/T323263 (10Vgutierrez) 05Open→03Stalled reported to upstream in https://github.com/haproxy/haproxy/issues/1934 [16:40:44] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqsin, 10Wikimedia-Incident: asw1-eqsin: VC mastership change - https://phabricator.wikimedia.org/T323094 (10ayounsi) The analysis of the core dump by JTAC showed that we were victim of this bug https://prsearch.juniper.net/problemreport/PR1080132 Even... [16:59:15] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Create mw-web helmfile deployment - https://phabricator.wikimedia.org/T321900 (10Clement_Goubert) 05In progress→03Resolved [16:59:28] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 3 others: Deploy mediawiki kubernetes services - https://phabricator.wikimedia.org/T321786 (10Clement_Goubert) [16:59:37] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Create mw-jobrunner helmfile deployment - https://phabricator.wikimedia.org/T321897 (10Clement_Goubert) 05In progress→03Resolved [16:59:49] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 3 others: Deploy mediawiki kubernetes services - https://phabricator.wikimedia.org/T321786 (10Clement_Goubert) [17:00:09] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 3 others: Deploy mediawiki kubernetes services - https://phabricator.wikimedia.org/T321786 (10Clement_Goubert) [17:00:19] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Create mw-api-ext helmfile deployment - https://phabricator.wikimedia.org/T321896 (10Clement_Goubert) 05In progress→03Resolved [17:00:30] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Create mw-api-int helmfile deployment - https://phabricator.wikimedia.org/T321895 (10Clement_Goubert) 05In progress→03Resolved [17:00:42] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 3 others: Deploy mediawiki kubernetes services - https://phabricator.wikimedia.org/T321786 (10Clement_Goubert) [17:00:58] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Stop spamming SAL with helmfile on scap deployments - https://phabricator.wikimedia.org/T323296 (10Clement_Goubert) [17:01:47] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Stop spamming SAL with helmfile on scap deployments - https://phabricator.wikimedia.org/T323296 (10Clement_Goubert) 05Open→03In progress p:05Triage→03Medium [17:01:59] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 3 others: Deploy mediawiki kubernetes services - https://phabricator.wikimedia.org/T321786 (10Clement_Goubert) [18:27:27] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 3 others: Stop spamming SAL with helmfile on scap deployments - https://phabricator.wikimedia.org/T323296 (10JMeybohm) helmfile_log_sal has support for that already: ` # Allow to explicitely suppress logging to SAL SUPPRESS_SAL=${SUPPRESS_SAL:-false} ` [20:13:31] 10HTTPS, 10SRE, 10Wikidata, 10wdwb-tech, 10wikiba.se website: Set HSTS on wikiba.se (force HTTPS) - https://phabricator.wikimedia.org/T232246 (10Dzahn) This is a task for WMDE but for WMF SRE anymore. wikiba.se is controlled by WMDE, not WMF now. [20:13:41] 10HTTPS, 10SRE, 10Wikidata, 10wdwb-tech, 10wikiba.se website: Set HSTS on wikiba.se (force HTTPS) - https://phabricator.wikimedia.org/T232246 (10Dzahn) also see https://gerrit.wikimedia.org/r/c/operations/puppet/+/858408/ [20:14:30] 10HTTPS, 10SRE, 10Wikidata, 10wdwb-tech, 10wikiba.se website: Set HSTS on wikiba.se (force HTTPS) - https://phabricator.wikimedia.org/T232246 (10Dzahn) @Addshore Is it really "External Realm" anymore? [20:14:44] 10HTTPS, 10Wikidata, 10wdwb-tech, 10wikiba.se website: Set HSTS on wikiba.se (force HTTPS) - https://phabricator.wikimedia.org/T232246 (10Dzahn) [20:24:56] 10HTTPS, 10Wikidata, 10wdwb-tech, 10wikiba.se website: Set HSTS on wikiba.se (force HTTPS) - https://phabricator.wikimedia.org/T232246 (10Addshore) >>! In T232246#8403811, @Dzahn wrote: > @Addshore Is it really "External Realm" anymore? Hey hey, this is external to the wikibase and wikidata development te... [20:26:28] 10HTTPS, 10Wikidata, 10wdwb-tech, 10wikiba.se website: Set HSTS on wikiba.se (force HTTPS) - https://phabricator.wikimedia.org/T232246 (10Dzahn) @Addshore Thank you! I wasn't sure if WMDE IT uses Phabricator. So that is appreciated. :) [20:34:17] 10HTTPS, 10Wikidata, 10wdwb-tech, 10wikiba.se website: Set HSTS on wikiba.se (force HTTPS) - https://phabricator.wikimedia.org/T232246 (10WMDE-leszek) hello @Alexander-Finger, subscribing you to this ticket, should you want to revise HSTS/HTTPS practices for WMDE run websites at some point. [21:07:52] 10Traffic, 10SRE, 10Patch-For-Review: ATS should alert if the number of total or active connections reached maximum - https://phabricator.wikimedia.org/T292815 (10BCornwall) 05Open→03Resolved The new patch which was just deployed addresses all these concerns. I'll close the ticket but please feel free to...