[01:27:37] 10Traffic, 10SRE, 10ops-codfw: codfw: cp2038 Correctable memory error on DIMM A3 - https://phabricator.wikimedia.org/T308459 (10ssingh) [05:12:17] 10netops, 10Infrastructure-Foundations, 10SRE: codfw: Provision a server script can not run without a cable ID" - https://phabricator.wikimedia.org/T308768 (10Marostegui) p:05Triage→03Medium a:03Papaul [05:57:56] (HAProxyEdgeTrafficDrop) firing: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [06:02:56] (HAProxyEdgeTrafficDrop) resolved: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [06:18:23] 10Traffic, 10Beta-Cluster-Infrastructure, 10SRE: Betacommons: 504, Connection Timed Out at 2022-05-02 13:35:16 GMT - https://phabricator.wikimedia.org/T307354 (10Marostegui) 05Open→03Resolved Closing this for now. Reopen if needed. Thanks for reporting! [08:01:49] bblack: to reply to your earlier comments: the patch doesn't change anything on how CI is run. We're already checking out the netbox-data (to let gdnsd perform the includes) in the current setup, see the 'Copying automatically generated zone files under target tree' step in a run of tox (with gdnsd) or tox -- -n (without gdnsd). They both already checkout the NEtbox data. Alternatively you can [08:01:55] set the DNS_INCLUDE_DIR env variable to ... [08:01:57] ... the path of your checkout of the Netbox data to use that instead of a fresh checkout. [13:41:14] 10netops, 10Infrastructure-Foundations, 10SRE: codfw: Provision a server script can not run without a cable ID" - https://phabricator.wikimedia.org/T308768 (10Papaul) @Volans thanks I have another server to install next week i will try and let you know. [13:54:34] 10Traffic, 10SRE, 10ops-codfw: codfw: cp2038 Correctable memory error on DIMM A3 - https://phabricator.wikimedia.org/T308459 (10ssingh) Hi @Papaul: Thanks for letting us know! The host is depooled and downtimed and so please proceed whenever you want. Thanks! [14:59:48] 10Traffic, 10SRE, 10ops-codfw: codfw: cp2038 Correctable memory error on DIMM A3 - https://phabricator.wikimedia.org/T308459 (10Papaul) @ssingh thanks will work on it when back on site next week [15:06:21] volans: ack, thanks, makes sense! I had forgotten that was already in there [15:06:36] np :) [17:58:10] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Finalise design extension of WMCS networks to new cloudsw in Eqiad rows E/F - https://phabricator.wikimedia.org/T304989 (10cmooney) Just a brief update here. I've completed the migration of the existing cloud realm networks configured on c...