[08:22:22] 10Traffic, 10SRE: Cannot edit wikipedia from my work computer - https://phabricator.wikimedia.org/T356799 (10Rijikk) [08:43:15] 10Traffic, 10SRE: Cannot edit wikipedia from my work computer - https://phabricator.wikimedia.org/T356799 (10taavi) Please include the details under "If you report this error to the Wikimedia System Administrators, please include the details below." in the footer. [08:50:56] 10Traffic, 10SRE: Cannot edit wikipedia from my work computer - https://phabricator.wikimedia.org/T356799 (10Rijikk) Sorry, I am not sure where is the footer in the report and what details should I include there. It is my first time on Phabricator... [10:04:44] 10Traffic, 10SRE, 10Patch-For-Review: Lower geodns TTLs from 600 (10min) to 300 (5min) - https://phabricator.wikimedia.org/T140365 (10akosiaris) Speaking with a appservers/wikikube clusters hat on, we don't see any problems with the lowering of the `dyna.wikimedia.org` from 10 minutes to 5 minutes. With an... [10:15:45] 10netops, 10Infrastructure-Foundations, 10sre-alert-triage: Alert in need of triage: BGP status (instance cr1-drmrs) - https://phabricator.wikimedia.org/T357389 (10LSobanski) [11:12:48] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, 10Release-Engineering-Team (Seen): Create parsoid mediawiki deployment - https://phabricator.wikimedia.org/T357392 (10akosiaris) [11:47:33] 10Traffic, 10Data-Persistence, 10SRE, 10SRE-swift-storage, and 3 others: Change default image thumbnail size - https://phabricator.wikimedia.org/T355914 (10ovasileva) Another step in the process here would be to look at how the change would visually affect current layouts on enwiki and across wikis, as wel... [12:52:08] 10Traffic, 10Data-Persistence, 10SRE, 10SRE-swift-storage, and 4 others: Change default image thumbnail size - https://phabricator.wikimedia.org/T355914 (10RHo) [12:53:00] 10Traffic, 10Data-Persistence, 10SRE, 10SRE-swift-storage, and 4 others: Change default image thumbnail size - https://phabricator.wikimedia.org/T355914 (10RHo) @DTorsani-WMF and @JScherer-WMF - per @Ladsgroup suggestions, wondering if you both can offer any ux recommendations for recommended thumbnail siz... [15:02:55] topranks: cp2027 and cp2028 are ready for maintenance. Thanks for your work! [15:05:26] Brett: that’s great thanks! [15:27:06] 10Traffic, 10Infrastructure-Foundations, 10SRE, 10vm-requests, 10Patch-For-Review: eqiad: 1 VM request for ncmonitor - https://phabricator.wikimedia.org/T356710 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host ncmonitor1001.eqiad.wmnet with OS bookworm [15:35:05] 10Traffic, 10Content-Transform-Team, 10MW-on-K8s, 10SRE, and 2 others: Create parsoid mediawiki deployment - https://phabricator.wikimedia.org/T357392 (10Jdforrester-WMF) [15:40:47] 10Traffic, 10Infrastructure-Foundations, 10SRE, 10vm-requests: eqiad: 1 VM request for ncmonitor - https://phabricator.wikimedia.org/T356710 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host ncmonitor1001.eqiad.wmnet with OS bookworm executed with errors: - ncm... [15:59:02] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10klausman) [16:02:38] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 2 others: Migrate servers in codfw rack A4 from asw-a4-codfw to lsw1-a4-codfw - https://phabricator.wikimedia.org/T355863 (10Jhancock.wm) Forgot to update earlier. Rack is physically ready [16:04:52] 10Acme-chief, 10Traffic: Puppetize deployment of ncmonitor - https://phabricator.wikimedia.org/T355190 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host ncmonitor1001.eqiad.wmnet with OS bookworm [16:06:02] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 2 others: Migrate servers in codfw rack A4 from asw-a4-codfw to lsw1-a4-codfw - https://phabricator.wikimedia.org/T355863 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=349240a0-30c3-4371-9418-7f1f46072237) set by cmooney@cumin1... [16:09:26] 10Domains, 10SRE: Request donatewiki redirect - https://phabricator.wikimedia.org/T357436 (10AKanji-WMF) [16:21:39] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A5 from asw-a5-codfw to lsw1-a5-codfw - https://phabricator.wikimedia.org/T355864 (10Jhancock.wm) rack is physically ready for tomorrow. [16:23:13] brett: those cp hosts are now moved if you want to repool them [16:49:12] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 2 others: Migrate servers in codfw rack A4 from asw-a4-codfw to lsw1-a4-codfw - https://phabricator.wikimedia.org/T355863 (10cmooney) All work completed, no issues to report :) [16:49:30] 10Acme-chief, 10Traffic: Puppetize deployment of ncmonitor - https://phabricator.wikimedia.org/T355190 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host ncmonitor1001.eqiad.wmnet with OS bookworm executed with errors: - ncmonitor1001 (**FAIL**) - Removed from Pup... [16:52:16] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 2 others: Migrate servers in codfw rack A4 from asw-a4-codfw to lsw1-a4-codfw - https://phabricator.wikimedia.org/T355863 (10MatthewVernon) Swift looks happy, thanks :) [16:55:14] 10Traffic, 10Data-Persistence, 10SRE, 10SRE-swift-storage, and 5 others: Change default image thumbnail size - https://phabricator.wikimedia.org/T355914 (10Ladsgroup) @RHo Thank you! Something to consider is also looking at thumbnail sizes in other areas. Mostly notably: - In categories and galleries wher... [17:09:39] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 2 others: Migrate servers in codfw rack A4 from asw-a4-codfw to lsw1-a4-codfw - https://phabricator.wikimedia.org/T355863 (10cmooney) >>! In T355863#9538876, @MatthewVernon wrote: > Swift looks happy, thanks :) great, thanks for the update! [17:55:55] 10Traffic, 10Automoderator, 10Data Products, 10Product-Analytics, and 4 others: Add revision ID to X-Analytics header - https://phabricator.wikimedia.org/T346350 (10jsn.sherman) >>! In T346350#9508853, @phuedx wrote: > Re-opening this while we wait for the patch to be deployed and for the key-value pair to... [18:08:40] (VarnishHighThreadCount) firing: Varnish's thread count on cp5022:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=99&var-site=eqsin&var-instance=cp5022 - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [18:13:40] (VarnishHighThreadCount) firing: (8) Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [18:23:40] (VarnishHighThreadCount) firing: (15) Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [18:28:40] (VarnishHighThreadCount) firing: (16) Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [18:31:03] 10Traffic, 10Automoderator, 10Data Products, 10Product-Analytics, and 3 others: Add revision ID to X-Analytics header - https://phabricator.wikimedia.org/T346350 (10mpopov) `lang=sql SELECT normalized_host.project, namespace_id IS NULL AS ns_id_is_null, element_at(x_analytics_map, 'ns') IS NULL AS x_... [18:37:01] 10Domains, 10Fundraising-Backlog, 10SRE: Request donatewiki redirect - https://phabricator.wikimedia.org/T357436 (10ssingh) [18:43:40] (VarnishHighThreadCount) firing: (9) Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [18:48:41] (VarnishHighThreadCount) resolved: (8) Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [19:17:15] 10Traffic, 10Automoderator, 10Data Products, 10Product-Analytics, and 3 others: Add revision ID to X-Analytics header - https://phabricator.wikimedia.org/T346350 (10jsn.sherman) >>! In T346350#9539289, @mpopov wrote: > `lang=sql > SELECT > normalized_host.project, > namespace_id IS NULL AS ns_id_is_nul... [19:38:07] 10Domains, 10Fundraising-Backlog, 10SRE, 10serviceops: Request donatewiki redirect - https://phabricator.wikimedia.org/T357436 (10RLazarus) Hi from Service Ops SRE! @AKanji-WMF How long would you like the redirect to stay active? Adding @Jgreen and @Dwisehaupt from FR-Tech SRE, as I'm not sure where we'v... [20:27:11] 10Traffic, 10SRE: Lower geodns TTLs from 600 (10min) to 300 (5min) - https://phabricator.wikimedia.org/T140365 (10ssingh) 05Open→03Resolved We have rolled this out today. For a complete list of domains affected, see the commit above. [20:27:53] 10Traffic, 10Patch-For-Review: Restarting fifo-log-demux should not restart nginx - https://phabricator.wikimedia.org/T355905 (10CodeReviewBot) brett opened https://gitlab.wikimedia.org/repos/sre/fifo-log-demux/-/merge_requests/1 fifo-log-demux: Exit(70) on unable to spawn server [22:27:45] 10Traffic, 10Security-Team, 10ContentSecurityPolicy: Stop sending X-Webkit-CSP and X-Webkit-CSP-Report-Only headers - https://phabricator.wikimedia.org/T357479 (10TheDJ) [22:39:30] 10Traffic, 10Security-Team, 10ContentSecurityPolicy, 10Patch-For-Review: Stop sending X-Webkit-CSP and X-Webkit-CSP-Report-Only headers - https://phabricator.wikimedia.org/T357479 (10TheDJ) Not really sure yet what to do with ./modules/profile/files/logstash/tests/node.yaml: "x-webkit-csp": "de...