[06:00:52] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE, 13Patch-For-Review: esams text cp nvme upgrade - https://phabricator.wikimedia.org/T360430#9664389 (10Fabfur) ESAMS DC started depooling @05:58UTC [07:14:52] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE, 13Patch-For-Review: esams text cp nvme upgrade - https://phabricator.wikimedia.org/T360430#9664436 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=e71791c7-a0fa-43b5-81ae-e92b275e5cc3) set by fabfur@cumin1002 for 1 day, 0:00:00 on 8 host(s) and... [07:24:47] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE, 13Patch-For-Review: esams text cp nvme upgrade - https://phabricator.wikimedia.org/T360430#9664439 (10Fabfur) [10:15:21] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE, 13Patch-For-Review: esams text cp nvme upgrade - https://phabricator.wikimedia.org/T360430#9664707 (10Fabfur) [10:34:03] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE, 13Patch-For-Review: esams text cp nvme upgrade - https://phabricator.wikimedia.org/T360430#9664786 (10RobH) ESAMS remote hands began hands on work at 11:10 CET and it is now ongoing. [11:50:24] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE, 13Patch-For-Review: esams text cp nvme upgrade - https://phabricator.wikimedia.org/T360430#9664951 (10Fabfur) [12:00:54] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE, 13Patch-For-Review: esams text cp nvme upgrade - https://phabricator.wikimedia.org/T360430#9664975 (10Fabfur) [12:12:42] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE, 13Patch-For-Review: esams text cp nvme upgrade - https://phabricator.wikimedia.org/T360430#9665004 (10Fabfur) [14:21:38] 06Traffic, 06collaboration-services, 06Data-Persistence, 06DC-Ops, and 5 others: ☂️ Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T357547#9665563 (10ops-monitoring-bot) jiji@cumin1002 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in codfw:... [14:41:09] 06Traffic, 06collaboration-services, 06Data-Persistence, 06DC-Ops, and 5 others: ☂️ Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T357547#9665627 (10ops-monitoring-bot) jiji@cumin1002 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in codfw:... [14:56:02] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE, 13Patch-For-Review: esams text cp nvme upgrade - https://phabricator.wikimedia.org/T360430#9665702 (10Fabfur) esams has been repooled at 12:15UTC [14:57:01] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE, 13Patch-For-Review: esams text cp nvme upgrade - https://phabricator.wikimedia.org/T360430#9665701 (10Fabfur) [15:00:06] 06Traffic, 06collaboration-services, 06Data-Persistence, 06DC-Ops, and 5 others: ☂️ Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T357547#9665753 (10jijiki) [15:30:05] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE, 13Patch-For-Review: esams text cp nvme upgrade - https://phabricator.wikimedia.org/T360430#9665926 (10RobH) a:05RobH→03Fabfur Reassigning from myself over to @fabfur for reimaging at #traffic's leisure. [15:47:10] 06Traffic, 06collaboration-services, 06Data-Persistence, 06DC-Ops, and 5 others: 14☂️ Northward Datacentre Switchover (March 2024) - 14https://phabricator.wikimedia.org/T357547#9665972 (10jijiki) 05Open→03Resolved a:03jijiki 14Switchover is done, it is Day 8, and we are back to Multi-DC. Thank y... [16:42:38] hello :) I've an acmechief question for you in -sre :) [17:21:36] I'm not working? :) [17:25:35] Oh damn.. I thought this was a privmsg :_) [17:25:49] I don't even know how to irc anymore [17:39:41] yep, was not expecting you at all here, sorry for the involuntary ping man ;) [18:42:18] vg: only you can answer that question on why you are here :P [19:26:29] 06Traffic, 10PageImages, 10WMF-General-or-Unknown, 07Regression: 14Miniature images from og:image not loading in social media links - 14https://phabricator.wikimedia.org/T359413#9666868 (10CDanis) 14Hello, This has been fixed with [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/1015051 | this... [19:26:30] 06Traffic, 10PageImages, 10WMF-General-or-Unknown, 07Regression: 14Miniature images from og:image not loading in social media links - 14https://phabricator.wikimedia.org/T359413#9666870 (10CDanis) 05Open→03Resolved a:03CDanis [19:33:57] 06Traffic, 10PageImages, 10WMF-General-or-Unknown, 07Regression: 14Miniature images from og:image not loading in social media links - 14https://phabricator.wikimedia.org/T359413#9666890 (10Theklan) 14Thanks! I assume they were trying to do evil! [22:15:22] 10netops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Decom asw-a-codfw switch stack - https://phabricator.wikimedia.org/T358244#9667344 (10Papaul)