[06:23:56] 10netops, 10Infrastructure-Foundations: cr2-eqdfw: PEM 1 Not Powered - https://phabricator.wikimedia.org/T289028 (10ayounsi) p:05Triage→03High [06:24:14] 10netops, 10Infrastructure-Foundations, 10ops-eqdfw: cr2-eqdfw: PEM 1 Not Powered - https://phabricator.wikimedia.org/T289028 (10ayounsi) [08:07:58] 10SRE-tools, 10DBA, 10Infrastructure-Foundations, 10Spicerack, 10Datacenter-Switchover: switchdc should verify active/active DBs are read-write in both datacenters - https://phabricator.wikimedia.org/T287129 (10LSobanski) @Legoktm since we are getting closer to the switch back, is this task a requirement... [09:37:54] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE: (Need By: TBD) rack/setup/install atlas-codfw.wikimedia.org - https://phabricator.wikimedia.org/T273114 (10cmooney) @robh no problem I will document the process there once it is fully clear to me. Update for now is that RIPE have gotten back to me a... [10:34:23] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqdfw: cr2-eqdfw: PEM 1 Not Powered - https://phabricator.wikimedia.org/T289028 (10ayounsi) 05Open→03Resolved Nevermind. It was an Equinix maintenance, it's now back to normal. [10:38:55] XioNox: RIPE came back with the image file download for our replacement anchor in codfw. [10:39:12] I'm downloading it here now to my own machine, I wonder is there somewhere we could store this in case we need it again? [10:46:19] topranks: apt1001:/srv/junos has all the Junos images, so maybe /srv/ripe-atlas ? dunno how/if it's puppetized though [10:46:51] Daniel might know more [10:47:02] or Moritz [11:25:31] the old ones are in /srv/firmware [12:09:03] ok cool thanks. [12:25:06] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE: (Need By: TBD) rack/setup/install atlas-codfw.wikimedia.org - https://phabricator.wikimedia.org/T273114 (10cmooney) RIPE have emailed back to confirm registration and provided image download ` From: RIPE Atlas (no reply) [mailto:no-reply@ripe.net] S... [12:25:28] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE: (Need By: TBD) rack/setup/install atlas-codfw.wikimedia.org - https://phabricator.wikimedia.org/T273114 (10cmooney) [14:35:36] 10SRE-tools, 10Infrastructure-Foundations, 10cloud-services-team (Kanban): Cookbooks repository: avoid stale code in master branch - https://phabricator.wikimedia.org/T287465 (10joanna_borun) Thank you all for such an active conversation. Myself and Nicholas would like to follow up on this thread and sum up... [16:02:37] 10netops, 10Infrastructure-Foundations, 10SRE, 10decommission-hardware, 10ops-eqiad: Decommission asw-c-eqiad - https://phabricator.wikimedia.org/T208734 (10Cmjohnson) 05Open→03Resolved Removed the serial console port (port 9) for old asw-c1.resolving [16:11:01] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Switch failure: asw2-a8-eqiad Aug 13th 2021 - https://phabricator.wikimedia.org/T288834 (10Cmjohnson) @ayounsi Did the fiber swap fix the issue or do we need to change optics? [17:04:30] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Switch failure: asw2-a8-eqiad Aug 13th 2021 - https://phabricator.wikimedia.org/T288834 (10cmooney) Hey @Cmjohnson. Link has been clean since, but it often goes a few weeks without issues, so we probably want to wait just to be sure. So far so g... [17:12:39] 10Puppet, 10Infrastructure-Foundations, 10Wikidata, 10wdwb-tech, and 2 others: Migrate wikibase-dispatch-changes crons to systemd timers - https://phabricator.wikimedia.org/T288175 (10RLazarus) [18:06:12] 10Puppet, 10Infrastructure-Foundations, 10Wikidata, 10wdwb-tech, and 2 others: Migrate wikibase-dispatch-changes crons to systemd timers - https://phabricator.wikimedia.org/T288175 (10Ladsgroup) Maybe after a while we should delete the old logs, I can put a reminder to delete them in three months. [19:12:31] 10SRE-tools, 10SRE, 10Spicerack, 10serviceops, 10Datacenter-Switchover: Clean up cron-specific elements of switchdc cookbooks - https://phabricator.wikimedia.org/T289078 (10RLazarus) [19:20:17] 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10Spicerack, and 2 others: Clean up cron-specific elements of switchdc cookbooks - https://phabricator.wikimedia.org/T289078 (10RLazarus)