[02:47:07] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:rack/setup/install 4 parsoid hosts - https://phabricator.wikimedia.org/T349874 (10VRiley-WMF) kubernetes1059 Rack: E 1 Position: U42 CableID: 5-02562 Port: 41 kubernetes1060 Rack: E 2 Position: U 43 CableID: 5-02561 Port: 44 kubernetes1061 Rack: E 3 Positio... [02:47:44] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:rack/setup/install 4 parsoid hosts - https://phabricator.wikimedia.org/T349874 (10VRiley-WMF) [09:33:07] 10serviceops, 10Data-Persistence, 10Infrastructure-Foundations, 10Traffic, 10conftool: Switch conftool to use the version 3 etcd datastore - https://phabricator.wikimedia.org/T350565 (10Volans) Untagged sre-tools and spicerack as I've created the dedicated sub-tasks for them. [09:49:05] 10serviceops, 10CX-cxserver, 10Citoid, 10Content-Transform-Team-WIP, and 10 others: Migrate node-based services in production to node18 - https://phabricator.wikimedia.org/T349118 (10elukey) I filed https://gerrit.wikimedia.org/r/c/mediawiki/services/recommendation-api/+/977751 for the recommendation-api s... [09:52:20] 10serviceops, 10MW-on-K8s, 10Release-Engineering-Team, 10Wikimedia-production-error: PHP Warning: geoip_country_code_by_name(): Required database not available at /usr/share/GeoIP/GeoIP.dat. - https://phabricator.wikimedia.org/T352156 (10hashar) p:05Triage→03Unbreak! [09:57:06] the MediaWiki on Kubernetes images are missing `/usr/share/GeoIP/GeoIP.dat` somehow [09:57:19] or refer to the wrong path in the config, I have investigated [09:57:49] 10serviceops, 10Data-Platform-SRE, 10Discovery-Search (Current work): Enable mediawiki.cirrussearch.page_rerender.v1 on all public wikis - https://phabricator.wikimedia.org/T351503 (10pfischer) a:03pfischer [10:01:30] 10serviceops, 10MW-on-K8s, 10Release-Engineering-Team, 10Wikimedia-production-error: PHP Warning: geoip_country_code_by_name(): Required database not available at /usr/share/GeoIP/GeoIP.dat. - https://phabricator.wikimedia.org/T352156 (10hashar) [10:54:53] 10serviceops, 10Data-Platform-SRE, 10Discovery-Search (Current work): Enable mediawiki.cirrussearch.page_rerender.v1 on all public wikis - https://phabricator.wikimedia.org/T351503 (10pfischer) @elukey, we would like to start populating this kafka topic on kafka-main. Enabling `page_rerender` is the last mis... [11:18:19] 10serviceops, 10MW-on-K8s, 10Release-Engineering-Team, 10Wikimedia-production-error: PHP Warning: geoip_country_code_by_name(): Required database not available at /usr/share/GeoIP/GeoIP.dat. - https://phabricator.wikimedia.org/T352156 (10Joe) The problem is that the new kubernetes nodes don't have a copy o... [11:24:09] 10serviceops, 10MW-on-K8s, 10MediaWiki-extensions-WikimediaEvents, 10Release-Engineering-Team, 10Wikimedia-production-error: PHP Warning: geoip_country_code_by_name(): Required database not available at /usr/share/GeoIP/GeoIP.dat. - https://phabricator.wikimedia.org/T352156 (10taavi) [11:25:28] 10serviceops, 10MW-on-K8s, 10MediaWiki-extensions-WikimediaEvents, 10Release-Engineering-Team, 10Wikimedia-production-error: PHP Warning: geoip_country_code_by_name(): Required database not available at /usr/share/GeoIP/GeoIP.dat. - https://phabricator.wikimedia.org/T352156 (10Joe) codesearch also says t... [12:14:57] 10serviceops, 10MW-on-K8s, 10MediaWiki-extensions-WikimediaEvents, 10Release-Engineering-Team, and 2 others: PHP Warning: geoip_country_code_by_name(): Required database not available at /usr/share/GeoIP/GeoIP.dat. - https://phabricator.wikimedia.org/T352156 (10jijiki) @jbond @Muehlenhoff It appears that w... [12:22:13] 10serviceops, 10MW-on-K8s, 10MediaWiki-extensions-WikimediaEvents, 10Release-Engineering-Team, and 2 others: PHP Warning: geoip_country_code_by_name(): Required database not available at /usr/share/GeoIP/GeoIP.dat. - https://phabricator.wikimedia.org/T352156 (10jbond) >>! In T352156#9363098, @jijiki wrote:... [13:09:40] _joe_: thank you for the GeoIP investigation :) [13:11:50] if I get it right the GeoIP.dat files are the proprietary one which are on the Puppet master and shipped to the kubernetes node. Then I imagine the path bind mounted from the k8s node inside the mediawiki container? [13:17:24] 10serviceops, 10LandingCheck, 10MW-on-K8s, 10MediaWiki-extensions-WikimediaEvents, and 3 others: PHP Warning: geoip_country_code_by_name(): Required database not available at /usr/share/GeoIP/GeoIP.dat. - https://phabricator.wikimedia.org/T352156 (10hashar) [13:18:49] 10serviceops, 10LandingCheck, 10MW-on-K8s, 10MediaWiki-extensions-WikimediaEvents, and 3 others: PHP Warning: geoip_country_code_by_name(): Required database not available at /usr/share/GeoIP/GeoIP.dat. - https://phabricator.wikimedia.org/T352156 (10hashar) >>! In T352156#9363024, @Joe wrote: > The problem... [13:27:03] hashar: exactly [13:27:19] so essentially restore the geoip files and we are set? :) [13:28:59] yes [14:00:01] hnowlan: o/ around? I have a restbase proxy question :D [14:22:54] elukey: hey, whassup? [14:23:28] already solved! [14:23:35] thanks <3 [14:25:49] heh, what was it? :D [14:27:59] so my team is porting a cloud vps Research ml model (python webapp) to Lift Wing, and some calls to /api/rest_v1 were made, so I was a little lost about how to map external to internal URIs [14:28:15] but before you ask, the final solution uses rest-gateway don't worry :D [14:40:41] 10serviceops, 10LandingCheck, 10MW-on-K8s, 10MediaWiki-extensions-WikimediaEvents, and 3 others: PHP Warning: geoip_country_code_by_name(): Required database not available at /usr/share/GeoIP/GeoIP.dat. - https://phabricator.wikimedia.org/T352156 (10Joe) >>! In T352156#9363316, @hashar wrote: >>>! In T3521... [14:41:29] ah heh, nice [14:42:23] 10serviceops, 10LandingCheck, 10MW-on-K8s, 10MediaWiki-extensions-WikimediaEvents, and 3 others: PHP Warning: geoip_country_code_by_name(): Required database not available at /usr/share/GeoIP/GeoIP.dat. - https://phabricator.wikimedia.org/T352156 (10kostajh) https://gerrit.wikimedia.org/r/c/mediawiki/exten... [15:31:01] 10serviceops, 10LandingCheck, 10MW-on-K8s, 10MediaWiki-extensions-WikimediaEvents, and 3 others: PHP Warning: geoip_country_code_by_name(): Required database not available at /usr/share/GeoIP/GeoIP.dat. - https://phabricator.wikimedia.org/T352156 (10jijiki) 05Open→03Resolved a:03jijiki Files copied o... [15:55:52] 10serviceops, 10Machine-Learning-Team, 10Patch-For-Review: Bump istio Docker images to Bookworm - https://phabricator.wikimedia.org/T351933 (10calbon) p:05Triage→03Medium [16:07:20] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:rack/setup/install 4 parsoid hosts - https://phabricator.wikimedia.org/T349874 (10VRiley-WMF) [21:14:02] 10serviceops, 10LandingCheck, 10MW-on-K8s, 10MediaWiki-extensions-WikimediaEvents, and 4 others: PHP Warning: geoip_country_code_by_name(): Required database not available at /usr/share/GeoIP/GeoIP.dat. - https://phabricator.wikimedia.org/T352156 (10hashar) >>! In T352156#9363645, @kostajh wrote: > https:/...