[02:56:14] PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1385 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [03:01:14] PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1685 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [03:06:14] PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1985 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [03:11:14] RECOVERY - check_mysql on frdb1006 is OK: Uptime: 787168 Threads: 14 Questions: 154812338 Slow queries: 2013691 Opens: 3198 Open tables: 1743 Queries per second avg: 196.670 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [06:08:20] PROBLEM - check_load on frav1003 is CRITICAL: CRITICAL - load average: 40.44, 29.28, 15.19 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_load [07:08:20] RECOVERY - check_load on frav1003 is OK: OK - load average: 3.41, 3.31, 4.92 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_load [08:15:10] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11670 MB (94% inode=98%): /var 2112 MB (7% inode=99%): /srv 96909 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 14598 MB (55% [08:15:10] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): /run/user/33 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [08:20:10] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11669 MB (94% inode=98%): /var 1034 MB (3% inode=99%): /srv 96909 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 14588 MB (54% [08:20:10] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [08:25:10] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11669 MB (94% inode=98%): /var 0 MB (0% inode=99%): /srv 96909 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 14569 MB (54% ino [08:25:10] /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [08:30:10] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11673 MB (94% inode=98%): /var 0 MB (0% inode=99%): /srv 96909 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 14558 MB (54% ino [08:30:10] /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): /run/user/33 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [08:35:10] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11673 MB (94% inode=98%): /var 0 MB (0% inode=99%): /srv 96909 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 14531 MB (54% ino [08:35:10] /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [08:40:10] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11671 MB (94% inode=98%): /var 0 MB (0% inode=99%): /srv 96909 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 14617 MB (55% ino [08:40:10] /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [08:45:10] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11669 MB (94% inode=98%): /var 0 MB (0% inode=99%): /srv 96909 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 14597 MB (55% ino [08:45:10] /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): /run/user/33 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [08:50:10] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11669 MB (94% inode=98%): /var 0 MB (0% inode=99%): /srv 96909 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 14571 MB (54% ino [08:50:10] /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [08:55:10] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11669 MB (94% inode=98%): /var 0 MB (0% inode=99%): /srv 96909 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 14552 MB (54% ino [08:55:10] /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [09:00:10] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11669 MB (94% inode=98%): /var 0 MB (0% inode=99%): /srv 96909 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 14533 MB (54% ino [09:00:10] /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): /run/user/33 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [09:05:10] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11669 MB (94% inode=98%): /var 0 MB (0% inode=99%): /srv 96900 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 14703 MB (55% ino [09:05:10] /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [09:10:10] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11669 MB (94% inode=98%): /var 0 MB (0% inode=99%): /srv 96907 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 14700 MB (55% ino [09:10:10] /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [09:15:10] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11669 MB (94% inode=98%): /var 0 MB (0% inode=99%): /srv 96907 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 14689 MB (55% ino [09:15:10] /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): /run/user/33 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [09:20:10] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11669 MB (94% inode=98%): /var 0 MB (0% inode=99%): /srv 96907 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 14671 MB (55% ino [09:20:10] /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [09:25:10] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11669 MB (94% inode=98%): /var 0 MB (0% inode=99%): /srv 96907 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 14764 MB (55% ino [09:25:10] /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [09:30:10] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11673 MB (94% inode=98%): /var 0 MB (0% inode=99%): /srv 96895 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 14746 MB (55% ino [09:30:10] /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): /run/user/33 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [09:35:10] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11673 MB (94% inode=98%): /var 0 MB (0% inode=99%): /srv 96895 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 14709 MB (55% ino [09:35:10] /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [09:40:10] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11670 MB (94% inode=98%): /var 0 MB (0% inode=99%): /srv 96895 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 14714 MB (55% ino [09:40:10] /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [09:45:10] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11670 MB (94% inode=98%): /var 0 MB (0% inode=99%): /srv 96895 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 14696 MB (55% ino [09:45:10] /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): /run/user/33 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [09:50:10] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11669 MB (94% inode=98%): /var 0 MB (0% inode=99%): /srv 96895 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 14670 MB (55% ino [09:50:10] /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [09:55:10] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11669 MB (94% inode=98%): /var 0 MB (0% inode=99%): /srv 96895 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 14659 MB (55% ino [09:55:10] /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [10:00:10] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11669 MB (94% inode=98%): /var 0 MB (0% inode=99%): /srv 96895 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 14737 MB (55% ino [10:00:10] /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): /run/user/33 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [10:05:10] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11669 MB (94% inode=98%): /var 0 MB (0% inode=99%): /srv 96893 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 17545 MB (66% ino [10:05:10] /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): /run/user/571 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [10:10:10] RECOVERY - check_disk on civi1002 is OK: DISK OK - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11669 MB (94% inode=98%): /var 25845 MB (97% inode=99%): /srv 71047 MB (24% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 17638 MB (66% inode=99% [10:10:10] og/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): /run/user/571 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [10:48:06] 10fundraising-tech-ops: Fix disk capacity issues on civi1002 - https://phabricator.wikimedia.org/T343071 (10Jgreen) [16:29:36] 10fundraising-tech-ops: Fix disk capacity issues on civi1002 - https://phabricator.wikimedia.org/T343071 (10Jgreen) [16:37:27] 10Fundraising-Backlog, 10fundraising-tech-ops: Process-control fails to handle a zero-byte job status file - https://phabricator.wikimedia.org/T343096 (10Jgreen) [16:40:01] 10Fundraising-Backlog, 10fundraising-tech-ops: Process-control fails to handle a zero-byte job status file - https://phabricator.wikimedia.org/T343096 (10Jgreen) [18:39:36] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+2] Replace deprecated MWException [extensions/CentralNotice] (wmf_deploy) - 10https://gerrit.wikimedia.org/r/928607 (https://phabricator.wikimedia.org/T328220) (owner: 10Daimona Eaytoy) [18:42:11] (03Merged) 10jenkins-bot: Replace deprecated MWException [extensions/CentralNotice] (wmf_deploy) - 10https://gerrit.wikimedia.org/r/928607 (https://phabricator.wikimedia.org/T328220) (owner: 10Daimona Eaytoy)