Long-running WSGI processes on keystone
Seeing messages like this sporadically in the keystone endpoint:
2018-02-20 15:04:01.688269 Timeout when reading response headers from daemon process 'keystone-admin': /usr/bin/keystone-wsgi-admin
2018-02-20 15:04:13.683543 Timeout when reading response headers from daemon process 'keystone-admin': /usr/bin/keystone-wsgi-admin
2018-02-20 15:04:25.651847 Timeout when reading response headers from daemon process 'keystone-admin': /usr/bin/keystone-wsgi-admin
2018-02-20 15:04:37.655803 Timeout when reading response headers from daemon process 'keystone-admin': /usr/bin/keystone-wsgi-admin
2018-02-20 15:04:49.741519 Timeout when reading response headers from daemon process 'keystone-admin': /usr/bin/keystone-wsgi-admin
2018-02-20 15:05:00.927068 mod_wsgi (pid=19048): Exception occurred processing WSGI script '/usr/bin/keystone-wsgi-admin'.
2018-02-20 15:05:00.927246 IOError: failed to write data
2018-02-20 15:05:01.741614 Timeout when reading response headers from daemon process 'keystone-admin': /usr/bin/keystone-wsgi-admin
2018-02-20 15:06:01.388416 mod_wsgi (pid=19047): Exception occurred processing WSGI script '/usr/bin/keystone-wsgi-admin'.
2018-02-20 15:06:01.388516 IOError: failed to write data
2018-02-20 15:06:17.229608 mod_wsgi (pid=19049): Exception occurred processing WSGI script '/usr/bin/keystone-wsgi-admin'.
2018-02-20 15:06:17.229731 IOError: failed to write data
2018-02-20 15:06:22.375812 mod_wsgi (pid=19046): Exception occurred processing WSGI script '/usr/bin/keystone-wsgi-admin'.
2018-02-20 15:06:22.376036 IOError: failed to write data
2018-02-20 15:06:26.457472 mod_wsgi (pid=19050): Exception occurred processing WSGI script '/usr/bin/keystone-wsgi-admin'.
2018-02-20 15:06:26.457683 IOError: failed to write data
2018-02-20 15:07:27.366785 mod_wsgi (pid=19041): Exception occurred processing WSGI script '/usr/bin/keystone-wsgi-public'.
2018-02-20 15:07:27.367624 IOError: failed to write data
2018-02-20 15:08:27.770928 mod_wsgi (pid=19042): Exception occurred processing WSGI script '/usr/bin/keystone-wsgi-public'.
2018-02-20 15:08:27.771082 IOError: failed to write data
2018-02-20 15:09:36.917050 mod_wsgi (pid=19043): Exception occurred processing WSGI script '/usr/bin/keystone-wsgi-public'.
2018-02-20 15:09:36.917224 IOError: failed to write data
2018-02-20 15:28:11.820002 Timeout when reading response headers from daemon process 'keystone-admin': /usr/bin/keystone-wsgi-admin
2018-02-20 15:28:11.950547 Timeout when reading response headers from daemon process 'keystone-admin': /usr/bin/keystone-wsgi-admin
2018-02-20 15:29:12.332066 Timeout when reading response headers from daemon process 'keystone-admin': /usr/bin/keystone-wsgi-admin
2018-02-20 15:29:12.373945 Timeout when reading response headers from daemon process 'keystone-admin': /usr/bin/keystone-wsgi-admin
2018-02-20 15:30:13.355878 Timeout when reading response headers from daemon process 'keystone-admin': /usr/bin/keystone-wsgi-admin
2018-02-20 15:30:13.460062 Timeout when reading response headers from daemon process 'keystone-admin': /usr/bin/keystone-wsgi-admin
2018-02-20 15:30:14.001521 Timeout when reading response headers from daemon process 'keystone-admin': /usr/bin/keystone-wsgi-admin
2018-02-20 15:30:18.737638 Timeout when reading response headers from daemon process 'keystone-admin': /usr/bin/keystone-wsgi-admin
2018-02-20 15:30:24.223370 Timeout when reading response headers from daemon process 'keystone-admin': /usr/bin/keystone-wsgi-admin
2018-02-20 15:30:25.989679 Timeout when reading response headers from daemon process 'keystone-admin': /usr/bin/keystone-wsgi-admin
These threads suggest a few different things:
https://bugs.launchpad.net/mos/+bug/1566802
https://bugs.launchpad.net/mos/+bug/1583095
We already have the application group set to global. I have temporarily disabled rsyslog on keystone to observe the behavior. Anyone else have any ideas while we see how that works out?